Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement basic support for pure-Python tools #1091

Merged
merged 5 commits into from Jul 26, 2022
Merged

Conversation

nmoroze
Copy link
Contributor

@nmoroze nmoroze commented Jul 19, 2022

This PR implements support for pure-Python tools, a feature we talked about a while back when discussing how to more cleanly port other flows into SC. With both the ORFS and Caravel flows, we've noticed cases where we need to do some intermediate processing on files being passed around the flow. The existing options we explored each have flaws:

  • Modify pre_process() or post_process() in a tool driver
    • This isn't great since we want to keep these functions as flow-agnostic as possible, so that we can reuse the tool drivers broadly. Forking the tool drivers per-flow isn't a great solution either, since this results in lots of code duplication.
  • Break the run into multiple run() calls that each use a partial steplist, with Python logic in between
    • This is what the Caravel example currently does, but it breaks some of SC's nice features for debugging (steplist is overwritten, flowgraph viz doesn't capture intermediate processing, the intermediate processing results aren't captured in the build directory)

Natively supporting pure Python steps seems like a cleaner solution to this problem. As a basic approach, I changed SC to treat any tool driver that defines a run(chip) function as a pure-Python tool. The SC runtime will then call this function instead of an executable.

The one problem with this implementation is that it introduces a third code path that doesn't include support for some nice features, like output redirection, memory tracking, and task timeouts. If we run into a situation where we want these features, I think we could explore running these tools in a separate process that can then be monitored by the runtime in order to provide this functionality, but this was the simplest first approach.

As a proof-of-concept, I updated the Caravel example to use these features. To validate the changes, I ran the flow before and after and made sure the input DEF to KLayout and that the final DEF passed into KLayout are the same. If you'd like more certainty that the changes don't break anything, I can try re-running the MPW prechecks using the new flow.

Don't try to load run() function when it won't be possible.
@nmoroze nmoroze requested a review from WRansohoff July 19, 2022 19:24
Copy link
Contributor

@WRansohoff WRansohoff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! I really like the approach, and it demonstrably simplifies the Caravel flow's custom Python logic.

@aolofsson aolofsson merged commit 7ce0aea into main Jul 26, 2022
@aolofsson aolofsson deleted the noah/python-tool branch July 26, 2022 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants