Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Routing of parameters in a Pipeline #1096

Open
zbarry opened this issue Feb 13, 2020 · 2 comments
Open

Routing of parameters in a Pipeline #1096

zbarry opened this issue Feb 13, 2020 · 2 comments

Comments

@zbarry
Copy link

@zbarry zbarry commented Feb 13, 2020

Given a Pipeline comprising multiple stages, it seems like the only way to move parameters from stage to stage currently is sequentially, i.e., stage 1 to 2, 2 to 3, etc.

In the case that stage 1 and 2 are both simply for configuration of different types of options, for example, in order for stage 3 to receive the configuration specified in stage 1, stage 2 must contain all those parameters in its definition, as well. This now results in unnecessary coupling between stages that may not need to know anything about the other.

If the Pipeline class had the ability to keep track of all the parameters across all stages and then selectively route them to stages that need them, that would be useful to avoid the coupling problem (and the duplication of code that results).

@jbednar

This comment has been minimized.

Copy link
Member

@jbednar jbednar commented Feb 13, 2020

Sounds useful. Are you imagining having access to the previous stage objects from each subsequent stage (e.g. stage3 could access stage1.param1), or having values from previous stages available in some flat format (e.g. self.previous_params in stage 3 would have a flat dictionary of all previous stage's parameters), or...?

@zbarry

This comment has been minimized.

Copy link
Author

@zbarry zbarry commented Feb 13, 2020

Hmm. If you wanted to avoid clobbering of parameter names, the stage1.param1 formulation makes sense. I'm not sure how you'd then specify in your downstream class that a particular parameter depends on it, though.

Would it be possible to somehow specify a dependson='StageClassName.param_name' keyword argument when defining parameters in the class body of a downstream stage and have the Pipeline construct a nested outputs dictionary of something very roughly like params[stage.__class__.__name__][param_name] = the_param for each parameter deemed an output in the upstream stage object?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.