How do I ensure that my pipeline runs/results are unaffected by changing contents of external config/parameter files? #2605
noam-insait
started this conversation in
Idea
Replies: 1 comment 1 reply
-
Hi @noam-insait!
That's the most likely answer, I don't think Kedro supports changing parameters during the run very well. I'm paging @ankatiyar just in case she can add more from the engineering side of things. |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi Kedro community!
My team and are I trying to create an optimal setup for running experiments in parallel. Concerningly, it appears as though if we are to change the contents of a parameters file (i.e.
conf/local/parameters.yml
) during a run, the results of the run may be affected.For example, let's say I set hyper_tune: False in
parameters.yml
, and run kedro run in the terminal. If I change the text inparameters.yml
to hyper_tune: True (for example, if I am setting up the parameters for my next experiment) before the "training" node begins executing, it appears that Kedro will then read hyper_tune: True. In this example, that would mean that Kedro would execute hyperparameter tuning (despite being instructed not to do so at the beginning of the run).Am I missing something? Is the answer as simple as passing all parameters to the pipeline one time as a whole (i.e. using a before_pipeline_runs hook) rather than to each node?
Beta Was this translation helpful? Give feedback.
All reactions