-
Notifications
You must be signed in to change notification settings - Fork 897
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable overriding configuration via variable interpolation for all configurations other than just parameters
for OmegaConfigLoader
#2531
Comments
Needs to be completed after #2507 |
There's two (potentially more) solutions to this ticket - Solution 1I have a draft PR with this one - #2931 runtime_params = {"_base_path": "<some_other_path>", "random_state":5} Catalog.yml - cars:
filepath: ${_base_path}
type: pandas.CSVDataSet
_base_path: <some_path> parameters.yml - random_state : 45
split: 0.2
Things to note :
Solution 2Use a similar custom resolver approach to the globals implementation.
cars:
filepath: "${runtime_params:path, <some_default_path>}"
type: pandas.CSVDataSet Comments from @noklam on Slack
Things to note:
❓ If we go with this solution
|
One thought - what if we inject whatever exist in Pro:
Con:
|
I like the resolver solution, but also think it's important to have consistency between how runtime resolution happens for parameters and other config files. So if the decision is to go with the resolver I'd vouch for also implementing that for parameters. |
I really like the resolver approach, because it makes the approach less magic and more declarative with fine grained level. Eventually this will facilitate maitenance* because we can create helpers to show which variables are available for overriding (e.g. a CLI I have a question though. How will the (potential) Should we expect users be able to do something like that: cars:
filepath: "${runtime_params:path, ${globals:path, <some_default_path>}}"
type: pandas.CSVDataSet The syntax feels a bit cluttered and hard to read for beginners. Should we :
|
@Galileo-Galilei I have the same thought before and I try to summarize @ankatiyar thought.
In earlier comment I have an idea that we can inject I try to summarise the feature that we want (I think we need to improve the terminology too):
I think the question here is do we want to support overriding
Essentially the more specific scope should takes higher priority. We had a separate ticket that we explore using an extra CLI argument to make things more explicit, but in practice users may not really care about the difference. |
Summary from discussion on 8/09/2023 with @merelcht @noklam @amandakys:
|
Description
Is your feature request related to a problem? A clear and concise description of what the problem is: "I'm always frustrated when ..."
With #2467 we resolves OCL earlier and parameters propogate correctly for
parameters
. We cannot do the same forcatalog
or other configuration yet due to the problem of "residual" configurations. The problem is specific toOmegaConfigLoader
, because it requires merging the dictionaries before resolving.Take this simple example
Expected Result
kedro run --params base_path:my_bucket
this should work and replace the value correctly.Current Result
The value will not propagate for configurations other than
parameters
. This is because we need to mergeaggregated_configs
withruntime_params
, which produce invalid catalog entries.The compiled catalog may look like this
Context
Why is this change important to you? How would you use it? How can it benefit other users?
This is essential to enable common use case like providing a bucket name for
catalog
via CLI.kedro run --params base_path:my_bucket
, with TemplatedConfigLoader it doesn't work either, but there are workarounds. In contrast, there is no workaround for OmegaConfigLoader.Possible Implementation
(Optional) Suggest an idea for implementing the addition or change.
The implementation should make sure above case will work correctly. It should also work for arbitrary configurations, i.e.
config_loader["spark"] or
config_loader["my_random_config"]as long as it's registered with
config_patterns`Possible Alternatives
(Optional) Describe any alternative solutions or features you've considered.
Detail context
The text was updated successfully, but these errors were encountered: