Is it possible to define dependency between two different config groups?

[bilzard]

In my use case, I want to load specific config groups depending on the value stored on the other config groups. For example, I defined two config groups preprocess and feature_extract. The parameter defined in preprocess/v1 generates some asset named preprocess_v1, and the parameter defined in feature_extract/v1 uses these assets (preprocess_v1) and generates asset named feature_extract_v1. Since the parameter used to generate feature_extract_v1 are dependent on the parameter preprocess/v1, I want to define dependency between two parameter groups preprocess/v1 and feature_extract/v1. Namely, if I select feature_extract/v1, I want hydra automatically select preprocess/v1. Is it possible using hydra?

• preprocess
  • v1
  • v2
• feature_extract
  • v1
  • v2

[odelalleau]

Yes, basic interpolation support is available in the defaults list, see doc at https://hydra.cc/docs/advanced/defaults_list/#interpolation-in-the-defaults-list

[bilzard]

@odelalleau Thanks for the link!

Well, it seems possible using interpolation if preprocess and feature versions are exactly same. However, my use case is a bit more complicated: they can have different versions.

For example if feature_extract/v2 depends on preprocess/v1, how to automatically select preprocess/v1 if feature_extract/v2 is selected?

Can interpolation by parameter also possible?

I want to do like below:

feature_extract/v2.yaml

version: v2
preprocess_version: v1

config.yaml

defaults:
  - feature_extract: v1
  - preprocess: ${feature_extract.preprocess_version}

[odelalleau]

No that shouldn't work, because when parsing the defaults list the content of the config itself is not available yet.

I think what you're trying to achieve would better be done by overriding the defaults list from your feature_extract/v2.yaml, as below.

feature_extract/v2.yaml

defaults:
  - override /preprocess@_global_.preprocess: v2

config.yaml

defaults:
  # Note that the order matters since the second one may override the first one.
  - preprocess: v1
  - feature_extract: v1

Edit: see comment below for a slightly better syntax

[bilzard]

@odelalleau Thanks. It worked!

By the way, I cannot fully understand why we should specify preprocess twice?
What is the difference of these three?

1. override /preprocess@_global_.preprocess: v2
2. override /preprocess: v2
3. override /preprocess@_global_: v2

[odelalleau]

1. Will dump the preprocess/v2 config into the preprocess node of the root config
2. Will dump the preprocess/v2 config into the preprocess node of the current config (i.e., feature_extract.preprocess)
3. Will dump the preprocess/v2 config directly into the root config (not under a specific node)

But actually I just realized the better option is

4. override /preprocess@_group_: v2

which will do the same as 1 but without having to specify the name of the target node (it will default to preprocess).

[bilzard]

Thanks. Now I can have more clear understanding on packages and default list.
Is my following understanding correct?

Basically, we should specify source and destination config group in the override syntax.

1. The first part /preprocess specifies how to search source config group from the current config file. And /specifies absolute path.
2. The latter part @_global_.preprocess or @_group_ specifies where to place the config group (destination).

[omry]

Yes.
The default list is documented here, the package (@) is documented here.

Take the time to read and understand those two pages.

[omry]

The solution above is correct. To make feature_extract/v2 select preprocess/v1, put this in feature_extract/v2.yaml:

defaults:
  - override /preprocess@_group_: v1

The primary Defaults List must select preprocess before feature_extract.

/preprocess identifies the config group using an absolute path. @_group_ places its contents in that group's default absolute package—preprocess here.

This sets the default dependency selection; an explicit command-line override still takes precedence.