[Bug] interpolation to hydra.job.override_dirname containing another interpolation is not possible

[immanuelweber]

🐛 Bug

Now that #725 is fixed, I post an issue for the behavior I described in (#725 (comment))
In the following example an interpolation to hydra.job.override_dirname containing another interpolation results in a crash:

To reproduce

test.py:

import hydra
from omegaconf import DictConfig


@hydra.main(config_path="conf", config_name="config")
def run(config: DictConfig):
    print("c", config.c)


if __name__ == "__main__":
    run()

conf/config.yaml:

a: 10
b: 20

c: override_${hydra:job.override_dirname}

call:
python test.py b=\${a} (I have to escape $)

** Stack trace/error message **

Traceback (most recent call last):
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/hydra/_internal/utils.py", line 197, in run_and_report
    return func()
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/hydra/_internal/utils.py", line 271, in <lambda>
    lambda: hydra.run(
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 105, in run
    return run_job(
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/hydra/core/utils.py", line 122, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/mtadmin/projects/debug_hydra/interpolation_jobdir/test.py", line 16, in run
    print("c", config.c)
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 315, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/base.py", line 95, in _format_and_raise
    format_and_raise(
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/_utils.py", line 555, in format_and_raise
    raise_(ex, cause)
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/_utils.py", line 548, in raise_
    raise ex
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 313, in __getattr__
    return self._get_impl(key=key, default_value=DEFAULT_VALUE_MARKER)
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 377, in _get_impl
    return self._resolve_with_default(
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 56, in _resolve_with_default
    resolved = self._resolve_interpolation(
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/base.py", line 370, in _resolve_interpolation
    return self._resolve_simple_interpolation(
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/base.py", line 344, in _resolve_simple_interpolation
    self._format_and_raise(key=inter_key, value=None, cause=e)
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/base.py", line 95, in _format_and_raise
    format_and_raise(
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/_utils.py", line 555, in format_and_raise
    raise_(ex, cause)
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/_utils.py", line 548, in raise_
    raise ex
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/base.py", line 335, in _resolve_simple_interpolation
    value = resolver(root_node, inter_key)
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 335, in caching
    cache[key] if key in cache else resolver(*OmegaConf._tokenize_args(key))
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/hydra/core/utils.py", line 150, in <lambda>
    lambda path: OmegaConf.select(cast(DictConfig, HydraConfig.get()), path),
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 531, in select
    format_and_raise(node=cfg, key=key, value=None, cause=e, msg=str(e))
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/_utils.py", line 623, in format_and_raise
    raise_(ex, cause)
  File "/home/mtadmin/miniconda3/envs/debug_hydra/lib/python3.8/site-packages/omegaconf/_utils.py", line 548, in raise_
    raise ex
omegaconf.errors.ConfigKeyError: str interpolation key 'a' not found
        full_key: hydra.job.override_dirname
        reference_type=Optional[HydraConf]
        object_type=HydraConf

Expected Behavior

the hydra.job.override_dirname should be able to interpolate a via b

System information

• Hydra Version : master/1.0.0rc2
• Python version : 3.8.3
• Virtual environment type and version : miniconda
• Operating system : ubuntu 18.04

Additional context

This can be worked arround with something like this:

hdr = HydraConfig.get()
override_dirname= hdr.job.override_dirname

[omry]

please explain in words what exactly you are trying to achieve here.

[immanuelweber]

In my actual code, I use the hydra:job.override_dirname in reporting for easier identification of what has been overridden.
So I have a config parameter run_${hydra:job.num}_${hydra:job.override_dirname}, which is used in reporting.
But as I (at least wanted to) describe in the report above, if I have an override of one of my actual parameters with an interpolation, then I can not interpolate to hydra:job.override_dirname anymore.
Hope this clarifies it a bit.

Btw. it works when the interpolation is part of the config.yaml and not overridden.

[omry]

I see, I will take a look.
Please clean up your example and make it minimal.

[immanuelweber]

like so?

[omry]

Yes, this is good.

[immanuelweber]

even though that you fixed it already a while ago, I just wanted to say thank you for the fix!