Enable %f pattern in now resolver

[forcecore]

Motivation

I sometimes manually run multiple instances of Hydra configured programs within a fraction of a second. In that case, their output directory's timestamp collide.

The user manual points to python2 strftime document which supports %f pattern (micro seconds), which would be one way of solving this issue.
https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior
However, in Python 3,
https://docs.python.org/3/library/time.html?highlight=strftime#time.strftime
time.strftime does not implement %f format and datetime.stftime does.

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

(How should this PR be tested? Do you require special setup to run the test or repro the fixed bug?)

Simple tutorial codes in the user manual would do the job

main.py:

#!/usr/bin/env python

import logging
import hydra
from omegaconf import OmegaConf

log = logging.getLogger(__name__)


@hydra.main(config_name='config')
def main(cfg):
    log.info(OmegaConf.to_yaml(cfg))
    log.debug("hello world!")


if __name__ == "__main__":
    main()

config.yaml:

db:
  driver: mysql
  user: omry
  password: secret

hydra:
  run:
    dir: ./outputs/${now:%Y%m%d_%H%M%S_%f}

Related Issues and PRs

(Is this PR part of a group of changes? Link the other relevant PRs and Issues here. Use https://help.github.com/en/articles/closing-issues-using-keywords for help on GitHub syntax)

None related

[shagunsodhani]

Thanks @forcecore Could you add a test for checking that %f is correctly resolved? Maybe we could add this to the docs at https://hydra.cc/docs/next/configure_hydra/workdir

[forcecore]

@shagunsodhani
I think I might write a test that uses %f and assert %f is not found in the actual run dir path name.
However, I'm unsure where to put the test. Could you recommend me where to put the test code?

Maybe along with
test_hydra.py: get_run_output(["tests/test_apps/run_dir_test/my_app.py"])
and write a new app like this?
"tests/test_apps/run_dir_test_microseconds/my_app.py"

[shagunsodhani]

Maybe something like this https://github.com/facebookresearch/hydra/blob/master/tests/test_hydra.py#L718 ?

[omry]

remove the prints please.

[omry]

You can use integration_test() instead of this decicated app.
Take a look at how it's being used in other places.

[forcecore]

It seems to expect something exact, but with microseconds I'm unsure how timestamp will go?

[omry]

You could print something like:
'%f' not in os.getcwd() and ensures that it's "True".

[forcecore]

Then I might try assert_regex_match, then it seems to only have use cases with dedicated programs

[forcecore]

You could print something like:
'%f' not in os.getcwd() and ensures that it's "True".

OK I see

[forcecore]

I think the new test works. It fails with the old code and passes with the new one.

[omry]

1. Please update the doc at https://hydra.cc/docs/configure_hydra/intro
2. Please add a news fragment file. See this.

[forcecore]

• Document updated to point to Python 3's datetime.strftime
• Added "1287.bugfix" with content Fixed hydra.job.id and hydra.job.num not getting passed to jobs in multirun

[omry]

Wrong doc link, should be https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior

[forcecore]

Doc link updated

[omry]

Awesome, thanks!