Hydra Configuration for Pytorch Lightning #2639

anthonytec2 · 2020-07-18T15:32:14Z

What does this PR do?

This merge request is the template for some initial discussion for using Hydra==1.0.0rc2 and Pytorch Lightning. @omry and I have been working on an example using the best features of Hydra to configure Pytorch Lightning. We want to understand what is the right place to put something like this MR. We have one file, trainer_conf.py which we think would be a good addition to the main repository if it could be maintained. This file consists of the base trainer configuration used for Pytorch Lightning. Users can then extend this base configuration with their own settings.

We highlight one style of configuring Pytorch Lightning with Hydra, which is using structured configs. Structured configs are mostly data classes that define the types and variables for a given argument group. This enables argument parsing to have type safety. This MR, enables configuring all of the argparse setting defined for Pytorch Lightning's Trainer class and shows an example of a user configuration which extend a base configuration. We also highlight Hydra's ability to instantiate objects, by creating ObjectConfs, which is really useful for something like changing out your, optimizer, scheduler, model or dataset easily. One other feature we show is callbacks can still be defined and used with PL using the ObjectConf structure.

For example a user can change the dataset in this MR by:

python -m pl_examples.hydra_examples.pl_template data=fashionmnist
python -m pl_examples.hydra_examples.pl_template  data=kmnist
python -m pl_examples.hydra_examples.pl_template  logger=testtube data=mnist

should a user make an error in the configuration:

python -m pl_examples.hydra_examples.pl_template  data=fashionmnista
Could not load data/fashionmnista.
Available options:
        fashionmnist
        kmnist
        mnist

We experience error #2519 which limits tensorboard support at the moment.

Fixes #2322
Fixes #807

Before submitting

Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?
Did you verify new and existing tests pass locally with your changes?
If you made a notable change (that affects users), did you update the CHANGELOG?

pep8speaks · 2020-07-18T15:32:19Z

Hello @anthonytec2! Thanks for updating this PR.

In the file pl_examples/hydra_examples/pl_template_simple.py:

Line 10:1: E265 block comment should start with '# '
Line 11:1: E265 block comment should start with '# '
Line 30:47: E231 missing whitespace after ','

In the file pl_examples/hydra_examples/user_config.py:

Line 15:1: E302 expected 2 blank lines, found 1
Line 39:58: W291 trailing whitespace
Line 41:28: W291 trailing whitespace
Line 52:1: W293 blank line contains whitespace
Line 55:1: E303 too many blank lines (3)
Line 56:1: E305 expected 2 blank lines after class or function definition, found 3

In the file pl_examples/models/hydra_config_model.py:

Line 122:1: W391 blank line at end of file

Comment last updated at 2020-07-21 05:27:04 UTC

omry

@anthonytec2:
Can you squash all the commits? it will make for a better clean start for this review phase.

Big questions for the PL team:
A. Parts of this example belongs in PL core in my opinion, specifically the generic configuration classes.
B. The registration of the configs with the Hydra ConfigStore, as well as as the example itself - depends on Hydra 1.0.0rc2 or newer.

We need to enforce that we are running against a supported version somehow.
From my perspective, the correct way to achieve that is via a pip dependency.
1.1. If this is not viable, will an extra dependency work?
pip install pip pytorch-lightning[hydra]
1.2. if not viable, will a runtime check + warning on old version work?
Will the PL team accept the generic configs into the core and maintain them moving forward?

I know those are big decisions.
Please take your time evaluating this PR and consider the usefulness versus the cost of maintaining those config classes and adding a more explicit dependency on Hydra.

omry · 2020-07-19T00:12:10Z

pl_examples/hydra_examples/README.md

@@ -0,0 +1,21 @@
+## Hydra Pytorch Lightning Example


Needs a pass for capitalization and typos.

omry · 2020-07-19T00:13:01Z

pl_examples/hydra_examples/README.md

+
+All of the above hyperparameters are configured in the config.yaml file which contains the top level configuration for all these configurations. Under this file is a defaults list which highlights for each of these Hydra groups what is the default configuration.  Beyond this configuration file, all of the parameters defined can be overriden via the command line. 
+
+Additionally, for type safety we highlight in our file `user_config.py` an example of extending the `PLConfig` data class with a user configuration. Hence, we can get the benefits of type safety for our entire config.yaml. For further examples of this, [checkout](https://hydra.cc/docs/next/tutorials/structured_config/intro).


There are no more examples for this there, but it is recommended that people using Hydra read both the Basic tutorial and the Structured Configs tutorial.

omry · 2020-07-19T00:16:28Z

pl_examples/hydra_examples/conf/callbacks/basic.yaml

+# @package _group_
+
+functions:
+  print:
+    target: pl_examples.hydra_examples.user_config.MyPrintingCallback
+  message:
+    target: pl_examples.hydra_examples.user_config.MessageCallback
+    params:
+      iter_num: 12
+
+callbacks_list:
+  - ${callbacks.functions.print}
+  - ${callbacks.functions.message}


Since we are not using composition here it will be simpler to just inline the callbacks:

Suggested change

# @package _group_

functions:

print:

target: pl_examples.hydra_examples.user_config.MyPrintingCallback

message:

target: pl_examples.hydra_examples.user_config.MessageCallback

params:

iter_num: 12

callbacks_list:

- ${callbacks.functions.print}

- ${callbacks.functions.message}

# @package _group_

callbacks:

- target: pl_examples.hydra_examples.user_config.MyPrintingCallback

- target: pl_examples.hydra_examples.user_config.MessageCallback

params:

iter_num: 12

We may not want to use composition here, but we should make things more explicit to allow the user to use composition upon this quite easily...

OmegaConf replaces lists completely during merge:

c1 = OmegaConf.create({"list": [1,2,3]}) c2 = OmegaConf.create({"list": [4,5,6]}) c3 = OmegaConf.merge(c1,c2) assert c3 == {"list": [4,5,6]}

One way to compose list element is what this looked like before:

e1 = {"elements": {"e1": {"foo":"bar"}}} e2 = {"elements": {"e2": {"zoo":"var"}}} l1 = {"list": ["${elements.e1}", "${elements.e2}"]} cfg = OmegaConf.merge(l1, e1, e2) print(cfg.pretty(resolve=True))

outputs:

list: - foo: bar - zoo: var elements: e1: foo: bar e2: zoo: var

This lets you reuse e1 and e2 in different lists without repeating them.
This is a bit advanced, and for the sake of this example I think we can just keep things simple.

This usage pattern is a good candidate for a feature I am thinking about of hiding a config node. in this case we want to hide the elements node (but keep it there to support the runtime interpolation from list). This feature is planned for the next major version of OmegaConf.

omry · 2020-07-19T00:28:48Z

pl_examples/hydra_examples/pl_template.py

+    trainer = Trainer(
+        **cfg.trainer,
+        logger=instantiate(cfg.logger),
+        profiler=instantiate(cfg.profiler),
+        checkpoint_callback=instantiate(cfg.checkpoint),
+        early_stop_callback=instantiate(cfg.early_stopping),
+        callbacks=callbacks,
+    )


PL Team, any thoughts about simplifying this?
Is it possible to create standard config driven initialization method (or factory) for the trainer?

How do you feel about:

trainer = Trainer.from_cfg(cfg)

@anthonytec2 :
I think all the PL specific configs should be under the same node:

pl: trainer: ... logger: ... profiler: ... ...

This would allow more easier initialization of PL (without having to worry about the user adding his/her own things to the top level config:

trainer = Trainer.from_cfg(cfg.pl)

Actually I like this proposal, but I feel we should change the name to Trainer.from_hydra_config since otherwise it may not be clear, what kind of config to expect here.

I thought the same. I agree with Justus

The config object is just an object with some fields, there is nothing Hydra about it.

omry · 2020-07-19T00:30:10Z

pytorch_lightning/trainer/trainer_conf.py

+    num_tpu_cores: Optional[int] = None
+
+
+cs.store(group="trainer", name="trainer", node=LightningTrainerConf)


Please namespace all of the pl configs:
pl/trainer.

@omry Can you please explain what are benefits of this?

Won't this just only make the configuration deeper? And as a result, when the user will like to override trainer parameters from command line, he/she will have to add pl to each one of them, like pl.trainer.gpus=[2,3] etc.?

Asking as in NeMo (where we independently started to merge PT Lightning with Hydra like two weeks ago;)) we just yesterday did the opposite and got rid of pl...

@tkornuta-nvidia, yes:
You can think of the configuration object Hydra is composing as a shared object that is potentially the home for more than one user.
As an example, the configuration is already hosting both Hydra itself under the hydra node (and the hydra/ config groups) and the user config.
Different frameworks that are installed can make their configs accessible via Hydra (via a Config Searchpath plugin and/or by storing configs in the ConfigStore like in the example above).
To avoid collisions between themselves and with the end user, I am encouraging framework owners to place their configs in some kind of namespace.

Imagine the following scenario:
A user wants to use NeMo and Fairseq at the same time assuming they both use Hydra.
If both conveniently places their model configs in the config group model and in the config node model, the user can now easily try to configure the NeMo model with a configuration for a Fairseq model.
namespacing is solving this.

This of it as the equivalent of Java packages or Python modules, but for configuration.
The cost that you are mentioning about the command line being longer is real, but I think It's still the right choice.

omry · 2020-07-19T00:34:15Z

pl_examples/hydra_examples/conf/optimizer.py

@@ -0,0 +1,108 @@
+from dataclasses import dataclass


I think the dataclasses for optimizer and scheduler should probably be a part of PL. It makes more sense than evey user re-defining Adam, AdamW, CosineConf etc.

Speaking of which, we should be consistent. Why Either consistently add Conf suffic or consistently not add it.

1.) IMO you're right, this is not specific to any research area so this should be part of lightning if we decide to move major things from this pr to lightning core

2.) We should definitely add the suffix to make clear that it actually is a config

I'd like to add that in my present use cases, although I don't necessarily want to use the structured configs paradigm for everything at the moment as I'm kind of enjoying the simplicity of a terse yaml heirarchy, I really like the idea of supporting trainer_conf.py, optimizer.py, and scheduler.py via the structured configs.

It feels very similar in spirit to maintaining Trainer's static method add_argparse_args(). On that note, maybe I missed this in the earlier discussion, but what are the actual cons to including optimizer_conf.py and scheduler_conf.py in PTL? Perhaps aggregating them into a singular hydra_conf.py in core so there's only one file to maintain?

@romesc-CMU:
As a user you can still use yaml config files and they will be validated automatically against the dataclasses as long as their name and config group matches.

fix job name template change to model create hydra examples folder fix error with none values optimizers and lr schedules clean up model structure model has data included dont configure outputs document hydra example update readme rename trainer conf scheduler example schedulers update change out structure for opt and sched flatten config dirs reduce number of classes scheduler and opt configs spelling change group config store location change import and store structured conf remaining classes fix for date change location of trainer config fix package name trainer instantiation clean up init trainer type fixes clean up imports update readme add in seed Update pl_examples/hydra_examples/README.md Co-authored-by: Omry Yadan <omry@fb.com> Update pl_examples/hydra_examples/README.md Co-authored-by: Omry Yadan <omry@fb.com> change to model clean up hydra example data to absolute path update file name fix path isort run name change hydra logging change config dir use name as logging group load configs in init py callout callbacks fix callbacks empty list example param data params example with two other data classes fix saving params dataset path correction comments in trainer conf logic in user app better config clean up arguments multiprocessing handled by PL settings cleaner callback list callback clean up top level config wip user config add in callbacks fix callbacks in user config fix group names name config fix user config instantiation without + change type split for readability user config move master config yaml hydra from master changes remove init py clean up model configuration add comments add to readme function doc need hydra for instantiate defaults defined in config yaml remove to do lines issue note remove imports unused cfg init removal double define instantiate changes change back to full config Update pl_examples/hydra_examples/pl_template.py Co-authored-by: Omry Yadan <omry@fb.com> Revert "double define" This reverts commit 4a9a962. fix data configuration remove bug comment, fixed already fix callbacks instantiate

romesco · 2020-07-19T04:44:00Z

I'm very happy this is moving forward, thank you @anthonytec2 for all the hard work!

One thing I'm still wondering about (after testing the latest commit) is whether we can also support an alternative to passing only cfg.model, cfg.data, cfg.optimizer, cfg.scheduler to __init__() of any LightningModule.

When passing configs like this, it is somewhat incompatible with the current LightningModule best-practice which is to list out the model hyperparameters in the __init__ signature as in the current lightning_template.py:

def __init__(self,
             drop_prob: float = 0.2,
             batch_size: int = 2,
             in_features: int = 28 * 28,
             learning_rate: float = 0.001 * 8,
             optimizer_name: str = 'adam',
             data_root: str = './datasets',
             out_features: int = 10,
             num_workers: int = 4,
             hidden_dim: int = 1000,
             **kwargs
             ):

as opposed to:

self __init__(self,
             model,
             data,
             scheduler,
             opt
             ):

This was one of the recommendations following the removal of passing an hparams Namespace, and I feel made quickly understanding others' modules much easier. It also lets the author include good defaults in the top level of the module. Providing a good example of how to do this, but still using hydra would be ideal (and avoid reintroducing some of the issues associated with the hparams Namespace object).

With 2519, I believe we can also offer this option.

justusschock · 2020-07-19T09:21:15Z

Actually I like this PR a lot. We should be clear on the interface and we should check about where to put those configs to make them accessible from the package.

@omry how long until the hydra release? is there already an ETA?

A) I agree that some parts of this belong to lightning core. But we should carefully sort them out from this PR.
B) I feel we could add this as an optional dependency or an extra dependency but not like pip install pip pytorch-lightning[hydra] but more like pip install pip pytorch-lightning[extras] which will also include hydra but not limited to hydra.

However this is just my personal opinion, so we should decide this together @PyTorchLightning/core-contributors

omry · 2020-07-19T15:06:00Z

Actually I like this PR a lot. We should be clear on the interface and we should check about where to put those configs to make them accessible from the package.

@omry how long until the hydra release? is there already an ETA?

I released Hydra 1.0.0rc2 yesterday. No deadline for Hydra 1.0.0 but it's getting close (I think within a few weeks).

A) I agree that some parts of this belong to lightning core. But we should carefully sort them out from this PR.
B) I feel we could add this as an optional dependency or an extra dependency but not like pip install pip pytorch-lightning[hydra] but more like pip install pip pytorch-lightning[extras] which will also include hydra but not limited to hydra.

I realized that users can choose multiple extras yesterday: foo[e1,e2], which might make sense if you have additional extras you want to support explicitly. I have not actually used extras before so I don't have strong opinions either way.

However this is just my personal opinion, so we should decide this together @PyTorchLightning/core-contributors

Sounds good.

omry · 2020-07-19T15:12:00Z

One limitation I want to call out to avoid surprises:
OmegaConf does not currently support Union types in Structured Configs (except Optional).
Keep this in mind, it means that for now you can't use Union in these configs.
The alternative is to use Any and do a runtime check until a time this support is added (I am planning it, and unless I hit some very difficult implementation issues this will be supported).

justusschock · 2020-07-20T07:40:26Z

@omry regarding extras: I've used them a lot and I experienced users not to know about it at all. In most cases users will just install the plain package and when they hit an import error they will just install the missing package afterwards

omry · 2020-07-20T07:43:51Z

@omry regarding extras: I've used them a lot and I experienced users not to know about it at all. In most cases users will just install the plain package and when they hit an import error they will just install the missing package afterwards

I think if we go there it will have to be documented clearly.

williamFalcon · 2020-07-20T09:36:49Z

thank you for putting all these examples together!

I do feel like these configs should live on the main hydra repo and not the pl lightning repo for these reasons:

Lightning does need to be agnostic to configuration systems.
In my view, these help people using hydra use lightning easier.
Not every lightning user likes/uses hydra and so we need to remain neutral to other potential libraries who might also want to integrate with lightning easier.

So, my requested changes here are to:

Leave an example in Lightning under pl_examples.
Migrate all the configs to hydra under a folder (/integrations/lightning) probably.
we create a convenience function (suggested by @justusschock)

Trainer.from_hydra_config(...)

williamFalcon

please apply the requested changes

Borda · 2020-07-20T11:26:34Z

I feel we could add this as an optional dependency or an extra dependency but not like pip install pip pytorch-lightning[hydra] but more like pip install pip pytorch-lightning[extras] which will also include hydra but not limited to hydra.

I like this option to install even some from extras :]

omry · 2020-07-24T15:46:30Z

@yukw777, thanks for digging. I would like to understand this better. can you jump into the Hydra chat?

tkornuta-nvidia · 2020-07-24T23:08:15Z

One limitation I want to call out to avoid surprises:
OmegaConf does not currently support Union types in Structured Configs (except Optional).
Keep this in mind, it means that for now you can't use Union in these configs.
The alternative is to use Any and do a runtime check until a time this support is added (I am planning it, and unless I hit some very difficult implementation issues this will be supported).

@omry Actually Union is something that I wanted to ask you about - what is the reason you are not supporting them in OmegaConf?

I had a serious problem when I was implementing a prototype of a Structured Config for PL trainer in NeMo, in particular for gpus I did gpus: Optional[int] = None what was preventing cases when user wanted to pick GPU e.g. by name.

@anthonytec2 did better job with gpus: Optional[Any] = None 👍 (but on the other hand, it accepts anything now, right?)

omry · 2020-07-25T01:54:16Z

One limitation I want to call out to avoid surprises:
OmegaConf does not currently support Union types in Structured Configs (except Optional).
Keep this in mind, it means that for now you can't use Union in these configs.
The alternative is to use Any and do a runtime check until a time this support is added (I am planning it, and unless I hit some very difficult implementation issues this will be supported).

@omry Actually Union is something that I wanted to ask you about - what is the reason you are not supporting them in OmegaConf?

The reason is that I am working on releasing Hydra and I did not have the time to work on this.
This is something I am planing to look into at the next cycle of OmegaConf improvements.
Structured Configs in OmegaConf was a major effort that took many months. at some point you need to draw the line and move on.

I had a serious problem when I was implementing a prototype of a Structured Config for PL trainer in NeMo, in particular for gpus I did gpus: Optional[int] = None what was preventing cases when user wanted to pick GPU e.g. by name.

You can do something like gpus: Any = None, which will happily allow you to pass in anything you want.
Then, you can just validate the input yourself at runtime, e.g:

assert isinstance(cfg.gpus , (MutableSequence, int)), "gpus must be a list or an int"

Borda · 2020-07-25T08:15:16Z

@yukw777, thanks for digging. I would like to understand this better. can you jump into the Hydra chat?

let's hare hydra thread in PL slack...

tkornuta-nvidia · 2020-08-10T21:47:43Z

@williamFalcon What is the status of this PR? What is missing? What are the showstoppers? And when it can be merged?

Asking as I am one of people working on NVIDIA NeMo and recently we have started using a) PTL as NeMo backend for training and b) hydra for configuration management. So as you might guess, we have faced several issues that this PR is actually trying to solve... ;)

williamFalcon · 2020-08-10T21:54:06Z

yup. The main blocker is that we still need to figure out where these configs will be hosted. Since we don't want to couple PL to any config system, we have a few options:

add to hydra (best option in my book)
add to bolts
add to a new independent repo?

Basically, we want to make sure the hydra team is also involved in maintaining these configs, since it can shift a lot to us which we have already seen with the loggers.

But 100% agree, that i'd like to get this solved asap

tkornuta-nvidia · 2020-08-10T22:10:30Z

yup. The main blocker is that we still need to figure out where these configs will be hosted. Since we don't want to couple PL to any config system, we have a few options:

add to hydra (best option in my book)

add to bolts

add to a new independent repo?

Basically, we want to make sure the hydra team is also involved in maintaining these configs, since it can shift a lot to us which we have already seen with the loggers.

But 100% agree, that i'd like to get this solved asap

Thanks for reply. So most of the structured configs introduced by this PR is in fact associated with pure PyTorch. Tiny problem: PyTorch is not depending on hydra.

Moreover, hydra, by definition, is "framework-agnostic" - so IMO they should not be a part of hydra.

Option number two (bolts) doesn't sound good to me only by looking at package dependencies - PTL Bolts depend on PTL, not the opposite.

So if you do not want them to be part of PTL project, then I guess option 3 is the way: they should be moved to a separate repository. The question is who/where it should be hosted. If you don't want to host it under PTL organization, then the natural choice to me is pytorch organization. But I can also arrange this and start hosting it on NVIDIA open-source organization. @omry ? @anthonytec2 ?

Let's get that ball rolling...

williamFalcon · 2020-08-10T22:35:56Z

@tkornuta-nvidia thanks for the great suggestion. Yeah, i think hosting it at nvidia would be great. That way both frameworks can contribute to it as needed!

@PyTorchLightning/core-contributors thoughts?

romesco · 2020-08-10T23:18:49Z

@tkornuta-nvidia, @williamFalcon @anthonytec2 I'd be happy to help maintain if we host elsewhere. Sort of funny that the project I'm working on which uses the PTL+hydra combo is also parttime @ nvidia (the seattle robotics group).

If it's not too much trouble, it would definitely be great to get on the PTL slack and discuss in the #configs-hydra channel 😄

lkhphuc · 2020-08-13T16:31:36Z

Just want to chip in, the neovim project did something similar that they create a separate repo under the same org that hosts configurations for a specific functionality, in a best effort manner. https://github.com/neovim/nvim-lsp

snisarg · 2020-09-16T19:27:28Z

pl_examples/hydra_examples/pl_template.py

+    callbacks = [instantiate(c) for c in cfg.callbacks.callbacks] if cfg.callbacks else []
+
+    trainer = Trainer(
+        **cfg.trainer,
+        logger=instantiate(cfg.logger),
+        profiler=instantiate(cfg.profiler),
+        checkpoint_callback=instantiate(cfg.checkpoint),
+        early_stop_callback=instantiate(cfg.early_stopping),
+        callbacks=callbacks,


How about separating this into a TrainerConf class.

We use target to this class that does these lines. So all users have to do it select TrainerConf and when they instantiate() on it, they get the real Trainer?

edenlightning · 2020-10-19T16:30:48Z

@anthonytec2 should we keep this PR open? or is everything already in the nvidia repo?

tkornuta-nvidia · 2020-10-19T16:36:48Z

@anthonytec2 should we keep this PR open? or is everything already in the nvidia repo?

Hi, at the end the project was moved to PyTorchEcosystem:
https://github.com/pytorch/hydra-torch

and it is still work in progress. Once done, we will clean up this PR and remove the PT-related structured configs. Hopefully then we will merge.

stale · 2020-11-02T18:44:32Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If you need further help see our docs: https://pytorch-lightning.readthedocs.io/en/latest/CONTRIBUTING.html#pull-request or ask the assistance of a core contributor here or on Slack. Thank you for your contributions.

stale · 2020-11-07T19:22:34Z

This pull request is going to be closed. Please feel free to reopen it create a new from the actual master.

turian · 2020-12-16T04:03:35Z

@tkornuta-nvidia You write "Once done, we will clean up this PR and remove the PT-related structured configs. Hopefully then we will merge.", I am curious if this PR is still in progress? I am interested in good PL and Hydra support

romesco · 2020-12-19T01:46:42Z

Hey @turian, since this is a pretty broad goal that reaches beyond configuring just lightning classes, we've moved a lot of the work to: https://github.com/pytorch/hydra-torch

We are also working on the analogous: https://github.com/romesco/hydra-lightning.

If you want to start testing by using in your project, you can use a git dependency until the PyPI release! We would love the feedback.

Both projects follow the same patterns and are being generated via an automatic tool we've been developing. My goal is to provide a minimal package that projects using hydra with lightning can simply import and have all necessary configs available. It doesn't try to do anything beyond that. The beauty of this format is that all hydra-configs projects are setup as namespace packages which means you can have many different projects contributing to hydra-configs.foo.bar.

End user code will have something like:

from hydra_configs.torch.utils.data import DataLoaderConf
from hydra_configs.pytorch_lightning.trainer import TrainerConf

My timeline for initial releases of these config packages (minimal critical path classes for torch, torchvision, and lightning) is before the end of January. If you'd like to jump in and help, I could definitely use it and we could probably get it finished even sooner =].

turian · 2020-12-19T04:07:58Z

@romesco thank you so much for the pointers. I actually just started migrating a new pl project to hydra, following the conventions of @yukw777 in https://github.com/yukw777/leela-zero-pytorch

I will check out the nascent projects. I am curious if you participate on the zulip chat or similar, so we can discuss a little more. I am thrilled to see this

mergify bot requested a review from a team July 18, 2020 15:32

Borda added feature Is an improvement or enhancement Important discussion In a discussion stage labels Jul 18, 2020

Borda added this to the 0.9.0 milestone Jul 18, 2020

Borda requested review from ananyahjha93, awaelchli, jeremyjordan, justusschock, neggert, tullie and williamFalcon July 18, 2020 19:54

omry reviewed Jul 19, 2020

View reviewed changes

anthonytec2 force-pushed the hydra_conf branch from 4e394e4 to c7e7ca3 Compare July 19, 2020 02:39

anthonytec2 added 4 commits July 19, 2020 08:32

change out links

7429e01

conf consistency

a5a764e

support simpler callbacks

739ae2f

callbacks is in trainer config

ab71dd1

williamFalcon requested changes Jul 20, 2020

View reviewed changes

mergify bot requested a review from a team July 20, 2020 09:46

yukw777 mentioned this pull request Jul 24, 2020

Subprocess launched in ddp have the wrong cwd when using hydra. #2691

Closed

omry mentioned this pull request Jul 31, 2020

Hydra configs with multi GPU DDP training in Pytorch Lightning #2727

Closed

Borda assigned yukw777 Aug 7, 2020

romesco mentioned this pull request Aug 7, 2020

Support **DictConfig hparam serialization #2519

Merged

7 tasks

Borda assigned nateraw Aug 7, 2020

awaelchli modified the milestones: 0.9.0, 1.0.0 Aug 8, 2020

tkornuta-nvidia mentioned this pull request Aug 27, 2020

Add primer notebook on NeMo 1.x, PTL and Hydra/OmegaConf NVIDIA/NeMo#1080

Merged

snisarg reviewed Sep 16, 2020

View reviewed changes

edenlightning modified the milestones: 1.0, 1.1 Oct 4, 2020

stale bot added the won't fix This will not be worked on label Nov 2, 2020

stale bot closed this Nov 7, 2020


		All of the above hyperparameters are configured in the config.yaml file which contains the top level configuration for all these configurations. Under this file is a defaults list which highlights for each of these Hydra groups what is the default configuration. Beyond this configuration file, all of the parameters defined can be overriden via the command line.

		Additionally, for type safety we highlight in our file `user_config.py` an example of extending the `PLConfig` data class with a user configuration. Hence, we can get the benefits of type safety for our entire config.yaml. For further examples of this, [checkout](https://hydra.cc/docs/next/tutorials/structured_config/intro).

		num_tpu_cores: Optional[int] = None


		cs.store(group="trainer", name="trainer", node=LightningTrainerConf)

Hydra Configuration for Pytorch Lightning #2639

Hydra Configuration for Pytorch Lightning #2639

Conversation

anthonytec2 commented Jul 18, 2020 • edited by Borda

What does this PR do?

Before submitting

pep8speaks commented Jul 18, 2020 • edited

Comment last updated at 2020-07-21 05:27:04 UTC

omry left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omry Jul 19, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkornuta-nvidia Jul 24, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justusschock Jul 19, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

romesco commented Jul 19, 2020 • edited

justusschock commented Jul 19, 2020

omry commented Jul 19, 2020 • edited

omry commented Jul 19, 2020 • edited

justusschock commented Jul 20, 2020

omry commented Jul 20, 2020

williamFalcon commented Jul 20, 2020 • edited

williamFalcon left a comment

Choose a reason for hiding this comment

Borda commented Jul 20, 2020

omry commented Jul 24, 2020

tkornuta-nvidia commented Jul 24, 2020

omry commented Jul 25, 2020

Borda commented Jul 25, 2020

tkornuta-nvidia commented Aug 10, 2020

williamFalcon commented Aug 10, 2020

tkornuta-nvidia commented Aug 10, 2020 • edited

williamFalcon commented Aug 10, 2020

romesco commented Aug 10, 2020

lkhphuc commented Aug 13, 2020

Choose a reason for hiding this comment

edenlightning commented Oct 19, 2020

tkornuta-nvidia commented Oct 19, 2020

stale bot commented Nov 2, 2020

stale bot commented Nov 7, 2020

turian commented Dec 16, 2020

romesco commented Dec 19, 2020 • edited

turian commented Dec 19, 2020

anthonytec2 commented Jul 18, 2020 •

edited by Borda

pep8speaks commented Jul 18, 2020 •

edited

omry left a comment •

edited

omry Jul 19, 2020 •

edited

tkornuta-nvidia Jul 24, 2020 •

edited

justusschock Jul 19, 2020 •

edited

romesco commented Jul 19, 2020 •

edited

omry commented Jul 19, 2020 •

edited

omry commented Jul 19, 2020 •

edited

williamFalcon commented Jul 20, 2020 •

edited

tkornuta-nvidia commented Aug 10, 2020 •

edited

romesco commented Dec 19, 2020 •

edited