Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide built-in routines from loading config `.py` files #195

huzecong opened this issue Sep 6, 2019 · 0 comments

Provide built-in routines from loading config `.py` files #195

huzecong opened this issue Sep 6, 2019 · 0 comments


Copy link

@huzecong huzecong commented Sep 6, 2019

Configurations in examples (e.g. transformer/ is written in a separate file with config values as global variables. To be able to use these config files, we import them with importlib, and later convert them into dictionaries so we can pass it as hparams. An example here:

config_downstream = importlib.import_module(args.config_downstream)
config_downstream = {
k: v for k, v in config_downstream.__dict__.items()
if not k.startswith('__')}

While this approach works, it certainly has downsides:

  1. Importing a module from a user-given location is unsafe, as Python executes the code during import.
  2. As illustrated above, converting an imported module to dictionary is non-trivial (has to ignore magic dunder (__) variables).

I'm proposing that we provide built-in mechanisms to deal with importing config modules. This could be a simple wrapper of the above code snippet that imports the the config module and converts it into a dictionary. It should also be able to work with file paths such as configs/ (instead of dot notation configs.config_data).

This would solve the second issue but not the first one, which I think would be impossible to solve as long as we still use config modules. In the future, we might think of alternative ways to store configs while still being robust enough.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
1 participant
You can’t perform that action at this time.