[Bug] Initialisation pydantic BaseModel with List causing validation error

[lalitpagaria]

🐛 Bug

Description

First thank you for awesome project.
I am using compose API to initialise class which have only List[str] parameter. When passing config in init pydantic throwing validation error that value is not a valid list (type=type_error.list). I found that config is reading list as ListConfig[str] and pydantic expect it as `List[str].

Please suggest how to instruct compose to force parse config in python's fields.

Checklist

• [ X ] I checked on the latest version of Hydra
• [ X ] I created a minimal repro

To reproduce

** Minimal Code/Config snippet to reproduce **
My config.yaml is as follows -

allowed_list:
 - "8080"
 - "8081"

My Model class as follows -

class AllowedPorts(BaseModel):
     allowed_list: List[str]

This is how I am using hydra -

with initialize(config_path="/home/user"):
    cfg = compose("config.yaml")
    allowed_ports = AllowedPort(**cfg)

At allowed_ports = AllowedPort(**cfg), pydantic will throw validation error.
** Stack trace/error message **

// Paste the bad output here!
Traceback (most recent call last):
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py", line 1477, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
...
...

  value is not a valid list (type=type_error.list)
python-BaseException

Expected Behavior

compose should have way to parse config in python default types.

System information

• Hydra Version : 1.0.4
• Python version : 3.8
• Virtual environment type and version : 20.0.31
• Operating system : MacOS

Additional context

Add any other context about the problem here.

[omry]

Hi, There are some major changes in instantiating in the next version of Hydra (in master now).
Take a look at the doc here.
Specifically, I think using using _convert_ all or partial will solve your problem.

Can you share details about why you need to use pydantic here?

[lalitpagaria]

Hi, There are some major changes in instantiating in the next version of Hydra (in master now).
Take a look at the doc here.
Specifically, I think using using _convert_ all or partial will solve your problem.

Yes this will solve my problem. Any idea when these will be released?

Can you share details about why you need to use pydantic here?

I am using pydantic because of easy integration with FastApi and less boiler plate code related to validation like http endpoint, ip address, secret string, etc.

Not related to this, is there any plan to support recursive initialisation via instantiate? Let say I have custom class, which have another field as different custom class.

For example

class Class2:
      name:str

class Class1:
       var: Class2

[omry]

Take a look at the docs I linked, the big change is recursive instantiation support. conversion strategies are just just a side thing.

No target data for 1.1. You can follow the milestone progress (there are a lot of things not yet done in the list).

In the mean time you can install from master.
Sometime down there line there will be some dev releases and later release candidate releases.

[lalitpagaria]

Cool thank you. Closing issue as feature request is already under development.

BTW I am using hydra in two ways -

• Rest API configuration (Guvicorn/Uvicorn app) -https://github.com/lalitpagaria/obsei/blob/master/rest_api/application.py#L115
• SDK configuration - https://github.com/lalitpagaria/obsei/blob/master/obsei/configuration.py

For rest API it is working like charm, and for SDK master branch will fix the issue which I am facing.
So if you want to quote example then you can use it.

[omry]

Cool thank you. Closing issue as feature request is already under development.

Actually this is developed already. It would be good if you try it out and provide any feedback.

About validating things like IP addresses etc:
There is an enhancement planned in OmegaConf [https://github.com/omry/omegaconf/issues/131] which should enable support for things like that.

[lalitpagaria]

Actually this is developed already. It would be good if you try it out and provide any feedback.

Just now tested master branch and _convert_ fixed this issue.

Thanks for pointing about OmegaConf, I will check this out.

[omry]

Just now tested master branch and convert fixed this issue.

Great. try recursive instantiation too! :)

Thanks for pointing about OmegaConf, I will check this out.

Hydra is using OmegaConf for the config.
I suggest that you read the docs of OmegaConf (or check the slides in the readme).
It has many great features that you can already use (interpolation, missing values support and much more).

[lalitpagaria]

@omry Is there possibility support _convert_ as parameter in compose function?
Idea to convert DictConfig to dict and ListConfig to list.

[omry]

You can use OmegaConf.to_container() directly.
Allow me to repeat the suggestion:
read the docs of OmegaConf.

[lalitpagaria]

Thanks. My apologies, I was only focused on Hydra to achieve my configuration need.

[omry]

Hydra is utilizing OmegaConf extensively. It's a good use of your time to learn about OmegaConf as well.