Support of pydantic to use in structured config

[espdev]

Currently, omegaconf supports dataclasses to create structured configs.

Pydantic is a library for Python 3.6+ for data parsing and validation using Python type hints.

Would you mind considering support of pydantic BaseModel/BaseSettings for structured configs? Using pydantic would allow to shift the parsing/validation functionality to it and supporting a lot of data types in configs (IP addresses, URLs, email, Path etc).

[omry]

Hi @espdev,
Thanks for the suggetion.
OmegaConf is a low level library without external dependencies except for the bare minimum.
While pydantic share some functionality with it, OmegaConf is more of a runtime data model than a data validation system.
In addition, OmegaConf support features not present in Pydantic (e..g Interpolation, MISSING values, custom resolvers).
This is not a direction I am interesting in taking OmegaConf to.

In terms of more powerful validation, take a look at #131.
The proposal there is to utilize PEP-593 and support additional metadata and validation constraints for supported types.

[espdev]

Got it. Thanks!

In my some projects I use OmegaConf and pydantic together. OmegaConf loads config files and performs interpolation. pydantic model represents a typed config structure and validates config dictionary after OmegaConf.to_container.

[omry]

1. Are you using Hydra?
2. What are the features missing in OmegaConf that you are using pydantic for?

In terms of config composition:
OmegaConf support for Structured Configs allow strong runtime type safety during config composition. This is something that you are not getting from pydantic in this scenario.

[espdev]

Yes, I know about Hydra. However, Hydra is a high-level solution with a focus to app-oriented projects with complex configuration. In one of the future cli-based app project I would like to use Hydra.

With pydantic model based configs I have statically defined config objects. IDE knows about it: auto-completion and type checking work fine. And pydantic allows me to define more config value types: pathlib.Path, Set, IPv4Address for example. I can write custom validators with relationship between config values. I usually use Strict* types in pydantic models for strong type safety.

[omry]

Yup, Hydra is higher level. I wanted to understand how you are using OmegaConf better.
Structured Configs with OmegaConf are also statically defined and allow for IDE and static type checking via duck typing.
They are still however DictConfig objects, which means that for the most part they just look like the underlying object (with some exceptions : no functions, no Union for now, and the type does not actually match).
You can still use duck typing though to get the benefits of static type checking:

@dataclass
class User:
  age: int
  name: str

user : User = OmegaConf.structured(User(age=7, name="bond"))

OmegaConf adds runtime type safety in addition to static type safety:

user.age = "seven" # Validation exception at runtime.

This type safety extends to config merges as well with somewhat different semantics.
Finally, OmegaConf indeed does not allow all object types.
With Structured Configs you get a lot of flexibility though.
I recommend that you go over the OmegaConf 2.0 slides linked from the project README to get a good idea Structured Configs and their benefits.

[espdev]

Thanks for your explanation and recommendations! I will take a look. Maybe I can really do without pydantic.

[elgalu]

@espdev could you share what have you learned wrt to this topic in the last ~6 months? I'm also facing this decision, whether to let go on pydantic.PostgresDsn, pydantic.HttpUrl, etc and be happy with Hydra/OmegaConf Structured Configs or try to setup a hybrid like you did with OmegaConf.to_container.

[dasturge]

there is also pydantic.dataclasses.dataclass, which is meant as a drop-in replacement for the stdlib dataclass.

https://pydantic-docs.helpmanual.io/usage/dataclasses/

this might serve to add some pydantic features while still being omegaconf compatible. I may try this with a project I'm currently working on, I'll try and remember to report back any findings.

[Jasha10]

I seem to recall that I tried OmegaConf+Pydantic at some point and that it worked.
Edit: see the answers to this stackoverflow question about OmegaConf+Pydantic