Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add _target_wrapper_ to instantiate and add pydantic parsing layer #666

Merged
merged 12 commits into from Apr 2, 2024

Conversation

rsokl
Copy link
Contributor

@rsokl rsokl commented Apr 1, 2024

Adds a new _target_wrapper_ argument to hydra_zen.instantiate, which enables users to introduce a custom wrapping layer of processing to the recursive instantiation process. I.e. specifying instantiate(cfg, _target_wrapper_=f) means that for every target(*args, **kwargs) that would have been called by the recursive instantiation process, instead f(target)(*args, **kwargs) will be called 1. This is also plumbed through hydra_zen.zen via the new instantiation_wrapper argument.

This is a powerful addition to the instantiate & zen APIs as it means that special parsing/validation can be introduced at the layer where CLI-configured values are passed to the constructors of the objects that are actually used by your program.

Along these lines, this PR also adds from hydra_zen.third_party.pydantic.pydantic_parser, which can be passed in as a _target_wrapper_. This adds pydantic-backed runtime type checking and annotation-based value conversions to all instantiation layers2.

E.g.

from hydra_zen import builds, instantiate
from hydra_zen.third_party.pydantic import pydantic_parser

from pydantic import PositiveInt

def f(x: PositiveInt): return x

good_conf = builds(f, x=10)
bad_conf = builds(f, x=-3)

>>> instantiate(good_conf, _target_wrapper_=pydantic_parser)
10
>>> instantiate(bad_conf, _target_wrapper_=pydantic_parser)
ValidationError: 1 validation error for f. x: Input should be greater than 0

Tearing out Hydra's type-checking and replacing it with pydantic parsing

First, tell hydra-zen's config store to strip out all type-annotations / structure from the stored configs. This ensures that Hydra is unable to get in our way when it comes to type-h

from hydra_zen import ZenStore
from hydra_zen.wrapper import default_to_config
from dataclasses import is_dataclass
from omegaconf import OmegaConf

def destructure(x):
    x = default_to_config(x)  # apply the default auto-config logic of `store`
    if is_dataclass(x):
        # Recursively converts: 
        # dataclass -> omegaconf-dict (backed by dataclass types) 
        #           -> dict -> omegaconf dict (no types)
        return OmegaConf.create(OmegaConf.to_container(OmegaConf.create(x)))  # type: ignore
    return x

store = ZenStore()
store = store(to_config=destructure)  # all stored configs will have their type-info stripped out

next, whenever you make your zen-wrapped task function, specify pydantic_parser as the instantiation_wrapper:

zen(task_function, 
   instantiation_wrapper=pydantic_parser,
).hydra_main(...)

and... that's it! Now only pydantic will perform runtime type checking and value conversions. Note that all of this type-information lives in the targets of the configs that you create and in the signature of the task function that is wrapped by zen.

Footnotes

  1. This is arguably the proper place for things like type-checking / value validation to occur. Hydra runs validation at the config-composition phase, which means that configs need to be annotated with the types of other configs; this leads to config-centric code producing more config-centric code. It is much simpler and more powerful to defer all validation to the actual code that is being configured and run by the Hydra application.

  2. Pydantic has more robust support for type-checking than Hydra. It can also perform annotation-based conversions, so that, e.g., a string-valued path specified at the CLI can be automatically be converted to a pathlib.Path during instantiation.

@rsokl rsokl added the enhancement New feature or request label Apr 1, 2024
@rsokl rsokl merged commit 9a1781b into main Apr 2, 2024
18 checks passed
@rsokl rsokl deleted the pydantic-prototype branch April 2, 2024 01:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant