Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace config validation with pydantic validation #332

Closed
sverhoeven opened this issue Jan 23, 2023 · 1 comment
Closed

Replace config validation with pydantic validation #332

sverhoeven opened this issue Jan 23, 2023 · 1 comment

Comments

@sverhoeven
Copy link
Member

In #324 I wanted to print a deprecation message when container_engine: singularity was used in the config file. So I wanted to add an extra validation rule.

Current validation code is copied from matplotlib and is not easy to extend. It would be nice to use pydantic.

Something like

ContainerEngineEnum(str, Enum):
  docker= 'docker'
  apptainer = 'apptainer'
  singularity = 'singularity'

class Config(BaseModel):
  container_engine:ContainerEngineEnum = ContainerEngineEnum.docker
  parameterset_dir: DirectoryPath
  parameter_sets: dict[str,ParameterSet]
  ...

  @validator('container_engine')
  def singularity_deprecated(self, value):
    if value == ContainerEngineEnum.singularity:
       warnings.warn(
            "singularity is deprecated, use apptainer instead",
            DeprecationWarning
        )

raw_config = <dict loaded from yaml string>
config = Config(**raw_config)
@Peter9192
Copy link
Collaborator

I would like this very much. In the example, I think the Enum object is a bit overkill, why not just use Literal['docker', 'apptainer', 'singularity']?

sverhoeven added a commit that referenced this issue Feb 21, 2023
@sverhoeven sverhoeven mentioned this issue Feb 21, 2023
3 tasks
Peter9192 added a commit that referenced this issue Mar 22, 2023
* Require gprc4bmi version with BmiClientApptainer class

* Replace Singularity with Apptainer

* Fix link

* Use pydantic for config validation

Refs #332

* Drop unused validator

* Fix available_parameter_sets()

* Drop tests for non-existing code

* Move tests for parametersetdb to own sub dir

* Move test fixtures for parametersetdb to own file

* Implement overwrite

* Add tests for Config

* Numba 0.56.4 does not support Python 3.11

* Bump versions of pre-commit

Was getting
```
RuntimeError: The Poetry configuration is invalid:
            - [extras.pipfile_deprecated_finder.2] 'pip-shims<=0.3.4' does not match '^[a-zA-Z-_.0-9]+$'

```

* Use CFG.bla instead of CFG['bla']

Also
* made . the default value for directories
* renamed Config class to Configuration, as mypy was confused by nested Config.Config class
* added filename to validation error
* create dir before assigning to CFG field
* docstrings for Configuration fields
* dropped test in tests/parameter_sets/__init__.py, as it rewrote my config file

* Prepend parameterset_dir to parameter set paths

* Make ParameterSet and ExampleParameterSet pydantic models

* Fix tests

* Fix tests even more

* Fix lisvap apptainer check

* Update changelog

* Test docs

* Mention pydantic in changelog

* Remember py3.11 is not supported due to mamba

* Bump versions of pre-commit

Was getting
```
RuntimeError: The Poetry configuration is invalid:
            - [extras.pipfile_deprecated_finder.2] 'pip-shims<=0.3.4' does not match '^[a-zA-Z-_.0-9]+$'

```

* Add expand user for paths + forbid extra attribs

To sync config code with docs

* Bump versions of pre-commit

Was getting
```
RuntimeError: The Poetry configuration is invalid:
            - [extras.pipfile_deprecated_finder.2] 'pip-shims<=0.3.4' does not match '^[a-zA-Z-_.0-9]+$'

```

* Use latest grpc4bmi

* Repalce basic_modeling_interface with bmipy

* Start to centralize way to start container

* Move starting of container to own module + each model declares versions/images in same way

* Fix tests

* Simplify typings

* Register models as plugins

* fixup merge

* fixup plugins

* mypy happy

* fix lisvap test

* Replace singularity with apptainer in example notebooks

* Install pydantic in mypy precommit hook

Otherwise get:
setup.cfg:121: error: Error importing plugin "pydantic.mypy": No module named 'pydantic'

* XDG_CONFIG_HOME env var should not have ~/ prepended to it.

* Move content of src/ewatercycle/config/_config_object.py to src/ewatercycle/config/__init__.py

Also added link from setup docs to config api docs.

* Fix import

* Implement __str__, __repr__ in AbstractModel and DefaultForcing similar to ParameterSet

* Depend on grpc4bmi>=0.4.0

* Test __str__ of forcing subclasses

* Dont pin protobuf here, grpc4bmi does it already

* Use old typing as python 3.8 does not implement PEP 585

* Extract start_apptainer_container() and start_docker_container() methods

* Update CHANGELOG

* Make AbstractModel also work when self.bmi is not an OptionalDestBmi instance

* Fix more tests + use newer mypy in precommit

* Also add pydantic to nbqa pre commit hook

---------

Co-authored-by: Stefan Verhoeven <s.verhoeven@esciencecenter.nl>
Co-authored-by: Stefan Verhoeven <stefan.verhoeven@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants