Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add YAML support #267

Merged
merged 2 commits into from Jan 18, 2023
Merged

Add YAML support #267

merged 2 commits into from Jan 18, 2023

Conversation

jcrist
Copy link
Owner

@jcrist jcrist commented Jan 18, 2023

This adds a new msgspec.yaml submodule for encoding/decoding YAML messages using the type system provided by msgspec. This can be useful for configuration files, or for validating YAML before encoding it in some other format.

For example, in a past job I could have used this to load and validate k8s manifests from YAML, before processing and sending them to the k8s server as JSON.

This relies on the third-party PyYAML library for YAML support. A yaml extra has been added for providing this dependency.

For now only encode and decode functions have been provided; no Encoder/Decoder types. The builtin msgpack/json integration can benefit from reusing those objects, but the overhead of PyYAML and the type conversations makes the use of such types useless beyond interface compatibility. We can add them later if a user need arises.

This adds a new `msgspec.yaml` submodule for encoding/decoding YAML
messages using the type system provided by msgspec. This can be useful
for configuration files, or for validating YAML before encoding it in
some other format. For example, in a past job I could have used this to
load and validate k8s manifests from YAML, before processing and sending
them to the k8s server as JSON.

This relies on the third-party PyYAML library for YAML support. A `yaml`
extra has been added for providing this dependency.

For now only `encode` and `decode` functions have been provided; no
`Encoder`/`Decoder` types. The builtin `msgpack`/`json` integration can
benefit from reusing those objects, but the overhead of PyYAML and the
type conversations makes the use of such types useless beyond interface
compatibility. We can add them later if a user need arises.
@jcrist
Copy link
Owner Author

jcrist commented Jan 18, 2023

Example usage:

In [1]: import msgspec

In [2]: class Config(msgspec.Struct):
   ...:     host: str
   ...:     port: int
   ...:     threads: int = 1
   ...: 

In [3]: msg = """
   ...: host: 127.0.0.1
   ...: port: 8786
   ...: """

In [4]: config = msgspec.yaml.decode(msg, type=Config)  # load and validate yaml

In [5]: config
Out[5]: Config(host='127.0.0.1', port=8786, threads=1)

In [6]: bad_msg = """
   ...: host: 127.0.0.1
   ...: port: oops
   ...: """

In [7]: msgspec.yaml.decode(bad_msg, type=Config)
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
Cell In[7], line 1
----> 1 msgspec.yaml.decode(bad_msg, type=Config)

File ~/Code/msgspec/msgspec/yaml.py:165, in decode(buf, type, dec_hook)
    163 if type is Any:
    164     return obj
--> 165 return _from_builtins(
    166     obj, type, builtin_types=(_datetime.datetime, _datetime.date), dec_hook=dec_hook
    167 )

ValidationError: Expected `int`, got `str` - at `$.port`

@jcrist
Copy link
Owner Author

jcrist commented Jan 18, 2023

This PR adds the code, tests, docstrings, and API docs for YAML support. Adding it to the prose docs (e.g. "Usage") will require a reorganization of the docs content, and will be done in a follow-up PR.

@jcrist jcrist merged commit e66ab96 into main Jan 18, 2023
@jcrist jcrist deleted the yaml branch January 18, 2023 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant