Using yamlpath to alter non-YAML dictionaries?

[AndydeCleyre]

Using YAML Paths to access and change specific, arbitrarily deep nodes, in a dict-like structure is super cool. Occasionally I find the fact that it's a YAML document, only supporting YAML-friendly values, to be restrictive.

For example, I have a small project that reads serialized data into a dict, only containing dicts, strings, and lists. From there, I use yamlpath to replace certain nodes with other types of values. Finally, I turn the result into a serialized form again (currently JSON/YAML/TOML), using a deep unstructuring method (via cattrs) that takes into account what types are supported by the target serialization form.

I found that I can skip creation of a proper YAML doc, and feed a plain dict to yamlpath's Processor, then change nested values with Processor.set_value, as long as the values are YAML-compatible.

In this case, the only (?) output type not supported by YAML is TOML's time (not datetime, plain time) type, so I am unable to insert time items into the dict this way.

A workaround for my case may be to set an unlikely-to-be-intentional str value, like a generated UUID or something, which the cattrs unstructure method will check each str value for the presence of, and when finding do its own conversion into a time or str format (depending on the target serialization format).

But I want to ask you, @wwkimball, if you think it might be worthwhile for yamlpath to have a mechanism by which to .set_value even if the value isn't supported by YAML, for cases in which the document/dict being altered is not going to be used directly to output YAML anyway.

I understand if that's plainly out of scope for this tool, which is made for managing proper YAML. But it also seems a shame to restrict the querying and setting abilities available here if they can reasonably be applied to non-YAML dicts.

What do you think?

[wwkimball]

Your question is interesting and could be a basis for expanding on implementations of the underlying YAML Path standard.

Indeed, the primary use-case of this project is to identify nodes, query nodes, and update nodes in YAML and compatible streams (files and pipes). This directly includes JSON because YAML is a superset of JSON, and indirectly supports other compatible data stream types (though I don't have any concrete tests for others). I recognize the temptation to want to extend YAML Path to other data stream types like TOML, and others. On that point, I am very often asked, "How do I use yamlpath on XML files?" I normally advise those asking about XML to use other purpose-built tools like xpath and xslt, but they very often come back with, "I'd rather use YAML Path syntax."

Rather than continue to dismiss this growing desire to use YAML Path against more data stream types, what can we do?

I can think of two options, today. First, consumers (like yourself) could monkeypatch select methods of the library to offer the additional capability(ies) you need. Second, I don't immediately imagine a trivial means of adding support for non-YAML data types. This is rooted in the fact that the implementation today is very tightly married to ruamel.yaml. In preserving its core use-cases, the yamlpath library can only safely offer what ruamel.yaml directly provides. By "safely", I'm referring to continuing to guarantee that yamlpath tools always properly emit YAML and JSON (the original intended use-case). But... can we add support for other data stream types?

My first thought takes us into the realm of extensions. This could look like a purely abstract core yamlpath library, offering deliberate hooks at the points of data parsing/loading, data change, and stream emission. The abstract core would continue to offer path processing, searching, and so on. As I imagine it, this would be a massive refactoring effort. We'd have to pull out ruamel.yaml and change its relationship to that of a just one concrete YAML/JSON consumer-level extension/implementation. At the end of that effort, we could offer other concrete implementations for TOML, XML, and any others.

That's my first take. As always, I'm open to anyone else's input, ideas, refinements, and so on.

[AndydeCleyre]

Hmm, thanks for this response. I had imagined something more like glom, where we operate on dicts/lists/any-python-thing, and then have a serialization process act on that, recursively. I have not thought at all about how comments can be supported, or I'm sure many other things that yamlpath and ruamel.yaml currently handle.

FWIW I did add TOML time support to my project using the string UUID marker strategy:

• https://github.com/AndydeCleyre/nestedtextto/blob/develop/nt2/casters.py#L78-L103
• https://github.com/AndydeCleyre/nestedtextto/blob/develop/nt2/converters.py#L36-L46

[wwkimball]

I'm glad you were able to trivially implement TOML time support while still using yamlpath! I might have tried encapsulating the time type by using a YAML timestamp with a 0001-01-01 date to indicate that only the time part is valuable (0001-01-01 is the minimum date which Python supports and is very unlikely to be a useful value to most users of otherwise valid timestamps). Your novel solution should work well.

As for glom, I've never seen that before. At its heart, yamlpath works very similarly; it indeed works recursively over complex embedded containers while parsing path segments in a loop. However, YAML is quite a bit more complex than native Python data types, so the yamlpath core loop has several more branches. The YAML Path parser is also quite a bit more complex to support more node matching capabilities than glom seems to support. While yamlpath has to be aware of more data types than glom, it does also -- as you've found -- support traversing the simpler native Python containers (dict, list, set), as well. This is deliberate to extend utility of the yamlpath module code as far as possible.

However, YAML requires explicit metadata support. XML (for example) would, too. Data streams using metadata have comments, attributes, user-defined data-types (YAML tags), namespaces, anchors/aliases, and white-space to preserve while round-tripping their streams. If we only had to deal with serialization, we could trivialize most of this. But yamlpath has a commitment to support full round-trips.

I'm still thinking about yamlpath version 4.x where the core handles native Python data types while extensions add handlers for YAML, JSON, TOML, XML, etc.