"include" directive support to embed yaml structure in yaml

[bedge]

There are cfg-mgmt type systems that extend YAML with an "include" directive to facilitate creating distributed/hierarchical data structures that provide

• data deduplication
• inheritance
• overloading
  eg:
  atmos describe subcommand: https://docs.cloudposse.com/fundamentals/atmos/#subcommand-describe
  gitlab include directive:
• https://microfluidics.utoronto.ca/gitlab/help/ci/yaml/includes.md
• https://docs.gitlab.com/ee/ci/yaml/

This is a common use case, but with no unifying implementation and, to use it, one incurs significant additional baggage (see above utility scopes).

yq supports file operators on the command line: file-operators, as well as the explode operator, but this could be embedded in the yaml directly with an include directive to eliminate the need to explicit merges, and be able to express the inheritance relationships within the yaml itself rather than via external file aggregation commands.

This feels like something that would be beneficial to include in generic YAML handling, such that it's handling and merge policies are defined in one place, rather than "tool specific variations", or "external to the yaml content".

To say it another way, I don't want to have to do this myself (lazy), and I also don't want to pull in all of atmos, or similar, just to benefit from it's include & deep merge support.

One last pass at the sales job - One could express the relationship between yaml files within the files themselves, rather than requiring external file aggregation commands.

There's other questions here that would be helped with this as well:
[how to combine multiple files, include only references.yml](#1291

And externally: All of these are ways around the fact that there's no include option within yaml itself:
How can I include a YAML file inside another?
Avoid external bash logic to control yaml expansion
Working With Multiple Files
My Cheatsheet for the “yq” Tool
Support for the include keyword to include other YAML files

Related: if include supported variable name expansion:
Use placeholders in YAML

Existing implementations:
Magnolia's !inherit, !include and !override

[mikefarah]

There's nothing official in yq to do that, but you can definitely come up with a naming convention and use yq expressions to automatically process the content.

a:
    yq_load_file: blah.yaml

yq '(.. | select(has("yq_load_file"))) |= load(.yq_load_file)'

This will allow you to add whatever template is required, and use yq to perform any transformations needed.

You could support multiple template commands by doing:

yq '
    (.. | select(has("yq_load_file"))) |= load(.yq_load_file)
|  (.. | select(has("yq_merge_base"))) |= load(.yq_merge_base) * load(.yq_merge_target)
'

And so on.