Multiple Include in the same level

[jbek7]

Consider this, a.yml:

a: 1

And b.yml:

b: 2

Template ab.yml:

ab:
  !Include 1.yml
  !Include 2.yml

emrichen -f ab.yml is stuck and hangs.

But expected, ab.yml:

ab:
   a: 1
   b: 2

It might be related to this #20

[japsu]

Remember Emrichen works within the YAML data model. You are constructing a dict with the key ab. What should be its value after deserializing the template but before applying the Emrichen transformation? If there were only one !Include, the value would be that !Include tag, ie. an instance of emrichen.tags.Include. For two !Include tags though, in YAML there is no such thing as putting two tags one after another on the same level. You could make a list in which each element is a tag, or a dictionary in which each value is a tag, but the YAML syntax for that is different (see below).

Another issue: "stuck and hang" is because there is no template file in your emrichen -f ab.yml invocation. The -f command line parameter defines variable files, not template files. Because there is no template file, Emrichen expects the template data on the standard input.

Try this:

1.yml

a: 5
b: 6

2.yml

b: 7
c: 8

ab.yml

ab: !Merge
  - !Include 1.yml
  - !Include 2.yml

Here ab.yml contains a single document. This document is a dictionary with the key ab whose value is the YAML tag !Merge that receives a list as its parameter (YAML tags can only have one parameter, but that parameter can be also be list or a dict). Here the parameter to !Merge is the list [!Include 1.yml, !Include 2.yml].

!Merge takes a list of dicts and merges them. If there are duplicate keys, the latter takes precedence. When you run emrichen ab.yml (notice the absence of -f), this first expands into

ab: !Merge
  - a: 5
    b: 6
  - b: 7
    c: 8

and finally the output should be

ab:
  a: 5
  b: 7
  c: 8

I assume this is what you wanted to do? Please close if this solved your problem.

[jbek7]

Thanks! Yes, my output supposed to be:

ab:
   - a: 1
   - b: 2

I tried to use the

ab: !Merge
  - !Include 1.yml
  - !Include 2.yml

But I kept getting ValueError: too many values to unpack (expected 2). And I went ahead and created another tag called MultiInput that globs directory and flat merges all dicts into an array.

I'm using the library for concourse pipeline generation, for example, I've many resources that repeat but can be templatized like:
resources/git-resources.in.yaml:

!Loop
  over: !Var deployments
  as: props
  template:
      name: !Format "source-{props.env}-{props.flavor}"
      type: git
      source:
        uri: git@github.domain.com:org/project
        branch: !Lookup props.branch
        private_key:  ((ghe-private-key))
      check_every: !Lookup props.check_every
      webhook_token: ((ghe-webhook-token))

And resources/helm-resources.in.yaml:

!Loop
  over: !Var deployments
  as: props
  template:
      name: !Format "project-{props.env}-{props.flavor}"
      type: helm
      source:
        release: !Format "project-{props.env}-{props.flavor}"
        cluster_url: https://myurl:6443
        namespace: my-ns
        token: ((token))

And in pipeline.yaml, I do:

resources:
    !MultiInclude ./resources/*.in.yaml

And it produces the result as:

resources:
- name: source-dev-vanilla
  type: git
  source:
    .....
- name: source-dev-choco
  type: git
  source:
      ....
- name: version-dev-vanilla
  type: semver
  source:
   ......
- name: version-dev-choco
  type: semver
  source:
#etc...

[japsu]

I checked the example I gave, it works. No ValueError: too many values to unpack (expected 2).

[japsu]

Hi @jbek7 and thanks for the idea! We had a design discussion and @akx went ahead and implemented !IncludeGlob that is similar in purpose to your !MultiInclude. !IncludeGlob will be available in the next release of Emrichen.

[jbek7]

Thanks @japsu and @akx ! I've tried using that option while it accepts multiple files with glob. However it still doesn't support concourse resource pipeline use case.

So I created a repo with examples where it's failing: https://github.com/jbek7/emrichen-concourse-use-case

[japsu]

Thanks @jbek7 for an excellent example. This is fixed easily by changing !IncludeGlob to !Concat,IncludeGlob. , here stands for tag composition as YAML won't let you do !Concat !IncludeGlob … – for some reason you can't have a tag as the parameter to another tag so we need to work around it with a composition syntax of our own.

So this

resources:
  !IncludeGlob ./resources/*.in.yaml

becomes this

resources:
  !Concat,IncludeGlob ./resources/*.in.yaml

and now resources is a list of dicts instead of a list of list of dicts.

[jbek7]

Awesome!! I overlooked Concat tag. Thanks a lot, @japsu !!