Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiplying documents (merge) doesn't work when it's not the first expression #1675

Open
tnozicka opened this issue May 26, 2023 · 1 comment
Labels

Comments

@tnozicka
Copy link

Describe the bug
. as $item ireduce({}; . * $item) can't merge multiple documents when preceded by other commands.

Version of yq: v4.34.1
Operating system: linux
Installed via: go install

Input Yaml

$ input_data=$( echo -e 'unique:  value\ndata:\n  foo:\n    alpha\n---\ndata:\n  bar:\n    beta' )
unique: value
data:
  foo: alpha
---
data:
  bar: beta

Command

$ yq eval-all -e '. as $all | $all as $item ireduce({}; . * $item)' < <( echo "${input_data}" )

Actual behavior

unique: value
data:
  foo: alpha
data:
  bar: beta

Expected behavior

unique: value
data:
  foo: alpha
  bar: beta

Additional context
Based on https://mikefarah.gitbook.io/yq/operators/reduce#merge-all-yaml-files-together

When . as $item ireduce is the first command it all works as expected.

$ yq eval-all -e '. as $item ireduce({}; . * $item)' < <( echo "${input_data}" )
unique: value
data:
  foo: alpha
  bar: beta

When the ireduce is preceded by an expression, it no longer works the same way and instead of merging it will output duplicate keys (.data). This is similar to what happens when using eval instead of eval-all. (Maybe it's no longer evaluated as multifile internally or something.)

Another manifestation may be a document duplication with a similar command, or maybe I misunderstand how it should work there.

$ yq eval-all -e '. as $all | . as $item ireduce({}; . * $item)' < <( echo "${input_data}" )
unique: value
data:
  foo: alpha
  bar: beta
unique: value
data:
  foo: alpha
  bar: beta

I tried to simplify the command for a reproducer but for a context I was after something like
yq eval-all -e '. as $all | select(fi==0) | .data = null | $all as $item ireduce({}; . * $item)' < <( echo "${input_data}" )
that would merge keys on .data from all files except the first one which should be discarded.

@mikefarah
Copy link
Owner

This is actually intentional when using variables - despite how odd it looks.

The expression ". as $all | ..." is actually like a for loop (this is how jq does it) - where each "." (in this case a document) is set to "$all" then the "..." expression is evaluated with that value. That's why you're seeing two outputs, it went through the loop twice, once with the first doc, then the next.

You can 'trick' by doing wrapping the '.' in an array:

 yq ea '[.] as $all | ($all | .[]) as $item ireduce({}; . * $item)' file.yaml

This will now work because its wrapping both documents into an array, setting that as "$all" and then in the loop, it splats it out again.

@mikefarah mikefarah reopened this May 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants