Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge arrays #35

Closed
tmaier opened this issue Jan 3, 2017 · 15 comments

Comments

@tmaier
Copy link

commented Jan 3, 2017

As a user, I want to be able to re-use and extend aliased arrays.

Example yaml:

array1: &my_array_alias
  - foo
  - bar

array2:
  << *my_array_alias
  - baz

Expected output:

array1:
  - foo
  - bar

array2:
  - foo
  - bar
  - baz
@ingydotnet

This comment has been minimized.

Copy link
Member

commented Jan 4, 2017

This will probably never happen for a few reasons.

First off, merge keys for objects use regular YAML syntax. '<<' is a string key which some YAML loaders perform a merge operation on, and often in a manner incompatible with other YAML loaders. Many loaders treat a << key as a normal string key. Your YAML happens to be valid, but the value of array2 is the string '<< *my_array_alias - baz'.

In YAML 1.3, loaders will be encouraged not to support merge keys by default. It will become an optional feature. YAML is a data language in the same sense that JSON is. Programatic features should not have ever been encouraged.

You might want to do something like this:

array1: &my_array_alias
  - foo
  - bar

array2:
  - <: *my_array_alias
  - baz

and then post process < keys to do what you want. Depending on your loader implementation you might be able to hook into the loading process to do that.

@ingydotnet ingydotnet closed this Jan 4, 2017
@tmaier

This comment has been minimized.

Copy link
Author

commented Jan 4, 2017

Hi @ingydotnet, thanks for your explanation. Much appreciated.

@amomchilov

This comment has been minimized.

Copy link

commented Nov 6, 2018

@ingydotnet Now every application that needs this feature will have it... except it will be proprietary, and it would have wasted the developer's time. To what end?

It's absolutely acceptable to want to say "hey, this config over here is the same as the one over there, except toss this in too."

@ulope

This comment has been minimized.

Copy link

commented Jan 24, 2019

Please reconsider this decision.
Yaml is used in many places where one has no control over the parsing side.

Very common examples are:

  • CI systems (Travis CI, Circle CI, AppVeyor, Jenkins, etc.).
  • Docker Stack / Swarm / Compose configurations

Without the ability to merge lists complex configurations often devolve into copy&paste hells that are very difficult to maintain.

@perlpunk

This comment has been minimized.

Copy link
Member

commented Jan 24, 2019

First: Yes, I can see why one would want to have such a feature. I would want it too.

Here I just want to talk about this specific suggestion in this issue.

So let's talk about merge keys for mappings first.

As @ingydotnet said, << is just a string key. It's special meaning is enabled by applying a schema.

# implicit merge key
<<: *alias
# string key
'<<': *alias
# explicit merge key
!!merge '<<': *alias

That means the parser does not know anything about merging. It is not syntax.
Only the loader knows what to do with the scalar elements. It also decides if it should convert the string TRUE to a boolean or not.

Now to the suggested merge array.
The original suggestion was:

array2:
  << *my_array_alias
  - baz

First, this is already valid syntax. It is a simple string '<< *my_array_alias - baz'.
Second, to support this, we would also have to invent a completely new syntax element.
YAML syntax is already (too) complex, and suggestions for 1.3 exist to make it simpler.
From experience we know that the complex syntax and the absence of a test suite resulted in many parser differences.

I think it is not a good idea to invent a new syntax element for a single programmatic feature.

There are suggestions on how to get more generic programmatic features into YAML that would just add one new syntax element (set) and leave the implementation to a higher level.

@creatorKoo

This comment has been minimized.

Copy link

commented Apr 29, 2019

I really want to re-open this issue.

@bluepeter

This comment has been minimized.

Copy link

commented May 8, 2019

I really want to re-open this issue.

+1

@antegulin

This comment has been minimized.

Copy link

commented May 22, 2019

Please reconsider this decision.
Yaml is used in many places where one has no control over the parsing side.

Very common examples are:

  • CI systems (Travis CI, Circle CI, AppVeyor, Jenkins, etc.).
  • Docker Stack / Swarm / Compose configurations

Without the ability to merge lists complex configurations often devolve into copy&paste hells that are very difficult to maintain.

I can confirm, this is happening to me. I'm using both Gitlab CI & Bitbucket Pipelines, and I always end up copy/pasting blocks of configuration, resulting in huge unreadable yaml files.

+1

@perlpunk

This comment has been minimized.

Copy link
Member

commented May 27, 2019

I think it was explained well enough that:

  • We understand that merging sequences is something useful
  • The proposed syntax in this issue is close to impossible. (1)

I personally understand that it would be useful to have a standard way to do this, and new features in YAML 1.3 sound nice but don't exist yet. But this issue talks about a specific syntax.

(1) Everyone who wants this syntax, feel free to implement a parser that does this. But first look at the YAML Test Matrix http://matrix.yaml.io/ and think about why we have so few parsers that are working correctly, and if you really feel able to implement a correct parser (at least as correct as existing ones) with this new feature.

@perlpunk

This comment has been minimized.

Copy link
Member

commented May 27, 2019

I created issue #48

@christhomas

This comment has been minimized.

Copy link

commented Sep 11, 2019

This is terrible reasoning. Not wanting to put symbols that mean things into the language because they are data processing? Hmmm, all symbols represent actions

an_array:
  - 1
  - 2
  - 3

Means create an array called "an_array", then each - symbol means add a numerical index to the array it's defined under with the value: 1, 2, 3

Adding <<: or whatever here is just another syntax element, just because it's data processing doesn't mean it's not worthy and this helps in a lot of really basic cases that have real world benefits.

Saying that because something is data processing, well, damn, all symbols represent data processing. All elements do things according to what the specification says it means

@perlpunk

This comment has been minimized.

Copy link
Member

commented Sep 11, 2019

@christhomas I think it was already explained well that << is not a special syntax element. To the YAML parser it's just a string.
The constructor turns it into something special (if supported and enabled).

Please read the explanations above and the linked issue #48

@christhomas

This comment has been minimized.

Copy link

commented Sep 11, 2019

Then it should be a special syntax element, because it is in the position of syntax and not data, << could be a part of a string, but if you're in an array syntax and you can't find a "-" or a "key:" then "<<:" is obviously not a string.

this_array:
  - a sequence key
  key: a value for a mapping (key: being the key and this string is the value)
  <<: this is obviously a completely different thing

Each time the processor is scanning through, it's taking care of indentation, it comes to <<: and then it thinks that this is a string?

Does anybody use << as a string whilst defining arrays? Anybody? and under what conditions? I can't even think why I'd want to use << as a string key in an array

@perlpunk

This comment has been minimized.

Copy link
Member

commented Sep 11, 2019

This YAML is already invalid:

sequence:
  - item1
  key: value

But let's take your <<: proposal:

sequence:
  - item1
  <<: *alias

This would be a new syntax, because currently this is invalid. I can give you reasons why this still isn't practicable and would need a lot of changes in existing YAML processors. An obvious problem though is: let's say you want that merge element at the top of the sequence:

sequence:
  <<: *alias
  - item2

Now the node under sequence will be parsed as a mapping, and the - item2 is now invalid.

@christhomas

This comment has been minimized.

Copy link

commented Sep 11, 2019

I wasn't giving a single example, I was giving an array with the three possible options below it, either a sequence key (-), a mapping (key:), or a merge key (<<:)

@henryiii henryiii referenced this issue Sep 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.