Support: Discussions | Gitter chat
Merges yaml configuration files, preserving comments and whitespaces. Assumed to be used for microservice configuration updates.
By default, targets the most common use-case: add all new properties, without removing or changing existing values. Lists not merged because list is a value and all current values must remain.
Comments preserved using custom (simple) yaml parser. Snakeyaml parser used for source files validation, comments parser self-validation (compares parse trees) and result validation.
Due to complete validation, merged file correctness is guaranteed.
Supports:
- Multiline values (all syntax variations)
- Reformatting (changed paddings in new config in both directions)
- Quoted property names with escaped characters (according to yaml spec)
- Properties reordering according to new config
- Variables replacement in new config before merge (environment-specific config adoption)
- Current values remove (e.g. deprecated values or for value replacement)
- Object list items update (lists not merged, but new properties could be added to list items)
- Backup current configuration (only if content changes)
- Migration testing (dry run in cli and testing api)
IMPORTANT: this is not a general-purpose yaml merge tool because yaml features like multiple documents in one file and object references are not supported (only common subset of features, used in configs)
Original config:
# top comment
# something
prop:
one: 1
two: 2
lists:
# sub comment
list:
- one
- two
obj:
- one: 1
two: 2
large: multi-line
value
# trailing commentUpdate file:
# changed comment
# something else
prop:
one: 3
two: 3
three: 3 # new property
lists:
# changed sub comment
list: # padding changed
- two # order changed (ignored)
- one
- three # new value ignored
obj:
- two: 2
one: 1
three: 3 # new value
large: multi-line
value
# changed trailing commentMerge result:
# changed comment
# something else
prop:
one: 1
two: 2
three: 3 # new property
lists:
# changed sub comment
list:
- one
- two
obj:
- two: 2
one: 1
three: 3 # new value
large: multi-line
value
# changed trailing commentMerge report (shown by cli modules):
Configuration: /var/somewhere/config.yml (185 bytes, 23 lines)
Updated from source of 497 bytes, 25 lines
Resulted in 351 bytes, 25 lines
Added from new file:
prop/three 7 | three: 3 # new property
lists/obj[0]/three 20 | three: 3 # new value
Could be used as:
- Library through API (embedding tool or perform testing migration)
- Command line util (fat jar, native binary or docker image)
- Dropwizard plugin (same arguments as in cli module)
Read exact module's readme for setup instructions.
- All values from current configuration preserved
- All new properties added
- Order of properties from update file used (current config properties could be re-ordered)
- For existing values, top comments updated from new config (if property exists in new config).
- If new property does not contain any comment - old comment is preserved
- In-line comments (after value) not updated (and so may be used for persistent marks)
- Trailing comment at the end of file preserved as-is (not updated)
- Paddings applied from update file (original config could be re-formatted)
- Padding change works in both directions (reduce or enlarge)
- Possible whitespace between property name and colon is removed (
name : valuebecomesname: value) - Property writing style taken from target file: e.g. if property was quoted and not quoted in new file then it would be not quoted in merged file
- Lists not merged
- Object list items could be updated with new properties (if matched item found)
- Dash style for object item could change in both directions: empty dash (property on new line) or property on the same line with dash
- Flow style lists and objects considered as single value and not merged:
[1, 2, 3]or{ one:1, two: 2 }
| Case | Current config | Update file | Merge result |
|---|---|---|---|
| New properties added |
one: 1
two: 2
|
one: 3
two: 3
three: 3
|
one: 1
two: 2
three: 3
|
| Order changed |
one: 1
two: 2
|
three: 3
two: 3
one: 3
|
three: 3
two: 2
one: 1
|
| Padding changed |
one:
two: 2
|
one:
two: 3
|
one:
two: 2
|
| Comment updated |
one:
# Old comment
two: 2
|
one:
# New comment
two: 3
|
one:
# New comment
two: 2
|
| Style changed |
"one": 1
|
one: 1
|
one: 1
|
| List not merged, but padding updated |
list:
- one
- two
|
list:
- one
- three
|
list:
- one
- two
|
| Object list item updated |
list:
- one: 1
two: 2
|
list:
- one: 1
two: 2
three: 3
|
list:
- one: 1
two: 2
three: 3
|
| List declaration style could change |
list:
- one: 1
two: 2
|
list:
-
one: 1
two: 2
|
list:
-
one: 1
two: 2
|
Object list items updated because in some cases lists could be quite complex and contain updatable configuration blocks.
Items match is searched by "the most similarity": selects item containing more properties with the same value.
- Item containing property with different value would never match
- If item contains sub-lists then at least one list item in current list must match
- Scalar list values ignored (only object lists counted)
- Only exact match counts: if multiple items match with the same amount of matched properties then no item would be updated (avoid guessing, work only on exact matches)
| Case | List item | Candidates | Match |
|---|---|---|---|
| Match by value |
one: 1
two: 2
|
- one: 1
two: 3
- one: 1
two: 2
|
Item 2 |
| Match by "weight" |
one: 1
two: 2
|
- one: 1
three: 3
- one: 1
two: 2
three: 3
|
Item 2 (technically both matches, but first item with one and second item with 2 props) |
| Multiple matches |
one: 1
|
- one: 1
two: 3
- one: 1
two: 2
|
No |
| Scalar list ignored |
one: 1
sub:
- a
- b
|
- one: 1
sub:
- c
- d
- one: 1
sub:
- e
- f
|
No (both matches, scalar list values ignored) |
| Sub list (one sub list element matched) |
one: 1
sub:
- a: 1
|
- one: 1
sub:
- a: 2
- a: 1
- one: 1
sub:
- b: 1
- b: 2
|
Item 1 (sub list has (at least) one match) |
Update config could contain variables with syntax: #{name}.
Such syntax used instead of more common ${..} because:
- Config may rely on environment variables and so will collide with replacing variables
- Such syntax recognized as comment and so not processed value would be empty value after parsing.
obj:
some: #{name}IMPORTANT: only known placeholders would be replaced (unknown variables will remain unprocessed)
Feature supposed to be used to "adopt" config for current environment before processing. Cli modules would automatically use environment variables.
- Read update config
- replace variables
- parse with snakeyaml (validation)
- parse with comments parser
- compare trees (self-validation)
- Read current config
- if not exists, simply copy update file
- parse with snakeyaml and comments parsers and compare trees
- Perform merge
- Write to tmp file
- Read merged result with snakeyaml (validate file correctness)
- Check all new properties added and current values preserved (validation)
- Backup current config (in the same folder)
- Replace config
Two last steps performed only if file content changes, otherwise current config remain untouched (to avoid producing redundant backups).
Parser assume everything above property as property comment (even empty lines):
# Large comment
# With empty lines
prop: 1Exactly because of this comment is always replaced from new file.
For example, assume reordering case:
Original config:
# Large header
# property comment
prop1: 1
prop2: 2New config:
# Large header
prop2: 2
# property comment
prop1: 1Without using comments from new file, entire header would be moved below the first property, but with comments update header would remain untouched.
But still, some edge cases possible (with removed properties).
Parser stores property values as everything after colon (including possible in-line comments). The same for multi-line values.
This way, current value could be restored exactly the same as it was in the original file (to avoid re-formatting).
And because of this in-line comments are not recognized and so could "survive" update.
