combine filter: fine list handling (option b)

[tchernomax]

SUMMARY

see:

• combine filter: allow merging dict and list #57623
• Public Ansible Project Meeting Agenda - Jun 2019 community#474 (comment)
• Public Ansible Project Meeting Agenda - Jun 2019 community#474 (comment)
• fixes Deep merge of dictionaries #59386
• fixes Appending or "stacking" variable lists #17714
  Solution (b)

args:

• recursive: same as before
• list_merge:
  • keep: same as before
  • append, prepend: list are appended
  • append_np, prepend_np: only elements that isn't already present in left list are append (_np stands for "non present")

README: will come if this implementation is validated.

I didn't implement 'uniq' as I think 'append_np' and 'prepend_np' are more useful, and are equivalent to unique if list are already filtered from duplication.

non backward compatible: [dict1, [dict2]] | combine doesn't work anymore, because I thought it shouldn't. If you disagree, flatten(terms, levels=1) → flatten(terms) and it's fixed.

ISSUE TYPE

• Feature Pull Request

COMPONENT NAME

component =lib/ansible/plugins/filter/core.py

ADDITIONAL INFORMATION

max@mde-oxalide % cat toto.yml
- hosts: localhost
  gather_facts: false
  vars:
    toto_default:
      a:
        a:
          a: default_value
          b: default_value
      b:
        - 1
        - 1
        - 2
    toto:
      a:
        a:
          b: toto_value
          c: toto_value
      b:
        - 2
        - 3
        - 3
        - 4: toto_value
  tasks:
    - name: recursive=False, list_merge='keep'
      debug:
        msg: "{{toto_default|toto_combine(toto, recursive=False, list_merge='keep')}}"
    - name: recursive=False, list_merge='append'
      debug:
        msg: "{{toto_default|toto_combine(toto, recursive=False, list_merge='append')}}"
    - name: recursive=False, list_merge='prepend'
      debug:
        msg: "{{toto_default|toto_combine(toto, recursive=False, list_merge='prepend')}}"
    - name: recursive=False, list_merge='append_np'
      debug:
        msg: "{{toto_default|toto_combine(toto, recursive=False, list_merge='append_np')}}"
    - name: recursive=False, list_merge='prepend_np'
      debug:
        msg: "{{toto_default|toto_combine(toto, recursive=False, list_merge='prepend_np')}}"

    - name: recursive=True, list_merge='keep'
      debug:
        msg: "{{toto_default|toto_combine(toto, recursive=True, list_merge='keep')}}"
    - name: recursive=True, list_merge='append'
      debug:
        msg: "{{toto_default|toto_combine(toto, recursive=True, list_merge='append')}}"
    - name: recursive=True, list_merge='prepend'
      debug:
        msg: "{{toto_default|toto_combine(toto, recursive=True, list_merge='prepend')}}"
    - name: recursive=True, list_merge='append_np'
      debug:
        msg: "{{toto_default|toto_combine(toto, recursive=True, list_merge='append_np')}}"
    - name: recursive=True, list_merge='prepend_np'
      debug:
        msg: "{{toto_default|toto_combine(toto, recursive=True, list_merge='prepend_np')}}"

PLAY [localhost] ***********************************************************************

TASK [recursive=False, list_merge='keep'] **********************************************
ok: [localhost] => {
    "msg": {
        "a": {
            "a": {
                "b": "toto_value",
                "c": "toto_value"
            }
        },
        "b": [
            2,
            3,
            3,
            {
                "4": "toto_value"
            }
        ]
    }
}

TASK [recursive=False, list_merge='append'] ********************************************
ok: [localhost] => {
    "msg": {
        "a": {
            "a": {
                "b": "toto_value",
                "c": "toto_value"
            }
        },
        "b": [
            1,
            1,
            2,
            2,
            3,
            3,
            {
                "4": "toto_value"
            }
        ]
    }
}

TASK [recursive=False, list_merge='prepend'] *******************************************
ok: [localhost] => {
    "msg": {
        "a": {
            "a": {
                "b": "toto_value",
                "c": "toto_value"
            }
        },
        "b": [
            1,
            1,
            2,
            2,
            3,
            3,
            {
                "4": "toto_value"
            }
        ]
    }
}

TASK [recursive=False, list_merge='append_np'] *****************************************
ok: [localhost] => {
    "msg": {
        "a": {
            "a": {
                "b": "toto_value",
                "c": "toto_value"
            }
        },
        "b": [
            2,
            3,
            3,
            {
                "4": "toto_value"
            },
            1,
            1
        ]
    }
}

TASK [recursive=False, list_merge='prepend_np'] ****************************************
ok: [localhost] => {
    "msg": {
        "a": {
            "a": {
                "b": "toto_value",
                "c": "toto_value"
            }
        },
        "b": [
            1,
            1,
            2,
            3,
            3,
            {
                "4": "toto_value"
            }
        ]
    }
}

TASK [recursive=True, list_merge='keep'] ***********************************************
ok: [localhost] => {
    "msg": {
        "a": {
            "a": {
                "a": "default_value",
                "b": "toto_value",
                "c": "toto_value"
            }
        },
        "b": [
            2,
            3,
            3,
            {
                "4": "toto_value"
            }
        ]
    }
}

TASK [recursive=True, list_merge='append'] *********************************************
ok: [localhost] => {
    "msg": {
        "a": {
            "a": {
                "a": "default_value",
                "b": "toto_value",
                "c": "toto_value"
            }
        },
        "b": [
            1,
            1,
            2,
            2,
            3,
            3,
            {
                "4": "toto_value"
            }
        ]
    }
}

TASK [recursive=True, list_merge='prepend'] ********************************************
ok: [localhost] => {
    "msg": {
        "a": {
            "a": {
                "a": "default_value",
                "b": "toto_value",
                "c": "toto_value"
            }
        },
        "b": [
            1,
            1,
            2,
            2,
            3,
            3,
            {
                "4": "toto_value"
            }
        ]
    }
}

TASK [recursive=True, list_merge='append_np'] ******************************************
ok: [localhost] => {
    "msg": {
        "a": {
            "a": {
                "a": "default_value",
                "b": "toto_value",
                "c": "toto_value"
            }
        },
        "b": [
            2,
            3,
            3,
            {
                "4": "toto_value"
            },
            1,
            1
        ]
    }
}

TASK [recursive=True, list_merge='prepend_np'] *****************************************
ok: [localhost] => {
    "msg": {
        "a": {
            "a": {
                "a": "default_value",
                "b": "toto_value",
                "c": "toto_value"
            }
        },
        "b": [
            1,
            1,
            2,
            3,
            3,
            {
                "4": "toto_value"
            }
        ]
    }
}

PLAY RECAP *****************************************************************************
localhost                  : ok=10   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

[ansibot]

The test ansible-test sanity --test pylint [explain] failed with 1 error:

lib/ansible/plugins/filter/core.py:314:8: undefined-variable Undefined variable '_validate_mutable_mappings'

The test ansible-test sanity --test pep8 [explain] failed with 3 errors:

lib/ansible/plugins/filter/core.py:320:17: E117 over-indented
lib/ansible/plugins/filter/core.py:362:22: E261 at least two spaces before inline comment
lib/ansible/plugins/filter/core.py:374:1: E302 expected 2 blank lines, found 1

click here for bot help

[tchernomax]

ping @bcoca (I made the change you asked for)

[samccann]

@tchernomax can you please rebase so I can review the docs portion of this PR? thanks!

[tchernomax]

Can someone relaunch the CI, I think the fail has nothing to do with my commit.

ping @samccann @felixfontein

[felixfontein]

CI restarted.

[tchernomax]

thanks felix

@samccann can you review ?

(it would be awesome to see this one year old PR moving forward !)

[felixfontein]

LGTM

[samccann]

Docs LTGM

[samccann]

Thanks @tchernomax for your patience on this Ansible fix!