allow 'ignore_missing' for fileglob lookups

[philfry]

SUMMARY

There are situations where it's useful to hide the Unable to find ... in expected paths (use -vvvvv to see paths) message. This PR adds the ability to pass ignore_missing to the fileglob plugin. See below for example.

ISSUE TYPE

• Feature Pull Request

COMPONENT NAME

fileglob

ANSIBLE VERSION

ansible 2.8.0.dev0 (fileglob-ignore_missing e0389d0aab) last updated 2018/10/19 10:44:40 (GMT +200)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/phil/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/phil/projects/3rdparty/ansible/lib/ansible
  executable location = /home/phil/projects/3rdparty/ansible/bin/ansible
  python version = 2.7.15 (default, Sep 21 2018, 23:26:48) [GCC 8.1.1 20180712 (Red Hat 8.1.1-5)]

ADDITIONAL INFORMATION

echo -e "foo\nbar\nbaz" > inventory
for f in foo baz; do install -D /dev/null files/${f}/iptables; done
cat > test.yml <<EOF
---
- hosts: all
  gather_facts: no
  tasks:
    - name: "without ignore_missing"
      debug: var=item
      loop: '{{lookup("fileglob",
        "files/"+inventory_hostname+"/iptables",
        wantlist=true)}}'
    - name: "with ignore_missing"
      debug: var=item
      loop: '{{lookup("fileglob",
        "files/"+inventory_hostname+"/iptables",
        wantlist=true, ignore_missing=true)}}'
...
EOF
ansible-playbook -i inventory test.yml 

output:

PLAY [all] ********************************************************************************

TASK [without ignore_missing] *************************************************************
 [WARNING]: Unable to find 'files/bar' in expected paths (use -vvvvv to see paths)

ok: [foo] => (item=/home/phil/projects/3rdparty/ansible/tmp/files/foo/iptables) => {
    "item": "/home/phil/projects/3rdparty/ansible/tmp/files/foo/iptables"
}
ok: [baz] => (item=/home/phil/projects/3rdparty/ansible/tmp/files/baz/iptables) => {
    "item": "/home/phil/projects/3rdparty/ansible/tmp/files/baz/iptables"
}

TASK [with ignore_missing] ****************************************************************
ok: [foo] => (item=/home/phil/projects/3rdparty/ansible/tmp/files/foo/iptables) => {
    "item": "/home/phil/projects/3rdparty/ansible/tmp/files/foo/iptables"
}
ok: [baz] => (item=/home/phil/projects/3rdparty/ansible/tmp/files/baz/iptables) => {
    "item": "/home/phil/projects/3rdparty/ansible/tmp/files/baz/iptables"
}

PLAY RECAP ********************************************************************************
bar                        : ok=0    changed=0    unreachable=0    failed=0    skipped=2   
baz                        : ok=2    changed=0    unreachable=0    failed=0    skipped=0
foo                        : ok=2    changed=0    unreachable=0    failed=0    skipped=0

[ansibot]

Hi @philfry, thank you for submitting this pull-request!

click here for bot help

[bcoca]

this is redundant with the 'errors' option available to all lookups. https://docs.ansible.com/ansible/latest/plugins/lookup.html#using-lookup-plugins

[philfry]

this is redundant with the 'errors' option available to all lookups.

The errors option does not eleminate the warnings for me. Am I missing something?

---
- hosts: all
  gather_facts: no
  tasks:
    - name: "with errors=strict"
      debug: var=item
      loop: '{{lookup("fileglob",
        "files/"+inventory_hostname+"/iptables",
        wantlist=true, errors="strict")}}'
    - name: "with errors=ignore"
      debug: var=item
      loop: '{{lookup("fileglob",
        "files/"+inventory_hostname+"/iptables",
        wantlist=true, errors="ignore")}}'
    - name: "with errors=warn"
      debug: var=item
      loop: '{{lookup("fileglob",
        "files/"+inventory_hostname+"/iptables",
        wantlist=true, errors="warn")}}'
...

ansible-playbook -i inventory test2.yml 

PLAY [all] ***************************************************************************************************************************************************************************

TASK [with errors=strict] ************************************************************************************************************************************************************
 [WARNING]: Unable to find 'files/bar' in expected paths (use -vvvvv to see paths)

ok: [foo] => (item=/home/phil/projects/3rdparty/ansible/tmp/files/foo/iptables) => {
    "item": "/home/phil/projects/3rdparty/ansible/tmp/files/foo/iptables"
}
ok: [baz] => (item=/home/phil/projects/3rdparty/ansible/tmp/files/baz/iptables) => {
    "item": "/home/phil/projects/3rdparty/ansible/tmp/files/baz/iptables"
}

TASK [with errors=ignore] ************************************************************************************************************************************************************
 [WARNING]: Unable to find 'files/bar' in expected paths (use -vvvvv to see paths)

ok: [foo] => (item=/home/phil/projects/3rdparty/ansible/tmp/files/foo/iptables) => {
    "item": "/home/phil/projects/3rdparty/ansible/tmp/files/foo/iptables"
}
ok: [baz] => (item=/home/phil/projects/3rdparty/ansible/tmp/files/baz/iptables) => {
    "item": "/home/phil/projects/3rdparty/ansible/tmp/files/baz/iptables"
}

TASK [with errors=warn] **************************************************************************************************************************************************************
 [WARNING]: Unable to find 'files/bar' in expected paths (use -vvvvv to see paths)

ok: [foo] => (item=/home/phil/projects/3rdparty/ansible/tmp/files/foo/iptables) => {
    "item": "/home/phil/projects/3rdparty/ansible/tmp/files/foo/iptables"
}
ok: [baz] => (item=/home/phil/projects/3rdparty/ansible/tmp/files/baz/iptables) => {
    "item": "/home/phil/projects/3rdparty/ansible/tmp/files/baz/iptables"
}

PLAY RECAP ***************************************************************************************************************************************************************************
bar                        : ok=0    changed=0    unreachable=0    failed=0   
baz                        : ok=3    changed=0    unreachable=0    failed=0   
foo                        : ok=3    changed=0    unreachable=0    failed=0   

[bcoca]

ah, nvmd thought you wanted to handle an error, not a warning

[s-hertel]

@philfry You can test this with a runme.sh style test or by running ansible-playbook from within the play (you'd just need to escape variables so they're templated at the right time). I put together a test for you:

diff --git a/test/integration/targets/lookup_fileglob/non_existent/play.yml b/test/integration/targets/lookup_fileglob/non_existent/play.yml
index e92dff5acc..6733952991 100644
--- a/test/integration/targets/lookup_fileglob/non_existent/play.yml
+++ b/test/integration/targets/lookup_fileglob/non_existent/play.yml
@@ -4,3 +4,22 @@
     - name: fileglob should be empty
       assert:
         that: q("fileglob", seed) | length == 0
+
+    - name: Test ignore_missing
+      vars:
+        query_with_warnings: "query('fileglob', 'missing/file')"
+        query_without_warnings: "query('fileglob', 'missing/file', ignore_missing=true)"
+        warning: "Unable to find 'missing' in expected paths"
+      block:
+        - name: Test a warning is given for a missing path
+          command: ansible localhost -m debug -a msg="{{ '{{' + query_with_warnings + '}}' }}"
+          register: warn_for_missing
+
+        - name: Test ignoring the warning
+          command: ansible localhost -m debug -a msg="{{ '{{' + query_without_warnings + '}}' }}"
+          register: ignore_missing
+
+        - assert:
+            that:
+              - warning in warn_for_missing.stderr
+              - warning not in ignore_missing.stderr

But while I was looking into this, I feel like a new option is redundant with the general lookup option errors which is handled in the core engine. @bcoca what do you think about adding the errors option instead, defaulting to warn for backwards compatibilty? Since fileglob is handling missing files itself and always warns, this effectively means support is missing for both errors=strict and errors=ignore and only behaves like errors=warn.

I also noticed the first_found plugin has done basically the same thing - in addition to skipping a missing file with errors=warn|ignore, it has it's own skip option, which also seems redundant to me - but adding an option with either of the existing names might be nice for consistency.

[bcoca]

first_found had preexisting options, we should really pass in the errors to the file finding function and suppress warnings when apropos.

[webknjaz]

The branch needs to be rebased for the CI to pick it up.

[webknjaz]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).