json_query filter fails when using the functions "contains", "starts_with", others

[ivandeex]

SUMMARY

This issue is migrated from ansible/ansible#27299

When using the json_query filter with the query [?interface.name=='irb.1111'].{interface: interface.name, address: address} everything functions fine and I get the correct results back. When that filter is changed to [?contains(interface.name,'irb')].{interface: interface.name, address: address} traceback errors with jmespath/functions.py start occurring that complain about invalid type for value.

ISSUE TYPE

• Bug Report

COMPONENT NAME

json_query

ANSIBLE VERSION

ansible 2.9.6

CONFIGURATION

Standard

OS / ENVIRONMENT

N/A

STEPS TO REPRODUCE

- name: json_query testing
  hosts: localhost
  connection: local
  gather_facts: no

  vars:
    json_file: "{{lookup('file','results.json') | from_json }}"
    jq_old: "[?interface.name=='irb.1111'].{interface: interface.name, address: address}"
    jq_new: "[?contains(interface.name,'irb')].{interface: interface.name, address: address}"

  tasks:

    - debug:
        var: item
      with_items: "{{ json_file.results | json_query(jq_new) }}"

EXPECTED RESULTS

[
  {
    "interface": "irb.1111",
    "address": "10.0.1.1/29"
  },
  {
    "interface": "irb.1114",
    "address": "10.0.2.1/29"
  },
  {
    "interface": "irb.1114",
    "address": "fe80::b/112"
  }
]

ACTUAL RESULTS

JMESPathTypeError: In function contains(), invalid type for value: irb.1111, expected one of: ['array', 'string'], received: "unknown"

fatal: [localhost]: FAILED! => {
    "failed": true,
    "msg": "Unexpected failure during module execution.",
    "stdout": ""
}

[ansibullbot]

Files identified in the description:
None

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibullbot]

Files identified in the description:

• plugins/filter/json_query.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[imjoseangel]

@ivandeex

Use the following to convert the data properly for now. I think the issue comes from the way ansible manages the JSON files.

    - debug:
        var: item
      with_items: "{{ json_file.results | to_json | from_json | json_query(jq_old) }}"

    - debug:
        var: item
      with_items: "{{ json_file.results | to_json | from_json | json_query(jq_new) }}"

The plugin just opens the result and parses with jmespath. When doing the same with Python directly, something like:

import json
import jmespath

expr = "results[?contains(interface.name,'1114')]"

openrojson = open("results.json", "r")
data = json.loads(openrojson.read())
outputorig = jmespath.search(expr, data)

It works fine. IMHO, this issue is more related to Ansible Core than the json_query.py plugin.

[ivandeex]

Thank you for workaround. I'd love to see this workaround described in the module documentation, since this issue is pretty old and going to stay for a while. Unless you have a fix to merge soon, of course.

[zeitounator]

It looks like maintenance as restarted on jmespath side and the author actually asked a question on the corresponding related ticket. This has drifted a lot so I'm not sure anyone implicated at start is still following this. Meanwhile, looks like we have an open window to get this fixed. Is someone still activelly following this issue ?

[ivandeex]

I am following jmespath/jmespath.py#158 and #320, and would love to see it fixed eventually. I'm sure I'm not the only one.

[Akasurde]

I added documentation around this in PR ansible/ansible#72821. Please feel free to comment.

[Akasurde]

resolved_by_pr ansible/ansible#72821