Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python3 - Intersect filter change in behavior (docs + code) #26494

Closed
evrardjp opened this issue Jul 6, 2017 · 2 comments · Fixed by #29171
Closed

python3 - Intersect filter change in behavior (docs + code) #26494

evrardjp opened this issue Jul 6, 2017 · 2 comments · Fixed by #29171
Labels
affects_2.3 This issue/PR affects Ansible v2.3 bug This issue/PR relates to a bug. c:plugins/filter python3 support:core This issue/PR relates to code supported by the Ansible Engineering Team.
Projects
Python 3 compatibility

Comments

@evrardjp
Copy link
Contributor

evrardjp commented Jul 6, 2017 •
edited

ISSUE TYPE
  • Bug Report
  • Documentation Report
COMPONENT NAME

intersect filter

ANSIBLE VERSION
ansible 2.3.1.0
  config file =
  configured module search path = Default w/o overrides
  python version = 3.6.1 (default, Mar 27 2017, 00:27:06) [GCC 6.3.1 20170306]
CONFIGURATION

Nothing special

OS / ENVIRONMENT

I am currently running ArchLinux, but same behavior caught with Ubuntu Xenial.
Different behavior with python3 and python2.

SUMMARY

First, the documentation about filters says (1) All these functions return a unique set from sets or lists.
You can read, on the same page, To get the intersection of 2 lists (unique list of all items in both).

I don't know if I should expect a set or a list if I intersect with a set. There is an expectation issue here.

However, fun stuff happens when you realize it doesn't behave the same way under python2 and python3.

I get an intersection under python2 and a warning (it doesn't work) in python3.

STEPS TO REPRODUCE
---
- hosts: nova_all
  connection: local
  gather_facts: no
  tasks:
    - debug:
        msg: "{{ groups['nova_all'] | intersect(ansible_play_hosts) }}"

Here is my inventory:

[all]
host1
localhost

[nova_all]
localhost
EXPECTED RESULTS

Output: "localhost"

ACTUAL RESULTS
TASK [debug] ************************************************************************************************************************************************************************
task path: /home/jean-philippe/evrardjp/pocs/intersect.yml:6
 11311 1499361217.87448: sending task start callback
 11311 1499361217.87454: entering _queue_task() for localhost/debug
 11311 1499361217.87462: Creating lock for debug
 11311 1499361217.87657: worker is 1 (out of 1 available)
 11311 1499361217.87697: exiting _queue_task() for localhost/debug
 11311 1499361217.87712: done queuing things up, now waiting for results queue to drain
 11311 1499361217.87720: waiting for pending results...
 11319 1499361217.90017: running TaskExecutor() for localhost/TASK: debug
 11319 1499361217.90069: in run()
 11319 1499361217.90111: calling self._execute()
 11319 1499361217.90326: Loading FilterModule 'core' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/core.py (found_in_cache=True, class_only=False)
 11319 1499361217.90348: Loading FilterModule 'ipaddr' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/ipaddr.py (found_in_cache=True, class_only=False)
 11319 1499361217.90362: Loading FilterModule 'json_query' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/json_query.py (found_in_cache=True, class_only=False)
 11319 1499361217.90374: Loading FilterModule 'mathstuff' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/mathstuff.py (found_in_cache=True, class_only=False)
 11319 1499361217.90420: Loading TestModule 'core' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/test/core.py (found_in_cache=True, class_only=False)
 11319 1499361217.90434: Loading TestModule 'files' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/test/files.py (found_in_cache=True, class_only=False)
 11319 1499361217.90446: Loading TestModule 'mathstuff' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/test/mathstuff.py (found_in_cache=True, class_only=False)
 11319 1499361217.90804: Loading FilterModule 'core' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/core.py (found_in_cache=True, class_only=False)
 11319 1499361217.90823: Loading FilterModule 'ipaddr' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/ipaddr.py (found_in_cache=True, class_only=False)
 11319 1499361217.90842: Loading FilterModule 'json_query' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/json_query.py (found_in_cache=True, class_only=False)
 11319 1499361217.90855: Loading FilterModule 'mathstuff' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/mathstuff.py (found_in_cache=True, class_only=False)
 11319 1499361217.90893: Loading TestModule 'core' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/test/core.py (found_in_cache=True, class_only=False)
 11319 1499361217.90906: Loading TestModule 'files' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/test/files.py (found_in_cache=True, class_only=False)
 11319 1499361217.90920: Loading TestModule 'mathstuff' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/test/mathstuff.py (found_in_cache=True, class_only=False)
 11319 1499361217.91143: Loading Connection 'local' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/connection/local.py
 11319 1499361217.91231: Loading ShellModule 'csh' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/shell/csh.py
 11319 1499361217.91306: Loading ShellModule 'fish' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/shell/fish.py
 11319 1499361217.91378: Loading ShellModule 'powershell' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/shell/powershell.py
 11319 1499361217.91420: Loading ShellModule 'sh' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/shell/sh.py
 11319 1499361217.91448: Loading ShellModule 'sh' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/shell/sh.py (found_in_cache=True, class_only=False)
 11319 1499361217.91468: Loading ActionModule 'debug' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/action/debug.py (searched paths: /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/action:/home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/action/__pycache__) (found_in_cache=True, class_only=False)
 11319 1499361217.91483: starting attempt loop
 11319 1499361217.91494: running the handler
 11319 1499361217.91506: handler run complete
 11319 1499361217.91526: attempt loop complete, returning result
 11319 1499361217.91537: _execute() done
 11319 1499361217.91546: dumping result to json
 11319 1499361217.91557: done dumping result, returning
 11319 1499361217.91568: done running TaskExecutor() for localhost/TASK: debug
 11319 1499361217.91584: sending task result
 11319 1499361217.91627: done sending task result
 11319 1499361217.91683: WORKER PROCESS EXITING
 [WARNING]: Failure using method (v2_runner_on_ok) in callback plugin (<ansible.plugins.callback.default.CallbackModule object at 0x7ffad0014a20>): Object of type 'set' is not JSON
serializable

 11311 1499361217.91893: Callback Exception:
  File "/home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/executor/task_queue_manager.py", line 373, in send_callback
    method(*args, **kwargs)
   File "/home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/callback/default.py", line 95, in v2_runner_on_ok
    msg += " => %s" % (self._dump_results(result._result),)
   File "/home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/callback/__init__.py", line 98, in _dump_results
    return json.dumps(abridged_result, indent=indent, ensure_ascii=False, sort_keys=sort_keys)
   File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
   File "/usr/lib64/python3.6/json/encoder.py", line 201, in encode
    chunks = list(chunks)
   File "/usr/lib64/python3.6/json/encoder.py", line 430, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
   File "/usr/lib64/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
   File "/usr/lib64/python3.6/json/encoder.py", line 437, in _iterencode
    o = _default(o)
   File "/usr/lib64/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)

 11311 1499361217.91910: no more pending results, returning what we have
 11311 1499361217.91917: results queue empty
 11311 1499361217.91923: checking for any_errors_fatal
 11311 1499361217.91930: done checking for any_errors_fatal
 11311 1499361217.91935: checking for max_fail_percentage
 11311 1499361217.91941: done checking for max_fail_percentage
 11311 1499361217.91947: checking to see if all hosts have failed and the running result is not ok
 11311 1499361217.91952: done checking to see if all hosts have failed
 11311 1499361217.91957: getting the remaining hosts for this loop
 11311 1499361217.91964: done getting the remaining hosts for this loop
 11311 1499361217.91972: building list of next tasks for hosts
 11311 1499361217.91979: getting the next task for host localhost
 11311 1499361217.91987: done getting next task for host localhost
 11311 1499361217.91997:  ^ task is: TASK: meta (flush_handlers)
 11311 1499361217.92004:  ^ state is: HOST STATE: block=3, task=1, rescue=0, always=0, run_state=ITERATING_TASKS, fail_state=FAILED_NONE, pending_setup=False, tasks child state? (None), rescue child state? (None), always child state? (None), did rescue? False, did start at task? False
 11311 1499361217.92010: done building task lists
@evrardjp evrardjp changed the title Intersect filter documentation is confusing Intersect filter documentation is confusing and the filter works differently under python3 Jul 6, 2017
@evrardjp evrardjp changed the title Intersect filter documentation is confusing and the filter works differently under python3 python3 - Intersect filter change in behavior (documentation is confusing and the filter works differently under python3) Jul 6, 2017
@evrardjp evrardjp changed the title python3 - Intersect filter change in behavior (documentation is confusing and the filter works differently under python3) python3 - Intersect filter change in behavior (docs + code) Jul 6, 2017
@ansibot ansibot added affects_2.3 This issue/PR affects Ansible v2.3 bug_report c:plugins/filter needs_triage Needs a first human triage before being processed. python3 support:core This issue/PR relates to code supported by the Ansible Engineering Team. labels Jul 6, 2017
@nitzmahone nitzmahone removed the needs_triage Needs a first human triage before being processed. label Jul 7, 2017
@odyssey4me
Copy link
Contributor

It appears to me that the same happens for the difference filter.

@evrardjp
Copy link
Contributor Author

evrardjp commented Jul 7, 2017

Probably all math ones?

@abadger abadger mentioned this issue Sep 9, 2017
Merged
abadger added a commit to abadger/ansible that referenced this issue Sep 10, 2017
@abadger
Fix set theory filters for python3
a9c2ffa 
The set theory filters need to use lists rather than generators on python3.

Also add unit tests for most of the mathstuff filters

Fixes ansible#26494
abadger added a commit that referenced this issue Sep 10, 2017
@abadger
Fix set theory filters for python3
8eab6a4 
The set theory filters need to use lists rather than generators on python3.

Also add unit tests for most of the mathstuff filters

Fixes #26494

(cherry picked from commit 75249e3)
@abadger abadger moved this from TODO to Done in Python 3 compatibility Sep 10, 2017
abadger added a commit that referenced this issue Sep 10, 2017
@abadger
Fix set theory filters for python3
75249e3 
The set theory filters need to use lists rather than generators on python3.

Also add unit tests for most of the mathstuff filters

Fixes #26494
Akasurde pushed a commit to Akasurde/ansible that referenced this issue Sep 10, 2017
@abadger @Akasurde
Fix set theory filters for python3
21092b3 
The set theory filters need to use lists rather than generators on python3.

Also add unit tests for most of the mathstuff filters

Fixes ansible#26494
prasadkatti pushed a commit to prasadkatti/ansible that referenced this issue Oct 1, 2017
@abadger
Fix set theory filters for python3
aa60c52 
The set theory filters need to use lists rather than generators on python3.

Also add unit tests for most of the mathstuff filters

Fixes ansible#26494
BondAnthony pushed a commit to BondAnthony/ansible that referenced this issue Oct 5, 2017
@abadger @BondAnthony
Fix set theory filters for python3
629842d 
The set theory filters need to use lists rather than generators on python3.

Also add unit tests for most of the mathstuff filters

Fixes ansible#26494
@ansibot ansibot added bug This issue/PR relates to a bug. and removed bug_report labels Mar 7, 2018
@ansible ansible locked and limited conversation to collaborators Apr 26, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
affects_2.3 This issue/PR affects Ansible v2.3 bug This issue/PR relates to a bug. c:plugins/filter python3 support:core This issue/PR relates to code supported by the Ansible Engineering Team.
Projects
No open projects
Python 3 compatibility
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix set theory filters for python3 abadger/ansible
4 participants
@odyssey4me @evrardjp @ansibot @nitzmahone