New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] move callback processing into it's own process #47899

Open
wants to merge 12 commits into
base: devel
from

Conversation

Projects
None yet
3 participants
@sivel
Member

sivel commented Oct 31, 2018

SUMMARY

move callback processing into it's own process

ISSUE TYPE
  • Feature Pull Request
COMPONENT NAME
ANSIBLE VERSION

ADDITIONAL INFORMATION

@sivel

This comment has been minimized.

Member

sivel commented Oct 31, 2018

bot_skip

@bcoca

This comment has been minimized.

Member

bcoca commented Oct 31, 2018

nice, also, sharing _callback_queue with the workers would allow for easy extension of callback events and action/connection/shell plugins to use that instead of display.

@sivel

This comment has been minimized.

Member

sivel commented Nov 1, 2018

I've resolved 2 of these so far, but it appears there are going to be a lot more.

Traceback (most recent call last):
  File "/usr/lib64/python2.6/multiprocessing/queues.py", line 242, in _feed
    send(obj)
RuntimeError: dictionary changed size during iteration

The cause is adding an object, task/host/play/result/etc to the queue for processing by the new process, but immediately following that, mutating the object before it is picked out of the queue.

Just mentioning it here for visibility.

The next comment will be collapsed, but will have the most recent example

@sivel

This comment was marked as resolved.

Member

sivel commented Nov 1, 2018

2 problems that I have traced:

('v2_playbook_on_task_start', (TASK: Test that retrieving all facts filtered to empty filter_spec works,), {'is_conditional': False})
('v2_runner_on_ok', (<ansible.executor.task_result.TaskResult object at 0x1089f9b90>,), {})

For v2_runner_on_ok it is task_result._task._parent that is being mutated.

sivel added some commits Nov 1, 2018

@sivel sivel force-pushed the sivel:callback-process branch 14 times, most recently from e637137 to 774da9e Nov 2, 2018

@sivel sivel force-pushed the sivel:callback-process branch from 774da9e to 27638c5 Nov 5, 2018

@sivel

This comment has been minimized.

Member

sivel commented Nov 5, 2018

This work is going to be paused for now. I think we have some larger problems that need solved here, instead of just trying to play whack-a-mole.

@mattclay mattclay added the ci_verified label Nov 5, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment