Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No space on device throws interessting error message #578

Closed
r3ap3rpy opened this issue Apr 10, 2019 · 8 comments

Comments

Projects
None yet
2 participants
@r3ap3rpy
Copy link

commented Apr 10, 2019

I'm using mitogen 0.2.6 on a CentOS system (Linux version 3.10.0-957.10.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Mon Mar 18 15:06:45 UTC 2019)
The ansible is at version 2.7.9, the python is the default which comes with the OS. No hacking or any modification happened to either ansible/python or mitogen.

The problem I have is when I use the mitogen with ANY strategies enabled and it runs against a linux system which has ran out of space.

With mitogen enabled this is the output:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: 'paths': '\n '.join(paths),
fatal: []: FAILED! =>
msg: Unexpected failure during module execution.
stdout: ''

When I disable mitogen:

fatal: []: UNREACHABLE! => changed=false
msg: |-
mkdir: cannot create directory /home/ansible/.ansible/tmp/ansible-tmp-1554900772.42-86330059306148: No space left on device
unreachable: true

@dw

This comment has been minimized.

Copy link
Owner

commented Apr 10, 2019

Ouch! I think that is occurring early on while finding a stable temporary directory. Would you mind re-running with "-vvv" and pasting the full exception?

@dw

This comment has been minimized.

Copy link
Owner

commented Apr 10, 2019

I guess I can reproduce locally by making a user with a homedir on a tiny ramdisk

@r3ap3rpy

This comment has been minimized.

Copy link
Author

commented Apr 10, 2019

Yes you can reproduce it, however I turned up -vvv and rerun, this is the output.
[mux 27919] 05:21:36.566109 W mitogen: mitogen.service.Pool(0x7f6d8b347910, size=32, th='mitogen.service.Pool.7f6d8b347910.worker-0'): call error: Message(0, 1, 0, 110, 1000, "\x80\x02X'\x00\x00\x00ansible_mitogen.services.ContextServiceq\x01X\x03"..540): exceptions.KeyError: u'namelist'
File "", line 3107, in _dispatch_one
File "master:/etc/ansible/mitogen-0.2.6/ansible_mitogen/target.py", line 383, in init_child
good_temp_dir = find_good_temp_dir(candidate_temp_dirs)
File "master:/etc/ansible/mitogen-0.2.6/ansible_mitogen/target.py", line 331, in find_good_temp_dir
'paths': '\n '.join(paths),

[WARNING]:
[task 27954] 05:21:36.568856 W ansible_mitogen.connection: Connection failed; stack configuration was:
({u'kwargs': {u'check_host_keys': u'ignore',
u'compression': True,
u'connect_timeout': 10,
u'hostname': u'',
u'identities_only': False,
u'identity_file': None,
u'password': None,
u'port': None,
u'python_path': [u'/usr/bin/python'],
u'ssh_args': [u'-C',
u'-o',
u'ControlMaster=auto',
u'-o',
u'ControlPersist=60s'],
u'ssh_debug_level': None,
u'ssh_path': u'ssh',
u'username': None},
u'method': u'ssh'},)

The full traceback is:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 140, in run
res = self._execute()
File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 612, in _execute
result = self._handler.run(task_vars=variables)
File "/etc/ansible/mitogen-0.2.6/ansible_mitogen/mixins.py", line 116, in run
return super(ActionModuleMixin, self).run(tmp, task_vars)
File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 46, in run
result = merge_hash(result, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))
File "/etc/ansible/mitogen-0.2.6/ansible_mitogen/mixins.py", line 354, in _execute_module
self._temp_file_gibberish(module_args, wrap_async)
File "/etc/ansible/mitogen-0.2.6/ansible_mitogen/mixins.py", line 333, in _temp_file_gibberish
self._connection.get_good_temp_dir()
File "/etc/ansible/mitogen-0.2.6/ansible_mitogen/connection.py", line 702, in get_good_temp_dir
self._connect()
File "/etc/ansible/mitogen-0.2.6/ansible_mitogen/connection.py", line 721, in _connect
self._connect_stack(stack)
File "/etc/ansible/mitogen-0.2.6/ansible_mitogen/connection.py", line 675, in _connect_stack
stack=mitogen.utils.cast(list(stack)),
File "/etc/ansible/mitogen-0.2.6/mitogen/core.py", line 1859, in call_service
return recv.get().unpickle()
File "/etc/ansible/mitogen-0.2.6/mitogen/core.py", line 835, in unpickle
raise obj
CallError: exceptions.KeyError: u'namelist'
File "", line 3107, in _dispatch_one
File "master:/etc/ansible/mitogen-0.2.6/ansible_mitogen/target.py", line 383, in init_child
good_temp_dir = find_good_temp_dir(candidate_temp_dirs)
File "master:/etc/ansible/mitogen-0.2.6/ansible_mitogen/target.py", line 331, in find_good_temp_dir
'paths': '\n '.join(paths),

| FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}

@dw

This comment has been minimized.

Copy link
Owner

commented Apr 10, 2019

Aah, this is already fixed on master :) I'll keep the ticket open to remember to update the changelog, and run a quick reproduction to ensure there are no other problems when out of disk space. Thanks for reporting this!

@r3ap3rpy

This comment has been minimized.

Copy link
Author

commented Apr 10, 2019

My bad for not downloading from git. By the way I used the download link from here: https://networkgenomics.com/ansible/ the link was: https://networkgenomics.com/try/mitogen-0.2.6.tar.gz

Is it not always the actual master?

Anyways this project is freaking amazing!!!!! I am using it in my daily work and at least 2x speed increase was noticed be it a playbook or a random ad-hoc command! Keep up the good work! Love from Hungary !

@dw

This comment has been minimized.

Copy link
Owner

commented Apr 10, 2019

0.2.6 is the correct default release just now. Master is generally quite stable, but it can be unstable before a release -- there are two huge branches waiting to land at present, so there is always a risk 'master' means 'code nobody else is using' if you download it on the wrong day. There are regularly big branches landing that need some time to settle down.

You can use master for now, or wait another week or so for 0.2.7, a new release is coming to handle some OpenStack fixes

Thanks again for reporting this :)

@r3ap3rpy

This comment has been minimized.

Copy link
Author

commented Apr 10, 2019

Great! Thanks for the info, I will wait for the next release.

@dw dw added user-reported ansible and removed ansible labels May 11, 2019

dw added a commit that referenced this issue Jun 3, 2019

@dw

This comment has been minimized.

Copy link
Owner

commented Jun 3, 2019

Thanks for reporting this!

@dw dw closed this Jun 3, 2019

dw added a commit that referenced this issue Jun 4, 2019

Merge remote-tracking branch 'origin/ci280'
* origin/ci280:
  ci: Another round of fixes for random Ansible UI breakage in 2.7/2.8
  ci: work around various broken aspects of Travis VM image
  Use virtualenv Python for stub connections to workaround problem
  ci: Ansible 2.8 requires Python 2.7.
  tests: add 2.8 format async error timeout message
  ansible: prevent tempfile.mkstemp() leaks.
  update gitignore again
  ci: try bumping more Travis jobs to Ansible 2.8.
  add .*.pid to gitignore
  tests: allow running without hdrhistograms library.
  issue #578: update Changelog.
  travis: exclude docs-master from CI
  issue #589: ensure real FileService/PushFileService are in the docs
  issue #589: ensure real FileService/PushFileService are in the docs
  docs: add new contributor entry
  issue #589: remove outdated/incomplete examples
  issue #589: split services example out and make it run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.