Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of disk space condition gives error: 'temp_path' is not defined #17215

Closed
geertn opened this issue Aug 24, 2016 · 6 comments · Fixed by #25394
Closed

Out of disk space condition gives error: 'temp_path' is not defined #17215

geertn opened this issue Aug 24, 2016 · 6 comments · Fixed by #25394

Comments

@geertn
Copy link

@geertn geertn commented Aug 24, 2016

ISSUE TYPE
  • Bug Report
COMPONENT NAME

core assembly

ANSIBLE VERSION
ansible 2.1.1.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
CONFIGURATION

Using ANSIBLE_KEEP_REMOTE_FILES=1:
fatal: [tw0]: UNREACHABLE! => {"changed": false, "msg": "mkdir: cannot create directory '/home/x/.ansible': No space left on device\n", "unreachable": true}

OS / ENVIRONMENT

Ubuntu 16.04

SUMMARY

When running ansible on a machine which has a 100% full disk the error is not indicative of the problem.

STEPS TO REPRODUCE
x@x:~/work-fl/cm-ansible$ ansible -i inventories/ota tw0 -m setup
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: NameError: name 'temp_path' is not defined
tw0 | FAILED! => {
    "changed": false, 
    "failed": true, 
    "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 123, in <module>\nNameError: name 'temp_path' is not defined\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "parsed": false
}
x@x:~/work-fl/cm-ansible$ ANSIBLE_KEEP_REMOTE_FILES=1 ansible -i inventories/ota tw0 -m setup 
tw0 | UNREACHABLE! => {
    "changed": false, 
    "msg": "mkdir: cannot create directory '/home/x/.ansible': No space left on device\n", 
    "unreachable": true
}
x@x:~/work-fl/cm-ansible$ 

EXPECTED RESULTS

I expect an error saying "No space left on device"

ACTUAL RESULTS
NameError: name 'temp_path' is not defined
@nitzmahone

This comment has been minimized.

Copy link
Member

@nitzmahone nitzmahone commented Aug 25, 2016

Might be an easy fix to the ansiballz wrapper, but surprised that mkdtemp() isn't giving us the "out of space" error (and in fact possibly is failing silently?)...

cc @abadger

@ansibot ansibot added the affects_2.1 label Sep 8, 2016
@jimi-c jimi-c added affects_2.2 and removed P2 labels Sep 23, 2016
@ralphct

This comment has been minimized.

Copy link

@ralphct ralphct commented Jan 13, 2017

I'm not sure if this is related, but I'm seeing similar errors when using 'become' on the git module. It looks like it's not even checking arguments before barfing. I got my test case down to this:

ansible my_server -m git -a "" -b --become-user=some_user -vvv

(obviously replace my_server and some_user with something valid). I get the following output:

Using /opt/sysadmin/ansible/ansible.cfg as config file
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/source_control/git.py
<my_host> ESTABLISH SSH CONNECTION FOR USER: None
<my_host> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r my_host '/bin/sh -c '"'"'sudo -H -S -n -u some_user /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-doquzrgecuimuufnucirjrcblyxtxgkf; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "<stdin>", line 133, in <module>
NameError: name 'temp_path' is not defined
my_host | FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_name": "git"
    },
    "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 133, in <module>\nNameError: name 'temp_path' is not defined\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE"
}

If I remove the 'become' then it works fine.

Getting Ansible to leave temporary files and some find/grep lead me to the /usr/lib/python2.7/site-packages/ansible/executor/module_common.py which contains the bit of code that's really got the problem. I found that if I initialised temp_path with "" I got an "IOError: No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/root']" error, which lead me to look at my /tmp and /var/tmp directories - neither of which were group/world writable. Fixed that, then all was well with Ansible once again.

I'd say 'real' errors are being swallowed somewhere, which cost me a couple of hours of head-scratching. It's doing the right thing, just not telling you what that is ;-)

Using ansible-2.2.0.0-4.el7.noarch.

@berislavlopac

This comment has been minimized.

Copy link
Contributor

@berislavlopac berislavlopac commented May 2, 2017

I'm having a similar problem on Ansible 2.3 as well, but I'm getting two different errors depending on the task. I have two playbooks, one with a simple sync:

---
- name: sync the application directory
  hosts: "{{ hostgroup }}"
  gather_facts: yes
  become: yes
  become_user: root
  remote_user: "{{ remote_user }}"
  tasks:
    - name: synchronize the working directory
      synchronize:
        src: ../../src/
        dest: "/opt/{{ application_name }}"
        rsync_opts: -F
  vars_files:
    - global_vars.yml

When run with vvv, this one gives this output:

Using /home/berislav/Development/myapp/ansible.cfg as config file

PLAYBOOK: test_sync.yml **************************************************************************************************************************************************************************************************************************
1 plays in operations/ansible/test_sync.yml

PLAY [sync the application directory] ************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************************
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/system/setup.py
<54.171.119.161> ESTABLISH SSH CONNECTION FOR USER: ubuntu
<54.171.119.161> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/berislav/.ssh/mykey.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 -o ControlPath=/home/berislav/.ansible/cp/86838d2333 54.171.119.161 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-dlbcigtyfiwhwomtorrzqumscyfrmmum; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<54.171.119.161> (1, '', 'Traceback (most recent call last):\n  File "<stdin>", line 136, in <module>\nNameError: name \'temp_path\' is not defined\n')
The full traceback is:
Traceback (most recent call last):
  File "<stdin>", line 136, in <module>
NameError: name 'temp_path' is not defined

fatal: [54.171.119.161]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 136, in <module>\nNameError: name 'temp_path' is not defined\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1
}
  to retry, use: --limit @/home/berislav/Development/myapp/operations/ansible/test_sync.retry

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************
54.171.119.161             : ok=0    changed=0    unreachable=0    failed=1   

Then there is the other playbook, simply running raw to install Python 2 on Xenial before the playbook is actually executed:

---
- hosts: "{{ hostgroup }}"
  gather_facts: no
  become: yes
  become_user: root
  remote_user: ubuntu
  tasks:
    - name: install python2
      raw: sudo apt-get -y install python-simplejson
  vars_files:
      - global_vars.yml

This one gives this output, on the same inventory:

Using /home/berislav/Development/myapp/ansible.cfg as config file

PLAYBOOK: test_raw.yml ***************************************************************************************************************************************************************************************************************************
1 plays in operations/ansible/test_raw.yml

PLAY [tag_App_myapp:&tag_Stage_dev:&tag_Version_1_0_0b] ***************************************************************************************************************************************************************************************
META: ran handlers

TASK [install python2] ***************************************************************************************************************************************************************************************************************************
task path: /home/berislav/Development/myapp/operations/ansible/test_raw.yml:8
<54.171.119.161> ESTABLISH SSH CONNECTION FOR USER: ubuntu
<54.171.119.161> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/berislav/.ssh/myapp.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 -o ControlPath=/home/berislav/.ansible/cp/86838d2333 -tt 54.171.119.161 'sudo -H -S -n -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-fwwbahzcnmnyzclwqcjryriinozvukwg; sudo apt-get -y install python-simplejson'"'"''
<54.171.119.161> (100, '\rReading package lists... 0%\r\rReading package lists... 0%\r\rReading package lists... 16%\r\rReading package lists... Error!\r\r\nE: Write error - write (28: No space left on device)\r\nE: IO Error saving source cache\r\nE: The package lists or status file could not be parsed or opened.\r\n', 'Shared connection to 54.171.119.161 closed.\r\n')
fatal: [54.171.119.161]: FAILED! => {
    "changed": true, 
    "failed": true, 
    "rc": 100, 
    "stderr": "Shared connection to 54.171.119.161 closed.\r\n", 
    "stdout": "\rReading package lists... 0%\r\rReading package lists... 0%\r\rReading package lists... 16%\r\rReading package lists... Error!\r\r\nE: Write error - write (28: No space left on device)\r\nE: IO Error saving source cache\r\nE: The package lists or status file could not be parsed or opened.\r\n", 
    "stdout_lines": [
        "", 
        "Reading package lists... 0%", 
        "", 
        "Reading package lists... 0%", 
        "", 
        "Reading package lists... 16%", 
        "", 
        "Reading package lists... Error!", 
        "", 
        "E: Write error - write (28: No space left on device)", 
        "E: IO Error saving source cache", 
        "E: The package lists or status file could not be parsed or opened."
    ]
}
  to retry, use: --limit @/home/berislav/Development/myapp/operations/ansible/test_raw.retry

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************
54.171.119.161             : ok=0    changed=0    unreachable=0    failed=1   

I can see differences in details like task path, but I'm not familiar with ins and out of Ansible; maybe this will help find the cause.

@azzeddinefaik

This comment has been minimized.

Copy link

@azzeddinefaik azzeddinefaik commented May 25, 2017

+1
I got the same in here ! any update ?

@one1zero1one

This comment has been minimized.

Copy link

@one1zero1one one1zero1one commented Jun 5, 2017

+1 "the error is not indicative of the problem"
Had the same error for a RPi in the wild that had a full drive, took me a while to get to 'No space left on device' from 'temp_path' is not defined.

abadger added a commit to abadger/ansible that referenced this issue Jun 6, 2017
…pdir wasn't created

If the temp directory creation failed in mkdtemp then temp_path is never
given a value.  This would lead to a NameError exception which would
obfuscate the original error (out of disk space being a common one).  By
catching NameError, python will raise the original exception as we want.

Fixes ansible#17215
@abadger

This comment has been minimized.

Copy link
Member

@abadger abadger commented Jun 6, 2017

Made a PR thatI think will solve this if anyone would like to test it out #25394

abadger added a commit that referenced this issue Jun 7, 2017
…pdir wasn't created

If the temp directory creation failed in mkdtemp then temp_path is never
given a value.  This would lead to a NameError exception which would
obfuscate the original error (out of disk space being a common one).  By
catching NameError, python will raise the original exception as we want.

Fixes #17215

(cherry picked from commit 80c1765)
abadger added a commit that referenced this issue Jun 7, 2017
…pdir wasn't created

If the temp directory creation failed in mkdtemp then temp_path is never
given a value.  This would lead to a NameError exception which would
obfuscate the original error (out of disk space being a common one).  By
catching NameError, python will raise the original exception as we want.

Fixes #17215
@ansibot ansibot added bug and removed bug_report labels Mar 7, 2018
@ansible ansible locked and limited conversation to collaborators Apr 25, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
9 participants
You can’t perform that action at this time.