subversion module: fail on https checkout

[miniwark]

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible_module_subversion

ANSIBLE VERSION

2.4.3.0

CONFIGURATION

DEFAULT_ASK_PASS(/home/moss/Documents/Developpement/Ansible-playbooks/ansible.cfg) = True
DEFAULT_HOST_LIST(/home/moss/Documents/Developpement/Ansible-playbooks/ansible.cfg) = [u'/home/moss/Documents/Developpement/Ansible-playbooks/hosts']

OS / ENVIRONMENT

• From: Ubuntu 16.04
• Manage: Debian 9.2
• Subversion version on manage: 1.9.5 (r1770682)

SUMMARY

The subversion module fail with error "'NoneType' object has no attribute 'group'"

STEPS TO REPRODUCE

With this task:

- name: Svn checkout of /usr/local/sbin/scripts
  subversion:
    repo: https://forge.reynholm-industries.co.uk/path/torepo/directory/trunk/scripts
    dest: /usr/local/sbin/script-cocktail
    username: "mauricemoss"
    password: "<redacted>"

(the top repo is forge.reynholm-industries.co.uk/path/torepo )

EXPECTED RESULTS

The repo is being checkout and Ansible give a green light

ACTUAL RESULTS

The checkout is done correctly on the managed host
but Ansible fail with:

Using module file /usr/lib/python2.7/dist-packages/ansible/modules/source_control/subversion.py
<moss.reynholm-industries.co.uk> ESTABLISH SSH CONNECTION FOR USER: None
<moss.reynholm-industries.co.uk> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o ConnectTimeout=10 -o ControlPath=/home/moss/.ansible/cp/493c6155b5 moss.reynholm-industries.co.uk '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<moss.reynholm-industries.co.uk> (0, '/home/moss\n', '')
<moss.reynholm-industries.co.uk> ESTABLISH SSH CONNECTION FOR USER: None
<moss.reynholm-industries.co.uk> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o ConnectTimeout=10 -o ControlPath=/home/moss/.ansible/cp/493c6155b5 moss.reynholm-industries.co.uk '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/moss/.ansible/tmp/ansible-tmp-1519208327.22-164280939884325 `" && echo ansible-tmp-1519208327.22-164280939884325="` echo /home/moss/.ansible/tmp/ansible-tmp-1519208327.22-164280939884325 `" ) && sleep 0'"'"''
<moss.reynholm-industries.co.uk> (0, 'ansible-tmp-1519208327.22-164280939884325=/home/moss/.ansible/tmp/ansible-tmp-1519208327.22-164280939884325\n', '')
<moss.reynholm-industries.co.uk> PUT /tmp/tmpmHJkjm TO /home/moss/.ansible/tmp/ansible-tmp-1519208327.22-164280939884325/subversion.py
<moss.reynholm-industries.co.uk> SSH: EXEC sshpass -d12 sftp -o BatchMode=no -b - -C -o ControlMaster=auto -o ControlPersist=60s -o ConnectTimeout=10 -o ControlPath=/home/moss/.ansible/cp/493c6155b5 '[dsi1app.oca.eu]'
<moss.reynholm-industries.co.uk> (0, 'sftp> put /tmp/tmpmHJkjm /home/moss/.ansible/tmp/ansible-tmp-1519208327.22-164280939884325/subversion.py\n', '')
<moss.reynholm-industries.co.uk> ESTABLISH SSH CONNECTION FOR USER: None
<moss.reynholm-industries.co.uk> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o ConnectTimeout=10 -o ControlPath=/home/moss/.ansible/cp/493c6155b5 moss.reynholm-industries.co.uk '/bin/sh -c '"'"'chmod u+x /home/moss/.ansible/tmp/ansible-tmp-1519208327.22-164280939884325/ /home/moss/.ansible/tmp/ansible-tmp-1519208327.22-164280939884325/subversion.py && sleep 0'"'"''
<moss.reynholm-industries.co.uk> (0, '', '')
<moss.reynholm-industries.co.uk> ESTABLISH SSH CONNECTION FOR USER: None
<moss.reynholm-industries.co.uk> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o ConnectTimeout=10 -o ControlPath=/home/moss/.ansible/cp/493c6155b5 -tt moss.reynholm-industries.co.uk '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=drrkxyafylgyufhguivcrimzobxvmthb] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-drrkxyafylgyufhguivcrimzobxvmthb; /usr/bin/python /home/moss/.ansible/tmp/ansible-tmp-1519208327.22-164280939884325/subversion.py; rm -rf "/home/moss/.ansible/tmp/ansible-tmp-1519208327.22-164280939884325/" > /dev/null 2>&1'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<moss.reynholm-industries.co.uk> (0, '\r\nTraceback (most recent call last):\r\n  File "/tmp/ansible_FqZ4hU/ansible_module_subversion.py", line 306, in <module>\r\n    main()\r\n  File "/tmp/ansible_FqZ4hU/ansible_module_subversion.py", line 284, in main\r\n    before = svn.get_revision()\r\n  File "/tmp/ansible_FqZ4hU/ansible_module_subversion.py", line 190, in get_revision\r\n    rev = re.search(r\'^Revision:.*$\', text, re.MULTILINE).group(0)\r\nAttributeError: \'NoneType\' object has no attribute \'group\'\r\n', 'Shared connection to moss.reynholm-industries.co.uk closed.\r\n')
fatal: [moss.reynholm-industries.co.uk]: FAILED! => {
    "changed": false, 
    "module_stderr": "Shared connection to moss.reynholm-industries.co.uk closed.\r\n", 
    "module_stdout": "\r\nTraceback (most recent call last):\r\n  File \"/tmp/ansible_FqZ4hU/ansible_module_subversion.py\", line 306, in <module>\r\n    main()\r\n  File \"/tmp/ansible_FqZ4hU/ansible_module_subversion.py\", line 284, in main\r\n    before = svn.get_revision()\r\n  File \"/tmp/ansible_FqZ4hU/ansible_module_subversion.py\", line 190, in get_revision\r\n    rev = re.search(r'^Revision:.*$', text, re.MULTILINE).group(0)\r\nAttributeError: 'NoneType' object has no attribute 'group'\r\n", 
    "msg": "MODULE FAILURE", 
    "rc": 0
}

Hi highly suspect a similar problem to issue #3255 because the managed host have French as default language.

By the way the svn result from command line is:

> /usr/bin/svn --non-interactive --trust-server-cert --no-auth-cache --username mauricemoss --password '<redacted>' checkout -r HEAD https://forge.reynholm-industries.co.uk/path/torepo/directory/trunk/scripts /usr/local/sbin/scripts
> Révision 14332 extraite.

[ansibot]

Files identified in the description:

• lib/ansible/modules/source_control/subversion.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibot]

cc @dsummersl
click here for bot help

[ansibot]

Files identified in the description:

• lib/ansible/modules/subversion.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[samdoran]

The command being run is essentially svn info [repo], the output of which is affected by LC_* variables. I can duplicate this failure by setting LC_ALL=es_ES.UTF-8. The output the becomes:

Ruta: /tmp/xml-crimson
Working Copy Root Path: /tmp/xml-crimson
URL: http://svn.apache.org/repos/asf/xml/crimson/trunk
Relative URL: ^/xml/crimson/trunk
Raíz del repositorio: http://svn.apache.org/repos/asf
UUID del repositorio: 13f79535-47bb-0310-9956-ffa450edef68
Revisión: 1889132
Tipo de nodo: directorio
Agendado: normal
Autor del último cambio: edwingo
Revisión del último cambio: 313192
Fecha de último cambio: 2002-04-22 20:09:22 -0400 (lun 22 de abr de 2002)

The regexp that is parsing this output is '^Revision:.*$' which will not match anything. This results in a None object, which has no group() attribute.

That regular expression is used in several places in the module.

[bcoca]

always need to force 'C' locale when scraping ....

[samdoran]

Yup. I was just looking at doing that. 😁

[samdoran]

The solution is we need to explicitly set the locale in the module. I'll put together a PR.

[samdoran]

Turns out we already set this, just not completely. LC_ALL needs to be set as well.

ansible/lib/ansible/modules/subversion.py

Line 298 in 7fd989d

module.run_command_environ_update = dict(LANG='C', LC_MESSAGES='C')

[samdoran]

Wow, there's a lot of history here.

ansible/ansible-modules-core#3257 - added dict(LANG='C', LC_ALL='C', LC_MESSAGES='C', LC_CTYPE='C') which broke paths with non-ASCII characters.

ansible/ansible-modules-core#4358 - partially reverted the above change, which is how the code is today.

Setting LC_ALL='C' like I was thinking breaks paths with non-ASCII characters in the path. 🤦