Error using become in a Windows environment

[jezol]

Hi, guys,
We're deploying different playbooks in windows environment and facing some strange behavior of become function. When we're first trying to use newly added to local admins user account, it fails with "incorrect username password error".
The second turn of playbook always works like a charm with no errors or problems at all.

ISSUE TYPE

• Bug Report

COMPONENT NAME

lib/ansible/playbook/become.py

ANSIBLE VERSION

ansible 2.9.6
  config file = /home/oz/iac/iac/playbooks/single_server/ansible.cfg
  configured module search path = ['/home/oz/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.9 (default, Nov  7 2019, 10:44:02) [GCC 8.3.0]

CONFIGURATION

ANSIBLE_PIPELINING(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = True
ANSIBLE_SSH_ARGS(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = -o ControlMaster=auto -o ControlPersist=30s
ANSIBLE_SSH_CONTROL_PATH(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = /tmp/ansible-%%r@%%h:%%p
CACHE_PLUGIN(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = jsonfile
CACHE_PLUGIN_CONNECTION(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = ./.ansible/cache
CACHE_PLUGIN_TIMEOUT(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = 3600
DEFAULT_GATHERING(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = smart
DEFAULT_HOST_LIST(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = ['/home/oz/iac/iac/playbooks/single_server/hosts.yaml']
DEFAULT_LOAD_CALLBACK_PLUGINS(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = True
DEFAULT_LOG_PATH(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = /home/oz/iac/iac/playbooks/single_server/log/ansible.log
DEFAULT_ROLES_PATH(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = ['/home/oz/iac/iac/roles']
DEFAULT_SCP_IF_SSH(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = True
DEFAULT_STDOUT_CALLBACK(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = yaml
DEFAULT_TIMEOUT(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = 60
DEFAULT_UNDEFINED_VAR_BEHAVIOR(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = True
HOST_KEY_CHECKING(/home/oz/iac/iac/playbooks/single_server/ansible.cfg) = False

OS / ENVIRONMENT

VMware virtual machines with clean Windows Server 2019 Standard OS

STEPS TO REPRODUCE

Add chocolatey with win_chocolatey module with become.

- name: Windows | Chocolatey 
  win_chocolatey:
    name: chocolatey
    source: 'https://chocolatey.org/'
  become: yes
  become_user: "{{ admin_login }}"
  become_method: runas

EXPECTED RESULTS

Successful choco installed

ACTUAL RESULTS

An exception occurred during task execution. To see the full traceback, use -vvv. The error was:    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
fatal: [dev-testoz-01]: FAILED! => changed=false 
  msg: 'internal error: failed to become user ''ourdomain\adminuser'': Exception calling "CreateProcessAsUser" with "9" argument(s): "Failed to logon ourdomain\adminuser (The user name or password is incorrect, Win32ErrorCode 1326 - 0x0000052E)"'

[ansibot]

Files identified in the description:

• lib/ansible/playbook/become.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[jborean93]

Sorry for the delay here but this is the error returned by the call to LogonUser. Windows is saying the username or password is invalid so I'm not sure what we can do fix that.

What I recommend you try is to explicitly set the username/password using the vars directive to rule out something else setting the wrong password.

- win_whoami:
  become: yes
  become_method: runas
  vars:
    ansible_become_user: ourdomain\adminuser
    ansible_become_pass: my password

If that also fails I'm not sure what else it could be, we are really at the mercy of what Windows validates the password as.

[janitza-frmue]

Hi,
I don't think this is caused by Windows.
I just had the same error on Fedora 32 with Ansible package version 2.9.10-1.fc32. After a downgrade to 2.9.6-1.fc32 it is working again.

[johlandabee]

I can confirm this behavior. Seems like the bug was introduced with 2.9.2. It essentially stops working from there on. 2.8.* are working fine. 2.9.1 too.

[johlandabee]

I did not take a closer look, but maybe something slipped through here:
cd37b2d

[jborean93]

If you have an issue please share more information about

• How you are configuring the variables for become
• What's the error you are getting back
• Any other info that might be relevant to the problem

The original issue is literally Windows reporting the username/password is incorrect and there's not much we can do about it. Maybe there's a bug with passing in the password correctly but without knowing how that was done in the first place (or even a reply with my suggestion) there's not much we can do to help.

From my perspective everything works fine and I need more info to try and track down why it might be happening it you.

[johlandabee]

Hi @jborean93,
I've the exact same issue as the original. The error is the same. The configuration is correct. As i mentioned, it works flawlessly on the same development environment (Ubuntu 18.04.4 LTS Docker container) and target systems (domain joined Windows Server 2019) up to 2.9.1.

Here the relevant variables:

ansible_connection: winrm
ansible_port: 5986
ansible_winrm_transport: kerberos
ansible_winrm_server_cert_validation: ignore
ansible_user: "{{ admin_user }}"
ansible_password: "{{ admin_password }}"
ansible_become_method: runas
ansible_become_password: "{{ admin_password }}"
ansible_become_user: "{{ nt_domain }}\\{{ admin_user }}"

This is the error I get if I use a version higher than 2.9.1:

ansible 2.9.2
  config file = None
  configured module search path = ['/home/vscode/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/vscode/.local/lib/python3.6/site-packages/ansible
  executable location = /home/vscode/.local/bin/ansible
  python version = 3.6.9 (default, Jul 17 2020, 12:50:27) [GCC 8.4.0]

TASK [win-mssql : Create SQL Server directories]
An exception occurred during task execution. To see the full traceback, use -vvv. The error was:    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
failed: [server.company.tld] (item=SQLData00) => changed=false 
  ansible_loop_var: item
  item: SQLData00
  msg: 'internal error: failed to become user ''mydomain\myuser'': Exception calling "CreateProcessAsUser" with "9" argument(s): "Failed to logon mydomain\myuser (The user name or password is incorrect, Win32ErrorCode 1326 - 0x0000052E)"'

Again, with 2.9.1, everything works fine without a change to the playbook. Same account, same everything.

ansible 2.9.1
  config file = None
  configured module search path = ['/home/vscode/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/vscode/.local/lib/python3.6/site-packages/ansible
  executable location = /home/vscode/.local/bin/ansible
  python version = 3.6.9 (default, Jul 17 2020, 12:50:27) [GCC 8.4.0]

TASK [win-mssql : Create SQL Server directories]
ok: [server02.company.tld] => (item=SQLData00)
ok: [server01.company.tld] => (item=SQLData00)
ok: [server02.company.tld] => (item=SQLTLog)
ok: [server01.company.tld] => (item=SQLTLog)
ok: [server01.company.tld] => (item=SQLBackup)
ok: [server02.company.tld] => (item=SQLBackup)
ok: [server01.company.tld] => (item=MSSQLServer)
ok: [server02.company.tld] => (item=MSSQLServer)
ok: [server01.company.tld] => (item=SQLTempDB01)
ok: [server02.company.tld] => (item=SQLTempDB01)
ok: [server01.company.tld] => (item=SQLTempDB02)
ok: [server02.company.tld] => (item=SQLTempDB02)
ok: [server01.company.tld] => (item=SQLTempDB03)
ok: [server02.company.tld] => (item=SQLTempDB03)
ok: [server01.company.tld] => (item=SQLTempDB04)
ok: [server02.company.tld] => (item=SQLTempDB04)

You are correct, Windows just tells us our credentials are wrong which is probably correct from the OSs point of view. I think the become module does not pass them correctly. There was a larger change merged into 2.9.2 (cd37b2d) that I suspect to have introduced this bug. Again, I did not take closer look, just compared releases and searched for commits related to the become module and this one looks suspicious.

[jborean93]

Can you set ansible_become_password in your inventory without it using a templated var? Can you also run the following tasks (avoiding any templated vars).

- win_whoami:
  become: yes
  become_method: runas
  vars:
    ansible_become_user: ourdomain\adminuser
    ansible_become_pass: my password

- win_whoami:
  become: yes
  become_method: runas
  vars:
    ansible_become_user: ourdomain\adminuser
    ansible_become_password: my password

[johlandabee]

Both tasks work with 2.9.1. On 2.9.2 the second task using ansible_become_password fails. I did not use any templated vars.

[jborean93]

Thanks for narrowing down the problem a bit more, on your normal playbook can you run the following

- debug:
    var: ansible_become_password

- debug:
    var: ansible_become_pass

- debug:
    var: ansible_runas_pass

The precedence for each var goes from lowest to highest. If ansible_runas_pass and ansible_become_pass is defined, then the former is what is used. There's a small chance that ansible_become_pass is also defined for your host but with a different value causing the failure. This would explain why the 2nd task with ansible_become_password failed but ansible_become_pass worked.

One last thing to try is to run a single task with become with ANSIBLE_KEEP_REMOTE_FILES=1 ansible-playbook -i inventory.ini main.yml. After this run there should be a folder at C:\Users\<connection user>\AppData\Local\Temp\ansible-tmp-* and inside it contains a PowerShell script called AnsiballZ_<module>.ps1. Open up that file and at the end there should be a json manifest like the following (there is a lot of data here)

{
  "module_entry": "module base64",
  "powershell_modules": {
    "Ansible.ModuleUtils.Legacy": "module util base64",
    "Ansible.ModuleUtils.CamelConversion": "module util base64",
    "Ansible.ModuleUtils.AddType": "module util base64"
  },
  "csharp_utils": {
    "Ansible.AccessToken": "module util base64",
    "Ansible.Become": "module util base64",
    "Ansible.Process": "module util base64"
  },
  "csharp_utils_module": [],
  "module_args": {
    "_ansible_check_mode": false,
    "_ansible_no_log": false,
    "_ansible_debug": false,
    "_ansible_diff": false,
    "_ansible_verbosity": 3,
    "_ansible_version": "2.11.0.dev0",
    "_ansible_module_name": "win_whoami",
    "_ansible_syslog_facility": "LOG_USER",
    "_ansible_selinux_special_fs": [
      "fuse",
      "nfs",
      "vboxsf",
      "ramfs",
      "9p",
      "vfat"
    ],
    "_ansible_string_conversion_action": "warn",
    "_ansible_socket": null,
    "_ansible_shell_executable": "/bin/sh",
    "_ansible_keep_remote_files": true,
    "_ansible_tmpdir": "C:\\Users\\vagrant-domain\\AppData\\Local\\Temp\\ansible-tmp-1599766908.8073065-12246-163912062193480\\.",
    "_ansible_remote_tmp": "%TEMP%"
  },
  "actions": [
    "become_wrapper",
    "module_powershell_wrapper"
  ],
  "environment": {},
  "encoded_output": false,
  "become_user": "vagrant-domain@DOMAIN.TEST",
  "become_password": "MyPass",
  "become_flags": "",
  "min_ps_version": null,
  "min_os_version": null,
  "module_powershell_wrapper": "module exec base64",
  "module_wrapper": "module exec base64",
  "exec_wrapper": "module exec base64",
  "become_wrapper": "module exec base64"
}

You can see the become information is under become_user and become_password. It would be great to know if that value is actually correct or has the wrong value. This could indicate there's a problem with getting the correct password passed down or whether there is an issue on the runas code side with logging in that user that we haven't seen before.

[johlandabee]

You are correct. The Playbook I use has ansible_become_pass set on groups_vars/all level. This is used for the Linux part of it. I forgot about it to be honest.

Running the debug tasks produces the output below. The variables have the same value no matter which Ansible version is used. ansible_become_pass is set incorrectly.

TASK [win-mssql : debug]
ok: [baw-lab-mssql01.langenfeld.controlexpert.firm] => 
  ansible_become_password: *correctpassword*
ok: [baw-lab-mssql02.langenfeld.controlexpert.firm] => 
  ansible_become_password: *correctpassword*

TASK [win-mssql : debug]
ok: [baw-lab-mssql01.langenfeld.controlexpert.firm] => 
  ansible_become_pass: *wrongpassword*
ok: [baw-lab-mssql02.langenfeld.controlexpert.firm] => 
  ansible_become_pass: *wrongpassword*
  
ok: [baw-lab-mssql01.langenfeld.controlexpert.firm] => 
  ansible_runas_pass: VARIABLE IS NOT DEFINED!
ok: [baw-lab-mssql02.langenfeld.controlexpert.firm] => 
  ansible_runas_pass: VARIABLE IS NOT DEFINED!

Changing ansible_become_pass to ansible_become_password on my group vars fixes it. My playbook now works with 2.9.13. It seems the priority of become password variables has changed between versions. I don't know if this is considered a bug as all _pass variables are deprecated and shouldn't be used any more as far as I understand. It might be helpful to add a warning if ansible_become_pass is set?

I didn't bother to run a play with ANSIBLE_KEEP_REMOTE_FILES=1 set as the result will probably the same.

[jborean93]

My playbook now works with 2.9.13. It seems the priority of become password variables has changed between versions

The precedence may have changed but that's largely as a result of how vars are now trying to conform to the same standard, e.g. _pass -> _password. There was also a massive shift from sourcing become variables from the play context to a plugin specific setup in 2.8 and 2.9.

I don't know if this is considered a bug as all _pass variables are deprecated

Using ansible_become_pass is not deprecated, it's just that with the move to become plugins those vars are now controlled by the actual plugin itself. It's been kept for backwards compatibility and probably won't ever be removed. In the case of runas you can set the password with

• ansible_become_password
• ansible_become_pass
• ansible_runas_password

Other become plugins will have different vars but the standard when we split the become setup to a plugin was to have those 3 (the latter being ansible_<name>_password.

It might be helpful to add a warning if ansible_become_pass is set

The order of this matters as it defines the priority of each pass, in your case the ansible_become_pass set for your Linux hosts will override ansible_become_password set for your Windows host. That's just how the precedence works and now it's actually documented and enforced whereas before it was a confusing mixture. We can always add/remove vars for each option but in reality those 3 seem to strict a blend between name conformity and exclusivity.

Ultimately you should either

• Consolidate on the 1 var name, i.e. ansible_become_password or ansible_become_pass and ensure they are set per group and not in all
  • Mixing vars will lead to confusion around precedence and why things work sometimes like we have here
• Use the plugin specific option, ansible_runas_password, which has the highest priority
  • I'm not the biggest fan of this route as it makes it harder to use a different become plugin in the future without having to change your vars

Adding a dep warning will be hard because this precedence is controlled in a global fashion and is just how the plugin config system is designed to work.

Also because OP hasn't replied with any further information I am going to assume that they came across this same problem and will close the issue. Windows is reporting the correct error in that the password was correct. We cannot log those details because a password is a sensitive bit of information but the comments here can help others debug their setup if need be.

TLDR: Don't mix and match ansible_become_password and ansible_become_pass. Use one or the other but not both.