Installation error after #1397

[rodeodomino]

Describe the bug

If ipsec_enabled is set to false in config.cfg, then the install crashes with the error "The task includes an option with an undefined variable."

The script runs fine if ipsec_enabled is set to true.

To Reproduce

Steps to reproduce the behavior:

1. Set ipsec_enabled to false
2. Run ./algo

Expected behavior

The script runs as normal, but omits Strongswan.

Additional context

Full log

*****s-MacBook-Pro:algo-master *****$ cd ~/Downloads/algo-master
*****s-MacBook-Pro:algo-master *****$ nano config.cfg
*****s-MacBook-Pro:algo-master *****$ python -m virtualenv --python=`which python2` env &&
>     source env/bin/activate &&
>     python -m pip install -U pip virtualenv &&
>     python -m pip install -r requirements.txt
New python executable in /Users/*****/Downloads/algo-master/env/bin/python
Installing setuptools, pip, wheel...
done.
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: pip in ./env/lib/python2.7/site-packages (19.1)
Collecting virtualenv
  Using cached https://files.pythonhosted.org/packages/4f/ba/6f9315180501d5ac3e707f19fcb1764c26cc6a9a31af05778f7c2383eadb/virtualenv-16.5.0-py2.py3-none-any.whl
Installing collected packages: virtualenv
Successfully installed virtualenv-16.5.0
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting ansible==2.7.10 (from -r requirements.txt (line 1))
Requirement already satisfied: setuptools in ./env/lib/python2.7/site-packages (from ansible==2.7.10->-r requirements.txt (line 1)) (41.0.1)
Collecting paramiko (from ansible==2.7.10->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/cf/ae/94e70d49044ccc234bfdba20114fa947d7ba6eb68a2e452d89b920e62227/paramiko-2.4.2-py2.py3-none-any.whl
Collecting jinja2 (from ansible==2.7.10->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/1d/e7/fd8b501e7a6dfe492a433deb7b9d833d39ca74916fa8bc63dd1a4947a671/Jinja2-2.10.1-py2.py3-none-any.whl
Collecting cryptography (from ansible==2.7.10->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/f5/2a/48848dd65908190d68ef389ad8ee9d0010278f7a9118f40d8fed7d84a326/cryptography-2.6.1-cp27-cp27m-macosx_10_6_intel.whl
Collecting PyYAML (from ansible==2.7.10->-r requirements.txt (line 1))
Collecting pyasn1>=0.1.7 (from paramiko->ansible==2.7.10->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl
Collecting bcrypt>=3.1.3 (from paramiko->ansible==2.7.10->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/14/1b/f6f05b80783ff84c58bca5f2b7a4cef7c02570e4bb2e40d326abb2ba41d3/bcrypt-3.1.6-cp27-cp27m-macosx_10_6_intel.whl
Collecting pynacl>=1.0.1 (from paramiko->ansible==2.7.10->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/51/83/2db5b919bf9848fe25d301225a16faabc378419e7eaf00da0b7d200fe801/PyNaCl-1.3.0-cp27-cp27m-macosx_10_6_intel.whl
Collecting MarkupSafe>=0.23 (from jinja2->ansible==2.7.10->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/6d/d2/0ccd2c0e2cd93b35e765d9b3205cd6602e6b202b522fc7997531353715b3/MarkupSafe-1.1.1-cp27-cp27m-macosx_10_6_intel.whl
Collecting enum34; python_version < "3" (from cryptography->ansible==2.7.10->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography->ansible==2.7.10->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/16/f6/46a3dece43541b2cbf3776ec2299e370a2408d9380958401cacb6d101853/cffi-1.12.3-cp27-cp27m-macosx_10_6_intel.whl
Collecting six>=1.4.1 (from cryptography->ansible==2.7.10->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography->ansible==2.7.10->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting ipaddress; python_version < "3" (from cryptography->ansible==2.7.10->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography->ansible==2.7.10->-r requirements.txt (line 1))
Installing collected packages: pyasn1, six, pycparser, cffi, bcrypt, enum34, asn1crypto, ipaddress, cryptography, pynacl, paramiko, MarkupSafe, jinja2, PyYAML, ansible
Successfully installed MarkupSafe-1.1.1 PyYAML-5.1 ansible-2.7.10 asn1crypto-0.24.0 bcrypt-3.1.6 cffi-1.12.3 cryptography-2.6.1 enum34-1.1.6 ipaddress-1.0.22 jinja2-2.10.1 paramiko-2.4.2 pyasn1-0.4.5 pycparser-2.19 pynacl-1.3.0 six-1.12.0
(env) *****s-MacBook-Pro:algo-master *****$ ./algo

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [Verify Ansible meets Drupal VM's version requirements.] ******************
ok: [localhost] => {
    "changed": false, 
    "msg": "All assertions passed"
}

PLAY [Ask user for the input] **************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]
[Region prompt]
What provider would you like to use?
    1. DigitalOcean
    2. Amazon Lightsail
    3. Amazon EC2
    4. Vultr
    5. Microsoft Azure
    6. Google Compute Engine
    7. Scaleway
    8. OpenStack (DreamCompute optimised)
    9. Install to existing Ubuntu 18.04 server (Advanced)
  
Enter the number of your desired provider
:

TASK [Region prompt] ***********************************************************
ok: [localhost]

TASK [Set facts based on the input] ********************************************
ok: [localhost]
[VPN server name prompt]
Name the vpn server
[algo]
:

TASK [VPN server name prompt] **************************************************
ok: [localhost]
[DNS adblocking prompt]
Do you want to install an ad blocking DNS resolver on this VPN server?
[y/N]
:

TASK [DNS adblocking prompt] ***************************************************
ok: [localhost]
[SSH tunneling prompt]
Do you want each user to have their own account for SSH tunneling?
[y/N]
:

TASK [SSH tunneling prompt] ****************************************************
ok: [localhost]

TASK [Set facts based on the input] ********************************************
fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'user_input'\n\nThe error appears to have been in '/Users/*****/Downloads/algo-master/input.yml': line 115, column 9, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n      - name: Set facts based on the input\n        ^ here\n"}
included: /Users/*****/Downloads/algo-master/playbooks/rescue.yml for localhost

TASK [debug] *******************************************************************
ok: [localhost] => {
    "fail_hint": [
        "Sorry, but something went wrong!", 
        "Please check the troubleshooting guide.", 
        "https://trailofbits.github.io/algo/troubleshooting.html"
    ]
}

TASK [Fail the installation] ***************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed as requested from task"}

PLAY RECAP *********************************************************************
localhost                  : ok=10   changed=0    unreachable=0    failed=2 

[rodeodomino]

And this is the text of config.cfg

---

# This is the list of user to generate.
# Every device must have a unique username.
# You can generate up to 250 users at one time.
users:
  - phone
  - laptop
  - desktop

# NOTE: You must "escape" any usernames with leading 0's, like "000dan"

### Advanced users only below this line ###

# If True re-init all existing certificates. Boolean
keys_clean_all: False

# Clean up cloud python environments
clean_environment: false

# Deploy StrongSwan to enable IPsec support
ipsec_enabled: false

# StrongSwan log level
# https://wiki.strongswan.org/projects/strongswan/wiki/LoggerConfiguration
strongswan_log_level: 2

# rightsourceip for ipsec
# ipv4
strongswan_network: 10.19.48.0/24
# ipv6
strongswan_network_ipv6: 'fd9d:bc11:4020::/48'

# Deploy WireGuard
wireguard_enabled: true
wireguard_port: 51820
# If you're behind NAT or a firewall and you want to receive incoming connections long after network traffic has gone silent.
# This option will keep the "connection" open in the eyes of NAT.
# See: https://www.wireguard.com/quickstart/#nat-and-firewall-traversal-persistence
wireguard_PersistentKeepalive: 0

# WireGuard network configuration
_wireguard_network_ipv4:
  subnet: 10.19.49.0
  prefix: 24
  gateway: 10.19.49.1
  clients_range: 10.19.49
  clients_start: 2
_wireguard_network_ipv6:
  subnet: 'fd9d:bc11:4021::'
  prefix: 48
  gateway: 'fd9d:bc11:4021::1'
  clients_range: 'fd9d:bc11:4021::'
  clients_start: 2
wireguard_network_ipv4: "{{ _wireguard_network_ipv4['subnet'] }}/{{ _wireguard_network_ipv4['prefix'] }}"
wireguard_network_ipv6: "{{ _wireguard_network_ipv6['subnet'] }}/{{ _wireguard_network_ipv6['prefix'] }}"

# Reduce the MTU of the VPN tunnel
# Some cloud and internet providers use a smaller MTU (Maximum Transmission
# Unit) than the normal value of 1500 and if you don't reduce the MTU of your
# VPN tunnel some network connections will hang. Algo will attempt to set this
# automatically based on your server, but if connections hang you might need to
# adjust this yourself.
# See: https://github.com/trailofbits/algo/blob/master/docs/troubleshooting.md#various-websites-appear-to-be-offline-through-the-vpn
reduce_mtu: 0

# Algo will use the following lists to block ads. You can add new block lists
# after deployment by modifying the line starting "BLOCKLIST_URLS=" at:
# /usr/local/sbin/adblock.sh
# If you load very large blocklists, you may also have to modify resource limits:
# /etc/systemd/system/dnsmasq.service.d/100-CustomLimitations.conf
adblock_lists:
 - "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts"
 - "https://hosts-file.net/ad_servers.txt"

# Enable DNS encryption.
# If 'false', 'dns_servers' should be specified below.
dns_encryption: true

# DNS servers which will be used if 'dns_encryption' is 'true'. Multiple
# providers may be specified, but avoid mixing providers that filter results
# (like Cisco) with those that don't (like Cloudflare) or you could get
# inconsistent results. The list of available public providers can be found
# here:
# https://github.com/DNSCrypt/dnscrypt-resolvers/blob/master/v2/public-resolvers.md
dnscrypt_servers:
  ipv4:
    - cloudflare
#   - google
  ipv6:
    - cloudflare-ipv6

# DNS servers which will be used if 'dns_encryption' is 'false'.
# The default is to use Cloudflare.
dns_servers:
  ipv4:
    - 1.1.1.1
    - 1.0.0.1
  ipv6:
    - 2606:4700:4700::1111
    - 2606:4700:4700::1001

# IP address for the local dns resolver
local_service_ip: 172.16.0.1

# Your Algo server will automatically install security updates. Some updates
# require a reboot to take effect but your Algo server will not reboot itself
# automatically unless you change 'enabled' below from 'false' to 'true', in
# which case a reboot will take place if necessary at the time specified (as
# HH:MM) in the time zone of your Algo server. The default time zone is UTC.
unattended_reboot:
  enabled: false
  time: 06:00

# Block traffic between connected clients
BetweenClients_DROP: true

congrats:
  common: |
    "#                          Congratulations!                            #"
    "#                     Your Algo server is running.                     #"
    "#    Config files and certificates are in the ./configs/ directory.    #"
    "#              Go to https://whoer.net/ after connecting               #"
    "#        and ensure that all your traffic passes through the VPN.      #"
    "#                     Local DNS resolver {{ local_service_ip }}                    #"
  p12_pass: |
    "#        The p12 and SSH keys password for new users is {{ p12_export_password }}       #"
  ca_key_pass: |
    "#        The CA key password is {{ CA_password }}       #"
  ssh_access: |
    "#      Shell access: ssh -i {{ ansible_ssh_private_key_file|default(omit) }} {{ ansible_ssh_user|default(omit) }}@{{ ansible_ssh_host|default(omit) }}        #"

SSH_keys:
  comment: algo@ssh
  private: configs/algo.pem
  public: configs/algo.pem.pub

cloud_providers:
  azure:
    size: Basic_A0
    image: 18.04-LTS
  digitalocean:
    size: s-1vcpu-1gb
    image: "ubuntu-18-04-x64"
  # Change the encrypted flag to "true" to enable AWS volume encryption, for encryption of data at rest.
  # Warning: the Algo script will take approximately 6 minutes longer to complete.
  # Also note that the documented AWS minimum permissions aren't sufficient.
  # You will have to edit the AWS user policy documented at
  # https://github.com/trailofbits/algo/blob/master/docs/cloud-amazon-ec2.md to also allow "ec2:CopyImage".
  # See https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-edit.html
  ec2:
    encrypted: false
    size: t2.micro
    image:
      name: "ubuntu-bionic-18.04"
      owner: "099720109477"
  gce:
    size: f1-micro
    image: ubuntu-1804
    external_static_ip: false
  lightsail:
    size: nano_1_0
    image: ubuntu_18_04
  scaleway:
    size: START1-S
    image: Ubuntu Bionic Beaver
    arch: x86_64
  openstack:
    flavor_ram: ">=512"
    image:  Ubuntu-18.04
  vultr:
    os: Ubuntu 18.04 x64
    size: 1024 MB RAM,25 GB SSD,1.00 TB BW
  local:

fail_hint:
  - Sorry, but something went wrong!
  - Please check the troubleshooting guide.
  - https://trailofbits.github.io/algo/troubleshooting.html

booleans_map:
  Y: true
  y: true

[TC1977]

I can reproduce the error by running newest Algo commit on MacOS, cloud install onto Vultr, "IPsec_enabled" set to "false" as above. Here's ./algo -vvv output:

TASK [SSH tunneling prompt] ********************************************************************************************************************************************************
task path: /Users/********/Downloads/algo-master/input.yml:107
ok: [localhost] => {
    "changed": false, 
    "delta": 1, 
    "echo": true, 
    "rc": 0, 
    "start": "2019-04-27 12:08:33.121053", 
    "stderr": "", 
    "stdout": "Paused for 0.02 minutes", 
    "stop": "2019-04-27 12:08:34.570745", 
    "user_input": "y"
}
Read vars_file 'config.cfg'

TASK [Set facts based on the input] ************************************************************************************************************************************************
task path: /Users/********/Downloads/algo-master/input.yml:115
fatal: [localhost]: FAILED! => {
    "msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'user_input'\n\nThe error appears to have been in '/Users/admin/Downloads/algo-master/input.yml': line 115, column 9, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n      - name: Set facts based on the input\n        ^ here\n"
}
Read vars_file 'config.cfg'
Read vars_file 'config.cfg'
included: /Users/********/Downloads/algo-master/playbooks/rescue.yml for localhost
Read vars_file 'config.cfg'
Read vars_file 'config.cfg'

TASK [debug] ***********************************************************************************************************************************************************************
task path: /Users/********/Downloads/algo-master/playbooks/rescue.yml:2
ok: [localhost] => {
    "fail_hint": [
        "Sorry, but something went wrong!", 
        "Please check the troubleshooting guide.", 
        "https://trailofbits.github.io/algo/troubleshooting.html"
    ]
}
Read vars_file 'config.cfg'

TASK [Fail the installation] *******************************************************************************************************************************************************
task path: /Users/********/Downloads/algo-master/playbooks/rescue.yml:5
fatal: [localhost]: FAILED! => {
    "changed": false, 
    "msg": "Failed as requested from task"
}

PLAY RECAP *************************************************************************************************************************************************************************
localhost                  : ok=10   changed=0    unreachable=0    failed=2   

[namila007]

im getting this error even ipsec_enabled: true
this was done using a fresh clone from the master branch. only changed the user list. others are kept original. following is the relevant output;

TASK [Install the requirements] ****************************************************************************************
ok: [localhost -> localhost]
[local : pause]
Enter the IP address of your server: (or use localhost for local installation):
[localhost]
:

TASK [local : pause] ***************************************************************************************************
ok: [localhost]

TASK [local : Set the facts] *******************************************************************************************
ok: [localhost]

TASK [local : Set the facts] *******************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'user_input'\n\nThe error appears to have been in '/root/algo/roles/local/tasks/prompts.yml': line 25, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Set the facts\n  ^ here\n"}
included: /root/algo/playbooks/rescue.yml for localhost

TASK [debug] ***********************************************************************************************************
ok: [localhost] => {
    "fail_hint": [
        "Sorry, but something went wrong!",
        "Please check the troubleshooting guide.",
        "https://trailofbits.github.io/algo/troubleshooting.html"
    ]
}

TASK [Fail the installation] *******************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed as requested from task"}

PLAY RECAP *************************************************************************************************************
localhost                  : ok=19   changed=1    unreachable=0    failed=2

this was done on Ubuntu 18.04 x64

[TC1977]

It seems that replacing elif _variable.user_input with elif _variable.user_input is defined in input.yml will get past this. E.g.:

algo_ondemand_cellular: >-
            {% if ondemand_cellular is defined %}{{ ondemand_cellular | bool }}
            {%- elif _ondemand_cellular.user_input is defined %}{{ booleans_map[_ondemand_cellular.user_input] | default(defaults['ondemand_cellular']) }}
            {%- else %}false{% endif %}

but I have no idea if this is the best way to do it, and probably a lot of other files will have to be changed as well, like roles/local/tasks/prompts.yml for #1411. The version of this statement before #1397 was:

{%- elif _ondemand_cellular.user_input is defined and _ondemand_cellular.user_input != "" %}{{ booleans_map[_ondemand_cellular.user_input] | default(defaults['ondemand_cellular']) }}

[marcovn]

im getting this error even ipsec_enabled: true
this was done using a fresh clone from the master branch. only changed the user list. others are kept original. following is the relevant output;
fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'user_input'\n\nThe error appears to have been in '/root/algo/roles/local/tasks/prompts.yml': line 25, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Set the facts\n ^ here\n"}
included: /root/algo/playbooks/rescue.yml for localhost

TASK [debug] ***********************************************************************************************************
this was done on Ubuntu 18.04 x64

Can confirm this (on 18.04 as well). Last week I setup a server (local installation as well) using commit a60d49f and that one worked fine. Just tested it again and it still works on the original server, not the one I deployed with the latest commit.

[jackivanov]

#1415 should fix this

[marcovn]

I have tested again using the #1415 branch and the installation error is resolved. However, when running update-users I get a similar error. Will report on that tomorrow.

[marcovn]

Hey just an update as promised.

Just setup a fresh ubuntu 18.04 server and installed algo with defaults except for 'save CA password'. Works as expected and tested multiple clients.

Next I wanted to add 2 more users so added them in config.cfg and ran ./algo update-users. After entering localhost and my CA password it exits with an error. See below for details.

(env) root@london-algo:~/algo# ./algo update-users

PLAY [localhost] *************************************************************************************************
[Server address prompt]
Enter the IP address of your server: (or use localhost for local installation):

TASK [Server address prompt] *************************************************************************************
ok: [localhost]

TASK [Set facts based on the input] ******************************************************************************
ok: [localhost]

TASK [Import host specific variables] ****************************************************************************
ok: [localhost]
[CA password prompt]
Enter the password for the private CA key (output is hidden):

TASK [CA password prompt] ****************************************************************************************
ok: [localhost]

TASK [Set facts based on the input] ******************************************************************************
ok: [localhost]

TASK [Add the server to the vpn-host group] **********************************************************************
changed: [localhost]

PLAY [User management] *******************************************************************************************

TASK [Gathering Facts] *******************************************************************************************
ok: [localhost]

TASK [Install the requirements] **********************************************************************************
fatal: [localhost -> localhost]: FAILED! => {"changed": false, "msg": "Unable to find any of pip3 to use.  pip needs to be installed."}

PLAY RECAP *******************************************************************************************************
localhost                  : ok=7    changed=1    unreachable=0    failed=1

Details on my env:

(env) root@london-algo:~/algo# ./algo-showenv.sh
Algo running on: Ubuntu 18.04.2 LTS (Virtualized: kvm)
Created from git clone. Last commit: 78a6a3e Another fact fix
Python 2.7.15rc1

[TC1977]

@marcovn I can reproduce your error, but I think it may not be related and deserves its own issue.

[jackivanov]

Yes, it's not related, but I addressed this one in the same PR just to speed up the process.
@marcovn Could you check it out now, please? It worked on my end

[marcovn]

@jackivanov can confirm: update-users working fine now, tried several times with additional users too.