"Controller - Installing" failing with "msg": "Unexpected templating type error occurred on during

[mashoodakram]

Hi,
I'm getting error during the "Controller - Installing" step with below error! playbook that calls role is triggered from ADO pipeline.

2021-09-14T15:44:10.8653790Z fatal: [SECNGXCTLSR1003]: FAILED! => {"msg": "Unexpected templating type error occurred on (timeout --foreground 10m ./install.sh --accept-license --tsdb-volume-type '{{ nginx_controller_tsdb_volume_type }}' --smtp-host '{{ nginx_controller_smtp_host }}' --smtp-port '{{ nginx_controller_smtp_port | default(25) }}' --smtp-authentication '{{ nginx_controller_smtp_authentication }}' --smtp-use-tls '{{ nginx_controller_smtp_use_tls }}' --noreply-address '{{ nginx_controller_noreply_address }}' --fqdn '{{ nginx_controller_fqdn }}' --organization-name '{{ nginx_controller_organization_name }}' --admin-firstname '{{ nginx_controller_admin_firstname }}' --admin-lastname '{{ nginx_controller_admin_lastname }}' --admin-email '{{ nginx_controller_admin_email }}' --admin-password '{{ nginx_controller_admin_password }}' {{ (nginx_controller_tsdb_volume_type == \"nfs\") | ternary(\"--tsdb-nfs-path '\" + nginx_controller_tsdb_nfs_path + \"'\",'') }} {{ (nginx_controller_tsdb_volume_type == \"nfs\") | ternary(\"--tsdb-nfs-host '\" + nginx_controller_tsdb_nfs_host + \"'\",'') }} {{ (nginx_controller_tsdb_volume_type == \"aws\") | ternary(\"--tsdb-aws-volume-id '\" + nginx_controller_tsdb_aws_volume_id + \"'\",'') }} {{ (nginx_controller_configdb_volume_type != \"\" ) | ternary(\"--configdb-volume-type '\" + nginx_controller_configdb_volume_type + \"'\", '') }} {{ (nginx_controller_configdb_volume_type == \"nfs\") | ternary(\"--configdb-nfs-path '\" + nginx_controller_configdb_nfs_path + \"'\",'') }} {{ (nginx_controller_configdb_volume_type == \"nfs\") | ternary(\"--configdb-nfs-host '\" + nginx_controller_configdb_nfs_host + \"'\",'') }} {{ (nginx_controller_configdb_volume_type == \"aws\") | ternary(\"--configdb-aws-volume-id '\" + nginx_controller_configdb_aws_volume_id + \"'\",'') }} {{ ((nginx_controller_apigw_cert is defined and nginx_controller_apigw_cert | length > 0) and (nginx_controller_apigw_key is defined and nginx_controller_apigw_key | length > 0) ) | ternary(\"--apigw-cert '\" + nginx_controller_apigw_cert + \"'\",'') }} {{ ((nginx_controller_apigw_cert is defined and nginx_controller_apigw_cert | length > 0) and (nginx_controller_apigw_key is defined and nginx_controller_apigw_key | length > 0) ) | ternary(\"--apigw-key '\" + nginx_controller_apigw_key + \"'\",'') }} {{ (nginx_controller_smtp_authentication | bool) | ternary(\"--smtp-user '\" + nginx_controller_smtp_user + \"'\",'') }} {{ (nginx_controller_smtp_authentication | bool) | ternary(\"--smtp-password '\" + nginx_controller_smtp_password + \"'\",'') }} {{ (nginx_controller_self_signed_cert | bool) | ternary('--self-signed-cert','') }} {{ (nginx_controller_overwrite_existing_configs | bool) | ternary('--overwrite-existing-configs','') }} {{ (nginx_controller_auto_install_docker | bool) | ternary('--auto-install-docker','') }} {{ ((nginx_controller_bundled_db | bool) and ( nginx_controller_version is version('3.8', operator='ge', strict=True ))) | ternary(\"--use-bundled-db\",\"--database-host '\" + nginx_controller_db_host + \"' --database-port '\" + nginx_controller_db_port + \"' --database-user '\" + nginx_controller_db_user + \"' --database-password '\" + nginx_controller_db_password + \"'\" ) }} {{ ((nginx_controller_db_enable_ssl | bool) and ( not nginx_controller_bundled_db )) | ternary('--db-enable-ssl true','') }} {{ ((nginx_controller_db_enable_ssl | bool) and (nginx_controller_db_client_cert | length > 0)) | ternary(\"--db-client-cert '\" + nginx_controller_db_client_cert + \"'\",'') }} {{ ((nginx_controller_db_enable_ssl | bool) and (nginx_controller_db_client_key | length > 0)) | ternary(\"--db-client-key '\" + nginx_controller_db_client_key + \"'\",'') }} {{ ((nginx_controller_db_enable_ssl | bool) and (nginx_controller_db_ca | length > 0)) | ternary(\"--db-ca '\" + nginx_controller_db_ca + \"'\",'') }} {{ ((nginx_controller_version is version('3.5', operator='ge', strict=True) ) ) | ternary('--non-interactive','') }}\n): must be str, not NoneType"}

I can't tell from error above if it's complaining about the last line or one of the other variables.
Any suggestions would be greatly appreciated.
Thanks,

[aknot242]

Can you please provide the following?

To reproduce

• Your playbook(s) including variables with any sensitive information redacted

Your environment:

• Version of the NGINX Controller Installation role or specific commit
• Version of Ansible
• Version of Jinja2
• Target deployment platform

[brianehlert]

Unfortunately Ansible repeats the entire command without vars. So we don't know which var is missing and matches the error: must be str, not NoneType

[mashoodakram]

thank you Brian, went through the variables defined and found the nginx_controller_db_password was missing so added one directly to test and it's gotten past that error. I have now though stumbled onto another error, which I'm working through to figure out why it does not like key provided.

"ERROR: Parameter --apigw-key is invalid: File /etc/ssl/private/star_azure_defra_cloud.key does not exist or is not readable by current user."

Have defined below in the variables.

• nginx_controller_apigw_cert: /etc/ssl/certs/star_azure_defra_cloud.crt
• nginx_controller_apigw_key: /etc/ssl/private/star_azure_defra_cloud.key

Have checked key is there and permissions all look good but error persists. Is the current user root when installing?

[brianehlert]

Not all variables are required.

For example; the db password and user are tied to using an external PostgreSQL database server. Not required. You can use the internal-db flag to have the system support its own. using local file storage if not production, and using a remote NFS backed volume if production.

the api gateway cert and key are specific to using your own cert and key for the interface of Controller - both GUI and API.
Again, not required. You can also allow the system to self sign - and add a cert after install using helper.sh if you still require one.
The path you are providing for the cert and key assumes they are already stored in those places and the permissions are correct for the user that Ansible is using to execute the installer. So, who is the owner of the files. For example, you copied them over with root, but are executing the script as user ubuntu.

[brianehlert]

The user is not root when installing. And installing as root is not supported by default.
The install needs to be executed by an administrator who is not root following best practices.

[mashoodakram]

thanks again brian, apologies in advance if below question is not clear, new to ansible and so learning how it works as part of this install so excuse any nonsensical questions.

I've managed to sort out the certificate errors by creating certificates in locations specified and sorted out permissions, though then had error as docker was not already installed, have add docker role to playbook but it's not failing connecting to db.

I've tested the admin user and password on the managed Azure PostgreSQL with pgAdmin and have specified same user@servername and password. However running the install it's failing with below error.

Thu Sep 16 20:24:23 UTC 2021 2. Checking required ports... OK Thu Sep 16 20:24:23 UTC 2021 Last step took 00m:00s 3. Attempting to detect your Operating System... Found bionic Thu Sep 16 20:24:23 UTC 2021 Last step took 00m:00s 4. Checking for required tools: head openssl dirname tee rev ps envsubst mkdir jq id less tar getent base64 grep basename comm sort cat sed awk numfmt gunzip. All found. Thu Sep 16 20:24:23 UTC 2021 Last step took 00m:00s 5. Checking Docker version... Docker version 20.10.8, build 75249d8 We recommend setting native.cgroupdriver to systemd for Docker. WARNING! Docker configuration does not seem to have log rotation enabled. We recommend enabling log rotation for docker containers. For steps to enable log rotation follow this link: https://success.docker.com/article/how-to-setup-log-rotation-post-installation Loaded image: python:3.6-alpine Loaded image: postgres:12.5-alpine Loaded image: controller-infra/platform-mgr:2.28.24 Thu Sep 16 20:24:26 UTC 2021 Last step took 00m:03s 6. Config Database configuration Could not connect to the database.

Using below versions btw.
controller-installer-3.15.0.tar.gz
remote controller Ubuntu 18.04.5 LTS
Azure Database for PostgreSQL server PostgresSQL 9.6

ansible [core 2.11.4]
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/vmadmin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
ansible collection location = /home/vmadmin/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.6.9 (default, Jan 26 2021, 15:33:00) [GCC 8.4.0]
jinja version = 2.10
libyaml = True

[brianehlert]

the key is this line: Config Database configuration Could not connect to the database.

This is the Controller server component attempting to use the provided credentials to connect to the remote PostgreSQL endpoint.
The failure could be the credentials or a block at the network level from a policy not being defined between the Controller machine and service endpoint.

[mashoodakram]

hi Brian, thanks for all your help thus far we resolved the issue with connect to PostgreSQL by adding rule on the NSG where the controller resides to allow outbound connection to SQL over port 5432, however we then had issues trying to use SSL connecting to db not sure if this is in fact possible with a managed instance of postgreSQL that Azure provide.

I noted in the vars file the description below suggest certificate needs to be installed on both controller and db server but as we are using a managed postgreSQL instance we cannot install certificate on db server.

## Optional. Enable database ssl communications. Requires certificate installation at db server and controller. (true, false)

So we set - nginx_controller_db_enable_ssl: false

This got us past this error;

  6. Config Database configuration
psql: error: SSL error: certificate verify failed
 Failed to log into the database. Check that the PostgreSQL username and password are correct and that it allows connections from this host.

However now it's failing much further into the install with below error, is this result of not using ssl connectivity which I thought was optional.

NGINX Controller database has been initialized.
system: unable to upsert accounts: pq: SSL connection is required. Please specify SSL options and retry.
Failed to create the admin account in NGINX Controller.
Detailed information can be found in installation logs at /var/log/nginx-controller/failure. Retry NGINX Controller installation or contact support, providing all logs.

Any help understanding above would be greatly appreciated.

[brianehlert]

I don't know about managed PostgreSQL, but if you install it yourself - you have to set the listener on the PostgreSQL side to accept connections, and the machines to accept connections from.
I have never use the PostgreSQL ssl setting in my labs myself.
And I also don't use a PostgreSQL managed service.

Also the user account used has some expectations. The automation still has to meet all of these expectations: https://docs.nginx.com/nginx-controller/admin-guides/install/install-nginx-controller/

[mashoodakram]

Ok thanks for your input there, be good to understand if using a managed PostgreSQL instance is supported solution.
I did find it kind of strange there is option to turn SSL off but then the role expects it to be on where it failed.

If we could see what command is being passed that throws up the below and understand why it's asking for SSL connection when turned off earlier might help us understand the issue.

system: unable to upsert accounts: pq: SSL connection is required. Please specify SSL options and retry.

I'm also speaking to someone from F5 based in UK to align time zones, he's looking to try replicate the error.

[mashoodakram]

Hello again, just to update managed to get past above error by disabling SSL on the managed PostgreSQL instance. So seems setting it to nginx_controller_db_enable_ssl: false it will try use SSL connection when it finds it's enabled on the PostgreSQL server itself.

Also looking at the MS doc it mentions downloading and using MS certificate to PostgreSQL instance.
https://docs.microsoft.com/en-us/azure/postgresql/concepts-ssl-connection-security#applications-that-require-certificate-verification-for-tls-connectivity

I'm wondering if we can change the below vars provided to the role nginx_controller_install to point to the certificate MS are suggest is needed to connect to PostgreSQL instance.

• nginx_controller_db_client_cert: /etc/ssl/certs/star_azure_defra_cloud.crt
• nginx_controller_db_client_key: /etc/ssl/private/star_azure_defra_cloud.key

To something like this;

• nginx_controller_db_client_cert: /etc/ssl/certs/Baltimore_CyberTrust_Root.pem

Would that work do you think from role perspective?

[mashoodakram]

Hi @brianehlert so think we have confirmed SSL is not possible with PaaS instance as no way use client certificate. We have therefore decided to go bundled_db option so can close this particular issue as seems PaaS won't work with play books.