Bug fixes 2.x pr 2.x (#2280)

* Updating CI to 2.x.

* Defending against missing Ansible.

* Making the ce-provision-config branch in CI dynamic.

* We do not want a 'ce-dev provision' because it breaks our controller.

* Reverting 'ce-dev provision' change.

* Trying a different ansible_facts var.

* Testing using the source branch in ce-dev.

* Setting max_childen to an integer to avoid CI issues.

* Trying to change the python interpreter used.

* Adding platform and cgroup values to ce-dev compose template.

* Trying latest ubuntu containers in GitHub Actions.

* Fixing the test.sh script to work with venvs.

* Documentation for PHP in CI.

* Adding GitLab test back in.

* Fixing role namespaces.

* Minor bug fixes to ce-provision installer.

* Testing installing ce-provision in the GitHub Actions container directly.

* Using the submitted install script as well.

* Trying as runner user.

* Trying to use the ce-dev base container.

* Updating key name.

* Suppressing systemd actions in Docker.

* Seems Ansible flags have changed.

* Still trying to get --extra-vars right!

* Catching Ansible Galaxy upgrade timers for docker containers.

* Trying to force --roles-path for Galaxy.

* Trying different quotes.

* Missed a line.

* Trying a different approach to passing vars.

* Adding some debug.

* Running ce-python debug first.

* Trying moving to the ce-provision directory.

* Checking the specific path to galaxy roles in ce-provision.

* Trying as controller user again.

* Trying to make the roles dir.

* Being consistent about paths in bash.

* Removing debug lines for now.

* Allowing script to skip iptables.

* Misnamed flag.

* Adding user_provision role to configure controller user.

* Wrapping cleanup so it doesn't break GitHub Actions.

* Completing variables for user_provisin.

* Missed the sudoers var.

* Quoting vars.

* GitLab installer needs _domain_name.

* Logic error in clean-up script.

* Fixing paths to ce-provision in container.

* Trying to fix CI perms issues.

* Git dubious ownership error.

* Git dubious ownership error.

* Running the web server test as the controller user.

* Missed a controller var.

* Commenting out the CE container to test.

* Adding a separate step for Git actions.

* Need sudo for Ubuntu.

* Using a volume to persist data between steps.

* Adding debug commands to test volumes.

* Tweaking volumes.

* Adding the checkout command back in.

* Trying a different approach.

* ls command looks good, so putting web build back in.

* More Ansible Galaxy debug.

* Trying to make ansible-galaxy detect installed roles.

* Run galaxy command as controller.

* Trying galaxy command and cd wrapped in su.

* Specifically checking the contents of galaxy/roles.

* Trying a double-tap install process.

* Quick refactor and debug of SSH.

* Adding OpenSSH server package.

* Checking for a firewall.

* Checking listening packages.

* Starting SSHD especially.

* Starting SSHD without systemd.

* Pre-empting config a bit more.

* More galaxy path debug.

* Running a find to see if we can find the missing roles.

* More verbosity.

* Checking for missing requirements file.

* Removing eroneous when clause.

* Tidying up redundant debug lines.

* Creating a separate ci.yml play targeting localhost.

* Making sure sshd is running.

* Tidying up GitLab CI file and installing SSHD.

* Installing SSHD as a separate step.

* SSHD already installed, starting it instead.

* Don't create systemd timers in containers.

* Preparing a test GitLab build.

* Making builds nightly and fixing GitLab role bug.

* Ensuring is_local var exists and making lock behaviour optional.

* Fixing location and owner of Blackfire config so it is configurable.

* Documentation update.

* Removing all is defined checks for is_local since it is now always defined.

* Letting GitLab know it's on Docker earlier.

* Trying to run runsvdir-start to avoid container freezing.

* Temporarily skipping reconfigure of GitLab to test the rest.

* Trying to move GitLab reconfigure commands to CI.

* Fixing service namespace for runner and reinstating GitLab tasks.

* Trying to get config script working for GitLab in CI.

* No systemd, do not try to restart gitlab-runner.

* Removing firewall role from CI GitLab test, don't need it and it breaks CI.

* Outputting PostGreSQL logs to see if there are errors.

* Outputting PostGreSQL logs to see if there are errors.

* Trying the config script for GitLab again.

* Suppressing extra GitLab config for CI runs.

* Setting Blackfire CLI defaults to use ce-dev user.

* Improving GitLab vars and adding force stop feature.

* Not installed aws_credentials in meta roles if AWS support disabled.

* Most people will not want pam_ldap or pam_linotp, should not be in meta.

* If you don't create LDAP SSL certs you might not have a /etc/ldap directory.

* Adding a merge of the main branch into the docs branch to CI.

* Adding comment to not use hyphens in boto profile names.

* Supporting different key types to publish to AWS.

* Adding a git fetch before the merge in docs publishing.

* Using the safer _ce_provision_username var in AWS key role.

* Adding the --allow-unrelated-histories flag to git merge in CI.

* Commenting out some of the AWS ACL rulesets to leave them as examples.

* Trying to pull the docs branch to ensure it is up to date.

* Adding keyserver.ubuntu.com to the key servers we publish to by default in gpg_key.

* CI updates and improving SOPS role.

* Swapping hard coded branch names for vars in CI.

* Giving up on merging 2.x - will have to try another approach.

* Change of approach to Packer plugin management.

* Renaming task.

* Reorganising the GitLab Runner role into blocks.

* Updating amazon.aws collection for Ansible.

* Merging 2.x.

* Adding a generic launcher playbook.

* Updating default branch names to 'main'.

* Adding code to launcher playbook that adds new server to hosts.yml.

* Adding hosts.yml file handling to launcher.

Author: gregharvey

ce-dev/ansible/vars/_common/user_deploy.yml

@@ -1,8 +1,6 @@
 _user_deploy_username: ce-dev
 user_deploy:
   username: "{{ _user_deploy_username }}"
-  utility_host: "localhost"
-  utility_username: "{{ _user_deploy_username }}"
   sudo_config:
     entity_name: "{{ _user_deploy_username }}"
     hosts: "ALL"

ce-dev/ansible/vars/_common/user_provision.yml

@@ -1,8 +1,6 @@
 _user_provision_username: ce-dev
 user_provision:
   username: "{{ _user_provision_username }}"
-  utility_host: "localhost"
-  utility_username: "{{ _user_provision_username }}"
   sudo_config:
     entity_name: "{{ _user_provision_username }}"
     hosts: "ALL"

ce-dev/ansible/vars/gitlab/ce_deploy.yml

@@ -2,6 +2,5 @@ ce_deploy:
   own_repository: https://github.com/codeenigma/ce-deploy.git
   own_repository_branch: 1.x
   username: deploy
-  utility_username: deploy
   local_dir: /home/deploy/ce-deploy
-  ce_provision_dir: /home/ce-dev/ce-provision
+  ce_provision_dir: /home/ce-dev/ce-provision

install.sh

@@ -147,7 +147,6 @@ fi
 /usr/bin/su - "$CONTROLLER_USER" -c "/usr/bin/python3 -m venv /home/$CONTROLLER_USER/ce-python"
 /usr/bin/su - "$CONTROLLER_USER" -c "/home/$CONTROLLER_USER/ce-python/bin/python3 -m pip install --upgrade pip"
 /usr/bin/su - "$CONTROLLER_USER" -c "/home/$CONTROLLER_USER/ce-python/bin/pip install ansible netaddr python-debian"
-/usr/bin/su - "$CONTROLLER_USER" -c "/home/$CONTROLLER_USER/ce-python/bin/ansible-galaxy collection install ansible.posix -p /home/$CONTROLLER_USER/.ansible/collections/ansible_collections --force"
 if [ "$AWS_SUPPORT" = "true" ]; then
   /usr/bin/su - "$CONTROLLER_USER" -c "/home/$CONTROLLER_USER/ce-python/bin/pip install boto3"
 fi
@@ -167,6 +166,8 @@ else
   /usr/bin/echo "-------------------------------------------------"
 fi
 /usr/bin/mkdir -p "/home/$CONTROLLER_USER/ce-provision/galaxy/roles"
+/usr/bin/su - "$CONTROLLER_USER" -c "cd /home/$CONTROLLER_USER/ce-provision && /home/$CONTROLLER_USER/ce-python/bin/ansible-galaxy collection install ansible.posix -p /home/$CONTROLLER_USER/ce-provision/galaxy/ansible_collections --force"
+
 # Create playbook for ce-provision.
 /bin/cat >"/home/$CONTROLLER_USER/ce-provision/provision.yml" << EOL
 ---
@@ -233,8 +234,6 @@ user_provision:
   create: false
   create_home: false
   update_password: always
-  utility_username: "${CONTROLLER_USER}"
-  utility_host: localhost
   sudo_config:
     entity_name: "${CONTROLLER_USER}"
     hosts: "ALL"

plays/launcher/configure.yml

@@ -0,0 +1,57 @@
+---
+# Bare provisioning of a new server.
+# Use this command to execute:
+#
+#   cd /home/controller/ce-provision && \
+#     ./scripts/provision.sh --workspace /home/controller/ce-provision \
+#     --repo none --branch none \
+#     --playbook /home/controller/ce-provision/plays/launcher/configure.yml \
+#     --ansible-extra-vars "_provision_host=X.X.X.X"
+#
+# Replace X.X.X.X with your hostname or IP address.
+- hosts: localhost
+  vars:
+    _ce_provision_username: controller
+    _add_host: true # set to false if you do not want to add the new server to hosts.yml
+    _ce_provision_config_branch: 2.x # change this if you didn't use our example config and have a different main branch name
+  tasks:
+    - name: Add the server to Ansible hosts in memory.
+      ansible.builtin.add_host:
+        hostname: "{{ _provision_host }}"
+
+    - name: Add the server to hosts.yml.
+      when: _add_host
+      block:
+      - name: Edit hosts.yml file.
+        ansible.builtin.lineinfile:
+          path: "/home/{{ _ce_provision_username }}/ce-provision/config/hosts/hosts.yml"
+          line: "{{ _provision_host }}:"
+          create: true
+
+      - name: Add hosts.yml changes.
+        ansible.builtin.command: git add .
+        args:
+          chdir: "/home/{{ _ce_provision_username }}/ce-provision/config"
+
+      - name: Commit hosts.yml changes.
+        ansible.builtin.shell: "git diff --staged --quiet || git commit -m 'Ansible autogenerated - host {{ _provision_host }} added.'"
+        args:
+          chdir: "/home/{{ _ce_provision_username }}/ce-provision/config"
+
+      - name: Push the hosts file change to the config repository.
+        ansible.builtin.command: "git push origin {{ _ce_provision_config_branch | default('main') }}"
+        args:
+          chdir: "/home/{{ _ce_provision_username }}/ce-provision/config"
+
+- hosts: "{{ _provision_host }}"
+  vars:
+    ansible_user: admin
+    _ce_provision_build_id: 0
+  become: true
+  tasks:
+    - ansible.builtin.import_role:
+        name: _init
+    - ansible.builtin.import_role:
+        name: debian/user_provision
+    - ansible.builtin.import_role:
+        name: _exit

roles/_init/defaults/main.yml

@@ -7,6 +7,7 @@ _venv_path: "/home/{{ _ce_provision_username }}/ce-python"
 _venv_command: /usr/bin/python3 -m venv
 _venv_install_username: "{{ _ce_provision_username }}"
 _ce_ansible_timer_name: upgrade_ansible
+_env_type: unspecified
 
 # AWS variables - if you are using an AWS account, you can preset certain variables
 # Generally it is recommended to place these in your ce-provision-config repository under hosts/group_vars/all

roles/debian/ce_deploy/tasks/main.yml

@@ -47,7 +47,7 @@
   ansible.builtin.git:
     repo: "{{ ce_deploy.own_repository | default('https://github.com/codeenigma/ce-deploy.git') }}"
     dest: "{{ ce_deploy.local_dir }}"
-    version: "{{ ce_deploy.own_repository_branch | default('master') }}"
+    version: "{{ ce_deploy.own_repository_branch | default('main') }}"
     update: true
     accept_hostkey: true
   become: true
@@ -58,7 +58,7 @@
     repo: "{{ ce_deploy.config_repository }}"
     accept_hostkey: true
     dest: "{{ _ce_provision_build_tmp_dir }}/config"
-    version: "{{ ce_deploy.config_repository_branch | default('master') }}"
+    version: "{{ ce_deploy.config_repository_branch | default('main') }}"
   become: false
   delegate_to: localhost
   when: ce_deploy.config_repository is defined and ce_deploy.config_repository

roles/debian/ce_provision/tasks/main.yml

@@ -85,7 +85,7 @@
   ansible.builtin.git:
     repo: "{{ ce_provision.own_repository | default('https://github.com/codeenigma/ce-provision.git') }}"
     dest: "{{ ce_provision.local_dir }}"
-    version: "{{ ce_provision.own_repository_branch | default('master') }}"
+    version: "{{ ce_provision.own_repository_branch | default('main') }}"
     update: true
     accept_hostkey: true #@todo?
   become: true
@@ -98,7 +98,7 @@
     repo: "{{ ce_provision.config_repository }}"
     accept_hostkey: true
     dest: "{{ ce_provision.local_dir }}/config"
-    version: "{{ ce_provision.config_repository_branch | default('master') }}"
+    version: "{{ ce_provision.config_repository_branch | default('main') }}"
   become: true
   become_user: "{{ ce_provision.username }}"
   when:

roles/debian/user_ansible/defaults/main.yml

@@ -11,9 +11,6 @@ user_ansible:
   # This is shown for documentation, you should do this in your config repo
   # uid: 999
   # gid: 999
-  # Local username of the deploy user.
-  utility_host: "localhost"
-  utility_username: "{{ _user_ansible_username }}"
   sudo_config: {} # an empty dictionary will skip creating a sudo config
   # Example sudo config allowing full sudo permissions - see the debian/sudo_config role for more details.
   #  entity_name: "{{ _user_ansible_username }}"

roles/debian/user_deploy/defaults/main.yml

@@ -12,9 +12,6 @@ user_deploy:
   # This is shown for documentation, you should do this in your config repo
   # uid: 989
   # gid: 989
-  # Local username of the deploy user.
-  utility_host: "localhost"
-  utility_username: "{{ _user_deploy_username }}"
   sudo_config: {}
   # Example config allowing for feature branching. Allows manipulation of NGINX vhosts and cron.d files.
   # Uncomment to use.