docker: Automate deletion of docker static Jenkins nodes (#3561)

* initial

* initial

* finished deleteNodes script

* script works with ansible

* add no_log to python script task to hide variable from log

* add comments

* update comment

* typo

* cherry pick docs pr into this one

* Change http to https

* Change port to 443

---------

Co-authored-by: Martijn Verburg <martijnverburg@gmail.com>

ansible/playbooks/AdoptOpenJDK_Unix_Playbook/dockernode.yml

@@ -16,3 +16,5 @@
       tags: deploy
     - role: deploy_container
       tags: deploy
+    - role: remove_container
+      tags: remove

ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/DockerStatic/README.md

@@ -60,6 +60,20 @@ If you are creating a new container with the intention of replacing a container
 
 * Instead of creating a new node in Jenkins, simply modify the name and PORT number of the replaced node's entry in Jenkins accordingly
 
+## Retiring a DockerStatic container (automated)
+
+The [remove_container.yml](https://github.com/adoptium/infrastructure/blob/master/ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/remove_container/tasks/main.yml) role facilitates the removal of DockerStatic containers. Usage:
+
+```
+ansible-playbook -u root -i hosts AdoptOpenJDK_Unix_Playbook/dockernode.yml -t "remove" -e "delete_nodes=$NODE_1,$NODE_2,$NODE_3,..."
+```
+
+The hosts file should include the dockerhost machine which hosts the docker static nodes. Both the Jenkins nodes and the docker containers will be deleted (as long as the Jenkins nodes are idle).
+
+Use the [DockerInventory.json](https://github.com/adoptium/infrastructure/blob/master/ansible/DockerInventory.json) to lookup the docker static node names and the dockerhost machine to which they belong.
+
+After the playbook deletes the nodes, please update the DockerInventory.json file by following the instructions [here](https://github.com/adoptium/infrastructure/tree/master/ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/DockerStatic#inventory).
+
 ## Patching
 
 * The static containers are patched daily using this [script](https://github.com/adoptium/infrastructure/blob/master/ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/DockerStatic/scripts/updatepackages.sh) which runs on a daily cron job on each of the dockerhost machines.

ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/remove_container/scripts/deleteNodes.py

@@ -0,0 +1,59 @@
+import jenkins
+import sys
+import json
+import requests
+
+def createServer(username, password):
+    url = "https://ci.adoptium.net:443"
+    server = jenkins.Jenkins(url, username=username,
+                        password=password)
+    return server
+
+def returnPort(inventory, nodeName):
+    # Search dockerInventory.json for port numbers, return port number
+    for dockerhost in inventory:
+        for node in dockerhost["containers"]:
+            if node["nodeName"] == nodeName:
+                port = node["port"]
+                return port
+
+def deleteJenkinsNode(nodeName, USERNAME, TOKEN):
+    # Post request to doDelete jenkins url
+    headers = {"Content-Type": "application/xml"}
+    auth = (USERNAME, TOKEN)
+    deleteURL = f'https://ci.adoptium.net/computer/{nodeName}/doDelete'
+    r = requests.post(url=deleteURL, auth=auth, headers=headers)
+
+    if r.status_code == 200:
+        print(f'\n{nodeName} deleted\n')
+    else:
+        print(f'\nSomething went wrong. Check to see if {nodeName} is deleted\n')
+        sys.exit(1)
+
+def main():
+    USERNAME,TOKEN = sys.argv[1:3]
+    nodeList = sys.argv[3].split(',')
+
+    server = createServer(USERNAME, TOKEN)
+
+    with open('../../DockerInventory.json') as file:
+        inventory = json.load(file)
+
+    deletedNodesPorts = []
+    isNotIdleNodes = []
+    # Delete in Jenkins first
+    for node in nodeList:
+        testNode = server.get_node_info(node)
+        if testNode['idle']:
+            deleteJenkinsNode(node, USERNAME, TOKEN)
+            deletedNodesPorts.append(returnPort(inventory, node))
+        else:
+            isNotIdleNodes.append(node)
+
+    if len(isNotIdleNodes) > 0:
+        print(f'\nList of nodes that were not deleted: {isNotIdleNodes}\n')
+
+    print(*deletedNodesPorts)
+
+if __name__ == "__main__":
+    main()

ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/remove_container/tasks/delete_container.yml

@@ -0,0 +1,11 @@
+---
+
+# Get the container ID using the port number found in DockerhostInventory.json
+- name: Get container ID of container running on {{ docker_port }}
+  shell: docker ps | grep {{ docker_port }} | sed 's/ /\n/g' | head -n 1
+  register: docker_container_ID
+
+- name: Stop and remove container {{ docker_container_ID.stdout }}
+  community.docker.docker_container:
+    name: "{{ docker_container_ID.stdout }}"
+    state: absent

ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/remove_container/tasks/main.yml

@@ -0,0 +1,24 @@
+---
+# The deleteNodes.py script deletes the nodes in Jenkins
+# Ansible then removes the docker containers on the dockerhost via the delete_container role
+# If the node is not idle in Jenkins, the node is not deleted
+
+# no_log hides the api_token from the output
+- name: Run deleteNodes script
+  script: scripts/deleteNodes.py {{ jenkins_username }} {{ jenkins_api_token }} {{ delete_nodes }}
+  args:
+    executable: python3
+  delegate_to: localhost
+  no_log: true
+  register: docker_ports
+
+- name: Set docker_ports variable
+  set_fact:
+    ports_list: "{{ docker_ports.stdout_lines[0].split(' ') | list }}"
+
+# Iterate over each machine, removes container on dockerhost machine
+- name: Run delete_container.yml for every docker port
+  include_tasks: delete_container.yml
+  loop: "{{ ports_list }}"
+  loop_control:
+    loop_var: docker_port