Add the name of its dockerhost machine to the list of labels of a static docker container in Jenkins

[Haroon-Khel]

For example, on https://ci.adoptium.net/computer/test-docker-ubuntu2004-armv7l-1/ I have added the label dockerhost:dockerhost-equinix-ubuntu2004-armv8-1 hw.dockerhost.dockerhost-equinix-ubuntu2004-armv8-1 so that I can list the static docker containers hosted on dockerhost-equinix-ubuntu2004-armv8-1 in jenkins.

Adding dockerhost-equinix-ubuntu2004-armv8-1 as a label (without the dockerhost:) will not work; jenkins jobs scheduled to run on dockerhost-equinix-ubuntu2004-armv8-1 will instead run on any static docker container that has dockerhost-equinix-ubuntu2004-armv8-1 as a label, so I feel that dockerhost:dockerhost-equinix-ubuntu2004-armv8-1 is more appropriate.

I would like to implement this on the existing static docker containers in Jenkins, as well as make it standard practice when creating new static containers.

If anyone has any ideas for improvement, please feel free to share

[Haroon-Khel]

I guess this also acts as a way to schedule jenkins jobs on containers of a specific dockerhost machine, for troubleshooting purposes etc

[Haroon-Khel]

Using Python's Jenkins API im able to get the following list

[
    {
        "name": "dockerhost-equinix-ubuntu2004-armv8-1",
        "ip": "147.75.35.203",
        "containers": [
            "build-docker-ubuntu2004-armv7l-1",
            "test-docker-alpine313-aarch64-1",
            "test-docker-alpine314-aarch64-1",
            "test-docker-fedora35-armv8l-1",
            "test-docker-ubuntu1804-armv8l-4",
            "test-docker-ubuntu2004-armv7l-1",
            "test-docker-ubuntu2004-armv7l-2",
            "test-docker-ubuntu2004-armv7l-3",
            "test-docker-ubuntu2004-armv8l-2",
            "test-docker-ubuntu2004-armv8l-3",
            "test-docker-ubuntu2010-armv8l-2",
            "test-docker-ubuntu2110-armv8l-1",
            "test-docker-ubuntu2204-armv8l-2"
        ],
        "containersCount": 13
    },
    {
        "name": "dockerhost-equinix-ubuntu2004-x64-1",
        "ip": "145.40.114.58",
        "containers": [
            "test-docker-alpine314-x64-1",
            "test-docker-alpine317-x64-1",
            "test-docker-centos8-x64-1",
            "test-docker-debian11-x64-1",
            "test-docker-fedora35-x64-1",
            "test-docker-fedora37-x64-1",
            "test-docker-ubi8-x64-1",
            "test-docker-ubuntu2004-x64-1",
            "test-docker-ubuntu2204-x64-1"
        ],
        "containersCount": 9
    },
    {
        "name": "dockerhost-equinix-ubuntu2204-armv8-1",
        "ip": "139.178.86.243",
        "containers": [
            "test-docker-alma8-aarch64-1",
            "test-docker-alma8-armv8-2",
            "test-docker-alpine314-aarch64-2",
            "test-docker-alpine314-armv8-3",
            "test-docker-alpine315-armv8-2",
            "test-docker-alpine316-armv8-2",
            "test-docker-alpine317-armv8-1",
            "test-docker-centos8-armv8-1",
            "test-docker-debian11-armv8-1",
            "test-docker-fedora35-armv8-1",
            "test-docker-fedora36-armv8-1",
            "test-docker-fedora37-armv8-1",
            "test-docker-ubi8-armv8-1",
            "test-docker-ubuntu2004-armv8-1",
            "test-docker-ubuntu2204-armv8-2"
        ],
        "containersCount": 15
    },
    {
        "name": "dockerhost-equinix-ubuntu2204-x64-1",
        "ip": "145.40.113.173",
        "containers": [
            "test-docker-alpine314-x64-2",
            "test-docker-alpine317-x64-2",
            "test-docker-centos8-x64-2",
            "test-docker-debian11-x64-2",
            "test-docker-fedora35-x64-2",
            "test-docker-fedora37-x64-2",
            "test-docker-ubi8-x64-2",
            "test-docker-ubuntu2004-x64-2",
            "test-docker-ubuntu2204-x64-2"
        ],
        "containersCount": 9
    },
    {
        "name": "dockerhost-marist-ubuntu2204-s390x-1",
        "ip": "148.100.74.237",
        "containers": [
            "test-docker-sles12-s390x-1",
            "test-docker-sles15-s390x-1"
        ],
        "containersCount": 2
    },
    {
        "name": "dockerhost-osuosl-ubuntu2004-ppc64le-1",
        "ip": "140.211.168.214",
        "containers": [
            "docker-osuosl-ubuntu2004-ppc64le-1",
            "test-docker-fedora33-ppc64le-1",
            "test-docker-ubuntu1804-ppc64le-1",
            "test-docker-ubuntu2010-ppc64le-1"
        ],
        "containersCount": 4
    },
    {
        "name": "dockerhost-osuosl-ubuntu2204-aarch64-1",
        "ip": "140.211.167.67",
        "containers": [],
        "containersCount": 0
    },
    {
        "name": "dockerhost-skytap-ubuntu2004-ppc64le-1",
        "ip": "20.61.136.212",
        "containers": [
            "test-docker-debian11-ppc64le-1",
            "test-docker-debian11-ppc64le-2",
            "test-docker-debian11-ppc64le-3",
            "test-docker-debian11-ppc64le-4",
            "test-docker-ubuntu2204-ppc64le-1",
            "test-docker-ubuntu2204-ppc64le-2"
        ],
        "containersCount": 6
    }
]

The API doesnt look like it has a method for updating a nodes config

[steelhead31]

It does have a method called reconfig_node, which needs an xml input file ( https://python-jenkins.readthedocs.io/en/latest/api.html#jenkins.Jenkins.reconfig_node ) , I wonder if grabbing the nodes existing config as an xml file, updating the file, then using it to reconfig the node, would be a workable approach..

[sxa]

We could definitely benefit from having some way of tracking this, and the label is probably the most effective, even though I'm not a great fan of having such long labels! (Although I'm about to propose making it longer!)

I feel that dockerhost:dockerhost-equinix-ubuntu2004-armv8-1 is more appropriate.

As you've seen there are a few ways to approach collating this information (and the one I've used - finding the IP address and then searching for every other system with the same IP isn't as useful or open as what's being proposed here unless that's what your API calls are doing under the covers) I did wonder whether we could do it in nagios but a jenkins label is potentially the most useful. I'd recommend following the same label format used for selecting test machines and using a "three dots" separated label of hw.dockerhost.dockerhost-equinix-ubuntu2004-armv8-1

Related: There may be benefit in having a tag to indicate that a host is a docker static container. While you could filter on the test-docker- name that's not quite so simple for adding to a test label, although as per #2965 that may not easily allow them to be excluded anyway.

[Haroon-Khel]

finding the IP address and then searching for every other system with the same IP isn't as useful or open as what's being proposed here unless that's what your API calls are doing under the covers

Yep

https://ci.adoptium.net/label/hw.dockerhost.dockerhost-equinix-ubuntu2004-armv8-1/
https://ci.adoptium.net/label/hw.dockerhost.dockerhost-equinix-ubuntu2004-x64-1/
https://ci.adoptium.net/label/hw.dockerhost.dockerhost-equinix-ubuntu2204-armv8-1/
https://ci.adoptium.net/label/hw.dockerhost.dockerhost-equinix-ubuntu2204-x64-1/
https://ci.adoptium.net/label/hw.dockerhost.dockerhost-marist-ubuntu2204-s390x-1/
https://ci.adoptium.net/label/hw.dockerhost.dockerhost-osuosl-ubuntu2004-ppc64le-1/
https://ci.adoptium.net/label/hw.dockerhost.dockerhost-skytap-ubuntu2004-ppc64le-1/

Done

[Haroon-Khel]

A bit of confusion with dockerhost-osuosl-ubuntu2204-aarch64-1. Its in Jenkins, https://ci.adoptium.net/computer/dockerhost-osuosl-ubuntu2204-aarch64-1/, has no containers in Jenkins, and is not in the inventory.yml. I can ssh into it, and it has one centos8 container running

root@dockerhost-osuosl-ubuntu2204-aarch64-1:~# docker ps
CONTAINER ID   IMAGE       COMMAND               CREATED        STATUS       PORTS                                   NAMES
ee6bf41532c6   aqa_cent8   "/usr/sbin/sshd -D"   3 months ago   Up 2 weeks   0.0.0.0:2224->22/tcp, :::2224->22/tcp   cent8.2224

@sxa @steelhead31 Any idea about this machine?

[steelhead31]

It doesnt ring any bells, this seems to mention it.. #2808 (comment)

[sxa]

I suspect that was there for initial testing of testing and has now been superceded by the other dockerhost machine. We should probably create an issue for it to be set up with some real containers - CentOS8 isn't a supported OS any more so I'd suggest that container is not particularly useful. Feel free to create an issue to set it up properly since it seems possible that it's not being used at present.

[Haroon-Khel]

For the alpine containers on any of these dockerhosts, I've changed their labels to hw.dockerhost.alpine-linux.$DOCKERHOST. I noticed that if I kick off a grinder with hw.dockerhost.$DOCKERHOST and the Platform as x86-64_linux, there is an obvious chance that the job will pick up one of the alpine containers on the dockerhost and with that Platform parameter, the job will fail (and vice versa)

[Haroon-Khel]

I suspect that was there for initial testing of testing and has now been superceded by the other dockerhost machine. We should probably create an issue for it to be set up with some real containers - CentOS8 isn't a supported OS any more so I'd suggest that container is not particularly useful. Feel free to create an issue to set it up properly since it seems possible that it's not being used at present.

Created the issue here #2991

This issue can be closed