feat: build both agent and inbound agent

[lemeurherve]

This PR allows to build both agent and inbound-agent container images in this repository.

Note: each type of image ("agent" or "inbound-agent") has its corresponding tests suite.

I've splitted this pull requests in atomic operations:

• rename docker-inbound-agent files before import: lemeurherve@ab173cf
• rename docker-agent files before import: 77850e6
• import prepared docker-inbound-agent files: cfc94cc (merge commit to keep imported files history)
• feat: build both agent and inbound-agent images: 0ab91ff (this is the main one to review)

New/renamed/imported/modified files list:

New file

• README.md

Modified

• .gitignore
• alpine/Dockerfile
• build.ps1
• build.sh
• debian/Dockerfile
• debian/preview/Dockerfile
• docker-bake.hcl
• Jenkinsfile
• Makefile
• tests/agent.Tests.ps1
• tests/test_helpers.bash
• tests/test_helpers.psm1
• windows/nanoserver/Dockerfile
• windows/windowsservercore/Dockerfile

Renamed from agent

• README_agent.md

Renamed and modified from agent

• tests/tests_agent.bats

Imported from inbound-agent

• jenkins-agent
• jenkins-agent.ps1
• tests/netcat-helper/Dockerfile
• tests/netcat-helper/Dockerfile-windows

Imported and renamed from inbound-agent

• README_inbound-agent.md

Imported and modified from inbound-agent

• tests/inbound-agent.Tests.ps1

Imported, renamed and modified from inbound-agent

• .github/workflows/update-dockerhub-description.yaml (was sync-dockerhub-readme.yml)
• tests/tests_inbound-agent.bats

How I imported files from inbound-agent with their git history:

---

I created a script (adapted from https://gist.github.com/tsayen/f1c1c4d62d4fda77abf1586bd39f9b74) with the following content to retrieve git history of a file as a set of email patches and to replace `#n` references by `jenkinsci/docker-inbound-agent#n`:

#! /bin/bash
# Usage:
# ./git-patch.sh path/to/file/or/dir path/to/destination/repo
echo "creating patch for path ${1}"
git log --name-only --pretty="format:" --follow  "${1}" \
 | sort -u | \
 xargs git log --pretty=email --patch-with-stat --reverse --full-index --binary -m --first-parent -- > "${2}/_patch_" \
&& cat "${2}/_patch_" | sed 's/Merge pull request #/Merge pull request jenkinsci\/docker-inbound-agent#/' >> "${2}/_all_patches_"

As applying this script on multiple files at once resulted in a unordered history, I applied this script on every imported inbound-agent file from the list above, one after the other, manually (and painfully) reordering each generated patch in the correct order.

When all files have been proceed, I've got as a result a big patch file that I applied to the agent repository folder with the following command:

git am --committer-date-is-author-date --keep < ../_all_patches_

Here is the corresponding patches file (renammed in .txt to be attached here): [_all_patches_.txt](https://github.com/jenkinsci/docker-agent/files/13601317/_all_patches_.txt)

State of the agent repository after applying these patches and before renaming/modifying files if you want to check that the content of these files haven't been changed by this process and that no unwanted modifications have been introduced:
https://github.com/lemeurherve/docker-agent/tree/aa4262466bab1782f9c946357205c76a596c862f

The downside of this process is that even if I'm not the author of these inbound-agent commits, I didn't manage not to be the (co-)commiter of them.

---

EDIT: I wrote a script using https://github.com/newren/git-filter-repo allowing me to rework this PR and this import to get a proper history.

#!/usr/bin/env bash

set -ex

# Cleanup
rm -rf docker-agent
rm -rf docker-inbound-agent
rm -rf pullrequest

# Retrieve content of the pull request from my fork, will be used later
git clone https://github.com/lemeurherve/docker-agent pullrequest
cd pullrequest
git fetch --all
git checkout feat-build-both-agent-and-inbound-agent

cd ..

# Commit message replacement
echo "Merge pull request #==>Merge pull request jenkinsci/docker-inbound-agent#" > commits-msg-replacement.txt

# Prepare docker-inbound-agent for import
git clone https://github.com/jenkinsci/docker-inbound-agent

cd docker-inbound-agent

# Trim repo to keep only files to import, and replace GitHub pull request references in commit messages
git filter-repo \
    --path tests/netcat-helper/Dockerfile \
    --path tests/netcat-helper/Dockerfile-windows \
    --path tests/inboundAgent.Tests.ps1 \
    --path tests/tests.bats \
    --path .github/workflows/sync-dockerhub-readme.yaml \
    --path jenkins-agent \
    --path jenkins-agent.ps1 \
    --path README.md \
    --replace-message ../commits-msg-replacement.txt

# Rename files before import to avoid collisions
git mv README.md README_inbound-agent.md
git mv .github/workflows/sync-dockerhub-readme.yaml .github/workflows/update-dockerhub-description.yaml
git mv tests/inboundAgent.Tests.ps1 tests/inbound-agent.Tests.ps1
git mv tests/tests.bats tests/tests_inbound-agent.bats

git status
git commit -m "rename \`docker-inbound-agent\` files before import"

cd ..


# Prepare docker-agent before import
git clone https://github.com/jenkinsci/docker-agent

cd docker-agent

# Rename files before import to avoid collisions
git mv README.md README_agent.md
git mv tests/tests.bats tests/tests_agent.bats

git status
git commit -m "rename \`docker-agent\` files before import"

# Import prepared docker-inbound-agent into docker-agent'
git remote add docker-inbound-agent ../docker-inbound-agent
git fetch docker-inbound-agent --no-tags
git merge --allow-unrelated-histories docker-inbound-agent/master -m "import prepared \`docker-inbound-agent\` files"
git remote remove docker-inbound-agent

### Import finished
### Below are the commands used to retrieve the modifications from this pull request automatically

# Update docker-agent with the modified content from the pull request
## New file
cp ../pullrequest/README.md README.md

## Modified in agent
cp ../pullrequest/.gitignore .gitignore
cp ../pullrequest/alpine/Dockerfile alpine/Dockerfile
cp ../pullrequest/build.ps1 build.ps1
cp ../pullrequest/build.sh build.sh
cp ../pullrequest/debian/Dockerfile debian/Dockerfile
cp ../pullrequest/debian/preview/Dockerfile debian/preview/Dockerfile
cp ../pullrequest/docker-bake.hcl docker-bake.hcl
cp ../pullrequest/Jenkinsfile Jenkinsfile
cp ../pullrequest/Makefile Makefile
cp ../pullrequest/tests/agent.Tests.ps1 tests/agent.Tests.ps1
cp ../pullrequest/tests/test_helpers.bash tests/test_helpers.bash
cp ../pullrequest/tests/test_helpers.psm1 tests/test_helpers.psm1
cp ../pullrequest/windows/nanoserver/Dockerfile windows/nanoserver/Dockerfile
cp ../pullrequest/windows/windowsservercore/Dockerfile windows/windowsservercore/Dockerfile

## Renamed and modified from agent
cp ../pullrequest/tests/tests_agent.bats tests/tests_agent.bats

## Imported and modified from inbound-agent
cp ../pullrequest/tests/inbound-agent.Tests.ps1 tests/inbound-agent.Tests.ps1

## Imported, renamed and modified from inbound-agent
cp ../pullrequest/.github/workflows/update-dockerhub-description.yaml .github/workflows/update-dockerhub-description.yaml
cp ../pullrequest/tests/tests_inbound-agent.bats tests/tests_inbound-agent.bats

# Commit changes from pull request
git status
git add .
git commit -m "feat: build both \`agent\` and \`inbound-agent\` images"

# Create verification branch
git checkout -b verification-build-both-agent-and-inbound-agent

# Create branch for the pull request update
git checkout -b feat-build-both-agent-and-inbound-agent

# Set appropriate remotes
git remote remove origin
git remote add origin https://github.com/lemeurherve/docker-agent.git
git remote add upstream https://github.com/jenkinsci/docker-agent.git

As a bonus, I can reapply this script whenever I want then I just have to git push origin feat-build-both-agent-and-inbound-agent --force to update this PR.

Ref and details about proposed procedure for the next steps:

• Build both agent and inbound-agent container images in this repository #569

Testing done

• Ran on a Macintosh M2 for Linux images and on a Windows VM with the same image used by agents for Windows images.
• CI build: https://ci.jenkins.io/job/Packaging/job/docker-agent/job/PR-570/10/
• Compared files lists from sudo find / between these images and the existing ones
• Used docker scout compare between these images and the existing ones: https://docs.docker.com/engine/reference/commandline/scout_compare/
• As suggested by @basil, ran a basic job using https://github.com/jenkinsci/docker-plugin with both "Launch attached" (agent) and "Launch via JNLP" (inbound-agent) with the existing images, then with these images

Submitter checklist

Edit tasklist title

Beta Give feedback Tasklist Submitter checklist, more options

• Rename
• Copy markdown
• Delete tasklist

Delete tasklist

Delete tasklist block?

Are you sure? All relationships in this tasklist will be removed.

Cancel Delete

1. Make sure you are opening from a **topic/feature/bugfix branch** (right side) and not your main branch!

   Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
2. Ensure that the pull request title represents the desired changelog entry

   Ensure that the pull request title represents the desired changelog entry
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
3. Please describe what you did

   Please describe what you did
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
4. Link to relevant issues in GitHub or Jira

   Link to relevant issues in GitHub or Jira
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
5. Link to relevant pull requests, esp. upstream and downstream changes

   Link to relevant pull requests, esp. upstream and downstream changes
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
6. Ensure you have provided tests - that demonstrates feature works or fixes the issue

   Ensure you have provided tests - that demonstrates feature works or fixes the issue
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove

Add item to Submitter checklist

[basil]

Thank you, this sounds fine to me.

[basil]

(My comments about the merge of the other repository's commits remain unaddressed.)

[lemeurherve]

Currently trying git-filter-repo to see if I can properly reimport inbound-agent files and their git history without too much time and effort.

[lemeurherve]

Currently trying git-filter-repo

Got it working, thanks for the hint @basil.

I'm going to push with force on this branch to keep existing comments here, only timja review should be discarded.

EDIT: nothing lost, just broke my own review comments not visible anymore in the "Files" tabs. I can repost them if needed.

[lemeurherve]

Check the updated PR body if you want to see the script I used to get a proper import of docker-inbound-agent files and their history.

Force pushed twice as I initially forgot to replace Merge pull request # by Merge pull request jenkinsci/docker-inbound-agent# in imported docker-inbound-agent commits messages to get proper GitHub references.

[basil]

Very nice!

[lemeurherve]

Release of https://github.com/jenkinsci/docker-agent/releases/tag/3206.vb_15dcf73f6a_9-2 for both agent and inbound-agent successful!

The first build on trusted.ci.jenkins.io failed publishing the "linux" images with the following error:

unexpected status from HEAD request to https://registry-1.docker.io/v2/jenkins/agent/blobs/...: 429 Too Many Requests

After a few minutes, I replayed a build only on "linux" images, it finished with success.

--- old/Jenkinsfile
+++ new/Jenkinsfile
@@ -24,7 +24,7 @@
                 axes {
                     axis {
                         name 'IMAGE_TYPE'
-                        values 'linux', 'nanoserver-1809', 'nanoserver-ltsc2019', 'nanoserver-ltsc2022', 'windowsservercore-1809', 'windowsservercore-ltsc2019', 'windowsservercore-ltsc2022'
+                        values 'linux'
                     }
                 }

https://hub.docker.com/r/jenkins/agent/tags
https://hub.docker.com/r/jenkins/inbound-agent/tags

[ksalerno99]

Great work, guys! This eliminates 11 lines from my test build environment!

docker exec -ti docker-buildx-$ARCH /bin/bash -c "cd; \
			git clone https://github.com/jenkinsci/docker-inbound-agent.git; \
			cd docker-inbound-agent && \
			TAG=\$(grep -A1 PARENT_IMAGE_VERSION docker-bake.hcl | \
				head -2 | tail -1 | awk '{ print \$NF }' | \
				tr -d '\"') && \
			docker image tag jenkins/agent:jdk17 \
				jenkins/agent:\$TAG-jdk11 && \
			docker image tag jenkins/agent:jdk21 \
				jenkins/agent:\$TAG-jdk17 && \
			time make build"