Skip to content

HADOOP-19814: Create Hadoop 3.5.0 Docker image#8414

Closed
cnauroth wants to merge 1 commit into
apache:docker-hadoop-3.5.0from
cnauroth:HADOOP-19814-docker-hadoop-3.5.0
Closed

HADOOP-19814: Create Hadoop 3.5.0 Docker image#8414
cnauroth wants to merge 1 commit into
apache:docker-hadoop-3.5.0from
cnauroth:HADOOP-19814-docker-hadoop-3.5.0

Conversation

@cnauroth
Copy link
Copy Markdown
Contributor

@cnauroth cnauroth commented Apr 8, 2026

Description of PR

Create a Hadoop 3.5.0 Docker image. I pushed a new branch, docker-hadoop-3.5.0, based on current docker-hadoop-3.4.3. This pull request shows the incremental changes required:

  • Upgrade to a new base image with JDK 17 + Ubuntu 24.04 for alignment with the release image.
  • Change Hadoop version number from 3.4.3 to 3.5.0.
  • Remove fs.default.name from test core-site.xml to avoid log spam about deprecation of this property in 3.5.0.
  • Remove version from docker-compose.yaml to avoid deprecation warning.

This won't actually build successfully until we publish the new hadoop-runner:jdk17-u2404 base image, though you can build locally.

How was this patch tested?

I used the Docker Compose harness to test NameNode, DataNode, ResourceManager and NodeManager.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

If an AI tool was used:

@cnauroth cnauroth requested a review from ayushtkn April 8, 2026 20:55
@github-actions github-actions Bot added the trunk label Apr 8, 2026
@cnauroth
Copy link
Copy Markdown
Contributor Author

cnauroth commented Apr 8, 2026

#8413 is a prerequisite for this.

@cnauroth
Copy link
Copy Markdown
Contributor Author

cnauroth commented Apr 8, 2026

@pan3793 FYI

Copy link
Copy Markdown
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

cnauroth added a commit that referenced this pull request Apr 12, 2026
Closes #8414

Signed-off-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Cheng Pan <chengpan@apache.org>
@cnauroth
Copy link
Copy Markdown
Contributor Author

I merged this to docker-hadoop-3.5.0. Thank you for the reviews, @ayushtkn , @slfan1989 , @pan3793 .

@cnauroth cnauroth closed this Apr 12, 2026
@cnauroth
Copy link
Copy Markdown
Contributor Author

Hello @ayushtkn . Sorry to bother again, but I am not seeing a new build of the 3.5.0 tag after merging this PR.

https://hub.docker.com/layers/apache/hadoop/3.5.0/images/sha256-28b60103770cf4db0f7e745df6492734ea9bb18198d11ca35196eb097c851d65

Is there anything else I need to do?

@ayushtkn
Copy link
Copy Markdown
Member

Hi @cnauroth
Ideally the automated build should have picked up the new commit. I can trigger manually, but some issue with DockerHub
image

@ayushtkn
Copy link
Copy Markdown
Member

@cnauroth
It is failing with

2026-04-15T00:48:48Z Cloning into '.'...
2026-04-15T00:48:48Z Warning: Permanently added the RSA host key for IP address '140.82.114.4' to the list of known hosts.
2026-04-15T00:51:58Z Checking out files:  30% (4935/16394)   
Checking out files:  31% (5083/16394)   
Checking out files:  32% (5247/16394)   
Checking out files:  33% (5411/16394)   
Checking out files:  34% (5574/16394)   
Checking out files:  35% (5738/16394)   
Checking out files:  36% (5902/16394)   
Checking out files:  37% (6066/16394)   
Checking out files:  38% (6230/16394)   
Checking out files:  39% (6394/16394)   
Checking out files:  40% (6558/16394)   
Checking out files:  41% (6722/16394)   
Checking out files:  42% (6886/16394)   
Checking out files:  43% (7050/16394)   
Checking out files:  44% (7214/16394)   
Checking out files:  45% (7378/16394)   
Checking out files:  46% (7542/16394)   
Checking out files:  47% (7706/16394)   
Checking out files:  48% (7870/16394)   
Checking out files:  49% (8034/16394)   
Checking out files:  50% (8197/16394)   
Checking out files:  51% (8361/16394)   
Checking out files:  52% (8525/16394)   
Checking out files:  53% (8689/16394)   
Checking out files:  54% (8853/16394)   
Checking out files:  55% (9017/16394)   
Checking out files:  56% (9181/16394)   
Checking out files:  57% (9345/16394)   
Checking out files:  58% (9509/16394)   
Checking out files:  58% (9544/16394)   
Checking out files:  59% (9673/16394)   
Checking out files:  60% (9837/16394)   
Checking out files:  61% (10001/16394)   
Checking out files:  62% (10165/16394)   
Checking out files:  63% (10329/16394)   
Checking out files:  64% (10493/16394)   
Checking out files:  65% (10657/16394)   
Checking out files:  66% (10821/16394)   
Checking out files:  67% (10984/16394)   
Checking out files:  68% (11148/16394)   
Checking out files:  69% (11312/16394)   
Checking out files:  70% (11476/16394)   
Checking out files:  71% (11640/16394)   
Checking out files:  72% (11804/16394)   
Checking out files:  73% (11968/16394)   
Checking out files:  74% (12132/16394)   
Checking out files:  75% (12296/16394)   
Checking out files:  76% (12460/16394)   
Checking out files:  77% (12624/16394)   
Checking out files:  78% (12788/16394)   
Checking out files:  79% (12952/16394)   
Checking out files:  80% (13116/16394)   
Checking out files:  81% (13280/16394)   
Checking out files:  82% (13444/16394)   
Checking out files:  83% (13608/16394)   
Checking out files:  84% (13771/16394)   
Checking out files:  85% (13935/16394)   
Checking out files:  86% (14099/16394)   
Checking out files:  87% (14263/16394)   
Checking out files:  88% (14427/16394)   
Checking out files:  89% (14591/16394)   
Checking out files:  90% (14755/16394)   
Checking out files:  91% (14919/16394)   
Checking out files:  92% (15083/16394)   
Checking out files:  93% (15247/16394)   
Checking out files:  93% (15408/16394)   
Checking out files:  94% (15411/16394)   
Checking out files:  95% (15575/16394)   
Checking out files:  96% (15739/16394)   
Checking out files:  97% (15903/16394)   
Checking out files:  98% (16067/16394)   
Checking out files:  99% (16231/16394)   
Checking out files: 100% (16394/16394)   
Checking out files: 100% (16394/16394), done.
2026-04-15T00:52:00Z Switched to a new branch 'docker-hadoop-3.5.0'
2026-04-15T00:52:00Z Pulling cache layers for index.docker.io/apache/hadoop:runner-jdk17-u2404...
2026-04-15T00:52:29Z Done!
2026-04-15T00:52:29Z KernelVersion: 5.4.0-1068-aws
2026-04-15T00:52:29Z Components: [{u'Version': u'20.10.15', u'Name': u'Engine', u'Details': {u'KernelVersion': u'5.4.0-1068-aws', u'Os': u'linux', u'BuildTime': u'2022-05-05T13:17:24.000000000+00:00', u'ApiVersion': u'1.41', u'MinAPIVersion': u'1.12', u'GitCommit': u'4433bf6', u'Arch': u'amd64', u'Experimental': u'false', u'GoVersion': u'go1.17.9'}}, {u'Version': u'1.6.21', u'Name': u'containerd', u'Details': {u'GitCommit': u'3dce8eb055cbb6872793272b4f20ed16117344f8'}}, {u'Version': u'1.1.7', u'Name': u'runc', u'Details': {u'GitCommit': u'v1.1.7-0-g860f061'}}, {u'Version': u'0.19.0', u'Name': u'docker-init', u'Details': {u'GitCommit': u'de40ad0'}}]
2026-04-15T00:52:29Z Arch: amd64
2026-04-15T00:52:29Z BuildTime: 2022-05-05T13:17:24.000000000+00:00
2026-04-15T00:52:29Z ApiVersion: 1.41
2026-04-15T00:52:29Z Platform: {u'Name': u'Docker Engine - Community'}
2026-04-15T00:52:29Z Version: 20.10.15
2026-04-15T00:52:29Z MinAPIVersion: 1.12
2026-04-15T00:52:29Z GitCommit: 4433bf6
2026-04-15T00:52:29Z Os: linux
2026-04-15T00:52:29Z GoVersion: go1.17.9
2026-04-15T00:52:29Z Buildkit: Starting build for index.docker.io/apache/hadoop:3.5.0...
2026-04-15T00:52:30Z #1 [internal] load build definition from Dockerfile
2026-04-15T00:52:30Z #1 transferring dockerfile: 1.61kB done
2026-04-15T00:52:30Z #1 DONE 0.1s
2026-04-15T00:52:30Z 
2026-04-15T00:52:30Z #2 [internal] load .dockerignore
2026-04-15T00:52:30Z #2 transferring context: 865B done
2026-04-15T00:52:30Z #2 DONE 0.1s
2026-04-15T00:52:30Z 
2026-04-15T00:52:30Z #3 [internal] load metadata for ghcr.io/apache/hadoop-runner:jdk17-u2404
2026-04-15T00:52:30Z #3 ERROR: ghcr.io/apache/hadoop-runner:jdk17-u2404: not found
2026-04-15T00:52:30Z ------
2026-04-15T00:52:30Z  > [internal] load metadata for ghcr.io/apache/hadoop-runner:jdk17-u2404:
2026-04-15T00:52:30Z ------
2026-04-15T00:52:30Z ERROR: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: ghcr.io/apache/hadoop-runner:jdk17-u2404: not found
2026-04-15T00:52:30Z Build failed using Buildkit (1)

@cnauroth
Copy link
Copy Markdown
Contributor Author

@ayushtkn , OK, I see now there is supposed to be a GitHub action pushing the base runner image to GHCR:

https://github.com/apache/hadoop/blob/docker-hadoop-runner-jdk17-u2404/.github/workflows/build-hadoop-runner-image.yaml

I don't see the base image in GHCR:

https://github.com/apache/hadoop/pkgs/container/hadoop-runner

It looks like the GitHub action failed:

https://github.com/apache/hadoop/actions/runs/24204001269

The actions docker/metadata-action@8e5442c4ef9f78752691e2d8f8d19755c6f78e81, docker/setup-qemu-action@49b3bc8e6bdd4a60e6116a5414239cba5943d3cf, docker/setup-buildx-action@c47758b77c9736f4b2ef4073d4d51994fabfe349, docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567, and docker/build-push-action@4f58ea79222b3b9dc2c8bbdd6debcef730109a75 are not allowed in apache/hadoop because all actions must be from a repository owned by your enterprise, created by GitHub, or match one of the patterns: 1Password/load-secrets-action@13f58eec611f8e5db52ec16247f58c508398f3e6, 1Password/load-secrets-action@8d0d610af187e78a2772c2d18d627f4c52d3fbfb, 1Password/load-secrets-action@92467eb28f72e8255933372f1e0707c567ce2259, 1Password/load-secrets-action@dafbe7cb03502b260e2b2893c753c352eee545bf, AdoptOpenJDK/install-jdk@*, BobAnkh/auto-generate-changelog@*, DavidAnson/markdownlint-cli2-action@07035fd053f7be764496c0f8d8f9f41f98305101, DavidAnson/markdownlint-cli2-action@ce4853d43830c74c1753b39f3cf40f71c...

I'm thinking that we need to make sure our build-hadoop-runner-image.yaml has the same commit hash for every action aligned with what is being allowed by ASF infra:

https://github.com/apache/infrastructure-actions/blob/main/actions.yml

I'll try a PR for this later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants