KNOX-3256: Refactor Docker build to use local Maven artifacts and unify CI/Dev workflows by smolnar82 · Pull Request #1239 · apache/knox

smolnar82 · 2026-05-21T14:59:37Z

KNOX-3256 - Refactor Docker build to use local Maven artifacts and unify CI/Dev workflows

What changes were proposed in this pull request?

This PR refactors the Docker build process used in CI and local development to be more efficient and unified. Key changes include:

Unified Dockerfile: Refactored .github/workflows/build/Dockerfile to copy pre-built tarballs directly from the local target/ directory. This eliminates the need for external staging directories.
Removed Redundancy: Deleted Dockerfile.local which previously cloned the repository and built Knox from scratch inside the container. This was redundant and slow compared to using existing local Maven artifacts.
Simplified Docker Compose: Updated docker-compose.yml to use the project root as the build context, allowing the Dockerfile to access the target/ directory directly.
Streamlined CI Workflow: Simplified .github/workflows/tests.yml by removing manual artifact extraction steps and environment variables that were only needed for the old cloning-based build.

How was this patch tested?

Built Knox locally then ran integration tests:

$ docker compose -f ./.github/workflows/compose/docker-compose.yml up --exit-code-from tests tests
[+] up 1/1
 ! Image apache/knox-dev:master pull access denied for apache/knox-dev, repository does not exist or may require 'docker login'                                                                                                                  1.0s
[+] Building 6.3s (25/25) FINISHED                                                                                                                                                                                                                   
 => [internal] load local bake definitions                                                                                                                                                                                                      0.0s
 => => reading from stdin 546B                                                                                                                                                                                                                  0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                            0.0s
 => => transferring dockerfile: 2.50kB                                                                                                                                                                                                          0.0s
 => WARN: MaintainerDeprecated: Maintainer instruction is deprecated in favor of using label (line 18)                                                                                                                                          0.0s
 => [internal] load metadata for docker.io/library/eclipse-temurin:17-jre                                                                                                                                                                       1.0s
 => [auth] library/eclipse-temurin:pull token for registry-1.docker.io                                                                                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                               0.0s
 => => transferring context: 418B                                                                                                                                                                                                               0.0s
 => [ 1/17] FROM docker.io/library/eclipse-temurin:17-jre@sha256:0d79988c68791ce864fe39d149ab1dc84f680539dca77ee7f6f3b041ad7f2f43                                                                                                               0.0s
 => => resolve docker.io/library/eclipse-temurin:17-jre@sha256:0d79988c68791ce864fe39d149ab1dc84f680539dca77ee7f6f3b041ad7f2f43                                                                                                                 0.0s
 => [internal] load build context                                                                                                                                                                                                               2.8s
 => => transferring context: 931B                                                                                                                                                                                                               2.8s
 => CACHED [ 2/17] RUN useradd -ms /bin/bash gateway                                                                                                                                                                                            0.0s
 => CACHED [ 3/17] RUN mkdir -p /tmp/knox-artifacts /tmp/knoxshell-artifacts /knox-runtime /knoxshell /knox-runtime/knoxshell                                                                                                                   0.0s
 => CACHED [ 4/17] COPY target/*/knox-*.tar.gz /tmp/knox-artifacts/                                                                                                                                                                             0.0s
 => CACHED [ 5/17] COPY target/*/knoxshell-*.tar.gz /tmp/knoxshell-artifacts/                                                                                                                                                                   0.0s
 => CACHED [ 6/17] RUN tar -xvzf /tmp/knox-artifacts/knox-*.tar.gz -C /tmp/knox-artifacts/ &&     tar -xvzf /tmp/knoxshell-artifacts/knoxshell-*.tar.gz -C /tmp/knoxshell-artifacts/ &&     mv /tmp/knox-artifacts/knox-*/* /knox-runtime/ &&   0.0s
 => CACHED [ 7/17] ADD .github/workflows/build/master /knox-runtime/data/security/master                                                                                                                                                        0.0s
 => CACHED [ 8/17] ADD .github/workflows/build/gateway-site.xml /knox-runtime/conf/gateway-site.xml                                                                                                                                             0.0s
 => CACHED [ 9/17] ADD .github/workflows/build/conf/topologies/knoxtoken.xml /knox-runtime/conf/topologies/knoxtoken.xml                                                                                                                        0.0s
 => CACHED [10/17] ADD .github/workflows/build/conf/topologies/health.xml /knox-runtime/conf/topologies/health.xml                                                                                                                              0.0s
 => CACHED [11/17] ADD .github/workflows/build/conf/topologies/knoxldap.xml /knox-runtime/conf/topologies/knoxldap.xml                                                                                                                          0.0s
 => CACHED [12/17] ADD .github/workflows/build/conf/topologies/remoteauth.xml /knox-runtime/conf/topologies/remoteauth.xml                                                                                                                      0.0s
 => CACHED [13/17] RUN chown -R gateway /knox-runtime/                                                                                                                                                                                          0.0s
 => CACHED [14/17] ADD .github/workflows/build/ldap.sh /ldap.sh                                                                                                                                                                                 0.0s
 => CACHED [15/17] ADD .github/workflows/build/gateway.sh /gateway.sh                                                                                                                                                                           0.0s
 => CACHED [16/17] RUN chmod +x /ldap.sh                                                                                                                                                                                                        0.0s
 => CACHED [17/17] RUN chmod +x /gateway.sh                                                                                                                                                                                                     0.0s
 => exporting to image                                                                                                                                                                                                                          2.1s
 => => exporting layers                                                                                                                                                                                                                         0.0s
 => => exporting manifest sha256:1db7f1e6795f015e9a76f07bff25dac20fe22d1c08003ed40b262957528623b9                                                                                                                                               0.0s
 => => exporting config sha256:64efa91dda3f0d5d142a96a65553ebdd1a71e446b0a528670b56c4ceac91015a                                                                                                                                                 0.0s
 => => exporting attestation manifest sha256:19d358d43fd7c1e2209d51fc79a26a92a44c5ebb114f7fedd07397273b84926d                                                                                                                                   0.0s
 => => exporting manifest list sha256:632c393e51d0831e714b0630c50295c8d57656ac178a536ea890f5941c8025a8                                                                                                                                          0.0s
 => => naming to docker.io/apache/knox-dev:master                                                                                                                                                                                               0.0s
[+] up 6/6acking to docker.io/apache/knox-dev:master                                                                                                                                                                                            2.0s
 ✔ Image apache/knox-dev:master Built                                                                                                                                                                                                            8.0s
 ✔ Network compose_default      Created                                                                                                                                                                                                          0.0s
 ✔ Container compose-knox-dev-1 Created                                                                                                                                                                                                          0.5s
 ✔ Container compose-ldap-1     Created                                                                                                                                                                                                          0.1s
 ✔ Container compose-knox-1     Created                                                                                                                                                                                                          0.1s
 ✔ Container compose-tests-1    Created                                                                                                                                                                                                          0.0s
Attaching to tests-1
tests-1  | Collecting requests==2.32.4
tests-1  |   Downloading requests-2.32.4-py3-none-any.whl (64 kB)
tests-1  |      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.8/64.8 kB 1.0 MB/s eta 0:00:00
tests-1  | Collecting pytest==8.3.4
tests-1  |   Downloading pytest-8.3.4-py3-none-any.whl (343 kB)
tests-1  |      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 343.1/343.1 kB 3.6 MB/s eta 0:00:00
tests-1  | Collecting idna<4,>=2.5
tests-1  |   Downloading idna-3.15-py3-none-any.whl (72 kB)
tests-1  |      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 72.3/72.3 kB 6.1 MB/s eta 0:00:00
tests-1  | Collecting urllib3<3,>=1.21.1
tests-1  |   Downloading urllib3-2.6.3-py3-none-any.whl (131 kB)
tests-1  |      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 131.6/131.6 kB 9.5 MB/s eta 0:00:00
tests-1  | Collecting charset_normalizer<4,>=2
tests-1  |   Downloading charset_normalizer-3.4.7-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (200 kB)
tests-1  |      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 200.6/200.6 kB 10.4 MB/s eta 0:00:00
tests-1  | Collecting certifi>=2017.4.17
tests-1  |   Downloading certifi-2026.5.20-py3-none-any.whl (134 kB)
tests-1  |      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.1/134.1 kB 9.1 MB/s eta 0:00:00
tests-1  | Collecting pluggy<2,>=1.5
tests-1  |   Downloading pluggy-1.6.0-py3-none-any.whl (20 kB)
tests-1  | Collecting tomli>=1
tests-1  |   Downloading tomli-2.4.1-py3-none-any.whl (14 kB)
tests-1  | Collecting packaging
tests-1  |   Downloading packaging-26.2-py3-none-any.whl (100 kB)
tests-1  |      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.2/100.2 kB 13.0 MB/s eta 0:00:00
tests-1  | Collecting exceptiongroup>=1.0.0rc8
tests-1  |   Downloading exceptiongroup-1.3.1-py3-none-any.whl (16 kB)
tests-1  | Collecting iniconfig
tests-1  |   Downloading iniconfig-2.1.0-py3-none-any.whl (6.0 kB)
tests-1  | Collecting typing-extensions>=4.6.0
tests-1  |   Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB)
tests-1  |      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.6/44.6 kB 9.0 MB/s eta 0:00:00
tests-1  | Installing collected packages: urllib3, typing-extensions, tomli, pluggy, packaging, iniconfig, idna, charset_normalizer, certifi, requests, exceptiongroup, pytest
tests-1  | Successfully installed certifi-2026.5.20 charset_normalizer-3.4.7 exceptiongroup-1.3.1 idna-3.15 iniconfig-2.1.0 packaging-26.2 pluggy-1.6.0 pytest-8.3.4 requests-2.32.4 tomli-2.4.1 typing-extensions-4.15.0 urllib3-2.6.3
tests-1  | WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
tests-1  | 
tests-1  | [notice] A new release of pip is available: 23.0.1 -> 26.0.1
tests-1  | [notice] To update, run: pip install --upgrade pip
tests-1  | Waiting for knox...
tests-1  | ============================= test session starts ==============================
tests-1  | platform linux -- Python 3.9.25, pytest-8.3.4, pluggy-1.6.0
tests-1  | rootdir: /tests
tests-1  | collected 21 items
tests-1  | 
tests-1  | test_health.py .....                                                     [ 23%]
tests-1  | test_knox_auth_service_and_LDAP.py ..                                    [ 33%]
tests-1  | test_knox_configs.py .                                                   [ 38%]
tests-1  | test_knoxauth_preauth_and_paths.py ......                                [ 66%]
tests-1  | test_remote_auth.py ...                                                  [ 80%]
tests-1  | test_remoteauth_extauthz_additional_path.py ....                         [100%]
tests-1  | 
tests-1  | =============================== warnings summary ===============================
tests-1  | test_health.py: 5 warnings
tests-1  | test_knox_auth_service_and_LDAP.py: 2 warnings
tests-1  | test_knox_configs.py: 1 warning
tests-1  | test_knoxauth_preauth_and_paths.py: 6 warnings
tests-1  | test_remote_auth.py: 3 warnings
tests-1  | test_remoteauth_extauthz_additional_path.py: 4 warnings
tests-1  |   /usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host 'knox'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
tests-1  |     warnings.warn(
tests-1  | 
tests-1  | -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
tests-1  | ----------------- generated xml file: /tests/test-results.xml ------------------
tests-1  | ======================= 21 passed, 21 warnings in 0.67s ========================
tests-1 exited with code 0
Aborting on container exit...
Container compose-tests-1 Stopping 
Container compose-tests-1 Stopped

Integration Tests

Updated .github/workflows/tests.yml to use the new simplified build process. The integration tests themselves remain the same but now run against an image built more reliably from the local build output.

UI changes

N/A

…fy CI/Dev workflows

moresandeep · 2026-05-21T16:19:23Z

-      args:
-        knoxurl: ${knoxurl:-https://github.com/apache/knox.git}
-        branch: ${branch:-master}
+      context: ../../../


Add a .dockerignore at the root that allowlists only what the Dockerfile needs: e.g. * !target//knox-.tar.gz !target//knoxshell-.tar.gz !.github/workflows/build/

The build log shows about 200 MB of artifacts being transferred on every build. Looks like every build sends the full Maven target directory, .git history, docs, and any local credentials to the Docker daemon. This is both slow and a potential (I am not sure yet) secrets leakage.

moresandeep · 2026-05-21T16:25:41Z

      - name: Build Docker Images
        run: |
-          export knoxurl=${KNOX_URL}
-          export branch=${BRANCH:-master}


knox-dev service built the official image tagged apache/knox-dev:master. Service now builds an image tagged local-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ID:-local}. basically, there is no master tag.
This causes two issues, there will be lots of images in the Apache repo which will exhaust our quota, secondly, it is not easy to pickup a knox image will the most latest changes. The idea behind using apache/knox-dev:master was to save space and provide folks with an image with all commits on master.

github-actions · 2026-05-21T17:18:30Z

Test Results

21 tests 21 ✅ 1s ⏱️
1 suites 0 💤
1 files 0 ❌

Results for commit 8f2f248.

♻️ This comment has been updated with latest results.

smolnar82 · 2026-05-21T21:27:01Z

@moresandeep - I addressed your review comments; please give it another shot. Thanks!

KNOX-3256: Refactor Docker build to use local Maven artifacts and uni…

7e5e165

…fy CI/Dev workflows

smolnar82 requested a review from moresandeep May 21, 2026 14:59

smolnar82 assigned hanicz and smolnar82 and unassigned hanicz May 21, 2026

smolnar82 requested a review from hanicz May 21, 2026 15:00

smolnar82 added docker integrationTest labels May 21, 2026

moresandeep requested changes May 21, 2026

View reviewed changes

KNOX-3256: Address code review comments

cc4a1de

smolnar82 requested a review from moresandeep May 21, 2026 20:32

Added missing Apache License

8f2f248

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KNOX-3256: Refactor Docker build to use local Maven artifacts and unify CI/Dev workflows#1239

KNOX-3256: Refactor Docker build to use local Maven artifacts and unify CI/Dev workflows#1239
smolnar82 wants to merge 3 commits into
apache:masterfrom
smolnar82:KNOX-3256

smolnar82 commented May 21, 2026 •

edited

Loading

Uh oh!

moresandeep May 21, 2026

Uh oh!

smolnar82 May 21, 2026

Uh oh!

moresandeep May 21, 2026

Uh oh!

smolnar82 May 21, 2026

Uh oh!

github-actions Bot commented May 21, 2026 •

edited

Loading

Uh oh!

smolnar82 commented May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

smolnar82 commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Integration Tests

UI changes

Uh oh!

moresandeep May 21, 2026

Choose a reason for hiding this comment

Uh oh!

smolnar82 May 21, 2026

Choose a reason for hiding this comment

Uh oh!

moresandeep May 21, 2026

Choose a reason for hiding this comment

Uh oh!

smolnar82 May 21, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

smolnar82 commented May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

smolnar82 commented May 21, 2026 •

edited

Loading

github-actions Bot commented May 21, 2026 •

edited

Loading