fix(upgrade): move ingestion images to Python 3.12 and strip build-only tooling - #30878
fix(upgrade): move ingestion images to Python 3.12 and strip build-only tooling#30878harshsoni2024 wants to merge 8 commits into
Conversation
…park jar surgery The ingestion-slim image (ingestion/operators/docker/Dockerfile*) shipped vulnerable jars bundled by PySpark 3.5.6 that the earlier CVE fix (#30516) never patched — that fix only touched the airflow image (ingestion/Dockerfile*), so the PySpark jar surgery never reached the slim image. AWS Inspector kept flagging zookeeper, jackson-asl, netty, and a phantom black finding on it. Extract the jar patch into a single SHA256-pinned script and run it in all four ingestion Dockerfiles so the two images can no longer drift: - zookeeper 3.6.3 -> 3.7.2 CVE-2023-44981 (SASL quorum auth bypass) - jackson-mapper/core-asl 1.9.13 CVE-2019-10202 (removed; no upstream fix) - netty-codec-http 4.1.96 -> 4.1.135 CVE-2026-42581 / CVE-2026-42584 (both fixed in 4.1.133.Final; staying on the 4.1.x line keeps binary compatibility with PySpark's sibling netty 4.1.96 jars — verified pyspark and SparkSession still import) Also strip spacy's bundled tests/package/requirements.txt from the slim images: it pins an old black that scanners misreport as an installed package (CVE-2026-31900). The file is test-only and never imported at runtime. Left as documented residuals (no drop-in fix for this runtime): - derby 10.14.2.0 (CVE-2022-46337): only fix needs Java 21, image is Java 17; vulnerable LDAP path unused - jetty 9.4.x (CVE-2026-2332): shaded inside hadoop-client-runtime / spark-core uber-jars, not a swappable standalone jar; clears on a future pyspark bump Verified against a local build of the slim CI image: patched jars present, jackson-asl and the black fixture gone, pyspark imports cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…cleanup into jar patch Address review feedback on the PySpark CVE-jar remediation: - Detection no longer fails open. Previously `python -c 'import pyspark' 2>/dev/null || true` treated a broken-but-installed pyspark the same as an absent one, letting an image with unpatched jars ship silently. The script now distinguishes three cases via exit code: genuinely absent (skip), installed-but-unimportable (fail the build), importable (patch). - Fold the spaCy test-fixture cleanup (spacy/tests/**/requirements.txt, the phantom black CVE-2026-31900) into patch_pyspark_jars.sh so it runs wherever the patch runs, and drop the separate per-Dockerfile `find ... -delete` RUN lines that could drift. Self-guarding and idempotent: no-op when pyspark is absent (INGESTION_DEPENDENCY=slim builds), active when present (default `all`). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lerant delete Address review feedback on patch_pyspark_jars.sh: - The spaCy fixture cleanup was placed after the early `exit 0` that fires when PySpark is absent, so a build with spaCy but no PySpark (e.g. INGESTION_DEPENDENCY=pii-processor — spaCy comes from pii-processor/sample-data, PySpark from deltalake, independent extras) skipped the strip and reintroduced the CVE-2026-31900 phantom finding. Extract it into strip_spacy_scanner_fixture() and call it before the PySpark exit branches so it runs on every path. - Restore the best-effort delete: `find ... -delete` under `set -euo pipefail` aborted the build on any undeletable file (read-only layer). Back to `-delete 2>/dev/null || true` so the fixture strip stays non-fatal. ]
…ils closed instead of skipping
python:3.10-slim-bookworm is EOL and no longer rebuilt, so its Debian layer and bundled Python tooling are frozen. Move all four ingestion images to python:3.12-slim-bookworm / apache/airflow:3.2.2-python3.12. - ingestion/Dockerfile pinned the Airflow *Python 3.10* constraints file while building on a python3.12 base; point it at constraints-3.12.txt. - Regenerate the vendored airflow-constraints-3.2.2.txt from upstream's 3.12 variant. Both come from the same 2026-05-26 generation run, so this is the matching sibling: it drops the py<3.11 backports (backports.strenum, exceptiongroup, tomli) and refreshes numpy/scipy/scikit-learn. Verified openmetadata-airflow-apis still installs under it and that the new setuptools==82.0.1 pin stays inert against the deliberate setuptools<81. - Upgrade the interpreter-level pip to a pinned 26.2 before dropping privileges. The existing `pip install --upgrade pip` runs after the USER switch, so it only reached ~/.local and left the base image's copy on disk for scanners (CVE-2025-8869, CVE-2026-3219, CVE-2026-6357, CVE-2026-8643, fixed in 26.1.2). Python 3.12 alone is not enough - its bundled pip is 25.0.1. The airflow images need the explicit /usr/local/bin/python: as root, `pip` hits airflow's wrapper and exits 1, and bare `python` resolves to /home/airflow/.local/bin/python, so both would miss the very copy this line exists to fix. - Uninstall datamodel-code-generator and the black it depends on after the codegen step in the two CI images. Neither is imported at runtime; they exist only to run datamodel_generation.py, and account for eight HIGH findings. Bumping the pin is not an option - both codegen scripts monkeypatch datamodel_code_generator.model.pydantic.types.IMPORT_SECRET_STR (the CustomSecretStr swap), which no longer exists in 0.26+. The uninstall runs after the final pip install so no extra can pull them back in. Python 3.12 also ships ensurepip without setuptools/wheel, removing the vendored wheel-0.45.1 behind CVE-2026-24049. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
❌ PR checklist incompleteThis PR cannot be merged until the following are addressed on its linked issue:
The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically. Maintainers can bypass this check by adding the |
github-actions
Bot
added
Ingestion
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
| rm -f zookeeper-*.jar zookeeper-jute-*.jar \ | ||
| jackson-mapper-asl-*.jar jackson-core-asl-*.jar \ | ||
| netty-codec-http-*.jar | ||
|
|
||
| fetch_jar zookeeper-3.7.2.jar \ | ||
| org/apache/zookeeper/zookeeper/3.7.2/zookeeper-3.7.2.jar \ | ||
| b12d6fb4afd7b3849d3a9a5a38b9260c23a12e1ea58ca8c8d775880249cb8eac | ||
|
|
||
| fetch_jar zookeeper-jute-3.7.2.jar \ | ||
| org/apache/zookeeper/zookeeper-jute/3.7.2/zookeeper-jute-3.7.2.jar \ | ||
| ad15d812b1f01f373638443adcda0e23fda549d65ced2be8c4f64bef33b5d774 | ||
|
|
||
| fetch_jar netty-codec-http-4.1.135.Final.jar \ | ||
| io/netty/netty-codec-http/4.1.135.Final/netty-codec-http-4.1.135.Final.jar \ | ||
| 4018529d3d6aecf4044b98c75d9a90c91839ddf49c7aa484c5ac81c90a15da02 |
There was a problem hiding this comment.
⚠️ Edge Case: Netty module version skew after codec-http bump
The script deletes only netty-codec-http-*.jar and re-fetches netty-codec-http-4.1.135.Final, while PySpark 3.5.6's sibling Netty jars (netty-common, netty-buffer, netty-handler, netty-transport, netty-codec, netty-resolver) remain at 4.1.96.Final. Netty does not guarantee forward binary compatibility across patch releases within 4.1.x, so a 4.1.135 codec-http calling a method added after 4.1.96 in a sibling module can raise NoSuchMethodError/LinkageError at runtime on the Hive/Deltalake path. The build's SHA-pinned checks won't catch this since it's a runtime linkage issue. Consider bumping all bundled netty-*-4.1.96 jars to 4.1.135.Final together (or the lowest common release ≥4.1.133 that fixes the CVEs) so the module set stays version-consistent.
Was this helpful? React with 👍 / 👎
Code Review
|
| Compact |
|
Was this helpful? React with 👍 / 👎 | Gitar | Powered by Gitar — free for open source
There was a problem hiding this comment.
Pull request overview
This PR upgrades the ingestion Docker images to Python 3.12 bases (including Airflow 3.2.2 python3.12), refreshes interpreter-level pip early to reduce scanner findings, consolidates PySpark jar/CVE remediation into a shared script, and updates the Airflow constraints to match the Python 3.12 variant.
Changes:
- Bump ingestion image bases to
python:3.12-slim-bookwormandapache/airflow:3.2.2-python3.12, and align the Airflow constraints URL toconstraints-3.12.txt. - Pin/refresh interpreter-level
pipto26.2before dropping privileges, and run PySpark/spaCy CVE cleanup viaingestion/scripts/patch_pyspark_jars.sh. - Regenerate
ingestion/airflow-constraints-3.2.2.txtand uninstall build-only codegen tooling (datamodel-code-generator,black) in CI images.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| ingestion/scripts/patch_pyspark_jars.sh | New shared script to patch PySpark-bundled vulnerable jars and remove spaCy scanner fixture deterministically. |
| ingestion/operators/docker/Dockerfile.ci | Slim CI image moved to Python 3.12, pins interpreter pip, runs jar/fixture patch, and uninstalls codegen tooling. |
| ingestion/operators/docker/Dockerfile | Slim runtime image moved to Python 3.12, pins interpreter pip, and runs jar/fixture patch. |
| ingestion/Dockerfile.ci | Airflow CI image moved to Python 3.12, pins interpreter pip via explicit interpreter path, runs jar/fixture patch, and uninstalls codegen tooling. |
| ingestion/Dockerfile | Airflow runtime image moved to Python 3.12, fixes constraints URL to constraints-3.12.txt, pins interpreter pip, and runs jar/fixture patch. |
| ingestion/airflow-constraints-3.2.2.txt | Updated to the regenerated Python 3.12-compatible constraints set for Airflow 3.2.2. |
| # datamodel-code-generator (and the black it depends on) exist only to run the codegen | ||
| # step above -- nothing imports them at runtime. Drop them so the shipped image stops | ||
| # reporting their CVEs. Runs after the final pip install so an extra cannot pull them | ||
| # back in. Bumping the pin instead is not an option: the codegen script monkeypatches | ||
| # datamodel_code_generator.model.pydantic.types.IMPORT_SECRET_STR, gone in 0.26+. | ||
| RUN pip uninstall -y datamodel-code-generator black |
✅ Playwright Results — workflow succeededValidated commit ✅ 105 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 3 skipped · 🧰 0 lifecycle flaky PerformanceBlocking targets: ✅ met · Optimization targets: 🟡 in progress Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting. 🕒 Full workflow signal wall (to summary) 49m 20s ⏱️ Max setup 3m 13s · max shard execution 11m 22s · max shard-job elapsed before upload 19m 31s · reporting 4s 🌐 208.55 requests/attempt · 1.78 app boots/UI scenario · 0.00% common-shard skew Optimization targets still in progress:
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
Describe your changes:
Follow-up to #30751. That PR patched PySpark's vulnerable jars in place; this one
addresses the rest of the CVE surface in the ingestion images — the frozen base
layer and build tooling that ships into the runtime image.
python:3.10-slim-bookwormis EOL and no longer rebuilt, so its Debian layer andbundled Python tooling are frozen. All four ingestion Dockerfiles move to
python:3.12-slim-bookworm/apache/airflow:3.2.2-python3.12.Key changes
Base bump to Python 3.12 across
ingestion/Dockerfile{,.ci}andingestion/operators/docker/Dockerfile{,.ci}. 3.12'sensurepipships pip only —no setuptools, no wheel — which removes the vendored
wheel-0.45.1behind CVE-2026-24049.
Bug fix:
ingestion/Dockerfilepinned the Airflow Python 3.10 constraintsfile while building on a python3.12 base → now
constraints-3.12.txt.Regenerated
airflow-constraints-3.2.2.txtfrom upstream's 3.12 variant. Bothfiles come from the same 2026-05-26 generation run, so this is the matching sibling:
it drops the
py<3.11backports (backports.strenum,exceptiongroup,tomli) andrefreshes numpy/scipy/scikit-learn.
Interpreter-level pip pinned to 26.2, before the
USERswitch. The existingpip install --upgrade pipruns after that switch, so it only reached~/.localand left the base image's copy on disk for scanners (CVE-2025-8869, CVE-2026-3219,
CVE-2026-6357, CVE-2026-8643 — fixed in 26.1.2). Python 3.12 alone is not enough:
its bundled pip is 25.0.1.
datamodel-code-generator+blackuninstalled after codegen in the two CIimages. Neither is imported at runtime; they exist only to run
datamodel_generation.py, and account for 8 HIGH findings.Type of change:
High-level design:
Why the airflow images need
/usr/local/bin/python -m piprather than plainpip:three independent reasons, all verified in the base image —
pipresolves to airflow's/root/bin/pigwrapper, whichexit 1s (so aplain
pipline fails the build, it doesn't silently no-op);execs${HOME}/.local/bin/pipeven for non-root, so it can neverreach the interpreter-level install;
pythonas root resolves to/home/airflow/.local/bin/python, sopython -m pipalso targets~/.local.Only
/usr/local/bin/python(→/usr/python/bin/python3) reaches the copy scannersreport. The slim images are ordinary
python:3.12-slim-bookwormwith no wrapper, soplain
pipis correct there.Alternative rejected — bumping
datamodel-code-generatorinstead of removing it.0.64.0+ is CVE-free, but
scripts/datamodel_generation.pymonkeypatchesdatamodel_code_generator.model.pydantic.types.IMPORT_SECRET_STR(theCustomSecretStrswap applied to every generated password field), and that modulepath no longer exists in 0.26+ — probed on 0.25.6 / 0.64.0 / 0.71.0. The script also
post-processes generated output with exact string replacements that would fail
silently on a formatting change. A pin bump therefore needs a regenerate-and-diff of
the full
src/metadata/generated/tree; tracked separately. The uninstall runs afterthe final
pip installso no extra can pull the packages back in.Alternative rejected — lifting
setuptools<81. setuptools ≥83 fixesCVE-2026-59890 but no longer ships
pkg_resources, andsqlalchemy_redshift/__init__.py:1imports it unguarded at module top level, withsqlalchemy-redshift==0.8.14(2023) being the latest release. Left as a documentedresidual.
Rollout / compatibility.
cx_Oracle 8.3.0andmysqlclient==2.1.1have no cp312wheel but both build cleanly from sdist on 3.12 (the toolchain and
pkg-configarealready installed) — verified in a container. Bumping
mysqlclientto 2.2.x for aprebuilt wheel is a sensible follow-up but is not required here.
Tests:
Use cases covered
openmetadata-managed-apisunder the matching 3.12constraints file.
datamodel-code-generator/black, and theirinterpreter-level pip is ≥ 26.1.2.
Unit tests
Backend integration tests
Ingestion integration tests
Playwright (UI) tests
Manual testing performed
docker buildx build --checkclean on all four Dockerfiles.pip install --upgrade pipas root inapache/airflow:3.2.2-python3.12exits 1, and that
/usr/local/bin/python -m pip install "pip==26.2"exits 0 andlands in
/usr/python/lib/python3.12/site-packages.openmetadata-airflow-apisunder the regenerated constraints file in the3.12 airflow base — succeeds, setuptools stays at the pinned 80.10.2, and
pkg_resourcesstill imports.cx_Oracle 8.3.0andmysqlclient==2.1.1build from sdist onpython:3.12-slim-bookworm.pip uninstall -y datamodel-code-generator blackexits 0 when thepackages are absent, so the layer is a safe no-op.