fix(ingestion): resolve Trivy image CVEs (airflow 3.2.2, Spark hive jars, perl/nss) - #30516
fix(ingestion): resolve Trivy image CVEs (airflow 3.2.2, Spark hive jars, perl/nss)#30516harshsoni2024 wants to merge 7 commits into
Conversation
…ars, perl/nss) Clears HIGH/CRITICAL/MEDIUM findings from the Trivy image scans across three layers of the ingestion image. Apache Airflow 3.2.1 -> 3.2.2 (CVE-2026-42252, doc-only BashOperator fix; no runtime change): setup.py pin, both Dockerfile FROM lines, constraints URL/file, and regenerated airflow-constraints-3.2.2.txt. Patch Hive-metastore jars bundled in PySpark 3.5.6 (Deltalake calls .enableHiveSupport(), so they are live). After pip install, swap them in pyspark/jars/, SHA1-pinned to Maven Central: - derby 10.14.2.0 -> 10.17.1.0 (CVE-2022-46337; Java 17 in this image) - zookeeper 3.6.3 -> 3.7.2 (CVE-2023-44981) - jackson-mapper-asl / jackson-core-asl 1.9.13 removed (CVE-2019-10202, no patched release exists upstream) Add perl/nss (perl perl-base perl-modules-5.36 libperl5.36 libnss3) to the apt-get --only-upgrade block in operators/docker/Dockerfile.ci (CVE-2026-13221/-57433/-12087/-16389); the other ingestion Dockerfiles already carried this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
❌ PR checklist incompleteThis PR cannot be merged until the following are addressed on its linked issue:
The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically. Maintainers can bypass this check by adding the |
github-actions
Bot
added
Ingestion
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
This PR updates the ingestion build artifacts to address Trivy-reported CVEs by bumping the Apache Airflow base/pin to 3.2.2, patching vulnerable Hive-metastore jars bundled with PySpark in the ingestion images, and extending OS security upgrades in the operators CI image.
Changes:
- Bump Airflow from
3.2.1to3.2.2across ingestion packaging and Docker images, including the pinned constraints file. - Patch PySpark-bundled Hive-metastore jars in ingestion images by replacing Derby/ZooKeeper jars and removing legacy jackson-asl jars.
- Extend
apt-get --only-upgradesecurity upgrades in the operators CI Dockerfile to includeperl*andlibnss3.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
ingestion/setup.py |
Updates the shared Airflow version pin to 3.2.2. |
ingestion/operators/docker/Dockerfile.ci |
Extends the OS security upgrade block to cover additional CVE-impacted packages. |
ingestion/Dockerfile.ci |
Bumps Airflow base image/constraints usage and patches PySpark jars during image build. |
ingestion/Dockerfile |
Bumps Airflow base image/constraints URL and patches PySpark jars during image build. |
ingestion/airflow-constraints-3.2.2.txt |
Adds the regenerated Airflow 3.2.2 constraints snapshot used for constrained installs. |
Comments suppressed due to low confidence (4)
ingestion/airflow-constraints-3.2.2.txt:160
- PR description says
apache-airflow-providers-openaiwas removed because OpenMetadata doesn’t use it and it pulls in extra transitive deps/CVEs, but this regenerated constraints file includes it again. Either update the PR description or re-apply the omission to keep the constraints aligned with the stated CVE-reduction approach.
This issue also appears in the following locations of the same file:
- line 446
- line 490
- line 659
ingestion/airflow-constraints-3.2.2.txt:446
- PR description calls out removing the OpenAI provider chain (litellm/openai/tiktoken). This constraints regeneration reintroduces
litellm, which makes the file diverge from the described change set. Either update the PR description or omit this pin again to match the intended CVE surface reduction.
ingestion/airflow-constraints-3.2.2.txt:490 - PR description calls out removing the OpenAI provider chain (litellm/openai/tiktoken). This constraints regeneration reintroduces
openai, which makes the file diverge from the described change set. Either update the PR description or omit this pin again to match the intended CVE surface reduction.
ingestion/airflow-constraints-3.2.2.txt:659 - PR description calls out removing the OpenAI provider chain (litellm/openai/tiktoken). This constraints regeneration reintroduces
tiktoken, which makes the file diverge from the described change set. Either update the PR description or omit this pin again to match the intended CVE surface reduction.
✅ Playwright Results — workflow succeededValidated commit ✅ 107 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 3 skipped · 🧰 0 lifecycle flaky PerformanceBlocking targets: ✅ met · Optimization targets: 🟡 in progress Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting. 🕒 Full workflow signal wall (to summary) 38m 44s ⏱️ Max setup 3m 38s · max shard execution 11m 36s · max shard-job elapsed before upload 17m 43s · reporting 3s 🌐 208.62 requests/attempt · 1.76 app boots/UI scenario · 0.00% common-shard skew Optimization targets still in progress:
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
Follow-up to the CVE-batch commit, addressing code-review findings on the PySpark jar patch. - Derby: revert the 10.14.2.0 -> 10.17.1.0 swap. 10.17.1.0 requires Java 21 but the image ships Java 17 (default-jre-headless), so loading it under Hive's embedded metastore would throw UnsupportedClassVersionError. No Java-17 backport (10.14.3.0 / 10.15.2.1 / 10.16.1.2) is published to Maven Central, so there is no drop-in fix. Left at the bundled 10.14.2.0 and documented: the CVE is an LDAP-authenticator flaw and OM's embedded Derby metastore uses no LDAP auth, so the path is unreachable. - Checksums: append `|| exit 1` to the wget && sha1sum chain so a failed download or SHA1 mismatch on any jar aborts the build, instead of only the last loop iteration's status mattering. zookeeper 3.7.2 swap and jackson-asl removal are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GitHub Actions
Failure
GitHub Actions
Inactive
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (4)
ingestion/setup.py:22
- The new Airflow pin comment claims a "BashOperator Jinja2 injection" runtime issue, but the PR description notes CVE-2026-42252 is documentation-only. This wording is likely misleading for future maintainers triaging security impact; consider softening it to avoid implying an exploitable runtime vulnerability.
"airflow": "apache-airflow==3.2.2", # CVE-2026-42252 BashOperator Jinja2 injection
ingestion/airflow-constraints-3.2.2.txt:272
- This constraints file pins cryptography to 48.0.0, which is below the ingestion package’s security floor (ingestion/setup.py requires cryptography>=48.0.1 due to a bundled OpenSSL vulnerability in older wheels). Using this constraint during image builds can downgrade cryptography and reintroduce known CVEs; please bump the pin or regenerate constraints to keep cryptography at >=48.0.1.
This issue also appears in the following locations of the same file:
- line 466
- line 664
ingestion/airflow-constraints-3.2.2.txt:466
- This constraints file pins mistune to 3.2.1, but the ingestion package’s base requirements explicitly set mistune>=3.3.0 for CVE-2026-49851 (ReDoS in parse_link_text). Keeping mistune below that floor can reintroduce the vulnerability in the Airflow image; please bump the pin/regenerate constraints to at least 3.3.0.
ingestion/airflow-constraints-3.2.2.txt:664 - This constraints file pins tornado to 6.5.5, but ingestion/setup.py’s Airflow extra includes a security floor of tornado>=6.5.7 (multiple CVEs listed there). Pinning below that level can reintroduce those vulnerabilities in the Airflow image; please bump the pin/regenerate constraints to at least 6.5.7.
GitHub Actions
Inactive
GitHub Actions
Failure
GitHub Actions
Inactive
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (3)
ingestion/setup.py:22
- The inline comment on the Airflow pin claims a "BashOperator Jinja2 injection" runtime vulnerability, but the PR description notes CVE-2026-42252 is a documentation-only fix with no runtime/API change. To avoid misleading future maintainers, make the comment match the actual impact (or just reference the CVE ID).
"airflow": "apache-airflow==3.2.2", # CVE-2026-42252 BashOperator Jinja2 injection
ingestion/Dockerfile.ci:168
- The Maven Central jar integrity check uses SHA-1 (via
sha1sum). Elsewhere in these Dockerfiles (e.g., the IBM iAccess .deb and the ANTLR jar) checksums are pinned with SHA-256. Consider switching these jar pins to SHA-256/SHA-512 to avoid relying on a deprecated collision-prone hash for supply-chain verification.
name="${entry%%|*}"; rest="${entry#*|}"; path="${rest%%|*}"; sha="${rest##*|}"; \
wget -q "https://repo1.maven.org/maven2/${path}" -O "${name}" \
&& echo "${sha} ${name}" | sha1sum -c - || exit 1; \
done
ingestion/Dockerfile:131
- The Maven Central jar integrity check uses SHA-1 (via
sha1sum). Elsewhere in this Dockerfile (e.g., the IBM iAccess .deb) checksums are pinned with SHA-256. Consider switching these jar pins to SHA-256/SHA-512 to avoid relying on a deprecated collision-prone hash for supply-chain verification.
"zookeeper-3.7.2.jar|org/apache/zookeeper/zookeeper/3.7.2/zookeeper-3.7.2.jar|3b7c2c26b697094fc29e5a78a522cffa1a55b26d" \
"zookeeper-jute-3.7.2.jar|org/apache/zookeeper/zookeeper-jute/3.7.2/zookeeper-jute-3.7.2.jar|1ca84725b0e581c8fef7c54e5355f3b8f7e44410" \
; do \
name="${entry%%|*}"; rest="${entry#*|}"; path="${rest%%|*}"; sha="${rest##*|}"; \
wget -q "https://repo1.maven.org/maven2/${path}" -O "${name}" \
&& echo "${sha} ${name}" | sha1sum -c - || exit 1; \
done
The Hive-metastore jar patch ran `import pyspark` unconditionally, so a build that overrides INGESTION_DEPENDENCY to an extra without the deltalake/pyspark deps would fail the RUN at the import. Wrap the step in a pyspark-presence check: skip cleanly (exit 0) with a log line when the pyspark jars dir is not found, and only patch when it exists. Download/checksum failures within the patch path still abort the build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (2)
ingestion/setup.py:22
- The inline comment for the Airflow 3.2.2 pin states this CVE is a “BashOperator Jinja2 injection”, but the PR description indicates CVE-2026-42252 is a documentation-only hardening with no runtime/API change. This comment is likely misleading for future maintainers triaging security bumps.
"airflow": "apache-airflow==3.2.2", # CVE-2026-42252 BashOperator Jinja2 injection
ingestion/operators/docker/Dockerfile.ci:55
- This comment says the wget/perl*/libnss3 upgrades are a no-op “until Debian ships it”, but the Dockerfile is now explicitly attempting to upgrade these packages as part of the security remediation. Consider rewording so it’s clear the intent is to pick up security updates when they’re available (while still noting that --only-upgrade is a no-op if the package/fix isn’t present).
# wget, perl*, libnss3 -> tracked; upstream fix pending, no-op until Debian ships it).
SHA1 is collision-prone and inconsistent with the SHA256-verified downloads elsewhere in these Dockerfiles (e.g. the IBM iAccess driver). Switch the zookeeper jar pins to SHA256 digests (computed from the same artifacts, verified against the previously pinned SHA1) and use sha256sum -c - for verification. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
ingestion/setup.py:22
- The inline CVE note here asserts a “BashOperator Jinja2 injection”, but the PR description indicates CVE-2026-42252 is a documentation-only fix with no runtime/API behavior change. Consider rewording the comment to avoid overstating the vulnerability mechanism.
"airflow": "apache-airflow==3.2.2", # CVE-2026-42252 BashOperator Jinja2 injection
ingestion/operators/docker/Dockerfile.ci:55
- This comment says perl*/libnss3 upgrades are a “no-op until Debian ships it”, but the next line explicitly adds perl*/libnss3 to the --only-upgrade list as part of the security-fix block. Reword to reflect that these packages are intentionally included so Debian security updates are picked up when available.
# wget, perl*, libnss3 -> tracked; upstream fix pending, no-op until Debian ships it).
…w test CSV
Replace the pipe-delimited `for entry` loop in the PySpark Hive-metastore
jar patch with a positional-arg `fetch_jar()` shell function in both
ingestion/Dockerfile and Dockerfile.ci. The old `${entry%%|*}` parameter
expansion crashed Snyk's Dockerfile parser (regex "Nothing to repeat"),
causing ingestion-docker-scan to return a false 0 instead of scanning.
Jars, Maven paths, SHA256 pins, checksum fail-fast, and the pyspark-presence
guard are all unchanged.
Vendor winequality-red.csv into the mlflow integration test data and
read it from disk instead of fetching it from the openmetadata-demo GitHub
URL, which now 404s and fails test_mlflow at setup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GitHub Actions
Inactive
GitHub Actions
Failure
|
The Python checkstyle failed. Please run You can install the pre-commit hooks with |
Code Review ✅ Approved 4 resolved / 4 findingsResolves ingestion Trivy image CVEs by upgrading to Airflow 3.2.2, patching PySpark Hive jars, and updating OS security packages, addressing checksum failures, Java compatibility constraints, and silent jar patch skipping findings. ✅ 4 resolved✅ Bug: Checksum failures ignored for non-final jars in for-loop
✅ Bug: Derby 10.17.1.0 requires Java 21 but image ships Java 17
✅ Security: Jar CVE patch can be silently skipped in pyspark-bearing images
✅ Edge Case: Removing jackson-asl jars may break Hive at runtime
OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
ingestion/Dockerfile:134
- The jar patch step deletes jackson-*-asl jars (CVE-2019-10202) but doesn’t validate that Spark can still start with Hive support. Since the Deltalake connector always calls
SparkSession.builder.enableHiveSupport()(seeingestion/src/metadata/ingestion/source/database/deltalake/clients/pyspark.py), a missing transitive dependency will only be discovered at runtime. Add a small build-time smoke test after patching (only whenpysparkis present) that instantiates a local SparkSession withenableHiveSupport()so the image build fails fast if the classpath is broken.
cd "${JARS_DIR}" \
&& rm -f zookeeper-*.jar zookeeper-jute-*.jar \
jackson-mapper-asl-*.jar jackson-core-asl-*.jar \
&& fetch_jar zookeeper-3.7.2.jar \
org/apache/zookeeper/zookeeper/3.7.2/zookeeper-3.7.2.jar \
ingestion/Dockerfile.ci:171
- Same concern as the production ingestion image: this build removes jackson-*-asl jars without verifying that Spark can still initialize Hive support. Because
enableHiveSupport()is exercised in the Deltalake connector, consider failing fast during the image build by running a lightweight local SparkSession + Hive initialization after the jar swap (only whenpysparkis present).
cd "${JARS_DIR}" \
&& rm -f zookeeper-*.jar zookeeper-jute-*.jar \
jackson-mapper-asl-*.jar jackson-core-asl-*.jar \
&& fetch_jar zookeeper-3.7.2.jar \
org/apache/zookeeper/zookeeper/3.7.2/zookeeper-3.7.2.jar \
Changes
Apache Airflow
3.2.1 → 3.2.2— patch bump (CVE-2026-42252, a documentation-onlyBashOperator fix; no runtime/API change). Updated the
setup.pypin, both ingestionDockerfile
FROMlines, the pinned Airflow constraints URL/file, and regeneratedairflow-constraints-3.2.2.txtfrom upstream (git tracks it as a rename of the 3.2.1 file).Patch vulnerable Hive-metastore jars bundled inside PySpark 3.5.6 — the Deltalake
connector calls
.enableHiveSupport(), so these jars are live, not dormant. Afterpip install, the Dockerfile swaps them inpyspark/jars/(all SHA1-pinned to Maven Central):zookeeper 3.6.3 → 3.7.2— CVE-2023-44981 (SASL quorum auth bypass)jackson-mapper-asl/jackson-core-asl 1.9.13— CVE-2019-10202; removed (legacy Codehaus lib has no patched release upstream)perl / nss OS upgrades — added
perl perl-base perl-modules-5.36 libperl5.36 libnss3to the existing
apt-get --only-upgradesecurity block inoperators/docker/Dockerfile.ci(CVE-2026-13221/-57433/-12087/-16389). The other three ingestion Dockerfiles already carried this.
CVE coverage
apt-get --only-upgradeapt-get --only-upgradeAlready fixed at source on
main, cleared by rebuild (no change here): CVE-2026-2332(jetty
12.1.7), CVE-2026-42581 / -42584 (netty-bom4.1.135).black(CVE-2026-31900)does not apply — the dev extra uses
ruff.golang.org/x/net(CVE-2026-39821) is abase-image Go binary refreshed by the 3.2.2 bump; a
.trivyignoreis the fallback if it survives.Greptile Summary
This revision updates the ingestion images and package metadata for Airflow 3.2.2 while applying additional container security remediations.
Confidence Score: 3/5
The PR is not yet safe to merge because Deltalake’s Hive initialization can still fail after the image removes required Codehaus Jackson classes.
The Derby replacement that required Java 21 has been removed, but both primary ingestion images still delete the Jackson ASL jars without providing compatible classes, while the reachable Deltalake metastore path initializes Spark with Hive support.
Files Needing Attention: ingestion/Dockerfile and ingestion/Dockerfile.ci
Important Files Changed
Reviews (7): Last reviewed commit: "fix: ruff path fix" | Re-trigger Greptile