chore(agent-data-plane): add AIX build helper by thieman · Pull Request #2084 · DataDog/saluki

thieman · 2026-07-10T20:11:38Z

Human Summary

Adds a script and Makefile target to build a release ADP binary on AIX. This comes with two intentional divergences from how we build on the other platforms:

We can't use cargo auditable because it does not support XCOFF. Clanker tried to get it to work with some manual patching but wasn't able to. Seems like this will be a more significant contribution to upstream if we want this to work.
We can't use link-time optimization (LTO) because it doesn't seem to work on AIX at all, even for a hello-world crate. This is actually called out in IBM's docs here.

Neither of these should be showstoppers, but please comment if they are. The plan will be to call this script from within the Agent's packaging/aix machinery when we do AIX builds. Working on that follow-up now.

Summary

Adds make build-adp-aix for native AIX ADP builds using the IBM Rust SDK and AIX Toolbox GCC toolchain.
Adds ci/tooling/build-adp-aix.sh to record the AIX build environment and run the native Cargo build.
Adds an aix-optimized-release Cargo profile that keeps the optimized-release binary settings that work on AIX while disabling LTO.
Checks the major toolchain versions for cargo/rustc/gcc/g++ so AIX builds fail loudly if the toolchain drifts unexpectedly.

Notes

The AIX enablement and AWS-LC dependency changes from fix(process-memory): support RSS queries on AIX #2031, fix(stringtheory): support big-endian platforms for MetaString #2032, fix(datadog-agent-commons): add AIX platform settings #2034, fix(agent-data-plane): preserve DNS for hostname IPC endpoints #2060, and chore(deps): bump aws-lc #2082 are now on main; this PR is rebased on top of them and only carries the build helper/profile changes.
aws-lc-sys is built from the bumped dependency without patching Cargo's registry cache.
On the AIX test host, gcc-8 fails in an AWS-LC compiler feature probe; /opt/freeware/bin/gcc (gcc 13.3) succeeds, so the helper defaults to that compiler.
AIX currently does not use cargo auditable: cargo-auditable emits ELF/generic Unix linker inputs and flags that AIX's XCOFF linker rejects.
AIX currently uses aix-optimized-release instead of optimized-release: the IBM Rust SDK 1.92 fails LTO even for a tiny hello-world crate with failed to get bitcode from object file for LTO (Can't find section .ipa).
The helper still uses the same metadata inputs that affect the ADP binary (APP_FULL_NAME, APP_SHORT_NAME, APP_IDENTIFIER, APP_VERSION, APP_GIT_HASH, APP_BUILD_TIME, APP_DEV_BUILD) and passes BUILD_FEATURES through to Cargo.
Toolchain version prefixes are overrideable through ADP_AIX_EXPECTED_*_PREFIX variables when an intentional toolchain update occurs.

Test Plan

bash -n ci/tooling/build-adp-aix.sh
ADP_AIX_BUILD_DRY_RUN=true make build-adp-aix
cargo metadata --no-deps --format-version 1
AIX: CARGO_HOME=/opt/cargo-home-aix-adp-pr-stack CARGO_TARGET_DIR=/opt/saluki-aix-adp-pr-stack-target make build-adp-aix
AIX: /opt/saluki-aix-adp-pr-stack-target/aix-optimized-release/agent-data-plane --help
AIX: /opt/saluki-aix-adp-pr-stack-target/aix-optimized-release/agent-data-plane version
Pre-commit hook: fmt, check-clippy, check-licenses, check-deny, check-docs, generate-api-docs

AIX toolchain observed:

cargo 1.92.0 (IBM Open SDK for Rust on AIX 1.92.0.0)
rustc 1.92.0 (IBM Open SDK for Rust on AIX 1.92.0.0)
gcc (GCC) 13.3.0
g++ (GCC) 13.3.0

AIX build result:

Finished aix-optimized-release profile [optimized + debuginfo] target(s) in 26m 22s
Binary: 64-bit XCOFF executable or object module not stripped

pr-commenter · 2026-07-10T20:20:27Z

Binary Size Analysis (Agent Data Plane)

Baseline: 33d6ac6 · Comparison: 96e35a2 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.58 MiB (baseline) vs 41.58 MiB (comparison)
Size Change: +0 B (+0.00%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
`anon.104b1f618a14d6d60f2ebe1b651d81f5.1.llvm.7820091515578204338`	+129 B	1
`anon.104b1f618a14d6d60f2ebe1b651d81f5.1.llvm.9054900619091518955`	-129 B	1
`anon.104b1f618a14d6d60f2ebe1b651d81f5.4.llvm.7820091515578204338`	+114 B	1
`anon.104b1f618a14d6d60f2ebe1b651d81f5.4.llvm.9054900619091518955`	-114 B	1
`anon.104b1f618a14d6d60f2ebe1b651d81f5.3.llvm.7820091515578204338`	+108 B	1
`anon.104b1f618a14d6d60f2ebe1b651d81f5.3.llvm.9054900619091518955`	-108 B	1
`anon.104b1f618a14d6d60f2ebe1b651d81f5.0.llvm.7820091515578204338`	+96 B	1
`anon.104b1f618a14d6d60f2ebe1b651d81f5.0.llvm.9054900619091518955`	-96 B	1
`anon.104b1f618a14d6d60f2ebe1b651d81f5.2.llvm.7820091515578204338`	+94 B	1
`anon.104b1f618a14d6d60f2ebe1b651d81f5.2.llvm.9054900619091518955`	-94 B	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW]    +129  [NEW]     +40    anon.104b1f618a14d6d60f2ebe1b651d81f5.1.llvm.7820091515578204338
  [NEW]    +114  [NEW]     +25    anon.104b1f618a14d6d60f2ebe1b651d81f5.4.llvm.7820091515578204338
  [NEW]    +108  [NEW]     +19    anon.104b1f618a14d6d60f2ebe1b651d81f5.3.llvm.7820091515578204338
  [NEW]     +96  [NEW]      +7    anon.104b1f618a14d6d60f2ebe1b651d81f5.0.llvm.7820091515578204338
  [NEW]     +94  [NEW]      +5    anon.104b1f618a14d6d60f2ebe1b651d81f5.2.llvm.7820091515578204338
  [DEL]     -94  [DEL]      -5    anon.104b1f618a14d6d60f2ebe1b651d81f5.2.llvm.9054900619091518955
  [DEL]     -96  [DEL]      -7    anon.104b1f618a14d6d60f2ebe1b651d81f5.0.llvm.9054900619091518955
  [DEL]    -108  [DEL]     -19    anon.104b1f618a14d6d60f2ebe1b651d81f5.3.llvm.9054900619091518955
  [DEL]    -114  [DEL]     -25    anon.104b1f618a14d6d60f2ebe1b651d81f5.4.llvm.9054900619091518955
  [DEL]    -129  [DEL]     -40    anon.104b1f618a14d6d60f2ebe1b651d81f5.1.llvm.9054900619091518955
  [ = ]       0  [ = ]       0    TOTAL

Add a Makefile target and helper script for native AIX ADP builds. The helper records the IBM Rust SDK, AIX Toolbox GCC, system ar/ranlib, Cargo cache, and target directory settings needed on the AIX builder, with a dry-run test target for local validation.

pr-commenter · 2026-07-10T20:51:35Z

Regression Detector (Agent Data Plane)

Run ID: 490baf29-5a54-43a4-b928-df922211badd
Baseline: 33d6ac63 · Comparison: 96e35a23 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_idle	memory	⚪ +0.38	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ +0.03	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ -0.12	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ -0.12	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ -0.27	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 131 MiB ≤ 140 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 43.3 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 65 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 192 MiB ≤ 200 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 28.9 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

webern

Nice trailblazing.

webern · 2026-07-13T08:55:18Z

+if [[ "${ADP_AIX_BUILD_DRY_RUN}" == "true" ]]; then
+    print_environment
+    exit 0
+fi


This is definitely a nit, but I think a --dry-run arg is a little bit more canonical and visible, i.e. in the Makefile.

Since this is mainly intended for use via make the CLI flag ends up being less ergonomic than env vars since it'd require a separate Make target or a way to forward around arguments. I don't expect anyone will be running this directly, should be via Make.

Add lightweight version prefix checks for the AIX ADP build helper's Rust and GCC toolchain. The defaults pin the known-good IBM Rust SDK 1.92 and AIX Toolbox GCC 13 paths while allowing intentional toolchain updates through environment overrides.

Add an AIX-specific optimized release profile that keeps release optimization settings while disabling LTO, which is not currently supported by the IBM Rust SDK on AIX. Keep the AIX build helper focused on producing the release binary with plain cargo and the same embedded metadata inputs as other release builds.

Explain why the AIX optimized release profile disables LTO while the IBM Rust SDK does not support the existing release LTO settings on AIX.

thieman · 2026-07-13T18:11:39Z

@@ -0,0 +1,102 @@
+#!/usr/bin/env bash


This script doesn't seem like it's actually giving us much, can you remove it and its reference in the Makefile?

[GPT-5.6 Sol] Fixed in a3156d3. Removed ci/tooling/test-build-adp-aix.sh and the test-build-adp-aix-script Makefile target; the dry-run path remains available manually with ADP_AIX_BUILD_DRY_RUN=true make build-adp-aix.

thieman · 2026-07-13T18:12:35Z

+cargo fetch
+
+echo "[*] Building agent-data-plane for AIX..."
+cargo build --profile "${BUILD_PROFILE}" --bin agent-data-plane --features "${BUILD_FEATURES}"


Add an explanatory comment above this line explaining why we aren't using cargo auditable the way the other builds are (doesn't support XCOFF)

[GPT-5.6 Sol] Fixed in a3156d3. Added the XCOFF/cargo-auditable comment above the AIX cargo build invocation to explain why this path uses plain cargo build.

thieman · 2026-07-13T18:12:52Z

+inherits = "release"
+debug-assertions = false
+overflow-checks = false
+# IBM Open SDK for Rust 1.92 on AIX fails LTO during final linking with `Can't find section .ipa`.


Add a bit more information in this comment based on our investigation into these build failures

[GPT-5.6 Sol] Fixed in a3156d3. Expanded the LTO comment to include the IBM Rust SDK 1.92 failure mode and the minimal hello-world repro result (Can't find section .ipa).

Remove the dry-run helper test target and document the AIX-specific reasons for disabling cargo-auditable and LTO in the native AIX build path.

thieman · 2026-07-13T18:25:00Z

 codegen-units = 1

+[profile.aix-optimized-release]
+inherits = "release"


Make this inherit from optimized-release and then just keep the LTO override to false, if that works

[GPT-5.6 Sol] Updated in 96e35a2. The AIX profile now inherits from optimized-release and only overrides lto = false, which keeps the shared optimized-release settings centralized while preserving the AIX LTO workaround. I validated the profile parses with cargo metadata --no-deps --format-version 1, and the pre-commit checks passed.

Make the AIX-specific release profile inherit from the shared optimized-release profile and override only LTO. This keeps the optimized release settings centralized while preserving the AIX workaround for IBM Rust SDK LTO failures.

dd-octo-sts Bot added area/io General I/O and networking. area/memory Memory bounds and memory management. area/ci CI/CD, automated testing, etc. labels Jul 10, 2026

thieman force-pushed the thieman/aix-adp-pr-stack branch from 3d1c153 to 1f9d343 Compare July 10, 2026 20:27

dd-octo-sts Bot removed area/io General I/O and networking. area/memory Memory bounds and memory management. labels Jul 10, 2026

webern approved these changes Jul 13, 2026

View reviewed changes

thieman added 3 commits July 13, 2026 09:56

chore(agent-data-plane): document AIX LTO setting

d99f89d

Explain why the AIX optimized release profile disables LTO while the IBM Rust SDK does not support the existing release LTO settings on AIX.

thieman commented Jul 13, 2026

View reviewed changes

chore(agent-data-plane): address AIX build helper review

a3156d3

Remove the dry-run helper test target and document the AIX-specific reasons for disabling cargo-auditable and LTO in the native AIX build path.

thieman marked this pull request as ready for review July 13, 2026 18:21

thieman requested a review from a team as a code owner July 13, 2026 18:21

thieman commented Jul 13, 2026

View reviewed changes

chore(agent-data-plane): simplify AIX release profile

96e35a2

Make the AIX-specific release profile inherit from the shared optimized-release profile and override only LTO. This keeps the optimized release settings centralized while preserving the AIX workaround for IBM Rust SDK LTO failures.

thieman mentioned this pull request Jul 13, 2026

Add Agent Data Plane support to AIX packaging DataDog/datadog-agent#53596

Merged

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: queued labels Jul 13, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit b1ad2b6 into main Jul 13, 2026
87 checks passed

gh-worker-dd-devflow-36fce6 Bot removed the mergequeue-status: in_progress label Jul 13, 2026

gh-worker-dd-mergequeue-cf854d Bot deleted the thieman/aix-adp-pr-stack branch July 13, 2026 19:05

gh-worker-dd-devflow-36fce6 Bot added the mergequeue-status: done label Jul 13, 2026

Uh oh!

Conversation

thieman commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Human Summary

Summary

Notes

Test Plan

Uh oh!

pr-commenter Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Binary Size Analysis (Agent Data Plane)

✅ Binary size difference within threshold

Uh oh!

pr-commenter Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector (Agent Data Plane)

Optimization Goals: ✅ No significant changes detected

Uh oh!

webern left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thieman Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

thieman commented Jul 10, 2026 •

edited

Loading

pr-commenter Bot commented Jul 10, 2026 •

edited

Loading

pr-commenter Bot commented Jul 10, 2026 •

edited

Loading

thieman Jul 13, 2026 •

edited

Loading