Skip to content

Fix AotHloIdenticalTest false-positive failure on JAX nightly due to new source mapping format#3947

Merged
copybara-service[bot] merged 1 commit into
mainfrom
darisoy-align-jax-nightly
May 19, 2026
Merged

Fix AotHloIdenticalTest false-positive failure on JAX nightly due to new source mapping format#3947
copybara-service[bot] merged 1 commit into
mainfrom
darisoy-align-jax-nightly

Conversation

@darisoy
Copy link
Copy Markdown
Collaborator

@darisoy darisoy commented May 19, 2026

Description

This PR updates the Ahead-Of-Time (AOT) identical graph verification tests to ignore new XLA debug source location metadata. This fixes false-positive integration test failures occurring under newer JAX/XLA nightly environments.

The Problem Being Solved:

JAX/XLA nightly recently introduced a new debug source mapping metadata format in the compiled HLO headers (file_location_id and parent_frame_id). Because Ahead-Of-Time compilation (train_compile.py) and normal JIT compilation (train.py) have different call stacks, these generated frame IDs differ.

The test's sanitation regex location_map_pattern did not filter out file_location_id, triggering a false-positive AotHloIdenticalTest::test_default_hlo_match failure even though the actual compiled hardware instructions are 100% identical.

The Solution:

Updated the location_map_pattern regex in tests/integration/aot_identical_test.py to match and strip both file_name_id and the new file_location_id metadata blocks before comparing AOT and Real hashes:

location_map_pattern = re.compile(r"^\s*\d+\s+\{(file_name_id|file_location_id)=[^}]*\}\s*$")

FIXES: b/514475466
FIXES: #3941


Tests

Build Images workflow no longer fails with AOT identical graph test.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 19, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@copybara-service copybara-service Bot merged commit 3d1bf85 into main May 19, 2026
101 of 107 checks passed
@copybara-service copybara-service Bot deleted the darisoy-align-jax-nightly branch May 19, 2026 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MaxText Docker Image Build Failure

3 participants