Skip to content

Increase upgrade test tester shutdown timeout from 5s to 30s#13288

Merged
saintstack merged 1 commit into
apple:mainfrom
saintstack:asan_timeout
May 27, 2026
Merged

Increase upgrade test tester shutdown timeout from 5s to 30s#13288
saintstack merged 1 commit into
apple:mainfrom
saintstack:asan_timeout

Conversation

@saintstack
Copy link
Copy Markdown
Contributor

@saintstack saintstack commented May 27, 2026

Under ASAN, TaskQueue::clear() fires broken_promise to actors across multiple external client network threads. The cascading cleanup takes longer than the previous 5-second timeout, causing the test harness to SIGKILL the tester and report failure.

See in last nights' nightly. All tests pass but the cleanup on the end is timing out.

Increase upgrade test tester shutdown timeout from 5s to 30s
57d9262 
Under ASAN, TaskQueue::clear() fires broken_promise to actors across
multiple external client network threads. The cascading cleanup takes
longer than the previous 5-second timeout, causing the test harness to
SIGKILL the tester and report failure.
@saintstack saintstack added the nightlies Issues to address failures in the nighty runs. label May 27, 2026
@saintstack saintstack requested review from jbarkley-apple and ploxiln and removed request for jbarkley-apple May 27, 2026 18:26
@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: 57d9262
  • Duration 0:24:53
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: 57d9262
  • Duration 0:33:37
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: 57d9262
  • Duration 0:45:37
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: 57d9262
  • Duration 0:50:35
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 57d9262
  • Duration 0:50:44
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 57d9262
  • Duration 0:59:49
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 57d9262
  • Duration 1:03:40
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@saintstack
Copy link
Copy Markdown
Contributor Author

saintstack commented May 27, 2026

RandomSeed="229801209" SourceVersion="57d9262a25ff5be062b2d04d1d315b78539e1ffc" Time="1779908484" BuggifyEnabled="1" DeterminismCheck="0" FaultInjectionEnabled="1" TestFile="tests/fast/MoveKeysCycle.toml"

@saintstack saintstack closed this May 27, 2026
@saintstack saintstack reopened this May 27, 2026
@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: 57d9262
  • Duration 0:23:56
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: 57d9262
  • Duration 0:34:29
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: 57d9262
  • Duration 0:45:37
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: 57d9262
  • Duration 0:47:16
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 57d9262
  • Duration 0:53:56
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 57d9262
  • Duration 1:01:59
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 57d9262
  • Duration 1:02:50
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@saintstack
Copy link
Copy Markdown
Contributor Author

saintstack commented May 27, 2026

RandomSeed="4008309185" SourceVersion="57d9262a25ff5be062b2d04d1d315b78539e1ffc" Time="1779914942" BuggifyEnabled="0" DeterminismCheck="0" FaultInjectionEnabled="1" TestFile="tests/slow/SwizzledCycleTest.toml"

@saintstack saintstack closed this May 27, 2026
@saintstack saintstack reopened this May 27, 2026
@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: 57d9262
  • Duration 0:22:18
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: 57d9262
  • Duration 0:34:08
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: 57d9262
  • Duration 0:44:52
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 57d9262
  • Duration 0:51:55
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 57d9262
  • Duration 0:58:19
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 57d9262
  • Duration 1:02:19
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@saintstack saintstack merged commit e5276f5 into apple:main May 27, 2026
9 of 10 checks passed
@foundationdb-ci
Copy link
Copy Markdown
Contributor

foundationdb-ci commented May 27, 2026

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: 57d9262
  • Duration 1:19:03
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@saintstack saintstack mentioned this pull request May 28, 2026
Merged
saintstack pushed a commit to saintstack/foundationdb that referenced this pull request May 28, 2026
Add LSAN suppression for ThreadSafeTransaction indirect leaks
674b0c1 
The existing leak:ThreadSafeDatabase::createTransaction (added in apple#13278)
only catches the direct ThreadSafeTransaction allocation (2x2048 bytes).
Indirect leaks from transaction operations (TransactionState,
extractReadVersion, WriteMap entries) are attributed to their own allocation
sites, which have ThreadSafeTransaction methods in the call stack but don't
match the createTransaction suppression.

This causes the ASAN nightly to report 10,296 bytes in 48 allocations across
multiple fdb_c_api_test_* tests even with the fix from apple#13278 applied.

Related:
- apple#13188 — Initial LSAN suppressions + gRPC use-after-return fix
- apple#13242 — Remove explicit __lsan_do_leak_check() (symbolizer deadlock)
- apple#13255 — Client-side shutdown leak suppressions (30+ entries)
- apple#13278 — Fix root causes: TaskQueue::clear() + DatabaseContext destructor
- apple#13288 — Increase upgrade test shutdown timeout for ASAN
saintstack added a commit that referenced this pull request May 28, 2026
@saintstack
Add LSAN suppression for ThreadSafeTransaction indirect leaks (#13296)
685b3e8 
The existing leak:ThreadSafeDatabase::createTransaction (added in #13278)
only catches the direct ThreadSafeTransaction allocation (2x2048 bytes).
Indirect leaks from transaction operations (TransactionState,
extractReadVersion, WriteMap entries) are attributed to their own allocation
sites, which have ThreadSafeTransaction methods in the call stack but don't
match the createTransaction suppression.

This causes the ASAN nightly to report 10,296 bytes in 48 allocations across
multiple fdb_c_api_test_* tests even with the fix from #13278 applied.

Related:
- #13188 — Initial LSAN suppressions + gRPC use-after-return fix
- #13242 — Remove explicit __lsan_do_leak_check() (symbolizer deadlock)
- #13255 — Client-side shutdown leak suppressions (30+ entries)
- #13278 — Fix root causes: TaskQueue::clear() + DatabaseContext destructor
- #13288 — Increase upgrade test shutdown timeout for ASAN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ploxiln ploxiln ploxiln approved these changes

Assignees

No one assigned

Labels

nightlies Issues to address failures in the nighty runs.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@saintstack @foundationdb-ci @ploxiln