Skip to content

Exclude sharded RocksDB from DDPipelineSaturation to stop OOM failures - #13808

Merged
spraza merged 1 commit into
apple:release-7.4from
spraza:sim-triage-DDPipelineSaturation-2475633740-tosubmit
Aug 5, 2026
Merged

Exclude sharded RocksDB from DDPipelineSaturation to stop OOM failures#13808
spraza merged 1 commit into
apple:release-7.4from
spraza:sim-triage-DDPipelineSaturation-2475633740-tosubmit

Conversation

@spraza

@spraza spraza commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

DDPipelineSaturation fails roughly 3 times in 10000 simulation runs, and every failure is the same one: the process is killed by the 8GB memory watchdog in FDB.

All observed failures ran with storage_engine=ssd-sharded-rocksdb. Sharded rocks is currently not supported, and looks like it cannot absorb the shard explosion this test creates on purpose.

For now, just disabling sharded rocks in this test to reduce noise.

Ran DDPipelineSaturation test 10K times:

20260804-171331-praza-sim-triage-DDPipeline-74f05e489f85ca22 compressed=True data_size=40538476 duration=735887 ended=10000 fail_fast=10 max_runs=10000 pass=10000 priority=100 remaining=0 runtime=0:13:22 sanity=False started=10000 stopped=20260804-172653 submitted=20260804-171331 timeout=5400 username=praza-sim-triage-DDPipelineSaturation-2475633740-tosubmit-dba4fd02c15b35de022b2f42dd268a3bc017a8da

@spraza
spraza marked this pull request as draft July 31, 2026 19:58
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@spraza spraza closed this Aug 3, 2026
@spraza spraza reopened this Aug 3, 2026
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@gxglass

gxglass commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

@spraza I approved this despite its draft status due to a seeming low threshold for disabling anything sharded-rocksdb-related if it causes problems.

@spraza
spraza force-pushed the sim-triage-DDPipelineSaturation-2475633740-tosubmit branch from dba4fd0 to 94faa18 Compare August 4, 2026 17:12
@foundationdb-ci

This comment has been minimized.

Sign in to view
@spraza

spraza commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

@spraza I approved this despite its draft status due to a seeming low threshold for disabling anything sharded-rocksdb-related if it causes problems.

It was draft because I ran into some joshua issue and was ensuring it's noise rather than an actual issue. Confirmed it was noise, here's a subsequent 10K run (just for this test):

20260804-171331-praza-sim-triage-DDPipeline-74f05e489f85ca22 compressed=True data_size=40538476 duration=735887 ended=10000 fail_fast=10 max_runs=10000 pass=10000 priority=100 remaining=0 runtime=0:13:22 sanity=False started=10000 stopped=20260804-172653 submitted=20260804-171331 timeout=5400 username=praza-sim-triage-DDPipelineSaturation-2475633740-tosubmit-dba4fd02c15b35de022b2f42dd268a3bc017a8da

Marking as ready for review.

@spraza
spraza marked this pull request as ready for review August 4, 2026 17:55
@spraza spraza closed this Aug 4, 2026
@spraza spraza reopened this Aug 4, 2026
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@foundationdb-ci

This comment has been minimized.

Sign in to view
@neethuhaneesha

neethuhaneesha commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Will you be forward porting to main branch as well?

@spraza

spraza commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

Will you be forward porting to main branch as well?

Yes, saw the failure in main branch nightly as well

@spraza
Exclude sharded RocksDB from DDPipelineSaturation to stop OOM failures
a51eb98 
DDPipelineSaturation fails roughly 3 times in 10000 simulation runs, and every
failure is the same one: the process is killed by the 8GiB memory watchdog.
Because platform::outOfMemory() only logs SevWarn before calling
criticalError(FDB_EXIT_NO_MEM), these runs carry no SevError at all; joshua
reports them as PeakMemory ~8270000 KB with FailReason=ProducedErrors.

All observed failures ran with storage_engine=ssd-sharded-rocksdb. Sampling
VmRSS externally while re-running a failing seed (2475633740) shows RSS growing
monotonically at ~100MB/s and dying at simtime ~300. Joining those samples
against the trace timeline separates two contributions: the startup burst costs
~39MB per KVS instance as 33 ShardedRocksDB instances open (each reserving its
own SHARDED_ROCKSDB_BLOCK_CACHE_SIZE = 128MB LRU cache), while the remaining
+6.8GB arrives with the instance count flat and instead tracks column family
activity, ~7.5MB per shard event. The column families accumulate: 476 OPEN
against only 87 DESTROY over the run, leaving 389 live at death, which at
SHARDED_ROCKSDB_WRITE_BUFFER_SIZE = 16MB each accounts for the growth.

The test pins min_shard_bytes = 10000, a thousand times below the default, on
purpose - that shard explosion is how it drives pipelineSize() past
DD_MAX_PIPELINE_MOVES. Under an engine that spends a write buffer per shard, and
with every simulated process sharing one address space, those budgets sum past
the cap. The pipeline gate being tested does not depend on the storage engine, so
exclude engine 5 here as BulkDumping, BulkLoading and CacheTest already do.

The DD pipeline gate is not implicated: at the moment of death PipelineSize,
InFlight, InQueue, PendingGateRelocations and UnhealthyRelocations are all 0, and
the gate tripped only twice in the entire run.
@spraza
spraza force-pushed the sim-triage-DDPipelineSaturation-2475633740-tosubmit branch from 94faa18 to a51eb98 Compare August 4, 2026 23:30
@foundationdb-ci

foundationdb-ci commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS 14.x

  • Commit ID: a51eb98
  • Duration 0:36:18
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: a51eb98
  • Duration 0:44:00
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux RHEL 9

  • Commit ID: a51eb98
  • Duration 0:47:46
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS 14.x

  • Commit ID: a51eb98
  • Duration 0:53:16
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: a51eb98
  • Duration 0:55:31
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: a51eb98
  • Duration 1:29:00
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@spraza
spraza merged commit 2f5747b into apple:release-7.4 Aug 5, 2026
7 checks passed
@spraza spraza mentioned this pull request Aug 5, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@neethuhaneesha neethuhaneesha neethuhaneesha approved these changes
@gxglass gxglass gxglass approved these changes
@saintstack saintstack Awaiting requested review from saintstack
@akankshamahajan15 akankshamahajan15 Awaiting requested review from akankshamahajan15

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@spraza @foundationdb-ci @gxglass @neethuhaneesha