[ENH] Wire up garbage collector to do 3-phase GC. #4987

rescrv · 2025-06-28T22:04:05Z

Description of changes

Wire up the garbage collector to do 3-phase GC.

Test plan

CI

Documentation Changes

N/A

github-actions · 2025-06-28T22:04:26Z

propel-code-bot · 2025-06-28T22:04:34Z

Implement 3-Phase Garbage Collection for WAL Logs and Wire Up End-to-End GC Path

This PR introduces a full 3-phase garbage collection (GC) workflow across the WAL-backed log service and garbage collector in Chroma, extending the API, service implementations, and GC operator logic accordingly. The changes add a new GarbageCollectPhase2 RPC to the logservice API and wire all the GC phases (compute, truncate manifest, delete files) into both the log service and the garbage collector, updating configuration, orchestration, and operator layers to support this across distributed services. It also updates documentation and charts to expose the new API and workflow.

Key Changes:
• Introduces and wires up a new GarbageCollectPhase2 gRPC endpoint in the LogService proto and Rust/Go implementations, allowing phase 2 (manifest truncation) to be performed by the log service.
• Refactors the garbage collector's log deletion operators to use explicit three-phase GC: phase 1 (compute garbage), phase 2 (truncate manifest via RPC call to log service), and phase 3 (physical deletion of log files).
• Extends the log abstraction (Log types in Rust) and their trait implementations to support phase 2 and phase 3 of GC as atomic RPC operations.
• Updates several Kubernetes YAMLs (values/Chart/config/config.yaml) to route and expose the new phase2 GC endpoint, including Helm version bump.
• Documents the rationale for three-phase GC and the responsibilities of each phase in wal3/README.md and code comments, clarifying the architectural division of labor.
• Makes breaking config changes for the garbage collector, requiring explicit log service configuration, and propagates log client configuration throughout the stack.
• Updates test and proptest harness to account for the log API changes.
• Minor: NOP implementations of new endpoint in Go (for go log compatibility), proto/plumbing changes, and CI/test/plumbing fixes.

Affected Areas:
• Distributed garbage collection (Rust GC service, Rust log service, wal3)
• Proto API (logservice.proto, generated Rust/Go code, service interfaces)
• Cluster/k8s configuration (Helm charts, values.yaml, service configs)
• Rust log, log-service, garbage_collector, wal3, and associated glue/operators
• Documentation (wal3/README.md, comment updates)

This summary was automatically generated by @propel-code-bot

Sicheng-Pan · 2025-06-30T17:26:26Z

idl/chromadb/proto/logservice.proto

@@ -163,4 +173,6 @@ service LogService {
  rpc InspectLogState(InspectLogStateRequest) returns (InspectLogStateResponse) {}
  // This endpoint should route to the rust log service.
  rpc ScrubLog(ScrubLogRequest) returns (ScrubLogResponse) {}
+  // This endpoint should route to the rust log service.
+  rpc GarbageCollectPhase2(GarbageCollectPhase2Request) returns (GarbageCollectPhase2Response) {}


nit: can we rename this so that the name explains what this service does in more details?

What would you name it? It's phase2 of the process known as garbage collect.

from my understanding it's truncating the manifest? so maybe TruncateManifest

Sicheng-Pan · 2025-06-30T17:27:48Z

go/pkg/log/server/server.go

@@ -183,6 +183,10 @@ func (s *logServer) ScrubLog(ctx context.Context, req *logservicepb.ScrubLogRequ
 	return
 }

+func (s *logServer) GarbageCollectPhase2(ctx context.Context, req *logservicepb.GarbageCollectPhase2Request) (res *logservicepb.GarbageCollectPhase2Response, err error) {
+	return


does this return nil or an actual response?

It's an empty response. I'm copying other empty responses here. Should we set the struct?

I suppose it's implicitly converting nil to empty response in this case. We can set the struct for clarity or leave it as is for consistency, whichever you prefer

sanketkedia · 2025-06-30T19:31:20Z

rust/garbage_collector/src/operators/delete_unused_logs.rs

-                    match writer.garbage_collect(&GarbageCollectionOptions::default(), Some(*minimum_log_offset_to_keep)).await {
-                        Ok(()) => Ok(()),
+                    match writer.garbage_collect_phase1(&GarbageCollectionOptions::default(), Some(*minimum_log_offset_to_keep)).await {
+                        Ok(true) => {},


when is the writer expected to return true v/s false. And why do we skip phase2 and 3 in case it returns false?

why is phase1 not done by the log service? It can compute and write out the garbage and send a path to the garbage file as a response of the rpc?

sanketkedia · 2025-06-30T19:37:14Z

rust/garbage_collector/src/operators/delete_unused_logs.rs

+                        tracing::error!("Unable to garbage collect log for collection [{collection_id}]: {err}");
+                        return Err(DeleteUnusedLogsError::Gc(err));
+                    };
+                    if let Err(err) = writer.garbage_collect_phase3(&GarbageCollectionOptions::default()).await {


also would prefer to name these phases better instead of just phase1, 2 and 3

sanketkedia · 2025-06-30T19:39:03Z

rust/garbage_collector/src/operators/delete_unused_logs.rs

+                            return Err(DeleteUnusedLogsError::Wal3(err));
+                        }
+                    };
+                    if let Err(err) = logs.garbage_collect_phase2(*collection_id).await {


please add comments on why phase3 is done by the GC service i.e. the argument of keeping a single service that deletes all the files

github-actions · 2025-06-30T21:38:56Z

✅ The Helm chart's version was changed. Your changes to the chart will be published upon merge to main.

## Description of changes Wire up the garbage collector to do 3-phase GC. ## Test plan CI ## Documentation Changes N/A

## Description of changes Included PRs in order: - **[TST] Test for #4972 (#4983)** - **[CLN] Fix dedup in get_collections_with_new_data. (#4974)** - **[ENH] Implement three-phase garbage collection for WAL3 (#4984)** - **[CLN] Remove err(Display) from wal3. (#4992)** - **[ENH] Wire up garbage collector to do 3-phase GC. (#4987)** - **[ENH] Do not materialize all fragments to delete. (#5004)** ## Test plan CI through main and then CI on the hotfix branch. ## Documentation Changes N/A

rescrv requested a review from sanketkedia June 29, 2025 17:49

rescrv force-pushed the rescrv/three-phase-gc branch from ba3fe76 to 9bbea49 Compare June 29, 2025 17:51

rescrv force-pushed the rescrv/three-phase-gc-service branch from d5d740b to a1786d3 Compare June 29, 2025 17:51

Base automatically changed from rescrv/three-phase-gc to main June 30, 2025 16:06

rescrv force-pushed the rescrv/three-phase-gc-service branch from a1786d3 to 5475ab1 Compare June 30, 2025 16:08

Sicheng-Pan reviewed Jun 30, 2025

View reviewed changes

sanketkedia reviewed Jun 30, 2025

View reviewed changes

sanketkedia approved these changes Jun 30, 2025

View reviewed changes

rescrv added 7 commits June 30, 2025 13:40

[ENH] Wire up garbage collector to do 3-phase GC.

b027df3

code complete and cargo nextest run passes

b902089

Use default log config

790de19

use in-memory log for failing test

1385ee6

configure log

e7f10fb

add descriptors to the three phases

528faba

documentation

19793f0

rescrv force-pushed the rescrv/three-phase-gc-service branch from c8d817a to 19793f0 Compare June 30, 2025 20:43

rescrv added 2 commits June 30, 2025 14:07

configure log for garbage collector in tilt

a81a30d

sort out the grpc ordinal connection

7ff781a

rescrv added 2 commits June 30, 2025 14:42

bump helm chart

373d5a3

document what true/false means for phase1

fb61841

rescrv merged commit 76de464 into main Jun 30, 2025
57 checks passed

rescrv deleted the rescrv/three-phase-gc-service branch June 30, 2025 22:53

rescrv added a commit that referenced this pull request Jul 1, 2025

[ENH] Wire up garbage collector to do 3-phase GC. (#4987)

ba1ae89

## Description of changes Wire up the garbage collector to do 3-phase GC. ## Test plan CI ## Documentation Changes N/A

rescrv mentioned this pull request Jul 1, 2025

[HOTFIX] Wal3-gc related fixes from this weekend. #5006

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ENH] Wire up garbage collector to do 3-phase GC. #4987

[ENH] Wire up garbage collector to do 3-phase GC. #4987

Uh oh!

rescrv commented Jun 28, 2025

Uh oh!

github-actions bot commented Jun 28, 2025

Uh oh!

propel-code-bot bot commented Jun 28, 2025 •

edited

Loading

Uh oh!

Sicheng-Pan Jun 30, 2025

Uh oh!

rescrv Jun 30, 2025

Uh oh!

Sicheng-Pan Jun 30, 2025

Uh oh!

Sicheng-Pan Jun 30, 2025

Uh oh!

rescrv Jun 30, 2025

Uh oh!

Sicheng-Pan Jun 30, 2025

Uh oh!

sanketkedia Jun 30, 2025

Uh oh!

sanketkedia Jun 30, 2025 •

edited

Loading

Uh oh!

sanketkedia Jun 30, 2025

Uh oh!

sanketkedia Jun 30, 2025

Uh oh!

github-actions bot commented Jun 30, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

[ENH] Wire up garbage collector to do 3-phase GC. #4987

[ENH] Wire up garbage collector to do 3-phase GC. #4987

Uh oh!

Conversation

rescrv commented Jun 28, 2025

Description of changes

Test plan

Documentation Changes

Uh oh!

github-actions bot commented Jun 28, 2025

Reviewer Checklist

Testing, Bugs, Errors, Logs, Documentation

System Compatibility

Quality

Uh oh!

propel-code-bot bot commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanketkedia Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

propel-code-bot bot commented Jun 28, 2025 •

edited

Loading

sanketkedia Jun 30, 2025 •

edited

Loading

github-actions bot commented Jun 30, 2025 •

edited

Loading