Self-hosted ClickHouse disk fills up — lightweight-deleted rows never reclaimed

[KarolisKont]

Describe your question

TL;DR

We self-host Langfuse. The observations table on our ClickHouse cluster keeps growing even though we have a 14-day retention configured. Roughly 85% of the disk used by the active observations partition is rows that have already been deleted by retention — they just don't get physically removed. We have to log in and reclaim the space manually every few days.

We'd like to know whether this is expected, and what the recommended way to keep disk usage bounded is on a self-hosted setup.

Our setup

• Langfuse: v3.155.1. Web and worker run as Kubernetes deployments — they are the only components on Kubernetes.
• ClickHouse: v25.8 (Altinity Stable). Deployed on dedicated bare-metal servers, not in Kubernetes.
  • 1 shard × 3 replicas of clickhouse-server (ReplicatedMergeTree, full data on every node).
  • 3 ClickHouse Keeper nodes for replication coordination (separate ensemble from clickhouse-server).
  • ~3 TB of local disk per replica.
• PostgreSQL: on a dedicated bare-metal host.
• Redis: clustered on dedicated VMs.
• Tenant: one active project, retention set to 14 days.
• Ingest: ~70–90 GB compressed per day, almost all of it into observations.

What happens

When the retention window passes, Langfuse runs:

DELETE FROM observations WHERE project_id = '…' AND start_time < …

ClickHouse handles that as a lightweight delete — it marks the rows invisible but does not remove their bytes from disk. The space is supposed to come back later when ClickHouse merges the affected data parts in the background.

In practice that doesn't happen for us:

• The observations table is partitioned by month (toYYYYMM(start_time)).
• At our ingest rate, the current month's data parts grow to ~150 GB within a few days.
• ClickHouse has a default safety limit (max_bytes_to_merge_at_max_space_in_pool = 150 GiB): it won't merge parts above that size. Once a part hits the limit, the background merger never touches it again.
• The retention job keeps marking rows inside those frozen parts as deleted, but the parts never shrink.
• After a partition stops receiving writes (e.g. once the month is over), nothing drops it either — Langfuse only ever issues row-level deletes, never DROP PARTITION.

So the deleted rows just sit there, forever, taking up disk.

Concrete numbers from today

Among the parts that contain lightweight-deleted rows on our cluster right now:

Table	Partition	Size on disk	Share that is already deleted
observations	May 2026	~802 GiB	~85% (≈ 679 GiB dead)
traces	May 2026	~125 GiB	~79%
traces	April 2026	~2.2 GiB	100% (every row past retention)
scores	April 2026	~68 MiB	100%

The April partitions are interesting: every row in them is well past the 14-day cutoff, they are 100% logically deleted, but they still occupy full disk space because nothing ever drops them.

How we currently keep the disk from filling up

We manually run, against the cluster, whenever disk usage gets uncomfortable:

ALTER TABLE observations ON CLUSTER '<cluster>'
APPLY DELETED MASK IN PARTITION '<yyyymm>';

This rewrites only the affected parts and physically drops the deleted rows. The last run reclaimed about 360 GiB per replica in ~17 minutes. It works, but:

• It's a manual operation we have to repeat every few days.
• The bloat builds back up because new parts keep hitting the 150 GB ceiling.

OPTIMIZE TABLE … FINAL would also work but needs a partition's worth of free disk as scratch space, which we don't comfortably have.

Questions

1. Is there an official/intended way to reclaim space from lightweight-deleted rows in a self-hosted Langfuse? Is APPLY DELETED MASK the recommended answer, or are we missing something Langfuse provides out of the box?

2. What should we do with fully-aged-out partitions (like April above — 100% past retention, still on disk)? Should self-hosters be running ALTER TABLE … DROP PARTITION '<id>' themselves as a periodic cleanup? We couldn't find this documented anywhere.

3. Should self-hosters set up their own ClickHouse cron jobs to free up disk space? Concretely: is the expected operational pattern that every self-hosting team writes its own scheduled job that runs APPLY DELETED MASK on active partitions and DROP PARTITION on fully-aged ones? If so, having a reference example or a documented playbook in the Langfuse docs would help a lot. If not, what does Langfuse expect to handle this?

4. Would Langfuse consider an option to partition observations / traces / scores by day instead of by month? With daily partitions, a simple DROP PARTITION once a day is past retention would reclaim space instantly, with no scratch disk required, and disk usage would stay bounded by retention × daily_ingest.

5. Are we doing something wrong / missing a configuration option that would handle this automatically? Happy to be told we've misread the system 😊.

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

This is a known challenge with self-hosted ClickHouse deployments at scale. Here's what the documentation covers and some gaps:

What Langfuse Documents

For ClickHouse disk management, the recommended approach is to use ClickHouse's TTL feature to automatically remove data from the traces, observations, scores, and event_log tables. (1)

For slow or timing-out deletions, you can increase LANGFUSE_CLICKHOUSE_DELETION_TIMEOUT_MS (default 600000, i.e. 10 minutes) to give long-running deletes more headroom. (1)

On ClickHouse 25.7 and above, you can reduce mutation pressure by opting into lightweight deletes and updates: (1)

• Set CLICKHOUSE_LIGHTWEIGHT_DELETE_MODE from its default alter_update to lightweight_update (or lightweight_update_force) to resolve DELETE statements via lightweight deletes instead of ALTER TABLE ... DELETE mutations
• Set CLICKHOUSE_USE_LIGHTWEIGHT_UPDATE=true to route tracing-table updates through native UPDATE statements instead of ALTER TABLE ... UPDATE mutations

Both reduce the amount of background merge work ClickHouse performs on deletion-heavy workloads. (1)

For larger deployments where ClickHouse disk growth becomes an operational bottleneck, consider ClickHouse Cloud or BYOC. These deployments separate storage from compute, so storage can grow independently of the compute that serves Langfuse queries. (1)

What's Not Documented

Your specific scenario—where lightweight-deleted rows never get physically reclaimed due to parts exceeding max_bytes_to_merge_at_max_space_in_pool—is not explicitly addressed. The documentation does not provide:

• Guidance on running APPLY DELETED MASK or DROP PARTITION as periodic maintenance
• A recommended cron job or playbook for self-hosters
• Options to change partitioning granularity (daily vs monthly)

Recommendations Based on Available Info

1. Your manual APPLY DELETED MASK approach appears to be a valid workaround, though it's not officially documented as the recommended path.

2. For fully-aged-out partitions (like your April data at 100% deleted), DROP PARTITION would be effective, but this is also not documented as part of Langfuse's expected operational model.

3. You might consider raising max_bytes_to_merge_at_max_space_in_pool on your ClickHouse cluster if you have sufficient resources, though this is ClickHouse configuration rather than Langfuse-specific.

4. The questions about daily partitioning and official self-hosted maintenance playbooks would be good feature requests for the Langfuse team—these aren't covered in current documentation.

📚 Sources:

1. Scaling

---

Have another question? Just tag @inkeep.

[KarolisKont]

For ClickHouse disk management, the recommended approach is to use ClickHouse's TTL feature to automatically remove data from the traces, observations, scores, and event_log tables. (1)

Document recommend to use ClickHouse TTL for log tables. I don't think I can use/set TTL for observations, traces, etc. tables ("data" related tables), because per project bases Retention Period can be changed - make longer then TTL, or new project can be added with longer Retention period than table TTL. So there would be need always check if project retention period doesn't exceeds table TTL.

On ClickHouse 25.7 and above, you can reduce mutation pressure by opting into lightweight deletes and updates: (1)

Looking at how those flags are wired in Langfuse:

• CLICKHOUSE_LIGHTWEIGHT_DELETE_MODE is applied as a session-level setting in packages/shared/src/server/clickhouse/client.ts. It changes how a DELETE FROM … is executed (legacy ALTER UPDATE _row_exists=0 mutation vs. 25.7's patch-part mechanism). Either way the deleted rows are only physically removed when a later merge rewrites the affected part.
• CLICKHOUSE_USE_LIGHTWEIGHT_UPDATE only swaps the SQL form for updates on events_full / events_core in packages/shared/src/server/repositories/events.ts. It doesn't touch the observations / traces / scores delete path at all.

So both flags reduce delete-time mutation pressure (CPU / IO / merge churn during the retention sweep). Neither flag changes the symptom in this thread, which is physical disk not being reclaimed after rows have already been marked deleted. In our case the parts already exceed max_bytes_to_merge_at_max_space_in_pool = 150 GiB, so no background merge ever touches them again regardless of which delete mechanism was used to mark the rows.

[pratimapatel2008]

@KarolisKont Thank you for sharing detailed findings. I think the easiest way is run CRON Job (APPLY DELETED MASK) to force the deletion on rows which were marked as deleted by ClickHouse. from the disk.

Doc: https://clickhouse.com/docs/sql-reference/statements/alter/apply-deleted-mask

Let me check if we have any plans to change the partition scheme for ClickHouse tables.

[KarolisKont]

Thanks for response 🙇. Wanted to check if not doing something wrong and setting a Cron Job (APPLY DELETED MASK) best/good approach for now.

[sumerman]

Hi @KarolisKont I will explore adding an internal APPLY DELETED MASK routine. Meanwhile running it on cron is the way to go.

Should self-hosters be running ALTER TABLE … DROP PARTITION '' themselves as a periodic cleanup? We couldn't find this documented anywhere.

This should also be acceptable provided you don't have tenants with retention longer than 14 days.

[KarolisKont]

Regarding ALTER TABLE … DROP PARTITION, it have to be more elegant - find the longest retention period of existing project and use that when to drop partitions :)

[pratimapatel2008]

@KarolisKont Please let us know if your ask has been answered.

[KarolisKont]

Yes thank you.

[patrickherrmann]

FWIW, I ran into this as well. The data retention job was running, but disk space was not getting reclaimed. Running APPLY DELETED MASK in a cronjob is working for us, but the data retention job was not clearing space out of the box.