Bug: Out‑of‑order errors when inserting future data on new metric

[KrishnaJyothika]

What is the bug?

We are observing unexpected ingestion behavior in Mimir when inserting future data as a new series and on a completely new metric.

Configuration:

creation_grace_period = 7d
out_of_order = 6h

Observed Behavior: When future data is inserted with a new metric, all the existing live data already being ingested for that tenant is rejected with an "out of order" error resulting the data loss.

ts=2025-11-13T16:27:00.055561086Z caller=grpc_logging.go:76 level=warn method=/cortex.Ingester/Push duration=1.571926ms msg=gRPC err="user=test: the sample has been rejected because another sample with a more recent timestamp has already been ingested and this sample is beyond the out-of-order time window of 6h (err-mimir-sample-timestamp-too-old). The affected sample has timestamp 2025-11-13T16:27:00.002Z and is from series avalanche_metric_mmmmm_0_189{cycle_id=\"0\", grafana_alloy_agent=\"mimir-test-agent-0\", instance=\"localhost:9001\", job=\"avalanche-metrics\", label_key_kkkkk_0=\"label_val_vvvvv_0\", load_test_area=\"Avalanche-load-test-alloy\", pod=\"mimir-test-agent-0\", podname=\"mimir-test-agent-0\", series_id=\"0\"} (sampled 1/10)"

How to reproduce it?

1. Configure Mimir with creation_grace_period=7d and out_of_order=6h.
2. Insert live data for a tenant (normal ingestion works).
3. Insert future data for a completely new metric under the same tenant.
4. Observe that existing live data ingestion is rejected with "out of order".

What did you think would happen?

Since creation_grace_period is set to 7 days, inserting future data for a completely new metric and new series should be allowed without impacting ongoing ingestion. The live data already flowing into that tenant should continue to be accepted normally. The system should treat the future data as valid within the grace period, and not reject existing live data as “out of order.

What was your environment?

Kubernetes
Helms
Mimir 2.14.2v

Any additional context to share?

No response

[aknuds1]

2. Insert live data for a tenant (normal ingestion works).
3. Insert future data for a completely new metric under the same tenant.
4. Observe that existing live data ingestion is rejected with "out of order".

Can you clarify what you mean by "existing live data ingestion" being rejected? From your log message it looks as if you are writing samples to the time series avalanche_metric_mmmmm_0_189{cycle_id="0", grafana_alloy_agent="mimir-test-agent-0", instance="localhost:9001", job="avalanche-metrics", label_key_kkkkk_0="label_val_vvvvv_0", load_test_area="Avalanche-load-test-alloy", pod="mimir-test-agent-0", podname="mimir-test-agent-0", series_id="0"}, which are rejected because they are more than six hours older than the most recent sample of the same series.

[KrishnaJyothika]

Hi @aknuds1,

Once we had inserted the future data, the data which being sent to the tenant started to get rejected. Even though the incoming data aligned with the current timestamp, causing the active series to drop entirely since all new samples were being rejected.

Future data

Drop in active series after inserting the future data

Out-of-orders after future data insert

Kindly let me know if any additional information is required.

[56quarters]

This is expected behavior unless you enable out-of-order ingestion. Once you insert a sample for time T=1, all samples that follow must be for T > 1

There's a good explanation here.

[KrishnaJyothika]

We’re planning to increase our out‑of‑order tolerance for this. While inserting future data, we’ve noticed two new issues:

1. After inserting both existing data and future data into Mimir, everything looks fine initially — we can query and see both sets of data without any problems. However, after about 12 hours, when we attempt to read the data, no metrics or labels are returned. Interestingly, if we query a specific metric directly, the full data is available. Note: There’s no drop in ingestion rate or active series during this period.

We’re able to successfully read both the existing data and the future data

However, after some time, no metrics or labels are returned if we try for time range less 12hrs — even though the underlying data is still present

2. For some of the future data metrics, we’re observing gaps in the data points. This is unexpected because the data was fully available at the time of insertion.

Please let us know if any adjustments to the read path or compactor configuration are required in order to reliably maintain future data in Mimir without issues.