Bug: err-mimir-sample-out-of-order for seemingly no reason

[NRCan-LGariepy]

What is the bug?

Mimir is generating error logs for out of order samples a couple of times every day. The logs look like so :

The main error message is

the sample has been rejected because another sample with a more recent timestamp has already been ingested and out-of-order samples are not allowed (err-mimir-sample-out-of-order). The affected sample has timestamp 2025-10-27T09:01:20.626Z and is from series kube_pod_owner ..."

Full log line example:

    {
        "line": "ts=2025-10-27T09:01:25.687830499Z caller=grpc_logging.go:76 level=warn method=/cortex.Ingester/Push duration=1.005569ms msg=gRPC err=\"user=cost-analyzer_ccmeo-geoproc-dev: the sample has been rejected because another sample with a more recent timestamp has already been ingested and out-of-order samples are not allowed (err-mimir-sample-out-of-order). The affected sample has timestamp 2025-10-27T09:01:20.626Z and is from series kube_pod_owner{instance=\\\"opencost-1-1741878999.opencost.svc:9003\\\", job=\\\"prometheus.scrape.opencost\\\", namespace=\\\"cdtk-stage\\\", owner_is_controller=\\\"true\\\", owner_kind=\\\"ReplicaSet\\\", owner_name=\\\"cdtk-registry-api-stage-8f9bdc6cd\\\", pod=\\\"cdtk-registry-api-stage-8f9bdc6cd-mrg6w\\\"} (sampled 1/10)\"",
        "timestamp": "1761555685704581248",
        "fields": {
            "app": "mimir",
            "cluster": "ccmeo-geoproc-dev",
            "component": "ingester",
            "container": "ingester",
            "controller": "mimir-ingester",
            "detected_level": "warn",
            "filename": "/var/log/pods/mimir_mimir-ingester-0_eb0759a1-8db6-4c04-998f-1c376823c4e7/ingester/0.log",
            "instance": "mimir",
            "job": "mimir/ingester",
            "namespace": "mimir",
            "node": "redacted",
            "pod": "mimir-ingester-0",
            "service_name": "mimir",
            "stream": "stderr",
            "version": "2.16.0"
        }
    }

How to reproduce it?

1. Scrape OpenCost with Grafana Alloy and send it to Mimir using the following config:

// Scrape targets for services
discovery.kubernetes "opencost" {
    role = "service"

    namespaces {
        names = ["opencost"]
    }
}

// Scrape metrics only from the opencost service
prometheus.scrape "opencost" {
    targets      = discovery.kubernetes.opencost.targets
    forward_to   = [prometheus.remote_write.mimir.receiver]
    honor_labels = true
}

prometheus.remote_write "mimir" {
    endpoint {
        url = "http://mimir-nginx.mimir.svc/api/v1/push"

        headers = {
            "X-Scope-OrgID" = "cost-analyzer_ccmeo-geoproc-dev",
        }
    }
}

2. Observe the error logs

What did you think would happen?

No errors logs.

What was your environment?

Kubernetes RKE2 1.32.6

Managed by Rancher 2.11.3

Deployed via the Mimir Distributed 5.7.0 chart.

Any additional context to share?

No response

[aknuds1]

It's not a bug that Mimir rejects out-of-order samples. I suggest you configure out-of-order sample ingestion, or figure out what causes Alloy to send out-of-order samples. Converting the issue to a discussion, since there's no evidence of a bug.

[NRCan-LGariepy]

Indeed, the rejections themselves are not the bug, but there's not a clear reason what causes them to happen in the first place. You can find similar bug reports, for example here: #10830.

[aknuds1]

I think you need to figure out whether Alloy is sending out-of-order samples. Mimir isn't to blame for samples arriving out of order.

[aknuds1]

#10830 is also a discussion, not a bug report, for the record.

[NRCan-LGariepy]

I ended up fixing this issue, which was most likely caused by alloy not operating in clustering mode, such that all pods in the daemonset were scraping the same endpoint at the same time, which sometimes caused out-of-order errors when remote-writing to mimir.