Unexpected too far behind errors

[moray95]

Loki rejects logs with "too far behind" errors when reject_old_samples is disabled and logs are no older than reject_old_samples_max_age.

Configuration:

 limits_config:
  reject_old_samples: false
  reject_old_samples_max_age: 168h

Steps to reproduce the behavior:
I am not sure how to reproduce the issue but have noticed it mostly appears after a downtime either on Loki itself or the backing object storage.

Expected behavior
After setting reject_old_samples to false, the logs should never be rejected with "entry too far behind" errors, and especially not when logs are no older than the configured reject_old_samples_max_age.

Environment:

• Loki: 2.4.1
• Deployment tool: Installed using the Helm chart v2.8.4
• Infrastructure: RKE 1.21.6

Screenshots, Promtail config, or terminal output

level=warn ts=2022-04-15T09:25:11.8678788Z caller=grpc_logging.go:38 duration=498.136µs method=/logproto.Pusher/Push err="rpc error: code = Code(400) desc = entry with timestamp 2022-04-15 03:20:00.80086913 +0000 UTC ignored, reason: 'entry too far behind' for stream: {REDACTED},
entry with timestamp 2022-04-15 03:20:00.883413328 +0000 UTC ignored, reason: 'entry too far behind' for stream: {REDACTED},
entry with timestamp 2022-04-15 03:20:04.550525553 +0000 UTC ignored, reason: 'entry too far behind' for stream: {REDACTED},
entry with timestamp 2022-04-15 03:20:04.936669923 +0000 UTC ignored, reason: 'entry too far behind' for stream: {REDACTED},
entry with timestamp 2022-04-15 03:20:05.402723364 +0000 UTC ignored, reason: 'entry too far behind' for stream: {REDACTED},
entry with timestamp 2022-04-15 03:20:05.977017186 +0000 UTC ignored, reason: 'entry too far behind' for stream: {REDACTED},
entry with timestamp 2022-04-15 03:20:07.235240884 +0000 UTC ignored, reason: 'entry too far behind' for stream: {REDACTED},
entry with timestamp 2022-04-15 03:20:07.90163636 +0000 UTC ignored, reason: 'entry too far behind' for stream: {REDACTED},
entry with timestamp 2022-04-15 03:20:09.9954798 +0000 UTC ignored, reason: 'entry too far behind' for stream: {REDACTED},
entry with timestamp 2022-04-15 03:20:10.159147785 +0000 UTC ignored, reason: 'entry too far behind' for stream: {REDACTED},
total ignored: 926 out of 926" msg="gRPC
"

[slim-bean]

ordering of samples is always per stream, and for a given stream Loki will accept out of order samples for at most max_chunk_age/2 behind the most recent entry sent. This is a little confusing so I'll try to post some examples.

For a stream {job="foo",app="bar"} and a default max_chunk_age: 2h if you send a log to that stream with a timestamp of 2022-04-19T12:00:00Z, Loki will accept logs as far back as 2022-04-19T11:00:00Z. Anything older than that will be rejected with entry too far behind

It's important to note though that Loki will always accept much older samples, it's always the relationship to the most recent received timestamp for a given stream.

If you want to load older logs into Loki, you need to do so in a roughly chronological order, you can playback logs from years ago as long as you start with the older logs and work your way forward. Example, today is 2022-04-19, as long as you have reject_old_samples: false you can send a log with timestamp 2018-01-01T13:00:00Z, the same rules apply though, if that's the most recent log for the stream you can only send older logs that have a time newer than 2018-01-01T12:00:00Z (you can send anything newer)

Make sure you have a schema entry that has a from date older than the oldest logs you want to send.

[moray95]

Thanks a lot for the detailed explanation @slim-bean ! Didn't know max_chunk_age had such a use case. I'm closing this issue for now, will check back after fine-tuning our configuration according to the new info.

[slim-bean]

Welcome! I would comment that increasing max chunk age to work around this will have some limits and perhaps unintended consequences depending on how big you want to go. That is, if you wanted to make it more than 24 hours I don't know if that may cause you query problems? We've never tried it.

Another consideration would be adding another label to the older logs to send them as another stream.

Good luck!

[mhoyer]

Still to me it's not 100% clear how both config parameters are interacting. Is it A or B?

A: Is reject_old_samples=true ignoring reject_old_samples_max_age and rejecting everything?
B: Is reject_old_samples=true only rejecting when reject_old_samples_max_age is fulfilled?

Maybe worth a clarification also in the configuration section here? https://grafana.com/docs/loki/latest/configuration/#limits_config

[mhoyer]

I might found the answer (B) already in the sources? https://github.com/grafana/loki/blob/main/pkg/distributor/validator.go#L68

Can you confirm? Thus, reject_old_samples=false is ignoring reject_old_samples_max_age completely, right?

[slim-bean]

Can you confirm? Thus, reject_old_samples=false is ignoring reject_old_samples_max_age completely, right?

This is correct, these two configurations are a little bit legacy, when Loki needed a separate index store like dynamodb it was desirable to limit how far back you could ingest data because there was code that would reduce the provisioned write throughput on old tables to save cost.

[dorinand]

Hello, I have similar issue. This is my limit configuration:

limits_config:
  # 2 weeks
  retention_period: 336h
  reject_old_samples: false
  reject_old_samples_max_age: 43800h

I set reject old samples max age to 5 years by this configuration (max_chunk_age is default 2h) and also ignore rejecting old samples. I still getting entry too far behind error for logs old only few hours. I would expect from the configuration, I will be able to import logs from 5y ago.

So how does this work? Should I also configure max_chunk_age? I sending logs to Loki with FIlebeat (Filebeat -> Logstash -> Loki). Maybe Filebeat and Logstash change the order of the logs?

Loki 2.5.0

[chandramohan97911]

@dorinand I am also facing same issue with promtail while scraping the instance logs to loki. Please let me know if you fix the entry too far behind issue and could share working config.

thanks,
Chandra

[charles-horel]

I am having the same issue with fluentd:
2023-03-24 20:17:12 +0000 [warn]: #0 [clusterflow:logging:loki-flow:clusteroutput:logging:loki-output] failed to write post to http://loki-gateway.loki/loki/api/v1/push (400 Bad Request entry with timestamp 2023-03-24 08:10:52.973597146 +0000 UTC ignored, reason: 'entry too far behind, oldest acceptable timestamp is: 2023-03-24T18:40:47Z' for stream: ....

We have changed the following loki configs and they are not making a difference:

loki:
  limits_config:
    reject_old_samples: false
    
  ingester:
    max_chunk_age: 24h

[waney316]

Anyone slove the problem? I am having the same issue.

[adegoodyer]

ordering of samples is always per stream, and for a given stream Loki will accept out of order samples for at most max_chunk_age/2 behind the most recent entry sent. This is a little confusing so I'll try to post some examples.

For a stream {job="foo",app="bar"} and a default max_chunk_age: 2h if you send a log to that stream with a timestamp of 2022-04-19T12:00:00Z, Loki will accept logs as far back as 2022-04-19T11:00:00Z. Anything older than that will be rejected with entry too far behind

It's important to note though that Loki will always accept much older samples, it's always the relationship to the most recent received timestamp for a given stream.

If you want to load older logs into Loki, you need to do so in a roughly chronological order, you can playback logs from years ago as long as you start with the older logs and work your way forward. Example, today is 2022-04-19, as long as you have reject_old_samples: false you can send a log with timestamp 2018-01-01T13:00:00Z, the same rules apply though, if that's the most recent log for the stream you can only send older logs that have a time newer than 2018-01-01T12:00:00Z (you can send anything newer)

Make sure you have a schema entry that has a from date older than the oldest logs you want to send.

Such an insightful and useful post - this needs to be reflected clearly in the official docs somewhere.

So my situation was data corruption of stored logs data so needed to reload logs being ingested from AWS.

Data pipeline is Cloudwatch > Filebeats > Logstash > Loki - all deployed on Kubernetes.

After restarting Loki would only ingest data for last hour (so using above advice - max_chunk_time / 2 for most recently received log).

After reading that logs can be replayed but must be in order - I cleared Loki storage (pvc's in this case), scaled down all filebeats so no logs were being sent (that's critical as that's what determines the cut-off point) and made sure filebeat config was start_position: beginning to streams logs from oldest first.

I then scaled up filebeats pods which (roughly) sent logs in order from a date starting years ago (give or take the max_chunk_time) - meaning that all logs were considered by Loki (before retention logic) - not just the previous hour.

I did have to set reject_old_samples: false to do this.

Hope this helps someone else somewhere and big thanks to @slim-bean as wouldn't have been able to understand it fully without his info.

[gpadidala]

I was able to resolve this issue on Loki 3.4.x by enabling the new
time-based stream sharding feature that was introduced in later releases.

The problem I was facing was due to long out-of-order / historical log
ingestion, which caused "entry too far behind" errors. This was fixed by
enabling time sharding at the distributor level.

Configuration used:

shard_streams:
enabled: true
time_sharding_enabled: true
time_sharding_ignore_recent: 40m
logging_enabled: false
desired_rate: 1536

With this enabled, Loki automatically injects a synthetic __time_shard__
label and ensures that no stream spans more than ~max_chunk_age/2. This
allows ingestion of very old logs without violating ordering constraints.

After enabling this:

• Old log entries were ingested successfully
• No more "entry too far behind" errors
• Streams were time-bounded as expected
• Queries worked normally without changes

This matches the functionality added via:
#14711
0d6d68d

Posting here in case anyone else runs into the same issue.