Loki v3.5.5 ignores limits_config.reject_old_samples: false on specific host

[Ivan-LCH]

1. Environment Details:

Loki Version: 3.5.5

Agent Version: Grafana Alloy v1.8.1

Installation Method: Docker (also observed with direct binary execution)

Host OS: Ubuntu 20.04 LTS

2. Bug Description:

On our Ubuntu 20.04 server, Loki v3.5.5 successfully starts and parses its configuration file. However, at runtime, it completely ignores the reject_old_samples: false setting provided in limits_config.

When Grafana Alloy sends log entries with timestamps older than 7 days, Loki's ingester component consistently rejects them with an entry too far behind error.

Additionally, the error message contains a static, frozen oldest acceptable timestamp value (...is: 2025-08-14T13:59:45Z). This timestamp never changes and does not reflect the server's actual, correct current time, which is a strong indicator of a bug.

3. Expected Behavior:

When reject_old_samples: false is configured, Loki should accept all log entries from Alloy, regardless of their timestamp.

4. How to Reproduce:

This issue is consistently reproducible on our specific host environment.

Step 1: Run Loki v3.5.5 with the following minimal configuration.

loki-config.yaml:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: info
  grpc_server_max_concurrent_streams: 1000

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

limits_config:
  metric_aggregation_enabled: true
  reject_old_samples: false
  reject_old_samples_max_age: 8760h

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

pattern_ingester:
  enabled: true
  metric_aggregation:
    loki_address: localhost:3100

ruler:
  alertmanager_url: http://localhost:9093

frontend:
  encoding: protobuf

Docker Command:

root@ivan-ubuntu20:/DOCKAREA/ivan# more COMMAND.FILE.loki
docker stop loki
docker rm loki
docker ps -a | grep loki
rm -rf /DOCKAREA/loki/data
mkdir /DOCKAREA/loki/data
chown -R 10001:10001 /DOCKAREA/loki/data
chown 10001:10001 /DOCKAREA/loki/loki-config.yaml
chmod 644 /DOCKAREA/loki/loki-config.yaml

docker run -d --name loki -p 3100:3100 -v /DOCKAREA/loki:/mnt/config -v /DOCKAREA/loki/data:/tmp/loki grafana/loki:3.5.5 -config.file=/mnt/config/loki-config.yaml

alloy.yaml:

local.file_match "jeusAccess" {
        path_targets = [{
                __address__ = "localhost",
                __path__    = "/DATA/alloy/test_data/web_test_100.txt",
                job         = "jeusAccess",
                host_ip     = env("HOST_IP"),
                host_name   = env("HOST_NAME"),
                sys_name    = env("SYS_NAME1"),
                instance    = env("INSTANCE1"),
        }]
}

loki.process "jeusAccess" {
    forward_to = [loki.write.default.receiver]

    stage.regex {
        expression = `^(?P<ip>[\w.:]+)\s+\[(?P<ts>[^\]]+)\]\s+\"(?P<request>[^\"]+)\"\s+(?P<status>\d{3})\s+(?P<size>\S+)\s+(?P<duration_micro>\d+)\
s+\"(?P<forwarded>[^\"]*)\"\s+\"(?P<agent>[^\"]*)".*`
    }

    stage.timestamp {
        source   = "ts"
        format   = "02/Jan/2006:15:04:05 -0700"
        location = "Asia/Seoul"
    }

    stage.replace {
        expression = `^-$`
        source     = "size"
        replace    = "0"
    }

    stage.template {
        source   = "output_json"
        template = `{` +
                   `"ip":"{{.ip}}", ` +
                   `"x_forwarded_for":"{{.x_forwarded}}", ` +
                   `"timestamp":"{{.ts}}", ` +
                   `"request":"{{.request}}", ` +
                   `"status_code":{{.status}}, ` +
                   `"size_bytes":{{.size}}, ` +
                   `"response_microseconds":{{.duration_micro}}, ` +
                   `"user_agent":"{{.agent}}"` +
                   `}`
    }


    stage.output {
        source = "output_json"
    }

    stage.labels {
        values = {
            job = "jeusAccess",
        }
    }
}



loki.source.file "jeusAccess" {
        targets               = local.file_match.jeusAccess.targets
        forward_to            = [loki.process.jeusAccess.receiver]
        legacy_positions_file = "/tmp/positions_jeus.yaml"
}

loki.write "default" {
        endpoint {
                url = "http://***.***.***.***:3100/loki/api/v1/push"
        }
        external_labels = {}
}

Step 3: Observe the Loki container logs.

5. Relevant Log Output:

Loki successfully starts, but when Alloy sends the old logs, Loki produces the following error:

level=info ts=2025-09-15T07:09:35.624693325Z caller=recalculate_owned_streams.go:52 msg="completed recalculate owned streams job"
level=error ts=2025-09-15T07:09:35.84147881Z caller=manager.go:50 component=ingester path=write msg="write operation failed" details="entry too far behind, entry timestamp is: 2025-08-10T14:59:55Z, oldest acceptable timestamp is: 2025-08-14T13:59:45Z for stream {filename=\"/DATA/alloy/test_data/web_test_100.txt\", host_ip=\"***.***.***.***\", host_name=\"ivan-ubuntu20\", instance=\"jeus-8088-***.***.***.***\", job=\"jeusAccess\", service_name=\"jeusAccess\"}" org_id=fake
level=warn ts=2025-09-15T07:09:35.841867076Z caller=grpc_logging.go:76 method=/logproto.Pusher/Push duration=588.329µs msg=gRPC err="rpc error: code = Code(400) desc = entry with timestamp 2025-08-10 14:59:55 +0000 UTC ignored, reason: 'entry too far behind, entry timestamp is: 2025-08-10T14:59:55Z, oldest acceptable timestamp is: 2025-08-14T13:59:45Z',\nuser 'fake', total ignored: 1 out of 100 for stream: {filename=\"/DATA/alloy/test_data/web_test_100.txt\", host_ip=\"***.***.***.***\", host_name=\"ivan-ubuntu20\", instance=\"jeus-8088-***.***.***.***\", job=\"jeusAccess\", service_name=\"jeusAccess\"}"

6. Additional Context:

We have performed extensive debugging and can confirm this is not a standard configuration error:
The host system's time is correct and synchronized.
The issue persists when running the Loki binary directly on the host, eliminating Docker as the root cause.
The same behavior was also observed on older Loki versions (2.9.x series).
The combination of the ignored configuration and the frozen, incorrect timestamp in the error message points to a deep, environmental incompatibility or a rare bug within Loki's core logic.

[seregayoga]

@Ivan-LCH Maybe this is what you're looking for https://grafana.com/docs/loki/latest/configure/#accept-out-of-order-writes?

How far into the past accepted out-of-order log entries may be is configurable with max_chunk_age. max_chunk_age defaults to 2 hour. Loki calculates the earliest time that out-of-order entries may have and be accepted with
YAML

time_of_most_recent_line - (max_chunk_age/2)

Log entries with timestamps that are after this earliest time are accepted. Log entries further back in time return an out-of-order error.

[Ivan-LCH]

Thanks alot