Internal log [Received out of order log message.] is being rate limited. #17176
Replies: 3 comments 3 replies
-
2023-04-19T17:12:34.468466Z ERROR source{component_kind="source" component_id=logs component_type=docker_logs component_name=logs}: vector::internal_events::docker_logs: Received out of order log message. error_type="condition_failed" stage="receiving" container_id="6df0278a4ee6078133f0a0643d39e054839f250f52b8da9250dfae2c2e019afb" timestamp="2023-04-19T17:12:34.466010893Z" internal_log_rate_limit=true 2023-04-19T17:12:34.468435Z ERROR source{component_kind="source" component_id=logs component_type=docker_logs component_name=logs}: vector::internal_events::docker_logs: Internal log [Received out of order log message.] has been rate limited 2 times. |
Beta Was this translation helpful? Give feedback.
-
For clarification, the "rate limit" message indicates that the error has been rate limited, not the source or sink. The actual error you are observing there is that the logs coming from docker were observed to have timestamps that were not in chronological order. The events with older timestamps are dropped, but subsequent logs with newer timestamps will be let through. |
Beta Was this translation helpful? Give feedback.
-
Do you have any idea how often this is happening? Is it something you might be able to identify by visually scanning the output of |
Beta Was this translation helpful? Give feedback.
-
I have configured vector-Loki on the nomad cluster and received an Internal log rate limiting errors from the container. How can I increase the rate limit and solve this issue somehow? Maybe I need to disable rate limiting, Thanks.
Here is my config:
data_dir = "alloc/data/vector/"
[api]
enabled = true
address = "0.0.0.0:8686"
playground = true
[sources.logs]
type = "docker_logs"
[sinks.out]
type = "console"
inputs = [ "logs" ]
encoding.codec = "json"
[sinks.loki]
type = "loki"
inputs = ["logs"]
endpoint = "${var.loki_endpoint}"
encoding.codec = "json"
healthcheck.enabled = true
# since . is used by Vector to denote a parent-child relationship, and Nomad's Docker labels contain ".",
# we need to escape them twice, once for TOML, once for Vector
labels.task = "{{ label."com.hashicorp.nomad.task_name" }}"
labels.group = "{{ label."com.hashicorp.nomad.task_group_name" }}"
labels.namespace = "{{ label."com.hashicorp.nomad.namespace" }}"
labels.node = "{{ label."com.hashicorp.nomad.node_name" }}"
labels.job = "${var.datacenter}"
labels.stream = "{{ stream }}"
remove_label_fields = true
Beta Was this translation helpful? Give feedback.
All reactions