-
Notifications
You must be signed in to change notification settings - Fork 487
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
loki.source.docker error: could not set up a wait request to the Docker client #4403
Comments
Since the original problem occurred on an older system, I have also tested the same config on another machine running Docker with same results. The other machine is running I also checked the |
I'm having exactly the same problem as you. |
Can confirm. Additionally I see a CPU spike in the agent and loki containers every minute. Goes away when Interestingly, the default refresh interval for the prometheus docker SD object is 1 minute as well.
|
|
@mai-rnd Can you share an example of a working filter setup? |
I have filtered by port, for example:
But it's not universal solution. Still try to find some "univeral" way. |
Hey there, thanks for the extra datapoint @mai-rnd, will keep it in mind!
Are you referring to this piece of code? Indeed, the |
The documentation gives an example of using It seems that the workaround can be even more complicated because in my environment a single container belonged to multiple networks. If a solution is not available soon, it might be a good idea to mention the drawbacks of this approach in the documentation. |
@tpaschalis yes, you're right,
|
One way to resolve this would be to give us a way to simply take the exported targets, which is a list of maps, and then be able to create a new list of maps excluding all of the duplicate items. For example, we could iterate over the list of maps based on the Currently the standard library is quite limited in terms of data manipulation functions, but maybe this could be implemented in another way. Allow us to set one of the returned fields as a "filter" where only the first instance is returned and others discarded. |
I had the same issue. In my case was related to dual ipv4 and ipv6 addresses. Checked the positions file and got 2 entries for each container, with the only difference on:
Did a quick test disabling ipv6 off the system, rebooted, and no issues after that. |
I faced that issue too for different reasons (which mostly are covered above):
My hacky solution for my smaller home lab set up: discovery.docker "default" {
host = "unix:///var/run/docker.sock"
}
discovery.relabel "docker" {
targets = discovery.docker.default.targets
+ // Step 1. - generate a unique de-duplication key from common __meta labels
rule {
source_labels = ["__meta_docker_container_name", "__meta_docker_container_network_mode", "__meta_docker_port_public", "__meta_docker_port_public_ip"]
target_label = "__meta_deduplication_key"
}
}
+// Step 2. - keep only those targets where the de-dup key matches a custom label attached to the container
discovery.relabel "container_filter" {
targets = discovery.relabel.docker.output
rule {
action = "keepequal"
source_labels = ["__meta_deduplication_key"]
target_label = "__meta_docker_container_label_deduplication_key_filter"
}
} This first {
__address__ = "172.22.0.2:9001",
+ __meta_deduplication_key = "/minio;minio_minio;9001;0.0.0.0",
__meta_docker_container_name = "/minio",
__meta_docker_container_network_mode = "minio_minio",
...
__meta_docker_network_name = "minio_minio",
__meta_docker_port_private = "9001",
__meta_docker_port_public = "9001",
__meta_docker_port_public_ip = "0.0.0.0"
} Considering such a version: "2"
networks:
minio:
services:
minio:
image: quay.io/minio/minio:RELEASE.2023-12-07T04-16-00Z
+ labels:
+ deduplication-key-filter: "grafana-agent;grafana-ag>
entrypoint: ["sh", "-euc", "minio server /data --console-address ':9001'"]
networks:
minio:
default:
ports:
- "9000:9000"
- "9001:9001" ... we see such {
__address__ = "172.22.0.2:9001",
__meta_deduplication_key = "/minio;minio_minio;9001;0.0.0.0",
+ __meta_docker_container_label_deduplication_key_filter = "/minio;minio_minio;9001;0.0.0.0",
__meta_docker_container_name = "/minio",
__meta_docker_container_network_mode = "minio_minio",
...
__meta_docker_network_name = "minio_minio",
__meta_docker_port_private = "9001",
__meta_docker_port_public = "9001",
__meta_docker_port_public_ip = "0.0.0.0"
} ... which is matching with the A pretty explicit approach as I now have to label all containers accordingly with a proper So or so. This is issue really should see some solution. E.g. some kind of |
This bug is absolutely insane. I have containers without exposed ports, with only one network, and only an IPv4 address. The bug still appears, and the Grafana Agent Flow does not push logs from some containers. Solved it via removing the Docker integration completely (I need only logs). River example:
Also you need to add option to docker logging driver: Btw without Docker integration - logs appears much faster (almost isntantelly). With integration I have delay about 8-12 sec (even if I change interval to 1sec). |
Hey folks, happy new year! 🎉 We've merged #6055 which allows loki.source.docker to automatically deduplicate targets with matching container IDs which should alleviate the problem based on our tests. The fix is out on main, and will be available on the Agent's v0.40 release in about 6 weeks. In the meantime, you can try using one of the latest images in our dockerhub repo, or build your own from main. Let us know how this works for you, and thank you for your patience! |
LGTM (I picked the latest commit on main 6965ae2 for testing) |
What's wrong?
I have a very basic Docker log scraping config with the agent running in flow mode. The logs seem to be transmitted normally, however I get constant error/warning messages from the
grafana-agent
complaining about the inability to read container logs. The error messages are produced and repeated for all targeted containers on a regular basis.Steps to reproduce
Simply run
AGENT_MODE=flow grafana-agent --server.http.listen-addr=0.0.0.0:12345 run /etc/grafana-agent.river
with the provided config on the affected host.Agent is running as user
grafana-agent
who is a member of the following groups:grafana-agent adm systemd-journal docker
System information
Linux 4.4.0-92-generic x86_64
Software version
Grafana Agent v0.34.3
Configuration
Logs
The text was updated successfully, but these errors were encountered: