operation was canceled during DNS lookup with remote_write

[errordeveloper]

What did you do?

With Prometheus v2.0.0 on Kubernetes I've set global: { scrape_inteval: 5s } and enabled remote_write and remote_read, with scrape_confgs previosly used with v1.7.1.

What did you expect to see?

What did you see instead? Under which circumstances?

Environment

• System information: GKE

• Prometheus version: prometheus/prom:v2.0.0

• Prometheus configuration file:

        global:
          scrape_interval: 5s
        remote_write:
          - url: 'https://cloud.weave.works/api/prom/push'
            basic_auth:
              password: <secret>
        remote_read:
          - url: 'https://cloud.weave.works/api/prom/read'
            basic_auth:
              password: <secret>
        scrape_configs: <ommited>

• Logs:

I see the following line repeated very frequently.

evel=warn ts=2017-11-24T12:41:28.49545111Z caller=queue_manager.go:485 component=remote msg="Error sending samples to remote storage" count=100 err="Post https://cloud.weave.works/api/prom/push: dial tcp: lookup cloud.weave.works on 10.59.240.10:53: dial udp 10.59.240.10:53: operation was canceled"

[bboreham]

I see that Prometheus 2.0 has set MaxIdleConns; however Go has rather unexpected behaviour when you open several sockets to the same address, as may well happen with scraping: it runs around closing sockets and I believe this behaviour results in the "operation was canceled" message too.

So I would recommend to add a line setting MaxIdleConnsPerHost to something higher, maybe 100?

[errordeveloper]

@bboreham as per our discussion offline, I've made this change and tested it, can confirm it solves the issue (see errordeveloper@1f74477). I'm happy to open a PR and we can take it from there.

[bboreham]

FWIW I now believe the underlying issue is golang/go#22724, fixed in Go 1.10.1, but it was valid to make the problem go away by raising the number of open connections.

[brian-brazil]

We'll pick that up in the next release then, as that'll be with that version of Go.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.