vmagent 1.90.0 couldn't send a block with size

[ihard]

Describe the bug

On an absolutely unloaded cluster of version 1.90.0, periodically (once every 30 minutes approximately) I see in the agent logs
2023-04-17T17:11:29.482+0700 warn VictoriaMetrics/app/vmagent/remotewrite/client.go:370 couldn't send a block with size 203 bytes to "7:secret-url": Post "http:// 127.0.0.1:8581/insert/multitenant/prometheus": EOF; re-sending the block in 2.000 seconds
local vmagent sends traffic to vminsert via 127.0.0.1 and gets these errors (seems like things could go wrong here)
launched tcpdump, I see that at the time of the error, RST arrives from the vminsert port

To Reproduce

run Vmagent and VMcluster 1.90.0 on same virtual machine

Version

vmagent:v1.90.0
vminsert:v1.90.0-cluster

Logs

No response

Screenshots

Used command-line flags

  - '--promscrape.config=/etc/vmagent/vmagent.yml'
  - '--graphiteListenAddr=:3003'
  - '--httpListenAddr=:8429'
  - '--remoteWrite.maxDiskUsagePerURL=10GB'
  - '--maxConcurrentInserts=100000'
  - '--insert.maxQueueDuration=60s'
  - '--remoteWrite.url=http://127.0.0.1:8581/insert/multitenant/prometheus'
  - '--remoteWrite.urlRelabelConfig=/etc/vmagent/relabel/prometheus-60s.yml'
  - '--remoteWrite.url=http://127.0.0.1:8581/insert/multitenant/prometheus'
  - '--remoteWrite.urlRelabelConfig=/etc/vmagent/relabel/prometheus-30s.yml'
  - '--remoteWrite.url=http://127.0.0.1:8581/insert/multitenant/prometheus'
  - '--remoteWrite.urlRelabelConfig=/etc/vmagent/relabel/prometheus-10s.yml'
  - '--remoteWrite.url=http://127.0.0.1:8591/insert/multitenant/prometheus'
  - '--remoteWrite.urlRelabelConfig=/etc/vmagent/relabel/prometheus-60s.yml'
  - '--remoteWrite.url=http://127.0.0.1:8591/insert/multitenant/prometheus'
  - '--remoteWrite.urlRelabelConfig=/etc/vmagent/relabel/prometheus-30s.yml'
  - '--remoteWrite.url=http://127.0.0.1:8591/insert/multitenant/prometheus'
  - '--remoteWrite.urlRelabelConfig=/etc/vmagent/relabel/prometheus-10s.yml'
  - '--remoteWrite.url=http://127.0.0.1:8581/insert/multitenant/prometheus'
  - '--remoteWrite.urlRelabelConfig=/etc/vmagent/relabel/graphite-60s.yml'
  - '--remoteWrite.url=http://127.0.0.1:8581/insert/multitenant/prometheus'
  - '--remoteWrite.urlRelabelConfig=/etc/vmagent/relabel/graphite-30s.yml'
  - '--remoteWrite.url=http://127.0.0.1:8581/insert/multitenant/prometheus'
  - '--remoteWrite.urlRelabelConfig=/etc/vmagent/relabel/graphite-10s.yml'
  - '--remoteWrite.url=http://127.0.0.1:8591/insert/multitenant/prometheus'
  - '--remoteWrite.urlRelabelConfig=/etc/vmagent/relabel/graphite-60s.yml'
  - '--remoteWrite.url=http://127.0.0.1:8591/insert/multitenant/prometheus'
  - '--remoteWrite.urlRelabelConfig=/etc/vmagent/relabel/graphite-30s.yml'
  - '--remoteWrite.url=http://127.0.0.1:8591/insert/multitenant/prometheus'
  - '--remoteWrite.urlRelabelConfig=/etc/vmagent/relabel/graphite-10s.yml'

Additional information

No response

[Amper]

Hi @ihard.
Are there any errors in the vminsert log?
What command-line flags are used in vminsert?

[ihard]

vminsert flag

      - '--httpListenAddr=:8581'
      - '--storageNode=10.111.234.4:8682'
      - '--storageNode=10.111.224.4:8682'
      - '--storageNode=10.111.225.4:8682'
      - '--replicationFactor=2'

vminsert log is null

[IvanZenger]

Is there any news here? We have the same problem.

The error comes about every 30 minutes couldn't send a block with size 152176 bytes to "2:secret-url": Post "https://vminsert.example.com:8581/api/v1/write": EOF.

We do not have another error on vmagent,vminsert or vmstorage.

vmagent has following configuration:

vminsert has following configuration:

Dashboards:

[wjordan]

This kind of error occurs when the HTTP client attempts to reuse an idle connection that's already been closed on the server. This can happen if the client's idle-connection timeout is greater than or equal to the timeout on the server, and both vmagent's client IdleConnTimeout and vminsert's server IdleTimeout are set to 1 minute by default:

VictoriaMetrics/app/vmagent/remotewrite/client.go

Line 121 in 2e494e2

IdleConnTimeout: time.Minute,

VictoriaMetrics/lib/httpserver/httpserver.go

Line 122 in 2e494e2

IdleTimeout: *idleConnTimeout,

VictoriaMetrics/lib/httpserver/httpserver.go

Line 52 in 2e494e2

idleConnTimeout = flag.Duration("http.idleConnTimeout", time.Minute, "Timeout for incoming idle http connections")

I can think of two possible solutions:

1. The underlying Go issue is discussed in net/http: Client returns errors on POST if keep-alive connection closes at unfortunate time golang/go#22158, where a workaround was implemented that allows automatically retrying POST requests tagged with a special X-Idempotency-Key header.
   A PR could add this header for VictoriaMetrics client requests that are known to be idempotent. (I'm pretty sure remote-write requests are idempotent thanks to deduplication logic?)
   • See: set X-Idempotency-Key header on VMAgent client requests #4452
2. Setting a slightly higher vminsert -http.idleConnTimeout config than the hard-coded 1m vmagent-client timeout (e.g., -http.idleConnTimeout 1m5s) might also help prevent this error, since idle connections would then be more likely to get closed by the client before the server.

[hagen1778]

As @wjordan mentioned, such errors could happen when connection timeouts on client and server doesn't match. However, this doesn't result in data loss as vmagent will retry the write attempt with a newly established connection.

[valyala]

The commit 992a1c0 should instruct vmagent immediately retry requests to remote storage on closed idle connection, without logging the error. This should remove the couldn't send a block with size ... bytes to "...": Post "...": EOF error logs. This change can be tested by building and running vmagent from this commit according to these docs. If everything is OK, this commit will be included in the next release.

[valyala]

Starting from v1.94.0, vmagent shouldn't emit couldn't send a block error logs on the first unsuccessful attempt to send data to remote storage. It will retry sending the block immediately over a new connection to the remote storage in this case. Closing the issue as resolved then.