app/vmagent/remotewrite: skip sending empty block to downstream server

[viperstars]

Describe Your Changes

Occasionally, vmagent sends empty blocks to downstream servers. If a downstream server returns an unexpected response, vmagent gets stuck in a retry loop. While vmagent handles 400 and 409 errors, there are various prometheus remote write implementations that return different error codes. For example, vector returns a 422 error. To mitigate the risk of vmagent getting stuck in a retry loop, it is advisable to skip sending empty blocks to downstream servers.

Related logs

2024-04-26T08:21:11.302Z  error  VictoriaMetrics/app/vmagent/remotewrite/client.go:383   unexpected status code received after sending a block with size 0 bytes to "1:secret-url" during retry #1: 422; response body="{\"code\":422,\"message\":\"Failed decompressing payload with snappy decoder.\"}"; re-sending the block in 2.000 seconds
2024-04-26T08:21:11.302Z  error  VictoriaMetrics/app/vmagent/remotewrite/client.go:383   unexpected status code received after sending a block with size 0 bytes to "1:secret-url" during retry #1: 422; response body="{\"code\":422,\"message\":\"Failed decompressing payload with snappy decoder.\"}"; re-sending the block in 2.000 seconds
2024-04-26T08:21:11.303Z  error  VictoriaMetrics/app/vmagent/remotewrite/client.go:383   unexpected status code received after sending a block with size 0 bytes to "1:secret-url" during retry #1: 422; response body="{\"code\":422,\"message\":\"Failed decompressing payload with snappy decoder.\"}"; re-sending the block in 2.000 seconds

Checklist

The following checks are mandatory:

• My change adheres VictoriaMetrics contributing guidelines.

[hagen1778]

Thanks! See my comments below

[viperstars]

I have made some update to fix lint check failure.

[hagen1778]

Thanks for the contribution!

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 55.34%. Comparing base (8aaa828) to head (3b8ca24).
Report is 696 commits behind head on master.

❗ Current head 3b8ca24 differs from pull request most recent head a784711

Please upload reports for the commit a784711 to get more accurate results.

Files	Patch %	Lines
app/vmagent/remotewrite/client.go	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6241      +/-   ##
==========================================
- Coverage   60.37%   55.34%   -5.04%     
==========================================
  Files         411      586     +175     
  Lines       76609    77970    +1361     
==========================================
- Hits        46253    43150    -3103     
- Misses      27794    31839    +4045     
- Partials     2562     2981     +419     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.