Thanos generates a lot of warnings when Prometheus creates empty chunks

[jrRibeiro]

Versions used: Thanos v0.34.1 - Prometheus v2.46.0

What happened:

I noticed thousands of log lines (one per empty chunk) appearing in our logging system, e.g:

caller=streamed_block_writer.go:116 level=warn msg="empty chunks happened, skip series" series="{__name__='metric_name, <list_of_labels>}"

Stale data on the Prometheus side often generates these chunks, which are then ignored when writing the series on the Thanos side during a compaction:

thanos/pkg/compact/downsample/streamed_block_writer.go

Lines 115 to 118 in a97a6ff

	if len(chunks) == 0 {
	level.Warn(w.logger).Log("msg", "empty chunks happened, skip series", "series", strings.ReplaceAll(lset.String(), "\"", "'"))
	return nil
	}

What you expected to happen:

A message per each series is logged as a warning everytime empty chunks appear. Given this might generate a lot of logs (one per series) in larger environments and it's not a concern on the Thanos side, should the log level be reduced to debug?