[Chore] Update Prometheus

[alanprot]

Changes

• Update Prometheus to prometheus/prometheus@efbd6e4

We pinned github.com/sercand/kuberesolver/v4 => github.com/sercand/kuberesolver/v5 v5.1.1 in order to be able to update google.golang.org/grpc needed by opentelemetry-go (and consequently removed the replace for google.golang.org/grpc)

Breaking Changes on Prometheus

• The interface chunkenc.Chunk added a Reset method that needed to be implemented by the structs on thanos.
• Needed to make thanos use the jsoniter due:
  • Fix regression: Returning "null" for empty matrix results in some cases. prometheus/prometheus#13993
  • bugfix: API: encode empty Vector/Matrix as [] not null prometheus/prometheus#13997
• Pinned prometheus/common for now as we need: Expose secret as FileSecret from config package prometheus/common#653

Verification

Breaking Changes on the new go-grpc:

• From here FailOnNonTempDialError, WithBlock, and WithReturnConnectionError are three DialOptions that are only supported by Dial because they only affect the behavior of Dial itself. WithBlock causes Dial to wait until the ClientConn reports its State as connectivity.Connected. The other two deal with returning connection errors before the timeout (WithTimeout or on the context when using DialContext).
  • Those options were added to work around an regression on GRPC: see this and this

[yeya24]

@alanprot OK I just checked and I think the E2E test failure is related. The test tries to spin up a Prometheus locally using Prometheus binary.

I logged the command in my local path.

prometheus --storage.tsdb.retention=2d --storage.tsdb.path=/var/folders/rd/vh0cfrk13jsf8pnnqgz9yrg40000gr/T/prometheus-test3289092244 --web.listen-address=localhost:54196 --web.route-prefix= --web.enable-admin-api --config.file=/var/folders/rd/vh0cfrk13jsf8pnnqgz9yrg40000gr/T/prometheus-test3289092244/prometheus.yml

And it failed with error below.

ts=2024-06-05T06:07:29.433Z caller=main.go:521 level=error msg="Error loading config (--config.file=/var/folders/rd/vh0cfrk13jsf8pnnqgz9yrg40000gr/T/prometheus-test3289092244/prometheus.yml)" file=/var/folders/rd/vh0cfrk13jsf8pnnqgz9yrg40000gr/T/prometheus-test3289092244/prometheus.yml err="parsing YAML file /var/folders/rd/vh0cfrk13jsf8pnnqgz9yrg40000gr/T/prometheus-test3289092244/prometheus.yml: yaml: unmarshal errors:\n  line 2: field rule_query_offset not found in type config.plain"

The config is:

global:
    rule_query_offset: 0s
    external_labels:
        az: "1"
        region: eu-west

rule_query_offset is a new config and it doesn't have omit_empty so it makes old Prometheus version fail. https://github.com/prometheus/prometheus/blob/main/config/config.go#L402.

I think what we can do is, rather than using global config to write the Prometheus config file here, we can write with plain yaml similar to this

[yeya24]

Let's update Prometheus version to prometheus/prometheus#14216 and see if it fixes it?

[alanprot]

Let's update Prometheus version to prometheus/prometheus#14216 and see if it fixes it?

Will do.. but i think there is another braking change that we will need to PR on prometheus.. I will push the change with the new prometheus for now but the build will fail.

prometheus/common#538 (comment)

I pinned the common to work around this issue.

[alanprot]

=== NAME  TestRangeQueryShardingWithRandomData
    testutil.go:91: query_frontend_test.go:667: ""
        
        	exp: model.Matrix(nil)
        
        	got: model.Matrix{}
        
        Diff

Seems related to: prometheus/prometheus#13993

=/
Need to understand why.

[alanprot]

Ok..

We are not using jsoniter to serialize response as the upstream prometheus is... it explains why the null is being serialized as null as the fix for prometheus/prometheus#13993 was actually done prometheus/prometheus#13997 and it was a fix on the json serializer...

I'm changing thanos to use jsoniter as it has multiple benefits anyway.

[yeya24]

LGTM!

[alanprot]

@GiedriusS @fpetkovski Can you guys take a peek on this one?

[fpetkovski]

We are seeing problems with grpc 1.64.0: grpc/grpc-go#7314. The issue in Thanos happens when we try to close stale connections here: https://github.com/thanos-io/thanos/blob/main/pkg/query/endpointset.go#L442.

Maybe it's better to pin grpc to 1.63.2?

[alanprot]

We are seeing problems with grpc 1.64.0: grpc/grpc-go#7314. The issue in Thanos happens when we try to close stale connections here: https://github.com/thanos-io/thanos/blob/main/pkg/query/endpointset.go#L442.

Maybe it's better to pin grpc to 1.63.2?

Ok.. lemme do this.

[yeya24]

Thanks. Merge on green

[yeya24]

LGTM