Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thanos with duplicate results more than 5 minutes #18422

Closed
antikilahdjs opened this issue Aug 14, 2023 · 2 comments
Closed

Thanos with duplicate results more than 5 minutes #18422

antikilahdjs opened this issue Aug 14, 2023 · 2 comments
Assignees
Labels
solved tech-issues The user has a technical issue about an application thanos triage Triage is needed

Comments

@antikilahdjs
Copy link

antikilahdjs commented Aug 14, 2023

Name and Version

bitnami/thanos:v0.31.0

What architecture are you using?

amd64

What steps will reproduce the bug?

1.Install the helm using the normal way
2.Configure S3 miniO as bucket
3.I have the Prometheus CRD and I configured the Thanos Sidecar
4.Everything works perfectly and the remote_write has been configured to sent all data to S3

Are you using any custom parameters or values?

        query:
        ## @param query.enabled Set to true to enable Thanos Query component
        ##
        enabled: true
        ## @param query.logLevel Thanos Query log level
        ##
        logLevel: info
        ## @param query.logFormat Thanos Query log format
        ##
        logFormat: logfmt
        ## @param query.replicaLabel Replica indicator(s) along which data is de-duplicated
        ##
        replicaLabel: [prometheus_replica]
        ## Dynamically configure store APIs using DNS discovery
        ## @param query.dnsDiscovery.enabled Enable store APIs discovery via DNS
        ## @param query.dnsDiscovery.sidecarsService Sidecars service name to discover them using DNS discovery
        ## @param query.dnsDiscovery.sidecarsNamespace Sidecars namespace to discover them using DNS discovery
        ##
        dnsDiscovery:
          enabled: true
          sidecarsService: "prometheus-k8s"
          sidecarsNamespace: "kubesphere-monitoring-system"

What is the expected behavior?

If I search a query before 5 minutes I got the correct result (dedup)

image

What do you see instead?

I have 2 Prometheus as operator replicas with Thanos then send data to my storage like miniO.

If searching some query for more than 1 or 2 hours the result is twice duplicated like below

image

It brings many results and the dudup is not working or I did something wrong.

Additional information

I have the Thanos sidecar enabled in my Prometheus CRD and RemoteWrite to Storegateway.

Please help me to identify it or if it is some bug in the version that I am using.

My Prometheus wasn't configured with the external labe then the default label is prometheus_replica. I don't know if I put some unnecessary values into the helm

image

Thanks

@antikilahdjs antikilahdjs added the tech-issues The user has a technical issue about an application label Aug 14, 2023
@bitnami-bot bitnami-bot added this to Triage in Support Aug 14, 2023
@github-actions github-actions bot added the triage Triage is needed label Aug 14, 2023
@carrodher
Copy link
Member

It appears that the issue may not be directly related to the Bitnami Thanos container image or Helm chart, but rather to how the application is being utilized or configured in your specific environment.

This seems to be a unique use case that might be challenging to replicate on our end, as it appears closely tied to your particular scenario.

For inquiries about the application itself, customizing its content, or questions about technology and infrastructure usage, we highly recommend referring to the forums and user guides provided by the project responsible for the application or technology.

With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.

@github-actions github-actions bot moved this from Triage to Pending in Support Aug 16, 2023
@antikilahdjs
Copy link
Author

antikilahdjs commented Aug 17, 2023

@carrodher thank you so much and the issue was the thanos version in 0.31 now the Team worked and released the 0.32 rc main and was fixed. I believe in some days you guys from Bitnami will update it too.
The reference thanos-io/thanos#6257

Thank you

@bitnami-bot bitnami-bot moved this from Pending to Triage in Support Aug 17, 2023
@bitnami-bot bitnami-bot moved this from Triage to Solved in Support Aug 17, 2023
@github-actions github-actions bot removed this from Solved in Support Aug 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
solved tech-issues The user has a technical issue about an application thanos triage Triage is needed
Projects
None yet
Development

No branches or pull requests

2 participants