Skip to content

Comments

Bump the limit of traces we see in e2e tests#8247

Merged
nrfox merged 5 commits intokiali:masterfrom
nrfox:fix-traces-flake
Mar 26, 2025
Merged

Bump the limit of traces we see in e2e tests#8247
nrfox merged 5 commits intokiali:masterfrom
nrfox:fix-traces-flake

Conversation

@nrfox
Copy link
Contributor

@nrfox nrfox commented Mar 18, 2025

Describe the change

Attempts to fix tracing flakes like this one: https://github.com/kiali/kiali/actions/runs/13909420482/job/38920315175#step:8:766

Steps to test the PR

N/A

Automation testing

Included

@nrfox nrfox requested a review from josunect March 18, 2025 00:04
Copy link
Contributor

@josunect josunect left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix, @nrfox ! It looks good.

Should we also fix this issue? https://github.com/kiali/kiali/actions/runs/13912310385/job/38929093053?pr=8247 It is not related but it is also related to tracing tests.

I think this https://github.com/kiali/kiali/blob/master/frontend/cypress/integration/featureFiles/app_details.feature#L48 should be details, probably, as we see in the image, maybe not all the spans are loaded yet, so productpage app is not found:

image

This is not happening with Jaeger, because, for what I understand, it is related to how the Tempo backend works, as results are indexed and might not be available at the time of the search.

@nrfox
Copy link
Contributor Author

nrfox commented Mar 18, 2025

@josunect thanks for taking a look.

This is not happening with Jaeger, because, for what I understand, it is related to how the Tempo backend works, as results are indexed and might not be available at the time of the search.

I assumed that what was happening here was that we are only seeing a portion of the overall traces because of the limit so we might be missing traces from certain apps altogether? Would adjusting the duration down to 1m help?

@jshaughn
Copy link
Collaborator

@leandroberetta Is working on a new test app that provides predictable logs for our cypress tests. Would it be useful to ry and also have it generate predictable traces?

@nrfox
Copy link
Contributor Author

nrfox commented Mar 18, 2025

@leandroberetta Is working on a new test app that provides predictable logs for our cypress tests. Would it be useful to ry and also have it generate predictable traces?

I think what makes the traces unpredictable is the tracing backend and not the app.

@josunect
Copy link
Contributor

@leandroberetta Is working on a new test app that provides predictable logs for our cypress tests. Would it be useful to ry and also have it generate predictable traces?

I think what makes the traces unpredictable is the tracing backend and not the app.

Yes, I think that's the issue. Unless we select a custom time period (For example, getting from the last 10 to the last 5 minutes), I think it would be more likely to have all the results indexed.

@jshaughn
Copy link
Collaborator

I think what makes the traces unpredictable is the tracing backend and not the app.

Yes, I think that's the issue. Unless we select a custom time period (For example, getting from the last 10 to the last 5 minutes), I think it would be more likely to have all the results indexed.

I get that the indexing time is probably the issue. I didn't know whether it would be useful to generate something simple and consistent that could be sampled at 100%, or something. But, it doesn't sound like it would be helpful.

@nrfox
Copy link
Contributor Author

nrfox commented Mar 25, 2025

@josunect I'm going to update this PR to instead pick a trace that is not in progress, one that has all of it's spans. I think that's the more direct cause for the flakes. I'll open a separate issue for the behavior of selecting an incomplete trace.

@nrfox
Copy link
Contributor Author

nrfox commented Mar 26, 2025

@josunect hopefully this run passes if you want to take another look

@josunect josunect self-requested a review March 26, 2025 07:50
Copy link
Contributor

@josunect josunect left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @nrfox ! It looks good

@nrfox nrfox merged commit 09e98c3 into kiali:master Mar 26, 2025
10 checks passed
@nrfox nrfox deleted the fix-traces-flake branch March 26, 2025 09:45
nrfox added a commit to nrfox/kiali that referenced this pull request May 5, 2025
@jshaughn jshaughn moved this from 📋 Backlog to ✅ Done in Kiali Sprint 26-03 | Kiali v2.23 May 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

3 participants