-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
jaeger-v2 e22 tests are unstable #5418
Comments
Another example, this time from ES:
I wonder if this is due to the fact that when a large batch is sent to the exporter, since our v1 storage implementations only save one span at a time we may have the situation that storage loops through all the 10k spans but the exporter meanwhile times out on the context deadline. It's not completely clear why 'rpc' would be mentioned in this case, however. |
## Which problem is this PR solving? - Helps debugging unstable e2e tests in #5418 ## Description of the changes - Add more visibility to CI workflows by dumping related storage docker logs if the tests failed. - Made changes to Cassandra, Elasticsearch, and Opensearch. ## How was this change tested? - Not tested, unable to test locally. ## Checklist - [x] I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md - [x] I have signed all commits - [ ] I have added unit tests for the new functionality - [x] I have run lint and test steps successfully - for `jaeger`: `make lint test` - for `jaeger-ui`: `yarn lint` and `yarn test` --------- Signed-off-by: James Ryans <james.ryans2012@gmail.com>
## Which problem is this PR solving? - Helps to debug unstable e2e tests in #5418 - Current logs are still hard to follow even it already dump storage's docker and OTEL col binary logs because we can't see which part of the test related to which storage docker/ OTEL col binary logs. ## Description of the changes - Add more logging to e2e tests that capture the timestamp of each read/write run. ## How was this change tested? - Run `STORAGE=grpc SPAN_STORAGE_TYPE=memory make jaeger-v2-storage-integration-test` ## Checklist - [x] I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md - [x] I have signed all commits - [ ] I have added unit tests for the new functionality - [x] I have run lint and test steps successfully - for `jaeger`: `make lint test` - for `jaeger-ui`: `yarn lint` and `yarn test` Signed-off-by: James Ryans <james.ryans2012@gmail.com>
Tests have been good recently, closing. |
@akagami-harsh since merging #5398 I noticed this test failing sporadically. Here's one failed run: https://github.com/jaegertracing/jaeger/actions/runs/8955823698/job/24596965281?pr=5416
There are a couple notable things here:
DeadlineExceeded
gRPC error earlier, probably during writing of the spansI don't know if we have any retries in the writing pipeline, but even if we did the writes are supposed to be idempotent since we derive a primary key from trace/span ID and the content hash. It may be useful to debug this further and perhaps log the difference between written and loaded trace/span IDs (trace ID is supposed to be the same), and perhaps this test needs to be more resilient to duplicates.
The text was updated successfully, but these errors were encountered: