Retry failed RBE tests and get verbose logs #15456
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
titusfortner
requested review from
shs96c,
AutomatedTester,
jimevans,
p0deje,
cgoldberg,
bonigarcia,
diemol,
pujagani and
nvborisenko
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Explore these optional code suggestions:
|
||||||||||||
cgoldberg
requested changes
Mar 19, 2025
There was a problem hiding this comment.
The Python configuration isn't quite right. You also need to add a logging handler, so I think it should look like:
if os.environ.get("DEBUG"):
logger = logging.getLogger("selenium")
logger.addHandler(logging.StreamHandler())
logger.setLevel(logging.DEBUG)
I updated the docs with similar instructions recently:
https://www.selenium.dev/documentation/webdriver/troubleshooting/logging/
Other than that, it looks fine. It might be interesting to make the log level configurable via environment variable (like LOG_LEVEL=DEBUG or LOG_LEVEL=WARNING), but it's understandable to just want a single DEBUG flag that works across all languages.
nvborisenko
reviewed
Mar 19, 2025
|
|
||
| if (!string.IsNullOrEmpty(enableDebugging)) | ||
| { | ||
| Log.SetLevel(LogEventLevel.Debug); |
There was a problem hiding this comment.
It is already set to Trace level here by default:
Usually I see RBE logs directly in EngFlow:
There was a problem hiding this comment.
Ah, ok. Why isn't it showing more output when the tests are passing?
In general the behavior I'm proposing is for the logging not to be turned on by default for faster execution / cleaner logs, and then allow the user to turn that on when needed (or automatically when failing). But it looks like current .NET is already not logging unnecessarily, so curious how you have it working.
CI Feedback 🧐A test triggered by this PR failed. Here is an AI-generated analysis of the failure:
|
pujagani
reviewed
Mar 24, 2025
| Logger rootLogger = Logger.getLogger(""); | ||
| rootLogger.setLevel(Level.FINE); | ||
| Arrays.stream(rootLogger.getHandlers()) | ||
| .forEach( |
There was a problem hiding this comment.
I don't think it is required since we are not registering any handlers but again having the code is not a problem. Looks fine to me.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
User description
Motivation and Context
Chrome 134 introduced a bunch of weird errors to the RBE run that were not obvious and were not perfectly repeatable. They couldn't be reproduced outside of RBE. I needed to see verbose details to understand what the problem was, and I wanted to be able to compare what good and bad runs look like with that additional logging.
Description
retry_failed_teststask that parses output of a bazel log to determine failed targets, and reruns each with log level set to debuggingDiscussion
Demo
This is what it looks like: https://github.com/SeleniumHQ/selenium/actions/runs/13935331133
There were RBE failures, the 3 failing targets were rerun on both RBE and on Github Actions runner
The logs are available as artifacts to download at the bottom of that page
You can see just how much larger the logs are of just those endpoints with the verbose logging
This is what it looks like with no RBE failures: https://github.com/SeleniumHQ/selenium/actions/runs/13939172170
Types of changes
PR Type
Enhancement, Tests
Description
Introduced environment-based debug logging across Java, .NET, and Python tests.
Enhanced Bazel build scripts and workflows for better logging and retry mechanisms.
Added support for retrying failed tests with detailed logs in CI workflows.
Improved artifact handling in GitHub Actions for better traceability.
Changes walkthrough 📝
7 files
Added DEBUG environment variable for Java loggingEnabled debug logging in .NET tests via environment variableAdded Debug property to TestEnvironment configurationEnhanced Bazel execution to support logging to filesEnabled debug logging in Python tests via environment variableAdded Bazel log file handling and conditional debug tracingAdded task to retry failed tests from Bazel logs3 files
Enhanced artifact handling and added Bazel log supportAdded retry mechanisms for failed RBE and GHA testsUpdated artifact handling for pinned browsers