feat: added console bidi events #450
Conversation
|
Warning Rate limit exceeded@nirtal85 has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 21 minutes and 35 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (1)
WalkthroughThe changes update the test configuration in Changes
Sequence Diagram(s)sequenceDiagram
participant T as Test Runner
participant S as pytest_runtest_setup
participant B as Firefox Browser
participant E as pytest_exception_interact
participant A as Allure Report
T->>S: Begin test setup
S->>B: Launch Firefox with firefox_options (Bidi enabled)
B-->>S: Stream console messages & JS errors
S-->>S: Store logs in global variables
T->>E: Test error occurs
E->>A: Attach console messages & JS errors (if browser is not "remote")
Suggested reviewers
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
tests/conftest.py(4 hunks)
🔇 Additional comments (3)
tests/conftest.py (3)
187-189: Good implementation of Bidi protocol for Firefox.Enabling Bidirectional protocol for Firefox matches the Chrome implementation at line 161, providing consistent browser communication capabilities across both browsers. This is a good practice for test stability.
222-226: LGTM! Well-structured initialization of console event handlers.Good approach to initialize the console message and JavaScript error handlers conditionally when not using remote browsers. Using
appendmethod provides a clean way to collect logs during test execution.
346-364:❓ Verification inconclusive
Verify that console messages are captured correctly.
This implementation replaces the previous browser logs collection with more specific console and JavaScript error logging. Make sure all important information from the previous implementation is still being captured.
🌐 Web query:
What's the difference between Selenium's browser logs and the new WebDriver BiDi console messages?💡 Result:
Selenium's traditional browser logs and WebDriver BiDi's console message handling differ fundamentally in their communication models, real-time capabilities, and integration with modern browser automation needs. Here's a breakdown of key differences:
Communication Architecture
Browser Logs (Classic):
Relied on unidirectional HTTP requests where clients had to manually poll for log data[5][6]. This created latency and made real-time monitoring impractical.WebDriver BiDi:
Uses WebSockets for bidirectional communication[2][5], allowing browsers to push events like console messages directly to test scripts as they occur.Real-Time Capabilities
Browser Logs:
Required post-facto retrieval after page loads/actions[1][4], making it impossible to react to errors during test execution.WebDriver BiDi:
Enables immediate response to events through handlers like:driver.script.add_console_message_handler(log_entries.append) # [8][10]This allows assertions on live console output during test execution[1][4].
Event Granularity
Browser Logs:
Typically provided aggregated log data without context about specific user actions triggering them[6].WebDriver BiDi:
Captures rich metadata including:
- Message types (log/warn/error)[1]
- Timestamps[8]
- Source line numbers[8]
- Associated browsing contexts[9]
Integration with Testing Workflows
Browser Logs:
Required separate log collection mechanisms that weren't natively integrated with test assertions[6].WebDriver BiDi:
Directly integrates with test scripts through event listeners:page.on('console', (message) => { console.log(`[${message.type()}] ${message.text()}`); // [1][3] });Enables real-time validation of application behavior during complex user flows[2][10].
Cross-Browser Consistency
Browser Logs:
Implementation varied significantly between browsers due to lack of standardization[3][5].WebDriver BiDi:
W3C-standardized protocol ensures consistent console message handling across Chrome, Firefox, and Edge[3][9].The shift to WebDriver BiDi represents a paradigm change from passive log collection to active event monitoring, particularly valuable for modern web apps with dynamic content and client-side error handling. While traditional logs remain useful for post-mortem analysis, BiDi's real-time capabilities enable proactive error detection during test execution[1][3][6].
Citations:
- 1: https://developer.chrome.com/blog/webdriver-bidi-2023
- 2: https://automatedtester.hashnode.dev/using-selenium-4-newest-webdriver-bidi
- 3: https://www.lambdatest.com/blog/webdriver-bidi-future-of-browser-automation/
- 4: https://saucelabs.com/resources/blog/bidirectional-apis
- 5: https://codoid.com/selenium-testing/an-introductory-guide-to-the-webdriver-bidi-protocol/
- 6: https://www.browserstack.com/guide/selenium-4-features
- 7: https://www.sitespeed.io/documentation/sitespeed.io/scripting/tutorial-12-Bidi.html
- 8: https://www.selenium.dev/documentation/webdriver/bidi/logging/
- 9: https://www.w3.org/TR/webdriver-bidi/
- 10: https://www.browserstack.com/docs/automate/selenium/bidi-event-driven-testing
Ensure Comprehensive Validation of Console & JavaScript Error Logging
The updated logging strategy in
tests/conftest.py(lines 346-364) now leverages WebDriver BiDi to capture console messages and JavaScript errors. As confirmed by the web query, this modern approach allows for real-time and metadata-rich logging compared to the traditional Selenium browser logs—which were based on HTTP polling and often lacked detailed context (e.g., timestamps, log levels, source information).
- Confirm that all essential log details previously available via browser logs are either captured by the new event-driven mechanism or are no longer required.
- Verify that the log collections (both for console messages and JavaScript errors) are properly formatted and attached to Allure, ensuring that critical debugging information isn’t inadvertently lost.
- Consider running tests under various scenarios to ensure that the real-time logging accurately reflects the necessary details for troubleshooting.
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
tests/conftest.py (2)
137-137: Consider reducing the use of global variables.Adding global variables for
console_messagesandjavascript_errorsworks, but global state can make testing and debugging harder. Consider refactoring to use a class or context manager to manage this state.
222-226: Add a comment explaining why remote browsers are excluded.The code correctly sets up handlers for console messages and JavaScript errors, but it would be helpful to add a brief comment explaining why remote browsers are excluded from this functionality.
if browser != "remote": + # Bidi protocol handlers not compatible with remote WebDriver console_messages = [] driver.script.add_console_message_handler(console_messages.append) javascript_errors = [] driver.script.add_javascript_error_handler(javascript_errors.append)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
tests/conftest.py(4 hunks)
🔇 Additional comments (2)
tests/conftest.py (2)
187-189: LGTM! Bidi support for Firefox properly enabled.The implementation correctly enables the Bidirectional protocol for Firefox, which is needed to capture console messages and JavaScript errors.
349-363: Properly implemented console and error logging.The code correctly formats and attaches console messages and JavaScript errors to the Allure report. The previous issue with JavaScript error attachment has been fixed.
I notice that you've implemented the suggested fix from the previous review by correctly joining and converting the JavaScript errors to strings before attaching them to the report.
Description
Motivation and Context
How Has This Been Tested?
Screenshots (if appropriate):
Types of changes
Checklist:
Summary by CodeRabbit