Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update chromium logger to identify subtest failures. #21763

Conversation

@LukeZielinski
Copy link
Contributor

LukeZielinski commented Feb 12, 2020

This also fixes a bug where a subtest failure would be hidden because
the expected_status was evaluated too late in test_end. For a
subtest failure the test status is Pass. But for Chromium, we flip this
"actual" status to a failure.
But the actual status is also used as a fall-back for the expected status
(see explanation in chromium.py:126). So if expected is evaluated after the
"flip", it will just match actual and the failure will be marked as
an "expected" failure.

This also fixes a bug where a subtest failure would be hidden because
the expected_status was evaluated too late in `test_end`. For a
subtest failurei the test status is Pass. But for Chromium, we flip this
"actual" status to a failure.
But the actual status is also used as a fall-back for the expected status
(see explanation in chromium.py:126). So if "expected" is evaluated after the
"flip", it will just match "actual" and the failure will be marked as
an "expected" failure.
@LukeZielinski
Copy link
Contributor Author

LukeZielinski commented Feb 12, 2020

@KyleJu FYI, this was carved off from crrev.com/c/2025694 so it's seen some exercise already.

@KyleJu
Copy link

KyleJu commented Feb 13, 2020

@LukeZielinski I see one test failure in TaskCluster, ::Sample HTML5 API Tests <- tests/functional/api-tests-1.html FAILED [ 15%]. Not sure if its related?

@LukeZielinski
Copy link
Contributor Author

LukeZielinski commented Feb 13, 2020

Doesn't seem related. Having trouble running those tests locally, but lets retry here.

@LukeZielinski
Copy link
Contributor Author

LukeZielinski commented Feb 14, 2020

Azure failure seems unrelated, retrying.

@KyleJu
KyleJu approved these changes Feb 14, 2020
Copy link

KyleJu left a comment

LGTM

@LukeZielinski LukeZielinski merged commit 22a7994 into web-platform-tests:master Feb 14, 2020
35 checks passed
35 checks passed
build-and-publish
Details
build-and-publish
Details
build-and-publish
Details
build-and-tag
Details
build-and-tag
Details
build-and-tag
Details
Azure Pipelines Build #20200214.29 succeeded
Details
Azure Pipelines (./wpt test-jobs) ./wpt test-jobs succeeded
Details
Azure Pipelines (affected tests without changes: Safari Technology Preview) affected tests without changes: Safari Technology Preview succeeded
Details
Azure Pipelines (affected tests: Safari Technology Preview) affected tests: Safari Technology Preview succeeded
Details
Azure Pipelines (infrastructure/ tests: macOS) infrastructure/ tests: macOS succeeded
Details
Azure Pipelines (tools/ unittests: Windows (Python 3.6)) tools/ unittests: Windows (Python 3.6) succeeded
Details
Azure Pipelines (tools/ unittests: Windows (Python 3.8)) tools/ unittests: Windows (Python 3.8) succeeded
Details
Azure Pipelines (tools/ unittests: Windows) tools/ unittests: Windows succeeded
Details
Azure Pipelines (tools/ unittests: macOS (Python 3.6)) tools/ unittests: macOS (Python 3.6) succeeded
Details
Azure Pipelines (tools/ unittests: macOS (Python 3.8)) tools/ unittests: macOS (Python 3.8) succeeded
Details
Azure Pipelines (tools/ unittests: macOS) tools/ unittests: macOS succeeded
Details
Azure Pipelines (tools/wpt/ tests: Windows (Python 3.6)) tools/wpt/ tests: Windows (Python 3.6) succeeded
Details
Azure Pipelines (tools/wpt/ tests: Windows (Python 3.8)) tools/wpt/ tests: Windows (Python 3.8) succeeded
Details
Azure Pipelines (tools/wpt/ tests: Windows) tools/wpt/ tests: Windows succeeded
Details
Azure Pipelines (tools/wpt/ tests: macOS (Python 3.6)) tools/wpt/ tests: macOS (Python 3.6) succeeded
Details
Azure Pipelines (tools/wpt/ tests: macOS (Python 3.8)) tools/wpt/ tests: macOS (Python 3.8) succeeded
Details
Azure Pipelines (tools/wpt/ tests: macOS) tools/wpt/ tests: macOS succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: Windows (Python 3.6)) tools/wptrunner/ unittests: Windows (Python 3.6) succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: Windows (Python 3.8)) tools/wptrunner/ unittests: Windows (Python 3.8) succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: Windows) tools/wptrunner/ unittests: Windows succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: macOS (Python 3.6)) tools/wptrunner/ unittests: macOS (Python 3.6) succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: macOS (Python 3.8)) tools/wptrunner/ unittests: macOS (Python 3.8) succeeded
Details
Azure Pipelines (tools/wptrunner/ unittests: macOS) tools/wptrunner/ unittests: macOS succeeded
Details
Azure Pipelines (wpt.fyi hook: safari-preview-affected-tests) wpt.fyi hook: safari-preview-affected-tests succeeded
Details
Azure Pipelines (wpt.fyi hook: safari-preview-affected-tests-without-changes) wpt.fyi hook: safari-preview-affected-tests-without-changes succeeded
Details
Community-TC (pull_request) TaskGroup: success
Details
wpt.fyi - chrome[experimental] Chrome results
Details
wpt.fyi - firefox[experimental] Firefox results
Details
wpt.fyi - safari[experimental] Safari results
Details
@LukeZielinski LukeZielinski deleted the LukeZielinski:chromium-logger-subtest-fails branch Feb 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.