Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do a bit of test gardening of unstable results #29399

Merged
merged 1 commit into from Feb 22, 2023
Merged

Do a bit of test gardening of unstable results #29399

merged 1 commit into from Feb 22, 2023

Conversation

mrobinson
Copy link
Member

This change updates results that are reliably flaky on the bots and also adds bug references for previously updated results. The bug annotations here are the same that are used for Gecko.

  • ./mach build -d does not report any errors
  • ./mach test-tidy does not report any errors
  • These changes do not require tests because they simply update expectations.

@mrobinson
Do a bit of test gardening of unstable results
280ebaf 
This change updates results that are reliably flaky on the bots and also
adds bug references for previously updated results. The bug annotations
here are the same that are used for Gecko.
@mrobinson
Copy link
Member Author

@bors-servo try

@bors-servo
Copy link
Contributor

⌛ Trying commit 280ebaf with merge 67318b1...

bors-servo added a commit that referenced this pull request Feb 22, 2023
@bors-servo
Auto merge of #29399 - mrobinson:gardening-02-22, r=<try>
67318b1 
Do a bit of test gardening of unstable results

This change updates results that are reliably flaky on the bots and also adds bug references for previously updated results. The bug annotations here are the same that are used for Gecko.

<!-- Please describe your changes on the following line: -->

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes do not require tests because they simply update expectations.

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
@github-actions
Copy link

Results from try job (#4240560291):

Flaky unexpected result (21)
  • OK /css/CSS2/normal-flow/block-in-inline-hittest-002.html (#29057)
    • FAIL [expected PASS] subtest: elementFromPoint
  • TIMEOUT [expected PASS] /css/css-text-decor/text-underline-offset-overline.html
  • OK /css/css-transitions/properties-value-inherit-002.html (#21486)
    • FAIL [expected PASS] subtest: color color(rgba) / values
    • FAIL [expected PASS] subtest: font-size length(pt) / values
    • FAIL [expected PASS] subtest: font-size length(pc) / values
    • FAIL [expected PASS] subtest: font-size length(px) / values
    • FAIL [expected PASS] subtest: font-size length(em) / values
    • FAIL [expected PASS] subtest: font-size length(ex) / values
    • FAIL [expected PASS] subtest: font-size length(mm) / values
    • FAIL [expected PASS] subtest: font-size length(cm) / values
    • FAIL [expected PASS] subtest: font-size length(in) / values
    • FAIL [expected PASS] subtest: font-size percentage(%) / values
    • And 40 more unexpected results...
  • OK /css/cssom-view/CaretPosition-001.html (#21338)
    • PASS [expected FAIL] subtest: Element at (400, 100)
  • OK /css/cssom-view/MediaQueryList-extends-EventTarget.html (#25269)
    • FAIL [expected PASS] subtest: listeners for "change" type are called
  • OK /css/cssom-view/elementFromPoint-float-in-relative.html (#27753)
    • FAIL [expected PASS] subtest: elementFromPoint-float-in-relative
  • OK /css/cssom-view/elementFromPoint-float-in-table.html (#27002)
    • FAIL [expected PASS] subtest: float-in-div
  • OK /css/cssom-view/elementFromPoint-visibility-hidden-resizer.html (#28695)
    • PASS [expected FAIL] subtest: elementFromPoint on resizer area of an element with visibility:hidden
  • TIMEOUT [expected OK] /dom/events/webkit-transition-end-event.html (#26497)
    • TIMEOUT [expected FAIL] subtest: webkitTransitionEnd event listener should trigger for an animation
    • NOTRUN [expected PASS] subtest: webkitTransitionEnd event listener should not trigger if an unprefixed listener also exists
    • NOTRUN [expected PASS] subtest: webkitTransitionEnd event listener should not trigger if an unprefixed event handler also exists
    • NOTRUN [expected FAIL] subtest: event types for prefixed and unprefixed transitionend event listeners should be named appropriately
    • NOTRUN [expected PASS] subtest: webkitTransitionEnd event listener is case sensitive
  • OK /html/browsers/browsing-the-web/navigating-across-documents/initial-empty-document/load-pageshow-events-iframe-contentWindow.html (#28681)
    • PASS [expected FAIL] subtest: load & pageshow events do not fire on contentWindow of <iframe> element created with src='about:blank'
  • OK [expected TIMEOUT] /html/browsers/browsing-the-web/navigating-across-documents/javascript-url-referrer.window.html (#29081)
    • PASS [expected TIMEOUT] subtest: no-referrer referrer policy used to create the starting page
  • OK /html/browsers/browsing-the-web/navigating-across-documents/javascript-url-security-check-same-origin-domain.sub.html (#29122)
    • PASS [expected FAIL] subtest: javascript: URL security check for same-origin-domain but not same-origin
  • OK /html/browsers/browsing-the-web/navigating-across-documents/navigation-unload-same-origin.window.html (#29049)
    • PASS [expected FAIL] subtest: Same-origin navigation started from unload handler must be ignored
  • TIMEOUT [expected OK] /html/browsers/history/the-history-interface/traverse-during-unload.html (#28688)
    • TIMEOUT [expected PASS] subtest: Traversing the history during unload
  • OK /html/browsers/the-window-object/open-close/creating_browsing_context_test_01.html (#29046)
    • FAIL [expected PASS] subtest: first argument: absolute url
  • TIMEOUT [expected OK] /html/interaction/focus/the-autofocus-attribute/update-the-rendering.html (#24145)
    • TIMEOUT [expected FAIL] subtest: "Flush autofocus candidates" should be happen before a scroll event and animation frame callbacks
  • OK [expected CRASH] /html/semantics/embedded-content/the-iframe-element/iframe_sandbox_popups_nonescaping-2.html (#22154)
  • TIMEOUT [expected OK] /html/webappapis/scripting/processing-model-2/integration-with-the-javascript-job-queue/promise-job-entry-different-function-realm.html (#25805)
    • TIMEOUT [expected FAIL] subtest: Fulfillment handler on pending-then-fulfilled promise
    • TIMEOUT [expected FAIL] subtest: Rejection handler on pending-then-rejected promise
  • OK [expected TIMEOUT] /html/webappapis/scripting/processing-model-2/unhandled-promise-rejections/promise-rejection-events.html (#26371)
    • FAIL [expected TIMEOUT] subtest: delayed handling: delaying handling rejected promise created from createImageBitmap will cause both events to fire
  • ERROR [expected OK] /resource-timing/cors-preflight.any.html (#28694)
    • TIMEOUT [expected FAIL] subtest: PerformanceResourceTiming sizes fetch with preflight test
  • TIMEOUT [expected OK] /webmessaging/with-ports/018.html (#24485)
    • TIMEOUT [expected PASS] subtest: origin of the script that invoked the method, javascript:
Stable unexpected results that are known to be intermittent (26)
  • TIMEOUT /FileAPI/url/url-in-tags-revoke.window.html (#19978)
    • FAIL [expected TIMEOUT] subtest: Opening a blob URL in a new window immediately before revoking it works.
  • OK /_mozilla/css/stylesheet_media_queries.html (#17159)
    • FAIL [expected PASS] subtest: Media queries within stylesheets
  • OK /_mozilla/mozilla/task_queue_throttling.any.html (#22519)
    • PASS [expected FAIL] subtest: Throttling the performance timeline task queue.
  • TIMEOUT [expected OK] /_webgl/conformance/uniforms/out-of-bounds-uniform-array-access.html (#26225)
    • NOTRUN [expected PASS] subtest: Overall test
  • OK /css/CSS2/floats/hit-test-floats-004.html (#25804)
    • FAIL [expected PASS] subtest: Miss float below something else
  • OK /css/CSS2/linebox/inline-negative-margin-001.html (#23862)
    • PASS [expected FAIL] subtest: [data-expected-height] 1
    • PASS [expected FAIL] subtest: [data-expected-height] 2
    • FAIL [expected PASS] subtest: [data-expected-height] 4
  • OK /css/css-flexbox/hittest-anonymous-box.html (#27091)
    • PASS [expected FAIL] subtest: Hit-testing within an anonymous flex-item should return the flexbox as the hittest result.
  • OK /css/css-flexbox/hittest-before-pseudo.html (#26521)
    • FAIL [expected PASS] subtest: Hit-testing text within a pseudo-element flex-item should return the flexbox as the hittest result.
  • PASS [expected FAIL] /css/css-text/line-breaking/segment-break-transformation-unremovable-2.html (#28758)
  • PASS [expected FAIL] /css/css-text/line-breaking/segment-break-transformation-unremovable-3.html (#28758)
  • OK /css/css-text/white-space/trailing-space-position-001.html (#24585)
    • PASS [expected FAIL] subtest: CSS Test: Positions of trailing collapsible spaces 4
  • OK /css/cssom-view/elementFromPoint-list-001.html (#23915)
    • FAIL [expected PASS] subtest: <li>Inside 3</li>
    • FAIL [expected PASS] subtest: <li>Image Inside 1</li>
    • FAIL [expected PASS] subtest: <li>Image Inside 2</li>
  • OK /css/cssom-view/elementFromPoint-subpixel.html (#23623)
    • PASS [expected FAIL] subtest: Hit test top left corner of box
  • OK /css/cssom-view/elementsFromPoint-iframes.html (#19273)
    • FAIL [expected PASS] subtest: elementsFromPoint on inner documents
  • TIMEOUT /fetch/metadata/generated/css-images.sub.tentative.html (#29047)
    • TIMEOUT [expected PASS] subtest: background-image sec-fetch-dest - Not sent to non-trustworthy same-origin destination
    • PASS [expected FAIL] subtest: border-image sec-fetch-user - Not sent to non-trustworthy same-site destination
  • OK [expected TIMEOUT] /html/browsers/browsing-the-web/navigating-across-documents/initial-empty-document/iframe-src-aboutblank-navigate-immediately.html (#29048)
    • FAIL [expected TIMEOUT] subtest: Navigating to a different document with form submission
  • OK [expected TIMEOUT] /html/browsers/browsing-the-web/read-media/pageload-image-in-popup.html (#23849)
    • FAIL [expected NOTRUN] subtest: The document for a standalone media file should have one child in the body.
  • OK [expected TIMEOUT] /html/interaction/focus/the-autofocus-attribute/supported-elements.html (#24145)
    • FAIL [expected TIMEOUT] subtest: Host element with delegatesFocus including no focusable descendants should be skipped
    • FAIL [expected NOTRUN] subtest: Area element should support autofocus
  • OK [expected TIMEOUT] /html/semantics/embedded-content/the-iframe-element/iframe_sandbox_popups_escaping-3.html (#24057)
    • FAIL [expected TIMEOUT] subtest: Check that popups from a sandboxed iframe escape the sandbox if
      allow-popups-to-escape-sandbox is used
  • OK /html/semantics/forms/textfieldselection/select-event.html (#27980)
    • PASS [expected FAIL] subtest: input type text: selectionEnd a second time (must not fire select)
  • TIMEOUT [expected OK] /html/webappapis/scripting/processing-model-2/integration-with-the-javascript-job-queue/promise-job-entry.html (#25805)
    • TIMEOUT [expected FAIL] subtest: Fulfillment handler on pending-then-fulfilled promise
    • TIMEOUT [expected FAIL] subtest: Rejection handler on pending-then-rejected promise
  • OK [expected TIMEOUT] /wasm/jsapi/functions/entry.html (#27087)
    • FAIL [expected TIMEOUT] subtest: Start function
  • TIMEOUT [expected CRASH] /webmessaging/broadcastchannel/cross-partition.https.tentative.html (#29058)
  • OK [expected TIMEOUT] /webmessaging/without-ports/018.html (#24485)
    • PASS [expected TIMEOUT] subtest: origin of the script that invoked the method, javascript:
  • OK [expected TIMEOUT] /webstorage/localstorage-about-blank-3P-iframe-opens-3P-window.partitioned.tentative.html (#29053)
    • PASS [expected TIMEOUT] subtest: StorageKey: test 3P about:blank window opened from a 3P iframe
  • OK [expected ERROR] /workers/constructors/Worker/Worker-constructor.html (#22991)

@bors-servo
Copy link
Contributor

☀️ Test successful - checks-github
State: approved= try=True

@mukilan
Copy link
Member

mukilan commented Feb 22, 2023

@bors-servo r+

@bors-servo
Copy link
Contributor

📌 Commit 280ebaf has been approved by mukilan

@bors-servo
Copy link
Contributor

⌛ Testing commit 280ebaf with merge 4704970...

@mukilan mukilan self-requested a review February 22, 2023 09:57
@github-actions
Copy link

Results from try job (#4241500258):

Flaky unexpected result (18)
  • TIMEOUT [expected OK] /FileAPI/url/url-charset.window.html (#26997)
    • TIMEOUT [expected PASS] subtest: Blob charset should override any auto-detected charset.
  • OK /css/CSS2/floats/hit-test-floats-002.html (#25733)
    • FAIL [expected PASS] subtest: Hit test float
  • OK /css/CSS2/linebox/inline-negative-margin-001.html (#23862)
    • PASS [expected FAIL] subtest: [data-expected-height] 1
    • PASS [expected FAIL] subtest: [data-expected-height] 2
    • FAIL [expected PASS] subtest: [data-expected-height] 4
  • OK /css/CSS2/normal-flow/hit-test-anonymous-block.html (#25807)
    • FAIL [expected PASS] subtest: Hit test beside line of text inside anonymous block
  • PASS [expected FAIL] /css/css-text/white-space/trailing-other-space-separators-break-spaces-005.html (#25875)
  • PASS [expected FAIL] /css/css-text/white-space/trailing-other-space-separators-break-spaces-006.html (#25875)
  • OK /css/css-transforms/transform-scale-hittest.html (#24667)
    • FAIL [expected PASS] subtest: Hit test within unscaled box
  • OK /css/css-transitions/properties-value-implicit-001.html (#21486)
    • FAIL [expected PASS] subtest: border-top-width length-em(em) / values
    • FAIL [expected PASS] subtest: border-right-width length-em(em) / values
    • FAIL [expected PASS] subtest: border-bottom-width length-em(em) / values
    • FAIL [expected PASS] subtest: border-left-width length-em(em) / values
    • FAIL [expected PASS] subtest: padding-bottom length-em(em) / values
    • FAIL [expected PASS] subtest: padding-left length-em(em) / values
    • FAIL [expected PASS] subtest: padding-right length-em(em) / values
    • FAIL [expected PASS] subtest: padding-top length-em(em) / values
    • FAIL [expected PASS] subtest: margin-bottom length-em(em) / values
    • FAIL [expected PASS] subtest: margin-left length-em(em) / values
    • And 17 more unexpected results...
  • OK /css/cssom-view/CaretPosition-001.html (#21338)
    • PASS [expected FAIL] subtest: Element at (400, 100)
  • OK /css/cssom-view/elementFromPoint-float-in-relative.html (#27753)
    • FAIL [expected PASS] subtest: elementFromPoint-float-in-relative
  • OK /css/cssom-view/elementsFromPoint-invalid-cases.html (#21338)
    • PASS [expected FAIL] subtest: The root element is the last element returned for otherwise empty queries within the viewport
  • ERROR [expected TIMEOUT] /html/browsers/browsing-the-web/history-traversal/persisted-user-state-restoration/scroll-restoration-fragment-scrolling-cross-origin.html (#28541)
  • TIMEOUT [expected OK] /html/browsers/history/the-history-interface/traverse-during-unload.html (#28688)
    • TIMEOUT [expected PASS] subtest: Traversing the history during unload
  • TIMEOUT [expected OK] /html/semantics/embedded-content/the-iframe-element/iframe_sandbox_popups_escaping-2.html (#22667)
    • TIMEOUT [expected FAIL] subtest: Check that popups from a sandboxed iframe escape the sandbox if
      allow-popups-to-escape-sandbox is used
  • TIMEOUT [expected OK] /html/semantics/embedded-content/the-iframe-element/iframe_sandbox_popups_nonescaping-1.html (#24066)
    • NOTRUN [expected FAIL] subtest: Check that popups from a sandboxed iframe do not escape the sandbox
  • CRASH [expected OK] /html/semantics/embedded-content/the-iframe-element/iframe_sandbox_popups_nonescaping-3.html (#24066)
  • OK /html/semantics/forms/historical.html (#28568)
    • PASS [expected FAIL] subtest: <input name=isindex> should not be supported
  • TIMEOUT [expected OK] /html/webappapis/scripting/processing-model-2/integration-with-the-javascript-job-queue/promise-job-entry.html (#25805)
    • TIMEOUT [expected FAIL] subtest: Fulfillment handler on pending-then-fulfilled promise
    • TIMEOUT [expected FAIL] subtest: Rejection handler on pending-then-rejected promise
Stable unexpected results that are known to be intermittent (25)
  • TIMEOUT /FileAPI/url/url-in-tags-revoke.window.html (#19978)
    • FAIL [expected TIMEOUT] subtest: Opening a blob URL in a new window immediately before revoking it works.
  • OK /_mozilla/mozilla/scrollBy.html (#21321)
    • FAIL [expected PASS] subtest: Ensure that the window.scrollBy function affects scroll position as expected
  • OK /css/CSS2/normal-flow/block-in-inline-hittest-002.html (#29057)
    • FAIL [expected PASS] subtest: elementFromPoint
  • OK /css/css-flexbox/hittest-anonymous-box.html (#27091)
    • PASS [expected FAIL] subtest: Hit-testing within an anonymous flex-item should return the flexbox as the hittest result.
  • PASS [expected FAIL] /css/css-text/line-breaking/segment-break-transformation-unremovable-2.html (#28758)
  • PASS [expected FAIL] /css/css-text/line-breaking/segment-break-transformation-unremovable-3.html (#28758)
  • OK /css/css-text/white-space/trailing-space-position-001.html (#24585)
    • PASS [expected FAIL] subtest: CSS Test: Positions of trailing collapsible spaces 4
  • OK /css/cssom-view/elementFromPoint-list-001.html (#23915)
    • PASS [expected FAIL] subtest: <li>Inside 2</li>
  • OK /css/cssom-view/elementFromPoint-subpixel.html (#23623)
    • PASS [expected FAIL] subtest: Hit test top left corner of box
  • OK /css/cssom-view/elementsFromPoint-iframes.html (#19273)
    • FAIL [expected PASS] subtest: elementsFromPoint on inner documents
  • TIMEOUT /fetch/metadata/generated/css-images.sub.tentative.html (#29047)
    • PASS [expected FAIL] subtest: border-image sec-fetch-user - Not sent to non-trustworthy same-site destination
    • TIMEOUT [expected PASS] subtest: background-image sec-fetch-site - HTTPS downgrade (header not sent)
  • OK [expected TIMEOUT] /html/browsers/browsing-the-web/navigating-across-documents/initial-empty-document/iframe-src-aboutblank-navigate-immediately.html (#29048)
    • FAIL [expected TIMEOUT] subtest: Navigating to a different document with form submission
  • OK /html/browsers/browsing-the-web/navigating-across-documents/initial-empty-document/load-pageshow-events-iframe-contentWindow.html (#28681)
    • FAIL [expected PASS] subtest: load & pageshow events do not fire on contentWindow of <iframe> element created with src=''
    • PASS [expected FAIL] subtest: load & pageshow events do not fire on contentWindow of <iframe> element created with src='about:blank'
  • OK [expected TIMEOUT] /html/browsers/browsing-the-web/read-media/pageload-image-in-popup.html (#23849)
    • FAIL [expected NOTRUN] subtest: The document for a standalone media file should have one child in the body.
  • TIMEOUT [expected FAIL] /html/canvas/element/manual/drawing-text-to-the-canvas/canvas.2d.disconnected.html (#29224)
  • OK [expected TIMEOUT] /html/interaction/focus/the-autofocus-attribute/supported-elements.html (#24145)
    • FAIL [expected TIMEOUT] subtest: Host element with delegatesFocus including no focusable descendants should be skipped
    • FAIL [expected NOTRUN] subtest: Area element should support autofocus
  • OK [expected CRASH] /html/semantics/embedded-content/the-iframe-element/iframe_sandbox_popups_nonescaping-2.html (#22154)
  • OK /html/semantics/forms/textfieldselection/select-event.html (#27980)
    • PASS [expected FAIL] subtest: input type text: selectionEnd a second time (must not fire select)
  • OK [expected TIMEOUT] /html/semantics/links/links-created-by-a-and-area-elements/htmlanchorelement_noopener.html (#23205)
    • PASS [expected NOTRUN] subtest: Check that rel=noopener with target=_self does a normal load
  • OK [expected TIMEOUT] /wasm/jsapi/functions/entry.html (#27087)
    • FAIL [expected TIMEOUT] subtest: Start function
  • TIMEOUT [expected OK] /webmessaging/with-ports/017.html (#24486)
    • TIMEOUT [expected PASS] subtest: origin of the script that invoked the method, about:blank
  • OK [expected TIMEOUT] /webmessaging/without-ports/017.html (#24486)
    • PASS [expected TIMEOUT] subtest: origin of the script that invoked the method, about:blank
  • OK [expected TIMEOUT] /webmessaging/without-ports/018.html (#24485)
    • PASS [expected TIMEOUT] subtest: origin of the script that invoked the method, javascript:
  • OK [expected TIMEOUT] /webstorage/localstorage-about-blank-3P-iframe-opens-3P-window.partitioned.tentative.html (#29053)
    • PASS [expected TIMEOUT] subtest: StorageKey: test 3P about:blank window opened from a 3P iframe
  • OK [expected ERROR] /workers/constructors/Worker/Worker-constructor.html (#22991)

@bors-servo
Copy link
Contributor

☀️ Test successful - checks-github
Approved by: mukilan
Pushing 4704970 to master...

@bors-servo bors-servo merged commit 4704970 into servo:master Feb 22, 2023
@bors-servo bors-servo mentioned this pull request Feb 22, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mukilan mukilan mukilan approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@mrobinson @bors-servo @mukilan