Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a travis job for testing PR stability #3975

Merged
merged 2 commits into from Oct 20, 2016
Merged

Add a travis job for testing PR stability #3975

merged 2 commits into from Oct 20, 2016

Conversation

jgraham
Copy link
Contributor

@jgraham jgraham commented Oct 13, 2016

The purpose of the job is to identify poorly-written, unstable, tests as
early in the cycle as possible. To this end we use wptrunner to run all
the changed tests in the PR in Firefox. If 10 runs of the tests don't
all give a consisent result then the job fails.

Adding runs in Chrome should be quite straightforward, adding Edge and
Safari may be more difficult because of their strict platform dependencies.

@jgraham jgraham force-pushed the stability_check branch 30 times, most recently from 77b676b to 37f4262 Compare October 19, 2016 12:56
@jgraham jgraham force-pushed the stability_check branch 2 times, most recently from 19a3806 to 9f030d7 Compare October 19, 2016 14:45
@wpt-stability-bot
Copy link

test

@wpt-stability-bot
Copy link

Testing in firefoxFiles changed:
/home/travis/build/w3c/web-platform-tests/.travis.yml
/home/travis/build/w3c/web-platform-tests/check_stability.py
/home/travis/build/w3c/web-platform-tests/ci_lint.sh
/home/travis/build/w3c/web-platform-tests/ci_stability.sh
/home/travis/build/w3c/web-platform-tests/dom/example_unstable.html
/home/travis/build/w3c/web-platform-tests/dom/historical.htmlTest runs done| /dom/historical.html => OK: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => FAIL: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => FAIL: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| /dom/example_unstable.html => OK: 10| - /dom/example_unstable.html => FAIL: 3 PASS: 7Got unstable results:/dom/example_unstable.html | Unstable test | FAIL: 3 PASS: 7

@wpt-stability-bot
Copy link

Testing in chromeFiles changed:
/home/travis/build/w3c/web-platform-tests/.travis.yml
/home/travis/build/w3c/web-platform-tests/check_stability.py
/home/travis/build/w3c/web-platform-tests/ci_lint.sh
/home/travis/build/w3c/web-platform-tests/ci_stability.sh
/home/travis/build/w3c/web-platform-tests/dom/example_unstable.html
/home/travis/build/w3c/web-platform-tests/dom/historical.htmlTest runs done| /dom/historical.html => OK: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => FAIL: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => FAIL: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => FAIL: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => FAIL: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| - /dom/historical.html => PASS: 10| /dom/example_unstable.html => OK: 10| - /dom/example_unstable.html => FAIL: 2 PASS: 8Got unstable results:/dom/example_unstable.html | Unstable test | FAIL: 2 PASS: 8

@wpt-stability-bot
Copy link

Testing in Chrome
Files changed:

  • /home/travis/build/w3c/web-platform-tests/.travis.yml
    • /home/travis/build/w3c/web-platform-tests/check_stability.py
    • /home/travis/build/w3c/web-platform-tests/ci_lint.sh
    • /home/travis/build/w3c/web-platform-tests/ci_stability.sh
    • /home/travis/build/w3c/web-platform-tests/dom/example_unstable.html
    • /home/travis/build/w3c/web-platform-tests/dom/historical.html

Test runs done

Test Subtest Results
/dom/historical.html OK: 10
Historical DOM features must be removed: Entity PASS: 10
Historical DOM features must be removed: schemaTypeInfo PASS: 10
Node member must be nuked: attributes PASS: 10
Node member must be nuked: getUserData PASS: 10
Historical DOM features must be removed: xmlVersion FAIL: 10
Historical DOM features must be removed: setIdAttributeNS PASS: 10
Historical DOM features must be removed: DOMUserData PASS: 10
Historical DOM features must be removed: width PASS: 10
Historical DOM features must be removed: setIdAttributeNode PASS: 10
Historical DOM features must be removed: DOMError FAIL: 10
DocumentType member must be nuked: notations PASS: 10
Historical DOM features must be removed: DOMImplementationSource PASS: 10
Historical DOM features must be removed: DOMSettableTokenList PASS: 10
Historical DOM features must be removed: EventException PASS: 10
Node member must be nuked: setUserData PASS: 10
DOMImplementation.getFeature() must be nuked. PASS: 10
Historical DOM features must be removed: RangeException PASS: 10
Historical DOM features must be removed: renameNode PASS: 10
Historical DOM features must be removed: normalizeDocument PASS: 10
Historical DOM features must be removed: DOMConfiguration PASS: 10
Historical DOM features must be removed: DOMObject PASS: 10
Historical DOM features must be removed: NameList PASS: 10
Historical DOM features must be removed: xmlEncoding FAIL: 10
Historical DOM features must be removed: UserDataHandler PASS: 10
Node member must be nuked: hasAttributes PASS: 10
Historical DOM features must be removed: domConfig PASS: 10
Historical DOM features must be removed: DOMImplementationList PASS: 10
Window member must be nuked: attachEvent PASS: 10
Historical DOM features must be removed: DOMErrorHandler PASS: 10
Historical DOM features must be removed: strictErrorChecking PASS: 10
Historical DOM features must be removed: Notation PASS: 10
Node member must be nuked: getFeature PASS: 10
Historical DOM features must be removed: DOMLocator PASS: 10
Node member must be nuked: namespaceURI PASS: 10
Historical DOM features must be removed: xmlStandalone FAIL: 10
Node member must be nuked: prefix PASS: 10
Node member must be nuked: rootNode PASS: 10
Historical DOM features must be removed: setIdAttribute PASS: 10
Historical DOM features must be removed: TypeInfo PASS: 10
Historical DOM features must be removed: createEntityReference PASS: 10
Text member must be nuked: replaceWholeText PASS: 10
Historical DOM features must be removed: height PASS: 10
DocumentType member must be nuked: entities PASS: 10
Node member must be nuked: isSupported PASS: 10
Node member must be nuked: localName PASS: 10
DocumentType member must be nuked: internalSubset PASS: 10
Historical DOM features must be removed: EntityReference PASS: 10
Text member must be nuked: isElementContentWhitespace PASS: 10
Historical DOM features must be removed: defaultCharset PASS: 10
/dom/example_unstable.html OK: 10
Unstable test FAIL: 6 PASS: 4
Got unstable results:
/dom/example_unstable.html Unstable test FAIL: 6 PASS: 4

@wpt-stability-bot
Copy link

Testing in Firefox
Files changed:

  • /home/travis/build/w3c/web-platform-tests/.travis.yml
    • /home/travis/build/w3c/web-platform-tests/check_stability.py
    • /home/travis/build/w3c/web-platform-tests/ci_lint.sh
    • /home/travis/build/w3c/web-platform-tests/ci_stability.sh
    • /home/travis/build/w3c/web-platform-tests/dom/example_unstable.html
    • /home/travis/build/w3c/web-platform-tests/dom/historical.html

Test runs done

Test Subtest Results
/dom/historical.html OK: 10
Historical DOM features must be removed: Entity PASS: 10
Historical DOM features must be removed: schemaTypeInfo PASS: 10
Node member must be nuked: attributes PASS: 10
Node member must be nuked: getUserData PASS: 10
Historical DOM features must be removed: xmlVersion PASS: 10
Historical DOM features must be removed: setIdAttributeNS PASS: 10
Historical DOM features must be removed: DOMUserData PASS: 10
Historical DOM features must be removed: width PASS: 10
Historical DOM features must be removed: setIdAttributeNode PASS: 10
Historical DOM features must be removed: DOMError FAIL: 10
DocumentType member must be nuked: notations PASS: 10
Historical DOM features must be removed: DOMImplementationSource PASS: 10
Historical DOM features must be removed: DOMSettableTokenList PASS: 10
Historical DOM features must be removed: EventException PASS: 10
Node member must be nuked: setUserData PASS: 10
DOMImplementation.getFeature() must be nuked. PASS: 10
Historical DOM features must be removed: RangeException PASS: 10
Historical DOM features must be removed: renameNode PASS: 10
Historical DOM features must be removed: normalizeDocument PASS: 10
Historical DOM features must be removed: DOMConfiguration PASS: 10
Historical DOM features must be removed: DOMObject PASS: 10
Historical DOM features must be removed: NameList PASS: 10
Historical DOM features must be removed: xmlEncoding PASS: 10
Historical DOM features must be removed: UserDataHandler PASS: 10
Node member must be nuked: hasAttributes PASS: 10
Historical DOM features must be removed: domConfig PASS: 10
Historical DOM features must be removed: DOMImplementationList PASS: 10
Window member must be nuked: attachEvent PASS: 10
Historical DOM features must be removed: DOMErrorHandler PASS: 10
Historical DOM features must be removed: strictErrorChecking PASS: 10
Historical DOM features must be removed: Notation PASS: 10
Node member must be nuked: getFeature PASS: 10
Historical DOM features must be removed: DOMLocator PASS: 10
Node member must be nuked: namespaceURI PASS: 10
Historical DOM features must be removed: xmlStandalone PASS: 10
Node member must be nuked: prefix PASS: 10
Node member must be nuked: rootNode FAIL: 10
Historical DOM features must be removed: setIdAttribute PASS: 10
Historical DOM features must be removed: TypeInfo PASS: 10
Historical DOM features must be removed: createEntityReference PASS: 10
Text member must be nuked: replaceWholeText PASS: 10
Historical DOM features must be removed: height PASS: 10
DocumentType member must be nuked: entities PASS: 10
Node member must be nuked: isSupported PASS: 10
Node member must be nuked: localName PASS: 10
DocumentType member must be nuked: internalSubset PASS: 10
Historical DOM features must be removed: EntityReference PASS: 10
Text member must be nuked: isElementContentWhitespace PASS: 10
Historical DOM features must be removed: defaultCharset PASS: 10
/dom/example_unstable.html OK: 10
Unstable test FAIL: 4 PASS: 6
Got unstable results:
/dom/example_unstable.html Unstable test FAIL: 4 PASS: 6

@wpt-stability-bot
Copy link

Testing in Firefox
Files changed:

  • /home/travis/build/w3c/web-platform-tests/.travis.yml
  • /home/travis/build/w3c/web-platform-tests/check_stability.py
  • /home/travis/build/w3c/web-platform-tests/ci_lint.sh
  • /home/travis/build/w3c/web-platform-tests/ci_stability.sh
  • /home/travis/build/w3c/web-platform-tests/dom/example_unstable.html
  • /home/travis/build/w3c/web-platform-tests/dom/historical.html
Test Subtest Results
/dom/historical.html OK: 10
Historical DOM features must be removed: Entity PASS: 10
Historical DOM features must be removed: schemaTypeInfo PASS: 10
Node member must be nuked: attributes PASS: 10
Node member must be nuked: getUserData PASS: 10
Historical DOM features must be removed: xmlVersion PASS: 10
Historical DOM features must be removed: setIdAttributeNS PASS: 10
Historical DOM features must be removed: DOMUserData PASS: 10
Historical DOM features must be removed: width PASS: 10
Historical DOM features must be removed: setIdAttributeNode PASS: 10
Historical DOM features must be removed: DOMError FAIL: 10
DocumentType member must be nuked: notations PASS: 10
Historical DOM features must be removed: DOMImplementationSource PASS: 10
Historical DOM features must be removed: DOMSettableTokenList PASS: 10
Historical DOM features must be removed: EventException PASS: 10
Node member must be nuked: setUserData PASS: 10
DOMImplementation.getFeature() must be nuked. PASS: 10
Historical DOM features must be removed: RangeException PASS: 10
Historical DOM features must be removed: renameNode PASS: 10
Historical DOM features must be removed: normalizeDocument PASS: 10
Historical DOM features must be removed: DOMConfiguration PASS: 10
Historical DOM features must be removed: DOMObject PASS: 10
Historical DOM features must be removed: NameList PASS: 10
Historical DOM features must be removed: xmlEncoding PASS: 10
Historical DOM features must be removed: UserDataHandler PASS: 10
Node member must be nuked: hasAttributes PASS: 10
Historical DOM features must be removed: domConfig PASS: 10
Historical DOM features must be removed: DOMImplementationList PASS: 10
Window member must be nuked: attachEvent PASS: 10
Historical DOM features must be removed: DOMErrorHandler PASS: 10
Historical DOM features must be removed: strictErrorChecking PASS: 10
Historical DOM features must be removed: Notation PASS: 10
Node member must be nuked: getFeature PASS: 10
Historical DOM features must be removed: DOMLocator PASS: 10
Node member must be nuked: namespaceURI PASS: 10
Historical DOM features must be removed: xmlStandalone PASS: 10
Node member must be nuked: prefix PASS: 10
Node member must be nuked: rootNode FAIL: 10
Historical DOM features must be removed: setIdAttribute PASS: 10
Historical DOM features must be removed: TypeInfo PASS: 10
Historical DOM features must be removed: createEntityReference PASS: 10
Text member must be nuked: replaceWholeText PASS: 10
Historical DOM features must be removed: height PASS: 10
DocumentType member must be nuked: entities PASS: 10
Node member must be nuked: isSupported PASS: 10
Node member must be nuked: localName PASS: 10
DocumentType member must be nuked: internalSubset PASS: 10
Historical DOM features must be removed: EntityReference PASS: 10
Text member must be nuked: isElementContentWhitespace PASS: 10
Historical DOM features must be removed: defaultCharset PASS: 10
/dom/example_unstable.html OK: 10
Unstable test FAIL: 3 PASS: 7

Got unstable results:

Test Subtest Results
/dom/example_unstable.html Unstable test FAIL: 3 PASS: 7

@wpt-stability-bot
Copy link

Testing in Chrome
Files changed:

  • /home/travis/build/w3c/web-platform-tests/.travis.yml
  • /home/travis/build/w3c/web-platform-tests/check_stability.py
  • /home/travis/build/w3c/web-platform-tests/ci_lint.sh
  • /home/travis/build/w3c/web-platform-tests/ci_stability.sh
  • /home/travis/build/w3c/web-platform-tests/dom/example_unstable.html
  • /home/travis/build/w3c/web-platform-tests/dom/historical.html
Test Subtest Results
/dom/historical.html OK: 10
Historical DOM features must be removed: Entity PASS: 10
Historical DOM features must be removed: schemaTypeInfo PASS: 10
Node member must be nuked: attributes PASS: 10
Node member must be nuked: getUserData PASS: 10
Historical DOM features must be removed: xmlVersion FAIL: 10
Historical DOM features must be removed: setIdAttributeNS PASS: 10
Historical DOM features must be removed: DOMUserData PASS: 10
Historical DOM features must be removed: width PASS: 10
Historical DOM features must be removed: setIdAttributeNode PASS: 10
Historical DOM features must be removed: DOMError FAIL: 10
DocumentType member must be nuked: notations PASS: 10
Historical DOM features must be removed: DOMImplementationSource PASS: 10
Historical DOM features must be removed: DOMSettableTokenList PASS: 10
Historical DOM features must be removed: EventException PASS: 10
Node member must be nuked: setUserData PASS: 10
DOMImplementation.getFeature() must be nuked. PASS: 10
Historical DOM features must be removed: RangeException PASS: 10
Historical DOM features must be removed: renameNode PASS: 10
Historical DOM features must be removed: normalizeDocument PASS: 10
Historical DOM features must be removed: DOMConfiguration PASS: 10
Historical DOM features must be removed: DOMObject PASS: 10
Historical DOM features must be removed: NameList PASS: 10
Historical DOM features must be removed: xmlEncoding FAIL: 10
Historical DOM features must be removed: UserDataHandler PASS: 10
Node member must be nuked: hasAttributes PASS: 10
Historical DOM features must be removed: domConfig PASS: 10
Historical DOM features must be removed: DOMImplementationList PASS: 10
Window member must be nuked: attachEvent PASS: 10
Historical DOM features must be removed: DOMErrorHandler PASS: 10
Historical DOM features must be removed: strictErrorChecking PASS: 10
Historical DOM features must be removed: Notation PASS: 10
Node member must be nuked: getFeature PASS: 10
Historical DOM features must be removed: DOMLocator PASS: 10
Node member must be nuked: namespaceURI PASS: 10
Historical DOM features must be removed: xmlStandalone FAIL: 10
Node member must be nuked: prefix PASS: 10
Node member must be nuked: rootNode PASS: 10
Historical DOM features must be removed: setIdAttribute PASS: 10
Historical DOM features must be removed: TypeInfo PASS: 10
Historical DOM features must be removed: createEntityReference PASS: 10
Text member must be nuked: replaceWholeText PASS: 10
Historical DOM features must be removed: height PASS: 10
DocumentType member must be nuked: entities PASS: 10
Node member must be nuked: isSupported PASS: 10
Node member must be nuked: localName PASS: 10
DocumentType member must be nuked: internalSubset PASS: 10
Historical DOM features must be removed: EntityReference PASS: 10
Text member must be nuked: isElementContentWhitespace PASS: 10
Historical DOM features must be removed: defaultCharset PASS: 10
/dom/example_unstable.html OK: 10
Unstable test FAIL: 5 PASS: 5

Got unstable results:

Test Subtest Results
/dom/example_unstable.html Unstable test FAIL: 5 PASS: 5

@jgraham
Copy link
Contributor Author

jgraham commented Oct 19, 2016

@Ms2ger r?

@Ms2ger
Copy link
Contributor

Ms2ger commented Oct 20, 2016

Go ahead and merge.

@jgraham jgraham force-pushed the stability_check branch 3 times, most recently from a68cd0e to 93152fa Compare October 20, 2016 12:55
@wpt-stability-bot
Copy link

Testing in Firefox
Files changed:

  • /home/travis/build/w3c/web-platform-tests/.travis.yml
  • /home/travis/build/w3c/web-platform-tests/check_stability.py
  • /home/travis/build/w3c/web-platform-tests/ci_lint.sh
  • /home/travis/build/w3c/web-platform-tests/ci_stability.sh
  • /home/travis/build/w3c/web-platform-tests/dom/example_unstable.html
  • /home/travis/build/w3c/web-platform-tests/dom/historical.html

==Unstable results==

Test Subtest Results
/dom/example_unstable.html Unstable test FAIL: 7 PASS: 3

==All results==

Test Subtest Results
/dom/historical.html OK: 10
Historical DOM features must be removed: Entity PASS: 10
Historical DOM features must be removed: schemaTypeInfo PASS: 10
Node member must be nuked: attributes PASS: 10
Node member must be nuked: getUserData PASS: 10
Historical DOM features must be removed: xmlVersion PASS: 10
Historical DOM features must be removed: setIdAttributeNS PASS: 10
Historical DOM features must be removed: DOMUserData PASS: 10
Historical DOM features must be removed: width PASS: 10
Historical DOM features must be removed: setIdAttributeNode PASS: 10
Historical DOM features must be removed: DOMError FAIL: 10
DocumentType member must be nuked: notations PASS: 10
Historical DOM features must be removed: DOMImplementationSource PASS: 10
Historical DOM features must be removed: DOMSettableTokenList PASS: 10
Historical DOM features must be removed: EventException PASS: 10
Node member must be nuked: setUserData PASS: 10
DOMImplementation.getFeature() must be nuked. PASS: 10
Historical DOM features must be removed: RangeException PASS: 10
Historical DOM features must be removed: renameNode PASS: 10
Historical DOM features must be removed: normalizeDocument PASS: 10
Historical DOM features must be removed: DOMConfiguration PASS: 10
Historical DOM features must be removed: DOMObject PASS: 10
Historical DOM features must be removed: NameList PASS: 10
Historical DOM features must be removed: xmlEncoding PASS: 10
Historical DOM features must be removed: UserDataHandler PASS: 10
Node member must be nuked: hasAttributes PASS: 10
Historical DOM features must be removed: domConfig PASS: 10
Historical DOM features must be removed: DOMImplementationList PASS: 10
Window member must be nuked: attachEvent PASS: 10
Historical DOM features must be removed: DOMErrorHandler PASS: 10
Historical DOM features must be removed: strictErrorChecking PASS: 10
Historical DOM features must be removed: Notation PASS: 10
Node member must be nuked: getFeature PASS: 10
Historical DOM features must be removed: DOMLocator PASS: 10
Node member must be nuked: namespaceURI PASS: 10
Historical DOM features must be removed: xmlStandalone PASS: 10
Node member must be nuked: prefix PASS: 10
Node member must be nuked: rootNode FAIL: 10
Historical DOM features must be removed: setIdAttribute PASS: 10
Historical DOM features must be removed: TypeInfo PASS: 10
Historical DOM features must be removed: createEntityReference PASS: 10
Text member must be nuked: replaceWholeText PASS: 10
Historical DOM features must be removed: height PASS: 10
DocumentType member must be nuked: entities PASS: 10
Node member must be nuked: isSupported PASS: 10
Node member must be nuked: localName PASS: 10
DocumentType member must be nuked: internalSubset PASS: 10
Historical DOM features must be removed: EntityReference PASS: 10
Text member must be nuked: isElementContentWhitespace PASS: 10
Historical DOM features must be removed: defaultCharset PASS: 10
/dom/example_unstable.html OK: 10
Unstable test FAIL: 7 PASS: 3

@wpt-stability-bot
Copy link

Testing in Chrome
Files changed:

  • /home/travis/build/w3c/web-platform-tests/.travis.yml
  • /home/travis/build/w3c/web-platform-tests/check_stability.py
  • /home/travis/build/w3c/web-platform-tests/ci_lint.sh
  • /home/travis/build/w3c/web-platform-tests/ci_stability.sh
  • /home/travis/build/w3c/web-platform-tests/dom/example_unstable.html
  • /home/travis/build/w3c/web-platform-tests/dom/historical.html

==Unstable results==

Test Subtest Results
/dom/example_unstable.html Unstable test FAIL: 5 PASS: 5

==All results==

Test Subtest Results
/dom/historical.html OK: 10
Historical DOM features must be removed: Entity PASS: 10
Historical DOM features must be removed: schemaTypeInfo PASS: 10
Node member must be nuked: attributes PASS: 10
Node member must be nuked: getUserData PASS: 10
Historical DOM features must be removed: xmlVersion FAIL: 10
Historical DOM features must be removed: setIdAttributeNS PASS: 10
Historical DOM features must be removed: DOMUserData PASS: 10
Historical DOM features must be removed: width PASS: 10
Historical DOM features must be removed: setIdAttributeNode PASS: 10
Historical DOM features must be removed: DOMError FAIL: 10
DocumentType member must be nuked: notations PASS: 10
Historical DOM features must be removed: DOMImplementationSource PASS: 10
Historical DOM features must be removed: DOMSettableTokenList PASS: 10
Historical DOM features must be removed: EventException PASS: 10
Node member must be nuked: setUserData PASS: 10
DOMImplementation.getFeature() must be nuked. PASS: 10
Historical DOM features must be removed: RangeException PASS: 10
Historical DOM features must be removed: renameNode PASS: 10
Historical DOM features must be removed: normalizeDocument PASS: 10
Historical DOM features must be removed: DOMConfiguration PASS: 10
Historical DOM features must be removed: DOMObject PASS: 10
Historical DOM features must be removed: NameList PASS: 10
Historical DOM features must be removed: xmlEncoding FAIL: 10
Historical DOM features must be removed: UserDataHandler PASS: 10
Node member must be nuked: hasAttributes PASS: 10
Historical DOM features must be removed: domConfig PASS: 10
Historical DOM features must be removed: DOMImplementationList PASS: 10
Window member must be nuked: attachEvent PASS: 10
Historical DOM features must be removed: DOMErrorHandler PASS: 10
Historical DOM features must be removed: strictErrorChecking PASS: 10
Historical DOM features must be removed: Notation PASS: 10
Node member must be nuked: getFeature PASS: 10
Historical DOM features must be removed: DOMLocator PASS: 10
Node member must be nuked: namespaceURI PASS: 10
Historical DOM features must be removed: xmlStandalone FAIL: 10
Node member must be nuked: prefix PASS: 10
Node member must be nuked: rootNode PASS: 10
Historical DOM features must be removed: setIdAttribute PASS: 10
Historical DOM features must be removed: TypeInfo PASS: 10
Historical DOM features must be removed: createEntityReference PASS: 10
Text member must be nuked: replaceWholeText PASS: 10
Historical DOM features must be removed: height PASS: 10
DocumentType member must be nuked: entities PASS: 10
Node member must be nuked: isSupported PASS: 10
Node member must be nuked: localName PASS: 10
DocumentType member must be nuked: internalSubset PASS: 10
Historical DOM features must be removed: EntityReference PASS: 10
Text member must be nuked: isElementContentWhitespace PASS: 10
Historical DOM features must be removed: defaultCharset PASS: 10
/dom/example_unstable.html OK: 10
Unstable test FAIL: 5 PASS: 5

@jgraham
Add a travis job for testing PR stability
be27f89 
The purpose of the job is to identify poorly-written, unstable, tests as
early in the cycle as possible. To this end we use wptrunner to run all
the changed tests in the PR in Firefox and Chrome. If 10 runs of the
tests don't all give a consisent result then the job fails.

Adding Edge and Safari should be possible, but will be challenging
because of their strict platform dependencies.
@wpt-stability-bot
Copy link

Testing in Firefox
Files changed:

  • /home/travis/build/w3c/web-platform-tests/.travis.yml
  • /home/travis/build/w3c/web-platform-tests/check_stability.py
  • /home/travis/build/w3c/web-platform-tests/ci_lint.sh
  • /home/travis/build/w3c/web-platform-tests/ci_stability.sh

All results were stable

All results

Test Subtest Results

@wpt-stability-bot
Copy link

Testing in Chrome
Files changed:

  • /home/travis/build/w3c/web-platform-tests/.travis.yml
  • /home/travis/build/w3c/web-platform-tests/check_stability.py
  • /home/travis/build/w3c/web-platform-tests/ci_lint.sh
  • /home/travis/build/w3c/web-platform-tests/ci_stability.sh

All results were stable

All results

Test Subtest Results

@wpt-stability-bot
Copy link

Testing in Chrome
Files changed:

  • /home/travis/build/w3c/web-platform-tests/.travis.yml
  • /home/travis/build/w3c/web-platform-tests/check_stability.py
  • /home/travis/build/w3c/web-platform-tests/ci_lint.sh
  • /home/travis/build/w3c/web-platform-tests/ci_stability.sh

No tests run.

@wpt-stability-bot
Copy link

Testing in Firefox
Files changed:

  • /home/travis/build/w3c/web-platform-tests/.travis.yml
  • /home/travis/build/w3c/web-platform-tests/check_stability.py
  • /home/travis/build/w3c/web-platform-tests/ci_lint.sh
  • /home/travis/build/w3c/web-platform-tests/ci_stability.sh

No tests run.

@jgraham jgraham merged commit 50075bf into master Oct 20, 2016
@ayg ayg deleted the stability_check branch October 26, 2016 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@jgraham @wpt-stability-bot @Ms2ger