Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: run the addon tests last #12062

Closed
wants to merge 1 commit into from

Conversation

@seppevs
Copy link
Contributor

commented Mar 27, 2017

Running the addon tests before the parallel, sequential, etc. tests
can be a problem if there is a bug in node that prevents the
addon tests from running properly. When the addon tests fail for any
reason, then none of the other tests (e.g. parallel, etc.)
are executed.

Running the addon tests last fixes this.

See #12031

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • commit message follows [commit guidelines][]
Affected core subsystem(s)

test

@aqrln

This comment has been minimized.

Copy link
Member

commented Mar 27, 2017

I think that "See ..." line should be formatted as a proper "Refs:" or "Fixes:" metadata field. Also, there's probably a typo in the commit message (s/lasts/last ?).

@seppevs seppevs force-pushed the seppevs:test_run_addon_tests_last branch Mar 27, 2017

@seppevs

This comment has been minimized.

Copy link
Contributor Author

commented Mar 27, 2017

Thanks for the tips, I've changed the commit message.

@seppevs seppevs changed the title test: run the addon tests lasts test: run the addon tests last Mar 27, 2017

@vsemozhetbyt

This comment has been minimized.

Copy link
Contributor

commented Mar 27, 2017

@vsemozhetbyt

This comment has been minimized.

Copy link
Contributor

commented Mar 27, 2017

Does this affect Windows builds? Should the vcbuild.bat also be fixed?

@vsemozhetbyt

This comment has been minimized.

Copy link
Contributor

commented Mar 27, 2017

Some tests failed, trying another CI: https://ci.nodejs.org/job/node-test-pull-request/7055/

@aqrln

This comment has been minimized.

@aqrln

This comment has been minimized.

Copy link
Member

commented Mar 27, 2017

@vsemozhetbyt those tests are unrelated to this PR and I don't think another CI run will help, tbh :)
Failed test on Windows is already marked as flaky in #12054, and I've seen parallel/test-async-wrap-check-providers as well as other tests failing with timeout on CentOS 5 for a while too (just open any of the recent PRs).

@seppevs seppevs force-pushed the seppevs:test_run_addon_tests_last branch Mar 27, 2017

@seppevs

This comment has been minimized.

Copy link
Contributor Author

commented Mar 27, 2017

@aqrln Thanks for the tip. I have modified the vcbuild.bat now. Can you review it, I'm not completely sure I did it correctly (and I have no Windows machine to verify).

@aqrln

This comment has been minimized.

Copy link
Member

commented Mar 27, 2017

@seppevs heh, neither do I, so I hope my suggestion was correct :)

/cc @nodejs/build @nodejs/platform-windows

EDIT: whoops, only members can mention teams. @vsemozhetbyt can you do that, please?

@vsemozhetbyt

This comment has been minimized.

Copy link
Contributor

commented Mar 27, 2017

/cc @nodejs/build @nodejs/platform-windows

@Fishrock123
Copy link
Member

left a comment

Yes please.

cc @bnoordhuis probably?

Makefile Outdated
@@ -195,7 +195,7 @@ test: all
$(MAKE) build-addons
$(MAKE) cctest
$(PYTHON) tools/test.py --mode=release -J \
addons doctool inspector known_issues message pseudo-tty parallel sequential
doctool inspector known_issues message pseudo-tty parallel sequential addons

This comment has been minimized.

Copy link
@aqrln

aqrln Mar 27, 2017

Member

One more thing. CI runs make test-ci, not make test, so you need to change line 311 too.

This comment has been minimized.

Copy link
@seppevs

seppevs Mar 27, 2017

Author Contributor

Thanks, I've changed the order on line 311 as well.

@seppevs seppevs force-pushed the seppevs:test_run_addon_tests_last branch Mar 27, 2017

@jasnell
Copy link
Member

left a comment

Note that failures in cctest also suffer this problem.

@refack

This comment has been minimized.

Copy link
Member

commented Mar 29, 2017

I have modified the vcbuild.bat now. Can you review it, I'm not completely sure I did it correctly (and I have no Windows machine to verify).

So on windows, it's only a half solution since build-addons is run before the tests, and so it the build fails no tests are run 😞

@aqrln

This comment has been minimized.

Copy link
Member

commented Mar 29, 2017

@refack hmm, not only only on Windows. The same goes for Unix.

Makefile Outdated
@@ -195,7 +195,7 @@ test: all
$(MAKE) build-addons

This comment has been minimized.

Copy link
@refack

refack Mar 29, 2017

Member

maybe split this to "all other test" -> build -> "test addons"... 🤔

vcbuild.bat Outdated
if /i "%1"=="test" set test_args=%test_args% addons doctool known_issues message parallel sequential -J&set cpplint=1&set jslint=1&set build_addons=1&goto arg-ok
if /i "%1"=="test-ci" set test_args=%test_args% %test_ci_args% -p tap --logfile test.tap addons doctool inspector known_issues message sequential parallel&set cctest_args=%cctest_args% --gtest_output=tap:cctest.tap&set build_addons=1&goto arg-ok
if /i "%1"=="test" set test_args=%test_args% doctool known_issues message parallel sequential addons -J&set cpplint=1&set jslint=1&set build_addons=1&goto arg-ok
if /i "%1"=="test-ci" set test_args=%test_args% %test_ci_args% -p tap --logfile test.tap doctool inspector known_issues message sequential parallel addons&set cctest_args=%cctest_args% --gtest_output=tap:cctest.tap&set build_addons=1&goto arg-ok

This comment has been minimized.

Copy link
@refack

refack Mar 29, 2017

Member

Is you decide to split the test, you'll need it here too...

@refack
refack approved these changes Mar 29, 2017
Copy link
Member

left a comment

Too much hassle to refactor vcbuild.bat
Current LGTM (works on Windows)

@seppevs seppevs force-pushed the seppevs:test_run_addon_tests_last branch Apr 4, 2017

@jasnell

This comment has been minimized.

Copy link
Member

commented Apr 4, 2017

This needs a rebase before it can be landed.

test: run the addon tests last
Running the addon tests before the parallel, sequential, etc. tests
can be a problem if there is a bug in node that prevents the
addon tests from running properly. When the addon tests fail for any
reason, then none of the other tests (e.g. parallel, etc.)
are executed.

Running the addon tests last fixes this.

Refs: #12031

@seppevs seppevs force-pushed the seppevs:test_run_addon_tests_last branch to 9149610 Apr 4, 2017

@seppevs

This comment has been minimized.

Copy link
Contributor Author

commented Apr 4, 2017

@jasnell rebase done

@mscdex

This comment has been minimized.

Copy link
Contributor

commented Apr 5, 2017

@aqrln

This comment has been minimized.

Copy link
Member

commented Apr 12, 2017

@seppevs can you please rebase it again?

@jasnell

This comment has been minimized.

Copy link
Member

commented Apr 13, 2017

This needs to be rebased before it can land! :-)

ha! posted then saw @aqrln's comment. guess I should read all the way through before posting ;-)

@refack

This comment has been minimized.

Copy link
Member

commented Apr 13, 2017

Poor guy already rebased one. I'm taking care of this.

refack added a commit to refack/node that referenced this pull request Apr 13, 2017
test: run the addon tests last
Running the addon tests before the parallel, sequential, etc. tests
can be a problem if there is a bug in node that prevents the
addon tests from running properly. When the addon tests fail for any
reason, then none of the other tests (e.g. parallel, etc.)
are executed.

Running the addon tests last fixes this.

Refs: nodejs#12031
PR-URL: nodejs#12062
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>

@refack refack closed this Apr 13, 2017

@refack

This comment has been minimized.

Copy link
Member

commented Apr 13, 2017

Landed in 3d2181c

@jasnell jasnell referenced this pull request May 11, 2017
@gibfahn gibfahn referenced this pull request Jun 15, 2017
2 of 3 tasks complete
gibfahn added a commit that referenced this pull request Jun 18, 2017
test: run the addon tests last
Running the addon tests before the parallel, sequential, etc. tests
can be a problem if there is a bug in node that prevents the
addon tests from running properly. When the addon tests fail for any
reason, then none of the other tests (e.g. parallel, etc.)
are executed.

Running the addon tests last fixes this.

Refs: #12031
PR-URL: #12062
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
gibfahn added a commit that referenced this pull request Jun 20, 2017
test: run the addon tests last
Running the addon tests before the parallel, sequential, etc. tests
can be a problem if there is a bug in node that prevents the
addon tests from running properly. When the addon tests fail for any
reason, then none of the other tests (e.g. parallel, etc.)
are executed.

Running the addon tests last fixes this.

Refs: #12031
PR-URL: #12062
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
MylesBorins added a commit that referenced this pull request Jul 11, 2017
test: run the addon tests last
Running the addon tests before the parallel, sequential, etc. tests
can be a problem if there is a bug in node that prevents the
addon tests from running properly. When the addon tests fail for any
reason, then none of the other tests (e.g. parallel, etc.)
are executed.

Running the addon tests last fixes this.

Refs: #12031
PR-URL: #12062
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
@MylesBorins MylesBorins referenced this pull request Jul 18, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.