Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

N-API test failure #173

Closed
targos opened this issue Sep 14, 2020 · 37 comments
Closed

N-API test failure #173

targos opened this issue Sep 14, 2020 · 37 comments

Comments

@targos
Copy link
Member

targos commented Sep 14, 2020

https://ci.nodejs.org/job/node-test-commit-linux/37061/nodes=ubuntu1804-64/console

10:35:08 not ok 2879 js-native-api/test_object/test
10:35:08   ---
10:35:08   duration_ms: 0.330
10:35:08   severity: crashed
10:35:08   exitcode: -11
10:35:08   stack: |-
10:35:08   ...
@targos
Copy link
Member Author

targos commented Sep 14, 2020

/cc @nodejs/n-api

@mhdawson
Copy link
Member

@targos
Copy link
Member Author

targos commented Sep 15, 2020

I cannot reproduce on my system (CentOS 8, with GCC 8.3.1)

@mhdawson
Copy link
Member

This did not reproduce on my system either (Ubuntu 18.04.4, GCC 7.5.0)

@mhdawson
Copy link
Member

I think one of the machines in the build matched the ubuntu/gcc level that I have:

03:11:46 Detected C++ compiler (CXX=ccache g++) version: 7.5.0
03:11:46 Detected C compiler (CC=ccache gcc) version: 7.5.0

from https://ci.nodejs.org/job/node-test-commit-linux/nodes=ubuntu1804-64/37061/consoleFull

So maybe we should re-run after the security release to see if it re-occurs

@targos
Copy link
Member Author

targos commented Sep 15, 2020

It's been occuring for some time (I haven't checked when it started to fail). Here's an other example from one week ago: https://ci.nodejs.org/view/MyJobs/job/node-test-commit-node-v8/600/

@mhdawson
Copy link
Member

So its intermittent or fails every time in the CI ?

@mhdawson
Copy link
Member

Also to confirm, the job just checks out the canary branch (the default) of node-v8 and does make test ?

@targos
Copy link
Member Author

targos commented Sep 15, 2020

It fails every time in CI. History: https://ci.nodejs.org/view/MyJobs/job/node-test-commit-node-v8/
The job triggers a node-test-commit subjob with the canary branch of nodejs/node-v8

@mhdawson
Copy link
Member

Strange since that should match what we've run. I wonder if it would make sense to clear the workspace and make sure it recreates.

@mhdawson
Copy link
Member

@targos are you ok if we clear the workspace to see if that makes any difference?

@targos
Copy link
Member Author

targos commented Sep 17, 2020

I'm ok

@mhdawson
Copy link
Member

Ok, have cleared the workspace for https://ci.nodejs.org/job/node-test-commit-node-v8/. We'll see if that makes any difference.

@gabrielschulhof
Copy link

I can't reproduce this.

@gabrielschulhof
Copy link

Looks like Ubuntu 16.04 64-bit with gcc-7.5.0.

@mhdawson
Copy link
Member

Using test-joyent-ubuntu1804-x64-1 on which we saw the failure recently to investigate

@mhdawson
Copy link
Member

Compiling in directory: https://github.com/nodejs/node-v8.git

@mhdawson
Copy link
Member

did not recreate with a make node followed by make test, so likely means that its related to config set by the CI job, for example compiler or something else

@mhdawson
Copy link
Member

Called set-compiler but not the ccache config still did not recreate, but not much seemed to be compiled either. Having said that it did not seem to have changed the compiler either

@mhdawson
Copy link
Member

mhdawson commented Sep 25, 2020

Double checked last failure was on https://ci.nodejs.org/job/node-test-commit-linux/nodes=ubuntu1804-64/37204/

@mhdawson
Copy link
Member

Remaining thing that is not like ci would be the ccache. Next week I'll create a duplicate job that would only run on the same machine and validate it fails under CI, and if so then try to remove the part of the job which configures ccache.

@mhdawson
Copy link
Member

Create this job: https://ci.nodejs.org/job/node-test-commit-linux-mhdawson/ as clone of main job which only runs on test-joyent-ubuntu1804-x64-1 and uses node-v8 and the canary branch.

Run to see if it recreates: https://ci.nodejs.org/job/node-test-commit-linux-mhdawson/2/console

@mhdawson
Copy link
Member

Unfortunately that job is not working, need to look further

@mhdawson
Copy link
Member

Recreated job as https://ci.nodejs.org/job/node-test-commit-linux-mhdawson/, does not default to node-v8/canary, but seems to be running having specified that in the build parameters

@mhdawson
Copy link
Member

Tried recreating through a clone of the job. Tried the follwing:

  1. run the job specifying node-v8/canary - no recreate
  2. first run the job for node/master, followed with run for node-v8/canary to more closely match what we see in the community runs - no recreate

@mhdawson
Copy link
Member

Looking at the build jobs they look pretty much the same. The only thing I can see is that my new job does not have the NODE_MAJOR_VERSION and IGNORE_FLAKY_TESTS parameters defined. I've added those and kicked off another run

@mhdawson
Copy link
Member

Still no recreate, the only other thing I can think of trying is to clear the workspace for node-test-commit-linux but that would definitely have an impact on the next run.

@mhdawson
Copy link
Member

mhdawson commented Oct 1, 2020

Wiping workspace for node-test-commit-linux

@mhdawson
Copy link
Member

mhdawson commented Oct 1, 2020

node-v8 run to see if wiping workspace had any effect: https://ci.nodejs.org/view/All/job/node-test-commit-node-v8/622/

@mhdawson
Copy link
Member

mhdawson commented Oct 1, 2020

second run as first one seems to fail, missing some config after workspace clean https://ci.nodejs.org/job/node-test-commit-node-v8/623/

@mhdawson
Copy link
Member

mhdawson commented Oct 1, 2020

@mhdawson
Copy link
Member

mhdawson commented Oct 1, 2020

Seems like you have to run 3 times after workspace clear, third one now seems to be building now.

@mhdawson
Copy link
Member

mhdawson commented Oct 1, 2020

Looks to me like the linux job on ubuntu 1804 passed in this run: https://ci.nodejs.org/job/node-test-commit-node-v8/624/

@mhdawson
Copy link
Member

mhdawson commented Oct 1, 2020

Lets see if it passes in the next regularly scheduled canary builds, if so it seems like the workspace clean removed whatever was causing the problem.

@targos
Copy link
Member Author

targos commented Oct 2, 2020

Thanks for looking into it @mhdawson ! Here is today's run: https://ci.nodejs.org/view/MyJobs/job/node-test-commit-node-v8/626/

@mhdawson
Copy link
Member

mhdawson commented Oct 2, 2020

@targos that link seems to 404 for me. Launched this one https://ci.nodejs.org/view/MyJobs/job/node-test-commit-node-v8/628/

@mhdawson
Copy link
Member

mhdawson commented Oct 2, 2020

Ok test_object passed in the last build I started as well. @targos I'm going to close this as I think it's resolved. Please re-open if it recreates.

@mhdawson mhdawson closed this as completed Oct 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants