node-daily-coverity failing on test-packetnet-ubuntu1604-x64-1

[richardlau]

https://ci.nodejs.org/job/node-daily-coverity/ is failing on https://ci.nodejs.org/computer/test-packetnet-ubuntu1604-x64-1/

e.g. https://ci.nodejs.org/job/node-daily-coverity/1421/console

+ getconf _NPROCESSORS_ONLN
+ V=1 cov-build --dir cov-int make -j 8
/tmp/jenkins5860511461133452759.sh: 6: /tmp/jenkins5860511461133452759.sh: cov-build: not found

According to https://github.com/nodejs/build/blob/97b004a672d17820976ea8eac91b4dde8a6a4697/ansible/MANUAL_STEPS.md#jenkins-workspace installing Coverity Build Tool for Linux x64 is a manual step. I've logged into Coverity but I can't download the build tool as apparently I need admin access to at least one project:

https://ci.nodejs.org/job/node-daily-coverity/1422/ ran on test-packetnet-ubuntu1604-x64-2 and passed.

[github-actions]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

[richardlau]

This is still failing as the Coverity Build Tool isn't available on the machines running the job:

cov-build: not found

I've applied to join the Node.js project on Coverity. As I originally posted it appears you need to have admin access to at least one project to be able to download the Coverity Build Tool. I'm not sure who the owners of the Node.js project on Coverity are -- hopefully they're still involved and able to respond to the application.

[mhdawson]

@richardlau is this the issue where you can update who you reached out to (@jbergstroem ?)

[richardlau]

Yes, it is -- I sent an email to @jbergstroem (and copying the build email alias) back on 9 July but I think he was out of office at the time.

[mhdawson]

@richardlau might make sense to follow up with @jbergstroem again.

[richardlau]

@jbergstroem has added me to the Node.js project on Coverity so I can now access the downloads for the Coverity build tools. I've proceeded to download and install them on the three jenkins-workspace hosts and updated the job so that the PATH points to the installation (as per https://github.com/nodejs/build/blob/master/ansible/MANUAL_STEPS.md#jenkins-workspace).

I've also updated the hosts (via rerunning our Ansible playbook against them) to install gcc-8 and updated the job to use the select-compiler.sh script.

Test build: https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-coverity/1979/console

[richardlau]

Build failed 😞.

From the corresponding build log:
https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-coverity/1979/console

[WARNING] No files were emitted. This may be due to a problem with your configuration
or because no files were actually compiled by your build command.
Please make sure you have configured the compilers actually used in the compilation.
 For more details, please look at: 
    /home/iojs/build/workspace/node-daily-coverity/cov-int/build-log.txt
+ ./node --version
+ git rev-parse --short HEAD
+ VERSION=v17.0.0-pre-48655e17e1
+ FILE=nodejs-v17.0.0-pre-48655e17e1.tar.gz
+ tar -czf nodejs-v17.0.0-pre-48655e17e1.tar.gz cov-int
+ set +x
Build successfully submitted.
+ rm -rf nodejs-v17.0.0-pre-48655e17e1.tar.gz cov-int

Unfortunately the job has deleted the cov-int directory and the build-log.txt inside it.

[richardlau]

The issue is that by using a binary other than gcc/g++, e.g. we use gcc-8/g++-8, Coverity fails to detect the compiler invocations. The recommendation is to use cov-configure but when we try to do that in a build it attempts to write to /var/cov-analysis-linux64-2020.09/config which the iojs user cannot do:

[ERROR] Could not create the path to the generated config: /var/cov-analysis-linux64-2020.09/config/gcc-config-0: mkdir: Permission denied

I ran (as root)

/var/cov-analysis-linux64-2020.09/bin/cov-configure --comptype gcc --compiler gcc-8

but ended up with ERRORS like the following in cov-init/build-log.txt

[ERROR] This gcc-8 compiler command specifies the following arguments which should be marked as required in your configuration:
        -m64 -m64
Please reconfigure this compiler as follows:
        cov-configure -co gcc-8 -- -m64 -m64
or create a template configuration for it (recommended).

so I ran (again as root)

/var/cov-analysis-linux64-2020.09/bin/cov-configure -co gcc-8 -- -m64 -m64
/var/cov-analysis-linux64-2020.09/bin/cov-configure -co g++-8 -- -m64 -m64 -fno-rtti -fno-exceptions -std=gnu++14

and this appears to have resulted in a build where the upload was accepted by Coverity, which now currently shows

I'll wait and see if we get a plausible analysis and look at updating the instructions for setting this up. Or maybe try setting up on new system where gcc/g++ are new enough to compile current Node.js master (perhaps Ubuntu 20.04?).

[richardlau]

I did some local tests and running in Ubuntu 20.04 allows us to use the default gcc/g++ binaries (gcc 9) without additional Coverity build tool configuration. Furthermore we don't get the previously mentioned ERRORs in the Coverity build log. So Ubuntu 20.04 is a potential option.

However I've also been able to get the Coverity CI job to work in the existing Ubuntu 18.04 jenkins-workspace's using symbolic links and PATH manipulation. Essentially:

1. Run select-compiler to set up CC and CXX for the version of Node.js being built (e.g. master for the daily runs).
2. Filter out the ccache symlink directory from PATH.
3. Determine the path to the actual compiler binary (e.g. /usr/bin/g++) from the CC/CXX env vars set by select-compiler.
4. Create symlinks called gcc and g++ to the actual compiler binaries.
5. Set PATH so that it contains the path to the Coverity build tools (as per the existing MANUAL_STEPS instruction) and
   a. The ccache symlink directory (so ccache is enabled).
   b. Our directory containing the gcc and g++ symlinks. This needs to come between the ccache symlink directory and the directory containing the actual compiler binary (e.g. /usr/bin) so that ccache picks up our symlink.
6. Reset CC and CXX to gcc and g++ respectively -- Coverity will not detect ccache gcc (for example).

I've changed the CI job to do those steps. No further additional Coverity build tool configuration is required (I cleared and reinstalled the tool to check). A test build with this setup has completed, submitted to Coverity and is currently being analyzed:
https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-coverity/2015/console (v17.0.0-pre-73d5f8a843).

The ERRORs in the Coverity build log are gone and as a result we process more "units",
e.g. compare
https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-coverity/2015/console

Emitted 3260 C/C++ compilation units (99%) successfully
[WARNING] Recoverable errors were encountered during 203 of these C/C++ compilation units.

3260 C/C++ compilation units (99%) are ready for analysis
The cov-build utility completed successfully.

to this morning's daily:
https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-coverity/1997/console

Emitted 1189 C/C++ compilation units (99%) successfully
[WARNING] Recoverable errors were encountered during 203 of these C/C++ compilation units.

1189 C/C++ compilation units (99%) are ready for analysis
The cov-build utility completed successfully.

[richardlau]

Good news is that I think the Coverity CI build is now fixed (🎉). We may need to fix up the exclusion patterns on the Coverity analysis side, currently:
https://scan.coverity.com/projects/node-js?tab=analysis_settings

which doesn't cover all of the deps:

but that should probably be a separate issue.