Analysis phase fails

[marcomicera]

Describe the bug

The analysis phase hangs indefinitely when running it locally, and fails silently when running in the cloud.

How to Reproduce

The analysis branch contains a CI/CD pipeline that automatically deploys DGraph + QMSTR in a dedicated Kubernetes namespace.

However, this can also be reproduced manually by following the following guide:

1. Navigate to the project's root folder and export the following variable to avoid any misunderstanding:

   export QMSTR=$(pwd -P)

2. Deploy DGraph:

   cd $QMSTR/deploy
   kubectl apply -k dgraph

3. Launch QMSTR.
   • In the cluster:
     1. Set the 4 image tags to be used in deploy/qmstr/kustomization.yam to sha-550b3eb.
     2. Deploy QMSTR:

        cd $QMSTR/deploy
        kubectl apply -k qmstr
        

     3. Observe the QMSTR master logs:

        kubectl logs --follow $(kubectl get pods --selector job-name=qmstr -o=name) qmstr-master

   • Locally (so that it is possible to debug the QMSTR master):
     1. Port-forward the following DGraph's Pod ports:

        kubectl port-forward statefulset/dgraph 5080:5080
        kubectl port-forward statefulset/dgraph 6080:6080
        kubectl port-forward statefulset/dgraph 8000:8000
        kubectl port-forward statefulset/dgraph 8080:8080
        kubectl port-forward statefulset/dgraph 9080:9080

     2. Clone a sample Maven project into a directory (e.g., $BUILDROOT/project):

        git clone https://github.com/okteto/java-maven-getting-started $BUILDROOT/project

     3. Perform the POM patch manually:

        mv $BUILDROOT/project/pom.xml $QMSTR/lib/pyqmstr/pom-patch
        python3 insert-build-plugin.py qmstr-maven-plugin.xml
        cat pom.xml | grep qmstr # should be not empty
        mv $QMSTR/lib/pyqmstr/pom-patch/pom.xml $BUILDROOT/project

     4. Edit deploy/qmstr/base/qmstr.yaml:
        • dbAddress: "localhost:9080"
        • workdir: "$BUILDROOT/project" (substitute manually)
        • resultfile: "$BUILDROOT/scancode.json" (substitute manually)
     5. Run the QMSTR server, possibly with a debugger, using
        • SERVER_BUILDPATH=$BUILDROOT as an environment variable, and
        • --config $QMSTR/deploy/qmstr/base/qmstr.yaml as a command-line argument.
     6. Launch an interactive QMSTR client container:

        docker run --tty -ti --rm --name clienttest \
          --env QMSTRADDRENV="host.docker.internal:50051" \
          --env BUILDROOT="/var/qmstr/buildroot" \
          --env QMSTR_MASTER="host.docker.internal:50051" \
          --mount type=bind,source=$BUILDROOT,target=/var/qmstr/buildroot \
          --network host \
          --user root \
          --entrypoint "/bin/sh" \
          endocodeci/qmstr-client-mvn:sha-550b3eb

     7. Start the building phase inside the QMSTR client container:

        mvn package -Dmaven.test.skip=true

        It should run successfully to completion.
     8. Start the analysis phase:

        export PATH=/usr/local/bin:$PATH # because you're root, so you can install packages
        qmstrctl analyze --verbose

Expected behavior

The analysis phase should run to completion, augmenting the build graph with the proper metadata.

Failing master logs

2020/09/10 14:51:49 Switching to phase 2
2020/09/10 14:51:49 Waiting for inserts
2020/09/10 14:51:49 Pending inserts [...]
2020/09/10 14:52:03 Closed insert queue
2020/09/10 14:52:03 Analysis activated
2020/09/10 14:52:04 Running analyzer scancode-analyzer ...
2020/09/10 14:52:04 Opened insert queue
2020/09/10 14:52:04 scancode-analyzer failed with:

2020/09/10 14:52:04 Waiting for inserts
2020/09/10 14:52:04 Closed insert queue
2020/09/10 14:52:04 Server entered failure phase due to Analyzer scancode-analyzer failed

The QMSTR client just reports the last error message.

Additional context

Please do not assume that my debugging guide and general setup is correct: the bug might be hiding somewhere there as well.

[marcomicera]

scancode-analyzer seems to be launching correctly (after forcing its working directory 19495b8), but its process doesn't do anything: 0.0% CPU, 0.1% RAM.

[marcomicera]

strace only shows condition variable primitives being called from time to time.

[marcomicera]

Getting the following after having installed scancode-analyzer, scancode and python on the qmstr-master image:

scancode-analyzer failed with:
    --> Calling [scancode --full-root -l -c --json /var/qmstr/buildroot/scancode.json /var/qmstr/buildroot/project]
    --> Scancode-toolkit failed fork/exec /usr/local/bin/scancode: no such file or directory
    --> scancode return code was not found. Something went seriously wrong.
Waiting for inserts
Closed insert queue
Server entered failure phase due to Analyzer scancode-analyzer failed: exit status 1 (exit status 1)

This line is the cause.

[marcomicera]

Installed scancode with Python3.6 (!) in the qmstr-master image.
Currently getting this error with this example repository:

Analyzer scancode-analyzer failed: exit status 2 (exit status 2)
scancode-analyzer failed with:
	--> Calling [scancode --full-root -l -c --json /var/qmstr/buildroot/scancode.json /var/qmstr/buildroot/project]
	--> Setup plugins...
	--> Collect file inventory...
	--> Removing temporary files...done.
	--> Usage: scancode [OPTIONS] <OUTPUT FORMAT OPTION(s)> <input>...
	--> Try 'scancode --help' for help on options and arguments.
	-->
	--> Error: Invalid value: ERROR: failed to collect codebase at: '/var/qmstr/buildroot/project'
	--> Traceback (most recent call last):
	-->   File "/usr/local/bin/scancode/src/scancode/cli.py", line 860, in run_scan
	-->     max_in_memory=max_in_memory
	-->   File "/usr/local/bin/scancode/src/scancode/resource.py", line 284, in __init__
	-->     self._populate()
	-->   File "/usr/local/bin/scancode/src/scancode/resource.py", line 443, in _populate
	-->     parent = parent_by_loc.pop(top)
	--> KeyError: '/var/qmstr/buildroot/project/target'
	-->
	--> Scancode-toolkit failed exit status 2

[marcomicera]

It's not a matter of missing packages.

[marcomicera]

Related issues.

[marcomicera]

Bug is introduced by the scancode-analyzer wrapper.

[marcomicera]

Getting the following after having installed the libgomp1 package:

Analyzer scancode-analyzer failed: exit status 1 (exit status 1)
scancode-analyzer failed with:
	--> Calling [scancode --full-root --license --copyright --json-pp /var/qmstr/buildroot/scancode.json /var/qmstr/buildroot/project]
	--> Setup plugins...
	--> Collect file inventory...
	--> Scan files for: licenses, copyrights with 1 process(es)...
	--> Scanning done.
	--> Summary:        licenses, copyrights with 1 process(es)
	--> Errors count:   0
	--> Scan Speed:     0.00 files/sec.
	--> Initial counts: 1 resource(s): 0 file(s) and 1 directorie(s)
	--> Final counts:   1 resource(s): 0 file(s) and 1 directorie(s)
	--> Timings:
	-->   scan_start: 2020-09-18T113513.017906
	-->   scan_end:   2020-09-18T113622.921141
	-->   setup_scan:licenses: 69.68s
	-->   setup: 69.68s
	-->   scan: 0.21s
	-->   total: 69.91s
	--> Removing temporary files...done.
	--> unexpected end of JSON input

[marcomicera]

workdir should have been set to /var/qmstr/buildroot. Still getting the same "unexpected end of JSON input" error.

[marcomicera]

It actually produces the JSON result, but the master crashes anyway.

[fullsushidev]

Quick question: was it working with the flag --json?

[marcomicera]

Quick question: was it working with the flag --json?

Good guess, that was the problem! Analysis phase completed in the analysis branch.