Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NullPointerException during parallel scans #1406

Closed
NicolasToussaint opened this issue Mar 27, 2019 · 10 comments
Closed

NullPointerException during parallel scans #1406

NicolasToussaint opened this issue Mar 27, 2019 · 10 comments
Labels
bug Issues that are considered to be bugs scanner About the scanner tool

Comments

@NicolasToussaint
Copy link

Running current master: 0f07ac1

I ran into a NPE while running two parallel scans.
Only the second crashed (the first might have hung, I had to stop it by hand).

Exception:

11:35:52.172 INFO  - Running ScannerDetails(name=ScanCode, version=2.9.7, configuration=--copyright --license --ignore *.ort.yml --info --strip-root --timeout 300 --ignore HERE_NOTICE --ignore META-INF/DEPENDENCIES --json-pp, data={}) on directory '/data/ort_download/maven-samples_ScanCode_20190327_113542/Maven/com.example.maven-samples/parent/1.0-SNAPSHOT'.
11:35:52.174 INFO  - Running '/home/<user>/.local/bin/scancode --copyright --license --ignore *.ort.yml --info --strip-root --timeout 300 --ignore HERE_NOTICE --ignore META-INF/DEPENDENCIES --processes 1 /data/ort_download/maven-samples_ScanCode_20190327_113542/Maven/com.example.maven-samples/parent/1.0-SNAPSHOT --json-pp /data/ort_home/maven-samples/030-scan_ScanCode_20190327_113542/native-scan-results/Maven/com.example.maven-samples/parent/1.0-SNAPSHOT/scan-results_ScanCode.json' in '/home/<user>/ORT/_tests/maven-samples'...
11:36:17.615 INFO  - Scanning package Maven:javax.servlet.jsp:jsp-api:2.2 in thread 'scan' (2/8).
11:36:17.948 INFO  - Trying to download source code for 'Maven:javax.servlet.jsp:jsp-api:2.2'.
11:36:18.160 INFO  - Trying to download 'Maven:javax.servlet.jsp:jsp-api:2.2' sources to '/data/ort_download/maven-samples_ScanCode_20190327_113542/Maven/javax.servlet.jsp/jsp-api/2.2' from VCS...
11:36:18.161 INFO  - Using VcsInfo(type=svn, url=https://jsp.dev.java.net/svn/jsp/tags/jsp-api-2.2, revision=, resolvedRevision=null, path=, data={}).
11:36:18.161 INFO  - Detected VCS type 'Subversion' from type name 'svn'.
11:36:18.499 INFO  - Running 'svn --version' in '/home/<user>/ORT/_tests/maven-samples'...
11:36:20.299 INFO  - Using Subversion version 1.9.3.
11:36:20.347 INFO  - Running 'svn checkout https://jsp.dev.java.net/svn/jsp/tags/jsp-api-2.2 --depth empty .' in '/data/ort_download/maven-samples_ScanCode_20190327_113542/Maven/javax.servlet.jsp/jsp-api/2.2'...
11:36:20.810 INFO  - Trying to download source artifact for 'Maven:javax.servlet.jsp:jsp-api:2.2' from https://repo.maven.apache.org/maven2/javax/servlet/jsp/jsp-api/2.2/jsp-api-2.2-sources.jar...
11:36:22.849 INFO  - Successfully downloaded source artifact for 'Maven:javax.servlet.jsp:jsp-api:2.2' to '/data/ort_download/maven-samples_ScanCode_20190327_113542/Maven/javax.servlet.jsp/jsp-api/2.2'...
11:36:22.849 INFO  - Running ScannerDetails(name=ScanCode, version=2.9.7, configuration=--copyright --license --ignore *.ort.yml --info --strip-root --timeout 300 --ignore HERE_NOTICE --ignore META-INF/DEPENDENCIES --json-pp, data={}) on directory '/data/ort_download/maven-samples_ScanCode_20190327_113542/Maven/javax.servlet.jsp/jsp-api/2.2'.
11:36:22.852 INFO  - Running '/home/<user>/.local/bin/scancode --copyright --license --ignore *.ort.yml --info --strip-root --timeout 300 --ignore HERE_NOTICE --ignore META-INF/DEPENDENCIES --processes 1 /data/ort_download/maven-samples_ScanCode_20190327_113542/Maven/javax.servlet.jsp/jsp-api/2.2 --json-pp /data/ort_home/maven-samples/030-scan_ScanCode_20190327_113542/native-scan-results/Maven/javax.servlet.jsp/jsp-api/2.2/scan-results_ScanCode.json' in '/home/<user>/ORT/_tests/maven-samples'...
Exception in thread "main" java.lang.NullPointerException
    at com.here.ort.scanner.scanners.ScanCode.generateSummary$scanner(ScanCode.kt:287)
    at com.here.ort.scanner.scanners.ScanCode.scanPath(ScanCode.kt:258)
    at com.here.ort.scanner.LocalScanner.scanPackage(LocalScanner.kt:306)
    at com.here.ort.scanner.LocalScanner.access$scanPackage(LocalScanner.kt:70)
    at com.here.ort.scanner.LocalScanner$scanPackages$2$invokeSuspend$$inlined$map$lambda$1$2.invokeSuspend(LocalScanner.kt:183)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
    at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: java.lang.NullPointerException
        ... 13 more
@sschuberth sschuberth added bug Issues that are considered to be bugs scanner About the scanner tool labels Mar 27, 2019
@kallesoranko
Copy link

I have also faced this issue of having NPE. I do not know is it because of parallel scans. It happens for some components and seems quite random to me. As a workaround I have just removed that component from analyzer-result.yml to be able to run scanner over the project, but of course this is not a sustainable solution.

Here is the stacktrace:

Exception in thread "main" java.lang.NullPointerException
	at com.here.ort.scanner.scanners.ScanCode.generateSummary$scanner(ScanCode.kt:347)
	at com.here.ort.scanner.scanners.ScanCode.scanPath(ScanCode.kt:323)
	at com.here.ort.scanner.LocalScanner.scanPackage(LocalScanner.kt:306)
	at com.here.ort.scanner.LocalScanner.access$scanPackage(LocalScanner.kt:70)
	at com.here.ort.scanner.LocalScanner$scanPackages$2$invokeSuspend$$inlined$map$lambda$1$2.invokeSuspend(LocalScanner.kt:183)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:238)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Also one thing that I noticed that before this NPE, there is usually a crash in ScanCode like:

Traceback (most recent call last):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\Scripts\scancode-script.py", line 11, in <module>
    load_entry_point('scancode-toolkit', 'console_scripts', 'scancode')()
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\site-packages\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\z00407wp\scancode-toolkit-2.9.7\src\scancode\utils.py", line 79, in main
    standalone_mode=standalone_mode, **extra)
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\site-packages\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\site-packages\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\z00407wp\scancode-toolkit-2.9.7\src\scancode\cli.py", line 717, in scancode
    quiet=quiet, verbose=verbose, kwargs=kwargs,
  File "c:\users\z00407wp\scancode-toolkit-2.9.7\src\scancode\cli.py", line 881, in run_scanners
    with_timing=timing, progress_manager=progress_manager)
  File "c:\users\z00407wp\scancode-toolkit-2.9.7\src\scancode\cli.py", line 1027, in scan_codebase
    pool.terminate()
  File "c:\python27\Lib\multiprocessing\pool.py", line 465, in terminate
    self._terminate()
  File "c:\python27\Lib\multiprocessing\util.py", line 207, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "c:\python27\Lib\multiprocessing\pool.py", line 513, in _terminate_pool
    p.terminate()
  File "c:\python27\Lib\multiprocessing\process.py", line 137, in terminate
    self._popen.terminate()
  File "c:\python27\Lib\multiprocessing\forking.py", line 312, in terminate
    _subprocess.TerminateProcess(int(self._handle), TERMINATE)
WindowsError: [Error 5] Access is denied

This crash with ScanCode can also be something else. For example one other that I have seen is:

ERROR: failed to collect codebase at: u'C:\\Users\\z00407wp\\Desktop\\ORT\\devISF06\\Poct1\\scanner\\downloads\\Maven\\com.android.support.test.espresso\\espresso-core\\3.0.0'
Traceback (most recent call last):
  File "c:\users\z00407wp\scancode-toolkit-2.9.7\src\scancode\cli.py", line 677, in scancode
    max_in_memory=max_in_memory
  File "c:\users\z00407wp\scancode-toolkit-2.9.7\src\scancode\resource.py", line 243, in __init__
    self._populate()
  File "c:\users\z00407wp\scancode-toolkit-2.9.7\src\scancode\resource.py", line 397, in _populate
    for top, dirs, files in os_walk(root.location, topdown=True, onerror=err):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Users\z00407wp\scancode-toolkit-2.9.7\lib\os.py", line 281, in walk
    onerror(err)
  File "c:\users\z00407wp\scancode-toolkit-2.9.7\src\scancode\resource.py", line 355, in err
    ('ERROR: cannot populate codebase: %(_error)r\n' % _error)
TypeError: format requires a mapping

I also attach full debug logs for both cases.
scanner1.log
scanner2.log

I am running ORT master branch at 5648c40

@sschuberth
Copy link
Member

Thanks for the additions, @kallesoranko. Could you please report the ScanCode crashes also to https://github.com/nexB/scancode-toolkit/issues and reference your report(s) from here?

This was referenced Aug 1, 2019
Closed
Closed
@kallesoranko kallesoranko mentioned this issue Aug 21, 2019
Closed
@sschuberth
Copy link
Member

@NicolasToussaint, @kallesoranko, do you still see this with current ORT and ScanCode 3.0.2?

@NicolasToussaint
Copy link
Author

Sorry @sschuberth, haven't updated ORT source code in a little while now :-/
But it's definitely in my todo-list :)

@kallesoranko
Copy link

@sschuberth I get same error (WindowsError: [Error 5] Access is denied) also with ScanCode 3.0.2 and that makes ORT fail

@sschuberth
Copy link
Member

@kallesoranko, please note that the "WindowsError: [Error 5] Access is denied" error is another issue than this one.

This issue is about a NullPointerException in ORT's ScanCode.generateSummary, probably caused by some race condition with parallel scans. The root cause is likely in ORT.

The NullPointerException in ORT's ScanCode.getFileCount triggered by ScanCode's "WindowsError: [Error 5] Access is denied" is handled in #1731. For that, the root cause is likely in ScanCode, not in ORT.

@kallesoranko
Copy link

@sschuberth you are correct and I haven't faced NPE in ScanCode.generateSummary after ScanCode update. However I updated ORT and ScanCode just yesterday and ran just few scans. Maybe we can keep this ticket for a while longer to be sure that the bug is gone.

@sschuberth
Copy link
Member

Maybe we can keep this ticket for a while longer to be sure that the bug is gone.

Sure. I'd like also @NicolasToussaint to confirm that.

@NicolasToussaint
Copy link
Author

Haven't had any parallel scan NPE, but I also avoid running parallel scans ...
I'll just close this issue, I suppose a new one can be opened if a new problem arises.

@sschuberth
Copy link
Member

I'll just close this issue, I suppose a new one can be opened if a new problem arises.

Right, or just re-open, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Issues that are considered to be bugs scanner About the scanner tool
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sschuberth @NicolasToussaint @kallesoranko