Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tern fails with ScanCode parsing error #1063

Closed
maxhbr opened this issue Oct 21, 2021 · 1 comment · Fixed by #1106
Closed

Tern fails with ScanCode parsing error #1063

maxhbr opened this issue Oct 21, 2021 · 1 comment · Fixed by #1106
Labels
bug Something went wrong

Comments

@maxhbr
Copy link

maxhbr commented Oct 21, 2021

Describe the bug
When scanning tomcat:10.0.6-jdk11-openjdk-buster with Tern, it fails after some time with the following output:

[...]
2021-10-21 14:17:33,554 - DEBUG - executor - Collecting file data...
2021-10-21 14:17:33,608 - DEBUG - common - Loading packages from cache: layer "139d3387e9"
2021-10-21 14:17:33,612 - DEBUG - common - Loading files from cache: layer "139d3387e9"
2021-10-21 14:17:33,625 - DEBUG - rootfs - Running command: sudo /usr/bin/scancode -ilpcu --quiet --timeout 300 -n 12 --json - /home/mhuber/.tern/temp/20da00f8ad0ba43d129dd08970e6649d63ca28412be31b0ff6d391862a085e68/contents
2021-10-21 14:18:56,208 - DEBUG - executor - Collecting file data...
2021-10-21 14:18:56,231 - DEBUG - common - Loading packages from cache: layer "bff998acef"
2021-10-21 14:18:56,297 - DEBUG - common - Loading files from cache: layer "bff998acef"
2021-10-21 14:18:58,745 - DEBUG - rootfs - Running command: sudo /usr/bin/scancode -ilpcu --quiet --timeout 300 -n 12 --json - /home/mhuber/.tern/temp/2306ce4f319a711c553242f8ab16abf709bc6a1853f122f50eafba0e2fa96487/contents
Traceback (most recent call last):
  File "/opt/tern.venv/lib/python3.8/site-packages/tern/extensions/scancode/executor.py", line 86, in filter_pkg_license
    return declared_license['license']
KeyError: 'license'

To Reproduce
Steps to reproduce the behavior:

  1. Run Tern with -x scancode on tomcat:10.0.6-jdk11-openjdk-buster and wait

Environment you are running Tern on
executed in dockerized environment. Observed both in container build from docker/Dockerfile.scancode and in a custom docker image.

$ tern --version
Tern version 2.8.0
   python version = 3.8.11 (default, Oct 16 2021, 17:24:33)
$ uname -a
Linux x1extremeG2 5.13.0-rc6 #1-NixOS SMP Sun Jun 13 21:43:10 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ scancode --version
/opt/scancode/src/cluecode/copyrights.py:3382: FutureWarning: Possible set difference at position 3
  remove_tags = re.compile(
ScanCode version 30.1.0
ScanCode Output Format version 1.0.0
SPDX License list version 3.14

tern.log
@rnjudge rnjudge added the bug Something went wrong label Jan 20, 2022
@rnjudge
Copy link
Contributor

rnjudge commented Jan 20, 2022

I can reproduce this. Working on a fix.

rnjudge added a commit to rnjudge/tern that referenced this issue Jan 21, 2022
@rnjudge
Fix scancode KeyError during license parsing
4b656c6 
When running Tern with Scancode, Tern tries to parse Scancode's
`declared_license` dictionary `license` key to determine the package
license. Sometimes this key is called `License` and sometimes the
`declared_license` dictionary is empty, however. In each of the
aforementioned cases Tern fails and throws a `KeyError`. This commit
fixes the error by checking for an empty dictionary before trying to
extract the license value and also checks for the `License` key in
addition to the `license` key.

Note that an issue[1] has been opened in the scancode repository asking
for clarity on whether the declared_license key should be `License` or
`license`. If any resolution comes from the scancode issue that has
been opened, it may make sense to revert parts of this change.

[1] nexB/scancode-toolkit#2803

Resolves tern-tools#1063

Signed-off-by: Rose Judge <rjudge@vmware.com>
rnjudge added a commit to rnjudge/tern that referenced this issue Jan 21, 2022
@rnjudge
Fix scancode KeyError during license parsing
bc96214 
When running Tern with Scancode, Tern tries to parse Scancode's
`declared_license` dictionary `license` key to determine the package
license. Sometimes this key is called `License` and sometimes the
`declared_license` dictionary is empty, however. In each of the
aforementioned cases Tern fails and throws a `KeyError`. This commit
fixes the error by checking for an empty dictionary before trying to
extract the license value and also checks for the `License` key in
addition to the `license` key.

Note that an issue[1] has been opened in the scancode repository asking
for clarity on whether the declared_license key should be `License` or
`license`. If any resolution comes from the scancode issue that has
been opened, it may make sense to revert parts of this change.

[1] nexB/scancode-toolkit#2803

Resolves tern-tools#1063

Signed-off-by: Rose Judge <rjudge@vmware.com>
@rnjudge rnjudge mentioned this issue Jan 21, 2022
Merged
rnjudge added a commit to rnjudge/tern that referenced this issue Jan 21, 2022
@rnjudge
Fix scancode KeyError during license parsing
08251cf 
When running Tern with Scancode, Tern tries to parse Scancode's
`declared_license` dictionary `license` key to determine the package
license. Sometimes this key is called `License` and sometimes the
`declared_license` dictionary is empty, however. In each of the
aforementioned cases Tern fails and throws a `KeyError`. This commit
fixes the error by checking for an empty dictionary before trying to
extract the license value and also checks for the `License` key in
addition to the `license` key.

Note that an issue[1] has been opened in the scancode repository asking
for clarity on whether the declared_license key should be `License` or
`license`. If any resolution comes from the scancode issue that has
been opened, it may make sense to revert parts of this change.

[1] nexB/scancode-toolkit#2803

Resolves tern-tools#1063

Signed-off-by: Rose Judge <rjudge@vmware.com>
rnjudge added a commit to rnjudge/tern that referenced this issue Jan 21, 2022
@rnjudge
Fix scancode KeyError during license parsing
c8ed91f 
When running Tern with Scancode, Tern tries to parse Scancode's
`declared_license` dictionary `license` key to determine the package
license. Sometimes this key is called `License` and sometimes the
`declared_license` dictionary is empty, however. In each of the
aforementioned cases Tern fails and throws a `KeyError`. This commit
fixes the error by checking for an empty dictionary before trying to
extract the license value and also checks for the `License` key in
addition to the `license` key.

Note that an issue[1] has been opened in the scancode repository asking
for clarity on whether the declared_license key should be `License` or
`license`. If any resolution comes from the scancode issue that has
been opened, it may make sense to revert parts of this change.

[1] nexB/scancode-toolkit#2803

Resolves tern-tools#1063

Signed-off-by: Rose Judge <rjudge@vmware.com>
nishakm pushed a commit that referenced this issue Jan 21, 2022
@rnjudge
Fix scancode KeyError during license parsing
ce5c763 
When running Tern with Scancode, Tern tries to parse Scancode's
`declared_license` dictionary `license` key to determine the package
license. Sometimes this key is called `License` and sometimes the
`declared_license` dictionary is empty, however. In each of the
aforementioned cases Tern fails and throws a `KeyError`. This commit
fixes the error by checking for an empty dictionary before trying to
extract the license value and also checks for the `License` key in
addition to the `license` key.

Note that an issue[1] has been opened in the scancode repository asking
for clarity on whether the declared_license key should be `License` or
`license`. If any resolution comes from the scancode issue that has
been opened, it may make sense to revert parts of this change.

[1] nexB/scancode-toolkit#2803

Resolves #1063

Signed-off-by: Rose Judge <rjudge@vmware.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something went wrong
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix scancode KeyError during license parsing rnjudge/tern
2 participants
@maxhbr @rnjudge