Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scancode not working any more with Fossology (Installer installs latest/unpinned version of Scancode) #2329

Closed
alpianon opened this issue Oct 5, 2022 · 2 comments · Fixed by #2436
Closed

Scancode not working any more with Fossology (Installer installs latest/unpinned version of Scancode) #2329

alpianon opened this issue Oct 5, 2022 · 2 comments · Fixed by #2436

Comments

@alpianon
Copy link
Contributor

alpianon commented Oct 5, 2022 •
edited

Description

I installed Fossology v4.1.0 from sources some time ago and it is working well, even with Scancode. The installed (working) version of Scancode was 30.1.0.

However, I created a new instance of Fossology v4.1.0 (always installing it from sources) and Scancode is not working any more with Fossology Scancode Agent:

immagine

click here to see the job log 
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "ERROR: failed to run output plugin: template:"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "Traceback (most recent call last):"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/scancode/cli.py", line 1079, in run_codebase_plugins"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    plugin.process_codebase(codebase, **kwargs)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/formattedcode/output_html.py", line 102, in process_codebase"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    write_templated(output_file, results, version, template_loc)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/formattedcode/output_html.py", line 113, in write_templated"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    for template_chunk in generate_output(results, version, template):"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/environment.py", line 1354, in generate"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    yield self.environment.handle_exception()"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/environment.py", line 936, in handle_exception"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    raise rewrite_traceback_stack(source=source)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/local/share/fossology/scancode/agent/scancode_template.html", line 64, in top-level template code"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    "value": {{ data.value|tojson }},"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/filters.py", line 1688, in do_tojson"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    return htmlsafe_json_dumps(value, dumps=dumps, **kwargs)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/utils.py", line 658, in htmlsafe_json_dumps"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    dumps(obj, **kwargs)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/__init__.py", line 238, in dumps"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    **kw).encode(obj)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/encoder.py", line 199, in encode"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    chunks = self.iterencode(o, _one_shot=True)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/encoder.py", line 257, in iterencode"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    return _iterencode(o, 0)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/encoder.py", line 179, in default"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    raise TypeError(f'Object of type {o.__class__.__name__} '"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "TypeError: Object of type Undefined is not JSON serializable"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "ERROR: failed to run output plugin: template:"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "Traceback (most recent call last):"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/scancode/cli.py", line 1079, in run_codebase_plugins"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    plugin.process_codebase(codebase, **kwargs)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/formattedcode/output_html.py", line 102, in process_codebase"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    write_templated(output_file, results, version, template_loc)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/formattedcode/output_html.py", line 113, in write_templated"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    for template_chunk in generate_output(results, version, template):"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/environment.py", line 1354, in generate"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    yield self.environment.handle_exception()"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/environment.py", line 936, in handle_exception"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    raise rewrite_traceback_stack(source=source)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/local/share/fossology/scancode/agent/scancode_template.html", line 64, in top-level template code"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    "value": {{ data.value|tojson }},"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/filters.py", line 1688, in do_tojson"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    return htmlsafe_json_dumps(value, dumps=dumps, **kwargs)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/utils.py", line 658, in htmlsafe_json_dumps"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    dumps(obj, **kwargs)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/__init__.py", line 238, in dumps"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    **kw).encode(obj)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/encoder.py", line 199, in encode"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    chunks = self.iterencode(o, _one_shot=True)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/encoder.py", line 257, in iterencode"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    return _iterencode(o, 0)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/encoder.py", line 179, in default"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    raise TypeError(f'Object of type {o.__class__.__name__} '"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "TypeError: Object of type Undefined is not JSON serializable"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "ERROR: failed to run output plugin: template:"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "Traceback (most recent call last):"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/scancode/cli.py", line 1079, in run_codebase_plugins"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    plugin.process_codebase(codebase, **kwargs)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/formattedcode/output_html.py", line 102, in process_codebase"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    write_templated(output_file, results, version, template_loc)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/formattedcode/output_html.py", line 113, in write_templated"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    for template_chunk in generate_output(results, version, template):"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/environment.py", line 1354, in generate"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    yield self.environment.handle_exception()"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/environment.py", line 936, in handle_exception"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    raise rewrite_traceback_stack(source=source)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/local/share/fossology/scancode/agent/scancode_template.html", line 64, in top-level template code"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    "value": {{ data.value|tojson }},"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/filters.py", line 1688, in do_tojson"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    return htmlsafe_json_dumps(value, dumps=dumps, **kwargs)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/utils.py", line 658, in htmlsafe_json_dumps"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    dumps(obj, **kwargs)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/__init__.py", line 238, in dumps"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    **kw).encode(obj)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/encoder.py", line 199, in encode"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    chunks = self.iterencode(o, _one_shot=True)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/encoder.py", line 257, in iterencode"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    return _iterencode(o, 0)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/encoder.py", line 179, in default"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    raise TypeError(f'Object of type {o.__class__.__name__} '"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "TypeError: Object of type Undefined is not JSON serializable"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "NOTICE scancode_wrapper.cc.151: No license found"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "NOTICE scancode_wrapper.cc.151: No license found"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "NOTICE scancode_wrapper.cc.151: No license found"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "NOTICE scancode_wrapper.cc.151: No license found"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "ERROR: failed to run output plugin: template:"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "Traceback (most recent call last):"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/scancode/cli.py", line 1079, in run_codebase_plugins"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    plugin.process_codebase(codebase, **kwargs)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/formattedcode/output_html.py", line 102, in process_codebase"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    write_templated(output_file, results, version, template_loc)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/formattedcode/output_html.py", line 113, in write_templated"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    for template_chunk in generate_output(results, version, template):"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/environment.py", line 1354, in generate"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    yield self.environment.handle_exception()"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/environment.py", line 936, in handle_exception"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    raise rewrite_traceback_stack(source=source)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/local/share/fossology/scancode/agent/scancode_template.html", line 64, in top-level template code"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    "value": {{ data.value|tojson }},"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/filters.py", line 1688, in do_tojson"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    return htmlsafe_json_dumps(value, dumps=dumps, **kwargs)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/home/fossy/pythondeps/jinja2/utils.py", line 658, in htmlsafe_json_dumps"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    dumps(obj, **kwargs)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/__init__.py", line 238, in dumps"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    **kw).encode(obj)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/encoder.py", line 199, in encode"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    chunks = self.iterencode(o, _one_shot=True)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/encoder.py", line 257, in iterencode"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    return _iterencode(o, 0)"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "  File "/usr/lib/python3.7/json/encoder.py", line 179, in default"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "    raise TypeError(f'Object of type {o.__class__.__name__} '"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "TypeError: Object of type Undefined is not JSON serializable"
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: "FATAL scancode_wrapper.cc.106: could not execute scancode command: PYTHONPATH='/home/fossy/pythondeps/' SCANCODE_CACHE=/var/local/cache/fossology/scancode /home/fossy/pythondeps/bin/scancode -lceu --custom-output - --custom-template scancode_template.html /srv/fossology/repository/localhost/files/64/78/7f/64787face2d527df6c78ba8754eb54c53dda6ac2.315c9e3e7dd1d24aeb9d6bc55b7de5fd.1847 --quiet  --license-text --license-score 50 "
2022-10-04 18:28:59 scancode [0] :: JOB[13].scancode[7818.localhost]: agent failed with error code 1
2022-10-04 18:29:00 scancode [0] :: JOB[13].scancode[7818.localhost]: agent failed, code: 1

I checked the installed Scancode version, which is now 31.1.0.

I tried to downgrade Scancode to version 30.1.0, and it works.

click for the commands I used to downgrade Scancode 
# as root

su - fossy -c 'PYTHONPATH="$HOME/pythondeps" pip3 uninstall -y attrs banal binaryornot boolean charset-normalizer click colorama commoncode cryptography debian-inspector dparse extractcode extractcode-7z extractcode-libarchive fasteners fingerprints ftfy future gemfileparser html5lib idna importlib-metadata isodate jaraco.functools javaproperties Jinja2 jsonstreams license-expression lxml MarkupSafe more-itertools normality packaging parameter-expansion-patched pdfminer.six pkginfo pluggy plugincode ply publicsuffix2 pygmars Pygments pymaven-patch pyparsing rdflib requests scancode-toolkit setuptools soupsieve spdx-tools text-unidecode toml typecode typecode-libmagic urllib3 wcwidth webencodings wheel'

su - fossy -c 'PYTHONPATH="$HOME/pythondeps" pip3 install --target=$HOME/pythondeps setuptools==62.1.0 wheel==0.37.1'

su - fossy -c 'PYTHONPATH="$HOME/pythondeps" pip3 install --target=$HOME/pythondeps attrs==21.4.0 banal==1.0.6 binaryornot==0.4.4 charset-normalizer==2.0.12 click==8.1.3 colorama==0.4.4 commoncode==30.2.0 cryptography==2.6.1 debian-inspector==30.0.0 dparse==0.5.1 extractcode==30.0.0 extractcode-7z==16.5.210531 extractcode-libarchive==3.5.1.210531 fasteners==0.17.3 fingerprints==1.0.3 ftfy==6.1.1 future==0.18.2 gemfileparser==0.8.0 html5lib==1.1 idna==2.6 importlib-metadata==4.11.3 isodate==0.6.1 jaraco.functools==3.5.0 javaproperties==0.8.1 Jinja2==2.10 jsonstreams==0.6.0 license-expression==21.6.14 lxml==4.8.0 MarkupSafe==1.1.0 more-itertools==8.12.0 normality==2.3.3 packaging==21.3 parameter-expansion-patched==0.2.1b4 pdfminer.six==20220319 pkginfo==1.8.2 pluggy==0.13.1 plugincode==21.1.21 ply==3.11 publicsuffix2==2.20191221 pygmars==0.7.0 Pygments==2.12.0 pymaven-patch==0.3.0 pyparsing==3.0.8 rdflib==6.1.1 requests==2.21.0 scancode-toolkit==30.1.0 soupsieve==2.3.2.post1 spdx-tools==0.7.0a3 text-unidecode==1.3 toml==0.10.2 typecode==21.6.1 typecode-libmagic==5.39.210531 urllib3==1.24.1 wcwidth==0.2.5 webencodings==0.5.1'

So the issue is apparently due to the newer version of Scancode.

Installer should install a pinned version of Scancode (and of its dependencies, that sometimes are not pinned in Scancode itself) instead of installing the latest, unpinned version.

How to reproduce

Install Fossology v4.1.0 (or any other more recent version) from sources. Upload a package and schedule Scancode Agent. Check the logs of the job (failure)

Downgrade Scancode to v 30.1.0 (see commands above). Upload another package and schedule Scancode Agent. Check the logs of the job (success)

Screenshots

See above

Versions

  • 4.1.0+
  • Debian 10 (but should apply also to other linux distros)

Job logs

See above
This was referenced Oct 5, 2022
Closed
Closed
@alpianon
Copy link
Contributor Author

alpianon commented Oct 5, 2022

I opened a PR to fix the problem alpianon#1

@alpianon alpianon changed the title Scancode not working any more with fossology (Installer installs latest/unpinned version of Scancode) Scancode not working any more with Fossology (Installer installs latest/unpinned version of Scancode) Oct 6, 2022
@dineshr93
Copy link
Contributor

I tried for 4.2.1.0 latest release , I got this error #2348

@GMishx GMishx mentioned this issue Apr 24, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@alpianon @dineshr93