"pip install scancode-toolkit" gives error

[AyanSinhaMahapatra]

Description

Please leave a brief description of the bug or feature request:

Creating a Python 3 virtualenv and running pip install scancode-toolkit generates this error

ERROR: Could not find a version that satisfies the requirement extractcode-libarchive (from scancode-toolkit) (from versions: none)
ERROR: No matching distribution found for extractcode-libarchive (from scancode-toolkit)

When I was testing a few days before, the Error was

ERROR: Could not find a version that satisfies the requirement extractcode-7z

The exact commands were

virtualenv -p /usr/bin/python3.6 venv-test-py3
source venv-test-py3/bin/activate
pip install scancode-toolkit

The scancode-toolkit downloaded from PyPI is of version 3.1.1
[On Python 3.6.8 and pip 19.3.1]

The python 2 pip install obviously works fine, and installs Scancode-Toolkit 3.0.2

System configuration

For bug reports, it really helps us to know:

• What OS are you running on? Ubuntu 18.04
• What version of scancode-toolkit was used to generate the scan file? v3.1.1
• What installation method was used to install/run scancode? pip

[pombredanne]

Thank you. Great catch!

[aj4ayushjain]

What's the actual issue here @pombredanne .
As in tried replicating it in my machine with almost same config only difference being ubuntu upgraded 16->18.Everthing was working fine.

[pombredanne]

@aj4ayushjain thank you for the feedback. All has been fixed now. The key thing was actual re-release of the latest binary plugins for extractcode/typecode

[filips123]

@pombredanne Repository's README says that PYthon 2 is required? Is this still true? If it is, you should really be fixing this as Python 2.7 EOL is in 2020.

[pombredanne]

@filips123 good catch!
@Abhishek-Dev09 completed the Python 3 port and we now fully support at least Python 3.6. For now we also still support Python 2.7 so we need to update the README for sure!.
Do you mind to support a pull request? that would be welcomed and very kind of you.

[filips123]

@pombredanne So the only need it to add a notice that boh Python 2.7 and 3.6+ are supported? If so, I can create PR. Are there any other places except README where this should be fixed? (update: Repository wiki also needs to be updated) Also, is 32-bit version of Python still needed?

[Abhishek-Dev09-zz]

@filips123 :Scancode supports upto Python 3.6.8 only. Soon we will be add the support Python 3.7+ .

[filips123]

@Abhishek-Dev09 Up to 3.6.8 or only 3.6.8? Also, is 64-bit on Windows now supported?

[Abhishek-Dev09-zz]

@filips123 Yes, upto 3.6.8 but 3.6.8(recommended) and 64 bit on window.

[filips123]

@Abhishek-Dev09 So 32-bit Python 2.7 and 64-bit Python up to 3.6.8? If so, I can create PR with corrections. Also, what exectly is wrong with 3.7+?

[Abhishek-Dev09-zz]

@filips123 Sorry, we don't support x64 https://github.com/nexB/scancode-toolkit/blob/develop/azure-pipelines.yml#L390 and for info see azure ci. For py 3.7 #1631 (comment)

[filips123]

Ok, so just 32-bit PYthon from 2.7 up to 3.6.8? I'm currently preparing PR so I will include this in it. And please support 64-bit and Python 3.7+ when possible as those limitations makes quite a lot harder to use ScanCode.

[Abhishek-Dev09-zz]

@filips123 : You can test on 64 bit architecture for confirmation and raise a ticket of that problem if exists. For now we are working on #1703 ,no need to look for version of Python and then we upgrade python version(3.7+) as well.

[filips123]

@pombredanne @Abhishek-Dev09 I created #1807.

[Abhishek-Dev09-zz]

@filips123 : @max-te already working on python 3.7 for intbitset inveniosoftware-contrib/intbitset#62 and inveniosoftware-contrib/intbitset#64

[pombredanne]

@filips123 Thank you ++... FWIW we are not really supporting 32 bits... or rather this is complicated:

• on Linux, 64 bits is the default and you should be able to recompile everything for Linux 32 bits alright with a pip install
• on macOS this is a non issue
• on Windows we have a pending issue to support 64 bits and for now only 32 bits Python works (and 32 Bits Python works also on 64 bits windows)

[NoobSolver]

@filips123 It s working fine now.

[pombredanne]

@AyanSinhaMahapatra can you double check things are OK now and then close this?

[AyanSinhaMahapatra]

@pombredanne python 3.6 virtualenv installation on ubuntu20.04 gives this error:

Traceback (most recent call last):
  File "/home/ayan/Desktop/nexB/vir_envs/venv-scancode-pip-test-3.6/bin/scancode", line 5, in <module>
    from scancode.cli import scancode
  File "/home/ayan/Desktop/nexB/vir_envs/venv-scancode-pip-test-3.6/lib/python3.6/site-packages/scancode/cli.py", line 108, in <module>
    plugin_classes, plugin_options = PluginManager.load_plugins()
  File "/home/ayan/Desktop/nexB/vir_envs/venv-scancode-pip-test-3.6/lib/python3.6/site-packages/plugincode/__init__.py", line 207, in load_plugins
    mgr_setup = manager.setup()
  File "/home/ayan/Desktop/nexB/vir_envs/venv-scancode-pip-test-3.6/lib/python3.6/site-packages/plugincode/__init__.py", line 231, in setup
    self.manager.load_setuptools_entrypoints(entrypoint)
  File "/home/ayan/Desktop/nexB/vir_envs/venv-scancode-pip-test-3.6/lib/python3.6/site-packages/pluggy/manager.py", line 299, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/home/ayan/Desktop/nexB/vir_envs/venv-scancode-pip-test-3.6/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 96, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/ayan/Desktop/nexB/vir_envs/venv-scancode-pip-test-3.6/lib/python3.6/site-packages/summarycode/generated.py", line 19, in <module>
    import typecode.contenttype
  File "/home/ayan/Desktop/nexB/vir_envs/venv-scancode-pip-test-3.6/lib/python3.6/site-packages/typecode/__init__.py", line 21, in <module>
    from typecode.contenttype import get_type
  File "/home/ayan/Desktop/nexB/vir_envs/venv-scancode-pip-test-3.6/lib/python3.6/site-packages/typecode/contenttype.py", line 43, in <module>
    from typecode import magic2
  File "/home/ayan/Desktop/nexB/vir_envs/venv-scancode-pip-test-3.6/lib/python3.6/site-packages/typecode/magic2.py", line 211, in <module>
    libmagic = load_lib()
  File "/home/ayan/Desktop/nexB/vir_envs/venv-scancode-pip-test-3.6/lib/python3.6/site-packages/typecode/magic2.py", line 90, in load_lib
    'CRITICAL: libmagic DLL and is magic database are not installed. '
Exception: CRITICAL: libmagic DLL and is magic database are not installed. Unable to continue: you need to install a valid typecode-libmagic plugin with a valid and proper libmagic and magic DB available.

[pombredanne]

@AyanSinhaMahapatra can you try with pip install scancode-toolkit[full] ? this is the require way now

[AyanSinhaMahapatra]

@pombredanne these are the exact commands I used - https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html#installation-as-a-library-via-pip, i.e. yes I used pip install scancode-toolkit[full]. It installed allright, this happened with scancode -h.

The same also happened on python3.8 also.

[pombredanne]

@AyanSinhaMahapatra Thanks!

Do you mind to try this too:

mkdir scancode-tmp
curl -o scancode-tmp/virtualenv.pyz https://bootstrap.pypa.io/virtualenv/virtualenv.pyz
python3.8 scancode-tmp/virtualenv.pyz scancode-tmp
source scancode-tmp/bin/activate
pip install scancode-toolkit[full]

and either with 3.6 or 3.8 ?

And also tell what show up with: python3 --version

[AyanSinhaMahapatra]

@pombredanne this^ works and runs fine though.

The python versions are 3.6.12 and 3.8.5 in both cases, the previous method and using the virtualenv.pyz .

Is this an issue on my side?

Similar to this report, pip install typecode-libmagic solves it.

[pombredanne]

@AyanSinhaMahapatra you need to run now scancode-toolkit[full] but I am starting to think the whole idea of extra_requires in Python packages is a failure. It should be rather an extra_ignore instead... sorry for this self-made mess. Let me think about a way out.

[pombredanne]

you need to run now scancode-toolkit[full] but I am starting to think the whole idea of extra_requires in Python packages is a failure. It should be rather an extra_ignore instead... sorry for this self-made mess. Let me think about a way out.

we reverted that in the latest release v21.3.31 and now pip install scancode-toolkit works again.
All clear now!