Issues with a clean pip install of intelmq on RHEL 8

[olekristoffer]

Hi

We encounter some problems with installing intelmq using pip on rhel8 where there seems to be a mismatch between the pip installation and source available on github under the 3.0.x tag.

bash-4.4$ pip3 --version
pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6)
bash-4.4$ pip3 show intelmq
Name: intelmq
Version: 3.0.2
Summary: IntelMQ is a solution for IT security teams for collecting and processing security feeds using a message queuing protocol.
Home-page: https://github.com/certtools/intelmq/
Author: IntelMQ Community
Author-email: None
License: AGPLv3
Location: /usr/local/lib/python3.6/site-packages
Requires: redis, ruamel.yaml, requests, pytz, python-dateutil, dnspython, python-termstyle, psutil
bash-4.4$ python3 --version
Python 3.6.8

Commands such as "Intelmqctl check" and "intelmqctl list bots" fails in a clean installation when installing the latest release v3.0.2 using pip. It seems that the error stems from expert bots present when using pip install that relies on logic from commit 7d4b9be in develop that defines the Class ExpertBot.

grep -r "import ExpertBot"
bots/experts/domain_valid/expert.py:from intelmq.lib.bot import ExpertBot
bots/experts/truncate_by_delimiter/expert.py:from intelmq.lib.bot import ExpertBot

example output from "intelmqctl check":

intelmqctl check
Reading configuration files.
Checking runtime and pipeline configuration.
Checking harmonization configuration.
Checking for bots.
Traceback (most recent call last):
File "/usr/local/bin/intelmqctl", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/intelmq/bin/intelmqctl.py", line 1909, in main
return x.run()
File "/usr/local/lib/python3.6/site-packages/intelmq/bin/intelmqctl.py", line 1048, in run
retval, results = args.func(**args_dict)
File "/usr/local/lib/python3.6/site-packages/intelmq/bin/intelmqctl.py", line 1564, in check
for group in utils.list_all_bots().values():
File "/usr/local/lib/python3.6/site-packages/intelmq/lib/utils.py", line 855, in list_all_bots
mod = importlib.import_module('.'.join(file.with_suffix('').parts))
File "/usr/lib64/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.6/site-packages/intelmq/bots/experts/domain_valid/expert.py", line 20, in
from intelmq.lib.bot import ExpertBot
ImportError: cannot import name 'ExpertBot'

Furthermore, there seems to be other issues with the release where there is a mismatch between bots available with the 3.0.x release tag in github differs from what is delivered when installing with pip. E.g in github the expert bot ripe is available, with pip install both expert bot "ripe" and "ripencc_abuse_contact" is present.

ls bots/experts/ripe*
bots/experts/ripe:
expert.py init.py pycache REQUIREMENTS.txt
bots/experts/ripencc_abuse_contact:
expert.py init.py pycache REQUIREMENTS.txt

Removing the domain_valid and truncate_by_delimiter bots from the install allows "intelmqctl list bots" and "intelmqctl check" commands to complete. However, the check command gives error on the additional bots available in the pip install

Checking for bots.
Incomplete installation: Executable 'intelmq.bots.collectors.xmpp.collector' for 'XMPP' not found in $PATH ('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin').
Incomplete installation: Executable 'intelmq.bots.parsers.abusech.parser_ransomware' for 'AbuseCHRansomwaretracker' not found in $PATH ('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin').
Incomplete installation: Executable 'intelmq.bots.parsers.bitcash.parser' for 'BitcashBlocklist' not found in $PATH ('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin').
Incomplete installation: Executable 'intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc' for 'FraunhoferDdosAttackCnc' not found in $PATH ('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin').
Incomplete installation: Executable 'intelmq.bots.parsers.fraunhofer.parser_ddosattack_target' for 'FraunhoferDdosAttackTarget' not found in $PATH ('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin').
Incomplete installation: Executable 'intelmq.bots.parsers.hphosts.parser' for 'HpHosts' not found in $PATH ('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin').
Incomplete installation: Executable 'intelmq.bots.parsers.malwaredomainlist.parser' for 'MalwareDomainList' not found in $PATH ('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin').
Incomplete installation: Executable 'intelmq.bots.parsers.malwaredomains.parser' for 'MalwareDomains' not found in $PATH ('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin').
Incomplete installation: Executable 'intelmq.bots.parsers.nothink.parser' for 'Nothink' not found in $PATH ('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin').
Incomplete installation: Executable 'intelmq.bots.parsers.urlvir.parser' for 'URLVir' not found in $PATH ('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin').
Incomplete installation: Executable 'intelmq.bots.experts.jinja.expert' for 'Jinja' not found in $PATH ('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin').
Incomplete installation: Executable 'intelmq.bots.experts.ripencc_abuse_contact.expert' for 'RIPENCCExpertDeprecated' not found in $PATH ('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin').
Incomplete installation: Executable 'intelmq.bots.outputs.xmpp.output' for 'XMPP' not found in $PATH ('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin').
Some issues have been found, please check the above output.

best regards,
Ole Kristoffer Dybvik Apeland
Nkom EkomCERT

[ghost]

I wonder why the two bots (domain_valid and truncate_by_delimiter) are at all in the PyPI package. Removing them is the correct workaround.

The Incomplete installation issues come from a failed pip installation. pip install intelmq needs to create the executables, that's nothing that intelmq can do itself. But I have no idea, why pip doesn't do that.

[ghost]

I wonder why the two bots (domain_valid and truncate_by_delimiter) are at all in the PyPI package. Removing them is the correct workaround.

It looks like I need to rm -r build/ before creating the packages for PyPI :/ I've re-uploaded the 3.0.2 sdist tarball with name intelmq-3.0.2-post1.tar.gz (I can't use the original name, PyPI prohibits that) without the two bots which shouldn't be there. Can you please re-try?

[olekristoffer]

I wonder why the two bots (domain_valid and truncate_by_delimiter) are at all in the PyPI package. Removing them is the correct workaround.

The Incomplete installation issues come from a failed pip installation. pip install intelmq needs to create the executables, that's nothing that intelmq can do itself. But I have no idea, why pip doesn't do that.

I believe the Incomplete Installation may have the same root cause as the other two bots present from the develop tree. The bots with incomplete installation are not part of the 3.0.x release tags in github, but were present in the old 2.x version. I.e. They should ultimately not be present in the pip 3.0.2 package, and I suspect they fail to install due to the code being incompatible with the changes introduced in 3.0

As a side-note: installing from source on the same rhel8 system, e.g. from the published tag 3.02 tar.gz asset from github, gives a complete install that runs without any errors.

[ghost]

Thanks for the feedback. I believe that removing the local build/ folder before generating the release tarballs for PyPI prevents all of the described errors - the missing executables and bots which shouldn't be there - in the future (I've documented that for myself in 2da82cf). I will also remove the bogus wheel package from PyPI tomorrow and replace it with a clean one.

Sorry for the troubles and thanks for the very detailed report and quick and smooth cooperation!

[ghost]

The release files on PyPI are fixed, so closing here.