Error loading bot queues information: {"errors": {"Invalid Authentication": "Provided Token verification credentials were invalid"} #287

aragorne007 · 2022-08-26T21:59:32Z

Hello,

I've installed a fresh version of IntelMQ with PyPi but I've an issue with the connection between Intelmq-manager and intelmq-api (i guess).

I can login in the intelmq-manager but when I click on "Configuration", "Management" and "Monitor", I get this :
error : Error loading bot queues information: {"errors": {"Invalid Authentication": "Provided Token verification credentials were invalid"}"

On "Check :
error getting check command output

On "Version":
error getting version
Error getting debugging information. Do you have IntelMQ >= 2.2.0?

i'm pretty sure it's a permission error but I don't see where I failed.

My intelmqsetup result :

Not overwriting existing 'runtime.yaml' with example.
Not overwriting existing 'harmonization.conf' with example.
Setting intelmq as owner for it's directories.
Calling `intelmqctl upgrade-config` to update/create state file.
Found previous version 3.0.1 in state file.
Checks if all harmonization fields and types are correct: Nothing to do.
Configuration upgrade successful!
Running setup for intelmq-api.
Detected Apache username 'www-data'.
Running webserver setup for intelmq-api.
Detected Apache configuration directory /etc/apache2/conf-available.
Unable to install webserver configuration api-config.conf: Neither /usr/lib/python3/dist-packages/etc/intelmq/api-apache.conf nor /etc/apache2/conf-available/api-apache.conf exists.
Setup of intelmq-api successful.
Running webserver setup for intelmq-manager.
Unable to install webserver configuration manager-config.conf: Neither /usr/lib/python3/dist-packages/etc/intelmq/manager-apache.conf nor /etc/apache2/conf-available/manager-apache.conf exists.
'intelmqsetup' completed.

I analyzed that :
Unable to install webserver configuration api-config.conf: Neither /usr/lib/python3/dist-packages/etc/intelmq/api-apache.conf nor /etc/apache2/conf-available/api-apache.conf exists.

275407 0 lrwxrwxrwx 1 root root   29 Sep  2  2021 intelmq-api.conf -> ../../intelmq/api-apache.conf
275419 0 lrwxrwxrwx 1 root root   33 Sep  2  2021 intelmq-manager.conf -> ../../intelmq/manager-apache.conf

root@Intelmq:~# ll /etc/intelmq/
total 60
274108  4 drwxrwxr-x  3 intelmq  intelmq   4096 Aug 26 22:51 .
129793  4 drwxr-xr-x 74 root     root      4096 Aug 26 23:01 ..
275402  4 -rw-rw-r--  1 www-data www-data   506 Sep  2  2021 api-apache.conf
275403  4 -rw-rw-r--  1 www-data www-data   241 Sep  2  2021 api-config.json
275404  4 -rw-rw-r--  1 www-data www-data   264 Sep  2  2021 api-sudoers.conf
274109 24 -rw-rw-r--  1 www-data www-data 21097 Sep 10  2021 harmonization.conf
276139  4 drwxrwxr-x  2 www-data intelmq   4096 Aug 26 22:51 manager
275418  4 -rw-rw-r--  1 www-data www-data   385 Sep  2  2021 manager-apache.conf
274110  8 -rw-rw-r--  1 www-data www-data  6313 Sep 10  2021 runtime.yaml

I've manually changed the owner (from "root" to "www-data" and "intelmq" but it doesn't work.

Thank in advance :)

The text was updated successfully, but these errors were encountered:

sebix · 2022-08-31T15:32:28Z

There was (or is) a similar report like this, but I currently can't find it.

aragorne007 · 2022-08-31T18:44:58Z

Hello @sebix,
I saw 1 or 2 other topic with this error but the solution isn't relevant to my case unfortnately.

I removed the filter "is:issue is:open " to find solved ticket ;)

sebix · 2022-08-31T18:47:33Z

I'm fairly sure it was identical to yours. After (and despite) a successful login, the the backend did not accept the token. Which is very strange of course.

aragorne007 · 2022-08-31T18:49:41Z

Yeah, and I'm pretty sure that it's a permission error because when it failed the first time, I tried to change, to some folder, the ownership passing through root, intelmq and www-data but with no success

sebix · 2022-08-31T18:52:09Z

If we check the ownerships: Please show a ls -la of the directory containing the session database (path is configured in the api-config.json)

aragorne007 · 2022-08-31T18:59:35Z

I don't know which one you need, so I "ls" both :

user@Intelmq:~$ cat /etc/intelmq/api-config.json
{
        "intelmq_ctl_cmd": ["sudo", "-u", "intelmq", "intelmqctl"],
        "allowed_path": "/opt/intelmq/var/lib/bots/",
        "session_store": "/var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi",
        "session_duration": 86400,
        "allow_origins": ["*"]
}
user@Intelmq:~$ ls -la /opt/intelmq/var/lib/bots/
total 12
drwxr-xr-x 3 intelmq root 4096 Aug 26 22:45 .
drwxr-xr-x 3 intelmq root 4096 Aug 26 22:45 ..
drwxr-xr-x 2 intelmq root 4096 Aug 26 23:27 file-output
user@Intelmq:~$ ls -la /var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi
-rw-r--r-- 1 intelmq intelmq 24576 Aug 26 23:18 /var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi

sebix · 2022-08-31T19:19:02Z

Please also ls -la /var/lib/dbconfig-common/sqlite3/intelmq-api/ (or instead of -a -d, to show the directory's permissions)

aragorne007 · 2022-08-31T19:25:39Z

The last try was a mix between "root" and "intelmq" owner permission

user@Intelmq:~$ ls -la /var/lib/dbconfig-common/sqlite3/intelmq-api/
total 32
drwxr-xr-x 2 intelmq intelmq  4096 Aug 26 23:18 .
drwxr-xr-x 3 intelmq intelmq  4096 Aug 26 23:17 ..
-rw-r--r-- 1 intelmq intelmq 24576 Aug 26 23:18 intelmqapi

sebix · 2022-08-31T20:02:08Z

Try chmod g+w /var/lib/dbconfig-common/sqlite3/intelmq-api/ /var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi. The webserver (should be member of the intelmq group) must write the database. Currently it can't.

aragorne007 · 2022-08-31T20:07:48Z

We move forward :D

I've a new error message but we are on the good way :D

22:07:17 50× Error loading bot queues information:
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.

sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required
sudo -u www-data sudo -u intelmq intelmqctl --type json list queues-and-status

22:06:30 Saved positions are not valid or not complete. The configuration has possibly been modified outside of the IntelMQ-Manager.

22:06:30 Get an error error, Internal Server Error when trying to obtain config file properly
http://192.168.0.241/intelmq/v1/api/positions
.

22:06:29 Get an error error, Internal Server Error when trying to obtain config file properly
http://192.168.0.241/intelmq/v1/api/bots
.

aragorne007 · 2022-08-31T20:23:56Z

Problem is : I don't have a password for www-data, so it's maybe not the good user to use ?

aragorne007 · 2022-08-31T21:10:29Z

Regarding my investigation, it's seems to be link to an askpass program and so, either a wrong user used or I need to add something in the sudoer file config but I'm not sure.

sebix · 2022-09-01T06:56:44Z

Your sudoers configuration for intelmq does not work properly. Is there a file in /etc/sudoers.d/ for intelmq? If not, move the api-sudoers.conf file there and set permissions to 440.

aragorne007 · 2022-09-01T18:12:16Z

Hello,
I've a file named 01_intelmq-api :

root@Intelmq:/home/user# ls -lisa /etc/sudoers.d/
total 16
 13623 4 drwxr-xr-x  2 root root 4096 Aug 26 22:51 .
129793 4 drwxr-xr-x 74 root root 4096 Sep  1 20:08 ..
 19237 4 -r--r-----  1 root root  264 Sep  2  2021 01_intelmq-api
 13624 4 -r--r-----  1 root root  958 Feb 27  2021 README

And the content is :

root@Intelmq:/home/user# cat /etc/sudoers.d/01_intelmq-api
# SPDX-FileCopyrightText: 2020 Birger Schacht
#
# SPDX-License-Identifier: CC0-1.0
#
# intelmq-api sudoers file, allowing the intelmq-api which usually
# is run by a webserver, to run intelmqctl as user intelmq
www-data ALL=(intelmq) NOPASSWD: /usr/bin/intelmqctl

sebix · 2022-09-01T18:14:42Z

Is the path to intelmqctl in this file correct?

aragorne007 · 2022-09-01T18:19:43Z

Yes it's is :)

root@Intelmq:/home/user# ls -lisa /usr/bin/intelmqctl
18325 4 -rwxr-xr-x 1 root root 964 Sep 10  2021 /usr/bin/intelmqctl

sebix · 2022-09-01T18:20:51Z

Please verify with which intelmqctl that it is actually the file that is found when intelmqctl is searched for in $PATH

aragorne007 · 2022-09-01T18:21:54Z

Ohhh strange

root@Intelmq:/home/user# which intelmqctl
/usr/local/bin/intelmqctl

aragorne007 · 2022-09-01T18:22:45Z

root@Intelmq:/home/user# ls -lisa /usr/local/bin/intelmqctl
144023 4 -rwxr-xr-x 1 root root 221 Aug 26 22:44 /usr/local/bin/intelmqctl

aragorne007 · 2022-09-01T18:24:59Z

And the internal script isn't same :

root@Intelmq:/home/user# cat /usr/local/bin/intelmqctl
#!/usr/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from intelmq.bin.intelmqctl import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

root@Intelmq:/home/user# cat /usr/bin/intelmqctl
#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'intelmq==3.0.2','console_scripts','intelmqctl'
import re
import sys

# for compatibility with easy_install; see #2198
__requires__ = 'intelmq==3.0.2'

try:
    from importlib.metadata import distribution
except ImportError:
    try:
        from importlib_metadata import distribution
    except ImportError:
        from pkg_resources import load_entry_point


def importlib_load_entry_point(spec, group, name):
    dist_name, _, _ = spec.partition('==')
    matches = (
        entry_point
        for entry_point in distribution(dist_name).entry_points
        if entry_point.group == group and entry_point.name == name
    )
    return next(matches).load()


globals().setdefault('load_entry_point', importlib_load_entry_point)


if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(load_entry_point('intelmq==3.0.2', 'console_scripts', 'intelmqctl')())

sebix · 2022-09-01T18:29:40Z

Running pip3 install intelmq should only install /usr/local/bin/intelmqctl, not /usr/bin/intelmqctl. I don't know where you got the latter from.

Anyway, the local bindir wins, if both exist. So make sure that the path in sudoers matches the active one, and optionally get rid of one.

aragorne007 · 2022-09-01T18:37:22Z

Yeah !! Works !!

One last error message on position xD

20:33:37 Saved positions are not valid or not complete. The configuration has possibly been modified outside of the IntelMQ-Manager.

20:33:36 Get an error error, Internal Server Error when trying to obtain config file properly
http://192.168.0.241/intelmq/v1/api/bots

sebix · 2022-09-01T18:38:36Z

Save the configuration once, that will also set the positions in place. (if you got the file permissions right).

aragorne007 · 2022-09-01T18:52:53Z

[Errno 13] Permission denied: '/opt/intelmq/etc/runtime.yaml'

root@Intelmq:/home/user# ls -lisa /opt/intelmq/etc/
total 52
260666  4 drwxrwxr-x 3 intelmq  intelmq   4096 Sep  1 20:40 .
259699  4 drwxr-xr-x 4 intelmq  root      4096 Aug 26 22:45 ..
260722 24 -rw-r--r-- 1 intelmq  intelmq  21097 Aug 26 22:45 harmonization.conf
390956  4 drwxr-xr-x 2 www-data www-data  4096 Sep  1 20:40 manager
260692  8 -rw-r--r-- 1 intelmq  intelmq   6530 Aug 26 22:45 runtime.yaml
259805  8 -rw-r--r-- 1 www-data www-data  6530 Sep  1 20:49 runtime.yaml.bak

I guess I need to change the owner from intelmq to www-data ?

sebix · 2022-09-01T18:54:20Z

www-data should be member of the intelmq group. And the group needs write permissions for all these files and the directories.

aragorne007 · 2022-09-01T18:58:45Z

www-data is in the intelmq group

intelmq:x:1001:www-data

So it should have access to the runtime.yaml file arf

and in "check" panel,
I've this error

error getting check command output

sebix · 2022-09-01T19:00:31Z

www-data is in the intelmq group

intelmqx1001:www-data

So it should have access to the runtime.yaml file arf

If you now gave the group write access to the file, yes.

aragorne007 · 2022-09-01T20:48:37Z

Doesn't work ^^

When I do a intelmqctl check :

root@Intelmq:/home/user# intelmqctl check
Reading configuration files.
Checking runtime and pipeline configuration.
Checking harmonization configuration.
Checking for bots.
Traceback (most recent call last):
  File "/usr/local/bin/intelmqctl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/dist-packages/intelmq/bin/intelmqctl.py", line 1909, in main
    return x.run()
  File "/usr/local/lib/python3.9/dist-packages/intelmq/bin/intelmqctl.py", line 1048, in run
    retval, results = args.func(**args_dict)
  File "/usr/local/lib/python3.9/dist-packages/intelmq/bin/intelmqctl.py", line 1564, in check
    for group in utils.list_all_bots().values():
  File "/usr/local/lib/python3.9/dist-packages/intelmq/lib/utils.py", line 855, in list_all_bots
    mod = importlib.import_module('.'.join(file.with_suffix('').parts))
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/dist-packages/intelmq/bots/experts/domain_valid/expert.py", line 20, in <module>
    from intelmq.lib.bot import ExpertBot
ImportError: cannot import name 'ExpertBot' from 'intelmq.lib.bot' (/usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py)

root@Intelmq:/home/user# ls -lisa /usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py
143598 60 -rw-r--r-- 1 root root 57581 Aug 26 22:44 /usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py

sebix · 2022-09-01T20:57:27Z

Does /usr/local/lib/python3.9/dist-packages/intelmq/lib/bot.py also contain the ExpertBot class?

May I ask why you did install with pip at all and not with the recommended deb/rpm packages?

aragorne007 · 2022-09-01T21:02:02Z

Arf, no it doesn't.

I followed the "Installation using pip" in the intelmq-manager installation guide (https://intelmq.readthedocs.io/en/maintenance/user/intelmq-manager.html#installation-using-pip)

Maybe I should used the rpm package ...

sebix · 2022-09-01T21:05:54Z

Arf, no it doesn't.

Then there must be some mix of old and new code. ExpertBot exists since 7d4b9be6bd, which is not in 3.0.x, but in the 3.1.0 RC. That raises the question which version you installed with pip.

aragorne007 · 2022-09-01T21:12:41Z

root@Intelmq:/home/user# intelmqctl -v
3.0.1

I think we find the issue

I'll create a new VM and use both "Native package" to intelmq and intelmq-manager.

Could you confirm that the following link will install all I need ? :

IntelMQ - Install Dependencies
https://intelmq.readthedocs.io/en/maintenance/user/installation.html#install-dependencies

IntelMQ - Native Package
https://software.opensuse.org/download.html?project=home:sebix:intelmq&package=intelmq

IntelMQ-Manager - Repo
https://software.opensuse.org/download.html?project=home:sebix:intelmq&package=intelmq-manager

sebix · 2022-09-02T06:37:40Z

I'll create a new VM and use both "Native package" to intelmq and intelmq-manager.

Could you confirm that the following link will install all I need ? :

IntelMQ - Install Dependencies https://intelmq.readthedocs.io/en/maintenance/user/installation.html#install-dependencies

IntelMQ - Native Package https://software.opensuse.org/download.html?project=home:sebix:intelmq&package=intelmq

IntelMQ-Manager - Repo https://software.opensuse.org/download.html?project=home:sebix:intelmq&package=intelmq-manager

You don't need the first linked section (it says If you are using native packages, you skip this section as all dependencies are installed automatically.) just follow https://intelmq.readthedocs.io/en/maintenance/user/installation.html#native-packages
The other two links are correct

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error loading bot queues information: {"errors": {"Invalid Authentication": "Provided Token verification credentials were invalid"} #287

Error loading bot queues information: {"errors": {"Invalid Authentication": "Provided Token verification credentials were invalid"} #287

aragorne007 commented Aug 26, 2022

sebix commented Aug 31, 2022

aragorne007 commented Aug 31, 2022

sebix commented Aug 31, 2022

aragorne007 commented Aug 31, 2022

sebix commented Aug 31, 2022

aragorne007 commented Aug 31, 2022

sebix commented Aug 31, 2022

aragorne007 commented Aug 31, 2022

sebix commented Aug 31, 2022

aragorne007 commented Aug 31, 2022

aragorne007 commented Aug 31, 2022

aragorne007 commented Aug 31, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 2, 2022

Error loading bot queues information: {"errors": {"Invalid Authentication": "Provided Token verification credentials were invalid"} #287

Error loading bot queues information: {"errors": {"Invalid Authentication": "Provided Token verification credentials were invalid"} #287

Comments

aragorne007 commented Aug 26, 2022

sebix commented Aug 31, 2022

aragorne007 commented Aug 31, 2022

sebix commented Aug 31, 2022

aragorne007 commented Aug 31, 2022

sebix commented Aug 31, 2022

aragorne007 commented Aug 31, 2022

sebix commented Aug 31, 2022

aragorne007 commented Aug 31, 2022

sebix commented Aug 31, 2022

aragorne007 commented Aug 31, 2022

aragorne007 commented Aug 31, 2022

aragorne007 commented Aug 31, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 1, 2022

aragorne007 commented Sep 1, 2022

sebix commented Sep 2, 2022