[BUG] ValueError: Incorrect decryption.

[tomadmiraal]

Description
We try to install salt on Almalinux 9 and run a first highstate on the salt master himself to install the base package "htop". Nothing special so far.

This is a clean install.

Master and Minion are running and config return any error.

Command: salt 'salt*' state.highstate -l debug

Output:

[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load log4mongo
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load raven
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Missing configuration file: /root/.saltrc
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load log4mongo
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load raven
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Closing AsyncReqChannel instance
[DEBUG   ] The functions from module 'local_cache' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/94/3a769634ca0f986dbc32fcbc7339829305868004dc1f37f0ba5d2c6f30956f/.minions.p
[DEBUG   ] get_iter_returns for jid 20221115081738180606 sent to {'salt.example.com'} will timeout at 09:27:38.237405
[DEBUG   ] jid 20221115081738180606 return from salt.example.com
[DEBUG   ] return event: {'salt.example.com': {'ret': 'The minion function caused an exception: Traceback (most recent call last):\n  File "salt/minion.py", line 1935, in _thread_return\n    return_data = minion_instance._execute_job_function(\n  File "salt/minion.py", line 1894, in _execute_job_function\n    return_data = self.executors[fname](opts, data, func, args, kwargs)\n  File "salt/loader/lazy.py", line 149, in __call__\n    return self.loader.run(run_func, *args, **kwargs)\n  File "salt/loader/lazy.py", line 1228, in run\n    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)\n  File "salt/loader/lazy.py", line 1243, in _run_as\n    return _func_or_method(*args, **kwargs)\n  File "/opt/saltstack/salt/run/salt/executors/direct_call.py", line 10, in execute\n    return func(*args, **kwargs)\n  File "salt/loader/lazy.py", line 149, in __call__\n    return self.loader.run(run_func, *args, **kwargs)\n  File "salt/loader/lazy.py", line 1228, in run\n    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)\n  File "salt/loader/lazy.py", line 1243, in _run_as\n    return _func_or_method(*args, **kwargs)\n  File "/opt/saltstack/salt/run/salt/modules/state.py", line 1102, in highstate\n    st_ = salt.state.HighState(\n  File "salt/state.py", line 4769, in __init__\n    self.state = State(\n  File "salt/state.py", line 768, in __init__\n    self.opts["pillar"] = self._gather_pillar()\n  File "salt/state.py", line 833, in _gather_pillar\n    return pillar.compile_pillar()\n  File "salt/pillar/__init__.py", line 353, in compile_pillar\n    ret_pillar = self.channel.crypted_transfer_decode_dictentry(\n  File "salt/utils/asynchronous.py", line 125, in wrap\n    raise exc_info[1].with_traceback(exc_info[2])\n  File "salt/utils/asynchronous.py", line 131, in _target\n    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))\n  File "salt/ext/tornado/ioloop.py", line 459, in run_sync\n    return future_cell[0].result()\n  File "salt/ext/tornado/concurrent.py", line 249, in result\n    raise_exc_info(self._exc_info)\n  File "<string>", line 4, in raise_exc_info\n  File "salt/ext/tornado/gen.py", line 1070, in run\n    yielded = self.gen.send(value)\n  File "salt/channel/client.py", line 187, in crypted_transfer_decode_dictentry\n    aes = cipher.decrypt(ret["key"])\n  File "Cryptodome/Cipher/PKCS1_OAEP.py", line 200, in decrypt\n    raise ValueError("Incorrect decryption.")\nValueError: Incorrect decryption.\n', 'out': 'nested', 'retcode': 1, 'jid': '20221115081738180606'}}
[DEBUG   ] The functions from module 'nested' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded nested.output
salt.example.com:
    The minion function caused an exception: Traceback (most recent call last):
      File "salt/minion.py", line 1935, in _thread_return
        return_data = minion_instance._execute_job_function(
      File "salt/minion.py", line 1894, in _execute_job_function
        return_data = self.executors[fname](opts, data, func, args, kwargs)
      File "salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "salt/loader/lazy.py", line 1228, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "salt/loader/lazy.py", line 1243, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/opt/saltstack/salt/run/salt/executors/direct_call.py", line 10, in execute
        return func(*args, **kwargs)
      File "salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "salt/loader/lazy.py", line 1228, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "salt/loader/lazy.py", line 1243, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/opt/saltstack/salt/run/salt/modules/state.py", line 1102, in highstate
        st_ = salt.state.HighState(
      File "salt/state.py", line 4769, in __init__
        self.state = State(
      File "salt/state.py", line 768, in __init__
        self.opts["pillar"] = self._gather_pillar()
      File "salt/state.py", line 833, in _gather_pillar
        return pillar.compile_pillar()
      File "salt/pillar/__init__.py", line 353, in compile_pillar
        ret_pillar = self.channel.crypted_transfer_decode_dictentry(
      File "salt/utils/asynchronous.py", line 125, in wrap
        raise exc_info[1].with_traceback(exc_info[2])
      File "salt/utils/asynchronous.py", line 131, in _target
        result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
      File "salt/ext/tornado/ioloop.py", line 459, in run_sync
        return future_cell[0].result()
      File "salt/ext/tornado/concurrent.py", line 249, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 4, in raise_exc_info
      File "salt/ext/tornado/gen.py", line 1070, in run
        yielded = self.gen.send(value)
      File "salt/channel/client.py", line 187, in crypted_transfer_decode_dictentry
        aes = cipher.decrypt(ret["key"])
      File "Cryptodome/Cipher/PKCS1_OAEP.py", line 200, in decrypt
        raise ValueError("Incorrect decryption.")
    ValueError: Incorrect decryption.
[DEBUG   ] jid 20221115081738180606 found all minions {'salt.example.com'}
[DEBUG   ] Closing IPCMessageSubscriber instance
ERROR: Minions returned with non-zero exit code
[DEBUG   ] Restoring sys.argv to: ['/opt/saltstack/salt/run/run', 'salt*', 'state.highstate', '-l', 'debug']

Setup

We run the salt setup on a promox VM. Other setups with Ubuntu (20.04 and 22.04) is running fine.

we use the "repo.saltproject.io/salt/py3/redhat/9/x86_64/latest" repo and every is updated to the last version.

Steps to Reproduce the behavior

Expected behavior
installing the given package

Versions Report

salt --versions-report

Salt Version: Salt: 3005.1

Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.0
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: Not Installed
Python: 3.9.15 (main, Nov 1 2022, 00:00:00)
python-gnupg: 0.4.8
PyYAML: 5.4.1
PyZMQ: 23.2.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4

System Versions:
dist: almalinux 9.0 Emerald Puma
locale: utf-8
machine: x86_64
release: 5.14.0-70.26.1.el9_0.x86_64
system: Linux
version: AlmaLinux 9.0 Emerald Puma

base-packages:
  pkg.installed:
    - pkgs:
      - htop

Does anybody know whats going wrong here?

[OrangeDog]

This seems to be a duplicate of #54092, but no definite cause was found.

[tomadmiraal]

has anyone any idea?

when I run "salt-call state.highstate test=true -l debug", the highstate is running correctly

I tried to install version 3004 and 3005 via the saltstack repo on Ubuntu 22.04, 20.04 and on Almalinux 9.0. On all the installations I get the same error "incorrect decryption"

is it posible that I forgot to install some packages?

[ottacom]

I have the same issue... on RH9 , do you have any updates?