[BUG] "salt" python module is not installed with onedir package

[hadouvex]

Description
"salt" python module is not installed with onedir package as it was with classic one. In that case some modules or states don't work, e.g. "file.serialize". Installing salt from classic packages repo or via pip fixes the problem, but we'd prefer to follow official recommendations and use onedir packages

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Ubuntu 22.04 Jammy

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior

• add repo "deb https://repo.saltproject.io/salt/py3/ubuntu/22.04/amd64/latest/ jammy main" to your repo list
• run:

   apt update
   apt install salt-common
   apt install salt-minion

• check if python module "salt" is available; run:
  python3 -c "import pkg_resources; installed_packages = [(d.project_name, d.version) for d in pkg_resources.working_set]; print(installed_packages)"
• see no mention of "salt" module

Expected behavior
run:
python3 -c "import pkg_resources; installed_packages = [(d.project_name, d.version) for d in pkg_resources.working_set]; print(installed_packages)"
see the following in printed out list:
('salt', '3005.1')

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

salt-master version is 3004.2, but it is not the case. It still works with 3005.1 version shipped via classic package repo

Salt Version:
          Salt: 3005.1
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
        Python: 3.9.16 (main, Dec 14 2022, 15:47:32)
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 22.04 jammy
        locale: utf-8
       machine: x86_64
       release: 5.15.0-56-generic
        system: Linux
       version: Ubuntu 22.04 jammy

Additional context
Example of incorrect behavior of "file.serialize" state:
running on minion:
sudo salt-call state.apply kubernetes.k8s.containerd_config test=0
produces following:

local:
The State execution failed to record the order in which all states were executed. The state return missing data is:
{'changes': {},
 'comment': 'An exception occurred in this state: Malformed state return. Data '
            'must be a dictionary type.',
 'name': 'later',
 'result': False}
----------
          ID: install_python_toml_module
    Function: pkg.installed
        Name: python3-toml
      Result: True
     Comment: All specified packages are already installed
     Started: 17:32:52.812550
    Duration: 42.545 ms
     Changes:   
----------
          ID: manage_containerd_config
    Function: file.serialize
        Name: /etc/containerd/config.toml
      Result: False
     Comment: An exception occurred in this state: Malformed state return. Data must be a dictionary type.
     Changes:   

Summary for local
------------
Succeeded: 1
Failed:    1

kubernetes.k8s.containerd_config content:

install_python_toml_module:
  pkg.installed:
    - name: python3-toml

manage_containerd_config:
  file.serialize:
    - name: /etc/containerd/config.toml
    - formatter: toml
    - merge_if_exists: True
    - user: root
    - group: root
    - mode: 644
    - require:
        - pkg: install_python_toml_module
    - dataset:
        metrics:
          address: "0.0.0.0:1338"

when switching to classic repo and reinstalling salt-common, salt-minion the state works perfectly:

sudo apt remove salt-minion
sudo apt remove salt-common

replace repo
deb https://repo.saltproject.io/salt/py3/ubuntu/22.04/amd64/latest/ jammy main
to
deb https://repo.saltproject.io/py3/ubuntu/20.04/amd64/3005/ focal main (since there's no classic package for 22.04)

install packages again:

apt update
apt install salt-common
apt install salt-minion

run state:
sudo salt-call state.apply kubernetes.k8s.containerd_config test=0
which produces:

local:
----------
          ID: install_python_toml_module
    Function: pkg.installed
        Name: python3-toml
      Result: True
     Comment: All specified packages are already installed
     Started: 17:43:01.855975
    Duration: 42.679 ms
     Changes:   
----------
          ID: manage_containerd_config
    Function: file.serialize
        Name: /etc/containerd/config.toml
      Result: True
     Comment: The file /etc/containerd/config.toml is in the correct state
     Started: 17:43:01.901573
    Duration: 14.839 ms
     Changes:   

Summary for local
------------
Succeeded: 2
Failed:    0

So with the classic package installation it works unlike with onedir

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[OrangeDog]

The onedir package does not touch your system python. It installs everything into one separate directory.

If you wish to install the salt module into your system python, then either install a classic package, or use pip install salt.

[hadouvex]

The onedir package does not touch your system python. It installs everything into one separate directory.

If you wish to install the salt module into your system python, then either install a classic package, or use pip install salt.

Could you please clarify if there is a correct way to do it? Is it implied that some functionality may only be made working via installing a pip module? Or it's possible to utilize module shipped with onedir package somehow?

I don't really need the module installed globally, I pointed it because it seemed like some components couldn't work without it

Thanks!

[OrangeDog]

A correct way to do what? The recommended installation method is with the provided packages.
As you are upgrading an existing installation to onedir, you should probably read this:
https://docs.saltproject.io/salt/install-guide/en/latest/topics/upgrade-to-onedir.html

The kubernetes issue is completely unrelated to whether Salt is installed globally or not.
I think there's already an issue filed, but if not feel free to create one.

[hadouvex]

Yes, I've misunderstood the concept of onedir package, thanks for pointing me in the right direction. I installed lacking dependencies via salt-pip and it works now.

[corycandia]

@hadouvex @OrangeDog

Yes, I've misunderstood the concept of onedir package, thanks for pointing me in the right direction. I installed lacking dependencies via salt-pip and it works now.

Did you use a state function to do that rather than execution module?

The only way I know how to do this is something like:
salt-pip install docker and /opt/saltstack/salt/bin/python3.10 -m pip install --upgrade pip

I am curious if there's a state function that installs modules into the onedir installation? I only found installing classic pip like:
`install_pip_stuff:
pkg.installed:
- name: python-pip/whatever

Can I upgrade and install salt onedir python modules? I found step 4 confusing:

https://docs.saltproject.io/salt/install-guide/en/latest/topics/upgrade-to-onedir.html

[OrangeDog]

@corycandia the documentation for pip.installed is a lie. It will install into the onedir installation by default.

[corycandia]

@corycandia the documentation for pip.installed is a lie. It will install into the onedir installation by default.

@OrangeDog LOL,

so:

/opt/saltstack/salt/bin/python3.10 -m pip install --upgrade pip is the same as: upgrade_onedir_python.sls:

minion_salt_pip_update:
  pip.uptodate:
    - name: pip (docs say doesn't matter)

and for managing docker with salt salt-pip install docker :

minion_salt_docker_manage.sls

minion_salt_docker_module:
  pip.installed:
    - name: docker