[BUG] The whole mongodb code is a mess

[dhs-rec]

Description
Using MongoDB for tops and pillars is currently broken, due to differences in how they connect to the database.

1. pillar/mongo.py can use mongo.uri, but throws an exception if mongo.host is also provided
2. tops/mongo.py ignores mongo.uri, so it is required to provide all the mongo.* options
3. none of the above provides an authSource parameter, so pymongo defaults to admin, resulting in authentication failures if the DB user has been created in the salt DB.
4. modules/mongodb.py provides a method for connecting, but that's not used
5. in general, the code for connecting to the DB looks quite different in pillar/mongo.py and tops/mongo.py
6. documentation doesn't tell anything about pymongo versions (Hint: Versions >= 3.13.0 don't work)

Setup

ext_pillar:
  - ...
  - mongo: {collection: pillar}
...
master_tops:
  mongo:
    collection: tops
    id_field: _id
    states_field: states
    environment_field: environment
mongo.db: salt
mongo.host: localhost
mongo.indexes: true
mongo.password: Secret
mongo.user: salt

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior

• Install/update Salt master to 3006.7 using onedir packages on Ubuntu Jammy
• Install pymongo==3.13.0 using Salts internal Python/pip

Expected behavior
Pillars/Tops in MongoDB work as before

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.7
 
Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:34:22) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.11
     gitpython: 3.1.42
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: 3.20.0
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 2.3.1
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.15.1
         smmap: 5.0.1
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 20.04.2 jammy
        locale: utf-8
       machine: aarch64
       release: 6.2.0-1018-aws
        system: Linux
       version: Ubuntu 20.04.2 jammy

Additional context
I have fixed/worked around point 3 above by adding an authSource=db parameter to pymongo.MongoClient() in both pillar/mongo.py and tops/mongo.py. However, this is not as clean as it could be.

[cplmayo]

Additional context I have fixed/worked around point 3 above by adding an authSource=db parameter to pymongo.MongoClient() in both pillar/mongo.py and tops/mongo.py. However, this is not as clean as it could be.

I am currently trying to implement a similar patch in my project; would be willing to provide an example of what you had to do to get it working? I've been poking at the files but I'm getting an error.

I manually rolled back to earlier than the version identified and it's working. So please ignore; and sorry for the spam emails.