Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MongoDB returner establishes a new database connection for every DB call #55999

Open
sjafferali opened this issue Jan 28, 2020 · 0 comments · May be fixed by #66238
Open

MongoDB returner establishes a new database connection for every DB call #55999

sjafferali opened this issue Jan 28, 2020 · 0 comments · May be fixed by #66238
Labels
Feature new functionality including changes to functionality and code refactors, etc. Returners
Milestone

Comments

@sjafferali
Copy link

Description of Issue

https://github.com/saltstack/salt/blob/master/salt/returners/mongo_future_return.py#L158

Every-time this function is called, a new connection to MongoDB is established. This results in large number of connections to MongoDB. This can result in the ephemeral port range on the server getting exhausted on busy saltmasters.

Other returner modules such as redis, https://github.com/saltstack/salt/blob/master/salt/returners/redis_return.py#L175 establish a connection only initially, then re-use that connection pool. It seems the pymongo returner could also use a connection pool via a similar method to reduce the number of connections that are made as pooling is supported by the module.

Setup

# master job cache and salt returns go here
master_job_cache: mongo
mongo.uri: "mongodb://USERNAME:PASSWORD@host1:27017,host2:27017,host3:27017/database?replicaSet=rs0&readPreference=nearest&journal=true&connectTimeoutMS=10000"
mongo.indexes: true

Steps to Reproduce Issue

Enable mongo returner module. Then attempt to run a salt command on many hosts and watch the connections to MongoDB (netstat -tunap). The number of connections keeps increasing proportional to the number minions being targeted. This does not occur when using other returners such as the redis returner.

Versions Report

root@saltmater:~# salt --versions-report
Salt Version:
           Salt: 2019.2.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.26.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.2
         Python: 3.6.8 (default, Jan 14 2019, 11:02:34)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-46-generic
         system: Linux
        version: Ubuntu 18.04 bionic

@Akm0d Akm0d added Feature new functionality including changes to functionality and code refactors, etc. Returners and removed needs-triage labels Feb 7, 2020
@Akm0d Akm0d added this to the Approved milestone Feb 7, 2020
@Akm0d Akm0d added this to Planning in Sodium Feb 7, 2020
@sagetherage sagetherage removed this from Planning in Sodium May 11, 2020
@ilinas ilinas linked a pull request Mar 15, 2024 that will close this issue
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature new functionality including changes to functionality and code refactors, etc. Returners
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants