FATAL: sorry, too many clients already

[jifox]

Environment

• Python version: 3.8.5
• Nautobot version: 1.0.0b2

Steps to Reproduce

1. invoke debug
2. Add some data using nautobot-ansible (pynautobot) - running on localhost

Questions:

• How to increase the number of allowed clients?
• How to check if everything is processed before sending a new request?

- name: "ANSIBLE PLAYBOOK - Add Nautobot Configuration"
  connection: local
  hosts: localhost
  gather_facts: False

  collections:
    - networktocode.nautobot
  vars:
    # nautobot_token see group_vars/all
    nbtoken: "{{ lookup('env', 'NAUTOBOT_TOKEN') if lookup('env', 'NAUTOBOT_TOKEN')|default(false) else nautobot_token }}"
    nburl: "{{ lookup('env', 'NAUTOBOT_URL') if lookup('env', 'NAUTOBOT_URL')|default(false) else nautobot_url }}"
    include_dir: "./include"
    snapshot_dir: "~/snapshot-transfer"

  tasks:
    - name: "Import ansible Fabric-Vars"
      include_vars:
        dir: vars/fabric

    - name: "Add Tags in Nautobot (nautobot_tags.yml)"
      networktocode.nautobot.tag:
        url: "{{ nburl }}"
        token: "{{ nbtoken }}"
        validate_certs: yes
        state: present
        data: "{{ item }}"
      loop: "{{ nbtags }}"

Data for import: ./vars/fabric/nautobot-tags.yml

nbtags:
  - { name: "name1", slug: "name1", description: "Tag name1", color: '4caf50' }
  - ...

Expected Behavior

Import all tags.

Observed Behavior

Environment:


Request Method: GET
Request URL: http://scvpconf01:8000/extras/tags/

Django Version: 3.1.7
Python Version: 3.8.5
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'cacheops',
 'corsheaders',
 'django_filters',
 'django_tables2',
 'django_prometheus',
 'mptt',
 'rest_framework',
 'social_django',
 'taggit',
 'timezone_field',
 'nautobot.core',
 'nautobot.circuits',
 'nautobot.dcim',
 'nautobot.ipam',
 'nautobot.extras',
 'nautobot.tenancy',
 'nautobot.users',
 'nautobot.utilities',
 'nautobot.virtualization',
 'django_rq',
 'drf_yasg',
 'graphene_django',
 'debug_toolbar']
Installed Middleware:
['debug_toolbar.middleware.DebugToolbarMiddleware',
 'django_prometheus.middleware.PrometheusBeforeMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'nautobot.core.middleware.ExceptionHandlingMiddleware',
 'nautobot.core.middleware.RemoteUserMiddleware',
 'nautobot.core.middleware.ExternalAuthMiddleware',
 'nautobot.core.middleware.APIVersionMiddleware',
 'nautobot.core.middleware.ObjectChangeMiddleware',
 'django_prometheus.middleware.PrometheusAfterMiddleware']



Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 233, in _get_session
    return self._session_cache

During handling of the above exception ('SessionStore' object has no attribute '_session_cache'), another exception occurred:
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python3.8/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)

The above exception (FATAL:  sorry, too many clients already
) was the direct cause of the following exception:
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.8/site-packages/debug_toolbar/middleware.py", line 67, in __call__
    panel.generate_stats(request, response)
  File "/usr/local/lib/python3.8/site-packages/debug_toolbar/panels/request.py", line 57, in generate_stats
    for k in sorted(request.session.keys())
  File "/usr/local/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 169, in keys
    return self._session.keys()
  File "/usr/local/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 238, in _get_session
    self._session_cache = self.load()
  File "/usr/local/lib/python3.8/site-packages/django/contrib/sessions/backends/db.py", line 43, in load
    s = self._get_session_from_db()
  File "/usr/local/lib/python3.8/site-packages/django/contrib/sessions/backends/db.py", line 32, in _get_session_from_db
    return self.model.objects.get(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 425, in get
    num = len(clone)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 269, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.8/site-packages/cacheops/query.py", line 273, in _fetch_all
    return self._no_monkey._fetch_all(self)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1308, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1154, in execute_sql
    cursor = self.connection.cursor()
  File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 259, in cursor
    return self._cursor()
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 235, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python3.8/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)

Exception Type: OperationalError at /extras/tags/
Exception Value: FATAL:  sorry, too many clients already

[jathanism]

Hello again @jifox! Thanks for the report.

Can you please provide the following:

• Output of the DATABASES section of your nautobot_config.py
• uWSGI config
• Operating system and version (e.g. Ubuntu 20.04.1 LTS)
• PostgreSQL version
• System memory/CPU

I'm asking for this because I'm trying to isolate if there might be issues with certain setups. Thanks!

[jifox]

Hello @jathanism

I'm using the docker development environment with default docker-compose files.

After a research I think I could find a solution by adding the --nothreading parameter to runserver command.

I have done few successful imports that had always lead to a traceback after changing the command.

File development/docker-compose.override.yml:

---
version: "3"
services:
  nautobot:
    command: "nautobot-server runserver 0.0.0.0:8000 --insecure --nothreading"

[jifox]

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 33
  Running: 19
  Paused: 0
  Stopped: 14
 Images: 4662
 Server Version: 20.10.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
 ...
 Kernel Version: 5.4.0-66-generic
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.61GiB

[jathanism]

Hello @jathanism

I'm using the docker development environment with default docker-compose files.

After a research I think I could find a solution by adding the --nothreading parameter to runserver command.

I have done few successful imports that had always lead to a traceback after changing the command.

File development/docker-compose.override.yml:

---
version: "3"
services:
  nautobot:
    command: "nautobot-server runserver 0.0.0.0:8000 --insecure --nothreading"

Ahhh I should have caught that when you said you did invoke debug.

If you're trying to do any legit data or performance testing, you might start bumping into issues like this with runserver since it's not the most performant.

[jathanism]

I think this would be valuable to have in the developer documentation in a "troubleshooting" section.