Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RemoteDisconnected: Remote end closed connection without response #725

Closed
joelwking opened this issue Jul 21, 2021 · 7 comments · Fixed by #745 or nautobot/nautobot-lab#45
Closed

Comments

@joelwking
Copy link

Environment

  • Python version: python:3.8.10-slim-buster
  • Nautobot version: 7cf9d60114ee (v1.1.0b1)
  • requirements.txt
pyyaml
ansible<3
pynautobot

Steps to Reproduce

The issue is not related to a specific module, it occurs on any of the Nautobot modules. This is used for example.

Install the Nautobot lab container at https://github.com/nautobot/nautobot-lab

Create input YAML file:

    vlans:
      - name: ADMIN
        vid: 33
        status: Active
      - name: BMC
        vid: 70
        status: Active
      - name: TOR
        vid: 100
        status: Active

Execute playbook containing task iterating over the VLANs defined.

    - name: Manage VLANS
      networktocode.nautobot.vlan:
        <<: *nautobot_opts
        data:
          name: Test VLAN
          vid: '{{ item.vid }}'
          name: '{{ item.name }}'
          status: '{{ item.status }}'
          description: '{{ item.name }}'
          site: '{{ site }}'
        state: '{{ state }}'
      loop: '{{ vlans }}'
      tags: ['vlan']

Expected Behavior

The API request would execute normally.

Observed Behavior

It would appear this issue is related to the Django server closing the connection. Refer to:

https://github.com/psf/requests/issues/4784

Note Two of the three VLANs are created successfully.

TASK [Manage VLANS] ************************************************************************************************************************************
ok: [localhost] => (item={'name': 'ADMIN', 'vid': 33, 'status': 'Active'})
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
failed: [localhost] (item={'name': 'BMC', 'vid': 70, 'status': 'Active'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "BMC", "status": "Active", "vid": 70}, "module_stderr": "Traceback (most recent call last):\n  File \"/opt/ansible210/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 699, in urlopen\n    httplib_response = self._make_request(\n  File \"/opt/ansible210/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 445, in _make_request\n    six.raise_from(e, None)\n  File \"<string>\", line 3, in raise_from\n  File \"/opt/ansible210/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 440, in _make_request\n    httplib_response = conn.getresponse()\n  File \"/usr/local/lib/python3.8/http/client.py\", line 1344, in getresponse\n    response.begin()\n  File \"/usr/local/lib/python3.8/http/client.py\", line 307, in begin\n    version, status, reason = self._read_status()\n  File \"/usr/local/lib/python3.8/http/client.py\", line 276, in _read_status\n    raise RemoteDisconnected(\"Remote end closed connection without\"\nhttp.client.RemoteDisconnected: Remote end closed connection without response\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/opt/ansible210/lib/python3.8/site-packages/requests/adapters.py\", line 439, in send\n    resp = conn.urlopen(\n  File \"/opt/ansible210/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 755, in urlopen\n    retries = retries.increment(\n  File \"/opt/ansible210/lib/python3.8/site-packages/urllib3/util/retry.py\", line 532, in increment\n    raise six.reraise(type(error), error, _stacktrace)\n  File \"/opt/ansible210/lib/python3.8/site-packages/urllib3/packages/six.py\", line 769, in reraise\n    raise value.with_traceback(tb)\n  File \"/opt/ansible210/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 699, in urlopen\n    httplib_response = self._make_request(\n  File \"/opt/ansible210/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 445, in _make_request\n    six.raise_from(e, None)\n  File \"<string>\", line 3, in raise_from\n  File \"/opt/ansible210/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 440, in _make_request\n    httplib_response = conn.getresponse()\n  File \"/usr/local/lib/python3.8/http/client.py\", line 1344, in getresponse\n    response.begin()\n  File \"/usr/local/lib/python3.8/http/client.py\", line 307, in begin\n    version, status, reason = self._read_status()\n  File \"/usr/local/lib/python3.8/http/client.py\", line 276, in _read_status\n    raise RemoteDisconnected(\"Remote end closed connection without\"\nurllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1626878994.0591378-48302-80611049494953/AnsiballZ_vlan.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/root/.ansible/tmp/ansible-tmp-1626878994.0591378-48302-80611049494953/AnsiballZ_vlan.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/root/.ansible/tmp/ansible-tmp-1626878994.0591378-48302-80611049494953/AnsiballZ_vlan.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.networktocode.nautobot.plugins.modules.vlan', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/local/lib/python3.8/runpy.py\", line 207, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/local/lib/python3.8/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/local/lib/python3.8/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_networktocode.nautobot.vlan_payload__nywcale/ansible_networktocode.nautobot.vlan_payload.zip/ansible_collections/networktocode/nautobot/plugins/modules/vlan.py\", line 223, in <module>\n  File \"/tmp/ansible_networktocode.nautobot.vlan_payload__nywcale/ansible_networktocode.nautobot.vlan_payload.zip/ansible_collections/networktocode/nautobot/plugins/modules/vlan.py\", line 218, in main\n  File \"/tmp/ansible_networktocode.nautobot.vlan_payload__nywcale/ansible_networktocode.nautobot.vlan_payload.zip/ansible_collections/networktocode/nautobot/plugins/module_utils/ipam.py\", line 35, in __init__\n  File \"/tmp/ansible_networktocode.nautobot.vlan_payload__nywcale/ansible_networktocode.nautobot.vlan_payload.zip/ansible_collections/networktocode/nautobot/plugins/module_utils/utils.py\", line 480, in __init__\n  File \"/tmp/ansible_networktocode.nautobot.vlan_payload__nywcale/ansible_networktocode.nautobot.vlan_payload.zip/ansible_collections/networktocode/nautobot/plugins/module_utils/utils.py\", line 811, in _change_choices_id\n  File \"/tmp/ansible_networktocode.nautobot.vlan_payload__nywcale/ansible_networktocode.nautobot.vlan_payload.zip/ansible_collections/networktocode/nautobot/plugins/module_utils/utils.py\", line 779, in _fetch_choice_value\n  File \"/opt/ansible210/lib/python3.8/site-packages/pynautobot/core/endpoint.py\", line 309, in choices\n    req = Request(base=self.url, token=self.api.token, http_session=self.api.http_session,).options()\n  File \"/opt/ansible210/lib/python3.8/site-packages/pynautobot/core/query.py\", line 362, in options\n    return self._make_call(verb=\"options\")\n  File \"/opt/ansible210/lib/python3.8/site-packages/pynautobot/core/query.py\", line 211, in _make_call\n    req = getattr(self.http_session, verb)(url_override or self.url, headers=headers, params=params, json=data)\n  File \"/opt/ansible210/lib/python3.8/site-packages/requests/sessions.py\", line 566, in options\n    return self.request('OPTIONS', url, **kwargs)\n  File \"/opt/ansible210/lib/python3.8/site-packages/requests/sessions.py\", line 542, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/opt/ansible210/lib/python3.8/site-packages/requests/sessions.py\", line 655, in send\n    r = adapter.send(request, **kwargs)\n  File \"/opt/ansible210/lib/python3.8/site-packages/requests/adapters.py\", line 498, in send\n    raise ConnectionError(err, request=request)\nrequests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
ok: [localhost] => (item={'name': 'TOR', 'vid': 100, 'status': 'Active'})

PLAY RECAP *********************************************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   
@jathanism
Copy link
Contributor

@joelwking Thanks for the report. This one seems to be popping up all over the place all of a sudden.

@nniehoff
Copy link
Contributor

nniehoff commented Jul 26, 2021

@joelwking I haven't been able to reproduce this locally however I do believe I have a fix for it. Can you please edit the uwsgi.ini file and add the following line:

http-keepalive = 1

If that doesn't do it can you please test:

http-keepalive = 1
add-header = Connection: close

I'd like confirmation of the fix and which one of the above resolves the issue, I'm hoping the keepalive is enough but before I submit a PR I'd like to know for sure.

@joelwking
Copy link
Author

Thanks, I will look at this later today or tomorrow morning.

@joelwking
Copy link
Author

@nniehoff just to be clear, on the running container, I docker cp uwsgi.ini out, modified, and copied back in the container, what service would I restart to pick up the configuration changes?

@nniehoff
Copy link
Contributor

@joelwking Once you have the file copied out and modified I would restart the entire container but override the config with a docker volume something like:

docker run -v /some/path/to/uwsgi.ini:/opt/nautobot/uwsgi.ini:ro ...

@joelwking
Copy link
Author

@nniehoff Using the configuration file below and the following:

docker run -v /home/ubuntu/uwsgi.ini:/opt/nautobot/uwsgi.ini:ro -itd --name nautobot2 -p 8002:8000 nautobot-lab

I was able to execute the playbook over 15 times without encountering any 'RemoteDisconnected' errors. You seem to have addressed the problem! Thank you.

[uwsgi]
; The IP address (typically localhost) and port that the WSGI process should listen on
http = 0.0.0.0:8000

; remote end closed connection
http-keepalive = 1
; add-header = Connection: close

; Fail to start if any parameter in the configuration file isn’t explicitly understood by uWSGI
strict = true

; Enable master process to gracefully re-spawn and pre-fork workers
master = true

; Allow Python app-generated threads to run
enable-threads = true

;Try to remove all of the generated file/sockets during shutdown
vacuum = true

; Do not use multiple interpreters, allowing only Nautobot to run
single-interpreter = true

; Shutdown when receiving SIGTERM (default is respawn)
die-on-term = true

; Prevents uWSGI from starting if it is unable load Nautobot (usually due to errors)
need-app = true

; By default, uWSGI has rather verbose logging that can be noisy
disable-logging = true

; Assert that critical 4xx and 5xx errors are still logged
log-4xx = true
log-5xx = true

;
; Advanced settings (disabled by default)
; Customize these for your environment if and only if you need them.
; Ref: https://uwsgi-docs.readthedocs.io/en/latest/Options.html
;

; Number of uWSGI workers to spawn. This should typically be 2n+1, where n is the number of CPU cores present.
; processes = 5

; If using subdirectory hosting e.g. example.com/nautobot, you must uncomment this line. Otherwise you'll get double paths e.g. example.com/nautobot/nautobot/.
; See: https://uwsgi-docs.readthedocs.io/en/latest/Changelog-2.0.11.html#fixpathinfo-routing-action
; route-run = fixpathinfo:

@nniehoff
Copy link
Contributor

@joelwking Thanks for validating that's super helpful! I'll work on a PR now

jathanism pushed a commit that referenced this issue Aug 4, 2021
Adding `http-keepalive = 1` resolves the remote disconnects reported in #725.
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
3 participants