SSL error: decryption failed or bad record mac & SSL SYSCALL error: EOF detected #127
Comments
|
Thanks for the report. Can you please also provide the following details:
Also, if you'd be willing, would you mind following the https://nautobot.readthedocs.io/en/latest/installation/nautobot/ I'd be very curious to see if you're still hitting this issue. But I also want to rule out that you might not be hitting an issue with system resources? We're trying to lock down some recommended system requirements. Thanks in advance! |
|
Type: VM (kvm) I've already switched from gunicorn to uWSGI. I'll get back on you after installing nautobot via new install instructions. fyi: Our nautobot database is currently populated with a handful of demo data (just 4 devices, some IPs, a Rack and so on). |
|
@mkrsn Thanks. Are you running the database server on localhost, or is it on a remote system? |
|
It's a single-instance - all localhost. |
|
Great, thanks. It looks like it may be related to our currently documented uWSGI configuration. If it comes up again, can you please try adding this to your # Workaround for SSL errors
lazy = true
lazy-apps = true |
|
@nniehoff yes but it was just a strange coincident (Murphy says hi) . I only stumbled over this error after visting the onboarding-plugin. So i thought this is related to the Plugin, that's why i reported that error there. Only my colleague saw this error in another module, so i don't think that this is related to the onboarding plugin. Unfortunately i never tried it without installing the Plugin. Only disabled it - error appeared in any case. Do you want me to reinstall and retest without any plugins installed (currently onboarding plugin and napalm)? @jathanism i've reinstalled nautobot via the new way. Same behavior. After a fresh nautobot start the error occurs. After some minutes it disappears... |
|
@jathanism I've added both parameters to the uwsgi.ini, stopped (checked that) and started nautobot but error still appears. |
|
@mkrsn Okay thanks. Going to try to reproduce this. Thanks. |
|
@mkrsn Can you please try replacing your [uwsgi]
; The IP address (typically localhost) and port that the WSGI process should listen on
http-socket = 127.0.0.1:8001
; Fail to start if any parameter in the configuration file isn’t explicitly understood by uWSGI
strict = true
; Enable master process to gracefully re-spawn and pre-fork workers
master = true
; Allow Python app-generated threads to run
enable-threads = true
;Try to remove all of the generated file/sockets during shutdown
vacuum = true
; Do not use multiple interpreters, allowing only Nautobot to run
single-interpreter = true
; Shutdown when receiving SIGTERM (default is respawn)
die-on-term = true
; Prevents uWSGI from starting if it is unable load Nautobot (usually due to errors)
need-app = true
; By default, uWSGI has rather verbose logging that can be noisy
disable-logging = true
; Assert that critical 4xx and 5xx errors are still logged
log-4xx = true
log-5xx = true
;
; Advanced settings (disabled by default)
; Customize these for your environment if and only if you need them.
; Ref: https://uwsgi-docs.readthedocs.io/en/latest/Options.html
;
; Number of uWSGI workers to spawn. This should typically be 2n+1, where n is the number of CPU cores present.
; processes = 5 |
|
I've tried your config. Problem disappears. I've enabled the "processes" variable and the Problem reappears. Thats why i've tested the original Config. Every time i set "processes" >1 the Problem appears. |
|
@mkrsn That's great feedback, thanks! I'd like to get to the bottom of the root cause to this other than just trying to circumvent it. This helps narrow it down. |
|
This only affects Ubuntu systems where PostgreSQL was installed via After more debugging, I can 100% get the issue to go away if I do any one of these:
Obviously the first two are not desirable for production use. The third option in my opinion should be our new default for the common case if we're going to continue to recommend uWSGI over Gunicorn. The root cause is the aggressive pre-forking of uWSGI which performs copy-on-write on subprocesses that often duplicates the database connection objects. This issue did not arise when using Gunicorn because its default |
|
I thought it was also worth noting what DID NOT work so far to alleviate this issue:
|
|
Nice work, thanks :) I'll try to disable Postgres encryption on Monday and report back. But wouldn't it be better to file a Bug at the uWSGI project instead of defining a workaround as new default? If you really want to change the default config to workaround that problem maybe it would be better to let the user choose the best solution. This would just need a small note in the Documentation. In a all-in-one Setup (bare metal & VM, not LXC, LXD or Docker), while just talking to localhost, the best solution would imho be to disable the encryption to the Postgres because:
|
uWSGI gratuitously documents that their aggressive pre-forking is working as intended and that if you don't like it, try the other options. The problem with uWSGI tho is that the configuration options are overwhelming and often confusing. I feel like there has to be some combination of settings for uWSGI that makes it work properly, but I've yet to find those.
Agreed here. My preference would be to have the defaults work for the common case, with callouts and warnings about "advanced" configurations and common pitfalls with those configurations. Like, if you want to use SSL on the database connection, that's the right thing to do, but it's going to also require that you do not use pre-forking, or if you do use pre-forking, that you disable persistent connections.
This is actually great. If you don't mind, I'll use this as inspiration for the forthcoming documentation changes. |
|
Confirmed - disabling the conn-encryption to postgres solves the issue.
sure |
|
@mkrsn Thanks for the confirmation. I'll be revising the docs with a quickness. |
|
@mkrsn and @jathanism, thanks for having this conversation. I've run into the same problem (using the new install instructions, on Ubutntu 20.04, using the distro-supplied PostgreSQL), and have worked around it by setting the database MAX_CONN_AGE to zero. |
Thanks for confirming. More motivation to get this into the docs and have a "common issues" or "troubleshooting" section. :) |
Something else occurred to me as well. If you review the original reason that So for example using the default My opinion is that for any deployment of reasonable size or complexity, |
|
See also: benoitc/gunicorn#996 |
jathanism
added
the
type: documentation
Environment
Steps to Reproduce
This only happens after a fresh start of nautobot. The error disappears after a while until a nautobot restart.
Expected Behavior
no HTTP500
Observed Behavior
fyi: /opt/nautobot/current is a symlink to /opt/nautobot/1.0.0b2/
Seconde error:
The text was updated successfully, but these errors were encountered: