-
Notifications
You must be signed in to change notification settings - Fork 264
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to stand up nautobot in a HA mode (one write instance & one read-only instance) #2477
Comments
We also have plans to bring up a read-only instance (with access to a database replica). I was hoping this was possible, but this issue lowers my expectations 😄 Is there any supported way to achieve something like this? A secondary Nautobot installation using the same backend database in a read-only fashion? |
You should be able to put your read-only instance into maintenance mode (see: https://nautobot.readthedocs.io/en/stable/configuration/optional-settings/?h=maint#maintenance_mode) which will stop the inserts and updates from that application instance to the connected database. |
Thanks for the suggestion @bryanculver , that's got a little further but I still see two problems.
This is what I did: Added these to the local.env file:
Altered a local copy of
Altered the docker-compose.yml to mount override the nautobot_config.py files in all containers except redis:
I tried exec'ing in to the container and can see my new config file is mounted and I see other files and directories under |
In debugging a little with @gneville-ot on Slack, I have a root cause at least for the issue with static files when nautobot/docker/docker-entrypoint.sh Lines 12 to 26 in 487b26e
We should consider making each of this post_upgrade steps discretely toggled in some way. |
@jathanism, I've raised #2600. I can now stand up the read-only instance, running in k8s. I can browse to the GUI and API access via Token works. I do however still have an outstanding issue, I'm unable to login via the GUI using local or SSO accounts. I was hoping to use a redis cluster that is local to where the read-only instance is running, the active/write instance of nautobot is in another site. We have cross-site replication for our redis clusters, but in one site it is set to a read mode. When the nautobot read-only instance points to its local read redis cluster, in read mode, I get the following error: Setting
I wonder if the In order to work around this I've changed the read-only instance to point back to the other site's redis cluster, but when I do this I get this issue when attempting to login using a local account:
When attempting to login using OKTA, our SSO provider, then I see this issue:
==================
|
I just want to thank @gneville-ot and @jathanism for all the troubleshooting done here. I managed to successfully bring up a read-only instance thanks to all the advice in this thread. Granted I'm not using authentication, so I'm not facing the same issues as @gneville-ot, but the instance itself is up and running. |
When it comes to that authentiction error @gneville-ot and @jathanism, I believe Nautobot still tries to update the last login time under The nautobot/nautobot/users/views.py Line 67 in 21c85f2
However remote user authentication doesn't seem to pass that check, it goes straight to nautobot/nautobot/core/middleware.py Line 23 in 21c85f2
I have fixed this by adding the following to
I have no clue if this is the right approach though, but it did fix it for me. |
Thanks @joaopsys for the in-depth debugging. Seems like this can result in some extra documentation and a small fix here. |
PR #2656 raised |
Thanks for the extra work @joaopsys and glad to see you were able to stand up a read-only instance. However with regards to the auth problem it appears your fix only applies if you are using HTTP headers for auth. I'm using SSO (OKTA) for auth, I did try adding your code to @jathanism, any ideas how to over come this issue?
|
@gneville-ot Can you enable I'm assuming all users already exist in the read-only database right? |
@joaopsys, with
|
@bryanculver , this is still an issue and pending #2600 |
@grahamneville Can you confirm again? @joaopsys made tweaks that further respected read-only earlier in the app startup. I see there is some valuable adds to #2600 for env flags but that can be a future follow-on since they can also be set in the settings.py. |
@bryanculver - I've just upgraded to 1.5.5 and tested this out. I'm still not able to login using SSO (OKTA) with a read-only instance. I do see this in the logs to show it's running in maintenance mode:
This is the error:
|
At a glance, looks like an issue with social_core/social_django |
Environment
Steps to Reproduce
Expected Behavior
I'm able to spin up a write instance and a separate read-only instance.
The idea being that nautobot can be hosted in two different sites, one site with write the other for read, the redis cluster is synced cross-site and the database has read-only replicas in the other site.
Observed Behavior
The read-only instance complains it doesn't have the permissions to perform INSERT/UPDATE statements in different tables.
The text was updated successfully, but these errors were encountered: