[BUG] LLDAP stuck in restart loop with UNIQUE constraint failures on fresh sqlite install - lldap exited code 132 #756
Comments
|
Something seems wrong here: normally, before doing the migration, we check
whether there are duplicate emails and we refuse to upgrade (or is that
only on latest and it hasn't been released?)
It's failing when trying to create the admin user, which doesn't come with
an email. But why is it trying to create an admin user? It should only do
that if there are no users in the lldap_admin group. And certainly, it
should work with an empty database (it's tested on every commit).
Can you:
- provide the verbose logs (verbose = true in the config)
- provide the user list (`SELECT user_id, email FROM users;`)
- list the members of lldap_admin (`SELECT user_id FROM memberships WHERE
group_id = 0;`)
You can also join Discord and ask over there to try to debug interactively,
we'll post the results back here.
…On Sun, 10 Dec 2023, 23:41 Tyler Pace, ***@***.***> wrote:
*Describe the bug*
lldap v0.5.0 is stuck in a restart loop on a fresh install due to failed
UNIQUE database constraints in sqlite.
*To Reproduce*
For context, I tried to upgrade from 0.4.3 to 0.5.0 and got the expected
UNIQUE constraint error on emails that I had shared across different users.
I updated the emails directly in users.db using sqlite3 but lldap
continued to restart with failed UNIQUE constraints.
So, I started with a fresh v0.5.0 install with a clean data directory with
the following docker compose specification:
lldap:
container_name: lldap
image: lldap/lldap:v0.5.0
restart: unless-stopped
environment:
TZ: ${TZ}
UID: ${PUID}
GID: ${PGID}
LLDAP_JWT_SECRET: ${LLDAP_JWT_SECRET}
LLDAP_LDAP_USER_PASS: ${LLDAP_LDAP_USER_PASS}
LLDAP_LDAP_BASE_DN: ${LLDAP_LDAP_BASE_DN}
volumes:
- ${DOCKER_DIR}/lldap:/data
lldap continues to get stuck in a restart loop with this output:
Loading configuration from /data/lldap_config.toml
2023-12-10T22:18:02.298303464+00:00 INFO set_up_server [ 1.21ms | 100.00% ]
2023-12-10T22:18:02.298319013+00:00 INFO ┝━ ｉ [info]: Starting LLDAP version 0.5.0
2023-12-10T22:18:02.305599549+00:00 WARN ┝━ 🚧 [warn]: Could not find an admin user, trying to create the user "admin" with the config-provided password
2023-12-10T22:18:02.306208556+00:00 ERROR ┕━ 🚨 [error]: | error: Database error: `Execution Error: error returned from database: (code: 2067) UNIQUE constraint failed: users.email`
2023-12-10T22:18:02.306269670+00:00 ERROR 🚨 [error]: Could not bring up the servers: while creating the admin user: Error setting up admin login/account: Error creating admin user: Database error: `Execution Error: error returned from database: (code: 2067) UNIQUE constraint failed: users.email`: Execution Error: error returned from database: (code: 2067) UNIQUE constraint failed: users.email: error returned from database: (code: 2067) UNIQUE constraint failed: users.email
2023-12-10T22:18:02.306326767+00:00 INFO ｉ [info]: End.
I stopped lldap and manually updated the admin email via sqlite3 with
this command:
update users set email = ***@***.***' where user_id = 'admin';
Then, I restarted lldap and got stuck in a similar restart loop due to a
failed UNIQUE constrains on users.user_id. The following output will loop
every few seconds.
Loading configuration from /data/lldap_config.toml
2023-12-10T22:24:48.328008495+00:00 INFO set_up_server [ 1.37ms | 100.00% ]
2023-12-10T22:24:48.328020708+00:00 INFO ┝━ ｉ [info]: Starting LLDAP version 0.5.0
2023-12-10T22:24:48.333270604+00:00 WARN ┝━ 🚧 [warn]: Could not find an admin user, trying to create the user "admin" with the config-provided password
2023-12-10T22:24:48.333914867+00:00 ERROR ┕━ 🚨 [error]: | error: Database error: `Execution Error: error returned from database: (code: 1555) UNIQUE constraint failed: users.user_id`
2023-12-10T22:24:48.333964970+00:00 ERROR 🚨 [error]: Could not bring up the servers: while creating the admin user: Error setting up admin login/account: Error creating admin user: Database error: `Execution Error: error returned from database: (code: 1555) UNIQUE constraint failed: users.user_id`: Execution Error: error returned from database: (code: 1555) UNIQUE constraint failed: users.user_id: error returned from database: (code: 1555) UNIQUE constraint failed: users.user_id
2023-12-10T22:24:48.334002630+00:00 INFO ｉ [info]: End.
*Expected behavior*
I expect lldap to instantiate with a default admin account that I can use
to login and to complete my configuration.
*Additional context*
I can't get 0.4.3 to work on a fresh install either. lldap immediately
falls into a restart loop even though logs appear to be successful. The
following output will loop every few seconds.
> Setup permissions..
> Starting lldap..
Loading configuration from /data/lldap_config.toml
2023-12-10T22:37:21.478722792+00:00 INFO set_up_server [ 1.11ms | 100.00% ]
2023-12-10T22:37:21.478731288+00:00 INFO ┝━ ｉ [info]: Starting LLDAP version 0.4.3
2023-12-10T22:37:21.482793217+00:00 INFO ┝━ ｉ [info]: Starting the LDAP server on port 3890
2023-12-10T22:37:21.484134880+00:00 INFO ┕━ ｉ [info]: Starting the API/web server on port 17170
2023-12-10T22:37:21.484198849+00:00 INFO ｉ [info]: starting 1 workers
2023-12-10T22:37:21.484202095+00:00 INFO ｉ [info]: Actix runtime found; starting in Actix runtime
—
Reply to this email directly, view it on GitHub
<#756>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAGCPWIQW47HS6RP7QGB25DYIY3CFAVCNFSM6AAAAABAO4H7XOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGAZTINJWGUZDQNQ>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
|
Verbose log output from The user list: lldap_admin members: |
|
There's something weird, that cannot be the logs from a fresh install with an empty data directory: the database already exists (you can see that no migration was done, it directly returned the latest schema version). I'm guessing that there is an issue, and you're not looking at the correct /data directory. Can you post the LLDAP section of your docker compose? That, or it's not actually the very first logs you get after deleting the DB, only the second time you start the service. |
|
My
But, you raise a good point -- I was grabbing the most recent copy of the Here's a lldap_v050_startup.txt |
|
Is this work if using volume? And about the command, |
|
Interesting! Maybe you can remove the "restart: unless-stopped" from compose? That way you'd only get the first start. I think grafana only captured stdout, but docker will also give you stderr. Or can you get that in grafana as well? |
|
@martadinata666 I modified my Here's the initial output from But, checking My docker volumes are hosted via NFS via TrueNAS. I know it's not ideal to run sqlite dbs on NFS, but this setup ran fine for a long while on v0.4.x and now I can't even rollback to that version family. I can try a setup without sqlite, but probably not until middle of next week. @nitnelave Progress! I removed Thank you both for the troubleshooting help! |
|
132? That's illegal instruction (unless you have a weird Linux). That would indeed stop the program in it's tracks with no hope of logging, to stderr or stdout. A potential way forward would be to recompile lldap yourself (which is very easy with cargo, check the readme). If you compile it on the machine itself, it shouldn't generate illegal instructions. Let's see if it solves the problem (you should be able to just copy the new binary into the container and restart it). |
|
@martadinata666 To clarify my earlier comment about NFS. My normal process is to host persistent docker volumes on NFS, but in your test case using the |
|
What's the output of |
|
@nitnelave I'm running virtualized Ubuntu on TrueNAS with
OS info: CPU info: Docker info: Container info: |
|
|
You could try the non-alpine container, but I'm not sure it'd change anything. Actually if you could get a coredump, we could see where the invalid instruction is and try to see if there's anything specific to the dependency that we could change. |
|
Note that given the state of the DB after the crash, it's almost certainly in the crypto operations to set the password. @martadinata666 v4 to v5, could it be due to the libc change? I remember something about musl, did we change anything with the Alpine container? |
|
Could be relevant: https://gitlab.torproject.org/tpo/core/arti/-/issues/571 Note that SIGILL can also occur when panicking while processing a panic. |
|
Oh interesting, 132 is about missing cpu instructions. Mostly 1st let's rule out a VM issue or a Host issue:
|
|
FYI I'll be AFK until Tuesday of next week. I don't want you to think that I've ghosted you after all the support. I'll take a swing at the latest suggestions upon my return. |
|
Good news, it appears the issue was related to my virtualization settings in TrueNAS. Changing Thanks for the troubleshooting! |
martadinata666
changed the title
Describe the bug
lldapv0.5.0 is stuck in a restart loop on a fresh install due to failed UNIQUE database constraints in sqlite.To Reproduce
For context, I tried to upgrade from 0.4.3 to 0.5.0 and got the expected UNIQUE constraint error on emails that I had shared across different users. I updated the emails directly in
users.dbusingsqlite3butlldapcontinued to restart with failed UNIQUE constraints.So, I started with a fresh v0.5.0 install with a clean data directory with the following
docker composespecification:lldapcontinues to get stuck in a restart loop with this output:I stopped
lldapand manually updated theadminemail viasqlite3with this command:Then, I restarted
lldapand got stuck in a similar restart loop due to a failed UNIQUE constrains onusers.user_id. The following output will loop every few seconds.Expected behavior
I expect
lldapto instantiate with a default admin account that I can use to login and to complete my configuration.Additional context
I can't get 0.4.3 to work on a fresh install either.
lldapimmediately falls into a restart loop even though logs appear to be successful. The following output will loop every few seconds.The text was updated successfully, but these errors were encountered: