[Installation] Missing DB after helm deploy. #16
Comments
|
After a bit of mucking around, I was able to get the UI to work.
I'll try to see why the initdb doesn't run. |
rubenhak
added
kind/bug
|
hi @LaurentDumont, Error 500 is explainable if there is missing DB. Are you upgrading from an earlier version? Seems like mysql crash restarted. Do you have logs from initial launch? |
|
I think this was from a clean state. Before manually fixing the issue and letting it run since, I deleted the mysql folder in the pv. It's a bit strange that the mysql container starts up and tries to recover stuff. Would there be anything else persistent beyond the volume that could play around with the mysql container? |
|
There is something a bit peculiar I saw. I was tailing the logs and it seems that the mysql container "restarts" three times during the helm install process. Initial logs Logs after a few minutes I can see that the PID changed three times to finally settle on 1. It's a bit strange because inside the volume, I can clearly see some bin files related to the kubevious database. |
|
I do believe this might be related to POSIX permissions on the volume? It's a gluster volume without any specific UID set. #################### MYSQL CONTAINER CRASH AND RESTARTS #################### This starts the XA crash recovery process What is the expected permissions + user/group for the volume? |
|
@LaurentDumont, I think mysql is running as root, and it needs 700 permissions for /var/lib/mysql which is mounted to pvc. If it was caused by invalid permissions, how would the server recover on its own after 3rd restart? I was googling around for solution and came across an issue request. Then after reading through realized it was you :) |
|
Not too sure why it works after the crash :( This is what a succesful database init and deployment looks like. It works if I change the volume type to HostPath instead of the Gluster server I'm using.
So it's probably something specific to my setup. I think we can close the issue and I'll dig around a bit more :) |
|
This is a little bit hard to debug considering specifics of each environment. I'll be in the loop at the docker-lib/mysql in case they have additional questions. Could you show exactly what you changed to make this work? I can expose the variable for a better control. |
|
Just the volume type. "Production" is using Gluster volumes as PV for Kubernetes. Just to test, I tried using HostPath (which binds a local path on the Kubernetes node to the container running on it ) and it allowed the mysql init to proceed correctly. |
|
Did you do that by changing the storage class or by completely replacing the volumeClaimTemplate with a volume? |
|
Alright, I posted in the SQL issue but it looks like the root cause was the Liveness probe that went over threshold on my setup. It worked with the local storage, probably because the performances we're better? I don't think it's anything specifically related to the Helm values. If anything, maybe the Liveness probe initial delay could be increased. I've increased mine to 180 seconds (but the actual init process is much quicker - maybe around 50-70 seconds) docker-library/mysql#658 (comment) I'll close this, thank you everyone! |
|
@LaurentDumont, That was quick to root cause. The timeout can be increased and even made editable, but I'm afraid that with such slow storage everything else would run super slow. There is quite a lot of stuff store in the database to make the time machine feature possible. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
To Reproduce
Steps to reproduce the behavior:
helm upgrade --atomic -i kubevious kubevious/kubevious --version 0.4.24 -n kubeviousExpected behavior
A clear and concise description of what you expected to happen.
Logs
Logs from the UI
Logs from the parser
Logs from mysql
Logs from the app
Environment Details:
The text was updated successfully, but these errors were encountered: