-
Notifications
You must be signed in to change notification settings - Fork 232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] OKD 4.12 does not start because of missing kubeadmin-password file #3635
Comments
another setup and start :
So no valid url shows and openshift does not start:
|
Workaround : set an arbitrary password in the kubeadmin-password file before start. It wil for some reason not update kubeconfig:
and not print the usual login links. This should work with the content of the kubeadmin-password file as password:
The issue seems to be that the kubeadmin-password file is not created and on top of that that the kubeconfig file cannot be found, although there is on in multiple sub directories:
This all works as expected on another machine with 4.11 okd and an older crc version. FYI I did try a full clean and reinstall multiple times with no succes, but somehow OCP 4.12.13 seemed to work without issues. |
@VGerris This file is created during |
I am also experiencing the same problem |
It's only done for the openshift preset at the moment, but okd also expects a kubeadmin file to be created. This should fix part of crc-org#3635
It's only done for the openshift preset at the moment, but okd also expects a kubeadmin file to be created. This should fix part of crc-org#3635
This branch should help, some codepaths are only used for openshift bundles when they should be used for both openshift and okd. |
I tried both https://github.com/cfergeau/crc "okd" branch and https://github.com/crc-org/crc HEAD of the master branch. There are some issues with both, but the fixes in the "okd" branch gave me some good hints on what to fix in the master branch. Bug #2857 also causes Given that the "okd" preset isn't receiving all that much love from maintainers would it perhaps make more sense to have
With that fix in current HEAD the "okd" preset seems to work ok. |
I just wanted to add, but already see people noticed: this concerns OKD We do not provide support or code changes for OKD. This is based on community support. So, if you can, please provide a PR to fix this. The reason for these codepaths is that this was originally maintained as a fork. We have since backported those changes due to lack of interest of and work on the fork. We wanted to prevent them from affected our critical flow of delivering the OCP-based release. |
There are numerous places in the CRC code base that calls IsOpenShift() to run logic that is not intended for Podman setups or other more limited setups. Using the "okd" preset gives you something that is very close to the "openshift" preset. CrcBundleInfo's IsOpenShift() now treats the "okd" preset as if it was OpenShift. This means we can avoid littering the code with checks for both OpenShift and OKD in a bunch of places, which also reduces the risk of forgetting to check for OKD. Fixes crc-org#3635
There are numerous places in the CRC code base that calls IsOpenShift() to run logic that is not intended for Podman setups or other more limited setups. Using the "okd" preset gives you something that is very close to the "openshift" preset. CrcBundleInfo's IsOpenShift() now treats the "okd" preset as if it was OpenShift. This means we can avoid littering the code with checks for both OpenShift and OKD in a bunch of places, which also reduces the risk of forgetting to check for OKD. Fixes crc-org#3635
I think in this case it might have been a forgotten case due to the addition of Microshift. Anyway, it runs the tests now. not sure if this will be included in the release of next week. |
Thanks for testing/improving my branch! :) |
I can confirm okd 4.12 works as expected by building : https://github.com/crc-org/crc .(UPDATE - does not start)
after cleanup I ran setup and started, but :
right before that at start :
so same issue but now with an ssh time out before. @praveenkumar there are no permission issues. Again, the workaround I described fixes it so it seems that kubeadmin file is still not being created?
thanks |
scratch that, it is down. I can try the other branch but would appreciate it if this can be looked into further, especially if nothing specific is needed for OKD. Thank you. |
Apologies for writing here again, but using the 2.19 version of crc works if one does something like: So if in the code there is a simply way to make that happen I think it is ok. |
crc/pkg/crc/machine/machine.go Lines 16 to 27 in 2468363
|
I made a PR : https://github.com/crc-org/crc/pull/3679/files I am not sure if it can be further improved given : N.B. Note the error I got before, not sure where that comes from. |
Just ran into this and assumed it was an issue with the ssl certs being too old. Would it be possible for crc to print the actual error instead of what I'd consider a "red herring" about ssl certs expiry? Only found this issue because of enabling |
There are several occurrences of IsOpenshift() checks that were problematic on OKD which is why I opted to have the IsOpenshift() check for both the "openshift" and "okd" presets in my PR rather than sprinkle |
I just installed the recently released 2.20. This specific issue is resolved, but OKD does not start with many errors like :
I wrote earlier what I tried to fix this and where I got stuck. I am happy to help out if someone can bring me up to speed with how the bundles work and are created so I can look further into this. pm me at github_username_at_gmail_ .thanks |
Are you using nested virtualization? |
hi, no I just run it as is on Fedora 38. With the VM I means the crc machine created by the installer. |
I was able to get crc w/ okd to work on fedora 38 (and windows) after building the version w/ your fix -- thanks! @VGerris timeouts could be related to it being too slow during hte bootstrapping; what are you using for CPU/Memory config on crc? The defaults are a bit too low IMO.. I set it for 8 cpu and 16 gb ram, but something closer to 4 cpu and 8 gb ram is probably a more reasonable minimum than the current default. After you change the defaults you will need to run If that doesn't work (and back to my original comment) try doing a fresh VM setup/init using the |
hi, I looked into it again, it is not a resource issue it is the bug I encountered and mentioned in the linked bug: login : ssh -i /home//.crc/machines/crc/id_ecdsa -p 22 core@192.168.130.11 I am unable to get it to start and work again ( now on 4.13 OKD 2.20 of crc) oc logs etcd-crc-cd8kd-master-0 -n openshift-etcd --kubeconfig=/opt/kubeconfig [update] [update3]
it comes straight under enp2s0 with 192.168.130.11 address. [update4] |
Ok, this is another ugly workaround that made my instance start.
The above got my cluster working but only the developer account seemed to exist. The authentication is configured with htacess. To have oc run as admin and fix the kubeadmin user you can do the following ( being logged in to the VM):
At restart I get : Failed to update pull secret on the disk: Temporary error: pull secret not updated to disk (x207) I cannot find how the mentioned var is set so not sure how this should be fixed. The root of the issue is that the kubelet service uses the wrong IP to look at and start etcd , which is set by KUBE_NODE_IP in the systemd file. I will leave this for a bit now and hope someone can clarify this further and/or fix it structurally, thank you. |
Looks like this is issue of our bundle generation script https://github.com/crc-org/snc/blob/master/createdisk.sh#L116-L122 we are setting right internal IP in case of OCP but not in case of OKD, we will put a PR and may be when next bundle is generated shouldn't have any issue. |
crc-org/snc#733 is how it should be handled as part of bundle generation. |
I tested with the latest crc and it works like a charm. |
General information
crc setup
before starting it (Yes)?CRC version
CRC status
crc status --log-level debug DEBU CRC version: 2.18.0+4ea3a1 DEBU OpenShift version: 4.12.13 DEBU Podman version: 4.4.1 DEBU Running 'crc status' CRC VM: Running OpenShift: Stopped (v4.12.0-0.okd-2023-02-18-033438) RAM Usage: 5.292GB of 25.22GB Disk Usage: 16.23GB of 32.68GB (Inside the CRC VM) Cache Usage: 38.11GB Cache Directory: /home/user/.crc/cache
CRC config
crc status --log-level debug DEBU CRC version: 2.18.0+4ea3a1 DEBU OpenShift version: 4.12.13 DEBU Podman version: 4.4.1 DEBU Running 'crc status' CRC VM: Running OpenShift: Stopped (v4.12.0-0.okd-2023-02-18-033438) RAM Usage: 5.292GB of 25.22GB Disk Usage: 16.23GB of 32.68GB (Inside the CRC VM) Cache Usage: 38.11GB Cache Directory: /home/user/.crc/cache
Host Operating System
Steps to reproduce
Expected
succesfull setup
Actual
fails with :
Failed to update kubeadmin user password: Cannot generate the kubeadmin user password: open /home/user/.crc/machines/crc/kubeadmin-password: no such file or directory
Indeed the directory and the file do not exist, which seems to be the cause of the issue.
If I set these up with an empty file, the start continues and then complains about the machine qemu-1-crc existing, which seems to be the crc machine. When I remove that, the same problem occurs.
The issue does not seem to happen with OCP 4.12, only OKD.
Logs
Before gather the logs try following if that fix your issue
Please consider posting the output of
crc start --log-level debug
on http://gist.github.com/ and post the link in the issue.Snippet of debug log:
DEBU error: Temporary error: pull secret not updated to disk - sleeping 2s
DEBU retry loop: attempt 41
DEBU Running SSH command:
DEBU SSH command succeeded
DEBU error: Temporary error: pull secret not updated to disk - sleeping 2s
DEBU retry loop: attempt 42
DEBU Running SSH command:
DEBU SSH command succeeded
then
Failed to update kubeadmin user password: Cannot generate the kubeadmin user password: open /home/user/.crc/machines/crc/kubeadmin-password: no such file or directory
Failed to update kubeadmin user password: Cannot generate the kubeadmin user password: open
The text was updated successfully, but these errors were encountered: