Bootstraping fails: node-image-pull.service fails

[ecatphi8]

Dear,

I'm trying to install a okd cluster on proxmox.

Bootstrap node:

• installer files created with openshift-install-mac-arm64-4.19.0-okd-scos.6
• Install coreos from image fedora-coreos-42.20250609.3.0-live-iso.x86_64.iso
• reboot
  during installation, service note-image-pull.service exits with an error (see log)

This looks like an "assert" in /var/usrlocal/bin/node-image-pull.sh

from /var/usrlocal/bin/node-image-pull.sh starting line 60

# ideally, `ostree container image pull` would support `--write-ref` or a
# command to escape a pullspec, but for now it's pretty easy to tell which ref
# it is since it's the only docker one
ref=$(ostree refs --repo "${ostree_repo}" | grep ^ostree/container/image/docker)
if [ $(echo "$ref" | wc -l) != 1 ]; then
    echo "Expected single docker ref, found:"
    echo "$ref"
    exit 1
fi

Any Idea on how to solve it ?

Failed Units: 1
  node-image-pull.service

core@bootstrap:~$ journalctl -b -f -u release-image.service -u bootkube.service -u node-image-pull.service
Jun 29 06:31:47 bootstrap.okd.phi8.ovh optimistic_dubinsky[3261]: quay.io/okd/scos-content@sha256:def6df145a7d1f834ccccbce7728a6abe88a043f08bede7a70dcf8621efb67d5
Jun 29 06:31:47 bootstrap.okd.phi8.ovh podman[2247]: 2025-06-29 06:31:47.938439041 +0000 UTC m=+18.119755861 container died 0f2e9394a4c2fa8c11d62873080b96b123dbe4cef1a4303e467f28674b1d4b56 (image=quay.io/okd/scos-release@sha256:0a16ec5bbc8e74b4a4286b96d385c5b65913ea2abb2146b93cb2e6287bc46319, name=optimistic_dubinsky, io.openshift.release.base-image-digest=sha256:7a347a24eee426990d9085e9c74a75dd0102d8d6f5b3c24abbd7894fe8b54b85, io.openshift.release=4.19.0-okd-scos.6)
Jun 29 06:31:47 bootstrap.okd.phi8.ovh podman[2247]: 2025-06-29 06:31:47.997299869 +0000 UTC m=+18.178616691 container remove 0f2e9394a4c2fa8c11d62873080b96b123dbe4cef1a4303e467f28674b1d4b56 (image=quay.io/okd/scos-release@sha256:0a16ec5bbc8e74b4a4286b96d385c5b65913ea2abb2146b93cb2e6287bc46319, name=optimistic_dubinsky, io.openshift.release=4.19.0-okd-scos.6, io.openshift.release.base-image-digest=sha256:7a347a24eee426990d9085e9c74a75dd0102d8d6f5b3c24abbd7894fe8b54b85)
Jun 29 06:31:48 bootstrap.okd.phi8.ovh node-image-pull.sh[2140]: Currently on CoreOS version 42.20250609.3.0
Jun 29 06:31:48 bootstrap.okd.phi8.ovh node-image-pull.sh[2140]: Target node image is quay.io/okd/scos-content@sha256:def6df145a7d1f834ccccbce7728a6abe88a043f08bede7a70dcf8621efb67d5
Jun 29 06:31:48 bootstrap.okd.phi8.ovh node-image-pull.sh[2140]: Not in live environment
Jun 29 06:31:48 bootstrap.okd.phi8.ovh node-image-pull.sh[2140]: Pulling quay.io/okd/scos-content@sha256:def6df145a7d1f834ccccbce7728a6abe88a043f08bede7a70dcf8621efb67d5
Jun 29 06:31:48 bootstrap.okd.phi8.ovh bootc[3297]: Fetching ostree-unverified-registry:quay.io/okd/scos-content@sha256:def6df145a7d1f834ccccbce7728a6abe88a043f08bede7a70dcf8621efb67d5
Jun 29 06:31:50 bootstrap.okd.phi8.ovh node-image-pull.sh[3297]: layers already present: 1; layers needed: 52 (1.4 GB)
Jun 29 06:31:50 bootstrap.okd.phi8.ovh bootc[3297]: layers already present: 1; layers needed: 52 (1.4 GB)
Jun 29 06:34:41 bootstrap.okd.phi8.ovh node-image-pull.sh[3297]: Image contains non-ostree compatible file paths: secrets: 2
Jun 29 06:34:41 bootstrap.okd.phi8.ovh node-image-pull.sh[3297]: Wrote: ostree-unverified-registry:quay.io/okd/scos-content@sha256:def6df145a7d1f834ccccbce7728a6abe88a043f08bede7a70dcf8621efb67d5 => 29bd5318a336a347f9a8ddc781c55c84b9fbe38efcc8bddf143299f5462e9e39
Jun 29 06:34:41 bootstrap.okd.phi8.ovh node-image-pull.sh[2140]: Expected single docker ref, found:
Jun 29 06:34:41 bootstrap.okd.phi8.ovh node-image-pull.sh[2140]: ostree/container/image/docker_3A__2F__2F_quay_2E_io/fedora/fedora-coreos_3A_stable
Jun 29 06:34:41 bootstrap.okd.phi8.ovh node-image-pull.sh[2140]: ostree/container/image/docker_3A__2F__2F_quay_2E_io/okd/scos-content_40_sha256_3A_def6df145a7d1f834ccccbce7728a6abe88a043f08bede7a70dcf8621efb67d5

[linuxcube]

Hi @ecatphi8 , are yo hosting your own images or hosting them via private registry. I ask because I am facing a similar issue when trying to deploy using a provate registry (harbor), but when I use quay as a private registry there is no issue is pulling the images into the bootstrap. Thoughts?

[ecatphi8]

Hi!

I'm trying to deploy using Internet located registries.
I made progress my using the iso advised by ./openshift-install

./openshift-install coreos print-stream-json | grep '\.iso[^.]' 

no success so far but the log is cleaner and this step is "solved"

hope it helps

[linuxcube]

Hi!

I'm trying to deploy using Internet located registries. I made progress my using the iso advised by ./openshift-install

./openshift-install coreos print-stream-json | grep '\.iso[^.]' 

no success so far but the log is cleaner and this step is "solved"

hope it helps

Hi @ecatphi8 , the point of using the registry is to avoid using the ISO images....I am not sure but is working fine under Quay, I would love to get it working under harbor. :-)

[linuxcube]

Reference:

[linuxcube]

Hi @ecatphi8 ,
I did observer the following under the okd bootstrap logs:

Jul 01 13:16:33 localhost.localdomain release-image-download.sh[890395]: Error: initializing source docker://registry.local.domain:443/origin/okd@sha256:a9ef388aca5e796b95ce0ee95609e4704080851fdc722bc473e386a3ec2b1f17: pinging container registry registry.local.domain:443: Get "https://registry.local.domain:443/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority

I am using a public certificate from digicert and when I browse the URL it loads find and I can see the secured connection and I validated the cert details via the URL as well so I am unsure why the okd bootstrap is not able to see the validaty of the cert. Please advise.

[linuxcube]

I finally got this sorted, the fix for me was to append the following to install-config.yaml

additionalTrustBundle

You may be able to read on this reference at the following URL: https://docs.okd.io/4.18/networking/configuring-a-custom-pki.html

Thank you all for the help and pointers.

[Nevah5]

Hey there, I got a similar issue, that I could not yet resolve. Would appreciate some ideas and fixes I should try.
I have the same log that is being logged: Expected single docker ref, found: (and then two refs)

See #2222 for more details.