Problems with disconnected agent-based single node installation of 4.19.0-okd-scos.9

[krzysztofpolito]

I can install 4.19.0-okd-scos.9 release with agent install without any problems in online mode, when all packages are downloaded from quay.io. However, problems appear when I try to mirror the packages to my own registry (I use harbor) and perform disconnected installation.

First, I do mirror data with the following script:

OCP_RELEASE=4.19.0-okd-scos.9
LOCAL_REGISTRY='hub.home.krzysp.ovh'
LOCAL_REPOSITORY='okd/4.19'
PRODUCT_REPO='okd'
LOCAL_SECRET_JSON='./secret.json'
RELEASE_NAME="scos-release"

oc adm release mirror -a ${LOCAL_SECRET_JSON} --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE} --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}

It works, 193 images are mirrored, at the end there is an information that I should add this section:

imageContentSources:
- mirrors:
  - hub.home.krzysp.ovh/okd/4.19
  source: quay.io/okd/scos-content
- mirrors:
  - hub.home.krzysp.ovh/okd/4.19
  source: quay.io/okd/scos-release

to the install-config.yaml. ISO is generated, but when booted from it, the installation tries to download a package, which is not in the mirror:

Aug 22 08:22:47 okd podman[13039]: 2025-08-22 08:22:47.305763427 +0000 UTC m=+0.243546609 image pull-error quay.io/okd/scos-content@sha256:5f9d2b6532fa485ae6effb1098ea7c051059c4a80a5d3dce2c3eff7afe710c1a initializing source docker://quay.io/okd/scos-content@sha256:5f9d2b6532fa485ae6effb1098ea7c051059c4a80a5d3dce2c3eff7afe710c1a: (Mirrors also failed: [hub.home.krzysp.ovh/okd/4.19@sha256:5f9d2b6532fa485ae6effb1098ea7c051059c4a80a5d3dce2c3eff7afe710c1a: reading manifest sha256:5f9d2b6532fa485ae6effb1098ea7c051059c4a80a5d3dce2c3eff7afe710c1a in hub.home.krzysp.ovh/okd/4.19: unknown: artifact okd/4.19@sha256:5f9d2b6532fa485ae6effb1098ea7c051059c4a80a5d3dce2c3eff7afe710c1a not found]): quay.io/okd/scos-content@sha256:5f9d2b6532fa485ae6effb1098ea7c051059c4a80a5d3dce2c3eff7afe710c1a: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io on 192.168.5.1:53: server misbehaving

What am I doing wrong? I understand that the mirror command should mirror all, but for me it looks like it's missing something.
I'm attaching full log of the mirror command, my agent-config.yaml and install-config.yaml files.
agent-config.yaml
install-config.yaml
mirror.log

[alexander-bruun]

I don't see any immediate issues with the agent or install config.

To me it sounds like your mirror is not including all manifests or entirely missing artifacts or looking for the artifacts in the wrong "directory" of the registry. I have experienced issues using v1 oc mirror for disconnected environments mirrors. Try using v2 oc mirror commands (oc mirror --v2 --help). If this does not fix the issue, then you could check the database status by running agent gather from the "bootstrap" node, thought i expect you would see the same log just in a different format.

ssh core@<node-ip> sudo /usr/local/bin/agent-gather -O > <local tmp path>/agent-gather.tar.xz

Agent gather example taken from: https://access.redhat.com/solutions/7016487

[krzysztofpolito]

Thank you @alexander-bruun for reply.
I tried to switch to v2 oc mirror plugin, but failed with specifying the ImageSetConfiguration file. In OKD docs: https://docs.okd.io/4.19/disconnected/mirroring/about-installing-oc-mirror-v2.html
there is an example of ISC:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
  platform:
    channels:
    - name: stable-4.19 
      minVersion: 4.19.2
      maxVersion: 4.19.2
    graph: true 

(I will skip operators and additional images, for the time being).

So, this example seems to download true OpenShift, not OKD. I tried this:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
mirror:
  platform:
    channels:
      - name: stable-4.19
        minVersion: 4.19.0-okd-scos.9
        maxVersion: 4.19.0-okd-scos.9

but it finds nothing to mirror:

2025/08/28 13:18:27  [INFO]   : 👋 Hello, welcome to oc-mirror
2025/08/28 13:18:27  [INFO]   : ⚙  setting up the environment for you...
2025/08/28 13:18:27  [INFO]   : 🔀 workflow mode: mirrorToMirror 
2025/08/28 13:18:27  [INFO]   : 🕵  going to discover the necessary images...
2025/08/28 13:18:27  [INFO]   : 🔍 collecting release images...
2025/08/28 13:18:28  [INFO]   : 🔍 collecting operator images...
2025/08/28 13:18:28  [INFO]   : 🔍 collecting additional images...
2025/08/28 13:18:28  [INFO]   : 🔍 collecting helm images...
2025/08/28 13:18:28  [INFO]   : ❗ No images to mirror...
2025/08/28 13:18:28  [INFO]   : mirror time     : 686.422428ms
2025/08/28 13:18:28  [INFO]   : 👋 Goodbye, thank you for using oc-mirror
2025/08/28 13:18:28  [ERROR]  : [Executor] no images to copy 

What values should I use for OKD?
Is there a place where I could find working examples or mirroring OKD with oc mirror v2?

[krzysztofpolito]

I think I'm now on the right track, but according to this doc:
https://github.com/openshift/oc-mirror/blob/release-4.21/v2/docs/okd-mirror.md
I have to set this env variable: OCP_SIGNATURE_URL="https://storage.googleapis.com/openshift-ci-release/releases/signatures/openshift/release/"
but to me it appears that there is no signature under this location. Where to find it?