RKE2 fails to import tarball produced by ctr image export

[nnewc]

Environmental Info:
RKE2 Version: v1.28.8-rke2r1

Node(s) CPU architecture, OS, and Version: SLE micro 5.5

Cluster Configuration: 3 servers, 3 agents

Describe the bug:
RKE2 fails to import a tarball in the specified folder

Steps To Reproduce:

• created tarball with ctr image export on another cluster
• copied tarball to /var/lib/rancher/rke2/agent/images

Expected behavior:

image tarball to be imported sucessfully

Actual behavior:
May 21 21:26:34 m-b1e67e24-9dea-4656-8838-8e546eb85963 rke2[12639]: time="2024-05-21T21:26:34Z" level=error msg="Error encountered while importing /var/lib/rancher/rke2/agent/images/cert-manager-1.13.6.tar: failed to import images from /var/lib/rancher/rke2/agent/images/cert-manager-1.13.6.tar: content digest sha256:a78a14ea69d0a1fe71eefc5ddabe6ed121db7bff2a284e0899000e6023e564b2: not found"

Additional context / logs:

tarball can be imported successfully with ctr -n k8s.io image import

[brandond]

ctr image export does not appear to provide the all the data required for the archive to be imported. We usually instruct users to use docker save:

brandond@dev01:~$ docker image save -o cert-manager-1.13.6.tar quay.io/jetstack/cert-manager-ctl:v1.13.6 quay.io/jetstack/cert-manager-webhook:v1.13.6 quay.io/jetstack/cert-manager-controller:v1.13.6 quay.io/jetstack/cert-manager-cainjector:v1.13.6
brandond@dev01:~$ tar -vtf cert-manager-1.13.6.tar
drwxr-xr-x 0/0               0 2024-04-25 09:19 blobs/
drwxr-xr-x 0/0               0 2024-05-21 23:34 blobs/sha256/
-rw-r--r-- 0/0            2822 2024-04-25 09:19 blobs/sha256/020ebb4d81616375dc72b85b2b0d13699b8adf3266acdafea12919c72010d424
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/070d36bbbda6b1c5475a57a7a4c5510a4ae04dd65dcda00dbc608fd6dba6817b
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/094a5ac05d8da7446a4307bb6a41afc7d47a086f2bb04b862472538039369612
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/0b83bb0d9474ad594e3b1c10aebf636ed19518aa205590f1d894e76b7bd6bb66
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/0f1cfd9cf807a59f33419eae37ce612e193ee692ec9c386a939d446242c899fe
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/0f252bc00e4c1e119d2f30d5443d712247e05e49a714a35b5fed9b07167f8586
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/174e1c1c14cdaa63cae68c6cb0724764b5e8c8f1cab87f0e26211d68bc691a5d
-rw-r--r-- 0/0           10240 2024-04-25 09:19 blobs/sha256/1a73b54f556b477f0a8b939d13c504a3b4f4db71f7a09c63afbc10acb3de5849
-rw-r--r-- 0/0            3584 2024-04-25 09:19 blobs/sha256/1aceb105f7d2bc5e4c9560069615ba3ca0cd45ef484acc162008b0623611acee
-rw-r--r-- 0/0             997 2024-04-25 09:19 blobs/sha256/1ca7d6f456528f447ca06f1a430e1a5544fe343b1132372ecda0b4c0ca9f816c
-rw-r--r-- 0/0        62868992 2024-04-25 09:19 blobs/sha256/230433ce6d90a3f202e1cb193197dae186c5efaa80d5f4cceb0438bb8487619c
-rw-r--r-- 0/0          225280 2024-04-25 09:19 blobs/sha256/2388d21e8e2b74e055216652e6af384768e97513ad568e10c27bacec389b7f0a
-rw-r--r-- 0/0            2410 1970-01-01 00:00 blobs/sha256/241011c1666259978d7416efeb9c9b3a86fe7018cf1e8042c7a6c5cfe22bc63b
-rw-r--r-- 0/0            1536 2024-04-25 09:19 blobs/sha256/2a92d6ac9e4fcc274d5168b217ca4458a9fec6f094ead68d99c77073f08caac1
-rw-r--r-- 0/0        53526016 2024-04-25 09:19 blobs/sha256/2b4857608f50fa003b2517df472c133441972040c66fec5b7b5ff6f7b5ad0c68
-rw-r--r-- 0/0          327680 2024-04-25 09:19 blobs/sha256/32ae37dc07be998c2ea491ff1a9826b2873cd0cf6b5c40ccfc65990bb649b7ad
-rw-r--r-- 0/0        46730752 2024-04-25 09:19 blobs/sha256/36b93c5fcf21065344a487b88771887d128dd816c645dd94586345b7bd03aa39
-rw-r--r-- 0/0            2410 1970-01-01 00:00 blobs/sha256/393c4424081b19ee81596a4060f327fa0beaaf45cb5a2747790be7af1f637c96
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/3d9f6452031afb205ea42d2fd5c6ea8445ae6cbf9cdb139b44c6a5c71f2c2a15
-rw-r--r-- 0/0            2773 2024-04-25 09:19 blobs/sha256/3fe5fc9c692924230b533fa473ccfdb4164257c4985c0fe1cef934d1f17b3925
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/472195ae382079c640905869d39f8ad470fa6b370f95d1fee21f7b51a40435cf
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/4852f8324416049b82fddeaec7f5ed73d97d617372f35ed1a7432b1a216ab438
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/4b11f105d1bbeae1667eca58f4fe79482804ca12e08f062ddec0c93ce1d69107
-rw-r--r-- 0/0            1536 2024-04-25 09:19 blobs/sha256/4d049f83d9cf21d1f5cc0e11deaf36df02790d0e60c1a3829538fb4b61685368
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/4ee90ac3fb04dfa1bc18f29d605703285819ec17b9b4404f84020eb7fdbeff1e
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/55eb4e308bdb36d114c4c367309b88f542ff95dd6de9d729b931399a21bbea61
-rw-r--r-- 0/0           51200 2024-04-25 09:19 blobs/sha256/577c8ee06f39e2bc276615f1058fa40081255ce5e2f072df4875e27868de5660
-rw-r--r-- 0/0            2560 2024-04-25 09:19 blobs/sha256/6c91c31541312a4eb1c1a486576b25c529aa2adc2e9e431b40f6eecc78a49bb8
-rw-r--r-- 0/0            3584 2024-04-25 09:19 blobs/sha256/72838896eb15cee4b9108ee943bcc7168d8784ada5474ad66ad1a2c03189bf09
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/72d10c7c4776b9f47fcc2df4ad9b78c7fa6d5260a57bebb1f1eece7c27d2d7c7
-rw-r--r-- 0/0            2410 1970-01-01 00:00 blobs/sha256/8c2710583724cef9d763b075025958d1906a865d75ebf059a0cbe9974e5d5f9b
-rw-r--r-- 0/0            1011 2024-04-25 09:19 blobs/sha256/919c5b0f3fa86c297b4cf7a63025d41bb00398a0f459af96dfed8e7ff5003289
-rw-r--r-- 0/0         3379200 2024-04-25 09:19 blobs/sha256/9ed498e122b248a801130d052c25418381ee7bf215cdf7990965bae0dc37dcc2
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/aa6519a19a1a6a6a64e6368894990d34d0c9196f28c903e41649ca3b4e09e088
-rw-r--r-- 0/0            2560 2024-04-25 09:19 blobs/sha256/ac805962e47900b616b2f4b4584a34ac7b07d64ac1fd2c077478cf65311addcc
-rw-r--r-- 0/0            2410 1970-01-01 00:00 blobs/sha256/adf2b3c5cee432ab8e03e353b430c1936d5f17388d0e4530f1b40cd80d7eed98
-rw-r--r-- 0/0            2560 2024-04-25 09:19 blobs/sha256/af5aa97ebe6ce1604747ec1e21af7136ded391bcabe4acef882e718a87c86bcc
-rw-r--r-- 0/0             401 2024-04-25 09:19 blobs/sha256/b2d9a5bab3100166e745285857953046b1f51084a7a585a8797444959dcb1583
-rw-r--r-- 0/0            2560 2024-04-25 09:19 blobs/sha256/bbb6cacb8c82e4da4e8143e03351e939eab5e21ce0ef333c42e637af86c5217b
-rw-r--r-- 0/0            2560 2024-04-25 09:19 blobs/sha256/bf97eafa40a2401f164df3147fc7c34efc2e5dddaa54452e94768a9483e5b847
-rw-r--r-- 0/0            3072 2024-04-25 09:19 blobs/sha256/c048279a7d9f8e94b4c022b699ad8e8a0cb08b717b014ce4af15afaf375a6ac2
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/c64c7ec879deaf8a49b27d4de6a4ea1380590b93e9cf6d5ba0c0584e2922bae0
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/ccc565ece537f9362fa2cb7b8bfc608daa4df53cc038e27e9635ffd3ed57c82f
-rw-r--r-- 0/0        39366144 2024-04-25 09:19 blobs/sha256/ce4e24779deffc93121567db6edf889cb801fd2abdd1eb5dd118600c07b41d77
-rw-r--r-- 0/0            1011 2024-04-25 09:19 blobs/sha256/d6af18dee33de4782dd06deba71be21e036286617f9f82a579b9e0c37f459871
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/d8816aeca80e4d8b680baabcea6be881cbe82b616ad2f7d5317cb9c50b8b744f
-rw-r--r-- 0/0            2801 2024-04-25 09:19 blobs/sha256/de523428601e4d357a76699a2384d4028f75b4ade1ebb8a9f2d035e50562cc5f
-rw-r--r-- 0/0             477 2024-04-25 09:19 blobs/sha256/e7c30dea6bbb3f209b5a1903dbe70715bca83cd93a0716d75f6ead0cc40234f9
-rw-r--r-- 0/0            2822 2024-04-25 09:19 blobs/sha256/ee4bf2cdf31433334433f7a62e2c6676a26a627cfefc3363e6d3654ae7510417
-rw-r--r-- 0/0            1005 2024-04-25 09:19 blobs/sha256/f5308b8d13ad5d15f565d68f39d8751bb8d5678ca80fa54252e8938798b22cdc
-rw-r--r-- 0/0            1449 2024-05-21 23:34 index.json
-rw-r--r-- 0/0           17800 1970-01-01 00:00 manifest.json
-rw-r--r-- 0/0              31 1970-01-01 00:00 oci-layout
-rw-r--r-- 0/0             480 1970-01-01 00:00 repositories

Compare that to the archive produced by ctr:

root@rke2-server-1:/# ctr image export cert-manager-1.13.6.tar quay.io/jetstack/cert-manager-ctl:v1.13.6 quay.io/jetstack/cert-manager-webhook:v1.13.6 quay.io/jetstack/cert-manager-controller:v1.13.6 quay.io/jetstack/cert-manager-cainjector:v1.13.6
root@rke2-server-1:/# tar -vtf cert-manager-1.13.6.tar
drwxr-xr-x 0/0               0 1970-01-01 00:00 blobs/
drwxr-xr-x 0/0               0 1970-01-01 00:00 blobs/sha256/
-r--r--r-- 0/0            2822 1970-01-01 00:00 blobs/sha256/020ebb4d81616375dc72b85b2b0d13699b8adf3266acdafea12919c72010d424
-r--r--r-- 0/0             270 1970-01-01 00:00 blobs/sha256/0a728406de952dcee21eb6793907b4469f6bcf2dd0b44f04354ec4e47c6d2e07
-r--r--r-- 0/0              75 1970-01-01 00:00 blobs/sha256/0f8b424aa0b96c1c388a5fd4d90735604459256336853082afb61733438872b5
-r--r--r-- 0/0              97 1970-01-01 00:00 blobs/sha256/1069fc2daed1aceff7232f4b8ab21200dd3d8b04f61be9da86977a34a105dfdc
-r--r--r-- 0/0            1728 1970-01-01 00:00 blobs/sha256/11c806b1561c32118b7e4c5017285a38296b0042e1927d41f68b1fa4987e7ebb
-r--r--r-- 0/0           20316 1970-01-01 00:00 blobs/sha256/2ae710cd8bfef4545fa3a6dc274d6b7a991ca379cdaa3cdf460d5cb5840a3c88
-r--r--r-- 0/0          122110 1970-01-01 00:00 blobs/sha256/307c1adadb60e6e9b8aca553ec620d77fedc112737cc54e9ee73ac165e7f3cbc
-r--r--r-- 0/0            3228 1970-01-01 00:00 blobs/sha256/37ec79b86e958581c2a7d9df36163828e391b97984294326a35aa0df8c7c88de
-r--r--r-- 0/0            2773 1970-01-01 00:00 blobs/sha256/3fe5fc9c692924230b533fa473ccfdb4164257c4985c0fe1cef934d1f17b3925
-r--r--r-- 0/0             327 1970-01-01 00:00 blobs/sha256/5318d93a3a6582d0351c833fa3cf04ab41352b2e6c77c9ec3d330581eb267683
-r--r--r-- 0/0            1728 1970-01-01 00:00 blobs/sha256/682fb3d893e04171f7c9651208ca59cfb2fbd555f0ea45988911f5acc68648c8
-r--r--r-- 0/0            1728 1970-01-01 00:00 blobs/sha256/689d029f027b8af34d37033a7bd1fb07f8a84048416a8f9540445c8dc43f19ad
-r--r--r-- 0/0        18268377 1970-01-01 00:00 blobs/sha256/68d0c4a82e6bfec36d7074bba2c9c76ec6a168d6eade204d037567f2b1857d1b
-r--r--r-- 0/0           83987 1970-01-01 00:00 blobs/sha256/7161956aee90ba03697f87be664e558a5a1331bbb6df61245273e874898d4da2
-r--r--r-- 0/0        17005642 1970-01-01 00:00 blobs/sha256/76cc199b8ab69b1dfaf82245720ed4b606e7e46977768c7dd1c33dd7aec27607
-r--r--r-- 0/0             633 1970-01-01 00:00 blobs/sha256/93411349b53323bae71aa24682c79ae1f6de508d3dac8a6aad54577e1ad47731
-r--r--r-- 0/0            3228 1970-01-01 00:00 blobs/sha256/9de910705c1251c5e7b282f0d80214cfac49659c150bab953a178026376715f1
-r--r--r-- 0/0             269 1970-01-01 00:00 blobs/sha256/a78d34652d6c56fb9e67567968f75ba937d3ca5ec8e2dae206c45c7f3e257bdb
-r--r--r-- 0/0             382 1970-01-01 00:00 blobs/sha256/b40161cd83fc5d470d6abe50e87aa288481b6b89137012881d74187cfbf9f502
-r--r--r-- 0/0             130 1970-01-01 00:00 blobs/sha256/c8022d07192eddbb2a548ba83be5e412f7ba863bbba158d133c9653bb8a47768
-r--r--r-- 0/0          634160 1970-01-01 00:00 blobs/sha256/d462aa3453675bb1f9a271a72cc72a53e628521a7d0e94b720bd07f9ca4962dc
-r--r--r-- 0/0             193 1970-01-01 00:00 blobs/sha256/d557676654e572af3e3173c90e7874644207fda32cd87e9d3d66b5d7b98a7b21
-r--r--r-- 0/0            3228 1970-01-01 00:00 blobs/sha256/d807aac5ea8cb08fce11c5f00b6f066310a9e2cad699e9680b6edf2f2eb92f98
-r--r--r-- 0/0             173 1970-01-01 00:00 blobs/sha256/d858cbc252ade14879807ff8dbc3043a26bbdb92087da98cda831ee040b172b3
-r--r--r-- 0/0        12502265 1970-01-01 00:00 blobs/sha256/dc8148130d5d037cc40819eedd20eca41a89ea796434904467a51887891d5f41
-r--r--r-- 0/0            2801 1970-01-01 00:00 blobs/sha256/de523428601e4d357a76699a2384d4028f75b4ade1ebb8a9f2d035e50562cc5f
-r--r--r-- 0/0             632 1970-01-01 00:00 blobs/sha256/e29cf116e5ea8e76ebb487a9966d2416c0c0cccfa99a9eae2726105929812576
-r--r--r-- 0/0            1728 1970-01-01 00:00 blobs/sha256/e77e12293f16da24744394c5525d570801c29c42cdae6cbd3f1089b63f42c04f
-r--r--r-- 0/0            3228 1970-01-01 00:00 blobs/sha256/e7fdbcae14d900d1cff763a5105125e97e3c0523752f399b74f9e7cf207e0f98
-r--r--r-- 0/0            2822 1970-01-01 00:00 blobs/sha256/ee4bf2cdf31433334433f7a62e2c6676a26a627cfefc3363e6d3654ae7510417
-r--r--r-- 0/0        14211275 1970-01-01 00:00 blobs/sha256/f7e27629f6bab88364158ffaac53d423acfebfc010134d8fa7b2fa8b8214e9ec
-rw-r--r-- 0/0            1259 1970-01-01 00:00 index.json
-rw-r--r-- 0/0            5135 1970-01-01 00:00 manifest.json
-r--r--r-- 0/0              30 1970-01-01 00:00 oci-layout

[brandond]

I believe this has do to with platform filtering.

docker pull pulls through the manifest list to retrieve only the image for the current platform, essentially transforming the target of the tag from a multi-platform manifest list, into a manifest. When you do docker image save, the resulting archive contains only the manifest and layers for the current platform.

ctr image pull does NOT lose the indirection provided by the manifest list, and saves out the original content - even if it is missing layers for platforms not present on the local machine. When you do ctr image export, the resulting tarball contains references to layers not present on the node, and the import fails.

I would probably recommend doing the following to ensure that all platforms referenced in the manifest are present in the tarball:

1. for IMAGE in quay.io/jetstack/cert-manager-ctl:v1.13.6 quay.io/jetstack/cert-manager-webhook:v1.13.6 quay.io/jetstack/cert-manager-controller:v1.13.6 quay.io/jetstack/cert-manager-cainjector:v1.13.6; do ctr image pull --all-platforms --local $IMAGE; done
2. ctr image export --all-platforms --local cert-manager-1.13.6.tar quay.io/jetstack/cert-manager-ctl:v1.13.6 quay.io/jetstack/cert-manager-webhook:v1.13.6 quay.io/jetstack/cert-manager-controller:v1.13.6 quay.io/jetstack/cert-manager-cainjector:v1.13.6

[brandond]

I've confirmed that the output produced by the above steps works:

May 22 00:15:49 systemd-node-1 rke2[14064]: time="2024-05-22T00:15:49Z" level=info msg="Importing images from /var/lib/rancher/rke2/agent/images/cert-manager-1.13.6.tar"
May 22 00:15:54 systemd-node-1 rke2[14064]: time="2024-05-22T00:15:54Z" level=info msg="Imported quay.io/jetstack/cert-manager-ctl:v1.13.6"
May 22 00:15:54 systemd-node-1 rke2[14064]: time="2024-05-22T00:15:54Z" level=info msg="Imported quay.io/jetstack/cert-manager-webhook:v1.13.6"
May 22 00:15:54 systemd-node-1 rke2[14064]: time="2024-05-22T00:15:54Z" level=info msg="Imported quay.io/jetstack/cert-manager-controller:v1.13.6"
May 22 00:15:54 systemd-node-1 rke2[14064]: time="2024-05-22T00:15:54Z" level=info msg="Imported quay.io/jetstack/cert-manager-cainjector:v1.13.6"
May 22 00:15:54 systemd-node-1 rke2[14064]: time="2024-05-22T00:15:54Z" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/cert-manager-1.13.6.tar in 4.688836673s"

If the OCI export format had some indicator of what platforms it contained content for, we could filter for those when importing… but instead we import with --all-platforms, which results in an error if there is content referenced for a platform not included in the export.

[brandond]

I think we could use containerd.WithSkipMissing to prevent this from failing the entire import if some content is missing.

I'll reopen to track this as an enhancement.

[nnewc]

FWIW, I tried the original above with docker save (though my issue may have been my Mac arm64 platform) and got the same unsuccessful results. However, the platform argument for ctr works and I have verified I can successfully import. Thanks so much for your help on this!

[mdrahman-suse]

Validation on master branch with commit 3aaa16c

Environment and config

Ubuntu 22.04, Single server

• config.yaml

write-kubeconfig-mode: 644
node-name: server1

Testing

• Install and start rke2 service
• Ensure cluster is up and running
• Deploy pod nginx (Can be from any other image)
• Check nginx is running and image is showed when listed
• Export the image
• Copy the image in to a new server and store it in path /var/lib/rancher/rke2/agent/images
• Install and start rke2 on the new server
• Ensure the image is shown when listed and no error related to image import is observed in the logs

Replication

$ rke2 -v
rke2 version v1.30.1+rke2r1 (e7f87c6dd56fdd76a7dab58900aeea8946b2c008)
go version go1.22.2 X:boringcrypto

• Create image and list image

$ kubectl run nginx --image=nginx
pod/nginx created

$ sudo /var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock --namespace k8s.io image export ngx-img.tar 
docker.io/library/nginx:latest

• Export image

$ sudo /var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock --namespace k8s.io image export nginx-image.tar docker.io/library/nginx:latest

• Check logs for error

$ sudo journalctl -u rke2-server | grep nginx-image
Jun 13 15:24:29 rke2[20027]: time="2024-06-13T15:24:29Z" level=info msg="Failed to find index.docker.io/rancher/rke2-runtime:v1.30.1-rke2r1 in /var/lib/rancher/rke2/agent/images/nginx-image.tar: tag rancher/rke2-runtime:v1.30.1-rke2r1 not found in tarball"
Jun 13 15:24:29 rke2[20027]: time="2024-06-13T15:24:29Z" level=info msg="Failed to find index.docker.io/rancher/rke2-runtime:v1.30.1-rke2r1 in /var/lib/rancher/rke2/agent/images/nginx-image.tar: tag rancher/rke2-runtime:v1.30.1-rke2r1 not found in tarball"
Jun 13 15:24:37 rke2[20027]: time="2024-06-13T15:24:37Z" level=info msg="Importing images from /var/lib/rancher/rke2/agent/images/nginx-image.tar"
Jun 13 15:24:39 rke2[20027]: time="2024-06-13T15:24:39Z" level=error msg="Error encountered while importing /var/lib/rancher/rke2/agent/images/nginx-image.tar: failed to import images from /var/lib/rancher/rke2/agent/images/nginx-image.tar: content digest sha256:c36751d7f6324d496c55c7fae626cda3efcc62b11a3b6cfb28d5ae53d8d6fc7a: not found"

$ sudo /var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock --namespace k8s.io images ls | grep nginx:latest
$

Validation

$ rke2 -v
rke2 version v1.30.1+dev.3aaa16c9 (3aaa16c9b17da45e9f3475ba5011ed90a49a2e42)
go version go1.22.2 X:boringcrypto

$ sudo journalctl -u rke2-server | grep nginx-image
Jun 13 15:32:11 rke2[28786]: time="2024-06-13T15:32:11Z" level=info msg="Failed to find index.docker.io/rancher/rke2-runtime:v1.30.1-dev.3aaa16c9 in /var/lib/rancher/rke2/agent/images/nginx-image.tar: tag rancher/rke2-runtime:v1.30.1-dev.3aaa16c9 not found in tarball"
Jun 13 15:32:19 rke2[28786]: time="2024-06-13T15:32:19Z" level=info msg="Importing images from /var/lib/rancher/rke2/agent/images/nginx-image.tar"
Jun 13 15:32:23 rke2[28786]: time="2024-06-13T15:32:23Z" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/nginx-image.tar in 3.43434656s"

$ sudo /var/lib/rancher/rke2/bin/ctr -a /run/k3s/containerd/containerd.sock -n k8s.io images ls | grep nginx:latest
docker.io/library/nginx:latest