Nextcloud - Failed to pull image while updating / fresh installing

[UnderEu]

App Name

Nextcloud

SCALE Version

22.02.1

App Version

15.2.20

Application Events

Application Events
2022-07-22 1:53:54
Back-off pulling image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0"
2022-07-22 1:55:06
Back-off pulling image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0"
2022-07-22 1:55:06
Error: ImagePullBackOff
2022-07-22 1:55:05
Error: ErrImagePull
2022-07-22 1:55:05
Failed to pull image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0": rpc error: code = Unknown desc = Error response from daemon: manifest for tccr.io/truecharts/nextcloud-fpm@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0 not found: manifest unknown: manifest unknown
2022-07-22 1:55:01
Pulling image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0"
2022-07-22 1:53:54
Error: ImagePullBackOff
2022-07-22 1:55:01
Add eth0 [172.16.13.195/16] from ix-net
Successfully assigned ix-nextcloud/nextcloud-cronjob-27641095-fr682 to ix-truenas
2022-07-22 1:55:00
Created pod: nextcloud-cronjob-27641095-fr682
2022-07-22 1:55:00
Created job nextcloud-cronjob-27641095
2022-07-22 1:53:26
Error: ErrImagePull
2022-07-22 1:53:42
Failed to pull image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0": rpc error: code = Unknown desc = Error response from daemon: manifest for tccr.io/truecharts/nextcloud-fpm@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0 not found: manifest unknown: manifest unknown
2022-07-22 1:53:13
Pulling image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0"
2022-07-22 1:53:26
Failed to pull image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0": rpc error: code = Unknown desc = Error response from daemon: Get "https://quay.io/v2/truecharts/nextcloud-fpm/manifests/sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0": received unexpected HTTP status: 504 Gateway Time-out
2022-07-22 1:52:57
Started container nextcloud-redis
2022-07-22 1:52:57
Created container nextcloud-redis
2022-07-22 1:52:56
Container image "ghcr.io/truecharts/redis:v7.0.2@sha256:5a76d7fa238ffbeb19f1620993fa0fa0015d2e4bb91b21f8ccfb961f595c7fc7" already present on machine
2022-07-22 1:52:56
Started container prepare
2022-07-22 1:52:55
Created container prepare
2022-07-22 1:52:46
Applied LoadBalancer DaemonSet ix-nextcloud/svclb-nextcloud
2022-07-22 1:52:55
Started container prepare
2022-07-22 1:52:55
Container image "ghcr.io/truecharts/multi-init:v0.0.1@sha256:41185dcea1e6f6a035d8090da40aecc6a69cef66b91dc1332a90c9d22861d367" already present on machine
2022-07-22 1:52:55
Created container prepare
2022-07-22 1:52:55
Add eth0 [172.16.13.194/16] from ix-net
2022-07-22 1:52:53
Container image "ghcr.io/truecharts/multi-init:v0.0.1@sha256:41185dcea1e6f6a035d8090da40aecc6a69cef66b91dc1332a90c9d22861d367" already present on machine
2022-07-22 1:52:53
Add eth0 [172.16.13.193/16] from ix-net
2022-07-22 1:52:52
Started container nextcloud-postgresql
2022-07-22 1:52:52
Created container nextcloud-postgresql
2022-07-22 1:52:51
Container image "ghcr.io/truecharts/postgresql:v14.4.0@sha256:420fb9ab413776cbde2b29ed942c24a8ddeb10e96086ed823299ee89b4c309c0" already present on machine
2022-07-22 1:52:51
Started container prepare
2022-07-22 1:52:51
Created container prepare
2022-07-22 1:52:50
LoadBalancer Ingress IP addresses updated: 192.168.15.101
2022-07-22 1:52:50
Container image "ghcr.io/truecharts/multi-init:v0.0.1@sha256:41185dcea1e6f6a035d8090da40aecc6a69cef66b91dc1332a90c9d22861d367" already present on machine
2022-07-22 1:52:50
Add eth0 [172.16.13.192/16] from ix-net
2022-07-22 1:52:48
Started container lb-tcp-10020
2022-07-22 1:52:48
Created container lb-tcp-10020
2022-07-22 1:52:48
Container image "rancher/klipper-lb:v0.3.4" already present on machine
2022-07-22 1:52:48
Add eth0 [172.16.13.191/16] from ix-net
Successfully assigned ix-nextcloud/nextcloud-postgresql-0 to ix-truenas
Successfully assigned ix-nextcloud/nextcloud-789d9d989d-wnjx5 to ix-truenas
Successfully assigned ix-nextcloud/nextcloud-redis-0 to ix-truenas
2022-07-22 1:52:47
Successfully provisioned volume pvc-b982c239-f407-4daa-8ce5-b257cf1bead6
2022-07-22 1:52:47
Successfully provisioned volume pvc-7f1c80c9-d2cd-4db2-83f9-c7e7226e3613
2022-07-22 1:52:47
Successfully provisioned volume pvc-9fc3a26e-9877-42f7-a621-2f0a5e628ced
2022-07-22 1:52:47
Successfully provisioned volume pvc-aa92e201-a6b8-40af-9ca9-f378f30f008b
2022-07-22 1:52:46
create Pod nextcloud-postgresql-0 in StatefulSet nextcloud-postgresql successful
2022-07-22 1:52:46
create Claim db-nextcloud-postgresql-0 Pod nextcloud-postgresql-0 in StatefulSet nextcloud-postgresql success
2022-07-22 1:52:46
waiting for a volume to be created, either by external provisioner "zfs.csi.openebs.io" or manually created by system administrator
0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
2022-07-22 1:52:46
External provisioner is provisioning volume for claim "ix-nextcloud/db-nextcloud-postgresql-0"
2022-07-22 1:52:46
create Pod nextcloud-redis-0 in StatefulSet nextcloud-redis successful
0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
2022-07-22 1:52:46
waiting for a volume to be created, either by external provisioner "zfs.csi.openebs.io" or manually created by system administrator
2022-07-22 1:52:46
External provisioner is provisioning volume for claim "ix-nextcloud/data-nextcloud-redis-0"
2022-07-22 1:52:46
create Claim data-nextcloud-redis-0 Pod nextcloud-redis-0 in StatefulSet nextcloud-redis success
0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
2022-07-22 1:52:46
waiting for a volume to be created, either by external provisioner "zfs.csi.openebs.io" or manually created by system administrator
2022-07-22 1:52:46
Created pod: nextcloud-789d9d989d-wnjx5
Successfully assigned ix-nextcloud/svclb-nextcloud-r9qnj to ix-truenas
2022-07-22 1:52:46
waiting for a volume to be created, either by external provisioner "zfs.csi.openebs.io" or manually created by system administrator
2022-07-22 1:52:46
Created pod: svclb-nextcloud-r9qnj
2022-07-22 1:52:46
Scaled up replica set nextcloud-789d9d989d to 1
2022-07-22 1:52:46
External provisioner is provisioning volume for claim "ix-nextcloud/nextcloud-data"
2022-07-22 1:52:46
External provisioner is provisioning volume for claim "ix-nextcloud/nextcloud-html"

Application Logs

Pod Name:nextcloud-postgresql-0

2022-07-22 04:52:52.535480+00:00�[38;5;6mpostgresql �[38;5;5m04:52:52.53 �[0m
2022-07-22 04:52:52.538039+00:00�[38;5;6mpostgresql �[38;5;5m04:52:52.53 �[0m�[1mWelcome to the Bitnami postgresql container�[0m
2022-07-22 04:52:52.539450+00:00�[38;5;6mpostgresql �[38;5;5m04:52:52.53 �[0mSubscribe to project updates by watching �[1mhttps://github.com/bitnami/containers�[0m
2022-07-22 04:52:52.541769+00:00�[38;5;6mpostgresql �[38;5;5m04:52:52.54 �[0mSubmit issues and feature requests at �[1mhttps://github.com/bitnami/containers/issues�[0m
2022-07-22 04:52:52.545723+00:00�[38;5;6mpostgresql �[38;5;5m04:52:52.54 �[0m
2022-07-22 04:52:52.614730+00:00�[38;5;6mpostgresql �[38;5;5m04:52:52.61 �[0m�[38;5;2mINFO �[0m ==> ** Starting PostgreSQL setup **
2022-07-22 04:52:52.644367+00:00�[38;5;6mpostgresql �[38;5;5m04:52:52.64 �[0m�[38;5;2mINFO �[0m ==> Validating settings in POSTGRESQL_* env vars..
2022-07-22 04:52:52.652670+00:00�[38;5;6mpostgresql �[38;5;5m04:52:52.65 �[0m�[38;5;2mINFO �[0m ==> Loading custom pre-init scripts...
2022-07-22 04:52:52.665644+00:00�[38;5;6mpostgresql �[38;5;5m04:52:52.66 �[0m�[38;5;2mINFO �[0m ==> Initializing PostgreSQL database...
2022-07-22 04:52:52.687786+00:00�[38;5;6mpostgresql �[38;5;5m04:52:52.68 �[0m�[38;5;2mINFO �[0m ==> pg_hba.conf file not detected. Generating it...
2022-07-22 04:52:52.689862+00:00�[38;5;6mpostgresql �[38;5;5m04:52:52.68 �[0m�[38;5;2mINFO �[0m ==> Generating local authentication configuration
2022-07-22 04:52:54.477188+00:00�[38;5;6mpostgresql �[38;5;5m04:52:54.47 �[0m�[38;5;2mINFO �[0m ==> Starting PostgreSQL in background...
2022-07-22 04:52:54.905522+00:00�[38;5;6mpostgresql �[38;5;5m04:52:54.90 �[0m�[38;5;2mINFO �[0m ==> Changing password of postgres
2022-07-22 04:52:54.932469+00:00�[38;5;6mpostgresql �[38;5;5m04:52:54.93 �[0m�[38;5;2mINFO �[0m ==> Creating user nextcloud
2022-07-22 04:52:54.965316+00:00�[38;5;6mpostgresql �[38;5;5m04:52:54.96 �[0m�[38;5;2mINFO �[0m ==> Granting access to "nextcloud" to the database "nextcloud"
2022-07-22 04:52:55.060995+00:00�[38;5;6mpostgresql �[38;5;5m04:52:55.06 �[0m�[38;5;2mINFO �[0m ==> Setting ownership for the 'public' schema database "nextcloud" to "nextcloud"
2022-07-22 04:52:55.101518+00:00�[38;5;6mpostgresql �[38;5;5m04:52:55.10 �[0m�[38;5;2mINFO �[0m ==> Configuring replication parameters
2022-07-22 04:52:55.144123+00:00�[38;5;6mpostgresql �[38;5;5m04:52:55.14 �[0m�[38;5;2mINFO �[0m ==> Configuring synchronous_replication
2022-07-22 04:52:55.146756+00:00�[38;5;6mpostgresql �[38;5;5m04:52:55.14 �[0m�[38;5;2mINFO �[0m ==> Configuring fsync
2022-07-22 04:52:55.179832+00:00�[38;5;6mpostgresql �[38;5;5m04:52:55.17 �[0m�[38;5;2mINFO �[0m ==> Loading custom scripts...
2022-07-22 04:52:55.186344+00:00�[38;5;6mpostgresql �[38;5;5m04:52:55.18 �[0m�[38;5;2mINFO �[0m ==> Enabling remote connections
2022-07-22 04:52:55.197972+00:00�[38;5;6mpostgresql �[38;5;5m04:52:55.19 �[0m�[38;5;2mINFO �[0m ==> Stopping PostgreSQL...
2022-07-22 04:52:55.302994+00:00waiting for server to shut down.... done
2022-07-22 04:52:55.303045+00:00server stopped
2022-07-22 04:52:55.309870+00:00�[38;5;6mpostgresql �[38;5;5m04:52:55.30 �[0m�[38;5;2mINFO �[0m ==> ** PostgreSQL setup finished! **
2022-07-22 04:52:55.309994+00:002022-07-22T04:52:55.309994497Z
2022-07-22 04:52:55.334158+00:00�[38;5;6mpostgresql �[38;5;5m04:52:55.33 �[0m�[38;5;2mINFO �[0m ==> ** Starting PostgreSQL **
2022-07-22 04:52:55.351997+00:002022-07-22 04:52:55.351 GMT [1] LOG:  pgaudit extension initialized
2022-07-22 04:52:55.360080+00:002022-07-22 04:52:55.359 GMT [1] LOG:  starting PostgreSQL 14.4 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2022-07-22 04:52:55.365378+00:002022-07-22 04:52:55.360 GMT [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2022-07-22 04:52:55.365407+00:002022-07-22 04:52:55.360 GMT [1] LOG:  listening on IPv6 address "::", port 5432
2022-07-22 04:52:55.366139+00:002022-07-22 04:52:55.366 GMT [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2022-07-22 04:52:55.374184+00:002022-07-22 04:52:55.374 GMT [148] LOG:  database system was shut down at 2022-07-22 04:52:55 GMT
2022-07-22 04:52:55.383166+00:002022-07-22 04:52:55.382 GMT [1] LOG:  database system is ready to accept connections



Pod Name:nextcloud-redis-0

2022-07-22 04:52:57.366172+00:00�[38;5;6mredis �[38;5;5m04:52:57.36 �[0m
2022-07-22 04:52:57.367630+00:00�[38;5;6mredis �[38;5;5m04:52:57.36 �[0m�[1mWelcome to the Bitnami redis container�[0m
2022-07-22 04:52:57.369049+00:00�[38;5;6mredis �[38;5;5m04:52:57.36 �[0mSubscribe to project updates by watching �[1mhttps://github.com/bitnami/containers�[0m
2022-07-22 04:52:57.370405+00:00�[38;5;6mredis �[38;5;5m04:52:57.36 �[0mSubmit issues and feature requests at �[1mhttps://github.com/bitnami/containers/issues�[0m
2022-07-22 04:52:57.371679+00:00�[38;5;6mredis �[38;5;5m04:52:57.37 �[0m
2022-07-22 04:52:57.373045+00:00�[38;5;6mredis �[38;5;5m04:52:57.37 �[0m�[38;5;2mINFO �[0m ==> ** Starting Redis setup **
2022-07-22 04:52:57.387660+00:00�[38;5;6mredis �[38;5;5m04:52:57.38 �[0m�[38;5;3mWARN �[0m ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
2022-07-22 04:52:57.394375+00:00�[38;5;6mredis �[38;5;5m04:52:57.39 �[0m�[38;5;2mINFO �[0m ==> Initializing Redis
2022-07-22 04:52:57.409851+00:00�[38;5;6mredis �[38;5;5m04:52:57.40 �[0m�[38;5;2mINFO �[0m ==> Setting Redis config file
2022-07-22 04:52:57.448975+00:00�[38;5;6mredis �[38;5;5m04:52:57.44 �[0m�[38;5;2mINFO �[0m ==> Configuring replication mode
2022-07-22 04:52:57.478185+00:00�[38;5;6mredis �[38;5;5m04:52:57.47 �[0m�[38;5;2mINFO �[0m ==> ** Redis setup finished! **
2022-07-22 04:52:57.478583+00:002022-07-22T04:52:57.478583870Z
2022-07-22 04:52:57.493987+00:00�[38;5;6mredis �[38;5;5m04:52:57.49 �[0m�[38;5;2mINFO �[0m ==> ** Starting Redis **
2022-07-22 04:52:57.519084+00:001:C 22 Jul 2022 04:52:57.518 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2022-07-22 04:52:57.519122+00:001:C 22 Jul 2022 04:52:57.518 # Redis version=7.0.2, bits=64, commit=00000000, modified=0, pid=1, just started
2022-07-22 04:52:57.519146+00:001:C 22 Jul 2022 04:52:57.518 # Configuration loaded
2022-07-22 04:52:57.519505+00:001:M 22 Jul 2022 04:52:57.519 * monotonic clock: POSIX clock_gettime
2022-07-22 04:52:57.520520+00:001:M 22 Jul 2022 04:52:57.520 * Running mode=standalone, port=6379.
2022-07-22 04:52:57.520549+00:001:M 22 Jul 2022 04:52:57.520 # Server initialized
2022-07-22 04:52:57.520565+00:001:M 22 Jul 2022 04:52:57.520 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2022-07-22 04:52:57.593505+00:001:M 22 Jul 2022 04:52:57.593 * Creating AOF base file appendonly.aof.1.base.rdb on server start
2022-07-22 04:52:57.594850+00:001:M 22 Jul 2022 04:52:57.594 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
2022-07-22 04:52:57.594879+00:001:M 22 Jul 2022 04:52:57.594 * Ready to accept connections



Pod Name:nextcloud-789d9d989d-wnjx5 Container Name:hpb

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"hpb\" in pod \"nextcloud-789d9d989d-wnjx5\" is waiting to start: PodInitializing","reason":"BadRequest","code":400}


Pod Name:nextcloud-789d9d989d-wnjx5Container Name:imaginary

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"imaginary\" in pod \"nextcloud-789d9d989d-wnjx5\" is waiting to start: PodInitializing","reason":"BadRequest","code":400}


Pod Name:nextcloud-789d9d989d-wnjx5Container Name:nextcloud

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"nextcloud\" in pod \"nextcloud-789d9d989d-wnjx5\" is waiting to start: PodInitializing","reason":"BadRequest","code":400}



Pod Name:nextcloud-789d9d989d-wnjx5Container Name:nginx

{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"nginx\" in pod \"nextcloud-789d9d989d-wnjx5\" is waiting to start: PodInitializing","reason":"BadRequest","code":400}

Application Configuration

Describe the bug

Update doesn´t work, it gets stuck at "Deploying 2/3"

To Reproduce

Fresh install or Update existing installation

Expected Behavior

Update/Installation works

Screenshots

Additional Context

Events:
Type Reason Age From Message

---

Normal Scheduled 17m default-scheduler Successfully assigned ix-nextcloud/nextcloud-cronjob-27641095-fr682 to ix-truenas
Normal AddedInterface 17m multus Add eth0 [172.16.13.195/16] from ix-net
Normal Pulling 15m (x4 over 17m) kubelet Pulling image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0"
Warning Failed 15m (x4 over 17m) kubelet Failed to pull image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0": rpc error: code = Unknown desc = Error response from daemon: manifest for tccr.io/truecharts/nextcloud-fpm@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0 not found: manifest unknown: manifest unknown
Warning Failed 15m (x4 over 17m) kubelet Error: ErrImagePull
Warning Failed 15m (x6 over 17m) kubelet Error: ImagePullBackOff
Normal BackOff 2m10s (x62 over 17m) kubelet Back-off pulling image "tccr.io/truecharts/nextcloud-fpm:v24.0.3@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0"

~# docker pull tccr.io/truecharts/nextcloud-fpm@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0
Error response from daemon: manifest for tccr.io/truecharts/nextcloud-fpm@sha256:5cc5f0d086e4ead89c1988f1cd4d9f6820cac63ac2647a012c46aab45fc9c0a0 not found: manifest unknown: manifest unknown

I've read and agree with the following

• I've checked all open and closed issues and my issue is not there.

[xstar97]

@UnderEu @smartynokies

from discord:
Due to migrating to another container backend, we're having some issues with images not being pull-able.

We've already fixed the issue, but it might take up-to 2 days to be fully restored for each-and-every App.

please check the discord and watch for #announcements

[Ornias1993]

We have a “do some basic research”requirement here.
Before jumping on a highhorse, follow our announcements, that are luterally peppered over all our socials and contain crucial information at times.

[Ornias1993]

@smartynokies +1’ing issues is generally not appreciated on github.

[truecharts-admin]

This issue is locked to prevent necro-posting on closed issues. Please create a new issue or contact staff on discord of the problem persists