Docker push EOF

[letran3691]

I deployed Harbor v2.10.2-1a741cb7 on NAS Synology OS 7.1.1. After deploying Harbor, I can access the web GUI normally. However, when I try to push Docker images, it returns EOF.

I've read the Harbor logs and searched Google; however, the issue remains unresolved

I hope someone can help me. Thanks.

[MinerYang]

Hi @letran3691 ,

Could you provide more contexts, like proxy logs, harbor-registry logs, docker daemon logs?

[letran3691]

As mentioned above, I deployed Harbor on a NAS Synology device with the following configurations:

• Docker version 20.10.3, build 55f0773
• Docker-compose version 1.28.5, build 24fb474e
  Here are the logs from the Harbor deployment:
  logs.zip

Hi @letran3691 ,

Could you provide more contexts, like proxy logs, harbor-registry logs, docker daemon logs?

[MinerYang]

Hi @letran3691 ,

I haven't' see any request at the harbor proxy about the pushing image harbor-registry.mbageas.life/test/test:latest

• What's your storage backend? filesystem or s3?
• Is there any proxy/firewall in front of Harbor or docker service?
• Could you check the docker daemon logs while you occurring this issue?

[letran3691]

Hi @letran3691 ,

I haven't' see any request at the harbor proxy about the pushing image harbor-registry.mbageas.life/test/test:latest

• What's your storage backend? filesystem or s3?
• Is there any proxy/firewall in front of Harbor or docker service?
• Could you check the docker daemon logs while you occurring this issue?

• The storage backend is set to filesystem. I've edited the storage_service -> redirect -> disable: true
• I don't have a proxy, but I do have a VLAN. I've tested it on a virtual machine within the same VLAN however, the error persists.
• The Docker daemon logs only display EOF. Details are provided below..

"May 10 10:18:53 master01 dockerd[2526225]: time="2024-05-10T10:18:53.505245247+07:00" level=error msg="Upload failed, retrying: EOF"
May 10 10:18:53 master01 dockerd[2526225]: time="2024-05-10T10:18:53.505417124+07:00" level=error msg="Upload failed, retrying: EOF"
May 10 10:18:53 master01 dockerd[2526225]: time="2024-05-10T10:18:53.505763625+07:00" level=debug msg="Assembling tar data for 3f75b4eed68d743d9d4e5fbba18337bc81f01eeb6914045990b046f27b7c8ac8"
May 10 10:18:53 master01 dockerd[2526225]: time="2024-05-10T10:18:53.509833916+07:00" level=error msg="Upload failed, retrying: EOF"
May 10 10:18:53 master01 systemd[1]: var-lib-docker-overlay2-3f75b4eed68d743d9d4e5fbba18337bc81f01eeb6914045990b046f27b7c8ac8-merged.mount: Deactivated successfully.
May 10 10:18:53 master01 dockerd[2526225]: time="2024-05-10T10:18:53.838300586+07:00" level=debug msg="Assembling tar data for 53c15b8073b7e13ff003b0d662a094eddbe59f996aed6187bc896f1bea089efd"
May 10 10:18:53 master01 dockerd[2526225]: time="2024-05-10T10:18:53.838846713+07:00" level=error msg="Upload failed, retrying: EOF"
May 10 10:18:54 master01 dockerd[2526225]: time="2024-05-10T10:18:54.049991205+07:00" level=debug msg="Assembling tar data for kvuf2havlofb8kvvpmyek8ufc"
May 10 10:18:54 master01 dockerd[2526225]: time="2024-05-10T10:18:54.053474797+07:00" level=error msg="Upload failed, retrying: EOF"
May 10 10:18:54 master01 systemd[1]: run-containerd-runc-k8s.io-a21a810c9e7c695423b5a26095f4775bc739f6f1bae53bcaef4e3eff9fa89bce-runc.TEfViB.mount: Deactivated successfully.
May 10 10:18:58 master01 systemd[1]: run-containerd-runc-k8s.io-64e9c11c66e28b4a6cfff2cb48e075b99f6aac368f55e3a2eab88fc08b8b9e58-runc.fmBacv.mount: Deactivated successfully.
May 10 10:19:00 master01 systemd[1]: run-containerd-runc-k8s.io-a21a810c9e7c695423b5a26095f4775bc739f6f1bae53bcaef4e3eff9fa89bce-runc.QfXN5i.mount: Deactivated successfully.
May 10 10:19:04 master01 systemd[1]: run-containerd-runc-k8s.io-a21a810c9e7c695423b5a26095f4775bc739f6f1bae53bcaef4e3eff9fa89bce-runc.0xhSK7.mount: Deactivated successfully.
May 10 10:19:08 master01 systemd[1]: run-containerd-runc-k8s.io-64e9c11c66e28b4a6cfff2cb48e075b99f6aac368f55e3a2eab88fc08b8b9e58-runc.sPzWDz.mount: Deactivated successfully.
May 10 10:19:10 master01 systemd[1]: run-containerd-runc-k8s.io-a21a810c9e7c695423b5a26095f4775bc739f6f1bae53bcaef4e3eff9fa89bce-runc.STbyul.mount: Deactivated successfully.
May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.506184169+07:00" level=debug msg="Pushing layer: sha256:b80974699bdb622b5cbb7ae31868637c5dc1d2ca6a5f61dbe654e4abb2e14632"
May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.506184285+07:00" level=debug msg="Pushing layer: sha256:2e0d92876e9f41196bbbb78eabd81a24ab66d43db4d681d910fba3b4a5bd7aa7"
May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.510738640+07:00" level=debug msg="Pushing layer: sha256:d75fd41110a448d9d52d83f6e415a4167f3ab4691a7e4948101cf64159cd1e96"
May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.839253494+07:00" level=debug msg="Pushing layer: sha256:3f69a32a2b313adbf3d5076a3be902a667b04783a3cdfb7aef74b1b62b0a8df4"
May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.935367150+07:00" level=debug msg="Assembling tar data for f00a150a4aef8d77e201fd0100bdc09fe513bd86c4254aa37e018cb03e94c6c1"
May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.941133148+07:00" level=error msg="Upload failed: EOF"
May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.942887625+07:00" level=debug msg="Assembling tar data for c79454a44ddc7a1f059fa157d4806c718a1862300a7a6fc2d9c460bed9c2b3b7"
May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.944553727+07:00" level=error msg="Upload failed: EOF"
May 10 10:19:13 master01 systemd[1]: var-lib-docker-overlay2-f00a150a4aef8d77e201fd0100bdc09fe513bd86c4254aa37e018cb03e94c6c1-merged.mount: Deactivated successfully.
May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.948713310+07:00" level=debug msg="Checking for presence of layer sha256:bf6cd01dee5edc1e179aeab7de1d24dd401c2cc798cdc61323030f8da0b778ee (sha256:4d84de5fb9b2d32bc417c2e043fcc88e3303970c61cd4916956e66431a9ce7d3) in harbor-registry.mbageas.life/test/login"
May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.949205790+07:00" level=debug msg="Checking for presence of layer sha256:a8a738834a4d15fbdc977804f7ed941d23b26b3ccd0364b390acb684a42fed24 (sha256:2818b7b6a9db4d0901d74850cf1096f76fa22d8a7fad4d711710fbf6ab1b48d1) in harbor-registry.mbageas.life/test/login"
May 10 10:19:13 master01 systemd[1]: var-lib-docker-overlay2-c79454a44ddc7a1f059fa157d4806c718a1862300a7a6fc2d9c460bed9c2b3b7-merged.mount: Deactivated successfully.
May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.968572425+07:00" level=debug msg="Pushing layer: sha256:bf6cd01dee5edc1e179aeab7de1d24dd401c2cc798cdc61323030f8da0b778ee"
May 10 10:19:13 master01 dockerd[2526225]: time="2024-05-10T10:19:13.970249678+07:00" level=debug msg="Pushing layer: sha256:a8a738834a4d15fbdc977804f7ed941d23b26b3ccd0364b390acb684a42fed24"
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.053908318+07:00" level=debug msg="Pushing layer: sha256:ecbd97428477513a42d7c54fa91b03e377cb232041b4bbbb4484d56857b3a8cc"
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.147648677+07:00" level=debug msg="Assembling tar data for 3f75b4eed68d743d9d4e5fbba18337bc81f01eeb6914045990b046f27b7c8ac8"
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.148546806+07:00" level=error msg="Upload failed: EOF"
May 10 10:19:14 master01 systemd[1]: var-lib-docker-overlay2-3f75b4eed68d743d9d4e5fbba18337bc81f01eeb6914045990b046f27b7c8ac8-merged.mount: Deactivated successfully.
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.192549944+07:00" level=debug msg="Checking for presence of layer sha256:52ec5a4316fadc09a4a51f82b8d7b66ead0d71bea4f75e81e25b4094c4219061 (sha256:b0a0cf830b12453b7e15359a804215a7bcccd3788e2bcecff2a03af64bbd4df7) in harbor-registry.mbageas.life/test/login"
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.215414543+07:00" level=debug msg="Pushing layer: sha256:52ec5a4316fadc09a4a51f82b8d7b66ead0d71bea4f75e81e25b4094c4219061"
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.643971107+07:00" level=debug msg="Assembling tar data for 53c15b8073b7e13ff003b0d662a094eddbe59f996aed6187bc896f1bea089efd"
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.645925541+07:00" level=error msg="Upload failed: EOF"
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.811944419+07:00" level=error msg="Upload failed, retrying: EOF"
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.818049736+07:00" level=debug msg="Assembling tar data for 2e3ac69c2e08ae61a2441df39d27051280c3cf7b62193097124fc6104d06630c"
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.818451492+07:00" level=error msg="Upload failed, retrying: EOF"
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.829230060+07:00" level=debug msg="Assembling tar data for 244b5054ff59424ea3fe13c89f35253cb4e83589be3e5e65a5c83875ddca6ceb"
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.927658513+07:00" level=error msg="Upload failed: EOF"
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.928044306+07:00" level=debug msg="Assembling tar data for kvuf2havlofb8kvvpmyek8ufc"
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.936099625+07:00" level=info msg="Attempting next endpoint for push after error: EOF"
May 10 10:19:14 master01 dockerd[2526225]: time="2024-05-10T10:19:14.941759145+07:00" level=debug msg="Assembling tar data for dd1425f918c1c4009babe0bba986e304d40da0d71e8880e46e18d554897fcdb7"

[wy65701436]

please share the log of registry.

[letran3691]

please share the log of registry.

it's here
registry.log

[letran3691]

So, does anyone have any ideas?

[MinerYang]

Hi @letran3691 ,

Could you check the blob existence on your filesystem(registry storage)?

[letran3691]

Hi @letran3691 ,

Could you check the blob existence on your filesystem(registry storage)?

I've checked, and nothing exists.

[MinerYang]

Hi @letran3691 ,
Did the push issue only happened in the repository/artifact or generally for all projects?

• If it is only happened for some of the specific image that you customize, then my best guess is it occurs at docker client side when process tar file, please check you the image fileyou build is not corrupted. Since the docker daemon log and the registry logs you provided is not relevant to a same artifact, that I didn't see any harbor-registry.mbageas.life/test/test:latest upload request arrived at the harbor egistry log.
• If it occurs for all the images including official docker image, then please try to disable all the FW and VPN etc.. (BTW, harbor only official shipped linux amd64 build, We are not testified on NAS Synology OS yet)
• Please double check with the image test/login:<sha256:123XXX> both in filesystem and redis cache
  cat /docker/harbor/registry/docker/registry/v2/sha245/12//data
• docker exec -it /bin/bash

$ redis-cli -n 1
127.0.0.1:6379[1]> keys "blobs:*"

127.0.0.1:6379[1]> keys "repository:*"

And one more thing, the logs indicates Harbor instance is on v2.7.3, Not v2.10.2-1a741cb7, Please make sure the log is consistent harbor instance you are using

May  9 12:45:37 192.168.208.1 core[9288]: 2024-05-09T05:45:37Z [INFO] [/core/main.go:259]: Version: v2.7.3, Git commit: 252a0b7c

[letran3691]

Hi @letran3691 , Did the push issue only happened in the repository/artifact or generally for all projects?

• If it is only happened for some of the specific image that you customize, then my best guess is it occurs at docker client side when process tar file, please check you the image fileyou build is not corrupted. Since the docker daemon log and the registry logs you provided is not relevant to a same artifact, that I didn't see any harbor-registry.mbageas.life/test/test:latest upload request arrived at the harbor egistry log.
• If it occurs for all the images including official docker image, then please try to disable all the FW and VPN etc.. (BTW, harbor only official shipped linux amd64 build, We are not testified on NAS Synology OS yet)
• Please double check with the image test/login:<sha256:123XXX> both in filesystem and redis cache
  cat /docker/harbor/registry/docker/registry/v2/sha245/12//data
• docker exec -it /bin/bash

$ redis-cli -n 1
127.0.0.1:6379[1]> keys "blobs:*"

127.0.0.1:6379[1]> keys "repository:*"

And one more thing, the logs indicates Harbor instance is on v2.7.3, Not v2.10.2-1a741cb7, Please make sure the log is consistent harbor instance you are using

May  9 12:45:37 192.168.208.1 core[9288]: 2024-05-09T05:45:37Z [INFO] [/core/main.go:259]: Version: v2.7.3, Git commit: 252a0b7c

it occurs for all the images including official docker image.

this path not exist cat /docker/harbor/registry/docker/registry/v2/sha245/12/data
Only the path exists.

The logs are different because I tried deploying multiple versions, but all have the same issue.