Bug: docker doesn't start error writing manifest to content store

[coutouly]

Describe the bug

the command line to run docker should start the docker. it doesnt start.
since few days after kernel update .
I try different version: latest , nightly, 1.0 none works
my server is:
CPU: Intel Atom N2800 (4) @ 1.862GHz
GPU: Intel Atom Processor D2xxx/N2xxx
I try different version of linux: arch , ubuntu but it fails
Linux ns3089902 6.5.0-21-generic #21-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 7 14:17:40 UTC 2024 x86_64 x86_64 x86_64

Steps to reproduce

just the command line from the doc of surrealdb like
docker run --rm --pull always -p 8000:8000 surrealdb/surrealdb:latest start

Expected behaviour

docker run

SurrealDB version

nightly, latest, v1.0

Contact Details

lucile.coutouly@gmail.com

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

[rushmorem]

Do you get any error messages @coutouly? What does it say when you try to run it?

[coutouly]

with the command line no output

docker run --rm --pull always -p 8000:8000 surrealdb/surrealdb:latest start --log
latest: Pulling from surrealdb/surrealdb
Digest: sha256:94cb1fcd10e2e6314c1a08414b3ab146450343b6962a79f487572709fa135dd6
Status: Image is up to date for surrealdb/surrealdb:latest

in the docker logs:

Mar 01 13:11:48 ns3089902 dockerd[8007]: time="2024-03-01T13:11:48.407709381Z" level=debug msg="Calling HEAD /_ping"
Mar 01 13:11:48 ns3089902 dockerd[8007]: time="2024-03-01T13:11:48.409982926Z" level=debug msg="Calling POST /v1.43/images/create?fromImage=surrealdb%2Fsurrealdb&tag=latest"
Mar 01 13:11:48 ns3089902 dockerd[8007]: time="2024-03-01T13:11:48.473819687Z" level=debug msg="Trying to pull surrealdb/surrealdb from https://registry-1.docker.io"
Mar 01 13:11:49 ns3089902 dockerd[8007]: time="2024-03-01T13:11:49.802431429Z" level=debug msg="Pulling ref from V2 registry: surrealdb/surrealdb:latest"
Mar 01 13:11:49 ns3089902 dockerd[8007]: time="2024-03-01T13:11:49.802923159Z" level=debug msg="docker.io/surrealdb/surrealdb:latest resolved to a manifestList object with 4 entries; looking for a unknown match"
Mar 01 13:11:49 ns3089902 dockerd[8007]: time="2024-03-01T13:11:49.803160417Z" level=debug msg="found match for linux/amd64 with media type application/vnd.oci.image.manifest.v1+json, digest sha256:404785fe54e59f0c05d7722b62d30fd7c369f1d100ac1420dcf4acf39d19f0cd"
Mar 01 13:11:49 ns3089902 dockerd[8007]: time="2024-03-01T13:11:49.935091623Z" level=debug msg="Adding content digest to lease" digest="sha256:94cb1fcd10e2e6314c1a08414b3ab146450343b6962a79f487572709fa135dd6" lease="moby-image-sha256:83827aa7aa2d190337b267cea5325505d02a795b2df65e755c01caadf6872e78" remote="docker.io/surrealdb/surrealdb:latest"
Mar 01 13:11:49 ns3089902 dockerd[8007]: time="2024-03-01T13:11:49.988825255Z" level=debug msg="Adding content digest to lease" digest="sha256:404785fe54e59f0c05d7722b62d30fd7c369f1d100ac1420dcf4acf39d19f0cd" lease="moby-image-sha256:83827aa7aa2d190337b267cea5325505d02a795b2df65e755c01caadf6872e78" remote="docker.io/surrealdb/surrealdb:latest"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.096866368Z" level=debug msg="Calling POST /v1.43/containers/create"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.101055091Z" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":[\"start\"],\"Domainname\":\"\",\"Entrypoint\":null,\"Env\":null,\"ExposedPorts\":{\"8000/tcp\":{}},\"HostConfig\":{\"AutoRemove\":true,\"Binds\":null,\"BlkioDeviceReadBps\":[],\"BlkioDeviceReadIOps\":[],\"BlkioDeviceWriteBps\":[],\"BlkioDeviceWriteIOps\":[],\"BlkioWeight\":0,\"BlkioWeightDevice\":[],\"CapAdd\":null,\"CapDrop\":null,\"Cgroup\":\"\",\"CgroupParent\":\"\",\"CgroupnsMode\":\"\",\"ConsoleSize\":[29,114],\"ContainerIDFile\":\"\",\"CpuCount\":0,\"CpuPercent\":0,\"CpuPeriod\":0,\"CpuQuota\":0,\"CpuRealtimePeriod\":0,\"CpuRealtimeRuntime\":0,\"CpuShares\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"DeviceCgroupRules\":null,\"DeviceRequests\":null,\"Devices\":[],\"Dns\":[],\"DnsOptions\":[],\"DnsSearch\":[],\"ExtraHosts\":null,\"GroupAdd\":null,\"IOMaximumBandwidth\":0,\"IOMaximumIOps\":0,\"IpcMode\":\"\",\"Isolation\":\"\",\"Links\":null,\"LogConfig\":{\"Config\":{},\"Type\":\"\"},\"MaskedPaths\":null,\"Memory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwappiness\":-1,\"NanoCpus\":0,\"NetworkMode\":\"default\",\"OomKillDisable\":false,\"OomScoreAdj\":0,\"PidMode\":\"\",\"PidsLimit\":0,\"PortBindings\":{\"8000/tcp\":[{\"HostIp\":\"\",\"HostPort\":\"8000\"}]},\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyPaths\":null,\"ReadonlyRootfs\":false,\"RestartPolicy\":{\"MaximumRetryCount\":0,\"Name\":\"no\"},\"SecurityOpt\":null,\"ShmSize\":0,\"UTSMode\":\"\",\"Ulimits\":null,\"UsernsMode\":\"\",\"VolumeDriver\":\"\",\"VolumesFrom\":null},\"Hostname\":\"\",\"Image\":\"surrealdb/surrealdb:latest\",\"Labels\":{},\"NetworkingConfig\":{\"EndpointsConfig\":{}},\"OnBuild\":null,\"OpenStdin\":false,\"StdinOnce\":false,\"Tty\":false,\"User\":\"\",\"Volumes\":{},\"WorkingDir\":\"\"}"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.304096164Z" level=debug msg="container mounted via layerStore: /var/lib/docker/overlay2/fe6d5446bb3d2a407c80165f9b8b5fd0b93f28ff0d8c220308a359eb2c3142be/merged" container=c3ee56600749885c7ab23ca47ecdeaeaf974b0364c813ec34b0b8b47e6bbc092
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.678819723Z" level=debug msg="Calling POST /v1.43/containers/c3ee56600749885c7ab23ca47ecdeaeaf974b0364c813ec34b0b8b47e6bbc092/attach?stderr=1&stdout=1&stream=1"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.682260776Z" level=debug msg="attach: stdout: begin"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.682550849Z" level=debug msg="attach: stderr: begin"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.685701648Z" level=debug msg="Calling POST /v1.43/containers/c3ee56600749885c7ab23ca47ecdeaeaf974b0364c813ec34b0b8b47e6bbc092/wait?condition=removed"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.690789720Z" level=debug msg="Calling POST /v1.43/containers/c3ee56600749885c7ab23ca47ecdeaeaf974b0364c813ec34b0b8b47e6bbc092/start"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.695003043Z" level=debug msg="container mounted via layerStore: /var/lib/docker/overlay2/fe6d5446bb3d2a407c80165f9b8b5fd0b93f28ff0d8c220308a359eb2c3142be/merged" container=c3ee56600749885c7ab23ca47ecdeaeaf974b0364c813ec34b0b8b47e6bbc092
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.699187484Z" level=debug msg="Assigning addresses for endpoint mystifying_wiles's interface on network bridge"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.699407681Z" level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[])"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.699687847Z" level=debug msg="Request address PoolID:172.17.0.0/16 Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0 Serial:false PrefAddress:invalid IP "
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.797052678Z" level=debug msg="Assigning addresses for endpoint mystifying_wiles's interface on network bridge"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.905266491Z" level=debug msg="Programming external connectivity on endpoint mystifying_wiles (3d51f1c1e3954f99cb98e7779c050780dc5f4c70689058e37b8db53bc99a57d4)"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.905550369Z" level=debug msg="/usr/sbin/iptables, [--wait -t nat -C DOCKER -p tcp -d 0/0 --dport 8000 -j DNAT --to-destination 172.17.0.2:8000 ! -i docker0]"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.913127574Z" level=debug msg="/usr/sbin/iptables, [--wait -t nat -A DOCKER -p tcp -d 0/0 --dport 8000 -j DNAT --to-destination 172.17.0.2:8000 ! -i docker0]"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.919956756Z" level=debug msg="/usr/sbin/iptables, [--wait -t filter -C DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 8000 -j ACCEPT]"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.926742670Z" level=debug msg="/usr/sbin/iptables, [--wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 8000 -j ACCEPT]"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.933167147Z" level=debug msg="/usr/sbin/iptables, [--wait -t nat -C POSTROUTING -p tcp -s 172.17.0.2 -d 172.17.0.2 --dport 8000 -j MASQUERADE]"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.941363809Z" level=debug msg="/usr/sbin/iptables, [--wait -t nat -A POSTROUTING -p tcp -s 172.17.0.2 -d 172.17.0.2 --dport 8000 -j MASQUERADE]"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.969779373Z" level=debug msg="DeleteConntrackEntries purged ipv4:0, ipv6:0"
Mar 01 13:11:50 ns3089902 dockerd[8007]: time="2024-03-01T13:11:50.970527547Z" level=debug msg="DeleteConntrackEntriesByPort for udp ports purged ipv4:0, ipv6:0"
Mar 01 13:11:51 ns3089902 dockerd[8007]: time="2024-03-01T13:11:51.013644735Z" level=debug msg="EnableService c3ee56600749885c7ab23ca47ecdeaeaf974b0364c813ec34b0b8b47e6bbc092 START"
Mar 01 13:11:51 ns3089902 dockerd[8007]: time="2024-03-01T13:11:51.013742643Z" level=debug msg="EnableService c3ee56600749885c7ab23ca47ecdeaeaf974b0364c813ec34b0b8b47e6bbc092 DONE"
Mar 01 13:11:51 ns3089902 dockerd[8007]: time="2024-03-01T13:11:51.050160315Z" level=debug msg="createSpec: cgroupsPath: system.slice:docker:c3ee56600749885c7ab23ca47ecdeaeaf974b0364c813ec34b0b8b47e6bbc092"
Mar 01 13:11:51 ns3089902 dockerd[8007]: time="2024-03-01T13:11:51.054228958Z" level=debug msg="bundle dir created" bundle=/var/run/docker/containerd/c3ee56600749885c7ab23ca47ecdeaeaf974b0364c813ec34b0b8b47e6bbc092 module=libcontainerd namespace=moby root=/var/lib/docker/overlay2/fe6d5446bb3d2a407c80165f9b8b5fd0b93f28ff0d8c220308a359eb2c3142be/merged
Mar 01 13:11:51 ns3089902 dockerd[8007]: time="2024-03-01T13:11:51.721964458Z" level=debug msg="sandbox set key processing took 90.27738ms for container c3ee56600749885c7ab23ca47ecdeaeaf974b0364c813ec34b0b8b47e6bbc092"
Mar 01 13:11:51 ns3089902 dockerd[8007]: time="2024-03-01T13:11:51.840814454Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/create
Mar 01 13:11:51 ns3089902 dockerd[8007]: time="2024-03-01T13:11:51.874105627Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/start
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.567114851Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/exit
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.573583345Z" level=debug msg="attach: stdout: end"
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.573583570Z" level=debug msg="attach: stderr: end"
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.573817176Z" level=debug msg="attach done"
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.624078477Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/delete
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.624227323Z" level=info msg="ignoring event" container=c3ee56600749885c7ab23ca47ecdeaeaf974b0364c813ec34b0b8b47e6bbc092 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.727892374Z" level=debug msg="Revoking external connectivity on endpoint mystifying_wiles (3d51f1c1e3954f99cb98e7779c050780dc5f4c70689058e37b8db53bc99a57d4)"
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.731558205Z" level=debug msg="/usr/sbin/iptables, [--wait -t nat -C DOCKER -p tcp -d 0/0 --dport 8000 -j DNAT --to-destination 172.17.0.2:8000 ! -i docker0]"
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.745643611Z" level=debug msg="/usr/sbin/iptables, [--wait -t nat -D DOCKER -p tcp -d 0/0 --dport 8000 -j DNAT --to-destination 172.17.0.2:8000 ! -i docker0]"
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.758717838Z" level=debug msg="/usr/sbin/iptables, [--wait -t filter -C DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 8000 -j ACCEPT]"
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.769747552Z" level=debug msg="/usr/sbin/iptables, [--wait -t filter -D DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 8000 -j ACCEPT]"
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.815893812Z" level=debug msg="/usr/sbin/iptables, [--wait -t nat -C POSTROUTING -p tcp -s 172.17.0.2 -d 172.17.0.2 --dport 8000 -j MASQUERADE]"
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.832034727Z" level=debug msg="/usr/sbin/iptables, [--wait -t nat -D POSTROUTING -p tcp -s 172.17.0.2 -d 172.17.0.2 --dport 8000 -j MASQUERADE]"
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.856928893Z" level=debug msg="DeleteConntrackEntries purged ipv4:0, ipv6:0"
Mar 01 13:11:52 ns3089902 dockerd[8007]: time="2024-03-01T13:11:52.857107953Z" level=debug msg="DeleteConntrackEntriesByPort for udp ports purged ipv4:0, ipv6:0"
Mar 01 13:11:53 ns3089902 dockerd[8007]: time="2024-03-01T13:11:53.096917524Z" level=debug msg="Releasing addresses for endpoint mystifying_wiles's interface on network bridge"
Mar 01 13:11:53 ns3089902 dockerd[8007]: time="2024-03-01T13:11:53.097044591Z" level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)"
Mar 01 13:11:53 ns3089902 dockerd[8007]: time="2024-03-01T13:11:53.097165452Z" level=debug msg="Released address Address:172.17.0.2 Sequence:Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:3"

[coutouly]

another log:

sudo journalctl -u docker.service
Mar 02 05:38:20 archlinux systemd[1]: Starting Docker Application Container Engine...
Mar 02 05:38:20 archlinux dockerd[14723]: time="2024-03-02T05:38:20.730780229Z" level=info msg="Starting up"
Mar 02 05:38:20 archlinux dockerd[14723]: time="2024-03-02T05:38:20.735677753Z" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Mar 02 05:38:21 archlinux dockerd[14723]: time="2024-03-02T05:38:21.603489957Z" level=info msg="Loading containers: start."
Mar 02 05:38:23 archlinux dockerd[14723]: time="2024-03-02T05:38:23.578320978Z" level=info msg="Loading containers: done."
Mar 02 05:38:23 archlinux dockerd[14723]: time="2024-03-02T05:38:23.974407423Z" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" storage-driver=overlay2
Mar 02 05:38:23 archlinux dockerd[14723]: time="2024-03-02T05:38:23.975791440Z" level=info msg="Docker daemon" commit=f417435e5f containerd-snapshotter=false storage-driver=overlay2 version=25.0.3
Mar 02 05:38:23 archlinux dockerd[14723]: time="2024-03-02T05:38:23.977088009Z" level=info msg="Daemon has completed initialization"
Mar 02 05:38:24 archlinux dockerd[14723]: time="2024-03-02T05:38:24.507642951Z" level=info msg="API listen on /run/docker.sock"
Mar 02 05:38:24 archlinux systemd[1]: Started Docker Application Container Engine.
Mar 03 06:29:15 archlinux dockerd[14723]: time="2024-03-03T06:29:15.725992700Z" level=warning msg="error aborting content ingest" digest="sha256:c9e454669ea19ca5909c73ab6203cf2c2f9df968fea4f2aea6f827048cc0c3eb" error="context canceled" remote="docker.io/surrealdb/surrealdb:nightly"
Mar 03 06:29:15 archlinux dockerd[14723]: time="2024-03-03T06:29:15.726539786Z" level=warning msg="Error persisting manifest" digest="sha256:c9e454669ea19ca5909c73ab6203cf2c2f9df968fea4f2aea6f827048cc0c3eb" error="error writing manifest to content store: failed to send write: EOF: unknown" remote="docker.io/surrealdb/surrealdb:nightly"
Mar 03 06:29:16 archlinux dockerd[14723]: time="2024-03-03T06:29:16.198085541Z" level=error msg="Not continuing with pull after error: context canceled"
Mar 03 06:29:38 archlinux dockerd[14723]: time="2024-03-03T06:29:37.717433212Z" level=info msg="ignoring event" container=08db21c84a2ff35029ed147b475ebdfd3ca976e38aeec8f17fa8cc7c3dc28561 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Mar 03 06:52:24 archlinux dockerd[14723]: time="2024-03-03T06:52:23.899081221Z" level=info msg="ignoring event" container=e01ec7c9cef629af840f47b51c096afe13a0d4fc74e8e0121e14fd8d11418046 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

[sgirones]

@coutouly we can't reproduce it ourselves.

Can you run the following?

$ docker run --rm --pull always -p 8000:8000 surrealdb/surrealdb:latest version
latest: Pulling from surrealdb/surrealdb
Digest: sha256:94cb1fcd10e2e6314c1a08414b3ab146450343b6962a79f487572709fa135dd6
Status: Image is up to date for surrealdb/surrealdb:latest
1.2.1 for linux on aarch64

If you don't get a similar output, then it may be related to your environment.

[coutouly]

i get this:

[arch@archlinux immich]$ docker run --rm --pull always -p 8000:8000 surrealdb/surrealdb:latest version
latest: Pulling from surrealdb/surrealdb
d41cd2d7d570: Pull complete
39e048e3444c: Pull complete
Digest: sha256:94cb1fcd10e2e6314c1a08414b3ab146450343b6962a79f487572709fa135dd6
Status: Downloaded newer image for surrealdb/surrealdb:latest

[sgirones]

in that case it seems to be a problem with your environment. I'm sorry, but there's little we can help with from here.

[coutouly]

could you please send me your dockerfile ?