Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build-next: failed to prune build cache #40480

Open
tao12345666333 opened this issue Feb 8, 2020 · 2 comments
Open

build-next: failed to prune build cache #40480

tao12345666333 opened this issue Feb 8, 2020 · 2 comments
Labels
area/builder version/master

Comments

@tao12345666333
Copy link
Contributor

Description

To debug #40398 issue. I update to nightly version. (I built it myself, because the official nightly channel has stopped updating.)

I want to check if #40398 be related to too many local files, so I run docker system prune command.

Steps to reproduce the issue:
issue happens only occasionally.

Describe the results you received:

When I executed the docker system prune --volumes command, I encountered a failed to prune build cache: failed to remove sha256:4ea5f8c9bd43a7e230f8090740fee2fd9ea04ce8fe 3ca5022a10c40c8648686c: lease "sha256:4ea5f8c9bd43a7e230f8090740fee2fd9ea04ce8fe3ca5022a10c40c8648686c": not found error.
But when I ran this command again, the error disappeared.

(MoeLove) ➜  docker system prune --volumes                                                                               
WARNING! This will remove:                                                                                                 
  - all stopped containers                                                                                                 
  - all networks not used by at least one container                                                                        
  - all volumes not used by at least one container                                                                         
  - all dangling images                                                                                                    
  - all dangling build cache                                                                                               
                                                                                                                           
Are you sure you want to continue? [y/N] y                                                                                 
Error response from daemon: failed to prune build cache: failed to remove sha256:4ea5f8c9bd43a7e230f8090740fee2fd9ea04ce8fe
3ca5022a10c40c8648686c: lease "sha256:4ea5f8c9bd43a7e230f8090740fee2fd9ea04ce8fe3ca5022a10c40c8648686c": not found
(MoeLove) ➜  docker system prune --volumes         
WARNING! This will remove:
  - all stopped containers                                   
  - all networks not used by at least one container
  - all volumes not used by at least one container
  - all dangling images
  - all dangling build cache
                              
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B

Then I tried to find the sha256:4ea5f8c9bd43a7e230f8090740fee2fd9ea04ce8fe3ca5022a10c40c8648686c in the error, and I found it!

[root@localhost]~# ls /var/lib/docker/image/overlay2/layerdb/sha256/4ea5f8c9bd43a7e230f8090740fee2fd9ea04ce8fe3ca5022a10c40c8648686c
cache-id  diff  parent  size  tar-split.json.gz

[root@localhost]/var/lib/docker# cat image/overlay2/layerdb/sha256/5ae054d728fa1d7d00ef344eed8218628653c6a06c700e63a3654ffce8b3865b/parent
sha256:4ea5f8c9bd43a7e230f8090740fee2fd9ea04ce8fe3ca5022a10c40c8648686c

Describe the results you expected:

I want to execute docker system prune --volumes normally without encountering errors.

Additional information you deem important (e.g. issue happens only occasionally):

yes, issue happens only occasionally.

Output of docker version:

Client: Docker Engine - Community                                                                                          
 Version:           0.0.0-20200208012753-cc3c0cce63                                                                        
 API version:       1.41                                                                                                   
 Go version:        go1.12.16                                                                                              
 Git commit:        cc3c0cce63                                                                                             
 Built:             Sat Feb  8 05:05:39 2020                                                                               
 OS/Arch:           linux/amd64                                                                                            
 Experimental:      true                                                                                                   
                                                                                                                           
Server: Docker Engine - Community                                                                                          
 Engine:                                                     
  Version:          0.0.0-20200208012753-cc3c0cce63                                                                        
  API version:      1.41 (minimum version 1.12)                                                                            
  Go version:       go1.12.16                                                                                              
  Git commit:       cc3c0cce63                               
  Built:            Sat Feb  8 05:02:56 2020                                                                               
  OS/Arch:          linux/amd64                                                                                            
  Experimental:     true
 containerd:                                                                                                               
  Version:          1.2.12                                                                                                 
  GitCommit:        35bd7a5f69c13e1563af8a93431411cd9ecf5021
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd  
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:                                                                                                                    
 Debug Mode: false                                                                                                         
 Plugins:                                                                                                                  
  app: Docker App (Docker Inc., v0.9.0-zeta1-96-g44932b629b)                                                               
  buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker)                                                              
  year: A docker plugin, Happy New Year! (Jintao Zhang, v1.0.0)    

Server:                                                                                                                    
 Containers: 1                                                                                                             
  Running: 1                                                                                                               
  Paused: 0                                                                                                                
  Stopped: 0                                                                                                               
 Images: 183
 Server Version: 0.0.0-20200208012753-cc3c0cce63
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 35bd7a5f69c13e1563af8a93431411cd9ecf5021
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd 
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.10-100.fc30.x86_64
 Operating System: Fedora 30 (Workstation Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.53GiB
 Name: localhost
 ID: XXXXX
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 33
  Goroutines: 54
  System Time: 2020-02-08T14:37:28.561671697+08:00
  EventsListeners: 0
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.1:5000
  127.0.0.0/8
 Live Restore Enabled: true

Additional environment details (AWS, VirtualBox, physical, etc.):
my laptop.
@tao12345666333
Copy link
Contributor Author

cc @AkihiroSuda @tonistiigi @tiborvass @thaJeztah

@tao12345666333
Copy link
Contributor Author

docker daemon logs here.

2月 08 17:13:23 localhost dockerd[5957]: time="2020-02-08T17:13:23.112655737+08:00" level=debug msg="Calling DELETE /v1.41/images/3a612a2371e5?force=1"                                                                                               
2月 08 17:13:23 localhost dockerd[5957]: time="2020-02-08T17:13:23.187919123+08:00" level=debug msg="Calling DELETE /v1.41/images/3a612a2371e5?force=1"                                                                                               
2月 08 17:13:23 localhost dockerd[5957]: time="2020-02-08T17:13:23.188755053+08:00" level=debug msg="Calling DELETE /v1.41/images/3d7ac13b921a?force=1"                                                                                               
2月 08 17:13:32 localhost dockerd[5957]: time="2020-02-08T17:13:32.704987548+08:00" level=debug msg="Calling HEAD /_ping"
2月 08 17:13:32 localhost dockerd[5957]: time="2020-02-08T17:13:32.705371934+08:00" level=debug msg="Calling GET /v1.41/images/json"    
2月 08 17:13:32 localhost dockerd[5957]: time="2020-02-08T17:13:32.746162504+08:00" level=debug msg="Calling HEAD /_ping"  
2月 08 17:13:35 localhost dockerd[5957]: time="2020-02-08T17:13:35.954857017+08:00" level=debug msg="Calling HEAD /_ping"  
2月 08 17:13:35 localhost dockerd[5957]: time="2020-02-08T17:13:35.955663201+08:00" level=debug msg="Calling GET /v1.41/images/json"                                                                                                                  
2月 08 17:13:38 localhost dockerd[5957]: time="2020-02-08T17:13:38.387934433+08:00" level=debug msg="Calling HEAD /_ping"  
2月 08 17:13:38 localhost dockerd[5957]: time="2020-02-08T17:13:38.388370446+08:00" level=debug msg="Calling GET /v1.41/info"                                                                                                                         
2月 08 17:13:53 localhost dockerd[5957]: time="2020-02-08T17:13:53.339662166+08:00" level=debug msg="Calling HEAD /_ping"
2月 08 17:13:54 localhost dockerd[5957]: time="2020-02-08T17:13:54.192832583+08:00" level=debug msg="Calling POST /v1.41/containers/prune"
2月 08 17:13:53 localhost dockerd[5957]: time="2020-02-08T17:13:53.339662166+08:00" level=debug msg="Calling HEAD /_ping"
2月 08 17:13:54 localhost dockerd[5957]: time="2020-02-08T17:13:54.192832583+08:00" level=debug msg="Calling POST /v1.41/containers/prune"
2月 08 17:13:54 localhost dockerd[5957]: time="2020-02-08T17:13:54.194215007+08:00" level=debug msg="Calling POST /v1.41/networks/prune"
2月 08 17:13:54 localhost dockerd[5957]: time="2020-02-08T17:13:54.195660679+08:00" level=debug msg="Calling POST /v1.41/volumes/prune"
2月 08 17:13:54 localhost dockerd[5957]: time="2020-02-08T17:13:54.195804363+08:00" level=debug msg=VolumeStore.Find ByType=service.andCombinator ByValue="[[local] false 0x55cd05df4740 0x55cd05df5d40]"
2月 08 17:13:54 localhost dockerd[5957]: time="2020-02-08T17:13:54.197039468+08:00" level=debug msg="Calling POST /v1.41/images/prune?filters=%7B%22dangling%22%3A%7B%22true%22%3Atrue%7D%7D"
2月 08 17:13:54 localhost dockerd[5957]: time="2020-02-08T17:13:54.198088507+08:00" level=debug msg="Calling POST /v1.41/build/prune?filters=&keep-storage=0"
2月 08 17:14:29 localhost dockerd[5957]: time="2020-02-08T17:14:29.609188056+08:00" level=debug msg="Calling HEAD /_ping"
2月 08 17:14:31 localhost dockerd[5957]: time="2020-02-08T17:14:31.255729569+08:00" level=debug msg="Calling POST /v1.41/containers/prune"
2月 08 17:14:31 localhost dockerd[5957]: time="2020-02-08T17:14:31.256984824+08:00" level=debug msg="Calling POST /v1.41/networks/prune"
2月 08 17:14:31 localhost dockerd[5957]: time="2020-02-08T17:14:31.258372886+08:00" level=debug msg="Calling POST /v1.41/volumes/prune"
2月 08 17:14:31 localhost dockerd[5957]: time="2020-02-08T17:14:31.258589622+08:00" level=debug msg=VolumeStore.Find ByType=service.andCombinator ByValue="[[local] false 0x55cd05df4740 0x55cd05df5d40]"
2月 08 17:14:31 localhost dockerd[5957]: time="2020-02-08T17:14:31.260123067+08:00" level=debug msg="Calling POST /v1.41/images/prune?filters=%7B%22dangling%22%3A%7B%22true%22%3Atrue%7D%7D"
2月 08 17:14:31 localhost dockerd[5957]: time="2020-02-08T17:14:31.261325656+08:00" level=debug msg="Calling POST /v1.41/build/prune?filters=&keep-storage=0"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/builder version/master
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@thaJeztah @tao12345666333