Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Referrer detection causes crash in v0.8.0 #458

Closed
PKizzle opened this issue Apr 25, 2023 · 3 comments · Fixed by #459
Closed

Referrer detection causes crash in v0.8.0 #458

PKizzle opened this issue Apr 25, 2023 · 3 comments · Fixed by #459

Comments

@PKizzle
Copy link
Contributor

PKizzle commented Apr 25, 2023 •
edited
Loading

With the latest version v0.8.0 whenever I try to pull an image that has been converted with nydusify I receive the following error:

time="2023-04-25T19:27:05.408619417+02:00" level=info msg="Start nydus-snapshotter. PID 1229759 Version  FsDriver fusedev DaemonMode multiple"
time="2023-04-25T19:27:05.424528086+02:00" level=info msg="Run daemons monitor..."
time="2023-04-25T19:27:05.427377273+02:00" level=info msg="setup image proxy keychain" target-image-service=/run/k3s/containerd/containerd.sock
time="2023-04-25T19:27:05.430258386+02:00" level=info msg="connected to backend CRI service"
time="2023-04-25T19:27:05.430551977+02:00" level=info msg="connected to backend CRI service"
panic: interface conversion: docker.dockerFetcher is not remotes.ReferrersFetcher: missing method FetchReferrers
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x64
golang.org/x/sync/singleflight.newPanicError({0x1341cc0?, 0x40005be5a0})
        /go/pkg/mod/golang.org/x/sync@v0.1.0/singleflight/singleflight.go:35 +0x28
golang.org/x/sync/singleflight.(*Group).doCall.func2.1()
        /go/pkg/mod/golang.org/x/sync@v0.1.0/singleflight/singleflight.go:184 +0x40
panic({0x1341cc0, 0x40005be5a0})
        /usr/local/go/src/runtime/panic.go:884 +0x1f4
github.com/containerd/nydus-snapshotter/pkg/referrer.(*referrer).checkReferrer.func1()
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/pkg/referrer/referrer.go:49 +0x94
github.com/containerd/nydus-snapshotter/pkg/referrer.(*referrer).checkReferrer(0x40005b68d0, {0x18090c8?, 0x40003c1410?}, {0x4000235420, 0x20}, {0x40003bfc70?, 0x40005b6888?})
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/pkg/referrer/referrer.go:95 +0x78
github.com/containerd/nydus-snapshotter/pkg/referrer.(*Manager).CheckReferrer.func1()
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/pkg/referrer/manager.go:55 +0x19c
golang.org/x/sync/singleflight.(*Group).doCall.func2(0x40005b69d6, 0x4000596aa0, 0x1?)
        /go/pkg/mod/golang.org/x/sync@v0.1.0/singleflight/singleflight.go:189 +0x60
golang.org/x/sync/singleflight.(*Group).doCall(0x131c740?, 0x40005be060?, {0x40003bfc70?, 0x47?}, 0x8?)
        /go/pkg/mod/golang.org/x/sync@v0.1.0/singleflight/singleflight.go:191 +0x84
golang.org/x/sync/singleflight.(*Group).Do(0x40000c5d50, {0x40003bfc70, 0x47}, 0x6?)
        /go/pkg/mod/golang.org/x/sync@v0.1.0/singleflight/singleflight.go:104 +0x1b8
github.com/containerd/nydus-snapshotter/pkg/referrer.(*Manager).CheckReferrer(0x40003bfc70?, {0x18090c8?, 0x40003c1410?}, {0x4000235420, 0x20}, {0x40003bfc70?, 0x3?})
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/pkg/referrer/manager.go:40 +0x80
github.com/containerd/nydus-snapshotter/pkg/filesystem.(*Filesystem).CheckReferrer(0x40000c27d0, {0x18090c8, 0x40003c1410}, 0x10e6960?)
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/pkg/filesystem/referer_adaptor.go:37 +0xc4
github.com/containerd/nydus-snapshotter/snapshot.(*snapshotter).findReferrerLayer.func1({0x0?, 0x0?}, {0x3, {0x4000592c60, 0x53}, {0x4000592d20, 0x53}, 0x40005be030, {0x1fbe89d9, 0xedbc5271d, ...}, ...})
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/snapshot/snapshot.go:385 +0x60
github.com/containerd/nydus-snapshotter/pkg/snapshot.IterateParentSnapshots({0x18090c8?, 0x40003c1410?}, 0x40005b7038?, {0x40003bf4f0, 0x4c}, 0x40005b6f58)
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/pkg/snapshot/storage.go:82 +0x250
github.com/containerd/nydus-snapshotter/snapshot.(*snapshotter).findReferrerLayer(0x0?, {0x18090c8?, 0x40003c1410?}, {0x40003bf4f0?, 0x1196d54?})
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/snapshot/snapshot.go:384 +0x7c
github.com/containerd/nydus-snapshotter/snapshot.chooseProcessor({0x18090c8?, 0x40003c1410}, 0x400028d500, 0x40000b9f80, {0x2, {0x40005486a0, 0x4}, {0x4000598740, 0x3, 0x4}}, ...)
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/snapshot/process.go:96 +0x4c0
github.com/containerd/nydus-snapshotter/snapshot.(*snapshotter).Prepare(0x40000b9f80, {0x18090c8, 0x40003c1410}, {0x40003bf4f0, 0x4c}, {0x4000592720, 0x53}, {0x0, 0x0, 0x0})
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/snapshot/snapshot.go:366 +0x338
github.com/containerd/containerd/contrib/snapshotservice.service.Prepare({{0x1810080?, 0x40000b9f80?}, {}}, {0x18090c8?, 0x40003c1410?}, 0x143cec0?)
        /go/pkg/mod/github.com/containerd/containerd@v1.7.0/contrib/snapshotservice/service.go:48 +0x140
github.com/containerd/containerd/api/services/snapshots/v1._Snapshots_Prepare_Handler({0x1494d20?, 0x40000aa2d0}, {0x18090c8, 0x40003c1410}, 0x4000171f80, 0x0)
        /go/pkg/mod/github.com/containerd/containerd@v1.7.0/api/services/snapshots/v1/snapshots_grpc.pb.go:229 +0x164
google.golang.org/grpc.(*Server).processUnaryRPC(0x40005241e0, {0x180f718, 0x40000c6d00}, 0x40003a5560, 0x4000539110, 0x25112c0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:1336 +0xb38
google.golang.org/grpc.(*Server).handleStream(0x40005241e0, {0x180f718, 0x40000c6d00}, 0x40003a5560, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:1704 +0x80c
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:965 +0x84
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:963 +0x278
goroutine 550 [running]:
golang.org/x/sync/singleflight.(*Group).doCall.func1()
        /go/pkg/mod/golang.org/x/sync@v0.1.0/singleflight/singleflight.go:161 +0x2cc
golang.org/x/sync/singleflight.(*Group).doCall(0x131c740?, 0x40005be060?, {0x40003bfc70?, 0x47?}, 0x8?)
        /go/pkg/mod/golang.org/x/sync@v0.1.0/singleflight/singleflight.go:196 +0xa4
golang.org/x/sync/singleflight.(*Group).Do(0x40000c5d50, {0x40003bfc70, 0x47}, 0x6?)
        /go/pkg/mod/golang.org/x/sync@v0.1.0/singleflight/singleflight.go:104 +0x1b8
github.com/containerd/nydus-snapshotter/pkg/referrer.(*Manager).CheckReferrer(0x40003bfc70?, {0x18090c8?, 0x40003c1410?}, {0x4000235420, 0x20}, {0x40003bfc70?, 0x3?})
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/pkg/referrer/manager.go:40 +0x80
github.com/containerd/nydus-snapshotter/pkg/filesystem.(*Filesystem).CheckReferrer(0x40000c27d0, {0x18090c8, 0x40003c1410}, 0x10e6960?)
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/pkg/filesystem/referer_adaptor.go:37 +0xc4
github.com/containerd/nydus-snapshotter/snapshot.(*snapshotter).findReferrerLayer.func1({0x0?, 0x0?}, {0x3, {0x4000592c60, 0x53}, {0x4000592d20, 0x53}, 0x40005be030, {0x1fbe89d9, 0xedbc5271d, ...}, ...})
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/snapshot/snapshot.go:385 +0x60
github.com/containerd/nydus-snapshotter/pkg/snapshot.IterateParentSnapshots({0x18090c8?, 0x40003c1410?}, 0x40005b7038?, {0x40003bf4f0, 0x4c}, 0x40005b6f58)
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/pkg/snapshot/storage.go:82 +0x250
github.com/containerd/nydus-snapshotter/snapshot.(*snapshotter).findReferrerLayer(0x0?, {0x18090c8?, 0x40003c1410?}, {0x40003bf4f0?, 0x1196d54?})
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/snapshot/snapshot.go:384 +0x7c
github.com/containerd/nydus-snapshotter/snapshot.chooseProcessor({0x18090c8?, 0x40003c1410}, 0x400028d500, 0x40000b9f80, {0x2, {0x40005486a0, 0x4}, {0x4000598740, 0x3, 0x4}}, ...)
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/snapshot/process.go:96 +0x4c0
github.com/containerd/nydus-snapshotter/snapshot.(*snapshotter).Prepare(0x40000b9f80, {0x18090c8, 0x40003c1410}, {0x40003bf4f0, 0x4c}, {0x4000592720, 0x53}, {0x0, 0x0, 0x0})
        /go/src/github.com/containerd/nydus-snapshotter-0.8.0/snapshot/snapshot.go:366 +0x338
github.com/containerd/containerd/contrib/snapshotservice.service.Prepare({{0x1810080?, 0x40000b9f80?}, {}}, {0x18090c8?, 0x40003c1410?}, 0x143cec0?)
        /go/pkg/mod/github.com/containerd/containerd@v1.7.0/contrib/snapshotservice/service.go:48 +0x140
github.com/containerd/containerd/api/services/snapshots/v1._Snapshots_Prepare_Handler({0x1494d20?, 0x40000aa2d0}, {0x18090c8, 0x40003c1410}, 0x4000171f80, 0x0)
        /go/pkg/mod/github.com/containerd/containerd@v1.7.0/api/services/snapshots/v1/snapshots_grpc.pb.go:229 +0x164
google.golang.org/grpc.(*Server).processUnaryRPC(0x40005241e0, {0x180f718, 0x40000c6d00}, 0x40003a5560, 0x4000539110, 0x25112c0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:1336 +0xb38
google.golang.org/grpc.(*Server).handleStream(0x40005241e0, {0x180f718, 0x40000c6d00}, 0x40003a5560, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:1704 +0x80c
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:965 +0x84
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:963 +0x278
nydus-snapshotter.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
@hsiangkao
Copy link
Contributor

@changweige

@imeoer
Copy link
Collaborator

imeoer commented Apr 26, 2023 •
edited
Loading

@PKizzle Hi, is it enable_referrer_detect = true enabled in your config.toml? It's still an experimental option, you may encounter the problem in this case, please try disabling it first, related fix: #459

imeoer added a commit to imeoer/nydus-snapshotter that referenced this issue Apr 26, 2023
@imeoer
referrer: fix panic on enable_referrer_detect enabled
829673d 
The panic occurs when `enable_referrer_detect = true` is enabled in config.toml:

```
panic: interface conversion: docker.dockerFetcher is not remotes.ReferrersFetcher:
missing method FetchReferrers runtime/debug.Stack()
```

Fixed it by replacing all `github.com/containerd/containerd/remotes` package to
`github.com/containerd/nydus-snapshotter/pkg/remote/remotes` in `remote` package.

Fix containerd#458

Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
@imeoer imeoer mentioned this issue Apr 26, 2023
Merged
@PKizzle
Copy link
Contributor Author

PKizzle commented Apr 27, 2023 •
edited
Loading

Yes, I enabled that option. Now that I have disabled it everything works as expected 👍🏻

Edit: I adjusted the title of the issue accordingly

@PKizzle PKizzle changed the title Unable to use nydus-snapshotter v0.8.0 in fusedev mode Referrer detection causes crash in v0.8.0 Apr 27, 2023
imeoer added a commit to imeoer/nydus-snapshotter that referenced this issue May 15, 2023
@imeoer
referrer: fix panic on enable_referrer_detect enabled
321fc7f 
The panic occurs when `enable_referrer_detect = true` is enabled in config.toml:

```
panic: interface conversion: docker.dockerFetcher is not remotes.ReferrersFetcher:
missing method FetchReferrers runtime/debug.Stack()
```

Fixed it by replacing all `github.com/containerd/containerd/remotes` package to
`github.com/containerd/nydus-snapshotter/pkg/remote/remotes` in `remote` package.

Fix containerd#458

Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
changweige pushed a commit to changweige/nydus-snapshotter that referenced this issue May 16, 2023
@imeoer @changweige
referrer: fix panic on enable_referrer_detect enabled
ca4603b 
The panic occurs when `enable_referrer_detect = true` is enabled in config.toml:

```
panic: interface conversion: docker.dockerFetcher is not remotes.ReferrersFetcher:
missing method FetchReferrers runtime/debug.Stack()
```

Fixed it by replacing all `github.com/containerd/containerd/remotes` package to
`github.com/containerd/nydus-snapshotter/pkg/remote/remotes` in `remote` package.

Fix containerd#458

Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
changweige pushed a commit to changweige/nydus-snapshotter that referenced this issue May 16, 2023
@imeoer @changweige
referrer: fix panic on enable_referrer_detect enabled
a613be6 
The panic occurs when `enable_referrer_detect = true` is enabled in config.toml:

```
panic: interface conversion: docker.dockerFetcher is not remotes.ReferrersFetcher:
missing method FetchReferrers runtime/debug.Stack()
```

Fixed it by replacing all `github.com/containerd/containerd/remotes` package to
`github.com/containerd/nydus-snapshotter/pkg/remote/remotes` in `remote` package.

Fix containerd#458

Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
billie60 pushed a commit to billie60/nydus-snapshotter that referenced this issue Aug 7, 2023
@imeoer @billie60
referrer: fix panic on enable_referrer_detect enabled
16aa647 
The panic occurs when `enable_referrer_detect = true` is enabled in config.toml:

```
panic: interface conversion: docker.dockerFetcher is not remotes.ReferrersFetcher:
missing method FetchReferrers runtime/debug.Stack()
```

Fixed it by replacing all `github.com/containerd/containerd/remotes` package to
`github.com/containerd/nydus-snapshotter/pkg/remote/remotes` in `remote` package.

Fix containerd#458

Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

referrer: fix panic on enable_referrer_detect enabled imeoer/nydus-snapshotter
3 participants
@imeoer @hsiangkao @PKizzle