Question about your content store plugin

[nealef]

When I build a plugin containerd will complain with "plugin was built with a different version ...". I am building with the same level of golang as containerd and my go.mod has the same level of containerd that I am running but I cannot get rid of this. How did you avoid it?

[hinshun]

You should use the gRPC plugin interface, so the compatibility is happening at a protobuf API level not a go plugin level. https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#proxy-plugins

[nealef]

I had created a content store based on the snapshotter plugin. However, it’s unclear how to connect my store implementation with the RPC service. For snapshotter we:

        // Convert the snapshotter to a gRPC service,
        snsvc := snapshotservice.FromSnapshotter(rs)

        // Register the service with the gRPC server
        snapshotsapi.RegisterSnapshotsServer(rpc, snsvc)

With that in place my snapshotter works fine.

However, for the content store I am a little lost. For content there's a RegisterContentServer() but that's something tha's implemented in content/contentserver/contentserver.go that uses a content store. So I am now sure how to make the association so my Info(), Status() functions are invoked.

[hinshun]

You need to disable the built in one. See the containerd config here as a reference: https://github.com/hinshun/ipcs/blob/master/cmd/containerd/config.toml#L4

[nealef]

It's done. When I try to do a pull I get an "unimplemented" error. I was wondering if the IPCS implementation chose to use the .so plugin method because they were having the same type of trouble. With the proxy method I am unclear how to let the grpc mechanism know we can handle those content store requests.

[hinshun]

I'm not sure I understand. Why not just use these:

cssvc := contentserver.New(cs)
cssvc.Register(rpc)

IPCS should be using the proxy mechanism, its been a few years so it's unclear why it chose the .so plugin method first. Probably to facilitate easier debugging / stacktraces when it was under heavy development.

[nealef]

With my gRPC implementation I had:

sv := contentserver.New(cs)
api.RegisterContentServer(rpc, sv)

Because using:

sv.Register(cs)

Resulted in:

sv.Register undefined (type "github.com/containerd/containerd/api/services/content/v1".ContentServer has no field or method Register)

As contentserver.New(cs) returns api.ContentServer not service for:

func (s *service) Register(server *grpc.Server)

The registration provides no feedback so it's not clear if things are set. I see a connection come in:

146069 accept4(7, {sa_family=AF_UNIX}, [112 => 2], SOCK_CLOEXEC|SOCK_NONBLOCK) = 8

But no traffic flows and when I do a pull:

# ctr image pull docker.io/library/httpd:latest
ERRO[0000] active check failed                           error="unknown service containerd.services.content.v1.Content: not implemented"
ERRO[0000] active check failed                           error="unknown service containerd.services.content.v1.Content: not implemented"

Which seems to tell me I really haven't registered properly.

[nealef]

Got it - i had incorrectly named the type content.store in the function that creates a content store. As a result the registration got garbage in and I go garbage out. I am now getting the gRPC messages. I am able to handle Delete/Walk/ListStatuses. Pull is proving less co-coperative where I am still getting "Not Implemented" messages but this should be easier to track down.

Thanks for the suggestions, they helped enormously.

[nealef]

I think the registration is still (a/the) problem. The value returned by sv := contentserver.New(cs) is:

fmt.Printf("sv - %q type: %s\n",sv,reflect.TypeOf(sv));
sv - &{%!q(*content.store=&{/var/lib/containerd/io.containerd.content.v1.content /var/lib/containerd-mssnap-grpc/cache 0xc0001351e0})} type: *contentserver.service

The significant part is that the type is *content.service, therefore it should be possible to invoke:

sv.Register(rpc)

So why does it tell me:

sv.Register undefined (type "github.com/containerd/containerd/api/services/content/v1".ContentServer has no field or method Register)

[nealef]

I have created a discussion on containerd to pursue this further. containerd/containerd#8611