Problem registering content server

[nealef]

I have written a proxy content store and am having trouble when registering a contentserver instance that uses it. Here's the relevant section of code:

[1]        cs, err := service.NewMyContentStore(ctx, *rootDir, *cacheDir)
[2]        if err != nil {
[3]                 log.G(ctx).WithError(err).Fatalf("Failed to create interface")
[4]        }
[5] 
[6]        sv := contentserver.New(cs)
[7]        if sv == nil {
[8]                 log.G(ctx).WithError(err).Fatalf("Failed to create content server")
[9]        }
[10]       log.G(ctx).Debug("Created content server")
[11]       fmt.Printf("sv - %q\ntype: %s\n",sv,reflect.TypeOf(sv));
[12]
[13]       sv.Register(rpc)

At line [11] the output is:

sv - &{%!q(*content.store=&{/var/lib/containerd/io.containerd.content.v1.content /var/lib/containerd-mssnap-grpc/cache 0xc0001351e0})} 
type: *contentserver.service

Based on the type of *contentserver.service it would appear sv is the correct type for invoking line [13]. However, the build objects with:

msstore/main.go:163:5: sv.Register undefined (type "github.com/containerd/containerd/api/services/content/v1".ContentServer has no field or method Register)

To get it to work I replaced line [13] with:

api.RegisterContentServer(rpc, sv)

Where api is api "github.com/containerd/containerd/api/services/content/v1" which builds cleanly. However, as the registration doesn't provide any feedback I am unsure if this is valid. When I run with this content store a pull fails with:

ERRO[0000] active check failed       error="unknown service containerd.services.content.v1.Content: not implemented"

Looking through the relevant code it shows that there is no matching service found in handleStream() in vendor/google.golang.org/grpc/server.go (I added code to iterate through the services):

srv, knownService := s.services[service]

Meanwhile the image delete operation is correctly invoking my content store functions.

[nealef]

The problem is that instantiating and registering a contentserver in my proxy is pointless. Unlike my content.Store which is proxied by content/proxy/... there is no equivalent for the contentserver. So the registration is working fine in my code but there's nothing to link it to the other side.

It would seem only .so-type plugins would work. The .so has to register two plugins so that both a content store is created and a content server is instantiated. This is because disabled_plugins = ["content"] is specified so that my content store is chosen and there are two plugins with the name content:

1. The content store in content/local
2. The service in services/content/service.go

Without the second there's nothing to create a content server even when I successfully create my content store.

There a few alternatives I can think of:

1. Avoid the disabled_plugins by not picking the 1st content plugin when registering the MetadataPlugin:

                        cs, err := ic.Get(plugin.ContentPlugin)
                        if err != nil {
                                return nil, err
                        }

Because the local store is the 1st registered this will get picked (assuming FIFO registration). Instead a default content store could be specified as you can do with a snapshotter:

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"

2. Rename the service plugin to "content-xxxx" so it doesn't get disabled along with the content store plugin. There are probably other repercussions that may occur.
3. Add a proxy for the content server.

Using the .so mechanism appears to work. However, this type of plugin appears fragile with messages such as:

plugin.Open("xxxxxx"): plugin was built with a different version of package internal/goarch

I assume the plugin needs to be built with the exact source code used to build containerd as there's a hash check against what was linked versus what is being run.

[dmcgowan]

What is the exact configuration value provided for disabled_plugins = ["content"]? The full URI is required and shouldn't pass validation without it. I find it odd that content server or grpc service is not loaded. Can you also show the plugin startup logs.

[nealef]

Sorry, I meant to put my long reply here rather than a separate post...

[nealef]

I just noticed the containerd instance I am using is defaulting to version 1 which uses an ID filter. If I change to version 2 which does use a URI filter, what does the format of the URI look like for plugins?

[dmcgowan]

The full URI is just "io.containerd.content.v1.content"

[nealef]

Thanks so much. That worked perfectly. Now my plugin is being used by the content server.

[nealef]

If I use disabled_plugins = ["io.containerd.content.v1.content"] it doesn't disable anything. if I use "content" it disables all plugins with that name. Here are 3 scenarios:

1. Running Vanilla containerd

Nothing disabled. The log shows these content-related plugins being loaded:

INFO[2023-06-01T22:31:34.025788667-04:00] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2023-06-01T22:31:34.030197991-04:00] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2023-06-01T22:31:34.030839493-04:00] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1

I added a print in server.go to see which services were being registered and see:

service: *contentserver.service

ctr plugins ls shows:

io.containerd.content.v1              content                  -              ok
io.containerd.service.v1              content-service          -              ok
io.containerd.grpc.v1                 content                  -              ok

2. Running containerd with "content" disabled with my .so plugin

The log shows:

INFO[2023-06-01T22:38:10.059795532-04:00] loading plugin "io.containerd.content.v1.mystore"...  type=io.containerd.content.v1
INFO[2023-06-01T22:38:10.064114160-04:00] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2023-06-01T22:38:10.067156243-04:00] loading plugin "io.containerd.grpc.v1.msstore-content"...  type=io.containerd.grpc.v1

My printf in server.go shows registration is happening:

service: *contentserver.service

ctr plugins ls shows:

io.containerd.content.v1              mystore                  linux/amd64    ok
io.containerd.service.v1              content-service          -              ok
io.containerd.grpc.v1                 mystore-content          -              ok

3. Running containerd with "io.containerd.content.v1' disabled with my .so plugin only registering content store

Log:

INFO[2023-06-01T22:52:11.126649165-04:00] loading plugin "io.containerd.content.v1.mystore"...  type=io.containerd.content.v1
INFO[2023-06-01T22:52:11.126954260-04:00] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2023-06-01T22:52:11.130980118-04:00] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2023-06-01T22:52:11.131705806-04:00] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1

Because my plugin is loaded first it gets to be the system-wide content store.

The content server was registerd:

service: *contentserver.service

ctr plugins ls reports:

io.containerd.content.v1              mystore                  linux/amd64    ok
io.containerd.content.v1              content                  -              ok
io.containerd.service.v1              content-service          -              ok
io.containerd.grpc.v1                 content                  -              ok

Is it always the 1st content that gets chosen? In which case there'd be no need to disable anything but can it be guaranteed?

My preference is to use a proxy content store (given the temperamental nature of the .so plugin)but that gets registered after the local store so never gets picked (or if it's non-deterministic it gets picked randomly). However, that requires selective disabling on the io.containerd.content.v1.content plugin which is desirable for other reasons related to the operation of my content store.