Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sandbox API #6703

Merged
merged 22 commits into from
Apr 19, 2022
Merged

Sandbox API #6703

merged 22 commits into from
Apr 19, 2022

Conversation

mxpv
Copy link
Member

@mxpv mxpv commented Mar 20, 2022

Last rebase didn't go well and Github won't let me to reopen the old PR (nor recognizes new commits). Context and discussion here: #5396

@containerd containerd deleted a comment from k8s-ci-robot Mar 20, 2022
@mxpv mxpv mentioned this pull request Mar 20, 2022
@containerd containerd deleted a comment from theopenlab-ci bot Mar 21, 2022
@dmcgowan dmcgowan added this to the 1.7 milestone Mar 21, 2022
@dmcgowan dmcgowan added this to New in Code Review via automation Mar 21, 2022

func init() {
plugin.Register(&plugin.Registration{
Type: plugin.ServicePlugin,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is cleanup we need to or should do across the service packages today, but in the case where there is a defined interface for service, it is better to export an implementation of that interface rather than that of the generated API client.

sandbox/controller.go Outdated Show resolved Hide resolved
var _ sb.Store = (*remoteSandboxStore)(nil)

// NewRemoteSandboxStore create client for sandbox store
func NewRemoteSandboxStore(client api.StoreClient) sb.Store {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note for followup: This can be moved to a subpackage such as sandbox/proxy

@@ -155,3 +159,17 @@ func WithIntrospectionService(in introspection.Service) ServicesOpt {
s.introspectionService = in
}
}

// WithSandboxStore sets the sandbox store.
func WithSandboxStore(client sandboxsapi.StoreClient) ServicesOpt {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note for follow up: We are trying to prevent directly using these generated interfaces in the client interface. The alternative would be to explicitly get an instance of the interface using the API client and passing in that instance.

@dmcgowan
Copy link
Member

Mostly just a few questions for me. I think we can handle some of the changes in follow-ups since they are similar to ongoing refactoring around the API and protobufs. If we can agree on the API we should just get this in.

@theopenlab-ci
Copy link

theopenlab-ci bot commented Mar 29, 2022

Build succeeded.

@containerd containerd deleted a comment from theopenlab-ci bot Mar 30, 2022
@containerd containerd deleted a comment from k8s-ci-robot Mar 30, 2022
@dmcgowan
Copy link
Member

/test pull-containerd-node-e2e

Copy link
Member

@cpuguy83 cpuguy83 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there some way to test this with ctr?

I see I can create a sandbox but then what?
I see there is a running v2 shim after.

Apologies, I'm a bit confused as to what's supposed to happen for these sandboxes.

When I create a sandbox using defaults I get a runc-v2 shim with no tasks.
How do I attach a task to it?

How would cri use this to implement the pod sandbox?

// When running the traditional containerd shim, the workflow looks as follows:
// For each new task we're about to run:
// 1. Invoke `shim_binary --start` to obtain `TaskService` address (printed in stdout)
// 2. Call TaskService.RunContainer(id=1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think these RunContainer calls are a real API? Is the doc out of date with the implementation?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, removed outdated docs

Name: "run",
Aliases: []string{"create", "c", "r"},
Usage: "run a new sandbox",
ArgsUsage: "[flags] <pod-config.json> <sandbox-id>",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No pod-config.json, I think?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be any spec recognized to a shim implementation.
These includes the container spec used when launching pause container:

spec, err := c.sandboxContainerSpec(id, config, &image.ImageSpec.Config, sandbox.NetNSPath, ociRuntime.PodAnnotations)

}

for _, sandbox := range list {
_, err := fmt.Fprintf(writer, "%s\t%s\t%s\t\n", sandbox.ID, sandbox.CreatedAt, sandbox.Runtime.Name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for follow up maybe, but this output is not easy to read.

ID   CREATED                                 RUNTIME
test 2022-03-30 21:39:50.043075106 +0000 UTC io.containerd.runc.v2


message ControllerStartRequest {
string sandbox_id = 1;
repeated containerd.types.Mount rootfs = 2;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does this rootfs come from? Who determines what the rootfs is?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Client is expected to specify this.
Same way as CRI does today via WithNewSnapshot call when creating a sandbox.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For VMs (incl. Firecracker), would we assume they boot Linux from the rootfs? How much compatibility between non-VM sandboxes and VM sandboxes is expected?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been working on a prototype with "pause" containers and this field was needed to pass a mount, so the shim could launch pause containers from the runtime via start sandbox request. For Firecracker this can be an image to boot from. Here containerd's client is required to provide proper values that will be recognized by shim implementations.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been working on a prototype with "pause" containers and this field was needed to pass a mount, so the shim could launch pause containers from the runtime via start sandbox request. For Firecracker this can be an image to boot from. Here containerd's client is required to provide proper values that will be recognized by shim implementations.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does this rootfs come from? Who determines what the rootfs is?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Client is expected to specify this.
Same way as CRI does today via WithNewSnapshot call when creating a sandbox.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been working on a prototype with "pause" containers and this field was needed to pass a mount, so the shim could launch pause containers from the runtime via start sandbox request. For Firecracker this can be an image to boot from. Here containerd's client is required to provide proper values that will be recognized by shim implementations.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been working on a prototype with "pause" containers and this field was needed to pass a mount, so the shim could launch pause containers from the runtime via start sandbox request. For Firecracker this can be an image to boot from. Here containerd's client is required to provide proper values that will be recognized by shim implementations.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Client is expected to specify this.
Same way as CRI does today via WithNewSnapshot call when creating a sandbox.

@mxpv
Copy link
Member Author

mxpv commented Mar 31, 2022

I see I can create a sandbox but then what?
I see there is a running v2 shim after.
Apologies, I'm a bit confused as to what's supposed to happen for these sandboxes.
When I create a sandbox using defaults I get a runc-v2 shim with no tasks.
How do I attach a task to it?

This is daemon side API to manage sandboxes.
To test it end to end, we require shim and CRI (or invoke it from ctr) to support this new APIs.
This will be done in subsequent PRs as this one already huge enough to review.

Is there some way to test this with ctr?

Yes, there is ctr subcommands that allow you to invoke the API. See sandboxes.go

How would cri use this to implement the pod sandbox?

This requires changes on CRI side. That's ongoing work.

@theopenlab-ci
Copy link

theopenlab-ci bot commented Mar 31, 2022

Build succeeded.


import "google/protobuf/any.proto";
import "google/protobuf/timestamp.proto";
import weak "gogoproto/gogo.proto";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remove weak after merge.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remove weak after merge.


message ControllerStartRequest {
string sandbox_id = 1;
repeated containerd.types.Mount rootfs = 2;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For VMs (incl. Firecracker), would we assume they boot Linux from the rootfs? How much compatibility between non-VM sandboxes and VM sandboxes is expected?


message ControllerShutdownRequest {
string sandbox_id = 1;
uint32 timeout_secs = 2;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

@mxpv mxpv Apr 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sandbox implementations are expected to shutdown both running containers (if any) and the sandbox itself on shutdown request. So timeout would be useful to try to shutdown containers properly first and in case if that take too long - kill them. There is similar logic in CRI:

func (c *criService) stopPodSandbox(ctx context.Context, sandbox sandboxstore.Sandbox) error {

Start request should just launch a new sandbox, so there is no need to have a dedicated timeout parameter.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems a little high-level to me. Shouldn't the client drive this?

mxpv added 3 commits April 8, 2022 13:33
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
mxpv added 6 commits April 8, 2022 13:33
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
@theopenlab-ci
Copy link

theopenlab-ci bot commented Apr 8, 2022

Build succeeded.

Copy link
Member

@cpuguy83 cpuguy83 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dmcgowan dmcgowan merged commit be60973 into containerd:main Apr 19, 2022
Code Review automation moved this from Ready For Review to Done Apr 19, 2022
mythi added a commit to mythi/containerd that referenced this pull request May 12, 2022
PR containerd#6366 implemented a tree-wide change to replace github.com/pkg/errors
to errors. The new sandbox API PR containerd#6703 had few errors.Wrap*() leftovers
and pulled github.com/pkg/errors back. This commit replaces those
leftovers by following the pattern in containerd#6366.

Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
@mxpv mxpv deleted the s branch June 16, 2022 19:50
@mxpv mxpv mentioned this pull request Aug 19, 2022
17 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

6 participants