specs: introduce the concept of a runtime.json

[philips]

Based on our discussion in-person yesterday it seems necessary to
separate the concept of runtime configuration from application
configuration. There are a few motivators:

• To support runtime updates of things like cgroups, rlimits, etc we
  should separate things that are inherently runtime specific from
  things that are static to the application running in the container.
• To support the goal of being able to move a bundle between hosts we
  should make it clear what parts of the spec are and are not portable
  between hosts so that upon landing on a new host the non-portable
  options may be rewritten or removed.
• In order to attach a cryptographic identity to a bundle we must not
  include details in the bundle that are host specific.

This is a WIP to generate discussion.

[wking]

You're missing backticks around runtime.json here. And I'm having trouble parsing this sentence as a whole, but I'm not sure how to rephrase it to make it read more clearly.

[philips]

Parsing this sentence is difficult because we don't have solid nouns in the spec yet, I think.

In the appc spec the thing that is contained and executed is called "the app". So, I could rephrase it this way:

"The goal is that the bundle can be moved, as a unit, to another machine and the app is executed in a similar manner on the new host if runtime.json is removed or reconfigured."

[wking]

On Tue, Aug 04, 2015 at 09:55:12PM -0700, Brandon Philips wrote:

In the appc spec the thing that is contained and executed is called "the app".

So you have:

• bundles, which get written ahead of time, distributed, and used to
  seed the runtime
• containers, which are the environment setup for the application
  (namespaces, cgroups, mounts, …)
• applications, which are the process trees launched inside the
  containers

That seems reasonable to me. Should I PR a glossary?

"The goal is that the bundle can be moved, as a unit, to another
machine and the app is executed in a similar manner on the new host
if runtime.json is removed or reconfigured."

That's easier for me to read, but “in a similar manner” is wishy
enough that I'm not sure what the sentence clarifies. But that may
just be because I don't see a clear line around “runtime-specific”
1.

[philips]

Sure, +1 on introducing the "application" as a noun in the spec.

[wking]

On Thu, Jul 30, 2015 at 12:43:27PM -0700, Brandon Philips wrote:

• To support runtime updates of things like cgroups, rlimits, etc we
  should separate things that are inherently runtime specific from
  things that are static to the application running in the
  container.
• To support the goal of being able to move a bundle between hosts
  we should make it clear what parts of the spec are and are not
  portable between hosts so that upon landing on a new host the
  non-portable options may be rewritten or removed.

“Inherently runtime specific” and “portable between hosts” both sound
like slippery slopes to me. For example, absolute-path hooks may not
be portable between hosts 1, but I doubt anyone would argue for the
hook declarations to be moved into a dynamic runtime.json. User/group
ID mappings are also likely to be imperfectly portable, and
environment variables may be runtime-specific. So I don't think
splitting the config into static/dynamic parts is going to add much
clarity or usability.

For dynamic cgroups, rlimits, etc., I've argued that we should just be
push/pulling that information from the kernel 2 or leaving
state-consumers to handle it directly 3.

For migration between hosts, I'm happy to jump through whatever hoops
CRIU requires, but it seems like any information that would need could
be copied out of the current config.json, /proc, and /sys. Is there
some part of this config/runtime split that makes CRIU easier?

• In order to attach a cryptographic identity to a bundle we must not
  include details in the bundle that are host specific.

For cryptographic identity, I don't see why we'd want to sign only
non-host-specific configs. A reasonable security model is probably to
have the agent making the host-specific adjustment do whatever they
feel is needed to verify the original bundle, and then have them vouch
for the host-adapted version that they right. In the absence of an
in-spec verification model (which I think is out of scope here [4,5]),
it's hard to address this directly. If you give more details about
the verification model you have in mind, I could try to sketch out the
re-signing in more detail.

[mrunalp]

I think namespaces are tricky because if the path isn't specified then they are portable.

[philips]

@mrunalp Any suggestions on how to resolve this?

[mrunalp]

How about breaking it into two different structs? One for new namespaces and other for joining namespaces. Validation could check that there is no overlap for same namespace type in the two or there could be documented behavior on what has higher preference.

Sent from my iPhone

On Aug 4, 2015, at 9:47 PM, Brandon Philips notifications@github.com wrote:

In runtime-config-linux.md:

+instance of the global resource. Changes to the global resource are visible to
+other processes that are members of the namespace, but are invisible to other
+processes. For more information, see the man page
+
+Namespaces are specified in the spec as an array of entries. Each entry has a
+type field with possible values described below and an optional path element.
+If a path is specified, that particular file is used to join that type of namespace.
+
+```json

• "namespaces": [

•    {
  

•        "type": "pid",
  

•        "path": "/proc/1234/ns/pid"
  

•    },
  

•    {
  

•        "type": "net",
  

  @mrunalp Any suggestions on how to resolve this?

—
Reply to this email directly or view it on GitHub.

[philips]

@mrunalp Hrm, is there a use case for the application to care what namespace it is in for pid, net, utc, ipc, or user? It seems like a runtime detail.

[LK4D4]

Btw I think that separate structs will be cleaner. Also I think it'd be pretty cool to have something portable about namespaces.

[philips]

@LK4D4 What is a use case for an application to ask for a namespace explicitly besides the mount propagation? I am not opposed, but if we add it I want a use case in mind.

[mrunalp]

@philips The primary use case is joining the namespaces of other containers. For e.g. like how the pod infra container in kubernetes serves as the base container whose net and namespaces the other containers in the pod join.

[dqminh]

i wonder if we can also use the same solution as mount points i.e. to give namespace a name and delegates the assignment of name to host path to the caller.
Or we requires the namespace path to be relative to the bundle i.e. user needs to bind-mount the namespaces.

[philips]

Does anyone else have comments on this approach? We need to establish a way of making the bundle actually portable across hosts and have a consistent way of establishing the identity of the bundle once it gets there.

[mrunalp]

I think user namespace mappings could be in the portable section.

[philips]

@mrunalp I don't think the mappings can be in the portable section because you won't know the uid/gid ranges available until you land on a host and find out the already consumed ranges. Perhaps you mean something else?

[mrunalp]

@philips What I was trying to say was that one could use the same mappings on any host and they would work assuming that the linux host has user namespaces enabled. I think you are referring to assigning a particular range to a container/cluster user and that may or may not be available on a host and hence isn't portable. One way to get around it is to just have users specify that they need a range of a particular length and then translate that to whatever is available on the host. However, that seems like a feature that belongs to higher level orchestration and it could be translated to non portable section of the OCI spec.

[crosbymichael]

Instead of two configs what do you think about making the single config portable by having the paths for binds and namespaces paths be relative to the container's dir?

[LK4D4]

@crosbymichael @philips yup, and caller can like create symlinks or bindmounts there.

[philips]

@crosbymichael We need separate files because there will always be things that are local mounts is just one of those things. Other things include cgroup constraints, userns mapping, SELinux labels, etc.

If we don't have two separate files then we can't hash the application environment variables, environment, uid, etc into the identity of the bundle.

[crosbymichael]

@philips maybe relative paths for the host specific paths then the runtime config for things that can be updated that include alot of the "settings" vs "paths"?

[philips]

@crosbymichael Another approach: in appc every mount point has a name see mountpoints then you can fulfill those mounts at runtime with a host mount. This feels like a cleaner abstraction instead of having multiple bind mount to fulfill a single container mount.

For example the config.json would say:

        "mountPoints": [
            {
                "name": "work",
                "path": "/var/lib/work",
            }
        ],

Then the runtime would resolve this to a mount on the host. The name could be resolved via a bind mount directory, as you suggest, or as an entry in a runtime.json.

[crosbymichael]

@philips that does sound like a much better idea. +1!

[vbatts]

@philips yes. nice.

[mrunalp]

+1 and we could use the same technique to resolve other similar issues.

[philips]

rebased and added the mountPoint object. cleaning up some other nits and feedback.

[wking]

Spun back into the main thread, because the 88/files comment anchor
was rebased away 1.

On Mon, Aug 10, 2015 at 10:08:32PM -0700, Brandon Philips wrote 1:

-One or more content directories may be adjacent to the configuration file. This must include at least the root filesystem…
+This must include at least the root filesystem…

@wking I removed the "one or more content directories" language
because we removed the idea of "content directories" a while back in
the spec.

Did we? I think you can still put as many directories as you like
into the bundle, and you can use those directories as mount points, or
for non-runtime tooling (e.g. an external config-selection tool, #76).
I think it's important to keep that flexiblity explicit.

[philips]

TODO: Add language about bundle filesystem should have mapped the userns mappings already pre-executed

[wking]

On Wed, Aug 12, 2015 at 10:25:29AM -0700, Brandon Philips wrote:

TODO: Add language about bundle filesystem should have mapped the
userns mappings already pre-executed

Is that “if you intend file $x to be owned by root inside your
container, it needs to be uid 0 in your bundled filesystem on the
host, regardless of which host-side user is being mapped to the
container's root”. Is that actually true? It seems like something
that userns should be handling for us.

[wking]

On Wed, Aug 12, 2015 at 11:30:25AM -0700, W. Trevor King wrote:

Wed, Aug 12, 2015 at 10:25:29AM -0700, Brandon Philips:

TODO: Add language about bundle filesystem should have mapped the
userns mappings already pre-executed

Is that “if you intend file $x to be owned by root inside your
container, it needs to be uid 0 in your bundled filesystem on the
host, regardless of which host-side user is being mapped to the
container's root”. Is that actually true? It seems like something
that userns should be handling for us.

And indeed, it is something that userns handles for us. Using
userns_child_exec (source in user_namespaces(7) 1)

$ id -u 1000
$ touch example-1
$ ls -ln example-1
-rw-r--r-- 1 1000 1000 0 Aug 24 09:22 example-1
$ ./userns_child_exec -p -m -U -M '0 1000 1' -G '0 1000 1' bash

ls -ln example-1

-rw-r--r-- 1 0 0 0 Aug 24 09:22 example-1

echo hi >example-1

echo there >example-2

ls -ln example*

-rw-r--r-- 1 0 0 3 Aug 24 09:24 example-1
-rw-r--r-- 1 0 0 6 Aug 24 09:24 example-2

exit

$ ls -ln example*
-rw-r--r-- 1 1000 1000 3 Aug 24 09:24 example-1
-rw-r--r-- 1 1000 1000 6 Aug 24 09:24 example-2

So I see no need to map the host-side filesystem uids and gids.