The purpose of sandbox vs. process vs. container config separation?

[jirutka]

I still don’t understand high-level concept of Lithos’ configuration. Why is configuration of a container separated into three configs (sandbox, process and container), specifically why is sandbox and process not a single config? What’s the idea behind this and how it’s supposed to be used? Could you please provide example of some non-trivial setup that demonstrates this?

In #8 (comment) you wrote:

Well, technically we have a concept of sandboxes just for that. I.e. you allow users to upload images to certain directory configured in a sandbox. And arrange some script to update their /etc/lithos/processes/<sandbox-name>.yaml. This allows them to freely update their images, add and remove services, but they can't escape a sandbox.

This quite makes sense, except one thing – cgroup limits are configured only in the container config (inside image). Cgroup limits are exactly the thing I’d like to enforce for users and not let them change them.

[tailhook]

specifically why is sandbox and process not a single config?

Sandbox is a static part of the config. It's usually deployed by admins and only changed by admins. Also, we don't make unique sandboxes for every project. Just a template with a unique directory and unique user/port range, and separate network bridge/vlan.

On the other hand, process config is dynamic. It says run N processes of version X. Where both N and X are supposed to be updated many times an hour. It's supposed to be generated by an orchestration system (or merely a deployment script).

Another way o think of it: in many other systems process config is the data transferred using runtime API call. Just we don't try to stick with HTTP API and think files are good API too.

And another thing we're trying to do (but not there yet) is to make it multi-tenant. I.e. think of "sandbox" written by hosting provider. "container" config written by actual user. And "process" config is created by CI/CD or orchestration system.

cgroup limits are configured only in the container config (inside image). Cgroup limits are exactly the thing I’d like to enforce for users and not let them change them.

TL;DR We haven't figured out a good way to make that limit(s) at the lithos level yet. We make it at orchestration system. And we're seeking a way to it in lithos too.

Memory Limit

Well, yes cgroup limits are underspecified for now. Let's talk about memory limit first, because this is the one I'm thinking of the most.

Let's consider we have a project called "plum". And it has two services:

1. plum-web -- a web(http) worker serving requests, with 500mb memory limit
2. plum-celery -- a queue for background tasks, with 2Gb memory limit

Now, let's talk about the following scenarios. The core idea is that all of them should work without changing sandbox:

1. We have 16 x plum-web and 2 x plum-celery at peak hours and 8 x plum-web and 4 x plum-celery at night
2. We want to scale the number of plum-web services beyond the normal value
3. Developers want to deploy 3 x "plum-ssr" with the memory limit of 1G instead of 6 x "plum-web" (because "plum-ssr" is partially replaces "plum-web")

Now (1) and (3) look like easy, just set 10Gb memory limit in a sandbox. Except for the one interesting question I've skipped before:

Are plum-web and plum-celery in the same sandbox or in different ones? What about plum-ssr?

The key point here is that interactions like "increase the number of instances of one service and decrease the number of instances of another" can happen between any services. Including services that belong to different projects.

So we're not yet sure how to set a memory limit for sandbox:

1. A cap for a single process (in this case user may overcome it using many instances)
2. A cap for a sandbox, this has three issues:
   a) doesn't scale well, because orchestration system is supposed to be able to move processes between different servers, effectively making fewer instances on this machine or more, this doesn't work well with static per-sandbox limit
   b) doesn't allow to quickly scale up the number of services without changing sandbox
   c) as in scenario (3) above, developers might need to split sandbox into two without clear idea of what good ratio is
3. A cap for a group of sandboxes (in this case we complicate config even more)

Okay, this is already too long...

CPU Limit

We only have cpu-shares configured in container. It's supposed to distinguish background and interactive tasks, that's probably it. We can add a cap for per-process cpu-shares easily. Adding limit on sum of cpu-shares for the sandbox is harder, but can be done. (please open separate issue if you wish this to be done)

There is also sched-bwc, but i haven't tried it yet. And I'm not sure how to configure it sensibly.

Other Limits

Well, I'm not sure about other ones. It looks like there aren't good way to limit disk throughtput (the cgroup one isn't effective in my practice). Any other ones?

Sandbox effectively limits other things: users, directories that can be mounted, network bridge that can be used. These things do not have shenanigans we have with memory.

Now

Currenly, we handle memory limit in an orchestration system. The update that contains invalid memory limit will be rejected. I.e. if we have a 100Gb memory limit for whole cluster, and 200 instances configured to run we reject config that has anything above 500mb as the memory limit.

You can also check it as a part of your CI/CD pipeline: i.e. don't allow to deploy anything which has limit larger than specified. This requires trusted code in CI which may be or may not be a problem for your use case.

Future

There are few other things that don't exactly match "sandbox" concept in current form:

1. We have a use-case for using multiple image-dir's
2. We may want two services to have distinct /persistent-storage directories without encoding that logic in the services itself
3. We imagine a use case where service might be in two networks simultaneoulsly
4. And a memory limit

So we're seeking a way to configure these things in a way that doesn't complicate config even more. And your input is valuable.

[tailhook]

I don't think there is anything actionable left in this issue. Feel free to open separate issues for specific proposals.