Prevent functions from stealing cycles

[markusthoemmes]

Problem

FaaS platforms in general are heavily request driven. Resources are handled on a per-request basis and metering/billing is usually also done per-request. But what happens if a user breaks out of the request's context?

An example:

function main(args) {
  setInterval(x => crunchNumbers(), 1000);
  return "I'm already done... or am I?";
}

The "request" will be immediatly closed in the above case, but if the container is still running, the function will steal cycles by running crunchNumbers() asynchronously. This is valid for any language with support for asynchronous tasks.

This problem is especially apparent in a multi-tenant platform, because one tenant can cause heavy noisy-neighbor effects due to the "untracked" nature of her system usage then.

Possible solutions

(This is of course an incomplete list and up for discussion)

1. Alternative metering

@grantr rightfully mentioned in a Slack discussion, that the platform could choose to meter based on the runtime of the container itself and not per-request. That would indeed solve this issue but I'm not sure if that'd be accepted in the FaaS world of only "paying for what you use on a millisecond basis". The scale-to-zero timeout would then determine the overhead of what I'm always paying, which may or may not make a dent in my bill.

2. Process freezing

A unix process can be frozen and thus prevented from consuming any cpu resources. There are several alternatives on how to do this, including the cgroup freezer and CRIU. A pause/resume command is built-in into docker, containerd/runc and kata. gvisor doesn't seem to support this OOTB. pause/resume is not part of the OCI spec. Kubernetes (today) does not expose a command to pause a container via its CRI.

Using any of the above solutions (some are much more involved than others) a container could be frozen in a very short timeframe after the last request finished "officially".

One more wrinkle is, that process freezing only prevents consumption of cpu resources, but leaves the process' pages in memory. A solution could be to use a more sophisticated checkpoint tool rather than just freezing the process. @jchesterpivotal mentioned previous work around that in riff.

As an example implementation, OpenWhisk on Kubernetes goes rogue on its executor nodes and talks directly to the underlying container daemon (implemented for docker and containerd) to invoke pause/unpause respectively. As a rough estimate, both operations take 15ms to complete.

It'd be desirable to have that operation baked into the K8s CRI.

3. Make container creation/removal so fast it doesn't matter

@julz mentioned that we could also make container creation/removal so fast, that cycle stealing becomes a non-issue. That would translate to a very low scale-to-zero timeout. That's certainly true, a cold start might involve more than just creating the container though (setting up connections in the function etc. etc.), so I'm wondering how far we can push the boundary on that end.

Certainly, having a container creation/removal lifecycle so fast makes the whole container lifecycle easier (less states to be in) and would possibly reduce overall complexity, so that'd be the nicest of them all.

[evankanderson]

/kind spec

This is a good discussion; I'm going to pull out the CPU freeze discussion from the runtime contract and reference this issue.

[jchesterpivotal]

Another motivating problem: someone manages to get cryptocoin mining logic into my function or base image. I want that code to only be able to run while a request is being served or event processed. It won't prevent the mining logic from executing, but it limits the scope and increases the odds of detection via latency/throughput metrics.

[jhanley-com]

Another motivating problem: someone manages to get cryptocoin mining logic into my function or base image. I want that code to only be able to run while a request is being served or event processed. It won't prevent the mining logic from executing, but it limits the scope and increases the odds of detection via latency/throughput metrics.

If someone can modify your image or function, then there is nothing to stop them from just delaying an HTTP request/response to the max time limit.

[jchesterpivotal]

@jhanley-com you're right, but I think the key is that delaying the HTTP response will quickly show up in response time metrics. @markusthoemmes was pointing out that in the execution model, the request could be processed normally and unobserved background work could continue without being connected to any request, until scale down occurs.

[mbehrendt]

thx @markusthoemmes -- this is a great topic.

In my mental model, the issue you're raising has some sub-aspects:

1. What determines the active, billable lifetime of an invocation, in today's blocking invocation model?
   Given today's blocking-only, multi-threaded model, I think the user would be charged by all container capacity available from serving the first to serving the last request. If invocations are run in a single-threaded model, the fundamental charging model would be the same, and as a result capacity the user is charged for matches the aggregate time it takes to serve each individual request.
   In this context, i think we only have to be concerned about cycle stealing if we deviated from using the request start and end times as the signals for starting/stopping metering. As we stick to them, the container would be destroyed once not needed anymore, in a very simple implementation and no matter what the customer would be running (incl things like bitcoin mining).
   The topic of freezing / thawing becomes (only) relevant if we want to avoid container creation since it is much more costly than just thawing suspended container. Obviously, based on our experience so far i think this is needed for a system that should be competitive.

2. Given the definition of a container being active from req begin to res end --- which mechanisms can we use in order to minimize cold starts -- e.g. by facilitating container reuse?
   i think there is just about a handful -- memory suspend/resume, optimize container provisioning times, pre-warm creation of containers, etc. . I think we should work on all of them, just one would likely not cut it.

3. What defines the execution end time, if we want to be able to run something asynchronously in the background?
   I believe we also need to support async invocations. For an async invocation, the time of the http response obviously doesn't make sense as the signal for charging, since the req returns pretty much immediately after the invocation request was received. The end would then have to be determined by the function finishing its work. Here, the customer would have to pay for the lifetime of the container while being active and from an optimization perspective (2) would apply here as well.

Long story short -- as long as the definitions for when metering starts/ends are clearly defined, avoiding stealing of cycles becomes a much easier topic. Because then the problem is reduced in my mind to whether i as a provider go with a suspend/remove or container-provisioning-time-optimization strategy (or both combined) . The former one is pretty proven and can be done today (with a few technology options, as you laid out above), the latter would require some more technical groundwork and therefore doesn't seem like a super short-term solution (i would love it to be wrong here -- if there is a competitive technology out there, pls bring it up).

WDYT?

[knative-housekeeping-robot]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle stale.
Stale issues rot after an additional 30 days of inactivity and eventually close.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle stale

[knative-housekeeping-robot]

Stale issues rot after 30 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle rotten.
Rotten issues close after an additional 30 days of inactivity.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle rotten

[knative-housekeeping-robot]

Rotten issues close after 30 days of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh by adding the comment /remove-lifecycle rotten.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/close

[knative-prow-robot]

@knative-housekeeping-robot: Closing this issue.

In response to this:

Rotten issues close after 30 days of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh by adding the comment /remove-lifecycle rotten.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.