No extension point at restore, so a restored actor cannot be reconciled against current policy

[Harihara04sudhan]

Problem

A snapshot captures execution state at a point in time. Any authorization derived
from outside the actor can change while it is checkpointed. On restore there is
no point at which an external component can reconcile the two.

Concretely, on the full-scope restore path:

• no container is created, so no OCI hook fires (restore.go calls neither
  CreateContainer nor StartContainer; only run.go does)
• the guest resumes with whatever in-guest state the snapshot captured
• the actor becomes reachable with no opportunity to re-validate it

So a policy decision made while the actor slept cannot be applied to it. The
actor comes back with the authority it had when the snapshot was taken.

This is not specific to any one policy system. Anything that establishes state
before a workload runs and needs it to still be valid afterwards has the same
problem: capability tokens, short-lived credentials, admission decisions, lease
or quota state, tenancy or residency constraints.

What already exists

The lifecycle point is arguably already there. restoreFullScope blocks before
returning:

// Block until every readyz-enabled container reports 200.
if err := readyz.WaitAll(ctx, containers, ateomnet.ActorVethIP); err != nil {

and durable volumes are re-materialised even earlier, before the VM starts:

// Restore the durable-dir volumes before anything can observe them
if hasDurableVolumes(p.containers) {
        if err := untarDurableVolumes(durableDir, restoreDir); err != nil {

So restore already has a stage where the actor is not yet reachable. What is
missing is a way for anything to run there.

Suggested shape

An optional, configured reconciliation step invoked on the restore path before
the actor becomes reachable, which can fail the restore. Roughly:

• runs after durable volumes are re-materialised and before the VM is resumed, so
  a rejected actor never executes rather than being stopped mid-flight
• receives enough identity to make a decision: actor ref and uid, template, and
  the restored durable directory
• a non-nil error fails the restore, and the existing teardown path already
  handles cleanup
• absent configuration means no behavioural change

Placing it before the VM resumes matters more than the exact interface. Doing it
after readyz means the workload has already run.

Why we are raising it

We are building kernel-level enforcement that binds a workload to an authority
epoch established before it runs, and we need that epoch re-checked when a
snapshot is restored so a revoked authority cannot come back with the snapshot.
We have this working locally by reading state the guest wrote onto the durable
share and comparing it at the untar point, which is enough for a demonstration
but relies on an implementation detail rather than a contract.

Happy to send a PR for whichever shape you prefer, or to adapt to something you
already have planned. Related: #1224 (guest_hook_path propagation), which is
the corresponding gap on the pre-activation side.

[BenTheElder]

cc Taahir Ahmed (@ahmedtd) this seems like it has a lot of overlap with the inbuilt identity mechanisms ... I'm not sure we want to optimize for a "non standard" solution

[ahmedtd]

HARIHARA SUDHAN. R (@Harihara04sudhan) Can you write out a very concrete example of what you're trying to achieve?

I don't think that attempting to delete credentials from the actor's filesystem is a good way to approach this (what if they are in memory, or uploaded to some external system?). Your credentials need to either be revocable, or short-lived.

Our current approach to identity is:

• For most purposes, you should rely on the egress gateway to inject credentials into outbound request (this is currently being worked on).
• If needed, we will provide SystemInformationVolume support for writing Substrate-issued actor JWTs and certificates into the actor's filesystem. These would be short-lived and automatically refreshed, but they also don't have any sort of epoch-based concerns --- as long as the Actor object exists in Substrate's database, it's valid to issue a credential that asserts its identity.

[Harihara04sudhan]

Thanks for engaging with this. I see what you're saying, and the identity point is fair.

What we're doing is containing agent objectives at the system layer. There are two ways to do that on Substrate, the guest VM or gVisor, and we picked the guest VM: an LSM in the Kata guest kernel to start with and minimize substrate changes. gVisor can be done too, it's a bit more work, since there's no guest kernel to load into and the check would have to live in the Sentry. If that's what it takes for this to be portable enough, we'll contribute it.

Taahir Ahmed (@ahmedtd) the concrete example you asked for. An agent is told to fix the failing unit tests in a repository. The rules allow reads and writes under /workspace/repo/ and running the compiler, and nothing else: not /home/agent/.ssh, not /etc, no other binary. The agent makes those syscalls itself. All the checks are good.

Now consider, if it goes idle part way through a test run and the platform suspends it. While it's asleep someone revokes the objective. On resume, it reads /home/agent/.ssh/id_ed25519.

For that not to work, something has to be checked at two points: before the workload first becomes active, and before a restored one becomes reachable. That's why there are two issues.

On credentials, we're not deleting or managing any. We install a set of rules that the guest kernel evaluates at the syscall boundary, so it sits alongside whatever Substrate issues. And you're right that they should be short-lived, our own tokens default to sixty seconds. What an expiry can't cover is a restore: a full-scope snapshot brings the guest back on its own frozen clock, so a restored guest can believe its rules are still valid when they expired an hour ago.

We raised these as issues rather than feature requests because there seem to be other use cases here, not just ours. Capability tokens, leases and the short-lived credentials in a SystemInformationVolume all hit the same restore problem. Happy for them to be reclassified however you prefer.

We're not blocked on these changes. We've made them in our fork and that's sufficient to address our issues. It does make our life considerably easier going forward if there is a standard way of addressing this, and it would mean nobody else has to fork for the same reason.

Would love to present this on one of your monthly calls if that helps. More details, including the gVisor sketch and answers to the questions on #1223: intentd on Agent Substrate: the use case, and what we are asking for

[BenTheElder]

What an expiry can't cover is a restore: a full-scope snapshot brings the guest back on its own frozen clock, so a restored guest can believe its rules are still valid when they expired an hour ago.

This isn't true, the clock is adjusted on restore. Try again @ HEAD