[Draft] Proposal: User Managed Session

[mattisonchao]

Summary

Let users manage Oxia sessions themselves. Today the Go client creates one session per shard behind the scenes and keeps it alive for as long as the client runs; Ephemeral() puts attach records to it. A user managed session is one the application controls directly:

• Lifecycle: the application creates it with its own timeout, keeps it alive, and closes it.
• Shared by ID: any client that knows the session's ID can use it, not just the client that created it.
• Inspectable: the application can read the session's remaining time and list the keys attached to it.

The server already does most of this. The proposal adds:

1. A client API for user managed sessions, built on the existing CreateSession, KeepAlive and CloseSession RPCs.
2. GetSession, a new RPC that returns a session's timeout, remaining time and client identity.
3. A session filter on List that returns the keys attached to a session.
4. A configurable maximum session timeout, replacing the fixed 5 minutes.

The first user is oxia-etcd, which serves the etcd v3 API on Oxia: an etcd lease maps directly onto a user managed session.

Motivation

An etcd lease and an Oxia session are the same thing. A client creates an ID with a TTL and keeps it alive. When the ID expires or is revoked, the keys attached to it are deleted.

etcd	Oxia today
LeaseGrant(TTL)	CreateSession(session_timeout_ms)
LeaseKeepAlive(ID)	KeepAlive(session_id)
LeaseRevoke(ID)	CloseSession(session_id)
Put(key, lease=ID)	Put with session_id
Expiry deletes the keys	Expiry deletes the ephemeral records
LeaseTimeToLive(ID)	❌ a session can't be read
LeaseTimeToLive(ID, keys=true)	❌ a session's keys can't be listed
A TTL of up to ~9e9 s	❌ at most 5 minutes
Any client can use a lease by ID	❌ a session is private to the client that created it

Because these gaps exist, oxia-etcd implements leases itself. It stores each lease as a plain record and tracks the expiry in its own memory. This has two costs:

• Only one gateway can serve a cluster's data. The expiry state is in one process's memory, so a second gateway can't keep the same leases alive.
• The gateway re-implements expiry that Oxia already has, including deleting the attached keys.

The timeout cap also matters on its own: the Kubernetes API server grants Event leases with the event TTL, which is 1 hour by default.

User managed sessions are useful beyond etcd too. Any application can share one lease-like lifetime across several processes, such as a job's workers whose records should disappear together.

Design

Client API

A new Session handle in the Go client:

// Create a session in the shard of a partition key
s, err := client.CreateSession(ctx, time.Hour, oxia.PartitionKey("cluster-a"))

// Or open an existing one by ID, possibly from another process
s := client.OpenSession(id, oxia.PartitionKey("cluster-a"))

s.ID() int64
s.KeepAlive(ctx) error             // one heartbeat; the application decides when
s.Info(ctx) (SessionInfo, error)   // GetSession: timeout, remaining time, identity
s.Close(ctx) error                 // CloseSession: deletes the attached records

// Attach a record to the session
client.Put(ctx, key, value, oxia.PartitionKey("cluster-a"), oxia.WithSession(s))

// List the records attached to the session
client.List(ctx, "", "", oxia.PartitionKey("cluster-a"), oxia.UseSession(s))

• A session is addressed by a partition key and an ID. Sessions live in a single shard, and session IDs are only unique within that shard. The partition key routes every call to the shard holding the session, and splits are followed the same way they are for records. A split child inherits its parent's sessions with the same IDs.
• No background keep-alive. The application sends the heartbeats. Wrapping this in a helper that keeps the session alive in the background is easy, but the default behavior stays explicit.
• Any client can keep alive, inspect or close a session. The server doesn't check the identity on these calls today, and this proposal keeps it that way.
• Ephemeral() is unchanged. It keeps using the client's internal sessions.
• WithSession requires a matching partition key. A put with WithSession(s) must carry the same partition key as the session, otherwise the client rejects it.

GetSession

rpc GetSession(GetSessionRequest) returns (GetSessionResponse);

message GetSessionRequest {
  int64 shard = 1;
  int64 session_id = 2;
}

message GetSessionResponse {
  uint32 session_timeout_ms = 1;  // the timeout the session was created with
  int64 remaining_ms = 2;         // time left before the session expires
  string client_identity = 3;
}

• The shard leader answers it from memory. The leader already tracks each session's expiry deadline (session.deadline), so the remaining time is deadline - now.
• An unknown or expired session returns ErrSessionNotFound, as KeepAlive does.
• The remaining time resets after a leader change. A new leader restores sessions from the database with their full timeout. This is the current behavior, and it matches etcd's default, where lease checkpointing is off.

Listing a session's keys

ListRequest gets an optional session ID:

message ListRequest {
  optional int64 shard = 1;
  string start_inclusive = 2;
  string end_exclusive = 3;
  optional string secondary_index_name = 4;
  bool include_internal_keys = 5;
  optional int64 session_id = 6;  // only the keys attached to this session
}

• It reads an existing index. For each ephemeral record, the shard already keeps a shadow entry ShadowKey(session, key) under the session key; expiry uses these entries to find the records to delete. The server scans only that session's range (shadowKeysRange), unescapes the keys and applies the [start, end) bounds.
• It streams. A session can hold many keys. For example, Kubernetes attaches every Event created within about a minute to the same lease. List already streams its results in batches.
• It needs a single shard. The request must name a shard or carry a partition key. A List across all shards with a session ID is rejected, and so is combining session_id with secondary_index_name.

Configurable maximum session timeout

constant.MaxSessionTimeout becomes a data server setting, defaulting to the current 5 minutes. The protocol needs no change: session_timeout_ms is a uint32, which already allows about 49 days.

A long timeout means a dead client's records stay around longer. That's the same trade-off etcd users make when they pick a lease TTL.

How oxia-etcd would use it

etcd API	oxia-etcd
LeaseGrant	CreateSession
LeaseKeepAlive	OpenSession(id).KeepAlive
LeaseRevoke	OpenSession(id).Close
LeaseTimeToLive	Info
LeaseTimeToLive(keys=true)	Info, then List with UseSession, merged into one response
LeaseLeases	the gateway's own lease records
Put(key, lease=ID)	Put with WithSession

Because sessions are shared by ID, every gateway can serve every lease, so the gateways no longer need an active/standby split for leases.

LeaseTimeToLive(keys=true) makes two calls. If the session expires between them, the key list may come back short. etcd can return the same result just before a lease expires, so the gateway doesn't retry.

Non-goals

• ListSessions. oxia-etcd keeps its own lease records, so it can answer LeaseLeases without it.
• Keeping the remaining time across leader changes (etcd's lease checkpointing).
• Client-chosen session IDs. etcd lets a client choose a lease ID; the gateway can map etcd IDs to session IDs itself.
• Access control on sessions. Anyone who knows a session's ID can use it, as with records today.

Related

• Session ids are re-minted from 0 when the WAL is empty or lags the database, colliding with sessions recovered from the DB #1333: session IDs are taken from the WAL offset and can be reused when the WAL is empty or lags the database. A session ID that other processes share must be unique, so this has to be fixed first.
• Session manager: per-session goroutine + timer costs ~15-20 KB of server RAM per live session #1235 (closed): sessions now cost little enough to keep one per lease.

Open questions

• Should GetSession also return the number of attached keys?
• Should the timeout cap be set per namespace instead of per data server?
• Should the client provide a helper that keeps a session alive in the background?