Proposal: Auth

[asher-pem-arm]

To facilitate the use of Labgrid in production environments we propose adding flexible authentication and authorisation functionality. This functionality will be backwards compatible so that Labgrid can continue to run in an insecure way for deployments that do not have security requirements (eg. local usage).

This proposal is dependant on Proposal: gRPC Refactor. However, this proposal does not reference any of the RPC protocol changes as they are yet to be implemented. The examples in this proposal can become more concrete once those changes are implemented.

Implementing authentication and authorisation requires the introduction of the following:

• Identity context

  The operations supported by labgrid-coordinator (eg. creating/deleting Places, acquiring/releasing Places, etc.) need to be restricted based on the calling user (the principal).

  We propose introducing a ContextVariable that provides access to the identity of the user and the capabilities they have (see Capability model below).

  The ContextVariable will be set in the Coordinator-side plugin (see Client/Exporter and Coordinator-side plugins for injecting and parsing authentication metadata below) and will be accessible from the RPC method handlers.

• Capability model

  The operations a Labgrid user can perform cannot be restricted only by the type of operation (eg. creating a Place, acquiring a Place, etc.). For example, whether a user can release a Place is determined by whether they are the owner of the Place lock.

  For this reason, we propose modelling the actions a user may be granted as capabilities that can be validated as part of the RPC method handler logic.

  For example, the release Place operation can be subdivided into the ReleasePlace_Owned (for releasing Places you own) and ReleasePlace_Any (for releasing any Place, regardless of whether you acquired it).

• Client/Exporter and Coordinator-side plugins for injecting and parsing authentication metadata

  With gRPC APIs, authentication information is passed in the gRPC metadata.

  We propose a pluggable interface in labgrid-client/labgrid-exporter for injecting the authentication into the gRPC metadata and in labgrid-coordinator for reading the metadata and populating the Identity context.

• Refactoring of existing RPCs to include capabilities checks

  The current labgrid-coordinator implementation embeds the authorisation logic into the RPC method handlers, based on the pseudo-identity provided in the StartupDone message.

  We propose refactoring the RPC method handlers to base their authorisation checks on the capabilities of the user, accessible from the identity context.

Identity context

We propose adding a ContextVariable for storing the per-request principal information:

client_identity_context: contextvars.ContextVar[Identity] = contextvars.ContextVar("client_identity")

We propose Identity as a Python Protocol, allowing flexibility of implementation by the labgrid-coordinator authentication plugin:

# labgrid/auth/auth.py
from typing import Collection, Protocol
from .capability import Capability
class Auth(Protocol):
    @property
    def id(self) -> str:
        """A unique identity of the user."""
        ...
    @property
    def capabilities(self) -> Collection[Capability]:
        """Capabilities this authenticated user can perform."""
        ...

Capability model

We propose the following capabilities:

Method (RPC)	Capability	Description
ClientStream	client_stream	Ability to establish the ClientStream.
ExporterStream	exporter_stream	Ability to establish the ExporterStream.
AddPlace	add_place	Ability to add a Place.
DeletePlace	delete_place	Ability to delete a Place.
GetPlaces	get_places	Ability to list Places.
AddPlaceAlias	add_place_alias	Ability to add an alias to a Place.
DeletePlaceAlias	delete_place_alias	Ability to delete a Place alias.
SetPlaceTags	set_place_tags	Ability to set Place tags.
SetPlaceComment	set_place_comment	Ability to set the comment on a Place.
AddPlaceMatch	add_place_match	Ability to add a Resource match to a Place.
DeletePlaceMatch	delete_place_match	Ability to delete a Resource match from a Place.
AcquirePlace	acquire_place	Ability to acquire a Place.
ReleasePlace	release_place_owned	Ability to release a Place that is owned by the calling principal.
ReleasePlace	release_place_any	Ability to release any Place. Effectively the ability to run labgrid-client release -k.
AllowPlace	allow_place_owned	Ability to allow another user access to a Place that is owned by the calling principal.
AllowPlace	allow_place_any	Ability to allow another user access to any acquired Place.
CreateReservation	create_reservation	Ability to create a Reservation for a Place.
CancelReservation	cancel_reservation_owned	Ability to cancel a Reservation that is owned by the calling principal.
CancelReservation	cancel_reservation_any	Ability to cancel any Reservation.
PollReservation	poll_reservation	Ability to poll the state of a Reservation.
GetReservations	get_reservations	Ability to list Reservations.

The capabilities could be represented as an enumerable:

# labgrid/auth/capability.py
from enum import StrEnum, auto
 
class Capability(StrEnum):
    client_stream = auto()
    exporter_stream = auto()
    add_place = auto()
    delete_place = auto()
    get_places = auto()
    add_place_alias = auto()
    delete_place_alias = auto()
    set_place_tags = auto()
    set_place_comment = auto()
    add_place_match = auto()
    delete_place_match = auto()
    acquire_place = auto()
    release_place_owned = auto()
    release_place_any = auto()
    allow_place_owned = auto()
    allow_place_any = auto()
    create_reservation = auto()
    cancel_reservation_owned = auto()
    cancel_reservation_any = auto()
    poll_reservation = auto()
    get_reservations = auto()

Client/Exporter and Coordinator-side plugins for injecting and parsing authentication metadata

gRPC metadata is the channel for transmitting authentication metadata. gRPC metadata is comprised of K/V pairs.

We propose a plugin architecture where, for a given identity service:

• a plugin is implemented for labgrid-client/labgrid-exporter that injects the identity service-specific metadata K/V pairs
• a plugin is implemented for labgrid-coordinator that processes the metadata K/V pairs, validates them and, if valid, emits the Identity for the principal

We propose Python Protocols for the plugins below. We propose these as callables to allow plugins to be used in a function-like way.

Client/Exporter-side

from typing import Mapping, Protocol
class ClientAuthPlugin(Protocol):
    def __call__(self) -> Mapping[str, List[str]]:
        """Produces the gRPC metadata pairs to add to each RPC call's metadata.
        """
        ...

Coordinator-side

from typing import Mapping, Protocol

class IdentityError(Exception):
    """Exception to be raised when the provided metadata cannot be resolved
    to a valid identity."""
    pass

class ServerAuthPlugin(Protocol):
    def __call__(self, metadata: Mapping[str, List[str]]) -> Identity:
        """Consumes the gRPC metadata, performs identity provider-specific
        validation and, if successful, emits a populated Identity."""
        ...

Default plugins

We propose the implementation of default plugin implementations that replicate the current functionality of Labgrid.

For the Client/Exporter-side plugin this will involve the injection of the hostname (and username, for the Client) into the metadata.

For the Coordinator-side plugin, this will involve generating an Identity-compatible object that uses the hostname/username as the id and a static set of capabilities that reflect the current Labgrid behaviour.

Refactoring of existing RPCs to include capabilities checks

For RPC methods that have a 1:1 mapping with a Capability, we suggest a method decorator that specifies the Capability (or potentially multiple) required to call the method:

@capability(Capability.add_place)
async def AddPlace(self, request, context): ...

For RPC methods that require more granular access, a decorator is not sufficient and the Capability-checking logic will need to be embedded into the method logic:

async def ReleasePlace(self, request, context):
    ...
    principal_id = client_identity_context.get().id
    principal_capabilities = client_identity_context.get().capabilities
    if Capability.release_place_any in principal_capabilities or \
        (Capability.release_place_owned in principal_capabilities and place.owned == principal_id):
            # Release place allowed
            ...
    ...

[Emantor]

Hi @pamolloy & @ozan956 would this also be a match for your infrastructure plans?

[gastmaier]

Hi @Emantor, on behalf of @pamolloy;

The changes above are welcome, but are not clear how the identity->capability look-up would be implemented; if there is a token in the grpc metadata, the encryption of the grpc channel needs to be handled, too.

To resolve that, I did a PoF at commit afd2d0f master...gastmaier:labgrid:tls

For the client|exporter-side, a new argument --ssl is added to enable TLS, where the system certificates are used, including SSL_CERT_FILE env var, or explicitly by passing a path. If --ssl is set, grpc.aio.secure_channel is used instead grpc.aio.insecure_channel, straight forward, already implemented by the lib.
For the coordinator, no changes are necessary, the user adds the coordinator behind a reverse proxy with TLS, like nginx. This is preferable to having labgrid|grpc managing the secret keys directly.

The second security concern from my side is ssh key-management, but https://github.com/openpubkey/opkssh seem to provide both OIDC for CI/CD workflows and and SSO for users; so changes to labgrid are also not necessary.

We still need to test opkssh + client|exporter ssl to confirm it fully solves the security+encryption needs.

[Emantor]

There is #1885 for TLS secured gRPC.

[jluebbe]

I've got a few questions for understanding:

• Are the credentials per gRPC channel or per call? What are the benefits for either approach?
• How is the relationship between the new Place.owned and the existing Place.acquired?
• How would you permit a user to only acquire a restricted set of places? As far as I can see, the capabilities are not Place/Resource-specific?

[asher-pem-arm]

Hi @jluebbe,

In the old API, several unary RPCs effectively depend on a prior stream-side identity registration. That makes auth awkward to review and implement cleanly, because authorization for a unary RPC partly depends on unrelated stream state. With the refactor, identity is available on the RPC being authorized, and the coordinator can make the authorization decision locally in that handler.

Keeping auth on top of the refactor avoids designing two auth paths, one for legacy stream/session identity and another for the new unary API. We planned to keep the legacy StartupDone fallback for backwards compatibility, but we do not want the new auth model to be constrained by that.

So I think the split we are aiming for is:

1. Land the gRPC/API refactor that detaches unary RPCs from ClientStream identity.
2. Add auth/capability checks on the refactored identity model.

[asher-pem-arm]

with response to the previous questions:

• Are the credentials per gRPC channel or per call? What are the benefits for either approach?
  that's per call as the only way you could do channel-level auth would be to move exclusively to mTLS.
  With our design we retain the flexibility to use any method we want.
• How is the relationship between the new Place.owned and the existing Place.acquired?
  They are intended to be identical. It made more sense to have "owners" and "shared_with" - see the bottom of the gRPC refactoring proposal: Proposal: gRPC Refactoring #1881
• How would you permit a user to only acquire a restricted set of places? As far as I can see, the capabilities are not Place/Resource-specific?
  The proposed capabilities are not Place/Resource-specific. That was a deliberate simplification where principals would have roles/relationships to specific assets such as Places or Reservations. The current proposal aims to cover the common/global cases first but open to discussions on how it could be extended

[jluebbe]

In the old API, several unary RPCs effectively depend on a prior stream-side identity registration. That makes auth awkward to review and implement cleanly, because authorization for a unary RPC partly depends on unrelated stream state. With the refactor, identity is available on the RPC being authorized, and the coordinator can make the authorization decision locally in that handler.

Keeping auth on top of the refactor avoids designing two auth paths, one for legacy stream/session identity and another for the new unary API. We planned to keep the legacy StartupDone fallback for backwards compatibility, but we do not want the new auth model to be constrained by that.

Hmm. I've discussed the gRPC refactor (unary RPCs, StartupDone) with @Emantor and @Bastian-Krause for most of the day and I'm not yet convinced that adding new RPCs for listing requests is the right way forward. There are several open questions which I'd prefer not to block adding authentication, as e.g. #1888 is also working in that direction.

It seems to me that the first two commits of #1918 basically contain the machinery so that individual RPC could use information provided by infer_peer_identity and pass it into a (to be added) policy mechanism/plugin.

We'll need to keep sending the StartupDone message for a while to support new clients on old coordinators, but this also allows us to keep creating the ClientSession in a single place (either from the peer identity if provided or from the StartupDone contents otherwise). This way, we have separate commits/PRs for metadata-based identity, adding authrization, then requiring authentication in the coordinator and finally dropping the StartupDone message in the client.

[asher-pem-arm]

Hi @jluebbe, we could certainly split down #1918 into the first two commits if that helps get that part in. We have split #1918 into that and #1919, so if we the first set of refactors are agreed on we could get those in and do #1919 after we start auth.

For the fallback identity path, do we need to support old clients when auth is enabled?
New clients should keep sending StartupDone for compatibility with old coordinators. But for a coordinator running in an auth-required mode, it seems reasonable to require new clients from release onwards? This would simplify the code and the possible test combos (i.e. saving integration/unti testing old clients with auth etc)

[Emantor]

We should not support old clients with auth enabled. We still support StartupDone for old clients when auth is disabled, but a coordinator requiring Auth will require the metadata based identity.

[jluebbe]

Building on top of auth, there could then be an optional credentials dict on Resources which is only send from the exporter to clients which have acquired a place with that resource.

[asher-pem-arm]

think that should be kept separate from this first proposal on auth/capabilities. But we will have a follow up proposal that will cover sensitive fields in resources