gap-8_http_auth_runtime.md

gap	title	description	author	status	type
8	HTTP authentication runtime	A gateway runtime for already-running HTTP-based services.	mf (@mfranciszkiewicz)	Draft	Feature

Abstract

This document describes a runtime gateway between the Golem Marketplace and an HTTP-based service, to be made publicly accessible over the Internet. Lifecycle and configuration management of that service lies with the authority of a Provider Agent administrator.

Inbound HTTP requests from the Internet to the service are routed via a custom reverse HTTP proxy implementation. The proxy authorizes the requests and collects per-user, per-endpoint usage statistics for the purpose of billing Requestors on the Golem Marketplace.

The runtime is responsible for adding and removing users authorized to use the service on Requestor's demand. The former is achieved via Proxy's Management API, available only on a private network address.

Motivation

The specification has been derived from the previous work on Erigon / Lighthouse service runtimes. It was concluded that a generic HTTP authentication runtime could be developed instead.

Specification

Key assumptions made while designing the runtime:

• Provider Agent administrator can add / remove services by adding / removing config files from a designated directory
• The runtime controls access to services
  • Enforces HTTP basic authentication by default; the runtime is extensible in this manner
  • There is a separate access list for every service
  • Requestor can pass in the access credentials (username + password); these should not be generated by the runtime. Username collision should be handled.
• Runtime should expose number of requests as a custom usage metric.
  • Breakdown of request count into particular service endpoints (i.e. by URI)

Key responsibilities of the reverse HTTP proxy:

• securing external HTTP communication with TLS

  • plain HTTP mode can be explicitly enabled via a configuration file / command line arguments

• forwarding authorized HTTP requests to services

• exposing a private Management API, utilized the Runtime, for:

  • exposing / hiding a service
  • listing registered services
  • adding / removing user credentials per service
  • listing registered users
  • reading usage statistics per service user

Runtime implementation

The implementation will be based on ya-runtime-sdk.

Command line arguments

Command line should accept a service name argument, according to the agreement.

An example can be found here.

Configuration

The runtime configuration file may contain the following values:

• proxy Management API network address
• proxy log directory path
• additional service configuration lookup directories

Background jobs

Implement the following in order to publish the custom runtime counter values:

1. Request counter fetching

   Periodically calls the statistics endpoints of proxy's Management API regarding all users registered by the current Requestor.

2. Publishing counter values to the ExeUnit Supervisor

   Periodically updates the custom counter value (total requests made by a user) via the interface provided by Runtime SDK. An example of a custom counter can be found here.

Runtime trait implementation

1. deploy

• Read the variant of the requested service from command line arguments
• Perform a service configuration file lookup and locate the requested service
• Save deployment metadata to the current working directory

2. start

• If not running, in background, start an instance of the proxy server; the proxy binary is expected to reside in the same directory as the runtime binary. The proxy binary is guarded via a lock file when starting up.
  • loop
    • if no lock file is present, create and lock the file
      • start the process and use the via Management API to check whether it's responding properly
      • break
    • if the file is present, wait for acquiring the lock
      • if the Management API is responding, break
      • else, create the process and await until it's responsive

3. run

   • user add <username> <password>

     Adds a new user via Management API and within local memory. If a username exists, returns an error (with a proper message outputted to stderr); the runtime is terminated.

   • user remove <username>

     Removes a user via Management API. Performs a lookup in a local memory store firstt.

   • user stats <username> <counter-name>

     Reads the specific counter value for the given user. The foreseen counter names will be set to the requested URIs (locations) configured in nginx.

   The commands above support an -a,--auth for setting the authentication method:

   • basic for HTTP basic authentication (default)
   • other forms of authentication are planned for future releases

4. transfer

   • none

5. offer-template

• extra Offer properties defined in service configuration
• supported proxy authentication methods as runtime capabilities
• (optional) partial service configuration as Offer properties (e.g. cpu threads, timeout values, user auth method)
• (optional) in case of self-signed certificates, add a cert string property in the runtime property namespace, containing the certificate checksum, in the following format: sha3:0badbeef

6. test

• perform a service configuration file lookup
• assert that the proxy binary is runnable
• assert that the service HTTP interface is available
• (optional) perform a public IP assertion check for the Provider

Shutdown procedure

Implement a process termination handler using ya-runtime-sdk:

1. Fetch last available counter values via Management API
2. Publish the latest known counter values via ya-runtime-sdk
3. De-register the user (if any) via Management API

Deployment

Each service should be assigned its own runtime name and use the common ya-runtime-basic-auth binary, e.g.:

[
  {
    "name": "ya-runtime-erigon",
    "version": "0.1.0",
    "supervisor-path": "exe-unit",
    "runtime-path": "ya-runtime-http-auth/ya-runtime-http-auth",
    "description": "The Basic-Auth runtime for the Erigon service",
    "extra-args": [
      "--runtime-managed-image",
      "--runtime-arg", "ya-runtime-erigon"
    ]
  }
]

depending on the installation method, located at:

• /usr/lib/yagna/plugins/ya-runtime-erigon.json (manual or deb installation)
• ~/.local/lib/yagna/plugins/ya-runtime-erigon.json (installer script)

HTTP proxy

Multi-threaded TCP server supporting HTTP 1.1 and HTTP 2.0 protocols.

By default, the proxy binds to a single local address:

• private (localhost:6668) for the Management API

For each service, a separate HTTPS (TLS 1.2, 1.3) server can be spawned. Each service can be configured with a custom listening socket address and certificate / key pair, otherwise it is expected to use the defaults. Domain configuration lies in Provider Agent administrator's responsibility.

The proxy supports a plain HTTP mode for development purposes, enabled explicitly via command line arguments.

Management API (private address space)

Management API implementation keeps track of the number of registered users and de-registers services when the user counter reaches 0 (i.e. at least one user needs to be registered first).

1. Service management

• GET /services -> List[Service] ? [pageSize, offset, count]

  Retrieves a list of registered services

• POST /services w/ CreateService

  Registers a service; returns 204 if the service already exists and all service parameters match.

  Tries to spawn an additional HTTP server thread when any of these conditions are met:

  • the listening address is not covered by the default configuration
  • service port is not covered by the default configuration

  In other cases, the default server thread will be used, but the service configuration must match the default proxy configuration (listening address, certificates, etc.).

• GET /services/{service_name} -> Service

  Retrieves service details

• DELETE /services/{service_name}

  De-registers a service and stops the extra HTTP server (if any)

2. User management per service

• GET /services/{service_name}/users -> List[User] ? [pageSize, offset, count]

  Lists users registered within the service

• POST /services/{service_name}/users w/ CreateUser

  Registers a new user within the service; returns 400 if a username already exists

• GET /services/{service_name}/users/{user_name} -> User

  Retrieves single registered user information

• DELETE /services/{service_name}/users/{user_name}

  De-registers a user from the service. If the user count drops to 0, de-registers the service and stops the extra HTTP server (if any)

3. User statistics

• GET /services/{service_name}/users/{user_name}/stats -> UserStats

  Per-service user statistics

• GET /services/{service_name}/users/{user_name}/endpoints/stats -> UserEndpointStats

  Per-endpoint, per-service user statistics

• GET /stats -> GlobalStats (optional)

  Proxy-wide (global) statistics

Runtime developers are expected to create an OpenAPI specification of the Management API.

Proxy (public address space)

The public TCP connection handler inspects and mangles each incoming HTTP request only to forward that request to the destination service server and tunnel the response.

The inspection and mangling module operates as follows:

1. Resolve the registered service basing on the requested URI
2. Authorize credentials included in the header
3. Rewrite the location in request's header to a relative service URI (the extra part of the proxied service endpoint)
4. Add / replace a X-Forwarded-Host header, if a domain name was set for the service
5. Add / replace a X-Forwarded-For header containing the connecting client's IP address
6. (optional, TBD) Add / replace a Forward header instead X-Forwarded-For
7. Forward the requested packet to the destination service server

The proxy server should update statistics for each unauthorized (global), failed (user / global) and successful request (user / global).

API models

• CreateService

  {
    "name": "service_name",
    "bind": "1.120.111.43:443",
    "domain": "example.com",
    "user": {
      "auth": {
        "method": "basic"
      },
      "requestTimeout": 10000,
      "responseTimeout": 2000
    },
    "auth": {
      "method": "basic"
    },
    "cert": {
      "path": "/etc/certs/cert.crt",
      "keyPath": "/etc/certs/cert.key"
    },
    "from": "/service",
    "to": "http://127.0.0.1:12345/api/v1/service",      
    "requestTimeout": 10000,
    "responseTimeout": 2000,
    "cpuThreads": 2
  }

• Service

  CreateService extended with a createdAt property (RFC3339 timestamp as string).

• CreateUser

  {
    "name": "username",
    "password": "dXNlcm5hbWU="
  }

  • password - base64 encoded password

• User

  {
    "name": "username",
    "createdAt": "2022-01-12T23:20:50.52Z"
  }

• UserStats

  {
    "total": 12,
    "failures": 0
  }

• UserEndpointStats

  {
    "/service/run": 1,
    "/service/build": 12
  }

• GlobalStats (optional)

  {
    "users": 43,
    "services": 3,
    "requests": {
      "total": 11314,
      "unauthorized": 44
    }
  }

CLI

The proxy server can be parameterized via the following command line arguments:

• (optional) Management API addresses
• (optional) log directory path
• (optional) default proxy API binding address (defaults to 0.0.0.0:443)
• (optional) TLS certificate path
• (optional) TLS certificate key path

Service configuration

Configuration files should contain the following service information in a JSON / YAML / TOML format:

• name (identifier)
• root proxy REST endpoint (public)
• root service URL (local)
• (optional) user request timeout
• (optional) user response timeout
• (optional) service request timeout
• (optional) service response timeout
• (optional) server binding addresses
• (optional) dedicated certificate entry (e.g. when binding to a separate network interface, resolvable under a different domain name)
  • certificate path
  • certificate key path
• (optional) number of designated CPU threads (defaults to all available logical cores)
• (optional) extra static properties to be included in the Offer
• (unused) user authentication method (currently basic auth only)
• (unused) service authentication method (currently none)

Rationale

Custom reverse HTTP proxy

Per-user, per-endpoint statistics in (e.g.) nginx OSS are not available out of the box. These statistics would need to be derived from server logs, which follow a specific (and often user-defined) format, so a custom parser would have to be implemented. While it's valid to spawn multiple instances of the runtime binary, there would be multiple instances of the parser running in parallel; this solution does not scale well enough for a large number of runtimes.

Although most web servers are pluggable, the plugin development process would require setting up additional CD infrastructure for building with multiple versions of that software. Additionally, if the server running on Provider's machine is used for other purposes, interoperability with other plugins will increase the difficulty and time required for development. Third-party server configuration would become a prerequisite that can be difficult to validate, becoming a source of issues for the end users.

With a custom proxy implementation, a full flexibility in implementing usage counters is gained. The proxy will be packaged as a portable binary, so that it does not interfere with system configuration.

Self-signed certificates

The proxy will allow self-signed certificates and provide the certificate hash for clients to verify the public key used; they need not verify the domain name and the CA signature.

Backwards Compatibility

No backward-compatibility issues were discovered in the design / development process.

Test Cases

N/A

Reference Implementation

• HTTP proxy

Security Considerations

• the provider needs to properly configure TLS certificates so that the authentication credentials cannot be sniffed (i.e. they won't be sent as plain text)
• self-signed certificate hashes need to be included in the offer in order to prevent Man In The Middle attacks
• a proxy server can operate in a plain HTTP mode, which is only recommended for development. Still, it's possible for the server itself to be proxied by e.g. nginx taking over the responsibility of managing certificates and encrypting connections.

Copyright

Copyright and related rights waived via CC0.