llard Service Design

llar Cloud Design

Background

LLAR is a cloud-based multi-language package manager built with XGo. It manages C/C++ and other native libraries through declarative formula files. A formula describes how to resolve dependencies and how to build a library.

llar make is the existing build command. It resolves the requested module, loads formulas, resolves dependencies, builds dependencies before dependents, runs formula build hooks, and records build results in LLAR's local cache.

llar install adds a prebuilt artifact path for the same package manager. In the llard design, the client is only a protocol wrapper: it sends one artifact request for one module@version plus matrix selection, consumes the command JSON line response stream, downloads the returned artifact set, and writes local install directories and .cache.json entries.

Dependency resolution, dependency artifact reuse, and dependency artifact return move to the llard-side LLAR build path. llard runs llar make; Builder resolves dependencies, checks local cache, checks completed artifact records, downloads dependency artifacts when available, and builds missing dependencies locally when needed.

Cloud build is the remote artifact path:

llar make
cache miss -> build locally

llar install
request -> llard -> artifact or error

llard is an HTTP server for one artifact request path. GET /v1/artifacts/<module>@<version>?<matrix query> performs artifact lookup, waits for an active local build when one exists, or starts a new local build when the artifact is missing.

Goal

The goal is to implement remote artifact lookup, remote artifact production, direct artifact download for llar install, and llard-side dependency artifact return/reuse.

The architecture provides:

• one request per requested root artifact;
• hash-affine routing by module, version, and matrix query;
• command JSON line streaming through the same HTTP response;
• llard-local in-progress coordination;
• Builder-owned dependency resolution, dependency artifact lookup, and dependency artifact return;
• persistent storage for completed artifact records and artifact bytes;
• canonical completed artifact selection through artifact.Store.Put.

The runtime model is a llard model behind nginx hash routing. In-progress build state is memory-local to a llard. Completed artifact records are stored in the artifact database. Artifact bytes are stored in the Artifact Store. GHCR is the default Artifact Store backend.

Concepts

Concept	Meaning
GHCR	GitHub Container Registry. GHCR is the default backend for the Artifact Store, not a hard dependency of the llard architecture.
Artifact	A completed LLAR build result for one module, version, and matrix. The record contains download URL, archive type, and checksum. LLAR build metadata lives inside the archive at .llar/metadata.json.
llard	The cloud build service behind nginx hash routing. A llard handles GET /v1/artifacts/<module>@<version>?<matrix query>, shares in-progress work within the process, runs remote builds for cache misses, uploads archives, and records completed artifacts.
Builder cloud mode	The llard-side llar make mode that resolves dependencies and uses completed artifact records as an additional cache source before local builds.

Data Flow Architecture

The runtime has one client request path. nginx provides cross-llard routing. llard owns HTTP handling, local build coordination, artifact record access, and upload orchestration. The artifact DB stores completed artifact records. The Artifact Store stores archive bytes; GHCR is the default backend.

Completed artifact lookup always happens before local building-entry lookup. After artifact.Put succeeds, the local entry serves requests that were already waiting. New requests resolve through completed artifact records in the artifact DB.

API Specification

llard-Service-Specification

llard Module Specification

llard has one HTTP entrypoint and three internal submodules. The entrypoint wires HTTP to the service modules. The submodules own build coordination, completed artifact records, and artifact upload.

cmd/llard
  internal/build
  internal/artifact
  internal/upload

cmd/llard

cmd/llard is HTTP glue. It starts Gin, registers GET /v1/artifacts/<module>@<version>, parses module name, version, and matrix query, calls internal/build, and writes the command JSON line response.

cmd/llard adapts raw build output into public info command lines. Response encoding stays at the HTTP boundary.

internal/build

internal/build owns llard-local build coordination:

• completed artifact lookup before local entry lookup;
• in-memory artifact identity -> build entry coordination;
• waiting for an existing local build;
• starting a new local build;
• raw info fanout;
• invoking the LLAR build path;
• calling upload;
• calling artifact.Store.Put;
• using the artifact returned by Put;
• removing local entries after terminal completion.

Public interface:

package build

type Target struct {
	Module  string
	Version string
}

type Matrix struct {
	Require map[string]string `json:"require"`
	Options map[string]string `json:"options,omitempty"`
}

type Request struct {
	Target    Target
	MatrixStr string
	Matrix    Matrix
}

type TargetArtifact struct {
	Target   string
	Artifact artifact.Artifact
}

type Result struct {
	Artifacts []TargetArtifact
}

type Builds struct {
	// internal state
}

func New(opts Options) *Builds
func (b *Builds) Build(ctx context.Context, req Request, info io.Writer) (Result, error)

info is optional:

nil
  non-streaming request; wait for final Result/error only

non-nil
  streaming request; write raw info text while waiting

Build never writes terminal artifact or error messages. The caller writes the terminal artifact set from Result or the terminal error from error.

Builder Cloud Mode

Dependency handling belongs inside the llard-side LLAR build path. When the llard runs llar make, Builder resolves dependencies, checks local cache, checks completed artifact records, downloads dependency artifacts when available, and builds missing dependencies locally when needed.

This is not a global dependency build lock. The same dependency may be built more than once when two root builds need it at the same time, either on the same llard instance or on different llard instances. After a local dependency build finishes, Builder uploads the candidate artifact and calls artifact.Store.Put. Builder must use the artifact returned by Put; if another build already stored an artifact for the same dependency key, the returned stored artifact replaces the local candidate for the rest of the build.

internal/artifact

internal/artifact owns completed artifact records. Upload, download, unpack, checksum calculation, local build coordination, and HTTP handling are owned by other modules.

package artifact

type Key struct {
	Module    string
	Version   string
	MatrixStr string
}

type Artifact struct {
	URL      string `json:"url"`
	Type     string `json:"type"`     // tar.gz by default
	Checksum string `json:"checksum"` // sha256
}

type Store interface {
	Get(ctx context.Context, key Key) (Artifact, bool, error)
	Put(ctx context.Context, key Key, artifact Artifact) (Artifact, error)
	Delete(ctx context.Context, key Key) error
}

Artifact table:

Column	Type	Key	Meaning
module	TEXT NOT NULL	Primary key	Module path
version	TEXT NOT NULL	Primary key	Resolved module version
matrix_str	TEXT NOT NULL	Primary key	LLAR matrix string used by cache and install directory layout
url	TEXT NOT NULL		Direct artifact download URL
type	TEXT NOT NULL		Archive type
checksum	TEXT NOT NULL		Artifact checksum
created_at	TIMESTAMP NOT NULL		Artifact record creation time
expires_at	TIMESTAMP NULL		Optional artifact record expiration time

Primary key: module, version, matrix_str.

Put is atomic and canonicalizes the completed artifact:

missing key
  insert artifact and return it

same key already exists
  return the existing stored artifact

database write/read failure
  return error

Callers must use the artifact returned by Put. They must not continue with a locally produced candidate if Put returns an already stored artifact. This is true even when the local candidate has a different checksum. The first stored artifact is the canonical completed artifact for that key.

internal/upload

internal/upload owns artifact byte upload to the configured Artifact Store backend. GHCR is the default backend.

package upload

type Options struct {
	Name  string
	Type  string // tar.gz by default
	Attrs map[string]string
}

type Result struct {
	URL      string
	Size     int64
	Checksum string // sha256
}

type Uploader interface {
	Upload(ctx context.Context, r io.ReadSeeker, opts Options) (Result, error)
}

type GHCRConfig struct {
	Owner string
	Token string
}

func NewGHCR(cfg GHCRConfig) Uploader

Options.Type is the archive type. The default archive type is tar.gz. upload computes checksum and size, uploads bytes, and returns the download URL. Artifact DB writes and build state remain outside the upload module. build writes .llar/metadata.json into the archive, combines upload URL, archive type, and checksum into a candidate artifact.Artifact, then calls artifact.Store.Put.

llar install selects download behavior from the artifact URL. For example, URLs under ghcr.io use GHCR download handling, including the required Authorization header. The protocol does not carry a separate artifact source field.

User Stories

Request Routing

All artifact requests enter through nginx. nginx hashes the request identity and routes requests for the same module, version, and matrix query to the same healthy llard. llard-local build sharing depends on this routing affinity. If the selected llard becomes unhealthy, later requests may reach another llard instance; that llard checks the artifact DB before starting a build.

sequenceDiagram
  participant Client as "llar install"
  participant Nginx as "nginx"
  participant Llard as "llard"

  Client->>Nginx: "GET /v1/artifacts/<module>@<version>?matrix"
  Nginx->>Nginx: "hash request identity"
  Nginx->>Llard: "route to selected healthy llard"

Loading

1. Install Artifacts That Already Exist

The user runs llar install <target>. The client sends one artifact request with the selected matrix. It does not resolve dependency graphs or submit dependency jobs.

After routing, the llard checks completed artifact records first. If the artifact exists, the llard returns an artifact message with Artifact.URL. The client chooses download handling from the URL, downloads from the Artifact Store, verifies checksum, extracts into the LLAR install directory, reads .llar/metadata.json, and writes .cache.json.

sequenceDiagram
  participant Client as "llar install"
  participant Llard as "llard"
  participant Artifact as "Artifact Layer"

  Client->>Llard: "GET /v1/artifacts/<module>@<version>?matrix"
  Llard->>Artifact: "read completed record"
  Artifact-->>Llard: "Artifact"
  Llard-->>Client: "artifact message"
  Client->>Artifact: "download each Artifact.URL"
  Artifact-->>Client: "archive bytes"
  Client->>Client: "install each target and write .cache.json"

Loading

2. Build A Missing Artifact

The client sends the same GET request. The llard checks the artifact DB and misses. If no local build entry exists for the artifact key, the llard creates one and starts a build. The build runs the LLAR build path, uploads the archive to the Artifact Store, and writes the completed artifact record through artifact.Store.Put.

After Put succeeds for the root and collected dependencies, all waiting requests receive artifact messages containing the ordered artifact set. The client then downloads every artifact directly from the Artifact Store.

sequenceDiagram
  participant Client as "llar install"
  participant Llard as "llard"
  participant Artifact as "Artifact Layer"

  Client->>Llard: "GET /v1/artifacts/<module>@<version>?matrix"
  Llard->>Artifact: "read completed record"
  Artifact-->>Llard: "miss"
  Llard->>Llard: "join or create local build entry"
  Llard->>Llard: "run LLAR build path"
  Llard-->>Client: "info messages when verbose"
  Llard->>Artifact: "upload archive and store completed record"
  Artifact-->>Llard: "canonical Artifact"
  Llard-->>Client: "artifact message"
  Client->>Artifact: "download each Artifact.URL"
  Artifact-->>Client: "archive bytes"
  Client->>Client: "install each target and write .cache.json"

Loading

3. Dependencies During llard Build

The client still sends one root artifact request. During that build, llard runs the LLAR build path. The Builder resolves dependencies, reuses dependency artifacts when metadata exists, locally builds missing dependencies when needed, and records every dependency artifact needed by the client.

sequenceDiagram
  participant Client as "llar install"
  participant Llard as "llard"
  participant Artifact as "Artifact Layer"

  Client->>Llard: "GET root artifact"
  Llard->>Llard: "run LLAR build path"
  Llard->>Llard: "resolve dependency graph"
  Llard->>Artifact: "read dependency artifact"
  alt "dependency artifact exists"
    Artifact-->>Llard: "Artifact + archive bytes"
    Llard->>Llard: "write dependency cache"
  else "dependency artifact missing"
    Artifact-->>Llard: "miss"
    Llard->>Llard: "build dependency locally"
    Llard->>Artifact: "upload candidate and store record"
    Artifact-->>Llard: "canonical Artifact"
  end
  Llard->>Llard: "append dependency artifact to result set"

Loading

Duplicate dependency builds are acceptable. artifact.Store.Put chooses the canonical completed artifact. If another build already stored the same dependency, Builder uses the artifact returned by Put, even when its local candidate has a different checksum. The terminal response still returns the canonical dependency artifact so the client can install it locally.

4. Multiple Clients Ask For The Same Root Artifact

Requests for the same request identity arrive at the same healthy llard. The first request creates the local build entry. Later requests for the same artifact key join that entry instead of starting another root build.

Streaming clients receive info output from the local entry. Non-streaming clients wait for the terminal message. The completed artifact record is persisted once through artifact.Store.Put; the terminal response contains the same ordered artifact set for all joined clients.

sequenceDiagram
  participant A as "llar install A"
  participant B as "llar install B"
  participant Llard as "same llard"
  participant Artifact as "Artifact Layer"

  A->>Llard: "GET /v1/artifacts/<module>@<version>?matrix"
  Llard->>Artifact: "read completed record"
  Artifact-->>Llard: "miss"
  Llard->>Llard: "create entry and start build"
  B->>Llard: "GET /v1/artifacts/<module>@<version>?matrix"
  Llard->>Artifact: "read completed record"
  Artifact-->>Llard: "miss"
  Llard->>Llard: "join existing entry"
  Llard->>Artifact: "upload archive and store completed record"
  Artifact-->>Llard: "canonical Artifact"
  Llard-->>A: "artifact message"
  Llard-->>B: "artifact message"

Loading

5. llard Fails During A Build

If a llard dies or is removed by nginx health checks, in-memory build entries and retained info output are lost. Later requests may reach another llard instance. The fallback llard instance checks the artifact DB first. If the artifact exists, it returns an artifact message. If the artifact is still missing, it starts a new build.

Duplicate builds are acceptable in these failure windows. artifact.Store.Put is the consistency boundary that prevents completed artifact records from becoming ambiguous.

sequenceDiagram
  participant Client as "llar install"
  participant Nginx as "nginx"
  participant W1 as "llard A"
  participant W2 as "llard B"
  participant Artifact as "Artifact Layer"

  Client->>Nginx: "GET /v1/artifacts/<module>@<version>?matrix"
  Nginx->>W1: "hash-routed request"
  W1->>Artifact: "read completed record"
  Artifact-->>W1: "miss"
  W1->>W1: "start local build"
  W1--xNginx: "llard removed by health check"
  Client->>Nginx: "retry same GET"
  Nginx->>W2: "fallback llard"
  W2->>Artifact: "read completed record"
  alt "artifact exists"
    Artifact-->>W2: "Artifact"
    W2-->>Client: "artifact message"
  else "artifact missing"
    Artifact-->>W2: "miss"
    W2->>W2: "start new local build"
    W2->>Artifact: "upload archive and store completed record"
    Artifact-->>W2: "canonical Artifact"
    W2-->>Client: "artifact message"
  end

Loading