Skip to content

Updated developer guide #458

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sircthulhu merged 4 commits into from
Mar 25, 2026
Merged

Updated developer guide #458

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3b8ab6c
Updated developer guide
myasnikovdaniil Mar 20, 2026
8585a24
Merge branch 'main' into developer-guide
myasnikovdaniil Mar 20, 2026
62c0272
Added docs about ocirepos and migrations
myasnikovdaniil Mar 24, 2026
ebaa512
Fixed PackageSource dependencies overview
myasnikovdaniil Mar 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
141 changes: 112 additions & 29 deletions content/en/docs/v1/development.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,46 +1,121 @@
---
linkTitle: Developer Guide
title: Cozystack Internals and Developer Guides
title: Cozystack Internals and Developer Guide
description: Cozystack Internals and Development
weight: 100
aliases:
- /docs/v1/development/development
---

## How it works

Cozystack is an operator-driven platform. The bootstrap and ongoing management are
handled by a set of controllers that run inside the cluster. The high-level flow is:

1. **Installer chart** (`packages/core/installer`) is applied via `helm install`.
It deploys the `cozystack-operator` Deployment into the `cozy-system` namespace.

2. **cozystack-operator** starts and performs one-time bootstrap:
- Installs Cozystack CRDs (`Package`, `PackageSource`) from embedded manifests
(`internal/crdinstall`).
- Installs Flux components (source-controller, helm-controller,
source-watcher) from embedded manifests (`internal/fluxinstall`).
- Creates the **initial OCIRepository** (`cozystack-platform`) from the
`platformSourceUrl` and `platformSourceRef` values configured in the installer.
- Creates a `PackageSource` that references the initial OCIRepository.

3. **Reconciliation loop** takes over. The operator watches `PackageSource` and
`Package` CRDs and translates them into Flux `HelmRelease` objects. Flux
then installs and manages the actual Helm charts.

4. **Platform chart** (`packages/core/platform`) is deployed as a regular
Package. It reads the cluster configuration from the
`cozystack.cozystack-platform`
[Package]({{% ref "/docs/v1/operations/configuration/platform-package" %}})
resource and templates bundle manifests that define which system components
should be installed.

The platform chart also creates the **secondary OCIRepository** (`cozystack-packages`)
by copying the spec from the initial OCIRepository. All PackageSources reference
this secondary repository. During upgrades, the platform chart runs migrations
as `pre-upgrade` hooks before creating or updating component HelmReleases.

5. **FluxCD** is the execution engine — it reconciles `HelmRelease` objects
created by the operator, pulling chart artifacts from `ExternalArtifact`
resources and applying them to the cluster.

For the full reconciliation chain (PackageSource → ArtifactGenerator → ExternalArtifact → Package → HelmRelease → Pods), dependency resolution, update and rollback flows, and the cozypkg CLI, see [Key Concepts]({{% ref "/docs/v1/guides/concepts" %}}).

### OCIRepositories and Migration Flow

Cozystack uses two OCIRepository resources to manage platform updates:

| OCIRepository | Created By | References |
|---|---|---|
| `cozystack-platform` | cozystack-operator | Configured via installer values (`platformSourceUrl`, `platformSourceRef`) |
| `cozystack-packages` | Platform chart (`repository.yaml`) | Copies spec from `cozystack-platform` |

All PackageSources in `packages/core/platform/sources/` reference `cozystack-packages`.

#### Migration Execution

Migrations run as Helm `pre-upgrade` hooks in the platform chart:

```yaml
# packages/core/platform/templates/migration-hook.yaml
metadata:
name: cozystack-migration-hook
annotations:
helm.sh/hook: pre-upgrade,pre-install
Comment on lines +62 to +69
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Description: Verify the actual hook types used in migration templates

# Check the platform chart migration hook configuration
fd -e yaml -e tpl . packages/core/platform/templates/ --exec grep -l "helm.sh/hook" | xargs grep -A2 -B2 "helm.sh/hook"

Repository: cozystack/website

Length of output: 183

🏁 Script executed:

#!/bin/bash
# Find concepts.md in the docs directory
fd -n "concepts.md" -t f .

Repository: cozystack/website

Length of output: 291

🏁 Script executed:

#!/bin/bash
# Find concepts.md in the docs directory
fd "concepts.md" .

Repository: cozystack/website

Length of output: 140

🏁 Script executed:

#!/bin/bash
# Read the relevant sections from both files
echo "=== development.md lines 60-75 ==="
sed -n '60,75p' ./content/en/docs/v1/development.md

echo -e "\n=== concepts.md migration-related content ==="
grep -n -A5 -B5 "helm.sh/hook\|pre-upgrade\|pre-install" ./content/en/docs/v1/guides/concepts.md | head -50

Repository: cozystack/website

Length of output: 1542

Inconsistent hook types between description and YAML example.

Line 62 states migrations run as pre-upgrade hooks, but the YAML example at line 69 shows helm.sh/hook: pre-upgrade,pre-install. This conflicts with concepts.md:125, which documents only pre-upgrade hooks for migrations.

Either update the YAML example to show only pre-upgrade, or clarify in the text why migrations need to run on both pre-upgrade and pre-install hooks.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@content/en/docs/v1/development.md` around lines 62 - 69, The documentation
text says migrations run as pre-upgrade hooks but the YAML example in
packages/core/platform/templates/migration-hook.yaml sets helm.sh/hook:
pre-upgrade,pre-install which is inconsistent; either change the YAML example to
only include pre-upgrade or update the surrounding text to explain why
migrations are also run on pre-install (e.g., to initialize a fresh install),
and ensure the helm.sh/hook annotation and the prose both reflect the same
behavior for migrations in migration-hook.yaml.

helm.sh/hook-weight: "1"
```

## How it works
The migration container reads the current version from the `cozystack-version` ConfigMap and executes migration scripts sequentially from `CURRENT_VERSION` to `TARGET_VERSION - 1`. Each migration updates the ConfigMap on success, ensuring migrations are idempotent and can resume after failures.

In short, cozystack is a seed container that bootstraps the entire platform. It consists of:
#### Why Two Repositories?

- **installer.sh** script: Contains minimal business logic, performs migrations, installs
FluxCD, and prepares Kubernetes to run the platform chart.
The separation ensures that:

- **platform chart**: A Helm chart that bootstraps the remaining configuration. It reads
the state from the Kubernetes cluster using Helm lookup functions and templates
all necessary manifests. The `installer.sh` script continuously runs the platform chart
to ensure changes in the cluster are properly reconciled.
1. The initial OCIRepository is managed by the operator (via installer values).
2. All PackageSources have a consistent reference (`cozystack-packages`) rather than pointing to the operator-managed source directly.
3. The platform chart can run migrations before creating the secondary OCIRepository, guaranteeing migrations execute before component updates.

- **HTTP server**: Serves assets (Helm charts and Grafana dashboards) built from Cozystack repository.
### Key binaries

- **FluxCD**: The main engine that applies all necessary Helm charts from the HTTP server
to Kubernetes according to the configuration applied by the platform chart.
| Binary | Source | Role |
|---|---|---|
| **cozystack-operator** | `cmd/cozystack-operator` | Bootstrap (CRDs, Flux, platform source), `PackageSource` and `Package` reconciliation, `cozystack-values` secret replication. |
| **cozystack-controller** | `cmd/cozystack-controller` | Workload and ApplicationDefinition reconciliation, dashboard management. |
| **cozystack-api** | `cmd/cozystack-api` | Kubernetes API aggregation layer for `apps.cozystack.io` and `core.cozystack.io` API groups. |
| **cozypkg** | `cmd/cozypkg` | CLI tool for managing packages — dependency visualization, interactive installation, deletion. |

## Repository Structure

The main structure of the [cozystack](https://github.com/cozystack/cozystack) repository is:

```shell
.
├── packages # Main directory for cozystack packages
│ ├── core # Core platform logic charts
│ ├── system # System charts used to configure the system
│ ├── apps # User-facing charts shown in the dashboard catalog
│ └── extra # Tenant-specific applications that can be deployed as tenant options
├── api # Go types for Cozystack CRDs (Package, PackageSource, etc.)
├── cmd # Entry points for all binaries
│ ├── cozystack-operator # Main platform operator
│ ├── cozystack-controller # Workload and application controllers
│ ├── cozystack-api # Aggregated API server
│ └── cozypkg # Package management CLI
├── internal # Controller and reconciler implementations
│ ├── operator # PackageSource and Package reconcilers
│ ├── controller # Workload, ApplicationDefinition controllers
│ ├── fluxinstall # Embedded Flux manifests and installer
│ ├── crdinstall # Embedded CRD manifests and installer
│ └── cozyvaluesreplicator # Secret replication logic
├── packages # Helm charts organized by layer
│ ├── core # Bootstrap and platform configuration
│ ├── system # Infrastructure operators and upstream charts
│ ├── apps # User-facing application charts
│ └── extra # Tenant-specific application charts
├── pkg # Shared Go libraries
├── dashboards # Grafana dashboards
├── hack # Helper scripts for local development
└── scripts # Scripts mainly used by the cozystack container
└── migrations # Migrations between versions
└── docs # Changelogs and release notes
```

Development can be done locally by modifying and updating files in this repository.
Expand All @@ -49,23 +124,31 @@ Development can be done locally by modifying and updating files in this reposito

### [core](https://github.com/cozystack/cozystack/tree/main/packages/core)

Core packages are used to bootstrap Cozystack and its configuration itself.

It consists of two packages:
Core packages handle bootstrap and platform-level configuration.

#### installer

Provides everything needed for the initial bootstrap of cozystack: the installer.sh script and an HTTP server with assets built from this repository.
A Helm chart that deploys the `cozystack-operator` Deployment. It creates the
`cozy-system` namespace, a ServiceAccount with cluster-admin privileges, and the
operator Deployment with flags that trigger CRD and Flux installation on startup.
The operator image and platform source URL are injected at build time.

#### platform

Reads the configuration from the cluster, templates the manifests, and applies them
back to the cluster.
A Helm chart deployed as a regular `Package` (not applied directly). It reads the
cluster configuration from the `cozystack.cozystack-platform`
[Package]({{% ref "/docs/v1/operations/configuration/platform-package" %}})
resource and templates manifests according to the specified
[variant]({{% ref "/docs/v1/operations/configuration/variants" %}}) and
component settings, defining which system components should be installed.

#### flux-aio

Flux components packaged for deployment by the operator.

#### talos

It reads the configuration from the `cozystack.cozystack-platform` [Package]({{% ref "/docs/v1/operations/configuration/platform-package" %}}) resource, and templates
manifests according to the specified options. The Package resource
specifies the [variant]({{% ref "/docs/v1/operations/configuration/variants" %}}) and component settings, detailing which system components should be
installed in the cluster.
Talos OS configuration assets.
Comment on lines +149 to +151
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Capitalize product name in heading

At Line 109, consider #### Talos (capitalized) for consistency with the product name and surrounding docs.

🧰 Tools
🪛 LanguageTool

[grammar] ~109-~109: Ensure spelling is correct
Context: ...d for deployment by the operator. #### talos Talos OS configuration assets. {{% aler...

(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@content/en/docs/v1/development.md` around lines 109 - 111, The heading "####
talos" should be capitalized to match product naming; update the heading string
used in the document (the line containing "#### talos") to "#### Talos" so it
reads with a capital T and remains consistent with surrounding docs.


{{% alert color="info" %}}
Core packages do not use Helm to apply manifests; they are intended to be used only as `helm template . | kubectl apply -f -`.
Expand Down
52 changes: 52 additions & 0 deletions content/en/docs/v1/guides/concepts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,46 @@ Read more: [Variants]({{% ref "/docs/v1/operations/configuration/variants" %}}).

Together, they form a declarative pipeline: external charts flow through Flux sources and artifact generators into ready-to-install Helm charts, which Packages then instantiate as running workloads.

### OCIRepositories: Platform and Packages

Cozystack uses two OCIRepository resources to manage the update flow and ensure migrations run before any component upgrades.

#### Initial OCIRepository (`cozystack-platform`)

Created by the cozystack-operator during bootstrap. The operator receives a platform source URL (e.g., `oci://ghcr.io/cozystack/cozystack/cozystack-packages`) and creates an OCIRepository named `cozystack-platform`. This repository points to the platform chart artifact, is configured via installer values (`platformSourceUrl`, `platformSourceRef`), and provides the platform chart that will create migrations and the secondary OCIRepository.

#### Secondary OCIRepository (`cozystack-packages`)

Created by the platform Helm chart (`packages/core/platform/templates/repository.yaml`). It copies the spec from `cozystack-platform` and creates a new OCIRepository named `cozystack-packages`. This repository is referenced by all PackageSources (networking, monitoring, postgres-operator, etc.), contains all system and application charts, and decouples the platform source from component PackageSources.

#### Migration Ordering

The two-repository design ensures that system migrations execute before any component updates:

```mermaid
flowchart TD
A["Installer Chart (helm install)"]
B["cozystack-operator starts"]
C["Initial OCIRepository (cozystack-platform)<br/>Created by operator"]
D["Platform Chart from initial OCIRepository"]
E["Pre-upgrade Hooks: Run migrations sequentially<br/>Update cozystack-version ConfigMap"]
F["Secondary OCIRepository (cozystack-packages)<br/>Created by platform chart"]
G["PackageSources reference cozystack-packages"]
H["System Components HelmReleases deploy"]

A --> B --> C --> D --> E --> F --> G --> H
```

When a new platform version is released and the cluster is upgraded:

1. The initial OCIRepository (`cozystack-platform`) provides the new platform chart.
2. During `helm upgrade`, the platform chart's `pre-upgrade` hooks execute migrations sequentially (from current to target version).
3. Each migration script performs necessary transformations and updates the `cozystack-version` ConfigMap.
4. After migrations complete, the platform chart creates or updates the `cozystack-packages` OCIRepository.
5. PackageSources reference `cozystack-packages` and trigger reconciliation of system components.

This guarantees migrations run before component upgrades, and the migration scripts come from the same chart version being deployed.

### Reconciliation Flow

The full reconciliation chain from an external registry to running Kubernetes resources:
Expand Down Expand Up @@ -136,6 +176,18 @@ flowchart LR

If all dependencies report a `Ready` status, the dependent Package proceeds to create its HelmRelease. Otherwise, the Package remains in a waiting state until the conditions are met.

Dependencies in PackageSource used at two levels:

- **Variant-level** (`spec.variants.dependsOn[]`): references other Package names. The PackageReconciler checks that all dependencies are ready before creating any HelmReleases. This ensures infrastructure packages (e.g., CNI, storage) are fully running before dependent packages attempt installation. The `spec.ignoreDependencies` field on a Package can override this check for specific dependencies.
- **Component-level** (`spec.variants.components.install.dependsOn[]`): translated into `spec.dependsOn[]` field on a HelmRelease resource. These dependencies enforce correct ordering of components during installation of package.

### Namespace and Values Management

When the PackageReconciler creates HelmReleases for a Package, it also:

- **Creates namespaces** declared in component `Install.namespace` fields, setting labels such as `cozystack.io/system=true` and `pod-security.kubernetes.io/enforce=privileged` where needed.
- **Injects cluster-wide configuration** via the `cozystack-values` Secret. The **CozyValuesReplicator** watches this Secret in `cozy-system` and replicates it to every namespace labeled `cozystack.io/system=true`. Each HelmRelease references this Secret through `valuesFrom`, ensuring all components receive consistent platform configuration.

### Update Flow

When a new chart version is pushed to the registry, updates propagate automatically through the reconciliation chain:
Expand Down