Skip to content

Conversation

@Wikid82
Copy link
Owner

@Wikid82 Wikid82 commented Jan 30, 2026

Summary

This PR addresses critical CI/CD infrastructure failures that were blocking the release pipeline and E2E test execution. The hotfix resolves four interconnected issues: Docker Compose E2E image references, SQLite driver compatibility, GoReleaser v2 syntax migration, and CGO removal for cross-platform binary compatibility.

🔴 Problems Identified

1. Docker Compose E2E Image Reference Mismatch

Symptom: E2E tests failed in CI with "image not found" errors.
Root Cause: The docker-compose.playwright-ci.yml referenced a hardcoded image tag that didn't match the dynamically built image from CI.

2. SQLite CGO Dependency Blocking Cross-Compilation

Symptom: GoReleaser nightly builds failed on macOS (darwin) and Windows targets.
Root Cause: The SQLite driver (mattn/go-sqlite3) requires CGO, which doesn't work with cross-compilation. Even with Zig toolchain configured, the CGO requirement prevented reliable cross-platform builds.

3. GoReleaser v2 Syntax Incompatibility

Symptom: GoReleaser failed with deprecation warnings and syntax errors.
Root Cause: The .goreleaser.yaml configuration used v1 syntax (format: instead of formats:) which is deprecated in GoReleaser v2.x.

4. CGO Inconsistency Across Platforms

Symptom: Binary incompatibility and runtime crashes on some platforms.
Root Cause: Different build configurations had inconsistent CGO settings, with some enabling CGO and others disabling it.

✅ Solutions Implemented

1. Docker Compose E2E Image Variable

File: .docker/compose/docker-compose.playwright-ci.yml

# Before:
image: charon:e2e-test

# After:
image: ${CHARON_E2E_IMAGE_TAG:-charon:e2e-test}

Benefit: CI can inject the actual built image tag via environment variable, with a sensible default for local development.

2. Pure-Go SQLite Driver Migration

File: backend/go.mod, backend/internal/database/database.go

// Before (CGO required):
import "gorm.io/driver/sqlite"  // Uses mattn/go-sqlite3

// After (Pure-Go, no CGO):
import "github.com/glebarez/sqlite"  // Uses modernc.org/sqlite

Database Configuration Changes:

  • PRAGMA settings now applied via db.Exec() instead of DSN parameters
  • Compatible with pure-Go modernc.org/sqlite driver
  • WAL mode, busy timeout, synchronous mode, and cache size properly configured
  • All existing database tests pass with the new driver

3. GoReleaser v2 Syntax Migration

File: .goreleaser.yaml

# Before (v1 syntax):
version: 1
archives:
  - format: tar.gz
    ...

# After (v2 syntax):
version: 2
archives:
  - formats:
      - tar.gz
    ...

4. CGO Disabled Across All Platforms

File: .goreleaser.yaml

builds:
  - id: linux
    env:
      - CGO_ENABLED=0  # Consistent across all builds
    # ...

  - id: windows
    env:
      - CGO_ENABLED=0  # No more Zig/CGO complexity
    # ...

  - id: darwin
    env:
      - CGO_ENABLED=0  # Pure-Go for macOS
    # ...

Benefit: Eliminates the need for Zig toolchain, C compilers, and platform-specific build configurations. Produces fully static, portable binaries.

📁 Files Changed

Core Changes

File Change Type Description
.goreleaser.yaml Modified v2 syntax migration, CGO=0 for all platforms
backend/go.mod Modified Added github.com/glebarez/sqlite (pure-Go SQLite)
backend/internal/database/database.go Modified Updated driver import and PRAGMA configuration
.docker/compose/docker-compose.playwright-ci.yml Modified Variable image reference for CI compatibility

Workflow Adjustments

File Change Type Description
.github/workflows/e2e-tests.yml Verified Image build and artifact save working correctly
.github/workflows/release-goreleaser.yml Verified Compatible with GoReleaser v2 configuration

Test Verification

File Status Description
backend/internal/database/database_test.go ✅ Passing All database connection tests pass
backend/internal/database/errors_test.go ✅ Passing Corruption detection tests pass
backend/internal/database/settings_query_test.go ✅ Passing Settings query behavior verified

🧪 Testing Verification

Backend Database Tests

cd /workspaces/Charon/backend && go test ./internal/database/... -v
# Result: PASS (all tests passing)

Pure-Go SQLite Functionality Verified

  • ✅ WAL mode enabled and verified
  • ✅ Connection pooling configured
  • ✅ Integrity check (PRAGMA quick_check) working
  • ✅ Corruption detection and logging functional
  • ✅ Settings queries with cache behavior verified

Cross-Compilation Test

CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /tmp/charon-linux ./cmd/api
CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o /tmp/charon-darwin ./cmd/api
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -o /tmp/charon.exe ./cmd/api
# Result: All builds succeed without CGO

⚠️ Breaking Changes

None for End Users

  • No API changes
  • No configuration file changes required
  • No database migration needed (SQLite files are compatible)

For Developers

  • Build Requirement: No longer need CGO, C compiler, or Zig toolchain for local builds
  • Driver Change: glebarez/sqlite replaces mattn/go-sqlite3 - this is transparent but changes import paths

📋 Checklist

Pre-Merge

  • All database tests pass (go test ./internal/database/...)
  • GoReleaser v2 syntax validated
  • Docker Compose E2E configuration uses variable image reference
  • CGO=0 consistent across all GoReleaser build configurations
  • Pure-Go SQLite driver properly integrated
  • PRAGMA settings applied correctly for modernc.org/sqlite compatibility

Post-Merge Verification

  • Nightly build workflow completes successfully
  • E2E tests pass in CI with new image reference pattern
  • Release workflow can produce binaries for all platforms
  • Docker image builds correctly with pure-Go SQLite

🔗 Related Issues

  • Fixes GoReleaser nightly build failures
  • Fixes E2E test container startup failures
  • Addresses cross-compilation blockers for beta release

📚 Technical References


Impact: This hotfix unblocks the beta release pipeline by ensuring all CI workflows can execute successfully without CGO dependencies or image reference mismatches.

Wikid82 and others added 5 commits January 29, 2026 22:37
chore(docker): migrate from Alpine to Debian Trixie base image
Propagate changes from development into feature/beta-release
- Updated Docker Compose files to use digest-pinned images for CI contexts.
- Enhanced Dockerfile to pin Go tool installations and verify external downloads with SHA256 checksums.
- Added Renovate configuration for tracking Go tool versions and digest updates.
- Introduced a new design document outlining the architecture and data flow for dependency tracking.
- Created tasks and requirements documentation to ensure compliance with the new digest pinning policy.
- Updated security documentation to reflect the new digest pinning policy and exceptions.
@Wikid82 Wikid82 changed the title Hotfix: Caddyfile Import Hotfix: CI Jan 30, 2026
@Wikid82 Wikid82 marked this pull request as ready for review January 30, 2026 12:31
Copilot AI review requested due to automatic review settings January 30, 2026 12:31

This comment was marked as resolved.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 30, 2026

⚠️ Supply Chain Verification Results

⚠️ WARNING

📦 SBOM Summary

  • Components: 838

🔍 Vulnerability Scan

Severity Count
🔴 Critical 0
🟠 High 7
🟡 Medium 20
🟢 Low 2
Total 409

📎 Artifacts

  • SBOM (CycloneDX JSON) and Grype results available in workflow artifacts

Generated by Supply Chain Verification workflow • View Details

@github-actions
Copy link
Contributor

github-actions bot commented Jan 30, 2026

✅ E2E Test Results: PASSED

All E2E tests passed!

Metric Result
Browser Chromium
Shards 4
Status PASSED

Per-Shard HTML Reports (easier to debug):

  • playwright-report-shard-1 through playwright-report-shard-4

📊 View workflow run & download reports


🤖 This comment was automatically generated by the E2E Tests workflow.

Fixes nightly build failures caused by:

GoReleaser v2 requiring version 2 config syntax
Zig cross-compilation failing for macOS CGO targets
SQLite Driver Migration:

Replace gorm.io/driver/sqlite with github.com/glebarez/sqlite (pure-Go)
Execute PRAGMA statements via SQL instead of DSN parameters
All platforms now build with CGO_ENABLED=0
GoReleaser v2 Migration:

Update version: 1 → version: 2
snapshot.name_template → version_template
archives.format → formats (array syntax)
archives.builds → ids
nfpms.builds → ids
Remove Zig cross-compilation environment
Also fixes Docker Compose E2E image reference:

Use CHARON_E2E_IMAGE_TAG instead of bare digest
Add fallback default for local development
All database tests pass with the pure-Go SQLite driver.
@codecov
Copy link

codecov bot commented Jan 30, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Adds automated workflow that creates a PR from nightly → main every
Monday at 9:00 AM UTC for scheduled release promotion.

Features:

Pre-flight health check verifies critical workflows are passing
Skips PR creation if nightly has no new commits
Detects existing PRs and adds comments instead of duplicates
Labels PRs with 'automated' and 'weekly-promotion'
Creates GitHub issue on failure for visibility
Manual trigger via workflow_dispatch with reason input
NO auto-merge - requires human review and approval
This gives early-week visibility into nightly changes and prevents
Friday surprises from untested code reaching main.
@Wikid82 Wikid82 changed the base branch from development to main January 30, 2026 14:47
The "Save Docker Image as Artifact" and "Upload Image Artifact" steps
were running even when skip_build=true, causing CI failures on Renovate
dependency update PRs.

Add skip_build check to artifact saving step condition
Add skip_build check to artifact upload step condition
Aligns artifact steps with existing build skip logic
- Add TestConnect_PRAGMAExecutionAfterClose to verify all PRAGMA settings
- Add TestConnect_JournalModeVerificationFailure for verification path
- Add TestConnect_IntegrityCheckWithNonOkResult for corruption detection branch
- Addresses Codecov patch coverage requirements for database.go
@Wikid82 Wikid82 changed the base branch from main to development January 30, 2026 15:20
@Wikid82 Wikid82 merged commit f1703ef into development Jan 30, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants