Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 12 additions & 111 deletions .circleci/README.md
Original file line number Diff line number Diff line change
@@ -1,130 +1,31 @@
# CircleCI

## CircleCI Status
CircleCI is kept for platform jobs that do not need to run on every local change.

| Component | Status | Details |
|-----------|--------|---------|
| Config validation | ✓ | YAML valid, orbs resolved |
| Multi-platform builds | ✓ | Linux CUDA, macOS Metal, Windows |
| Python matrix | ✓ | 3.9-3.12 testing |
| GPU resource allocation | ✓ | CUDA architectures 75;80;86 |
| Container builds | ✓ | CPU/GPU Docker images |
| Security scanning | ✓ | Trivy vulnerability checks |
| Benchmark execution | ✓ | Metal shim performance tests |
| Release automation | ✓ | Release-please integration |
The main repo checks live in GitHub Actions. CircleCI should stay quiet unless a job really needs it.

## Workflows
## Required Secrets

### ci
Triggers: push/PR to main
- Code quality
- Python 3.9-3.12 matrix
- Linux CUDA/macOS Metal/Windows builds
- Coverage + security scans
- Docker CPU/GPU builds
- Release automation

### nightly
Triggers: daily 02:00 UTC
- Full test suite
- Extended benchmarks
- Security scans

## Environment Variables

Required environment variables in CircleCI project settings:
Set these in the CircleCI project if Docker or coverage jobs are enabled:

```bash
GITHUB_TOKEN=<github_personal_access_token>
DOCKERHUB_USERNAME=<your_dockerhub_username>
DOCKERHUB_PASSWORD=<your_dockerhub_password_or_token>
CODECOV_TOKEN=<codecov_project_token>
DOCKERHUB_USERNAME=<dockerhub-user>
DOCKERHUB_PASSWORD=<dockerhub-token>
CODECOV_TOKEN=<codecov-token>
```

### Docker Hub Setup

1. **Create Docker Hub account** at https://hub.docker.com
2. **Generate access token** (recommended over password):
- Go to Account Settings → Security → Access Tokens
- Create new token with Read/Write permissions
3. **Add to CircleCI**:
- Project Settings → Environment Variables
- Add `DOCKERHUB_USERNAME` with your Docker Hub username
- Add `DOCKERHUB_PASSWORD` with your access token
4. **Update image namespace**:
- Replace `yourusername` in the config with your actual Docker Hub username
- Or set `DOCKERHUB_USERNAME` environment variable

## Executors
## Validate The Config

| Executor | Image | Resources |
|----------|-------|-----------|
| ubuntu-cuda | nvidia/cuda:13.1.1-devel-ubuntu22.04 | gpu.nvidia.small |
| ubuntu-cpu | ubuntu:24.04 | large |
| macos-metal | Xcode 15.0 | macos.m1.medium.gen1 |
| windows-cuda | Windows Server 2022 | windows.medium |

## Validation

Validate the CircleCI configuration:
Install the CircleCI CLI, then run:

```bash
# Install CircleCI CLI (if not installed)
curl -fLSs https://raw.githubusercontent.com/CircleCI-Public/circleci-cli/master/install.sh | bash

# Validate configuration
circleci config validate .circleci/config.yml
```

## Dependencies

```
code-quality
├── python-tests
├── build-linux-cuda
├── build-linux-cpu
├── build-macos-metal
└── build-windows-cuda
├── code-coverage
└── benchmark
├── docker-build
└── security-scan
└── release
```

## Codecov Integration

Both CI systems use identical Codecov v5 setup:

| CI System | Method | Token |
|-----------|--------|-------|
| GitHub Actions | `codecov-action@v5` | `${{ secrets.CODECOV_TOKEN }}` |
| CircleCI | `npx codecov@5` | `$CODECOV_TOKEN` |

## Artifacts

- Test results (JUnit)
- Coverage reports (Codecov)
- Build binaries
- Security scans (SARIF)
- Benchmarks

## Customization

### Python Versions
```yaml
python-tests:
matrix:
parameters:
python-version: ["3.9", "3.10", "3.11", "3.12"]
```
The local helper does the same check when the CLI is installed:

### CUDA Architectures
```bash
-DCMAKE_CUDA_ARCHITECTURES=75;80;86
bash scripts/validate-circleci.sh
```

### New Platforms
1. Add executor
2. Create build job
3. Update workflow
If the CLI is missing locally, the helper prints a warning and exits cleanly. In CI, missing validation tooling should fail.
38 changes: 19 additions & 19 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
version: 2.1

# ────────────────────────────────────────────────────────────────────────────────
# CircleCI Configuration for Hybrid Compute
# CircleCI configuration for Thread
# ────────────────────────────────────────────────────────────────────────────────

orbs:
Expand Down Expand Up @@ -208,7 +208,7 @@ commands:
# Install required tools
choco install cmake --installargs 'ADD_CMAKE_TO_PATH=System' -y --no-progress;
choco install opencv --version=4.5.5 -y --no-progress;
choco install python --version=3.10.0 -y --no-progress;
choco install python --version=3.12.0 -y --no-progress;
refreshenv;

# Install Python packages
Expand Down Expand Up @@ -801,9 +801,9 @@ jobs:
shell: bash
command: |
# Use DOCKERHUB_USERNAME or fallback to a default namespace
DOCKER_NAMESPACE="${DOCKERHUB_USERNAME:-harpertoken}"
TAG="${DOCKER_NAMESPACE}/hybrid-compute-gpu:${CIRCLE_SHA1}"
LATEST_TAG="${DOCKER_NAMESPACE}/hybrid-compute-gpu:latest"
DOCKER_NAMESPACE="${DOCKERHUB_USERNAME:?DOCKERHUB_USERNAME is required}"
TAG="${DOCKER_NAMESPACE}/thread-gpu:${CIRCLE_SHA1}"
LATEST_TAG="${DOCKER_NAMESPACE}/thread-gpu:latest"

docker build \
-f Dockerfile.cuda \
Expand All @@ -825,8 +825,8 @@ jobs:
name: Test CUDA container
shell: bash
command: |
DOCKER_NAMESPACE="${DOCKERHUB_USERNAME:-harpertoken}"
TAG="${DOCKER_NAMESPACE}/hybrid-compute-gpu:${CIRCLE_SHA1}"
DOCKER_NAMESPACE="${DOCKERHUB_USERNAME:?DOCKERHUB_USERNAME is required}"
TAG="${DOCKER_NAMESPACE}/thread-gpu:${CIRCLE_SHA1}"
echo "Testing CUDA container structure..."
docker run --rm --entrypoint="" $TAG ls -la /app/
echo "Testing CUDA upscale binary exists and is executable..."
Expand All @@ -843,9 +843,9 @@ jobs:
shell: bash
command: |
# Use DOCKERHUB_USERNAME or fallback to a default namespace
DOCKER_NAMESPACE="${DOCKERHUB_USERNAME:-harpertoken}"
TAG="${DOCKER_NAMESPACE}/hybrid-compute-cpu:${CIRCLE_SHA1}"
LATEST_TAG="${DOCKER_NAMESPACE}/hybrid-compute-cpu:latest"
DOCKER_NAMESPACE="${DOCKERHUB_USERNAME:?DOCKERHUB_USERNAME is required}"
TAG="${DOCKER_NAMESPACE}/thread-cpu:${CIRCLE_SHA1}"
LATEST_TAG="${DOCKER_NAMESPACE}/thread-cpu:latest"

docker build \
-f Dockerfile \
Expand All @@ -867,8 +867,8 @@ jobs:
name: Test CPU container
shell: bash
command: |
DOCKER_NAMESPACE="${DOCKERHUB_USERNAME:-harpertoken}"
TAG="${DOCKER_NAMESPACE}/hybrid-compute-cpu:${CIRCLE_SHA1}"
DOCKER_NAMESPACE="${DOCKERHUB_USERNAME:?DOCKERHUB_USERNAME is required}"
TAG="${DOCKER_NAMESPACE}/thread-cpu:${CIRCLE_SHA1}"
echo "Testing CPU container structure..."
docker run --rm --entrypoint="" $TAG ls -la /app/
echo "Testing CPU preprocess binary exists and is executable..."
Expand Down Expand Up @@ -902,11 +902,11 @@ jobs:
name: Scan CUDA image
shell: bash
command: |
DOCKER_NAMESPACE="${DOCKERHUB_USERNAME:-harpertoken}"
if docker image inspect "${DOCKER_NAMESPACE}/hybrid-compute-gpu:${CIRCLE_SHA1}" \
DOCKER_NAMESPACE="${DOCKERHUB_USERNAME:?DOCKERHUB_USERNAME is required}"
if docker image inspect "${DOCKER_NAMESPACE}/thread-gpu:${CIRCLE_SHA1}" \
>/dev/null 2>&1; then
trivy image --format sarif --output cuda-scan.sarif \
"${DOCKER_NAMESPACE}/hybrid-compute-gpu:${CIRCLE_SHA1}"
"${DOCKER_NAMESPACE}/thread-gpu:${CIRCLE_SHA1}"
else
echo "CUDA image not found, skipping scan"
echo '{"runs": [{"results": []}]}' > cuda-scan.sarif
Expand All @@ -916,11 +916,11 @@ jobs:
name: Scan CPU image
shell: bash
command: |
DOCKER_NAMESPACE="${DOCKERHUB_USERNAME:-harpertoken}"
if docker image inspect "${DOCKER_NAMESPACE}/hybrid-compute-cpu:${CIRCLE_SHA1}" \
DOCKER_NAMESPACE="${DOCKERHUB_USERNAME:?DOCKERHUB_USERNAME is required}"
if docker image inspect "${DOCKER_NAMESPACE}/thread-cpu:${CIRCLE_SHA1}" \
>/dev/null 2>&1; then
trivy image --format sarif --output cpu-scan.sarif \
"${DOCKER_NAMESPACE}/hybrid-compute-cpu:${CIRCLE_SHA1}"
"${DOCKER_NAMESPACE}/thread-cpu:${CIRCLE_SHA1}"
else
echo "CPU image not found, skipping scan"
echo '{"runs": [{"results": []}]}' > cpu-scan.sarif
Expand Down Expand Up @@ -951,7 +951,7 @@ workflows:
- python-tests:
matrix:
parameters:
python-version: ["3.9", "3.10", "3.11", "3.12"]
python-version: ["3.10", "3.11", "3.12"]
requires:
- code-quality

Expand Down
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# This file defines the code owners for the hybrid-compute project
# This file defines the code owners for the Thread project
# Code owners are automatically requested for review when a pull request changes files they own

# Global owners
Expand Down
Binary file added .github/thread-flow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 3 additions & 5 deletions .github/workflows/auto-maintenance.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
---
name: Automated Maintenance
name: Maintenance

on:
schedule:
- cron: '0 4 * * *' # Daily at 4 AM
workflow_dispatch:

jobs:
Expand Down Expand Up @@ -50,7 +48,7 @@ jobs:
echo "No changes to commit"
echo "changed=false" >> "$GITHUB_OUTPUT"
else
git commit -m "chore(maintenance): automated update $(date +%Y%m%d)"
git commit -m "chore(maintenance): update timestamp $(date +%Y%m%d)"
echo "changed=true" >> "$GITHUB_OUTPUT"
fi

Expand All @@ -65,7 +63,7 @@ jobs:
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
pr_title="chore(maintenance): automated update"
pr_title="chore(maintenance): update timestamp"
pr_body=$'This PR updates `.github/MAINTENANCE.yml`'
pr_body+=$' with the latest maintenance timestamp.\n\n'
pr_body+=$'It is opened by the scheduled maintenance workflow'
Expand Down
29 changes: 13 additions & 16 deletions .github/workflows/auto-project-assign.yml
Original file line number Diff line number Diff line change
@@ -1,31 +1,28 @@
---
name: Auto Assign to Project
name: Project Assignment

on:
pull_request:
types: [opened, reopened]
issues:
types: [opened, reopened]
workflow_dispatch:
inputs:
item_url:
description: 'Issue or pull request URL to add to the project'
required: true
type: string

jobs:
assign-to-project:
assign:
runs-on: ubuntu-latest
permissions:
contents: read
issues: write
pull-requests: write

steps:
- name: Add to Project
if: github.event_name == 'pull_request' || github.event_name == 'issues'
uses: actions/add-to-project@v2.0.0
with:
project-url: https://github.com/users/bniladridas/projects/14
github-token: ${{ secrets.PAT_TOKEN }}
env:
GH_TOKEN: ${{ secrets.PAT_TOKEN }}
ITEM_URL: ${{ inputs.item_url }}
run: gh project item-add 14 --owner bniladridas --url "$ITEM_URL"

- name: Log Assignment
if: github.event_name == 'pull_request' || github.event_name == 'issues'
run: |
item_num="${{ github.event.pull_request.number || github.event.issue.number }}"
echo "[INFO] Added #$item_num to Forge project"
echo "[INFO] Manual project assignment requested for ${{ inputs.item_url }}"
echo "[INFO] Project: https://github.com/users/bniladridas/projects/14"
Loading
Loading