Skip to content

chore: update Supabase dependencies and improve e2e test infrastructure #292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

chore: update Supabase dependencies and improve e2e test infrastructure #292

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 57 additions & 9 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,14 +54,62 @@ jobs:
run: pnpm nx affected -t build --configuration=production --parallel --exclude=playground --base="$NX_BASE" --head="$NX_HEAD"


# ────────────────────────────────── 2. DEPLOY PLAYGROUND ───────────────────────────
# ─────────────────────────────────────── 2. EDGE-WORKER E2E ──────────────────────────────────────
edge-worker-e2e:
runs-on: ubuntu-latest
env:
NX_CLOUD_ACCESS_TOKEN: ${{ secrets.NX_CLOUD_ACCESS_TOKEN }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- uses: ./.github/actions/setup

- name: Setup Deno
uses: denoland/setup-deno@v2
with:
deno-version: '1.45.2'

- name: Install sqruff
uses: quarylabs/install-sqruff-cli-action@main

- name: Setup Atlas
uses: ariga/setup-atlas@master
with:
cloud-token: ${{ secrets.ATLAS_CLOUD_TOKEN }}

- name: Set Nx SHAs for affected commands
uses: nrwl/nx-set-shas@v4

- name: Verify NX_BASE and NX_HEAD are set
run: echo "BASE=$NX_BASE HEAD=$NX_HEAD"

- name: Check if edge-worker e2e tests are affected
id: check-affected
run: |
if pnpm nx show projects --affected -t test:e2e --base="$NX_BASE" --head="$NX_HEAD" | grep -q "^edge-worker$"; then
echo "affected=true" >> $GITHUB_OUTPUT
echo "Edge-worker e2e tests are affected by changes"
else
echo "affected=false" >> $GITHUB_OUTPUT
echo "Edge-worker e2e tests are not affected by changes - skipping"
fi

- name: Run edge-worker e2e tests
if: steps.check-affected.outputs.affected == 'true'
run: pnpm nx affected -t test:e2e --parallel --base="$NX_BASE" --head="$NX_HEAD"


# ────────────────────────────────── 3. DEPLOY PLAYGROUND ───────────────────────────
deploy-playground:
needs: build-and-test
if: >-
${{
(github.event_name == 'pull_request') ||
(github.ref == 'refs/heads/main' && github.event_name == 'push')
}}
needs: [build-and-test, edge-worker-e2e]
if: false # Disabled
# if: >-
# ${{
# (github.event_name == 'pull_request') ||
# (github.ref == 'refs/heads/main' && github.event_name == 'push')
# }}
runs-on: ubuntu-latest
environment: ${{ github.event_name == 'pull_request' && 'preview' || 'production' }}
env:
Expand Down Expand Up @@ -95,9 +143,9 @@ jobs:
preview-url: https://pr-${{ github.event.pull_request.number }}--pgflow-demo.netlify.app
production-url: https://playground.pgflow.dev

# ────────────────────────────────── 3. DEPLOY WEBSITE ───────────────────────────
# ────────────────────────────────── 4. DEPLOY WEBSITE ───────────────────────────
deploy-website:
needs: build-and-test
needs: [build-and-test, edge-worker-e2e]
runs-on: ubuntu-latest
environment: ${{ github.event_name == 'pull_request' && 'preview' || 'production' }}
env:
Expand Down
35 changes: 28 additions & 7 deletions pkgs/edge-worker/CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,13 +53,34 @@ The Edge Worker:

## Testing

The package has different test types:

- **Unit tests** (`tests/unit/`) - For testing isolated components
- **Integration tests** (`tests/integration/`) - For testing integration with PostgreSQL
- **E2E tests** (`tests/e2e/`) - For testing end-to-end workflows (not automated for now)

Tests use Deno's built-in testing framework. Database tests require Docker to run a PostgreSQL instance.
The package has three test types with different infrastructure requirements:

- **Unit tests** (`tests/unit/`) - Testing isolated components
- No external dependencies required

- **Integration tests** (`tests/integration/`) - Testing integration with PostgreSQL
- Requires: Supabase database (via `db:ensure`)
- Runs in CI

- **E2E tests** (`tests/e2e/`) - Testing end-to-end workflows with Edge Functions
- Requires: Supabase database + Edge Functions server (via `serve:functions:e2e`)
- Uses `continuous: true` in Nx to keep server running during tests
- Tests worker scaling behavior, CPU limits, and concurrency (20k+ messages)
- Takes 2+ minutes to complete
- Runs in CI as separate parallel job (only when edge-worker is affected)
- Run locally with: `pnpm nx test:e2e edge-worker`

Tests use Deno's built-in testing framework. Database tests require Docker to run a PostgreSQL instance via Supabase CLI.

### E2E Test Architecture

E2E tests verify real Edge Function execution:
1. `sync-e2e-deps` copies built packages (core, dsl, edge-worker) to `supabase/functions/_vendor`
2. `serve:functions:e2e` starts Supabase Functions server in background (continuous task)
3. Tests make HTTP requests to spawn workers and verify behavior
4. Tests in `tests/e2e/`:
- `restarts.test.ts` - Worker auto-restart when CPU clock limit hits
- `performance.test.ts` - Processing thousands of queued messages concurrently

## Usage Examples

Expand Down
48 changes: 29 additions & 19 deletions pkgs/edge-worker/deno.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

37 changes: 28 additions & 9 deletions pkgs/edge-worker/project.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,6 @@
"cache": false,
"inputs": [
"{workspaceRoot}/pkgs/core/supabase/migrations/**/*.sql",
"{projectRoot}/sql/**/*.sql",
"^production"
],
"options": {
Expand All @@ -109,7 +108,6 @@
"mkdir -p supabase/migrations/",
"rm -f supabase/migrations/*.sql",
"cp ../core/supabase/migrations/*.sql supabase/migrations/",
"cp sql/*_*.sql supabase/migrations/",
"supabase db reset"
],
"parallel": false
Expand All @@ -123,7 +121,7 @@
"options": {
"cwd": "pkgs/edge-worker",
"commands": [
"supabase functions serve --env-file supabase/functions/.env"
"supabase functions serve --env-file supabase/functions/.env --no-verify-jwt"
],
"parallel": false
}
Expand All @@ -132,11 +130,7 @@
"executor": "nx:run-commands",
"local": true,
"dependsOn": ["^verify-migrations", "^dump-realtime-schema"],
"inputs": [
"default",
"^production",
"{projectRoot}/scripts/ensure-db"
],
"inputs": ["default", "^production", "{projectRoot}/scripts/ensure-db"],
"options": {
"cwd": "pkgs/edge-worker",
"commands": ["./scripts/ensure-db"],
Expand Down Expand Up @@ -169,9 +163,34 @@
"parallel": false
}
},
"sync-e2e-deps": {
"executor": "nx:run-commands",
"dependsOn": ["^build"],
"local": true,
"inputs": ["^production"],
"options": {
"cwd": "pkgs/edge-worker",
"commands": ["./scripts/sync-e2e-deps.sh"],
"parallel": false
}
},
"serve:functions:e2e": {
"executor": "nx:run-commands",
"continuous": true,
"dependsOn": ["sync-e2e-deps", "supabase:start"],
"local": true,
"cache": false,
"options": {
"cwd": "pkgs/edge-worker",
"commands": [
"supabase functions serve --env-file supabase/functions/.env --import-map supabase/functions/deno.json --no-verify-jwt"
],
"parallel": false
}
},
"test:e2e": {
"executor": "nx:run-commands",
"dependsOn": ["db:ensure", "^build"],
"dependsOn": ["supabase:reset", "serve:functions:e2e"],
"local": true,
"inputs": ["default", "^production"],
"options": {
Expand Down
73 changes: 73 additions & 0 deletions pkgs/edge-worker/scripts/sync-e2e-deps.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
#!/usr/bin/env bash
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
EDGE_WORKER_DIR="$(dirname "$SCRIPT_DIR")"
MONOREPO_ROOT="$(cd "$EDGE_WORKER_DIR/../.." && pwd)"
VENDOR_DIR="$EDGE_WORKER_DIR/supabase/functions/_vendor"

echo "🔄 Syncing edge function dependencies for e2e tests..."

# Clean and create vendor directory
rm -rf "$VENDOR_DIR"
mkdir -p "$VENDOR_DIR/@pgflow"

# Verify builds succeeded
if [ ! -d "$MONOREPO_ROOT/pkgs/core/dist" ]; then
echo "❌ Error: core package build failed - dist directory not found"
exit 1
fi

if [ ! -d "$MONOREPO_ROOT/pkgs/dsl/dist" ]; then
echo "❌ Error: dsl package build failed - dist directory not found"
exit 1
fi

# Copy core package
echo "📋 Copying @pgflow/core..."
mkdir -p "$VENDOR_DIR/@pgflow/core"
cp -r "$MONOREPO_ROOT/pkgs/core/dist/"* "$VENDOR_DIR/@pgflow/core/"
cp "$MONOREPO_ROOT/pkgs/core/package.json" "$VENDOR_DIR/@pgflow/core/"

# Copy dsl package
echo "📋 Copying @pgflow/dsl..."
mkdir -p "$VENDOR_DIR/@pgflow/dsl"
cp -r "$MONOREPO_ROOT/pkgs/dsl/dist/"* "$VENDOR_DIR/@pgflow/dsl/"
cp "$MONOREPO_ROOT/pkgs/dsl/package.json" "$VENDOR_DIR/@pgflow/dsl/"

# Copy edge-worker source (not built) - preserving directory structure
echo "📋 Copying @pgflow/edge-worker..."
mkdir -p "$VENDOR_DIR/@pgflow/edge-worker"
# Copy the entire src directory to maintain relative imports
cp -r "$MONOREPO_ROOT/pkgs/edge-worker/src" "$VENDOR_DIR/@pgflow/edge-worker/"

# Simple fix: replace .js with .ts in imports
find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.ts" -type f -exec sed -i 's/\.js"/\.ts"/g' {} +
find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.ts" -type f -exec sed -i "s/\.js'/\.ts'/g" {} +
Comment on lines +45 to +46
Copy link
Contributor

@graphite-app graphite-app bot Nov 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical: Platform compatibility issue with sed -i

The sed -i command behaves differently on macOS vs Linux. On macOS, sed -i requires a backup extension argument (even if empty), while Linux doesn't. This script will fail on macOS systems.

Fix:

# Use platform-independent approach
find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.ts" -type f -exec sed -i.bak 's/\.js"/\.ts"/g' {} +
find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.ts" -type f -exec sed -i.bak "s/\.js'/\.ts'/g" {} +
find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.bak" -type f -delete

Or use a more portable solution:

find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.ts" -type f | while read -r file; do
  sed 's/\.js"/\.ts"/g; s/\.js'\''/.ts'\'''/g' "$file" > "$file.tmp" && mv "$file.tmp" "$file"
done
Suggested change
find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.ts" -type f -exec sed -i 's/\.js"/\.ts"/g' {} +
find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.ts" -type f -exec sed -i "s/\.js'/\.ts'/g" {} +
find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.ts" -type f -exec sed -i.bak 's/\.js"/\.ts"/g' {} +
find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.ts" -type f -exec sed -i.bak "s/\.js'/\.ts'/g" {} +
find "$VENDOR_DIR/@pgflow/edge-worker" -name "*.bak" -type f -delete

Spotted by Graphite Agent

Fix in Graphite

Is this helpful? React 👍 or 👎 to let us know.


# Create a redirect index.ts at the root that points to src/index.ts
cat > "$VENDOR_DIR/@pgflow/edge-worker/index.ts" << 'EOF'
// Re-export from the src directory to maintain compatibility
export * from './src/index.ts';
EOF

# Create _internal.ts redirect as well since edge-worker exports this path
cat > "$VENDOR_DIR/@pgflow/edge-worker/_internal.ts" << 'EOF'
// Re-export from the src directory to maintain compatibility
export * from './src/_internal.ts';
EOF

# Verify key files exist
if [ ! -f "$VENDOR_DIR/@pgflow/core/index.js" ]; then
echo "⚠️ Warning: @pgflow/core/index.js not found after copy"
fi

if [ ! -f "$VENDOR_DIR/@pgflow/dsl/index.js" ]; then
echo "⚠️ Warning: @pgflow/dsl/index.js not found after copy"
fi

if [ ! -f "$VENDOR_DIR/@pgflow/edge-worker/src/index.ts" ]; then
echo "⚠️ Warning: @pgflow/edge-worker/src/index.ts not found after copy"
fi

echo "✅ Dependencies synced to $VENDOR_DIR"
11 changes: 0 additions & 11 deletions pkgs/edge-worker/sql/990_active_workers.sql
View file
Open in desktop

This file was deleted.

Loading
Loading