Workflow Patterns

Effective SparkEngine Development Workflows

Audience: Programmers | Mixed

Thread Context: N/A (development/process reference)

Platform/Backend Scope: All platforms (commands shown for Linux/bash; see notes for Windows/macOS equivalents)

Overview

Recurring workflows that have proven consistently effective for SparkEngine development tasks. These patterns reduce time spent on overhead and increase the reliability of outputs. Apply them proactively rather than waiting until something goes wrong.

They apply to all SparkEngine development sessions. Most correspond to gaps where the "obvious" approach is slower or less reliable than the pattern described here.

Codebase Exploration — Parallel Explore Agents

When a task touches multiple areas of the codebase (e.g., adding a feature that spans ECS + Graphics + Editor), launch 2-3 Explore agents in parallel rather than sequentially. Each agent gets a specific search focus.

Agent 1: "Search for existing implementations of X in SparkEngine/Source/Engine/ECS/"
Agent 2: "Find all usages of Y in SparkEditor/Source/"
Agent 3: "Identify patterns for Z in Tests/"

Do not use a single Explore agent for a broad multi-area search — it will either miss areas or spend too long on one area.

When to apply:

• Task involves code in 2+ directories
• Scope is uncertain and you need to map the codebase before planning
• Looking for an existing implementation before writing new code (always check first)

Notes:

• Use read-only exploration agents for codebase research; reserve planning agents for architectural design work after exploration.
• 3 agents maximum per parallel batch; quality over quantity.

Shared Build Directory — Serialize With tools/build-lock.sh

When several agents (or scripts) share one build tree, route every configure and build through the lock wrapper instead of an ad-hoc flock:

tools/build-lock.sh -- cmake --build build/linux-gcc-release --target SparkTests -j4
tools/build-lock.sh --timeout 600 -- cmake --preset linux-gcc-release -DBUILD_TESTS=ON
tools/build-lock.sh --status          # who holds it, and is the holder still alive?

The lock defaults to build/.spark-build.lock (--lock / SPARK_BUILD_LOCK override it). What the wrapper guarantees, and why each matters:

• Bounded wait. It gives up after --timeout seconds (default 1800, SPARK_BUILD_LOCK_TIMEOUT) with exit 75 and a holder report, instead of queueing forever.
• Holder record. The holder writes pid, host, start time, cwd, and command into the lock file; waiters print it every --report-every seconds (default 60).
• No inherited descriptors. The command runs with the lock fd closed (and in its own process group via setsid when available), so a daemon or detached helper it spawns cannot keep the lock after the wrapper exits.
• Orphan detection. If the recorded holder is dead but processes still have the lock file open, it says so and lists them. It never kills anything.

Do not guard a shared lock with a detached waiter that matches processes by command line (pgrep -f / pkill -f): the waiters' own command lines match the pattern. On 2026-09-24 such a helper inherited the build lock, waited for "no cmake --build build/... process", and deadlocked every queued build — including itself, since pkill -f on the same pattern also killed the invoking shell. Wait on a PID (kill -0 $PID) and hold locks only in the process that took them. Regression coverage: Tests/Tools/test_build_lock.py (CI job validate-ci-tools).

Fast Local Rebuilds — LTO Off, ccache, mold

linux-gcc-release keeps ENABLE_LTO=ON because shipping builds use it. With LTO every Release object carries -flto=auto: compiling is cheap and the optimizer runs again at link time over all of SparkTests (≈800 objects plus the whole-archive engine library). Every engine change repeats that link. For an iterate-and-test tree, turn it off and add a compiler cache and a faster linker:

sudo apt-get install -y ccache mold
ccache -o max_size=8G -o base_dir="$PWD" -o hash_dir=false \
       -o sloppiness=pch_defines,time_macros,include_file_mtime,include_file_ctime
cmake --preset linux-gcc-release -DBUILD_TESTS=ON -DENABLE_LTO=OFF \
      -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
      -DCMAKE_LINKER_TYPE=MOLD

• LTO off moves optimization back into each compile. Test coverage of the LTO configuration stays with the CI Release lanes.
• ccache turns a snapshot switch or a reconfigure into cache hits. The pch_defines,time_macros sloppiness is required, or ccache refuses to cache files that use a precompiled header. CI sets the same values in CCACHE_SLOPPINESS (.github/workflows/build.yml).
• mold (CMAKE_LINKER_TYPE, CMake 3.29+) links the large test binary in seconds instead of GNU ld's tens of seconds.
• SparkTests precompiled header. Tests/CMakeLists.txt precompiles TestFramework.h plus the standard headers the test files share. It holds no engine or platform header, so test-local #defines before engine includes keep their meaning. Sources compiled into SparkTests from outside Tests/ skip it. Lanes configured with -DCMAKE_DISABLE_PRECOMPILE_HEADERS=ON (the Windows sccache lanes and clang-tidy) build without it and still catch missing #includes.

Changing ENABLE_LTO, the launcher or the linker type changes every compile command, so the first build after the reconfigure is a full rebuild. Run it through tools/build-lock.sh when the tree is shared.

Disk Budget for Build Trees and Worktrees

Cloud sessions get a fixed writable-disk allowance, and parallel agents fill it fast. Measured on 2026-09-25 (Linux GCC):

Item	Size
build/linux-gcc-release (-DENABLE_LTO=OFF, tests and game modules)	≈2 GB
Sanitizer Debug tree (ASan/UBSan, -g1)	≈11 GiB (full -g is larger)
ccache store	its max_size (3–8 GB is plenty)
Detached git worktree of the repo (no build)	the checkout size, before any build

• Build privately only when needed. A private build per agent multiplies the first row. Prefer the shared tree under tools/build-lock.sh, restricted to the targets you need (--target SparkTests).
• Scratch locations. Put scratch worktrees and private builds in the session scratch directory, not in the repo, and git worktree remove them when done.
• Out-of-space errors. When "No space left on device" appears, delete regenerable outputs first: stale private builds, abandoned worktrees, then ccache -C. Delete a scratch folder only after proving its contents are committed.

Exploring the Codebase in 3D (Code City)

tools/architecture-viz/generate_code_city.py renders the tracked source tree as an interactive three.js city: projects are blocks, subsystem directories are districts, and each file is a building whose footprint and height grow with its line count. Selecting a building draws its resolved includes (blue) and includers (orange) and lists the readiness work items whose entry points name it; the color modes cover project, file kind, size, 180-day churn, readiness status and include fan-in.

python3 tools/architecture-viz/generate_code_city.py          # writes build/code-city/index.html
python3 Tests/Tools/test_code_city.py                         # layout, include resolution, embedded data

The page loads three.js from jsDelivr and embeds all data, so it opens straight from disk. The Code City Pages workflow (.github/workflows/code-city-pages.yml) regenerates it on pushes to Working and deploys it to GitHub Pages, which the README links to; Pages must be enabled with Settings → Pages → Source: GitHub Actions. Useful for orientation before a cross-subsystem change: search a file or class, then follow its include arcs.

Documentation Sync After Structural Changes

After any change that adds, renames, or removes public headers, ECS components, systems, editor panels, or tests — run the doc scripts before committing. The fastest reliable option is the master script:

docs/update-all-docs.sh           # runs all six doc scripts in order (~30s)
docs/update-all-docs.sh quick     # skips API docs + flowchart (~10s)

Or run only the two most commonly needed:

docs/generate-api-docs.sh check   # regenerates API pages if headers changed
docs/sync-wiki.sh sync            # updates wiki AUTO: sections (component/system/test counts)

Then git add any changed files under docs/api/ or wiki/ alongside the code change.

When to apply:

• Added or removed a .h file
• Added or removed an ECS component or system
• Added or removed an editor panel
• Added or removed a test

Notes:

• The check-format CI job will not catch stale docs — it only checks C++ formatting. Stale docs surface as unexpected diffs in later sessions.
• docs/generate-api-docs.sh check uses checksums — it is fast when nothing changed.
• docs/sync-wiki.sh sync updates <!-- AUTO:* --> markers in wiki files. Never edit those markers manually.

Pre-Push Checklist Order

Run pre-push checks in this specific order — each step catches a different error class, and the order minimizes wasted time:

# 1. Format first (fastest, most common failure)
find SparkEngine/Source GameModules SparkEditor/Source SparkConsole/src SparkShaderCompiler/src \
  -not -path '*/Metal/*' \( -name '*.h' -o -name '*.hpp' -o -name '*.cpp' \) | \
  xargs clang-format --dry-run --Werror 2>&1

# 2. CMake configure (catches missing headers, broken includes)
cmake --preset linux-gcc-release 2>&1 | tail -20

# 3. Build (catches compile errors)
cmake --build build/linux-gcc-release --parallel $(nproc) 2>&1 | tail -30

# 4. Tests (catches regressions)
ctest --test-dir build/linux-gcc-release --output-on-failure --no-tests=error

# 5. Docs (catches stale auto-generated content)
docs/update-all-docs.sh

Stop at the first failure; fix it before proceeding to the next step.

Notes:

• Step 1 (format) is cheap (~5 seconds). Do it even for tiny changes — MSVC code often reformats differently.
• Step 2 (configure) detects include-path issues before a full compile.
• Step 5 (docs) is only needed when step 1 touches headers or structural files.
• See Build Optimizations for the --parallel $(nproc) flag in step 3.

CMake Preset Over Manual Flags

Always prefer a named preset over manually constructing -B build -DCMAKE_BUILD_TYPE=... flags for local development.

# Preferred
cmake --preset linux-gcc-release

# Only use manual flags when a custom toggle is needed
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=ON -DENABLE_NETWORKING=ON \
  -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++

Notes:

• Presets are defined in CMakePresets.json and are authoritative.
• Available presets include: windows-debug, windows-release, windows-shipping, windows-development, linux-gcc-debug, linux-gcc-release, linux-clang-debug, linux-clang-release, linux-shipping, linux-development, linux-mingw-release, linux-mingw-debug, macos-debug, macos-release, macos-metal, macos-moltenvk, macos-shipping (Apple Silicon, experimental), linux-fuzz, minimal, plus the CI-specific ci-linux-asan and ci-linux-tsan.
• Note: the sanitizer/GCC/Clang CI jobs use manual cmake -B build flag invocations rather than the matching preset, so to reproduce those jobs exactly use the commands in CI Reproducible Builds, not the preset.
• If a preset produces unexpected behavior, check CMakePresets.json for the exact flags it sets rather than guessing.
• Delete build/ if switching between preset and manual configure — the cache will conflict.

Removal Before Addition

Before writing any new code in a file, check its size. SparkEngine's anti-bloat thresholds are ~500 lines for .cpp and ~300 for .h (guidelines for when to pause, not hard limits — see CLAUDE.md). If a file is over the threshold and doing multiple jobs, trim it first:

# Check size of a file before editing
wc -l SparkEngine/Source/Utils/SparkConsole.cpp

# If over threshold:
# 1. Find dead methods (no callers outside the class)
grep -n "void SimpleConsole::" SparkEngine/Source/Utils/SparkConsole.cpp | head -30
# 2. Delete them
# 3. Then make your actual change

For any new public method being added:

1. Search for existing methods that do something similar.
2. If one exists, extend it — don't add a new one.
3. If there are now 2 similar methods after your change, remove the older one.

For any new system/class being added:

1. Search for an existing class that could be extended.
2. If adding, remove something of equivalent complexity.

When to apply:

• Every time you open a file to edit it
• Before every PR — check net line delta: should be <= 0 for refactors, minimal positive for features
• When a file starts feeling hard to navigate

Notes:

• Removal is not "going backwards" — it is the primary maintenance activity.
• Delete dead code; don't comment it out. Git history exists.
• tools/check-bloat.sh enforces the size thresholds and runs as part of tools/validate-all.sh.

Session Start Checklist Order

Every session should start in this exact order before doing anything else:

1. git fetch origin Working && git log --oneline HEAD..origin/Working | wc -l — assess how far behind.
2. git rebase origin/Working — sync with upstream.
3. Resolve any rebase conflicts (see Git Rebase Conflicts).
4. cat wiki/_Sidebar.md — load persistent context from the wiki.
5. Read the wiki pages relevant to the current task.
6. Bloat check — run before touching anything:

   find SparkEngine/Source SparkEditor/Source SparkConsole/src GameModules \
     -name '*.cpp' | xargs wc -l | sort -rn | head -15

7. Then start reading code or planning.

Skipping steps 4-5 means starting each session without accumulated knowledge. Skipping step 6 means walking into a bloated file blind.

Notes:

• If wc -l output is 0 in step 1, the branch is up to date — skip the rebase.
• Never start reading or editing code before completing the git sync. Stale code leads to conflicts on push.
• The bloat check takes ~2 seconds.

Source & Freshness

• Original entry: Effective SparkEngine Development Workflows, last updated 2026-03-14.
• Verified against codebase 2026-06-08.
• 2026-09-24: added the Code City section (tools/architecture-viz).
• 2026-09-24: added the shared-build-directory lock section (tools/build-lock.sh) after a parallel-agent build deadlock.
• 2026-09-25: added the disk-budget section after repeated out-of-space stalls during parallel readiness work.
• Updated / found stale:
  • Doc-sync section now leads with docs/update-all-docs.sh (the master script), which is the current recommended one-shot; the two-script combo is kept as a faster subset.
  • Pre-push step 5 changed to docs/update-all-docs.sh to match current CLAUDE.md pre-commit guidance (was two separate scripts).
  • Anti-bloat thresholds updated to the current ~500 .cpp / ~300 .h guideline values (source quoted older 400/200 figures); noted tools/check-bloat.sh as the enforcer.
  • CMake preset list refreshed against CMakePresets.json; added new ci-linux-asan/ci-linux-tsan presets and the caveat that sanitizer/GCC/Clang CI jobs use manual flags, not presets.
  • Bloat-check find path generalized to GameModules (matches current CLAUDE.md; source pinned GameModules/SparkGame/Source).
  • Cross-references retargeted to the migrated wiki pages. Removed references to clang-format.md, ai-bloat-pattern.md, codebase-observations.md source files (not migrated to this folder).

Related Pages

• Git Rebase Conflicts
• Build Optimizations
• GitHub API and PR Checks
• CI Reproducible Builds
• Project conventions (CLAUDE.md)