our engine C++

our engine C++ is a fresh native engine project that starts from the first professional milestone we agreed on:

• timeline core
• multi-clip support
• multi-layer support
• playback session
• timeline resolver
• preview contracts

This repository is intentionally independent from UI code. A Qt/QML frontend can now be attached through a dedicated bridge without pulling Qt into the core engine itself.

Current milestone

The project now includes:

• step 1: timeline model, playback session, resolver, preview contracts
• step 2: editing commands, undo/redo history, project JSON serialization
• step 3: project JSON loading and round-trip verification
• step 4: FFmpeg-based preview frame extraction from real media
• step 5: timeline-driven preview extraction and media asset probing
• step 6: timeline decode session with preroll and cache behavior
• step 7: transport-driven preview loop that advances playback and refreshes buffered frames
• step 8: async decode worker with background scheduling and deterministic waiting
• step 9: async transport preview loop that queues decode work and applies completed frames
• step 10: presentation queue with paced frame delivery above async transport
• step 11: presentation handoff sink for UI-facing frame delivery without coupling Qt into the core
• step 12: CPU multi-layer composition core as the correctness reference before mobile GPU backends
• step 13: composition-aware handoff so the UI receives fully composited frames, not per-clip frames
• step 14: GPU resource abstraction with a backend-neutral upload contract and mock-GPU verification
• step 15: GPU presentation handoff so the UI-facing boundary can receive GPU texture bindings instead of CPU RGBA frames
• step 16: mobile GPU surface boundary so presented GPU textures can be attached to Android/iOS-style surfaces through a neutral contract
• step 17: first real backend adapter with optional Apple Metal texture upload and surface presentation verification
• step 18: first Android backend adapter with host-aware optional Vulkan texture allocation and Android-surface presentation verification
• step 19: shared GPU render graph planning for offscreen and surface targets above Metal/Vulkan backend boundaries
• step 20: shared GPU render executor that runs the render graph and feeds existing upload/presentation paths
• step 21: backend-aware GPU command encoding so shared timeline plans compile into deterministic command buffers before native backend submission
• step 22: device-level GPU command submission boundary so backend devices can accept validated command buffers before native renderer execution lands
• step 23: GPU resource binding boundary so encoded commands resolve into backend-owned textures and surfaces before bound submission
• step 24: backend-native GPU pass execution boundary so bound command buffers execute against real backend-owned render targets before deeper native renderer specialization
• step 25: GPU surface pass presentation boundary so executed bound passes can present their output texture to mobile-style surfaces through the same GPU-owned resource path
• step 26: transport GPU surface handoff integration so supported backends can drive presentation from the bound GPU pass path while older backends keep an explicit fallback
• step 27: native backend bound-surface adoption so Metal and Android Vulkan can execute bound passes into backend texture state and use the direct transport surface path when available
• step 28: timeline audio rendering, sink presentation, and playback-clock sync evaluation with real FFmpeg-decoded audio blocks
• step 29: first platform audio-device backend with Apple AVFoundation sink control above the step 28 audio transport boundary
• step 30: export pipeline that renders timeline video and audio into a real muxed output file through a dedicated FFmpeg export boundary
• step 31: mobile optimization planning that computes proxy scale, decode cache size, audio queue lead, and hardware-decode preference from memory pressure
• step 32: dedicated Qt/QML bridge with QObject transport control, QImage frame delivery, and QQuickImageProvider integration above the existing composited handoff
• step 33: extension foundations for templates, scripts, plugins, and marketplace packages with manifest validation, clip bindings, and JSON round-trip preservation
• step 34: template runtime boundary that resolves active template clips into deterministic parameter and script state at a timeline time

What exists in step 1

• Project / Sequence / Track / Layer / Clip data model
• multiple clips in the same timeline
• multiple layers per track
• separate video and audio tracks
• playback session with play, pause, seek, loop range, and preview requests
• resolver that turns the editable timeline into resolved segments and live snapshots
• smoke example and test target

What exists in step 2

• editing commands for:
  • split
  • trim left
  • trim right
  • move
  • transform update
  • mute update
  • delete
• snapshot-based undo/redo history
• project serialization to JSON
• example for editing flow and project save
• dedicated test coverage for command history and serialization

What is not implemented yet

• full Android runtime audio backend parity
• deeper native mobile GPU execution parity and hardware decode integration
• persistent command journal
• templates runtime beyond manifest and parameter foundations
• script runtime beyond manifest and package foundations
• plugins runtime beyond manifest and permission foundations
• marketplace packaging runtime beyond manifest registration and validation

These are intentionally planned for later phases so the core model stays clean.

How we verify the code

We currently validate the engine with:

• compile verification through cmake --build build
• unit and integration tests through ctest --test-dir build --output-on-failure
• runnable smoke examples for:
  • timeline and resolver
  • editing commands
  • JSON round-trip load/save
  • FFmpeg preview extraction from real media
  • timeline-driven preview extraction from active clips
  • preroll/buffering decode session behavior
  • transport-driven preview updates across seek, play, tick, and pause
  • background async decode requests with deterministic wait-based verification
  • async transport updates with enqueue, poll, and wait-for-frame verification
  • paced presentation delivery with queued frames and deterministic present intervals
  • UI-facing sink delivery for presented frames and clear events
  • multi-layer composition correctness with real media on multiple active layers
  • composition-aware UI handoff with fully composited frame delivery
  • backend-neutral GPU texture upload verification above the composited handoff boundary
  • GPU presentation handoff verification with texture delivery and clear events
  • GPU surface presentation verification with Android/iOS-ready surface contracts
  • optional Apple Metal backend verification through the same transport, upload, and surface pipeline
  • host-aware optional Android Vulkan backend verification through the same transport, upload, and surface pipeline
  • shared GPU render graph planning verification for offscreen and surface targets with deterministic layer ordering
  • shared GPU render execution verification with decoded inputs, executed draw nodes, and uploaded output texture
  • backend-aware GPU command-buffer encoding verification with Vulkan and Metal opcode plans above the shared executor
  • device-level GPU command submission verification with backend-matched queue labels, submission ids, and mismatch rejection
  • GPU resource-binding verification with per-layer sampled textures, render-target allocation, surface binding, and bound-command submission
  • backend-native GPU pass-execution verification with bound-resource validation, executed pass counts, and output render-target inspection
  • GPU surface pass-presentation verification with executed output texture delivery, composite metadata, and bound surface-state updates
  • transport GPU-surface handoff verification with bound-path reuse, stable surface ownership, texture turnover, and cleanup on clear
  • native-backend bound-path verification with Metal and host-available Android Vulkan shadow execution, direct surface presentation, and zero-upload transport delivery
  • timeline-audio verification with FFmpeg-decoded tone assets, mixed audio blocks, queued sink delivery, and playback-clock sync evaluation
  • platform-audio verification with Apple AVFoundation queueing, device start/stop control, and transport-driven playback on real rendered audio blocks
  • export-pipeline verification with real muxed media output, stream probing, preview-frame validation, and decoded audio verification from the exported file
  • mobile-optimization verification with deterministic playback-plan derivation under nominal, elevated, and critical memory pressure
• Qt-bridge verification with QObject transport control, composited QImage delivery, and QQuickImageProvider access to the current engine frame
• extension-foundation verification with template/script/plugin/package manifests, template and compound clip bindings, validator checks, and JSON round-trip preservation
  • template-runtime verification with sequence-backed clip snapshots, resolved template parameter overrides, script attachments, and runtime round-trip after JSON load

Reserved after Step 32

The following runtime families are now reserved in documentation and must land inside the engine after step 32, not inside UI code:

• motion and animation evaluation
• templates and presets
• script packages
• plugin boundaries
• marketplace packaging

The engine-side contracts that are intentionally reserved for those later phases are:

• animatable properties for clip, text, and effect parameters
• parameter schemas for templates and presets
• deterministic frame evaluation for motion
• a declarative script/animation package format
• plugin ABI and package manifest/versioning

Step 33 now fixes the first engine-owned layer for that reservation:

• template, script, plugin, and marketplace manifests now live in Project
• template clips can carry parameter overrides and attached script ids
• compound clips can bind nested sequences explicitly
• validation and JSON round-trip now preserve those contracts inside the core

The current strongest correctness check is round-trip testing:

1. create a project in memory
2. edit it
3. serialize it to JSON
4. load it again
5. resolve timeline state and compare expected behavior

For media-facing validation we now also:

1. generate a temporary video with ffmpeg
2. decode a preview frame at a requested timestamp
3. convert it to RGBA
4. write it to PPM
5. verify the output file exists and the frame size is correct

For timeline-media validation we now also:

1. probe media metadata from real files
2. import probed assets into the engine
3. place multiple clips on multiple layers
4. ask the engine for the active preview frame at a timeline time
5. verify that the top-most active clip was selected correctly

For decode-session validation we now also:

1. create a timeline decode session
2. request a center time with preroll slots
3. verify decoded slot count
4. verify cache hits on repeated requests
5. verify cache eviction behavior when the center moves

For transport validation we now also:

1. seek to a timeline position and prime the current frame
2. play from a fixed clock instant
3. tick the transport forward with deterministic time deltas
4. verify that the current preview frame follows the active clip across layers
5. pause and confirm the transport stops advancing while preview remains valid

For async-worker validation we now also:

1. enqueue decode work from the main thread
2. wait for a specific request id to complete
3. verify the worker publishes the expected center frame
4. verify cache reuse across consecutive async requests
5. verify an out-of-range request completes cleanly without a center clip

For async-transport validation we now also:

1. seek and enqueue a background frame request
2. poll transport state before completion
3. wait for the frame that matches the current transport position
4. tick transport forward and confirm later frames are applied only after completion
5. verify no-active-clip positions clear the current presentation frame

For presentation validation we now also:

1. wait for decoded frames to be staged into a presentation queue
2. present the first frame immediately
3. confirm later frames stay queued until the pacing interval elapses
4. present the queued frame only when the due time is reached
5. clear the presented frame when the transport moves to a no-clip region

For handoff validation we now also:

1. attach a recording presentation sink
2. verify no callback fires before a frame is actually presented
3. verify each newly presented frame is delivered exactly once
4. verify clip identity and frame dimensions across handoff events
5. verify a clear event is delivered when playback enters a no-clip region

For composition validation we now also:

1. generate two solid-color media sources
2. place them on two active layers at the same timeline time
3. compose them into a sequence-sized RGBA frame
4. verify base-layer pixels remain visible outside the overlay area
5. verify overlay-layer pixels dominate inside the overlay area

For composited-handoff validation we now also:

1. attach a sink that receives fully composited frames
2. verify no delivery happens before a frame is actually presented
3. verify the delivered frame contains multiple composited layers
4. verify pixel colors in the delivered frame match the expected composition
5. verify a clear event is delivered when playback enters a no-clip region

For GPU-resource validation we now also:

1. receive a fully composited frame from the handoff boundary
2. upload it into a backend-neutral GPU texture contract
3. verify the created texture shape, format, and usage flags
4. verify uploaded pixel colors still match the expected composition
5. verify repeated uploads reuse the same texture before reset destroys it

For GPU-presentation validation we now also:

1. attach a sink that receives GPU presentation packets
2. verify no GPU callback fires before a frame is actually presented
3. verify the delivered texture binding references a valid uploaded GPU texture
4. verify the uploaded texture pixels still match the expected composition
5. verify a clear event is delivered when playback enters a no-clip region

For GPU-surface validation we now also:

1. create a platform-tagged GPU surface contract for mobile-style output
2. present a delivered GPU texture into that surface
3. verify the surface metadata and backend expectations match the texture
4. verify the presented texture still contains the expected composition
5. verify the surface is cleared when playback enters a no-clip region

For GPU-command encoding validation we now also:

1. build one shared render graph from active timeline clips
2. encode that graph into a backend-aware command buffer for Vulkan
3. encode the same timeline into a second command buffer for Metal
4. verify opcode ordering and command counts for both targets without changing timeline logic
5. verify the shared render executor now carries the encoded command buffer alongside runtime execution

For GPU-command submission validation we now also:

1. submit a surface-target command buffer through a backend-matched GpuDevice
2. verify the submission report records backend name, queue label, and submission id
3. verify submitted clear, bind, draw, and present counts match the encoded command buffer
4. verify backend-mismatched command buffers are rejected at submission time
5. verify optional Metal and Android Vulkan device submission paths follow the same contract when available

For GPU-resource binding validation we now also:

1. bind sampled-input resources into backend textures using real decoded media frames
2. allocate a render-target texture for the graph output resource
3. bind an optional surface resource when the graph targets a presentation surface
4. verify bound submission rejects missing resource bindings
5. verify bound resources can be released cleanly after submission

For Apple-Metal validation we now also:

1. create a real Metal GPU device backend
2. drive it through the existing transport, composition, upload, and surface pipeline
3. verify the presented surface expects the Metal backend
4. verify the backend exposes a real native texture handle and uploaded texture contents
5. verify the surface clears correctly when playback enters a no-clip region

For Android-Vulkan validation we now also:

1. create a real Vulkan-backed Android device adapter from the official Android SDK runtime
2. drive it through the existing transport, composition, upload, and surface pipeline
3. verify the presented surface expects the Vulkan backend on the Android contract
4. verify the backend exposes a real native image handle and uploaded texture contents
5. verify the surface clears correctly when playback enters a no-clip region

For GPU-render-graph validation we now also:

1. build a backend-neutral render graph directly from active timeline clips
2. verify the graph creates one color target plus sampled inputs per active layer
3. verify draw-node ordering matches layer priority and z-index
4. verify surface targets append a final Present node while offscreen targets do not
5. verify the same timeline can target Mock, Metal, or Vulkan graph outputs without changing higher-level composition rules

For GPU-render-executor validation we now also:

1. execute the planned GPU render graph into a composited output packet
2. decode one input frame per active graph layer at the requested timeline time
3. blend the decoded inputs using the shared graph ordering rules
4. verify the executed output still matches the expected timeline composition
5. upload the executed output through the existing GPU uploader to prove the executor fits the current backend path

Phase Gate Policy

This project follows a strict phase-gate rule:

1. no later phase should be started before the current phase builds cleanly
2. every step must include a real test or runnable verification path
3. requests to skip ahead should be treated as a quality risk and explicitly flagged
4. architecture features planned for later must be accounted for, but not implemented early if doing so destabilizes the engine

The composition layer now exists as a CPU correctness reference. This is intentional: the future mobile GPU path for Android/iOS should match this behavior rather than replace the rules ad hoc.

The UI handoff path now also has a composited variant. This means a future Qt/QML layer can receive already-composited RGBA frames from the engine even before the dedicated mobile GPU backend lands.

Step 14 adds a backend-neutral GPU resource contract and a MockGpuDevice implementation. This is intentional: it lets us lock the upload and texture rules before we attach Metal or Vulkan paths for iOS/Android.

Step 15 moves the presentation boundary itself onto GPU-aware contracts. This means a future Qt/QML or mobile runtime can receive stable texture bindings from the engine rather than raw RGBA CPU frames whenever the GPU path is active.

Step 16 adds the next mobile-first boundary above that: a neutral GPU surface contract that can stand in for Android external textures or iOS layer-backed surfaces before the real platform backends land.

Step 17 is the first real backend implementation on top of those contracts. On Apple hosts, the engine can now create and upload real Metal textures while still using the same higher-level presentation path.

Step 18 adds the matching Android-side backend adapter using the official Android SDK Vulkan runtime available on the host. The adapter now performs a real availability probe before advertising itself as usable, so unsupported hosts degrade cleanly instead of failing later in the presentation path.

Step 19 adds the shared GPU render-graph layer above those backend boundaries. This is the piece that lets future Metal and Android Vulkan execution paths consume the same ordered composition plan instead of duplicating layer logic in each backend.

Step 20 adds the shared GPU render executor above that plan. This means the engine now has one place that actually runs the planned graph semantics and hands the result to the existing uploader and surface path, instead of calling the older direct compositor path from each GPU handoff separately.

Step 21 adds backend-aware GPU command encoding above that executor. This means the same timeline can now compile into deterministic Vulkan, Metal, or OpenGles command buffers before we replace CPU execution with native command submission in later phases.

Step 22 adds the device-level submission boundary above those command buffers. This means each backend device can now accept, validate, and report submitted command buffers through one shared contract before true native rendering and resource binding are attached in later phases.

Step 23 adds the resource-binding boundary above that submission layer. This means the engine now resolves sampled inputs, render targets, and optional surface targets into backend-owned handles before handing bound work to the GPU device.

Step 24 adds backend-native pass execution above those bound resources. This means the engine can now execute validated render passes against backend-owned targets and inspect the produced output before later phases deepen native renderer specialization.

Step 25 adds surface presentation above that execution boundary. This means the same bound GPU pass pipeline can now present its executed output texture to a mobile-style surface without falling back to the older CPU upload path.

Step 26 connects that path into the transport-facing GPU surface handoff. This means supported backends can now present transport-driven frames through the GPU-owned bound-pass path directly, while unfinished native backends still keep their older explicit uploader fallback until their execution path becomes real.

Step 27 deepens the native backends themselves. Metal and host-available Android Vulkan now execute bound passes into backend texture state and can adopt the same direct transport surface path instead of falling back to CPU upload-style delivery.

Step 29 closes the device-facing audio gap on Apple and leaves Android parity explicitly reserved behind the same sink boundary.

Step 30 turns the current compositor and audio renderer into a true export path. The engine now renders timeline frames and mixed audio blocks into temporary media inputs and muxes them into a real output file through a dedicated FFmpeg export boundary.

Step 31 adds the first mobile-runtime policy layer above the existing decode and audio transport systems. The engine can now derive a deterministic playback plan for proxy size, cache depth, queue lead, and hardware-decode preference from memory pressure instead of scattering those choices through UI code.

Step 32 adds the first true Qt/QML bridge above the engine. The bridge speaks through QObject, QImage, and QQuickImageProvider, while still consuming the same transport and composited-handoff path that the engine already tests.

Step 33 adds the first extension-model boundary above the current core. The engine now stores and validates:

• template manifests with parameter schemas and entry sequences
• script manifests with package format and output targets
• plugin manifests with runtime and permission declarations
• marketplace packages that bind those objects into distributable roots

This keeps the future motion/template/script/plugin runtime inside the engine instead of drifting into Qt/QML.

Step 34 adds the first true runtime boundary on top of that model. The engine can now resolve an active template clip into:

• its entry sequence binding
• its local playback time
• its final parameter values after overrides
• its attached script manifests

This keeps future motion and script execution deterministic and engine-owned, instead of rebuilding that state in the UI layer.

Future-ready direction

The step 1 model is already shaped to grow into:

• Compound clips
• Template clips
• reusable sequences
• script-driven transitions
• saved presets and packageable templates
• engine-to-Qt/QML bridge extensions

Structure

• include/our_engine/core
• include/our_engine/composition
• include/our_engine/extensions
• include/our_engine/gpu
• include/our_engine/export
• include/our_engine/model
• include/our_engine/mobile
• include/our_engine/playback
• include/our_engine/presentation
• include/our_engine/preview
• include/our_engine/qt
• include/our_engine/resolve
• include/our_engine/engine
• src
• examples
• tests
• docs

Build

cmake -S . -B build
cmake --build build
ctest --test-dir build --output-on-failure

Run examples

./build/our_engine_step1_smoke
./build/our_engine_step2_editing
./build/our_engine_step3_roundtrip
./build/our_engine_step4_ffmpeg_preview /path/to/video.mp4 1.0
./build/our_engine_step5_timeline_preview /path/to/video.mp4 1.0
./build/our_engine_step6_decode_session /path/to/video.mp4 1.0
./build/our_engine_step7_transport_preview /path/to/video.mp4 0.5
./build/our_engine_step8_async_decode_worker /path/to/video.mp4 0.5
./build/our_engine_step9_async_transport_preview /path/to/video.mp4 0.5
./build/our_engine_step10_presentation_queue /path/to/video.mp4 0.5
./build/our_engine_step11_presentation_handoff /path/to/video.mp4 0.5
./build/our_engine_step12_multi_layer_composition /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step13_composited_handoff /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step14_gpu_resources /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step15_gpu_presentation_handoff /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step16_gpu_surface_handoff /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step17_apple_metal_backend /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step18_android_vulkan_backend /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step19_gpu_render_graph /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step20_gpu_render_executor /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step21_gpu_command_encoder /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step22_gpu_command_submission /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step23_gpu_resource_binding /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step24_gpu_pass_execution /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step25_gpu_surface_pass_presentation /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step26_transport_gpu_surface_bound_path /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step27_native_backend_bound_surface /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step28_audio_output_sync /path/to/base.wav /path/to/overlay.wav
./build/our_engine_step29_platform_audio_output /path/to/base.wav /path/to/overlay.wav
./build/our_engine_step30_export_pipeline /path/to/base.mp4 /path/to/overlay.mp4 /path/to/output.mp4
./build/our_engine_step31_mobile_optimization
./build/our_engine_step32_qt_bridge /path/to/base.mp4 /path/to/overlay.mp4
./build/our_engine_step33_extension_foundations
./build/our_engine_step34_template_runtime