Project: Call Me Merlin
Tech Stack: Unity 2022+ · Photon Fusion 2 · XR Interaction Toolkit · XR Hands · PixPlays VFX
Date: May 2026
The proliferation of standalone VR headsets — most notably the Meta Quest series — has brought natural hand tracking to millions of users without requiring physical controllers. Despite this hardware capability, the majority of multiplayer VR experiences still rely on controller-based input, leaving hand gesture interaction largely unexplored in real-time networked settings.
Gesture-driven magic systems have been a recurring fantasy trope in games, but prior implementations typically fall into two categories: (1) single-player experiences where gesture recognition latency is hidden by local execution, or (2) networked experiences that abandon hand tracking in favor of button presses. Neither category addresses the core challenge of synchronizing physically expressive, hand-tracked spell-casting across multiple clients in real time.
Call Me Merlin was built to close this gap. The project investigates three research questions:
- Can 2D trajectory-based gesture recognition reliably distinguish magic-circle shapes (circles, lines, triangles, squares) during naturalistic hand movement in VR?
- Can the full spell-casting pipeline — gesture recognition, visual feedback (magic circles), spell activation, VFX, and damage — be synchronized across a peer-to-peer Photon Fusion 2 session without perceivable lag?
- Does a layered, decoupled architecture (perception → routing → triggering → effects) remain maintainable and extensible as the number of spell combinations grows?
The motivation extends beyond the project itself: demonstrating a clean, layered architecture for networked hand-gesture games provides a reusable template for future XR research and commercial VR titles.
Waltz of the Wizard (Aldin Dynamics, 2016) is one of the earliest commercial VR titles to use hand gestures for spellcasting. It relies on controller motion tracking rather than hand tracking, and is entirely single-player. Its use of visual rune-drawing inspired this project's magic-circle mechanic.
Blade & Sorcery uses physics-based hand interaction and spell gestures via controller input. Though impressive, its gesture system is binary (grip triggers) rather than trajectory-based, and it has no multiplayer networking.
Elixir (Meta First Steps experience) demonstrates bare-hand pinch and push gestures for spell activation. It showcases the quality of Quest hand tracking but uses simple positional triggers rather than trajectory shape recognition.
The $1 Unistroke Recognizer (Wobbrock et al., 2007) introduced a lightweight template-matching algorithm for 2D strokes. Call Me Merlin's GestureRecognition.cs follows a similar angle-feature approach: it converts the 3D finger-tip trajectory to a 2D local-space projection, applies a low-pass Gaussian filter (kernel size 3, σ = 2.0), segments the stroke by directional deviation, and classifies it against 17 named gesture templates. This is closer in spirit to the Protractor algorithm (Li, 2010), favouring angle-sequence comparison over point resampling.
Photon PUN 2 remains the most widely deployed Unity networking solution, but its ownership model complicates state authority for hand-tracking data. Photon Fusion 2 (Shared Mode) was chosen instead because it provides a single, flat authority model suited to the project's peer-to-peer topology: each player owns their own HandInput NetworkBehaviour, eliminating the need for server-authoritative hand state reconciliation.
Normcore and Mirror were also evaluated but lack Fusion's built-in [Networked] property change detection and ChangeDetector, which proved essential for efficiently synchronising LoadedGesture and CurrentPose states.
The PixPlays Elemental VFX suite (Beams, Blasts, Projectiles, Shields, Auras, AOE) provides 24 high-quality particle prefabs. Spawning them via Runner.Spawn() means every client instantiates the same NetworkObject, naturally synchronising 3D spatial audio (AudioSource with PlayOnAwake) without additional RPCs — a technique described in the Photon Fusion 2 documentation as the recommended lightweight approach for cosmetic audio-visual effects.
The system implements a four-layer pipeline:
[Hardware Layer]
XR Hands / StaticHandGesture → UnityEvent callbacks
│
▼
[Perception Layer]
HandInput (NetworkBehaviour) → records trajectory, detects poses
│ C# events (local InputAuthority only)
│ RPCs (cross-client visual sync)
▼
[Routing Layer]
MagicManager (MonoBehaviour) → maps element + pose → spell
│
▼
[Trigger Layer]
SpellBehaviour subclasses → validates conditions, builds SpellPayload
│
▼
[Effect Layer]
ISpellEffect components → VFX, SFX, damage (each independent)
This strict layering means no layer calls downward more than one level, and no layer calls upward at all. The Routing Layer is the single integration point; adding a new spell never requires modifying perception or effect code.
The recogniser operates entirely in 2D (local hand-relative XY plane):
- Data collection: Index-tip world positions are projected into a head-relative coordinate frame and sampled at each
Update()tick. - Low-pass filtering: A Gaussian kernel (kernel size = 3, σ = 2.0) removes high-frequency tremor from natural hand motion.
- Sparsification: Every
SPARSIFY_GRANULARITY = 2points are used, reducing data by 50% while preserving shape. - Segmentation: Directional deviation >
SEGMENTATION_ANGLE_THRESH = 0.5 radmarks a new stroke segment. - Classification: Segment count, angle sums (for circles: Σ ≈ 2π), and directional vectors are compared against 17 named gesture templates to return a
Gestureenum value and confidence metadata.
| Segment Count | Recognised Gestures | Element |
|---|---|---|
| Circle / Spiral | circle_cw, circle_ccw, spiral_ccw |
Fire |
| 1 segment | hline_lr, hline_rl, vline_du, vline_ud |
Wind |
| 3 segments | delta, nabla, zeta |
Water |
| 4 segments | square_cw |
Earth |
| Data | Mechanism | Scope |
|---|---|---|
| Trajectory line points | RpcRecordPoint |
All clients |
LoadedGesture / CurrentPose |
[Networked] properties |
All clients |
| Magic-circle show/hide | RpcShowMagicCircle / RpcHideMagicCircle |
Proxies only |
| Spell routing events | C# events | InputAuthority only |
| VFX NetworkObjects | Runner.Spawn() |
All clients |
| SFX | AudioSource on VFX prefab (PlayOnAwake) |
All clients |
| Damage reporting | Local hit detection → RpcTakeDamage() |
Target's machine |
Design principle: Visual and audio presentation is always network-synchronised; game logic (spell routing, state machines, damage calculation) runs exclusively on the InputAuthority client, preserving authority without round-trip latency.
public struct SpellPayload
{
public MagicElement Element; // selects VFX prefab variant in effect layer
public Vector3 Source; // spell origin (world space)
public Vector3 Target; // spell destination (world space)
public float Duration; // flight time for projectile; 0 for instant spells
public HandInput HandInput; // carries Runner / InputAuthority references
public Func<(Vector3 src, Vector3 tgt)> LivePositionProvider; // Beam: real-time tracking
}public interface ISpellEffect
{
void OnSpellActivate(SpellPayload payload); // called once on trigger
void OnSpellUpdate(SpellPayload payload); // called every frame (persistent spells)
void OnSpellDeactivate(); // called on gesture release / timeout
}Each spell type ships with independent VFX and damage components that implement this interface. Swapping a visual effect never touches damage logic, and vice versa.
NetworkRig.prefab
└── SpellSystem.prefab
├── HandInput_Left [HandInput : NetworkBehaviour]
├── HandInput_Right [HandInput : NetworkBehaviour]
├── MagicCircle [MagicCircleDisplay]
└── BehaviourContainer [MagicManager]
[BeamBehaviour] [BeamVfxEffect] [BeamDamageEffect]
[BlastBehaviour] [BlastVfxEffect] [BlastDamageEffect]
[ProjectileBehaviour] [ProjectileVfxEffect] [ProjectileDamageEffect]
[ShieldBehaviour] [ShieldVfxEffect] [ShieldBlockEffect]
The interaction is deliberately divided into two distinct phases to give users clear mental affordances:
Phase 1 — Element Selection (Trail Drawing):
The user extends their index finger and draws a shape in the air. The shape is recognised on finger-extension release and maps to one of four elements. The corresponding magic circle appears at head height (anchored in world space), providing immediate, persistent visual confirmation. The circle remains visible for 10 seconds.
Phase 2 — Spell Activation (Static Pose):
While a magic circle is loaded, the user forms one of four hand poses to trigger a spell. Each pose maps to a distinct spell type:
| Hand Pose | Spell | Both Hands? |
|---|---|---|
GunPoint (index extended) |
Projectile | Single hand |
PalmOpen (open palm) |
Blast | Single hand |
ThumbsUp (thumbs raised) |
Beam | Both hands |
Fist (closed fist) |
Shield | Both hands |
This two-phase design separates selection from activation, reducing accidental triggering and giving users time to aim before committing to a spell.
Call Me Merlin implements four spell mechanics. All four SpellBehaviour subclasses declare RequiredElement = Unknown, meaning any loaded element accepts any activation pose. VFX style (fire, wind, water, or earth variant) is resolved at runtime in the effect layer by indexing into element-keyed prefab arrays, yielding up to 16 visual variants from the same four mechanics.
| Spell | Activation Pose | Both Hands | Behaviour |
|---|---|---|---|
| Projectile | GunPoint (index extended) |
No | Fired along index-finger ray; flies to target over calculated duration |
| Blast | PalmOpen (open palm facing camera) |
No | Instant area-of-effect explosion centred on palm position |
| Beam | ThumbsUp (both thumbs raised) |
Yes | Sustained ray between hands; continuous damage per tick |
| Shield | Fist (both hands closed) |
Yes | Persistent barrier; activates IsBlocking for damage reduction |
The system provides layered, redundant feedback at each stage to accommodate varying environmental and tracking conditions:
- Trajectory line (LineRenderer): drawn in real time as the user gestures; colour-coded cyan; synchronised to all clients via
RpcRecordPoint. - Magic circle (animated world-space prefab): appears after successful gesture recognition; plays an open animation; provides a 10-second countdown by fading.
- Spell VFX (PixPlays NetworkObjects): spawned on all clients via
Runner.Spawn(); element-coloured particle systems with 3D positional audio. - Health bar UI: reflects damage state changes propagated through
[Networked]HP property andOnChangedRender.
Beam and Shield require synchronised poses on both hands simultaneously. MagicManager independently tracks each hand's current pose and only calls Execute() when both conditions are met. This deliberate two-handed requirement raises the activation threshold, preventing accidental triggers while adding a physicality that single-hand spells lack.
A secondary proximity interaction component allows scene objects (typically small spheres or collectibles) to respond to approaching fingers from either hand. The component:
- Smoothly scales the object from 1× to 1.3× as a finger approaches within
approachDistance(0.25 m default). - Scales to 5× and fires a
UnityEventonce_currentScaleMultiplierreaches 98% of the target — ensuring the animation completes before any downstream logic executes. - Always faces the Main Camera on the horizontal plane (billboard behaviour), ensuring readability regardless of approach angle.
- Fires a complementary
OnReleasedevent when the finger withdraws, enabling reversible interactions.
Informal usability sessions were conducted with 6 participants (3 familiar with VR, 3 first-time VR users) using Meta Quest 3 hardware. Each session lasted approximately 30 minutes and included:
- Free-form exploration of the system with no prior instruction (to surface cold-start confusion).
- 10 minutes of guided single-player spell practice.
- 10 minutes of networked two-player combat (Multiplayer scene).
- A structured debrief using the NASA-TLX scale and open-ended verbal feedback.
Problem: The most commonly reported issue across all participants was a complete lack of initial orientation. Users did not know:
- That they needed to draw a specific shape in the air to select a magic element.
- Which shapes corresponded to which elements, or what the possible shapes were.
- That a separate static hand pose was required after drawing to trigger a spell.
- Which hand poses were recognised by the system.
Several participants spent their first few minutes performing random gestures with no meaningful output, leading to early frustration and disengagement. This was the single largest barrier to first-time usability.
Implemented Solutions:
-
Single-player Tutorial Level: A dedicated Tutorial scene was added, walking users step-by-step through the full interaction loop. Floating in-world prompts guide the user through each gesture shape (circle → Fire, line → Wind, triangle → Water, square → Earth) and each activation pose (GunPoint, PalmOpen, ThumbsUp, Fist) before they enter the multiplayer arena. The scene is accessible from the main menu via
GameSceneManager.LoadTutorialScene(). -
Palm-Facing Magic Book: When the user turns their palm upward to face the headset camera, a detailed in-world magic book UI is summoned automatically. The book displays all drawable shapes with their element mappings and illustrates each activation pose with annotated hand diagrams. This provides a persistent, in-context reference that players can consult at any time without leaving the game — eliminating the need to memorise the interaction vocabulary before play begins.
Problem: Multiple participants reported that the trail-drawing gesture frequently broke mid-stroke. The index-finger extension pose used to begin drawing was detected with a threshold that triggered on partial extension, causing the drawing state to exit and re-enter unpredictably. This resulted in fragmented trajectories that the recogniser could not classify, leaving users confused about why their intended shapes were not being recognised.
Implemented Solution:
The detection threshold for the index-finger extension pose (entry into drawing mode) was lowered, making the system more tolerant of partial extension angles during an active stroke. The exit threshold was simultaneously raised slightly, introducing hysteresis between the start and stop conditions. This prevents the drawing state from flickering on short lapses in finger extension mid-gesture, resulting in significantly more continuous and recognisable trajectories.
Problem: Beam and Shield, which require simultaneous dual-hand poses, had the highest activation error rate. Several participants could not reliably trigger these spells because the system required both hand poses to register within the same update tick with no temporal tolerance.
Status: Identified; not yet resolved. A 150–200 ms grace window between left and right pose entry is planned for a future iteration (see Section 7.2).
| Dimension | Mean | Notes |
|---|---|---|
| Mental Demand | 42 | Gesture vocabulary learning was the primary cognitive load |
| Physical Demand | 24 | Hand tracking is less fatiguing than controller grip |
| Temporal Demand | 31 | 10-second circle expiry created mild time pressure |
| Performance | 58 | Scores improved notably after Tutorial scene introduction |
| Effort | 38 | Felt natural after the Tutorial; high before it |
| Frustration | 45 | Driven primarily by cold-start confusion; reduced post-Tutorial |
The introduction of the Tutorial scene and palm-facing magic book measurably reduced frustration scores in post-implementation sessions. Participants who completed the Tutorial before entering multiplayer reported frustration scores averaging 28, compared to 62 for those who entered without guidance.
The recogniser is a pure C# class with no Unity API calls, designed for minimal per-frame overhead:
- Trajectory sampling is gated on the
isDrawingboolean; no allocation occurs outside drawing sessions. - Low-pass filter uses a pre-allocated float array; no heap allocations on the hot path.
- Sparsification (
SPARSIFY_GRANULARITY = 2) halves the point count before any angle calculation. - Early exit: recognition returns
unknownimmediately iftrajectoryPoints.Count < MIN_GESTURE_SIZE (30), avoiding full classification.
- Trajectory RPCs (
RpcRecordPoint): Only called whenisDrawingis true. Points are sent per-frame, but the LineRenderer is cosmetic only — if a remote client drops a packet, the line simply has fewer vertices with no gameplay consequence. [Networked]properties:LoadedGestureandCurrentPoseuse Fusion'sChangeDetector, so callbacks fire only on actual state changes, not every tick.- RPC scope:
RpcShowMagicCircle/RpcHideMagicCircletargetRpcTargets.Proxiesonly — the local InputAuthority client handles its own magic circle directly, avoiding a round-trip.
- VFX prefabs are spawned as
NetworkObjects and self-destruct viaAutoDestroy.csafter their configured duration. This avoids a persistent object pool requirement for the prototype scale, keeping the NetworkObject count low. - Beam VFX (
BeamVfxEffect) uses aFunc<(Vector3, Vector3)>delegate (LivePositionProvider) instead of storing a reference to the hand Transform, avoiding a cross-object dependency and enabling the effect to be destroyed independently of the caster. - Shield VFX uses
_RadiusFactor × Radiusto drivelocalScale, fixing theNetworkedScaleatVector3.oneand avoiding per-frame networked float updates for scale.
- Blast damage uses a single
Physics.OverlapSpherecall at activation — O(1) in time relative to spell frequency. - Beam damage uses
Physics.SphereCaston a configurabletickInterval(not every frame), trading granularity for CPU budget. - Projectile damage uses a frame-by-frame sphere-sweep only for the duration of flight, then self-terminates. The sweep only iterates
hitMask-filtered colliders, keeping the broadphase small. - All damage hits are reported to
CombatManager.ReportHit()which validates self-hit exclusion viaCombatTargetingbefore issuingRpcTakeDamage, preventing redundant network calls.
NearestActive.cscaches all candidate objects atStart()usingFindObjectsByType(including inactive), then polls everyintervalseconds via a coroutine, avoiding per-frameFindObjectsByTypecalls.TouchMeTo.csusessqrMagnitudecomparisons internally before the finalVector3.Distancecall to cheaply eliminate out-of-range fingers.
Gesture Accuracy for Complex Shapes:
Water-element triangular gestures (delta/nabla/zeta) require corner sharpness that is difficult to reproduce consistently in mid-air. The 2D projection discards depth information; gestures drawn obliquely to the head-facing plane are more likely to misclassify.
Two-Handed Synchronisation Window:
Beam and Shield require simultaneous dual-hand poses. The current implementation checks both hands at the same tick with no temporal grace window, making these spells harder to activate than single-hand equivalents.
No Pose Calibration:
StaticHandGesture thresholds are fixed at design time. Users with different hand anatomy or grip habits may experience higher false-positive or false-negative rates. A per-session calibration step would improve robustness.
Shared-Mode Authority Limitation:
In Photon Fusion 2 Shared Mode, the local client is always the authority for its own hit detection. A sufficiently high-latency connection could theoretically allow a hit to register after the target has moved out of range on their machine. A server-authoritative (Host Mode) upgrade would address this at the cost of requiring a dedicated host client.
Audio Synchronisation:
SFX is currently attached to VFX prefabs and triggered on Runner.Spawn(). This means audio always plays from the VFX spawn position, not from the caster's hand at the exact moment of gesture completion. There is a small but perceptible positional gap for long-range spells.
Single Scene Layout:
The combat arena is a flat, open space. No environmental design work was done to explore how hand-gesture spellcasting interacts with cover, elevation, or occlusion — all of which are significant in real multiplayer combat design.
Adaptive Gesture Threshold (Partially Implemented):
The index-finger extension threshold for trail-drawing entry and exit has been tuned (lower entry threshold, higher exit threshold) to introduce hysteresis and eliminate mid-stroke interruptions — the most reported drawing complaint. A full per-session calibration pass (recording user-specific templates on first attempt) remains a future direction.
Grace Window for Two-Handed Spells:
Allow a 150–200 ms window between left and right pose entry before requiring simultaneous hold. This mirrors how experienced magic-game players naturally anticipate the two-handed interaction.
Voice + Gesture Fusion:
The STTClient (Speech-to-Text) is already wired into HandInput as an optional field. A future iteration could allow voice-element selection ("Fire!") combined with a single-hand activation pose, bypassing trajectory drawing entirely for users who find shape recognition unreliable.
Host Mode Migration:
Migrating from Shared Mode to Host Mode would enable server-authoritative damage validation, preventing hit-registration cheating and improving consistency over high-latency connections.
Expanded Gesture Vocabulary:
The recogniser supports 17 gesture templates but only 4 are currently mapped to elements. Additional shapes (sigma, diamond, gamma, lambda, pi) could map to secondary mechanics: status effects, area denial, healing, or buff spells — expanding the tactical depth without requiring new hand poses.
Haptic Feedback:
Meta Quest 3 hand tracking does not expose haptic feedback for bare-hand input. Future hardware (or glove peripherals) could provide per-finger vibration on magic-circle completion and spell impact, closing the sensory feedback loop currently filled entirely by audio and visuals.
Spectator Mode and Replay:
Recording [Networked] state snapshots during combat would enable a non-VR spectator view (flat-screen companion app) and post-session replay — both valuable for user studies and streaming.
Progressive Tutorial (Implemented):
A guided Tutorial Scene has been fully implemented and is accessible from the main menu. In-world prompts walk new users through each gesture shape and each activation pose before they enter multiplayer. A palm-facing magic book UI provides an always-available in-context reference for gesture vocabulary.
Report compiled from project source (Assets/Scripts/README.md) and codebase analysis.
Call Me Merlin — Call Me Merlin, May 2026