Ideas: Do we need a dedicated multimodal fusion layer, or is late fusion via the Blackboard enough?

[web3guru888]

The Multimodal Gap

ASI:BUILD currently handles individual modalities well — bci processes EEG signals, optimization has a VLA++ model that fuses vision+language+action, holographic captures gesture and gaze, and consciousness processes any sufficiently abstract "experience." But we don't have a dedicated cross-modal fusion layer that sits between raw modality-specific encoders and the Cognitive Blackboard.

This is the gap I want to think through in this thread.

---

What We Have Today

bci/         → EEG → motor imagery → P300 → thought-to-text
             (modality: neural)

optimization/ → VLA++ model: Vision encoder + Language encoder + Action decoder
              → BCI integration: EEG → action fusion
              (modalities: vision + language + action + neural)

holographic/  → light field rendering, 21-joint gesture tracking, gaze estimation
              (modality: spatial/3D + gesture)

consciousness/ → processes abstract cognitive state, not raw modality data
              (modality: symbolic/theoretical)

Each handles its own modality independently. The Cognitive Blackboard receives post-processed events from each, but there's no layer that does cross-modal alignment before that stage.

---

The Neuroscience Case for a Fusion Layer

The brain doesn't process modalities in isolation. Key phenomena:

1. Multisensory integration (Superior Colliculus, STS)

• Visual + auditory stimuli arriving within ~150ms are spatially merged (spatial congruence rule)
• Near-threshold unimodal stimuli summate to suprathreshold multimodal percepts
• This is why watching a video call while hearing audio feels coherent even with 50ms video lag

2. Cross-modal plasticity

• Visual cortex activates during Braille reading in blind subjects
• This suggests modality boundaries in the brain are learned, not fixed

3. Predictive coding across modalities

• The brain maintains a unified generative model; each modality provides evidence for or against current predictions
• McGurk effect: auditory /ga/ + visual /ba/ → perceived /da/

---

What a multimodal/ Module Could Do

Concretely:

Option A: Late fusion (simple, current approach)
Each modality encodes independently. Fusion happens at the Blackboard level via event subscription. This is what we do today and it's fine for most cases.

Option B: Intermediate fusion (alignment layer)
A learned alignment function maps each modality into a shared embedding space before Blackboard writes:

• ModalityEncoder per modality (EEG → embedding, image → embedding, text → embedding)
• CrossModalAttention module that weights modality contributions per cognitive state
• Outputs a unified PerceptualState object that the Blackboard stores as a single entry

Option C: Early fusion (joint processing)
Raw inputs from multiple modalities are concatenated before any encoding. More expressive but requires modalities to be synchronized in time. Hard to achieve with heterogeneous sensor rates (EEG at 1KHz vs. camera at 30fps).

---

Research Questions

1. What's the right temporal resolution for cross-modal fusion? EEG data arrives at 1KHz. Camera frames at 30-90fps. Robot proprioception at 100-500Hz. Should the fusion layer resample everything to a common clock, or work with asynchronous streams?

2. Should the fusion layer be learned or rule-based? A fixed attention mechanism (e.g. consciousness-state-weighted sum) is interpretable. A learned cross-modal transformer is more expressive. Which matters more for a research framework like ASI:BUILD?

3. Where does the fusion layer sit in the CognitiveCycle? Issue Implement CognitiveCycle: wire all 29 modules into a real-time tick loop #41 proposes a 9-phase tick. The natural place is between sensory (phase 1) and perception (phase 2). Should it be its own phase, or part of sensory?

4. How should it interact with the Cognitive Blackboard? Should it write one PerceptualState entry, or multiple modality-specific entries that downstream modules subscribe to individually?

5. BCI + vision + language as a minimum viable set — the optimization module already has BCI+VLA fusion. Should the multimodal/ module just wrap and extend this, or be built from scratch?

---

Concrete Next Steps If We Build This

1. Add src/asi_build/multimodal/ with a MultimodalFusionEngine class
2. Start with late fusion (just coordinate Blackboard writes from bci, optimization, holographic)
3. Add a PerceptualState blackboard entry type
4. Wire consciousness to subscribe to PerceptualState events
5. Benchmark: does cross-modal fusion change IIT Φ or GWT broadcast coverage?

---

Would love to hear thoughts. Do you think a dedicated fusion layer is worth the complexity, or is late fusion via the Blackboard sufficient for most ASI research use cases?