test: Verify backend abstraction with second backend implementation

[justinjoy]

Validate Backend Abstraction with Multiple Backend Implementations

Current State

The backend abstraction (wl_compute_backend_t vtable) has only been tested with a single backend implementation: Columnar.

// wirelog/backend.h:85-107
typedef struct {
  wl_session_t* (*session_create)(wl_compute_backend_t*, ...);
  void (*session_destroy)(wl_session_t*);
  int (*session_insert)(wl_session_t*, ...);
  int (*session_remove)(wl_session_t*, ...);
  int (*session_step)(wl_session_t*);
  int (*session_set_delta_cb)(wl_session_t*, ...);
  wl_snapshot_t* (*session_snapshot)(wl_session_t*);
} wl_compute_backend_t;

Only wl_backend_columnar() is implemented. No other backend exists to validate the interface.

Why This Matters

1. Hidden Assumptions

With only one implementation, it's impossible to detect:

• Columnar-specific assumptions baked into the vtable interface
• Missing operations needed for different compute paradigms
• Implicit dependencies on columnar data representation
• Operations that don't generalize (e.g., K-Fusion is columnar-specific)

Example: K-Fusion assumes column-major layout. FPGA or GPU backends might need different parallelization strategies.

2. Interface Design Risk

The vtable may have fundamental design flaws that only become apparent when:

• Adding an FPGA backend (different execution model)
• Adding a GPU backend (GPU memory management, kernel compilation)
• Adding a distributed backend (network I/O, fault tolerance)

These flaws would force retrofitting the interface, affecting all backends.

3. Delayed Problem Discovery

As noted in ARCHITECTURE.md:

"Architectural flaws in backend interface may be hidden until 2nd backend is added"

This is a known risk with no validation before major architectural changes.

4. Plan Representation Assumptions

Execution plans are generated assuming columnar backend semantics:

• Columnar-specific operators (K_FUSION, LFTJ, EXCHANGE) leak into shared enum (see issue refactor: Separate universal and backend-specific operators in exec_plan enum #495)
• Plan optimization assumes column-major layout
• Delta tracking assumes Z-set semantics

These may not apply to other backends.

Problem Scenario: FPGA Backend

If we implement an FPGA backend:

1. Data Transfer: Need Arrow IPC for host-FPGA data movement

   • Columnar: in-memory columnar arrays
   • FPGA: Arrow IPC serialization/deserialization overhead
   • Interface may need session_serialize/session_deserialize

2. Parallelization: K-Fusion assumes CPU workqueue parallelism

   • Columnar: deep-copy K-fusion workers
   • FPGA: streaming pipeline logic, no deep copies
   • K-fusion operator incompatible; needs different plan shape

3. Memory Management: Columnar uses malloc; FPGA needs host memory pinning

   • Need backend-specific allocation strategies
   • vtable may need memory_alloc/memory_free callbacks

4. Incremental Computation: Columnar uses differential arrangements

   • FPGA may use different delta tracking (on-device)
   • session_snapshot semantics may differ

5. Error Handling: FPGA execution errors vs. CPU memory errors are different

   • Columnar: returns error codes
   • FPGA: may need async error notification
   • vtable may need error callback

Solution: Implement FPGA Backend

Add a second backend implementation to validate the abstraction:

Phase 1: Basic FPGA Backend (Validation Phase)

• Implement fpga_backend.h with FPGA-specific context
• Implement wl_backend_fpga() vtable
• Migrate core logic to backend-agnostic
• Run existing tests against FPGA backend (fail if interface assumes columnarity)
• Identify and document interface gaps

Phase 2: Data Transfer

• Arrow IPC integration
• Host-FPGA serialization
• Benchmark vs. columnar backend

Phase 3: Plan Adaptation

• Separate universal operators from backend-specific (see issue refactor: Separate universal and backend-specific operators in exec_plan enum #495)
• Plan rewrite for FPGA (no K-Fusion, use pipeline operators)
• FPGA-specific optimization passes

Phase 4: Performance Validation

• Benchmark queries on both backends
• Identify columnar vs. FPGA performance characteristics
• Validate that abstraction doesn't introduce excessive overhead

Benefits

• Validates backend abstraction is truly pluggable
• Discovers hidden columnar-specific assumptions early
• Forces core engine to be truly backend-agnostic
• Proves new backends can be added without modifying core
• Establishes patterns for future backends (GPU, distributed)
• Catches interface design flaws before they spread

Risk if Not Done

• FPGA backend implementation becomes painful
• Core engine heavily assumes columnar layout
• New operators pollute shared enum (issue refactor: Separate universal and backend-specific operators in exec_plan enum #495 repeats)
• Backend interface retrofitting causes churn
• Architecture erosion: "backend-agnostic" becomes lie

Related Issues

• Issue refactor: Separate universal and backend-specific operators in exec_plan enum #495: Separate backend-specific operators (prerequisite)
• Architecture principle: "Backend Pluggability"

Definition of Done

1. FPGA backend compiles and links against core engine
2. FPGA backend passes existing test suite (or tests correctly fail due to FPGA-specific limitations)
3. No core engine changes needed to support FPGA (except interface bugs found and fixed)
4. Interface gaps documented and tracked as separate issues
5. Core engine is demonstrably backend-agnostic