[Optimize] Switch DirectoryProcessor to in-memory mode (bypassing intermediate disk writes)

[Parallelogramator]

Previously, sensor data was written to disk on every tick and then re-read by BaseDataset to form batches. This caused double I/O, serialization/deserialization overhead, and heavy dependency on disk speed - typical data preparation time ranged from 30–90 ms with spikes up to 350+ ms.

Changes:

• Refactored DirectoryProcessor into a hybrid class supporting both disk-based and direct in-memory input.
• Added optional input_data parameter to update_dataset:
  • If provided -> data is taken directly from memory, disk is completely bypassed;
• Implemented cleanup of the previous tick's in-memory data - only the current frame is kept, preventing memory leaks during long runs.
• Ensured in-memory data format is 100% identical to what BaseDataset expects from disk files -> no changes required in models or downstream pipeline.

Results (100-tick co-perception scenario):

• Average time per tick: was ~37–350 ms -> now 0.108 ms
• Median: 0.092 ms
• 95th percentile: 0.170 ms
• Maximum: 0.344 ms

Data preparation accelerated by more than 300× on average and median values. Completely eliminated fsync, file write/read, and directory scanning delays. System no longer depends on storage speed.

How to test:

• Run a scenario with in-memory mode enabled
• Compare timing logs for data preparation / dataset update
• Verify that detections, visualizations, and quality metrics remain identical to the previous version
• Run a long session (15–60 min) and confirm memory usage stays stable (no growth)

The main bottleneck has now shifted further down the pipeline (inference, post-processing, etc.). A major step toward real-time co-perception performance.