Skip to content

emc-mongoose/mongoose-storage-driver-preempt

Repository files navigation

Gitter chat Issue Tracker CI status Tag Maven metadata URL Sonatype Nexus (Releases) Docker Pulls

Contents

Preemptive multitasking (native threads pool) based storage driver. Uses thread-per-task approach for I/O.

Design

Batch Mode

A storage driver implementation can use the batch mode to work with the load operations to increase the efficiency.

Assuming

  • Qoutput: storage-driver-limit-queue-output
  • Qinput: storage-driver-limit-queue-input
  • Sbatch: load-batch-size

the following equation should always be true: Qoutput ≥ Qinput * Sbatch

otherwise, there may be the load operation results handling failures.

Heap Memory Consumption

The default Mongoose's load-batch-size configuration value is 32,768. The preemptive storage driver plugin which the task for each load operations batch. The default input queue size (storage-driver-limit-queue-input) is 1,000,000. This yields the 32,768,000,000 instances of the load operations in the runtime and require ~ 6 terabytes of the heap memory. To avoid this behavior, override the defaults:

Qinput * Sbatch ≤ 1,000,000