Skip to content

[FEATURE] Bulk import/export — high-throughput data migration (CSV, JSON, Parquet) #198

@ElioNeto

Description

@ElioNeto

Description

ApexStore has no bulk data import/export capability. Users must write custom scripts using the set/get API, which is slow for large datasets.

Proposed Implementation

Import

apexstore-cli import --format csv --file data.csv
apexstore-cli import --format json --file data.ndjson

Optimizations:

  • Skip WAL fsync during bulk import (direct SSTable generation)
  • Sort keys before writing for better compression
  • Parallel SSTable building across column families

Export

apexstore-cli export --format csv --output all_keys.csv
apexstore-cli export --format parquet --output data.parquet

Impact

  • Enables data migration from other databases
  • Enables ETL pipelines
  • Reduces import time from hours to minutes for large datasets

Labels

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions