electron-async-storage

A high-performance, type-safe asynchronous storage library for Electron applications, built on the unstorage architecture. Features a sophisticated driver-based system with advanced serialization, real-time watching, batching capabilities, full synchronous API support, and a powerful migration framework.

✨ Key Features

🏗️ Driver-Based Architecture: Mount multiple storage backends with different drivers
⚡ Dual API: Full synchronous and asynchronous API support for maximum flexibility
🚀 High Performance: Built-in batching, queueing, and optimization strategies
📡 Real-time Watching: File system monitoring with chokidar integration
🔄 Migration System: Version-based migrations with hooks and error handling
🏷️ Advanced TypeScript: Conditional typing, storage definitions, and full type safety
📦 Serialization Engine: Complex object serialization using superjson (Date, RegExp, Set, Map, Error, URL, bigint)
🔧 Tree-Shakable: Modular architecture with individual driver imports
⚡ Multiple Formats: ESM, CJS with optimized builds

🚀 Installation

# Using npm
npm install electron-async-storage

# Using pnpm
pnpm add electron-async-storage

# Using yarn
yarn add electron-async-storage

📖 Quick Start

Basic Usage

import { createStorage } from "electron-async-storage";

// Create storage with default memory driver
const storage = createStorage();

// Asynchronous operations
await storage.setItem("user:profile", {
  name: "John Doe",
  lastLogin: new Date(),
  preferences: new Set(["dark-mode", "notifications"]),
});

const profile = await storage.getItem("user:profile");
console.log(profile); // Full object with Date and Set preserved

// Synchronous operations (great for configuration and caching)
storage.setItemSync("config:theme", "dark");
storage.setItemSync("config:language", "en");

const theme = storage.getItemSync("config:theme");
const allConfigs = storage.getKeysSync("config:");
console.log(theme); // "dark"
console.log(allConfigs); // ["config:theme", "config:language"]

File System Storage

import { createStorage } from "electron-async-storage";
import fsDriver from "electron-async-storage/drivers/fs";
import { app } from "electron";

const storage = createStorage({
  driver: fsDriver({
    base: path.join(app.getPath("userData"), "storage"),
  }),
});

// All data persisted to disk with real-time watching
await storage.setItem("app:settings", { theme: "dark", version: "1.0.0" });

Multi-Driver Architecture

import { createStorage } from "electron-async-storage";
import fsDriver from "electron-async-storage/drivers/fs";
import memoryDriver from "electron-async-storage/drivers/memory";
import queueDriver from "electron-async-storage/drivers/queue";

const storage = createStorage({ driver: memoryDriver() });

// Mount different drivers for different data types
storage.mount("cache", memoryDriver()); // Fast in-memory cache
storage.mount("config", fsDriver({ base: "./config" })); // Persistent config
storage.mount(
  "logs",
  queueDriver({
    driver: fsDriver({ base: "./logs" }),
    batchSize: 100,
    flushInterval: 5000,
  })
); // Batched logging

// Data automatically routed to appropriate driver
await storage.setItem("cache:user-session", sessionData); // → Memory
await storage.setItem("config:app-settings", settings); // → File system
await storage.setItem("logs:error", errorData); // → Queued to disk

🏗️ Core Architecture

Storage Interface

The Storage interface provides a comprehensive dual API for data operations:

interface Storage<T extends StorageValue = StorageValue> {
  // Asynchronous Core Operations
  hasItem(key: string, opts?: TransactionOptions): Promise<boolean>;
  getItem<R = T>(key: string, opts?: TransactionOptions): Promise<R | null>;
  setItem(key: string, value: T, opts?: TransactionOptions): Promise<void>;
  removeItem(key: string, opts?: TransactionOptions): Promise<void>;

  // Synchronous Core Operations
  hasItemSync(key: string, opts?: TransactionOptions): boolean;
  getItemSync<R = T>(key: string, opts?: TransactionOptions): R | null;
  setItemSync(key: string, value: T, opts?: TransactionOptions): void;
  removeItemSync(key: string, opts?: TransactionOptions): void;

  // Batch Operations (Async)
  getItems(
    items: string[],
    commonOptions?: TransactionOptions
  ): Promise<Array<{ key: string; value: T | null }>>;
  setItems(
    items: Array<{ key: string; value: T; options?: TransactionOptions }>,
    commonOptions?: TransactionOptions
  ): Promise<void>;

  // Batch Operations (Sync)
  getItemsSync(
    items: string[],
    commonOptions?: TransactionOptions
  ): Array<{ key: string; value: T | null }>;
  setItemsSync(
    items: Array<{ key: string; value: T; options?: TransactionOptions }>,
    commonOptions?: TransactionOptions
  ): void;

  // Raw Value Operations (Async)
  getItemRaw<T = any>(
    key: string,
    opts?: TransactionOptions
  ): Promise<T | null>;
  setItemRaw<T = any>(
    key: string,
    value: T,
    opts?: TransactionOptions
  ): Promise<void>;

  // Raw Value Operations (Sync)
  getItemRawSync<T = any>(key: string, opts?: TransactionOptions): T | null;
  setItemRawSync<T = any>(
    key: string,
    value: T,
    opts?: TransactionOptions
  ): void;

  // Key Management (Async)
  getKeys(base?: string, opts?: GetKeysOptions): Promise<string[]>;
  clear(base?: string, opts?: TransactionOptions): Promise<void>;

  // Key Management (Sync)
  getKeysSync(base?: string, opts?: GetKeysOptions): string[];
  clearSync(base?: string, opts?: TransactionOptions): void;

  // Mount System, Watching, Migration, Lifecycle (Async only)
  mount(base: string, driver: Driver): Storage;
  unmount(base: string, dispose?: boolean): Promise<void>;
  watch(callback: WatchCallback): Promise<Unwatch>;
  migrate(): Promise<void>;
  dispose(): Promise<void>;

  // Aliases (Async)
  keys: typeof getKeys;
  get: typeof getItem;
  set: typeof setItem;
  has: typeof hasItem;
  del: typeof removeItem;
  remove: typeof removeItem;

  // Aliases (Sync)
  keysSync: typeof getKeysSync;
  getSync: typeof getItemSync;
  setSync: typeof setItemSync;
  hasSync: typeof hasItemSync;
  delSync: typeof removeItemSync;
  removeSync: typeof removeItemSync;
}

Driver System

All drivers implement the Driver interface with optional capabilities:

interface Driver<OptionsT = any, InstanceT = any> {
  name?: string;
  flags?: DriverFlags; // { maxDepth?: boolean; ttl?: boolean }
  options?: OptionsT;
  getInstance?: () => InstanceT;

  // Required Methods
  hasItem(key: string, opts: TransactionOptions): MaybePromise<boolean>;
  getItem(key: string, opts?: TransactionOptions): MaybePromise<StorageValue>;
  getKeys(base: string, opts: GetKeysOptions): MaybePromise<string[]>;

  // Optional Methods
  setItem?(
    key: string,
    value: string,
    opts: TransactionOptions
  ): MaybePromise<void>;
  removeItem?(key: string, opts: TransactionOptions): MaybePromise<void>;
  getMeta?(
    key: string,
    opts: TransactionOptions
  ): MaybePromise<StorageMeta | null>;
  clear?(base: string, opts: TransactionOptions): MaybePromise<void>;
  watch?(callback: WatchCallback): MaybePromise<Unwatch>;
  dispose?(): MaybePromise<void>;

  // Batch Operations (Performance Optimization)
  getItems?(
    items: Array<{ key: string; options?: TransactionOptions }>,
    commonOptions?: TransactionOptions
  ): MaybePromise<Array<{ key: string; value: StorageValue }>>;
  setItems?(
    items: Array<{ key: string; value: string; options?: TransactionOptions }>,
    commonOptions?: TransactionOptions
  ): MaybePromise<void>;

  // Raw Value Support
  getItemRaw?(key: string, opts: TransactionOptions): MaybePromise<unknown>;
  setItemRaw?(
    key: string,
    value: any,
    opts: TransactionOptions
  ): MaybePromise<void>;
}

🔧 Built-in Drivers

Memory Driver

High-performance in-memory storage using JavaScript Map:

import memoryDriver from "electron-async-storage/drivers/memory";

const storage = createStorage({
  driver: memoryDriver(),
});

// Instant read/write operations
// Data lost on process restart
// Perfect for caching and temporary data

File System Driver

Full-featured file system storage with watching capabilities:

import fsDriver from "electron-async-storage/drivers/fs";

const storage = createStorage({
  driver: fsDriver({
    base: "./data", // Base directory
    ignore: ["**/node_modules/**"], // Ignore patterns (anymatch)
    readOnly: false, // Read-only mode
    noClear: false, // Disable clear operations
    watchOptions: {
      // Chokidar options
      ignoreInitial: true,
      persistent: true,
    },
  }),
});

// Features:
// - Real-time file watching with chokidar
// - Path traversal protection
// - Recursive directory operations
// - File metadata (atime, mtime, size, birthtime, ctime)
// - Configurable ignore patterns

File System Lite Driver

Lightweight file system storage without watching:

import fsLiteDriver from "electron-async-storage/drivers/fs-lite";

const storage = createStorage({
  driver: fsLiteDriver({
    base: "./data",
    ignore: (path) => path.includes("temp"),
    readOnly: false,
    noClear: false,
  }),
});

// Smaller footprint, no chokidar dependency
// Same file operations as fs driver
// No real-time watching capability

Queue Driver

Performance-optimized batching wrapper for any driver:

import queueDriver from "electron-async-storage/drivers/queue";
import fsDriver from "electron-async-storage/drivers/fs";

const storage = createStorage({
  driver: queueDriver({
    driver: fsDriver({ base: "./logs" }),
    batchSize: 100, // Batch operations
    flushInterval: 1000, // Auto-flush interval (ms)
    maxQueueSize: 1000, // Maximum queue size
    mergeUpdates: true, // Merge duplicate key updates
  }),
});

// Benefits:
// - Dramatically improved write performance
// - Automatic batching and merging
// - Configurable flush strategies
// - Queue overflow protection
// - Maintains operation order

📊 Advanced Features

Complex Object Serialization

electron-async-storage uses superjson for sophisticated object serialization:

// All these types are preserved across storage operations
await storage.setItem("complex-data", {
  date: new Date(),
  regex: /pattern/gi,
  set: new Set([1, 2, 3]),
  map: new Map([["key", "value"]]),
  bigint: 123n,
  undefined: undefined,
  error: new Error("test"),
  url: new URL("https://example.com"),
});

const data = await storage.getItem("complex-data");
console.log(data.date instanceof Date); // true
console.log(data.regex instanceof RegExp); // true
console.log(data.set instanceof Set); // true
// All types perfectly preserved

Raw Value Operations

For binary data and custom serialization:

// Store raw binary data
const buffer = new Uint8Array([1, 2, 3, 4]);
await storage.setItemRaw("binary-data", buffer);

// Data is base64 encoded automatically
const restored = await storage.getItemRaw("binary-data");
console.log(restored instanceof Uint8Array); // true

Real-Time Watching

Monitor storage changes in real-time:

// Watch for all changes
const unwatch = await storage.watch((event, key) => {
  console.log(`${event}: ${key}`); // "update: user:profile"
});

// Watch specific keys
await storage.setItem("user:profile", userData); // Triggers watcher

// Cleanup
await unwatch();

Key Management and Filtering

Sophisticated key operations with depth control:

// Hierarchical keys with depth filtering
await storage.setItem("app:ui:theme", "dark");
await storage.setItem("app:ui:layout:sidebar", "collapsed");
await storage.setItem("app:data:cache:user", userData);

// Get keys at specific depths
const uiKeys = await storage.getKeys("app:ui", { maxDepth: 1 });
// Returns: ['app:ui:theme'] (excludes deeper nested keys)

const allAppKeys = await storage.getKeys("app");
// Returns: ['app:ui:theme', 'app:ui:layout:sidebar', 'app:data:cache:user']

Metadata System

Rich metadata support for advanced use cases:

// Set metadata
await storage.setMeta("cached-data", {
  ttl: Date.now() + 3_600_000, // TTL timestamp
  source: "api-v2",
  version: "1.2.0",
  priority: "high",
});

// Retrieve with metadata
const meta = await storage.getMeta("cached-data");
console.log(meta.ttl, meta.source, meta.atime, meta.mtime);

// File system drivers include native metadata
const fileMeta = await storage.getMeta("config.json");
console.log(fileMeta.size, fileMeta.birthtime, fileMeta.ctime);

⚡ Synchronous API

The library provides a complete synchronous API alongside the asynchronous methods, offering zero Promise overhead for performance-critical scenarios and simpler code when blocking operations are acceptable.

Core Benefits

🚀 Zero Promise Overhead: Direct return values without async/await
📝 Simplified Code: Cleaner logic for configuration and simple operations
🎯 Better Performance: Faster execution for memory and simple file operations
🔄 Seamless Integration: Works alongside async methods in the same storage instance

Usage Examples

import { createStorage } from "electron-async-storage";
import memoryDriver from "electron-async-storage/drivers/memory";

const storage = createStorage({ driver: memoryDriver() });

// Configuration at startup (synchronous)
storage.setItemSync("app:theme", "dark");
storage.setItemSync("app:language", "en");
storage.setItemSync("app:debug", true);

// Quick cache operations
const theme = storage.getItemSync("app:theme");
if (storage.hasItemSync("cache:user-preferences")) {
  const prefs = storage.getItemSync("cache:user-preferences");
  console.log("Cached preferences:", prefs);
}

// Batch operations
storage.setItemsSync([
  { key: "config:api-url", value: "https://api.example.com" },
  { key: "config:timeout", value: 5000 },
  { key: "config:retries", value: 3 },
]);

const allConfigs = storage.getItemsSync([
  "config:api-url",
  "config:timeout",
  "config:retries",
]);

// Key management
const configKeys = storage.getKeysSync("config:");
storage.clearSync("temp:"); // Clear temporary data

Driver Support

Driver	Sync Support	Performance	Use Cases
Memory	✅ Full	Excellent	Caches, sessions, config
FS	✅ Full	Good	Persistent config, small files
FS-Lite	✅ Full	Good	Lightweight file operations
Queue	⚠️ Partial	Variable	Immediate operations only

Performance Comparison

// Memory driver performance test
console.time("sync-operations");
for (let i = 0; i < 10_000; i++) {
  storage.setItemSync(`key-${i}`, { data: i });
  storage.getItemSync(`key-${i}`);
}
console.timeEnd("sync-operations"); // ~25ms

console.time("async-operations");
for (let i = 0; i < 10_000; i++) {
  await storage.setItem(`async-key-${i}`, { data: i });
  await storage.getItem(`async-key-${i}`);
}
console.timeEnd("async-operations"); // ~75ms (Promise overhead)

Best Practices

Use sync methods for: Configuration loading, simple caches, startup data
Use async methods for: I/O intensive operations, large datasets, non-blocking scenarios
Mixed usage: Combine both APIs as needed in the same application

// Application startup: sync for immediate needs
const theme = storage.getItemSync("app:theme") || "light";
const language = storage.getItemSync("app:language") || "en";

// Runtime: async for user data
const userProfile = await storage.getItem("user:profile");
const userSettings = await storage.getItem("user:settings");

// Caching: sync for quick access
if (!storage.hasItemSync("cache:theme-config")) {
  storage.setItemSync("cache:theme-config", buildThemeConfig(theme));
}

🔄 Migration System

Powerful version-based migration system with hooks:

const storage = createStorage({
  driver: fsDriver({ base: "./data" }),
  version: 3,
  migrations: {
    1: async (storage) => {
      // Migrate to version 1
      const oldData = await storage.getItem("legacy-format");
      await storage.setItem("new-format", transformData(oldData));
      await storage.removeItem("legacy-format");
    },
    2: async (storage) => {
      // Migrate to version 2
      const users = await storage.getKeys("users:");
      for (const key of users) {
        const user = await storage.getItem(key);
        user.version = 2;
        user.migrationDate = new Date();
        await storage.setItem(key, user);
      }
    },
    3: async (storage) => {
      // Migrate to version 3
      await storage.setItem("schema-version", {
        version: 3,
        features: ["new-api"],
      });
    },
  },
  migrationHooks: {
    beforeMigration: async (from, to, storage) => {
      console.log(`Starting migration from v${from} to v${to}`);
      await storage.setItem("migration:backup", await snapshot(storage, ""));
    },
    afterMigration: async (from, to, storage) => {
      console.log(`Migration complete: v${from} → v${to}`);
      await storage.removeItem("migration:backup");
    },
    onMigrationError: async (error, from, to, storage) => {
      console.error(`Migration failed: v${from} → v${to}`, error);
      // Restore from backup
      const backup = await storage.getItem("migration:backup");
      await restoreSnapshot(storage, backup);
    },
  },
});

// Migrations run automatically
await storage.migrate();

🏷️ Advanced TypeScript

Storage Definitions

Define typed storage schemas for compile-time safety:

interface AppStorageSchema {
  items: {
    "user:profile": UserProfile;
    "app:settings": AppSettings;
    "cache:api-response": ApiResponse;
  };
}

const storage = createStorage<AppStorageSchema>();

// Full type safety
await storage.setItem("user:profile", userProfile); // ✅ Typed
await storage.setItem("user:unknown", data); // ❌ Type error

const profile = await storage.getItem("user:profile"); // UserProfile | null

Conditional Types

The library uses sophisticated conditional types for flexible APIs:

// Automatically infer return types based on storage definition
type ProfileType = Awaited<ReturnType<typeof storage.getItem<"user:profile">>>;
// ProfileType = UserProfile | null

// Batch operations maintain type safety
const results = await storage.getItems(["user:profile", "app:settings"]);
// results[0].value is UserProfile | null
// results[1].value is AppSettings | null

Driver Type Safety

Custom drivers with full type safety:

interface CustomDriverOptions {
  endpoint: string;
  apiKey: string;
  timeout?: number;
}

const customDriver = defineDriver<CustomDriverOptions>((opts) => ({
  name: "custom-api",
  async getItem(key) {
    const response = await fetch(`${opts.endpoint}/${key}`, {
      headers: { Authorization: `Bearer ${opts.apiKey}` },
    });
    return response.json();
  },
  async setItem(key, value) {
    await fetch(`${opts.endpoint}/${key}`, {
      method: "PUT",
      headers: { Authorization: `Bearer ${opts.apiKey}` },
      body: JSON.stringify(value),
    });
  },
  // ... other methods
}));

⚡ Performance Optimization

Batching Operations

Optimize performance with batch operations:

// Instead of individual operations
for (const [key, value] of entries) {
  await storage.setItem(key, value); // Slow: N disk operations
}

// Use batch operations
await storage.setItems(entries.map(([key, value]) => ({ key, value }))); // Fast: 1 batch operation

Queue Driver Configuration

Optimize queue driver for your use case:

// High-throughput logging
const loggingStorage = createStorage({
  driver: queueDriver({
    driver: fsDriver({ base: "./logs" }),
    batchSize: 1000, // Large batches for efficiency
    flushInterval: 5000, // Less frequent flushes
    maxQueueSize: 10_000, // Large queue
    mergeUpdates: true, // Merge duplicate updates
  }),
});

// Real-time configuration
const configStorage = createStorage({
  driver: queueDriver({
    driver: fsDriver({ base: "./config" }),
    batchSize: 10, // Small batches for responsiveness
    flushInterval: 100, // Frequent flushes
    maxQueueSize: 100, // Small queue
    mergeUpdates: false, // Preserve all updates
  }),
});

Memory Management

// Dispose resources properly
const storage = createStorage({ driver: fsDriver({ base: "./data" }) });

// Use storage...

// Cleanup
await storage.dispose(); // Closes file watchers, flushes queues, etc.

🔧 Build Integration

Bundler Configuration

Webpack

module.exports = {
  resolve: {
    alias: {
      "electron-async-storage/drivers/fs": path.resolve(
        __dirname,
        "node_modules/electron-async-storage/drivers/fs.mjs"
      ),
    },
  },
};

Vite

export default defineConfig({
  resolve: {
    alias: {
      "electron-async-storage/drivers/fs":
        "electron-async-storage/drivers/fs.mjs",
    },
  },
});

Tree Shaking

Import only the drivers you need:

// ✅ Tree-shakable - only imports used drivers
import { createStorage } from "electron-async-storage";
import fsDriver from "electron-async-storage/drivers/fs";

// ❌ Imports all drivers
import { createStorage, builtinDrivers } from "electron-async-storage";

📚 Additional Documentation

Architecture Guide - Deep dive into the internal architecture
Driver Development - Creating custom drivers
Migration Guide - Advanced migration patterns
API Reference - Complete API documentation
Performance Guide - Optimization strategies
TypeScript Guide - Advanced TypeScript usage

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

📄 License

MIT License - see LICENSE for details.

🙏 Acknowledgments

Built on the solid foundation of unstorage by the UnJS team.

Name		Name	Last commit message	Last commit date
Latest commit History 6 Commits
.github/workflows		.github/workflows
docs		docs
scripts		scripts
src		src
test		test
.editorconfig		.editorconfig
.gitignore		.gitignore
.npmrc		.npmrc
.prettierignore		.prettierignore
.prettierrc		.prettierrc
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
README.md		README.md
build.config.ts		build.config.ts
eslint.config.mjs		eslint.config.mjs
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
tsconfig.json		tsconfig.json
vite.config.mjs		vite.config.mjs

License

YougLin-dev/electron-async-storage

Folders and files

Latest commit

History

Repository files navigation