Skip to content

9. Memory Conductor

Jimmy Pautz edited this page Nov 21, 2026 · 1 revision

Memory Conductor & Developer API

Storefront includes a built-in Memory Conductor (storefront_conductor) that coordinates background tasks, fork() subprocesses, and in-memory cache eviction across all installed KOReader plugins—preventing Kindle's memory watchdog (watchdogd / Linux OOM killer) from terminating KOReader when multiple plugins run simultaneously.


How Automatic Protection Works (Zero Configuration)

Even if other plugins have never heard of Storefront, the Memory Conductor protects your e-reader automatically as soon as Storefront loads:

  1. Background Timer Serialization (UIManager:scheduleIn): When plugins schedule background timers (onReaderReady, onNetworkConnected, onResume, periodic syncs), the Conductor identifies the calling plugin via stack inspection and checks whether it owns an active foreground window.
    • Foreground UI actions, core KOReader timers, micro-timers (<= 0.25s), and active subprocess polling timers pass through with zero delay.
    • Background timers are serialized into a single-file lane so two plugins never parse large JSON catalogs or scan chapters at the same time.
  2. Pre-fork() Copy-On-Write Stabilization (ffiutil.runInSubProcess): On Linux/Kindle, calling fork() shares LuaJIT's parent memory pages via Copy-On-Write (CoW). If the parent VM runs a garbage collection sweep or mutates tables while a child subprocess is alive, the kernel duplicates those 4 KB pages in RAM. Before any plugin forks a child process, the Conductor:
    • Reclaims soft caches from idle plugins if available RAM is below the safety watermark.
    • Runs collectgarbage("collect") to zero out parent GC debt before fork().
    • Holds other plugins' background timers until isSubProcessDone(pid) or terminateSubProcess(pid) completes.
  3. Kindle-Aware /proc/meminfo Thresholds: Legacy Kindle kernels (Linux 3.0) lack MemAvailable and report shared tmpfs memory (Shmem) inside Cached. Storefront computes true available memory as MemFree + Buffers + max(0, Cached - Shmem):
    • Low-Memory Devices (<= 512 MB RAM / Kindle): Soft watermark 45 MB, Critical floor 28 MB.
    • Standard Devices (> 512 MB RAM): Soft watermark 30 MB, Critical floor 18 MB.

Developer Integration (Optional, Zero Hard Dependency)

Plugin authors can integrate with the Memory Conductor at two levels without creating a hard dependency on Storefront.

Level 1: Zero-Import MemoryReclaim Event (3 Lines)

When RAM gets tight, Storefront broadcasts a standard KOReader Event:new("MemoryReclaim", level) to all active WidgetContainer plugins. Any plugin can shed its private cached tables without requireing Storefront:

--- Automatically called by Storefront Memory Conductor when system RAM is tight.
--- @param level string "soft" (pre-task cleanup) or "critical" (below critical RAM floor)
function MyPlugin:onMemoryReclaim(level)
    if not self:isMenuOpen() then
        self._cached_index = nil
    end
    if level == "critical" then
        self._chapter_buffer = nil
    end
    return false -- Return false so the event continues to other plugins
end

Level 2: storefront_conductor Module API

For background syncs, chapter analyzers, archive jobs, or subprocess workers, plugins can request a priority-managed execution slot via pcall(require, "storefront_conductor"):

local function runWithConductor(opts)
    local ok, Conductor = pcall(require, "storefront_conductor")
    if ok and Conductor and Conductor.requestSlot then
        return Conductor.requestSlot(opts)
    end
    -- Standalone fallback when Storefront is not installed
    opts.run(function() end)
    return nil
end

Conductor.requestSlot(opts)

Queues or immediately executes a task under Conductor supervision.

Field Type Default Description
plugin string Auto-detected Short plugin identifier (e.g. "xray", "sink", "nimbus").
task string "task" Descriptive task name for logs.
priority string "idle_bg" "foreground" (runs immediately, preempts idle work), "interactive_bg" (runs before idle tasks, e.g. dirty progress push on book close), or "idle_bg" (single-lane background queue with cooldown).
min_free_kb number Device threshold Minimum free RAM in KB required before starting. Triggers MemoryReclaim if below this floor.
subprocess boolean false Set true if the task spawns a child process (ffiutil.runInSubProcess) to run pre-fork() heap stabilization.
sync boolean false Set true for synchronous functions that don't invoke done() asynchronously.
run function(done) Required Task callback. Async tasks must call done() when finished (a 45s safety lease prevents deadlocks).
cancel function() nil Called if the Conductor preempts or aborts the task on suspend/low memory.
on_rejected function(reason) nil Called with "low_memory" or "suspended" if an idle_bg task cannot safely run.

Example (Async Background Subprocess):

runWithConductor({
    plugin      = "myplugin",
    task        = "sync_library",
    priority    = "idle_bg",
    min_free_kb = 40 * 1024, -- Require 40 MB free
    subprocess  = true,
    run = function(done)
        self:startBackgroundSync(function(ok, err)
            done() -- Release slot for the next plugin
        end)
    end,
    cancel = function()
        self:abortBackgroundSync()
    end,
    on_rejected = function(reason)
        logger.info("MyPlugin: deferred sync:", reason)
    end,
})

Conductor.registerPlugin(opts)

Registers a persistent plugin profile for targeted cache eviction and cancellation:

local ok, Conductor = pcall(require, "storefront_conductor")
if ok and Conductor and Conductor.registerPlugin then
    Conductor.registerPlugin({
        plugin = "myplugin",
        is_foreground = function() return self.menu_open == true end,
        on_reclaim = function(level)
            if not self.menu_open then
                self.cache = nil
            end
        end,
        on_cancel_bg = function()
            self:abortBackgroundSync()
        end,
    })
end

Conductor.reclaimMemory(level, exclude_plugin)

Triggers an immediate cross-plugin cache reclaim ("soft" or "critical") and garbage collection sweep.

Conductor.getMemoryStatus()

Returns a diagnostic snapshot table:

  • total_kb, available_kb, free_kb, buffers_kb, cached_kb, shmem_kb
  • luajit_heap_kb (current LuaJIT VM allocation in KB)
  • is_low_memory_device (boolean)
  • thresholds ({ soft_min_free_kb, critical_min_free_kb })
  • active_subprocesses, queue_depth, is_busy

Clone this wiki locally