Skip to content
This repository was archived by the owner on Jun 5, 2025. It is now read-only.
This repository was archived by the owner on Jun 5, 2025. It is now read-only.

[Task]: Improve DB insert logic #1085

@aponcedeleonch

Description

@aponcedeleonch

Description

These lines fire a concurrent db insert task for each new alert. This is not a problem in and of itself, since version 3+ of SQLite implements a multi-read/single-write concurrency model, but there's a catch.

The process of writing to the database requires the following logical steps

  1. obtain a SHARED lock -> still readable, new readers allowed
  2. obtain a RESERVED lock -> still readable, new readers allowed
  3. write some stuff durably to the transaction log -> still readable, new readers allowed
  4. obtain a PENDING lock -> still readable, no new readers allowed
  5. obtain an EXCLUSIVE lock -> not readable anymore

Here's SQLite docs page.

Each of the statements goes through this pipe, which means we have a bunch of (green) threads who likely all acquired a PENDING lock, so no new readers are allowed.

Thanks to @blkt for the above investigation

To avoid issues as the ones seen in #924 we should implement batch inserts into alerts table.

Handling all of the recording logic in a single transaction would also be ideal. Right now, we're inserting independently prompts, outputs and alerts. We could write into the 3 tables acquiring a lock for the transaction. Docs

BEGIN IMMEDIATE
-- write to prompts, output, and alerts
COMMIT;

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions