Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File based data connectors and accelerations - add notes about concurrency #226

Open
ewgenius opened this issue May 1, 2024 · 0 comments
Labels
bug Something isn't working documentation Improvements or additions to documentation

Comments

@ewgenius
Copy link
Contributor

ewgenius commented May 1, 2024

Both sqlite and duckdb allow multiple read connections, but only one write, we need to note about that in duckdb data connector, and duckdb/sqlite accelerator docs.

https://duckdb.org/docs/connect/concurrency

DuckDB has two configurable options for concurrency:

  1. One process can both read and write to the database.
  2. Multiple processes can read from the database, but no processes can write (access_mode = 'READ_ONLY').

When using option 1, DuckDB supports multiple writer threads using a combination of MVCC (Multi-Version Concurrency Control) and optimistic concurrency control (see Concurrency within a Single Process), but all within that single writer process.

https://www.sqlite.org/lockingv3.html

To write to a database, a process must first acquire a SHARED lock as described above (possibly rolling back incomplete changes if there is a hot journal). After a SHARED lock is obtained, a RESERVED lock must be acquired. The RESERVED lock signals that the process intends to write to the database at some point in the future. Only one process at a time can hold a RESERVED lock. But other processes can continue to read the database while the RESERVED lock is held.

SHARED | The database may be read but not written. Any number of processes can hold SHARED locks at the same time, hence there can be many simultaneous readers. But no other thread or process is allowed to write to the database file while one or more SHARED locks are active.

@digadeesh digadeesh added bug Something isn't working documentation Improvements or additions to documentation labels May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

2 participants