Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
We need a much smarter way of handling database connections.
Today, connections are simple: Datasette runs a number of threads (defaults to 3) and each thread gets a threadlocal read-only (or immutable) connection to each attached database - opened on demand.
I'd also like to be able to run joins across multiple databases (#283) which further complicates things.
Supporting thousands of open SQLite connections at once feels like it won't provide good enough performance (though I should benchmark that to be sure). Some kind of connection pooling is likely to be necessary.
I think what I want it a mechanism where a thread can say "give me a connection for database X" and it either gets back connection to X instantly OR a new connection to X is created and returned OR it blocks (because a certain number of connections to that database exist already) until another thread returns their connection OR it times out and returns an error.
Potential API design:
with pool.connection("fixtures") as conn: conn.set_time_limit(1000) conn.allow_all() conn.execute(...)
Within this block the current thread has exclusive action to the connection - which has essentially been "checked out" from the pool. When the block ends that connection is made available to be checked out by other threads.
This could accept multiple database names for the case where I want to join across databases:
with pool.connection("fixtures", "dogs") as conn: conn.execute(...)
I'm going to refactor the
They currently live on the
I think I'll put them on the