Repository navigation
Idiomatic Goose for a vectorized database engine (DuckDB port) #289
gustavoleo
started this conversation in
General
Replies: 1 comment
|
Certainly B) is what you do to overcome direct indexing to variable sized anything limitations, yes. Other than that, I don't know enough about databases.. best you experiment a little :) |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I'm looking at the feasibility of porting a vectorized analytical database (DuckDB) to Goose. I know Goose is great for games and compilers, but databases have some specific memory access patterns.
The main friction point is DuckDB's DataChunk architecture. It processes data in batches (e.g., 2048 rows) and relies on two things:
Fast, random access to fixed-width vectors (integers, floats).
Random access to variable-length strings within a vector (e.g., accessing string #500 to evaluate a filter).
Since Goose arrays of variable-sized elements can only be iterated and not randomly indexed, my plan is:
A)Use fixed-size arrays for the core numerical vectors (which I assume can be randomly indexed)
B)Use a parallel array of offsets (or a secondary index) to handle random access to string vectors.
C)Drop to C-interop (extern fn) for the absolute hottest memory-management/parsing loops.
Does this sound like the right idiomatic approach for Goose? If you were forced to write a high-performance columnar engine in Goose, how would you lay out the DataChunk? Is the parallel index array the way to go for strings, or is there a better pattern I'm missing?
Thanks!
All reactions