Repository navigation
Improve unordered table scans on rotational drives #1552
Description
An unordered table scan in Rethink is a very slow operation because we do a b-tree traversal without any awareness of disk layout. This is a problem because database dumps are extremely slow on rotational storage because of seeks.
We can improve this by making unordered scans disk-layout aware (this is obviously hard), or by somehow signaling the serializer to allow returning some blocks in a different order.
Leaving this in backlog.
Activity
jdoliner commentedon Oct 17, 2013
This isn't actually tough from a traversal perspective. This is basically already what backfilling does. We just need to make a way to hook that up to a ReQL API. So it's still kind of tough, but not as tough as OP makes it out to be.
danielmewes commentedon Oct 17, 2013
@jdoliner Can you explain a bit more how backfilling does that? Does it bypass the serializer interface or is there another trick which makes disk access more sequential (on average)?
I imagine that the solution to this would essentially be some kind of smart
rethinkdb extract, probably on a per-extent level.jdoliner commentedon Oct 17, 2013
Well, actually I'm overselling this quite a bit. It's just that backfilling works correctly with blocks returned in whatever order the serializer finds convenient. So that would probably already be a bit faster, and with some more optimization could request big batches of blocks and try to read them sequentially. It seems like we can only do so much here without changing the layout on disk though.