BZ1 Pt. II

[benbjohnson]

Overview

This pull request is another attempt at the bz1 engine. The original can be found at #3461.

This approach changes the engine to support an underlying WAL which is being moved out to a separate package. The WAL work is being done by @pauldix. The bz1 engine now supports a passthrough interface to a PointsWriter (which will be the WAL) and implements a function that the WAL periodically calls back to:

func (e *Engine) WriteIndex(pointsByKey map[string][][]byte) error

Implementation

The WriteIndex() function supports two separate write paths:

• "append" - Points for a key are all after the last point in the database. This is the fast path and it will simply append additional blocks onto the end.
• "insert" - Points for a key are intermixed with existing points. This is the slow path because it has to read out all the compressed blocks for the time range, combine them with the new points, and then rewrite all the blocks to disk.

There are two separate paths because most of the time users will be writing new data to the end of a series. The slow path mainly exists so that data can be backfilled.

TODO

• Implement append (fast path)
• Implement insert (slow path)
• Implement testing/quick black box testing

[benbjohnson]

This is the fast path. It grabs the last block in the bucket, reads the first 8 bytes (which is the ending timestamp), and if the minimum time for the inserted points is greater than that then it can simply append new blocks.

Otherwise it's a big PITA to read out and reinsert blocks as you can see below.

[pauldix]

just want to confirm that I'm reading this right. Because this happens in a transaction, the Delete and then the creation of a new block will be atomic?

[benbjohnson]

Yes

[pauldix]

Big one, but looks great. Just a few questions I had above.

[toddboom]

👏 👏 👏