WAL/marshalable chunks #2764
WAL/marshalable chunks #2764
Conversation
owen-d
force-pushed
the
wal/marshalable-chunks
branch
from
1f823dd
to
58b6e62
Compare
owen-d
force-pushed
the
wal/marshalable-chunks
branch
from
58b6e62
to
7ec0fb0
Compare
Codecov Report
@@ Coverage Diff @@
## master #2764 +/- ##
==========================================
- Coverage 61.32% 61.21% -0.11%
==========================================
Files 179 181 +2
Lines 14438 14547 +109
==========================================
+ Hits 8854 8905 +51
- Misses 4775 4824 +49
- Partials 809 818 +9
|
There was a problem hiding this comment.
Changes look good to me so far. The only thing I can't relate to is the need for toWireChunks and fromWireChunks. If I understand it correctly we would be needing just the encoding and decoding of records and series. Can you please help me understand why we need it or is it not related to WAL implementation?
|
The |
| @@ -106,11 +106,13 @@ type Chunk interface { | |||
| Blocks(mintT, maxtT time.Time) []Block | |||
| Size() int | |||
| Bytes() ([]byte, error) | |||
| BytesWith([]byte) ([]byte, error) // uses provided []byte for buffer instantiation | |||
There was a problem hiding this comment.
Fun fact we also need this for iterator, to be able to reuse them.
| // The passed wireChunks slice is for re-use. | ||
| func toWireChunks(descs []*chunkDesc, wireChunks []Chunk) ([]Chunk, error) { | ||
| if cap(wireChunks) < len(descs) { | ||
| wireChunks = make([]Chunk, len(descs)) |
There was a problem hiding this comment.
I wonder if it wouldn't be better to append what is missing. But without benchmark, I can't prove it so we shall see in the future.
| lastUpdated: time.Now(), | ||
| } | ||
|
|
||
| mc, err := chunkenc.NewByteChunk(c.Data, conf.BlockSize, conf.TargetChunkSize) |
There was a problem hiding this comment.
changing the config of this could be catastrophic for the wal ? I wonder if we should encode this in the chunk header now ? Not for this PR but just raising a point here.
There was a problem hiding this comment.
I was worried about this as well, but don't think it will adversely affect us. We'd simply flush on the next push if moving from a larger to a smaller block or target size.
…2764) Signed-off-by: Trevor Wood <Trevor.G.Wood@gmail.com> Co-authored-by: Marco Pracucci <marco@pracucci.com>
This PR adds more types/functions to prepare for WAL serialization. Includes some basic tests.