Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduces per stream chunks mutex #3000

Merged
merged 63 commits into from
Nov 27, 2020
Merged

Introduces per stream chunks mutex #3000

merged 63 commits into from
Nov 27, 2020

Commits on Oct 27, 2020

  1. marshalable chunks

    @owen-d
    owen-d committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    1b2b931 View commit details
    Browse the repository at this point in the history

  2. wal record types custom serialization

    @owen-d
    owen-d committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    ddecf02 View commit details
    Browse the repository at this point in the history

  3. proto types for wal checkpoints

    @owen-d
    owen-d committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    48fa77f View commit details
    Browse the repository at this point in the history

  4. byteswith output unaffected by buffer

    @owen-d
    owen-d committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    7ec0fb0 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2020

  1. wal & record pool ifcs

    @owen-d
    owen-d committed Oct 28, 2020
    Configuration menu
    Copy the full SHA
    d6ccccf View commit details
    Browse the repository at this point in the history

  2. wal record can hold entries from multiple series

    @owen-d
    owen-d committed Oct 28, 2020
    Configuration menu
    Copy the full SHA
    eb6a519 View commit details
    Browse the repository at this point in the history

  3. entry pool

    @owen-d
    owen-d committed Oct 28, 2020
    Configuration menu
    Copy the full SHA
    f089f09 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2020

  1. ingester uses noopWal

    @owen-d
    owen-d committed Oct 29, 2020
    Configuration menu
    Copy the full SHA
    211356d View commit details
    Browse the repository at this point in the history

  2. removes duplicate argument passing in ingester code. adds ingester co… 

    …nfig validation & derives chunk encoding.
    @owen-d
    owen-d committed Oct 29, 2020
    Configuration menu
    Copy the full SHA
    7adf966 View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2020

  1. Merge remote-tracking branch 'upstream/master' into wal/writes

    @owen-d
    owen-d committed Nov 4, 2020
    Configuration menu
    Copy the full SHA
    29fafd6 View commit details
    Browse the repository at this point in the history

  2. segment writing

    @owen-d
    owen-d committed Nov 4, 2020
    Configuration menu
    Copy the full SHA
    5ed8ffb View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2020

  1. [WIP] wal recovery from segments

    @owen-d
    owen-d committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    107b03b View commit details
    Browse the repository at this point in the history

  2. replay uses sync.Maps & preserves WAL fingerprints

    @owen-d
    owen-d committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    fb194f5 View commit details
    Browse the repository at this point in the history

  3. in memory wal recovery

    @owen-d
    owen-d committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    3d36b2d View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2020

  1. wal segment recovery

    @owen-d
    owen-d committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    9a7eaa5 View commit details
    Browse the repository at this point in the history

  2. ingester metrics struct

    @owen-d
    owen-d committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    30cdbf2 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2020

  1. wal replay locks streamsMtx in instances, adds checkpoint codec

    @owen-d
    owen-d committed Nov 9, 2020
    Configuration menu
    Copy the full SHA
    d723c0b View commit details
    Browse the repository at this point in the history

  2. ingester metrics

    @owen-d
    owen-d committed Nov 9, 2020
    Configuration menu
    Copy the full SHA
    897196e View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2020

  1. checkpointer

    @owen-d
    owen-d committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    8df2598 View commit details
    Browse the repository at this point in the history

  2. WAL checkpoint writer

    @owen-d
    owen-d committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    b5c2ef0 View commit details
    Browse the repository at this point in the history

  3. checkpointwriter can write multiple checkpoints

    @owen-d
    owen-d committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    60ebe5e View commit details
    Browse the repository at this point in the history

  4. reorgs checkpointing

    @owen-d
    owen-d committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    ed14e0c View commit details
    Browse the repository at this point in the history

  5. wires up checkpointwriter to wal

    @owen-d
    owen-d committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    5acf745 View commit details
    Browse the repository at this point in the history

  6. ingester SeriesIter impl

    @owen-d
    owen-d committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    6dd28b1 View commit details
    Browse the repository at this point in the history

  7. wires up ingesterRecoverer to consume checkpoints

    @owen-d
    owen-d committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    a7e40fd View commit details
    Browse the repository at this point in the history

  8. generic recovery fn

    @owen-d
    owen-d committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    957717b View commit details
    Browse the repository at this point in the history

  9. generic recovery fn

    @owen-d
    owen-d committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    9384e98 View commit details
    Browse the repository at this point in the history

  10. recover from both wal types

    @owen-d
    owen-d committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    a1beb69 View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2020

  1. cleans up old tmp checkpoints & allows aborting in flight checkpoints

    @owen-d
    owen-d committed Nov 12, 2020
    Configuration menu
    Copy the full SHA
    4e27f7c View commit details
    Browse the repository at this point in the history

  2. wires up wal checkpointing

    @owen-d
    owen-d committed Nov 12, 2020
    Configuration menu
    Copy the full SHA
    45840bc View commit details
    Browse the repository at this point in the history

  3. more granular wal logging

    @owen-d
    owen-d committed Nov 12, 2020
    Configuration menu
    Copy the full SHA
    d1b7f6c View commit details
    Browse the repository at this point in the history

  4. fixes off by 1 wal truncation & removes double logging

    @owen-d
    owen-d committed Nov 12, 2020
    Configuration menu
    Copy the full SHA
    2ca359d View commit details
    Browse the repository at this point in the history

  5. adds userID to wal records correctly

    @owen-d
    owen-d committed Nov 12, 2020
    Configuration menu
    Copy the full SHA
    ff6b35b View commit details
    Browse the repository at this point in the history

  6. wire chunk encoding tests

    @owen-d
    owen-d committed Nov 12, 2020
    Configuration menu
    Copy the full SHA
    d811aef View commit details
    Browse the repository at this point in the history

  7. more granular wal metrics

    @owen-d
    owen-d committed Nov 12, 2020
    Configuration menu
    Copy the full SHA
    6eb4ec0 View commit details
    Browse the repository at this point in the history

  8. checkpoint encoding test

    @owen-d
    owen-d committed Nov 12, 2020
    Configuration menu
    Copy the full SHA
    7962164 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2020

  1. Merge remote-tracking branch 'upstream/master' into wal/writes

    @owen-d
    owen-d committed Nov 13, 2020
    Configuration menu
    Copy the full SHA
    1961726 View commit details
    Browse the repository at this point in the history

  2. ignores debug bins

    @owen-d
    owen-d committed Nov 13, 2020
    Configuration menu
    Copy the full SHA
    67f298d View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'upstream/master' into wal/writes

    @owen-d
    owen-d committed Nov 13, 2020
    Configuration menu
    Copy the full SHA
    d4318c7 View commit details
    Browse the repository at this point in the history

  4. segment replay ignores out of orders

    @owen-d
    owen-d committed Nov 13, 2020
    Configuration menu
    Copy the full SHA
    d43ecde View commit details
    Browse the repository at this point in the history

  5. fixes bug between WAL reading []byte validity and proto unmarshalling… 

    … refs
    @owen-d
    owen-d committed Nov 13, 2020
    Configuration menu
    Copy the full SHA
    ce72f33 View commit details
    Browse the repository at this point in the history

  6. conf validations, removes comments

    @owen-d
    owen-d committed Nov 13, 2020
    Configuration menu
    Copy the full SHA
    c9614aa View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2020

  1. flush on shutdown config

    @owen-d
    owen-d committed Nov 18, 2020
    Configuration menu
    Copy the full SHA
    571f8fc View commit details
    Browse the repository at this point in the history

  2. POST /ingester/shutdown

    @owen-d
    owen-d committed Nov 18, 2020
    Configuration menu
    Copy the full SHA
    2c965f3 View commit details
    Browse the repository at this point in the history

  3. renames flush on shutdown

    @owen-d
    owen-d committed Nov 18, 2020
    Configuration menu
    Copy the full SHA
    1c4d526 View commit details
    Browse the repository at this point in the history

  4. wal & checkpoint use same segment size

    @owen-d
    owen-d committed Nov 18, 2020
    Configuration menu
    Copy the full SHA
    d9139de View commit details
    Browse the repository at this point in the history

Commits on Nov 19, 2020

  1. Merge remote-tracking branch 'upstream/master' into wal/writes

    @owen-d
    owen-d committed Nov 19, 2020
    Configuration menu
    Copy the full SHA
    9a4c810 View commit details
    Browse the repository at this point in the history

  2. writes entries to wal regardless of tailers

    @owen-d
    owen-d committed Nov 19, 2020
    Configuration menu
    Copy the full SHA
    bd9cee0 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2020

  1. makes wal checkpoing duration default to 5m

    @owen-d
    owen-d committed Nov 20, 2020
    Configuration menu
    Copy the full SHA
    e042812 View commit details
    Browse the repository at this point in the history

  2. recovery metrics

    @owen-d
    owen-d committed Nov 20, 2020
    Configuration menu
    Copy the full SHA
    f632022 View commit details
    Browse the repository at this point in the history

  3. encodes headchunks separately for wal purposes

    @owen-d
    owen-d committed Nov 20, 2020
    Configuration menu
    Copy the full SHA
    a948785 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2020

  1. merge upstream

    @owen-d
    owen-d committed Nov 23, 2020
    Configuration menu
    Copy the full SHA
    2e972c7 View commit details
    Browse the repository at this point in the history

  2. typo

    @owen-d
    owen-d committed Nov 23, 2020
    Configuration menu
    Copy the full SHA
    43d8a78 View commit details
    Browse the repository at this point in the history

  3. linting

    @owen-d
    owen-d committed Nov 23, 2020
    Configuration menu
    Copy the full SHA
    3166183 View commit details
    Browse the repository at this point in the history

  4. adds chunkMtx to ingester streams

    @owen-d
    owen-d committed Nov 23, 2020
    Configuration menu
    Copy the full SHA
    9474697 View commit details
    Browse the repository at this point in the history

  5. all flush locking locks streams -> chunks in order to prevent deadlocks

    @owen-d
    owen-d committed Nov 23, 2020
    Configuration menu
    Copy the full SHA
    9e97f48 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2020

  1. Merge remote-tracking branch 'upstream/master' into wal/chunks-mtx

    @owen-d
    owen-d committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    6a35f91 View commit details
    Browse the repository at this point in the history

  2. wal integration tests

    @owen-d
    owen-d committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    697ee85 View commit details
    Browse the repository at this point in the history

Commits on Nov 25, 2020

  1. addresses pr feedback 

    uses entry pool in stream push/tailer

removes unnecessary pool interaction

checkpointbytes comment

fillchunk helper, record resetting in tests via pool

redundant comment

defers wg done in recovery

s/num/count/

checkpoint wal uses a logger

encodeWithTypeHeader now creates its own []byte

removes pool from decodeEntries

wal stop can error
    @owen-d
    owen-d committed Nov 25, 2020
    Configuration menu
    Copy the full SHA
    ee38a1f View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2020

  1. Merge remote-tracking branch 'upstream/master' into wal/chunks-mtx

    @owen-d
    owen-d committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    a60252b View commit details
    Browse the repository at this point in the history

  2. cleans up merge

    @owen-d
    owen-d committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    844e0ba View commit details
    Browse the repository at this point in the history

  3. lint

    @owen-d
    owen-d committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    fd83964 View commit details
    Browse the repository at this point in the history

  4. inline functions to simplify unlocking

    @owen-d
    owen-d committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    d89431a View commit details
    Browse the repository at this point in the history