Skip to content
Permalink
Branch: master
Commits on Dec 4, 2019
  1. cut compaction outputs at grandparent boundaries

    ajkr committed Nov 28, 2019
    Previously range tombstones could cause output files that would later
    undergo excessively large compactions due to overlap with the
    grandparent level. This happened because only point keys were considered
    for cutting output files according to grandparent overlap. Thus if a
    range tombstone extended over a region devoid of point keys, compaction
    would never split files within that region.
    
    To fix this we now split output files at grandparent boundaries
    regardless of the presence of point keys. That means a region covered by
    a range tombstone and devoid of point keys can be split at grandparent
    boundaries if needed to prevent future huge compactions.
    
    Fixes #167.
Commits on Dec 2, 2019
  1. replace shouldStopBefore with stateless function

    ajkr committed Nov 28, 2019
    `shouldStopBefore()` was hard to change because it needed to be called
    every time before extending a file's range by adding a point key. But in
    the world of range data (i.e., range tombstones) this no longer makes
    sense because we do not want a simple yes/no answer to whether a point
    key is acceptable to add; instead, we want the exact key at which the
    file should be cut as it may be in the middle of a range.
    
    To provide this capability, this PR replaces `shouldStopBefore()` with a
    function, `findGrandparentLimit()`. It takes the start key of a file and
    returns the latest possible end key that can be used without violating
    grandparent overlap rules. Unlike `shouldStopBefore()`, it does not
    maintain any state about keys seen or files already passed, which I
    believe makes it easier to maintain.
    
    Also changed `finishOutput()` to take user key rather than internal key
    as that's all it needs (and all `findGrandparentLimit()` returns).
  2. pause compactionIter at range tombstone start keys

    ajkr committed Nov 30, 2019
    Previously compactionIter internally added range tombstones to the
    fragmenter as they were seen and then skipped over them. However, this did
    not give the client the option to make decisions based on the range
    tombstone data before adding it to fragmenter. This was inconsistent
    with point keys, which were always returned by compactionIter before
    being added to the output file by `runCompaction()`'s loop.
    
    This PR makes range tombstone start keys "non-skippable", i.e., returned
    by compactionIter as soon as they are seen, even if covered by another
    key in the same snapshot stripe. The ownership of the fragmenter is
    moved into the `compaction` object and it is now responsible for adding
    tombstones after they are returned by the `compactionIter`.
    
    There are also some miscellaneous cleanup changes in compactionIter
    related to making error handling and advancing logic clearer/more
    consistent.
Commits on Nov 26, 2019
  1. Merge pull request #417 from cockroachdb/pmattis/metamorphic

    petermattis committed Nov 26, 2019
    internal/metamorphic: compare output from runs
  2. Merge pull request #418 from cockroachdb/pmattis/archive-cleaner-waldir

    petermattis committed Nov 26, 2019
    internal/metamorphic: exit early on any background error
  3. Merge pull request #419 from cockroachdb/pmattis/history-logging

    petermattis committed Nov 26, 2019
    internal/metamorphic: address a TODO in history logging
  4. internal/metamorphic: address a TODO in history logging

    petermattis committed Nov 26, 2019
    Prefix all lines of `Infof` and `Fatalf` output with `// {INFO,FATAL}`.
  5. internal/metamorphic: exit early on any background error

    petermattis committed Nov 26, 2019
    Fix a bug where we were specifying the wrong path for cleaning when
    `Options.WALDir` is specified.
  6. internal/metamorphic: support SeekPrefixGE

    petermattis committed Nov 26, 2019
    Support `SeekPrefixGE` in the meta test. This is done by specifying a
    `Split` function on the comparer which returns the entire key. Fixed a
    few bits of brokenness around this which were previously preventing it
    from working.
    
    Fixes #409
  7. internal/metamorphic: compare output from runs

    petermattis committed Nov 26, 2019
    Output comparison is currently disabled by default as it is finding
    differences that appear to be due to bugs in Pebble.
    
    Pass `pebble.NoSync` to the various write methods as we're not relying
    on syncing and Pebble returns an error if you try to sync and the WAL is
    disabled.
    
    Fixes #408
Commits on Nov 25, 2019
  1. sstable/block: add comment about restart points and numRestarts

    itsbilal committed Nov 25, 2019
    Add a comment that makes reading / understanding this code and
    the interactions between offsets and restarts easier.
  2. Merge pull request #413 from cockroachdb/pmattis/level-iter

    petermattis committed Nov 25, 2019
    fix levelIter.SeekGE positioning
  3. add EstimateDiskUsage API for range space usage

    ajkr committed Nov 21, 2019
    Introduced a new DB API, `EstimateDiskUsage()`, which takes an
    inclusive-inclusive range of user keys, `[start, end]`, and returns the
    approximate space used in files for storing that range. See comment in
    "db.go" for details on how it is calculated.
  4. sstable: add more commentary around blockIter key stability

    petermattis committed Nov 25, 2019
Commits on Nov 24, 2019
  1. sstable: add commentary on blockIter fields

    petermattis committed Nov 24, 2019
  2. fix levelIter.{SeekGE,SeekLT,First,Last,Next,Prev} positioning

    petermattis committed Nov 21, 2019
    Fix the `levelIter` positioning routines to not move past (or before) an
    sstable which contains the upper/lower bound of iteration and the table
    contains range tombstones. The prior behavior allowed range tombstones
    to be unintentionally skipped.
Commits on Nov 22, 2019
  1. Merge pull request #414 from cockroachdb/pmattis/tool

    petermattis committed Nov 22, 2019
    tool: add --count flag to {db,sstable} scan
Commits on Nov 21, 2019
  1. tool: add --count flag to {db,sstable} scan

    petermattis committed Nov 21, 2019
    Fixes #411
  2. tool: add -v/--verbose to all commands

    petermattis committed Nov 21, 2019
    Fixes #412
  3. fix blockIter.SeekLT to provide the key stability guarantee

    petermattis committed Nov 21, 2019
    `blockIter.SeekLT` was violating the key stability guarantee that a key
    remains valid across positioning calls when the block has a restart
    interval of 1. This behavior is expected by `rangedel.Truncate`, the
    code to transform v1 format range-del blocks to v2 format on the fly,
    and by `rangedel.Seek{GE,LE}`.
    
    The specific problem in `blockIter.SeekLT` was that it was caching one
    entry too many in the common case where it stepped into the next restart
    block. This actually resulted in an extra unnecessary key
    comparison. The comparison was unnecessary because it had already been
    performed by the binary search earlier in the method.
  4. Merge pull request #407 from cockroachdb/pmattis/ingest

    petermattis committed Nov 21, 2019
    fix two ingestion bugs
  5. fix two ingestion bugs

    petermattis committed Nov 21, 2019
    Fix a bug where ingestion was blindly updating the seqnum for the
    largest key in an sstable. This unintentionally extended the bounds for
    an sstable when the largest key was a range tombstone sentinel key. This
    is a minor cosmetic issue as the range tombstone's effect was not
    changed by the extension of the sstable bounds.
    
    Fix a bug where concurrent ingestions could collide in choosing their
    target level. This collision occurred in `DB.ingestApply` which appears
    to have mutual exclusion via `DB.mu`, but that exclusion was being
    thwarted by `versionSet.logAndApply` which releases `DB.mu` during the
    write to the MANIFEST. Internally, `logAndApply` achieves mutual
    exclusion of MANIFEST operations by a separate mechanism. That mechanism
    is now made visible via `versionSet.log{Lock,Unlock}` calls.
Commits on Nov 20, 2019
  1. Merge pull request #400 from cockroachdb/pmattis/level-iter-seek-pref…

    petermattis committed Nov 20, 2019
    …ix-ge
    
    fix levelIter.SeekPrefiGE positioning
  2. fix levelIter.SeekPrefiGE positioning

    petermattis committed Nov 15, 2019
    Fix `levelIter.SeekPrefixGE` to not seek past an sstable which doesn't
    contain the target key if that table does contain range tombstones. The
    prior behavior allowed range tombstones to be unintentionally skipped.
Commits on Nov 19, 2019
  1. Do not recycle WAL files when a cleaner that needs the file contents,

    sumeerbhola committed Nov 18, 2019
    like ArchiveCleaner, is being used.
    
    Fixes #396
Commits on Nov 15, 2019
  1. Merge pull request #397 from cockroachdb/pmattis/db-tool

    petermattis committed Nov 15, 2019
    tool: automatically infer comparer/merger for db commands
  2. Merge pull request #398 from cockroachdb/pmattis/ingest-info-string

    petermattis committed Nov 15, 2019
    change TableIngestInfo.String to a single-line format
  3. change TableIngestInfo.String to a single-line format

    petermattis committed Nov 15, 2019
    `TableIngestInfo.String` was the only event info struct that used a
    multi-line format. A single-line format looks nicer in output to CRDB
    log files. Tweaked the format to include the file numbers of the
    ingested sstables.
  4. tool: automatically infer comparer/merger for db commands

    petermattis committed Nov 14, 2019
    Automatically infer the comparer/merger for the `db` commands by loading
    the `OPTIONS` file. Added support for skipping unknown options and
    sections which was necessary to make this work when running a `db`
    command on a RocksDB generated DB.
    
    Initialize the pretty formatter for `db scan` if the comparer has one
    specified.
    
    Fixes #391
Commits on Nov 13, 2019
  1. levelIter bug fix: Handle the case where index is valid, err is nil

    sumeerbhola committed Nov 13, 2019
    and iter is nil because the file pointed to by index is beyond the
    iteration bounds. This is not an error, and if the bounds change
    and the index does not, we should create an iterator.
    
    Fixes cockroach #42353
  2. Merge pull request #393 from cockroachdb/pmattis/memtable-sizing

    petermattis committed Nov 13, 2019
    only grow the memtable size if we filled it over 50%
  3. Merge pull request #392 from cockroachdb/pmattis/replay-flush

    petermattis committed Nov 13, 2019
    disable zeroing of seqnums when flushing
  4. only grow the memtable size if we filled it over 50%

    petermattis committed Nov 12, 2019
    Tweak the dynamic memtable sizing logic so that we only grow the
    memtable size if we filled it over 50% when a manual flush was
    performed. This reduces memtable memory pressure in workloads which
    frequently manually flush.
Commits on Nov 12, 2019
  1. disable zeroing of seqnums when flushing

    petermattis committed Nov 12, 2019
    RocksDB never zeroes seqnums when flushing. Doing so mostly works except
    for a bad interaction with WAL replaying at startup. While that
    interaction could be fixed, for now it seems safest to match the RocksDB
    behavior.
Older
You can’t perform that action at this time.