Skip to content
Commits on Feb 19, 2011
  1. @lelutin

    doc/import-rsnapshot: small corrections and clarification

    lelutin committed with
    There's a typo in the --dry-run option explanation.
    
    The form "[...] or only imports all [...]" is confusing. Turn it around
    a little bit so that the quantifiers are associated more easily to the
    right portions of the sentence.
    
    Also, add an example for using the backuptarget argument.
    
    Signed-off-by: Gabriel Filion <lelutin@gmail.com>
Commits on Feb 18, 2011
  1. cmd/midx, git.py: all else being equal, delete older midxes first.

    committed
    Previous runs of 'bup midx -f' might have created invalid midx files with
    exactly the same length as a newer run.  bup's "prune redundant midx" logic
    would quasi-randomly choose one or the other to delete (based on
    alphabetical order of filenames, basically) and sometimes that would be the
    new one, not the old one, so the 'bup midx -f' results never actually kicked
    in.
    
    Now if the file sizes are equal we'll use the mtime as a tie breaker; newer
    is better.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  2. t/test.sh: a test for the recently-uncovered midx4 problem.

    committed
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  3. _helpers.c: midx4 didn't handle multiple index with the same object.

    committed
    It *tried* to handle it, but would end up with a bunch of zero entries at
    the end, which prevents .exists() from working correctly in some cases.
    
    In midx2, it made sense to never include the same entry twice, because the
    only informatin we had about a particular entry was that it existed.  In
    midx4 this is no longer true; we might sometimes want to know *all* the idx
    files that contain a particular object (for example, when we implement
    expiry later).  So the easiest fix for this bug is to just include multiple
    entries when we have them.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  4. cmd/midx: add a --check option.

    committed
    Running this on my system does reveal that some objects return
    exists()==False on my midx even though they show up during iteration.
    
    Now to actually find and fix it...
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  5. Add git.shorten_hash(), printing only the first few bytes of a sha1.

    committed
    The full name is rarely needed and clutters the output.  Let's try this
    instead in a few places.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  6. tclient.py: add some additional tests that objcache.refresh() is called.

    committed
    ...which it is, so no bugs were fixed here.  Aneurin is sitll exposing a bug
    somehow though.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
Commits on Feb 17, 2011
  1. cmd/server: add a debug message saying which object caused a suggestion.

    committed
    Let's use this to try to debug Aneurin's problem (and potentially others).
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  2. cmd/list-idx: a quick tool for searching the contents of idx/midx files.

    committed
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  3. Add tests around the bloom ruin and check options

    Brandon Low committed with
    This generally improves our test coverage of bloom filter behavior and
    more specifically makes sure that check and ruin do something.
    
    Signed-off-by: Brandon Low <lostlogic@lostlogicx.com>
  4. Add a bloom --ruin for testing failure cases

    Brandon Low committed with
    This command option ruins a bloom filter by setting all of its bits to
    zero.
    
    Signed-off-by: Brandon Low <lostlogic@lostlogicx.com>
  5. One more constant for header lengths

    Brandon Low committed with
    I missed bloom header length in the last pass.
    
    Signed-off-by: Brandon Low <lostlogic@lostlogicx.com>
  6. Split PackMidx from git.py into a new midx.py.

    committed
    git.py is definitely too big.  It still is, but this helps a bit.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  7. bloom.py: move bloom.ShaBloom.create to just bloom.create.

    committed
    I don't really like class-level functions.  Ideally we'd just move all the
    creation stuff into cmd/bloom, but tbloom.py is testing them, so it's not
    really worth it.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  8. Merge branch 'bl/bloomcheck' into ap/cleanups

    committed
    * bl/bloomcheck:
      Bail out immediately instead of redownloading .idx
      Add a --check behavior to verify bloom
      Defines/preprocessor lengths > magic numbers
    
    Conflicts:
    	cmd/bloom-cmd.py
  9. Move bloom-related stuff from git.py to a new bloom.py.

    committed
    No other functionality changes other than that cmd/memtest now reports the
    number of bloom steps separately from the midx/idx steps.  (This is mostly
    so they don't have to share the same global variables, but it's also
    interesting information to break out.)
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  10. cmd/bloom: add a --force option to forget regenerating the bloom.

    committed
    This corresponds to midx's --force option.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  11. Use the new qprogress() function in more places.

    committed
    qprogress() was introduced in the last commit and has smarter default
    behaviour that automatically reduces progress output so we don't print too
    many messages per second.  Various commands/etc were doing this in various
    different ad-hoc ways, but let's centralize it all in one place.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  12. Bail out immediately instead of redownloading .idx

    Brandon Low committed with
    This should make diagnosing / fixing corrupted bloom filters and midx
    files easier, and is generally more sane behavior.
    
    Signed-off-by: Brandon Low <lostlogic@lostlogicx.com>
  13. Add a --check behavior to verify bloom

    Brandon Low committed with
    This new behavior is useful when diagnosing weird behavior, lets a bloom
    filter claiming to contain a particular idx be verified against that idx
    file.
    
    Signed-off-by: Brandon Low <lostlogic@lostlogicx.com>
  14. Defines/preprocessor lengths > magic numbers

    Brandon Low committed with
    This just changes some instances of "8", "12" and "20" to use the
    equivalent sizeof or #defined constants to make the code more readable.
    
    Signed-off-by: Brandon Low <lostlogic@lostlogicx.com>
  15. cmd/{bloom,midx}: clean up progress messages.

    committed
    bloom was printing messages more often than necessary on fast computers,
    which could overwhelm the stderr output a bit.  Also change to a percentage
    + number of objects, like midx and save do, rather than just printing the
    current file number.
    
    And don't print so many lines of output by default: now if bloom isn't
    end up doing anything, it doesn't print any output.  And if it does do
    something, it prints only one output line per file.
    
    bloom and midx now both print the name of the directory where they're
    creating their output files; if you have multiple directories in
    .bup/index-cache, it was a little confusing to see them doing
    multiple runs for no apparent reason.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  16. cmd/bloom: by default generate bloom filters in *all* index-cache dirs.

    committed
    This matches with 'bup midx -a' and 'bup midx -f' behaviour.  People might
    have been thinking they were regenerating bloom filters without actually
    doing them all.
    
    Moved some shared code to do this from cmd/midx to git.py.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
Commits on Feb 16, 2011
  1. cmd/newliner: avoid printing a blank line if the final output ended i…

    committed
    …n \r.
    
    If the last output was a progress message, we would blank out the line
    (which was correct) but then we'd print a newline, which was wrong.  Only
    print the leftover output, followed by a newline, if the last output was
    nonempty.
    
    'bup midx' suffered from this.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  2. cmd/index: make the default mode '-u'.

    committed
    I always forget the -u option, and it's by far the most common thing to do
    with 'bup index'.  So if no mode option is provided, just default to that
    one.
    
    While we're here, update the man page and usage message a bit.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  3. _helpers.c: don't cygwin doesn't set any win32 defines.

    committed
    ...so let's #ifdef for cygwin specifically.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  4. _helpers.c: don'g unpythonize_argv() on win32.

    committed
    Py_GetArgcArgv() doesn't exist on win32 platforms.  Which isn't so bad,
    since neither does the 'ps' command, really.
    
    Reported by Aneurin Price.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
Commits on Feb 15, 2011
  1. Remove .c and .o rules, apply flags to csetup.py

    Brandon Low committed with
    The .c and .o rules were not used and were misleadingly implying that we
    were already paying attention to LDFLAGS and CFLAGS.  Instead apply the
    flags to csetup.py where they will actually do something.
    
    Signed-off-by: Brandon Low <lostlogic@lostlogicx.com>
  2. Fix a valid warning that some compilers warned

    Brandon Low committed with
    And it's a good idea not to ignore fwrite's return value I suppose.
    
    Signed-off-by: Brandon Low <lostlogic@lostlogicx.com>
Commits on Feb 14, 2011
  1. Move .idx file writing to C

    Brandon Low committed with
    This was a remaining CPU bottleneck in bup-dumb-server mode.  In a quick
    test, writing 10 .idx files of 100000 elements on my netbook went from
    50s to 4s.  There may be more performance available by adjusting the
    definition of the PackWriter.idx object, but list(list(tuple)) isn't
    bad.
    
    Signed-off-by: Brandon Low <lostlogic@lostlogicx.com>
Commits on Feb 13, 2011
  1. main.py: fix whitespace in the usage string.

    committed
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  2. cmd/daemon: FD_CLOEXEC the listener socket and don't leak fd for the …

    committed
    …connection.
    
    Otherwise the listener gets inherited by all the child processes (mostly
    harmless) and subprograms run by bup-server inherit an extra fd for the
    connection socket (problematic since we want the connection to close as soon
    as bup-server closes).
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  3. cmd/daemon: close file descriptors correctly in parent process.

    committed
    The client wasn't getting disconnected when the server died, because the
    daemon was still hanging on to its copy of the original socket, due to some
    misplaced os.dup() calls.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  4. cmd/daemon: use SO_REUSEADDR.

    committed
    Otherwise we can't re-listen on that socket until the TIME_WAIT period ends,
    under certain conditions.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
  5. cmd/daemon: pass extra options along to 'bup server'.

    committed
    Currently 'bup server' doesn't take any options, but that might change
    someday.
    
    Also use a '--' to separate the bup mux command from its arguments, so it
    doesn't accidentally try to parse them.  This didn't matter before (since
    none of the options we were passing along started with a dash) but if the
    user provides extra options, it might matter.
    
    Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
Something went wrong with that request. Please try again.