Permalink
Commits on Jan 10, 2010
  1. Oops, 'bup save /' produced an invalid tree.

    apenwarr committed Jan 10, 2010
    Add a bunch of assertions to make sure that never happens.
  2. This adds the long-awaited indexfile feature, so you no longer have t…

    apenwarr committed Jan 10, 2010
    …o feed
    
    your backups through tar.
    
    Okay, 'bup save' is still a bit weak... but it could be much worse.
    
    Merge branch 'indexfile'
    
    * indexfile:
      Minor fix for python 2.4.4 compatibility.
      cmd-save: completely reimplement using the indexfile.
      Moved some reusable index-handling code from cmd-index.py to index.py.
      A bunch of wvtests for the 'bup index' command.
      Start using wvtest.sh for shell-based tests in test-sh.
      cmd-index: default indexfile path is ~/.bup/bupindex, not $PWD/index
      cmd-index: skip merging the index if nothing was written to the new one.
      cmd-index: only update if -u is given; print only given file/dirnames.
      cmd-index: correct reporting of deleted vs. added vs. modified status.
      Generalize the multi-index-walking code.
      cmd-index: indexfiles should start with a well-known header.
      cmd-index: eliminate redundant paths from index update command.
      cmd-index: some handy options.
      index: add --xdev (--one-file-system) option.
      Fix some bugs with indexing '/'
      cmd-index: basic index reader/writer/merger.
  3. cmd-save: completely reimplement using the indexfile.

    apenwarr committed Jan 10, 2010
    'bup save' no longer walks the filesystem: instead it walks the indexfile
    (which is much faster) and doesn't bother opening any files that haven't had
    an attribute change, since it can just reuse their sha1 from before.  That
    makes it *much* faster in the common case.
  4. Start using wvtest.sh for shell-based tests in test-sh.

    apenwarr committed Jan 10, 2010
    This makes the output a little prettier... at least in the common case where
    it passes :)
  5. cmd-index: only update if -u is given; print only given file/dirnames.

    apenwarr committed Jan 10, 2010
    cmd-index now does two things:
     - it updates the index with the given names if -u is given
     - it prints the index if -p, -s, or -m are given.
    
    In both cases, if filenames are given, it operates (recursively) on the
    given filenames or directories.  If no filenames are given, -u fails (we
    don't want to default to /; it's too slow) but -p/s/m just prints the whole
    index.
  6. cmd-index: correct reporting of deleted vs. added vs. modified status.

    apenwarr committed Jan 10, 2010
    A file with an all-zero sha1 is considered Added instead of Modified, since
    it has obviously *never* had a valid sha1.  (A modified file has an old
    sha1, but IX_HASHVALID isn't set.)
    
    We also now don't remove old files from the index - for now - so that we can
    report old files with a D status.  This might perhaps be useful eventually.
    
    Furthermore, we had a but where reindexing a particular filename would
    "sometimes" cause siblings of that file to be marked as deleted.  The
    sibling entries should never be updated, because we didn't check them and
    thus have no idea of their new status.  This bug was mostly caused by the
    silly way we current pass dirnames and filenames around...
Commits on Jan 9, 2010
  1. Generalize the multi-index-walking code.

    apenwarr committed Jan 9, 2010
    Now you can walk through multiple indexes correctly from anywhere, avoiding
    the need for merging a huge index just to update a few files.
Commits on Jan 8, 2010
  1. cmd-index: eliminate redundant paths from index update command.

    apenwarr committed Jan 7, 2010
    If someone asks to update "/etc" and "/etc/passwd", the latter is redundant
    because it's included in the first.  Don't bother updating the file twice
    (and thus causing two index merges, etc).
    
    Ideally we would only do one merge for *any* number of updates (etc /etc and
    /var).  This should be possible as long as we sort the entries correctly
    (/var/ and then /etc/), since a single sequential indexfile could just have
    one appended to the other.  But we don't do that yet.
  2. cmd-index: some handy options.

    apenwarr committed Jan 7, 2010
    New options:
    	--modified: print only files that aren't up to date
    	--status: prefix printouts with status chars
    	--fake-valid: mark all entries as up to date
    	--indexfile: override the default index filename
  3. index: add --xdev (--one-file-system) option.

    apenwarr committed Jan 7, 2010
    For not traversing across filesystem boundaries.
Commits on Jan 7, 2010
  1. On MacOS X, "wc -l" returns extra whitespace.

    apenwarr committed Jan 7, 2010
    ie. "    0" instead of "0".
    
    The easiest workaround is to compare as a number instead of as a string.
    This seems to work correctly on both MacOS and Linux.
  2. More compile options for MacOS X.

    apenwarr committed Jan 7, 2010
    Based on a patch from Dave Coombs.  I changed it to auto-detect the OS
    platform, so I might have broken it, however.
Commits on Jan 6, 2010
  1. splitting to a remote server would cause "already busy" errors.

    apenwarr committed Jan 6, 2010
    Specifically:
    	client.ClientError: already busy with command 'receive-objects'
    
    That's because recent changes removed the call to onclose() from
    PackWriter_Remote.  Now it's back, plus I added an extra unit test to reveal
    the problem.
  2. client: enhance the PATH when searching for the 'bup' binary.

    apenwarr committed Jan 6, 2010
    Automatically adds the *local* $PWD to the *remote* $PATH before trying to
    run 'bup server'.  That way, if you build the source in exactly the same
    folder on two machines - or if those two machines are actually the same
    machine and you're just doing a test against localhost - it'll work.
    
    I hereby curse both "sh -c <command>" and "ssh hostname -- <command>" for
    not allowing a sensible way to just set argv[] without doing any stupid
    quoting.  Nasty.
  3. Much more user-friendly error messages when bup can't exec the server.

    apenwarr committed Jan 6, 2010
    ...which happens unfortunately often, including in 'make test' when PATH
    doesn't include bup.  I'll fix that next.  But it makes sense to fix the
    error messages first :)
  4. Add a 'make stupid' target that does 'make test' with a minimal PATH.

    apenwarr committed Jan 6, 2010
    Because I'm stupid and I keep forgetting to test what happens if you don't
    have 'bup' in your PATH.
    
    Thanks to Dave Coombs and Andy Chong for reporting the problem.  And in
    v0.01 too.
  5. split: Prevent memory drain from excessively long shalists.

    apenwarr committed Jan 6, 2010
    This avoids huge RAM usage when you're splitting a really huge object, plus
    git probably doesn't work too well with single trees that contain millions
    of objects anyway.
  6. Split packs around 100M objects or 1G bytes.

    apenwarr committed Jan 6, 2010
    This will make pruning much easier later, plus avoids any problems with
    packs >= 2GB (not that we've had any of those yet, but...), plus avoids
    wasting RAM with an overly full MultiPackIndex.also{} dictionary.
  7. OOPS! Was writing one byte at a time to the server.

    apenwarr committed Jan 6, 2010
    _raw_write() expects a list, not a string, so it was iterating over it
    character by character.  Magically it worked anyway.  Which is sort of cool,
    and yet not.
  8. Fix compatibility with git 1.5.4.3 (Ubuntu Hardy).

    apenwarr committed Jan 6, 2010
    Thanks to Andy Chong for reporting the problem.
    
    Basically it comes down to two things that are missing in that version but
    exist in git 1.5.6:
    
      - git init --bare doesn't work, but git --bare init does.
      - git cat-file --batch doesn't exist in that version.
    
    Unfortunately, the latter problem is pretty serious; bup join is really slow
    without it.  I guess it might be time to implement an internal version of
    cat-file.
  9. Older git needs 'git --bare init' instead of 'git init --bare'

    apenwarr committed Jan 6, 2010
    Needed in at least 1.5.4 (Ubuntu Hardy), but not 1.5.6 (Debian Lenny).
Commits on Jan 5, 2010
  1. test-sh: don't truncate stderr.

    apenwarr committed Jan 5, 2010
    Normally you wouldn't notice this problem, unless you tried to run:
    
    	make test >&test.out
    
    and found that it was full of NUL characters.  Oops.
  2. Make now calls python2.5-config.

    David Wolever committed Jan 5, 2010