Permalink
Commits on Apr 3, 2014
  1. Builtin collectors need --no-clean too

    Shawn Halpenny committed Apr 3, 2014
Commits on Apr 2, 2014
  1. Add --no-clean flag

    - useful for debugging if you want to hang on to all input and output
    Shawn Halpenny committed Apr 2, 2014
  2. Make tempdir universally readable

    - this simplifies debugging tremendously
    Shawn Halpenny committed Apr 2, 2014
  3. Fix typos

    Shawn Halpenny committed Apr 2, 2014
Commits on Mar 16, 2012
  1. Add --compress to collector.

    - results in a bzip2-format compressed output file (since bzip2 chunks can
      be concatenated in their compressed form)
    committed Mar 16, 2012
Commits on Aug 1, 2010
Commits on Jul 23, 2010
  1. Fix typo.

    committed Jul 23, 2010
  2. Make sure ll-pipe tools can find themselves.

    They're always relateive to the directory of ll-pipe.  Don't use ${0:h}
    because that may take on a different value depending if a file was sourced
    or not (as in the case for ll-pipe-fns).
    
    Also make sure enableDebug is only called once it has been defined by
    sourcing ll-pipe-fns.
    committed Jul 23, 2010
  3. Redirect fd 3 to stderr if not yet open.

    Redirecting to /dev/tty was bad because if you didn't launch ll-pipe from a
    terminal, anything writing to fd 3 would fail.  This ensures something will
    at least work.
    committed Jul 23, 2010
Commits on Jul 14, 2010
  1. Verbose messaging includes total chunk count.

    As soon as all input is split, the total chunk count is known.  All verbose
    chunk processing messages will include the count, showing ??? if it is not
    yet known.
    committed Jul 14, 2010
  2. Remove unneeded comment.

    Files passed in will be readable because only readable files are passed in.
    committed Jul 14, 2010
  3. Improve collector processing.

    - properly escape collector command lines.
    - verbose messaging when skipping collection for a chunk
    - remove unneeded .done checking in ll-pipe-collect-by-suffix because the
      only files it receives are files that .done exists for.
    committed Jul 14, 2010
Commits on Jul 12, 2010
  1. Force stdout flush after printing each chunk name.

    This lets xargs start jobs on chunks as soon as they are available, rather
    than waiting for a bunch to be batched up.
    committed Jul 12, 2010
Commits on Jul 10, 2010
  1. Update README.

    ll-pipe is now self-contained, with no dependencies on shell functions
    defined externally.
    committed Jul 10, 2010
Commits on Jul 9, 2010
  1. Empty meta collector output file before writing.

    Since collectors append to their output file, the meta data output file
    needs to be emptied prior to anything being written to it.
    committed Jul 9, 2010
  2. Add more verbose messaging.

    committed Jul 9, 2010
  3. Better verbosity implementation.

    Route verbose output to fd #3, redirected to /dev/tty if not already
    redirected.  This prevents intermingling ll-pipe status messages with
    stdout/stderr from the parallel processes.
    committed Jul 9, 2010
Commits on Jul 8, 2010
  1. Always define common functions.

    By only defining the function if it does not exist, we risk using the
    function from the user's environment, which means we can't be certain about
    the results of the function.
    committed Jul 8, 2010
  2. No more JSON for meta information.

    The extra dependency on perl's JSON::XS module serves no real purpose.  The
    metadata output format can be easily parsed to JSON if necessary by some
    other external process.
    committed Jul 8, 2010
  3. Report jsonify failures to caller.

    Discovered that if your env is missing the JSON::XS perl module, metadata
    post-processing will not be done, stderr for each chunk will contain the
    failure messages from perl saying it can't find the module, but the chunks
    will still succeed.
    
    This change makes jsonifying failures be fatal for the chunk.
    
    I'm going to remove the JSON stuff anyway because the metadata can be
    post-processed from whatever form it gets written in as long as I choose
    something sane to parse.
    committed Jul 8, 2010
  4. Remove RCS keywords.

    committed Jul 8, 2010
  5. Support --verbose.

    The only messages output by ll-pipe and children will be verbose or error
    messages.
    committed Jul 8, 2010
  6. Define previously-implied shell functions.

    Supply definitions for shell functions that were only defined in my
    environment.
    committed Jul 8, 2010
Commits on Jun 21, 2010
  1. Correct comment.

    There is no second parameter to jsonifyFile.
    committed Jun 21, 2010
  2. Don't cd into temporary dir.

    Doing so means that callers have to be aware that any relative paths to
    tools (like collectors) that they use will likely not work.  By leaving the
    pwd unchanged, this surprise is avoided and it also means the tool works
    like other pipe-friendly tools as you'd expect.
    committed Jun 21, 2010