Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Viability of having source input from FIFO #59

GoogleCodeExporter opened this issue Mar 24, 2015 · 6 comments

Viability of having source input from FIFO #59

GoogleCodeExporter opened this issue Mar 24, 2015 · 6 comments


Copy link

I have two ~300MB .7zipped files with database dumps that, if
unzipped, would become ~30G in size each. Given that, I'd like
to avoid having to uncompress them, so I did this:

mkfifo first
mkfifo second
7z e -so first.7z       > first   &
7z e -so second.7z      > second  &
xdelta3 -s first second > delta    # delta turns out ~8GiB

But what I get are tons of warning saying  "input
window xxxx....yyyyy has no source copies'. The delta
gets generated, but they are two orders of magnitude
larger than what I get if I do:

7z e -so first.7z   > realfirst          # realfirst is ~30GiB
7z e -so second.7z  > realsecond         # realsecond is also ~30GiB
xdelta3 -s realfirst realsecond > delta  # delta comes at about 30MiB

The 'memory tuning' page on xdelta's web site says that
the input buffer allows the data to come sequentially without
seeking backwards and the source code also mentions something
about that and the possibility of the inputs coming from fifos,
so I'd guess it should be possible at least in principle.

So my question is: how viable would it be to add support for having
the file specified in the -s parameter to come from a FIFO at least
for the delta generation operation?

Original issue reported on by on 19 Dec 2007 at 6:18

Copy link

Your first example seems to be quite dangerous. :)

It's true that the encoder reads sequentially, so this should be possible.

The only thing is that it uses the source size to determine some parameters. 
However, this is not a real obstacle.  So I'll say that this isn't very hard and
should be done. 

Original comment by on 19 Dec 2007 at 6:51

  • Changed state: Accepted
  • Added labels: Type-Enhancement
  • Removed labels: Type-Defect

Copy link

Can it read sequentially from both the source and the target? I had a strange 
that it read sequentially from one and did random access on the other.

Original comment by on 30 Sep 2008 at 6:38

Copy link

I want to work on this issue--it's the most asked for by e-mail.

But, it's difficult!  There are assumptions here and there about knowing the 
size up front...  I'll keep at it.

Original comment by on 24 Oct 2009 at 1:31

Copy link

Progress: I removed all the stream->source->size dependencies with an API 
added a bunch of new assertions.  Subsequently, all tests are broken.

Original comment by on 24 Oct 2009 at 4:32

Copy link

SVN 292 allows source to be taken from a fifo.
Might need a little more testing, but tests are passing.

Original comment by on 26 Oct 2009 at 2:45

  • Changed state: Started

Copy link

Original comment by on 26 Oct 2009 at 6:24

  • Changed state: Fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

No branches or pull requests

1 participant