New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Viability of having source input from FIFO #59

Closed
GoogleCodeExporter opened this Issue Mar 24, 2015 · 6 comments

Comments

Projects
None yet
1 participant
@GoogleCodeExporter

GoogleCodeExporter commented Mar 24, 2015

I have two ~300MB .7zipped files with database dumps that, if
unzipped, would become ~30G in size each. Given that, I'd like
to avoid having to uncompress them, so I did this:

mkfifo first
mkfifo second
7z e -so first.7z       > first   &
7z e -so second.7z      > second  &
xdelta3 -s first second > delta    # delta turns out ~8GiB

But what I get are tons of warning saying  "input
window xxxx....yyyyy has no source copies'. The delta
gets generated, but they are two orders of magnitude
larger than what I get if I do:

7z e -so first.7z   > realfirst          # realfirst is ~30GiB
7z e -so second.7z  > realsecond         # realsecond is also ~30GiB
xdelta3 -s realfirst realsecond > delta  # delta comes at about 30MiB

The 'memory tuning' page on xdelta's web site says that
the input buffer allows the data to come sequentially without
seeking backwards and the source code also mentions something
about that and the possibility of the inputs coming from fifos,
so I'd guess it should be possible at least in principle.

So my question is: how viable would it be to add support for having
the file specified in the -s parameter to come from a FIFO at least
for the delta generation operation?

Original issue reported on code.google.com by kikocar...@gmail.com on 19 Dec 2007 at 6:18

@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

Your first example seems to be quite dangerous. :)

It's true that the encoder reads sequentially, so this should be possible.

The only thing is that it uses the source size to determine some parameters. 
However, this is not a real obstacle.  So I'll say that this isn't very hard and
should be done. 

Original comment by josh.mac...@gmail.com on 19 Dec 2007 at 6:51

  • Changed state: Accepted
  • Added labels: Type-Enhancement
  • Removed labels: Type-Defect

GoogleCodeExporter commented Mar 24, 2015

Your first example seems to be quite dangerous. :)

It's true that the encoder reads sequentially, so this should be possible.

The only thing is that it uses the source size to determine some parameters. 
However, this is not a real obstacle.  So I'll say that this isn't very hard and
should be done. 

Original comment by josh.mac...@gmail.com on 19 Dec 2007 at 6:51

  • Changed state: Accepted
  • Added labels: Type-Enhancement
  • Removed labels: Type-Defect
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

Can it read sequentially from both the source and the target? I had a strange 
idea 
that it read sequentially from one and did random access on the other.

Original comment by nicolas....@gmail.com on 30 Sep 2008 at 6:38

GoogleCodeExporter commented Mar 24, 2015

Can it read sequentially from both the source and the target? I had a strange 
idea 
that it read sequentially from one and did random access on the other.

Original comment by nicolas....@gmail.com on 30 Sep 2008 at 6:38

@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

I want to work on this issue--it's the most asked for by e-mail.

But, it's difficult!  There are assumptions here and there about knowing the 
source
size up front...  I'll keep at it.

Original comment by josh.mac...@gmail.com on 24 Oct 2009 at 1:31

GoogleCodeExporter commented Mar 24, 2015

I want to work on this issue--it's the most asked for by e-mail.

But, it's difficult!  There are assumptions here and there about knowing the 
source
size up front...  I'll keep at it.

Original comment by josh.mac...@gmail.com on 24 Oct 2009 at 1:31

@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

Progress: I removed all the stream->source->size dependencies with an API 
change,
added a bunch of new assertions.  Subsequently, all tests are broken.

Original comment by josh.mac...@gmail.com on 24 Oct 2009 at 4:32

GoogleCodeExporter commented Mar 24, 2015

Progress: I removed all the stream->source->size dependencies with an API 
change,
added a bunch of new assertions.  Subsequently, all tests are broken.

Original comment by josh.mac...@gmail.com on 24 Oct 2009 at 4:32

@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

SVN 292 allows source to be taken from a fifo.
Might need a little more testing, but tests are passing.

Original comment by josh.mac...@gmail.com on 26 Oct 2009 at 2:45

  • Changed state: Started

GoogleCodeExporter commented Mar 24, 2015

SVN 292 allows source to be taken from a fifo.
Might need a little more testing, but tests are passing.

Original comment by josh.mac...@gmail.com on 26 Oct 2009 at 2:45

  • Changed state: Started
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

Original comment by josh.mac...@gmail.com on 26 Oct 2009 at 6:24

  • Changed state: Fixed

GoogleCodeExporter commented Mar 24, 2015

Original comment by josh.mac...@gmail.com on 26 Oct 2009 at 6:24

  • Changed state: Fixed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment