Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

in-place zip editing support #38

Open
mingyuan-xia opened this issue Jul 4, 2018 · 1 comment
Open

in-place zip editing support #38

mingyuan-xia opened this issue Jul 4, 2018 · 1 comment

Comments

@mingyuan-xia
Copy link

mingyuan-xia commented Jul 4, 2018

We have been using yazl and yauzl for over a year and we very much like its stability and pure-JS-ness.
Yet, we face quite a lot use cases of editing an zip in-place. For now, we did what's done in #30: create a temp file with yazl, open the original zip with yauzl, transport all entries (and data) to the temp file and overwrite the original one with the temp file. This has quite a few drawbacks:

  1. performance: memory, IO
  2. sometimes it would be tricky to get a temp file path (original_file+'.tmp' / global temp folder are both not reliable for some cases
  3. quite a lot pitfalls (on macOS, empty folder shall be preserved; externalAttributes blablabla) and lengthy non-reusable code snippet.

We are searching for equivalence of zip -d, zip old.zip new.file, zip -ur. Delete, add, update in-place.
Can you share some thoughts?

@thejoshwolfe
Copy link
Owner

thejoshwolfe commented Jul 8, 2018

Editing existing zipfiles is a hard problem. Implementations are faced with lots of trafe offs that I've been trying to avoid. appending a file is fairly simple, but what about deleting the first file in the archive? Some possible approaches are:

  • shift everything else in the file back to make efficient use of space. This is about as expensive as completely recreating the whole archive.
  • leave unused space where the file was. This makes the archive unreadable to many non-conformant zipfile readers, notably including the system Archive Utility on Mac.

And if a file is updated in place, it's compressed size will probably change, which means you have to deal with the above problem for that case too.

The most conservative and compatible way to make zipfiles is to pack everything closely to leave no unused space. That strategy means that in the worst case you're going to be recreating the whole archive anyway. On average, you're probably recreating half of the archive.

The benefits of supporting in place edits seem very small to me, and the complexity of implementing the feature is very high. I don't plan to support it.

We are searching for equivalence of zip -d, zip old.zip new.file, zip -ur.

If those operations give a significant performance increase over the naive approach and the resulting files are readable by Archive Utility on Mac, I'll be curious to see how they did it. Maybe there's a trick that I'm not aware of that makes it easier than I think.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants