Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding support for split zip files #114

Open
Daniel451 opened this issue Apr 11, 2019 · 0 comments

Comments

@Daniel451
Copy link

@Daniel451 Daniel451 commented Apr 11, 2019

Large datasets (roughly >3GB) cannot be uploaded at once so that one has to split large datasets manually into either smaller datasets or multiple zip files (by hand or custom scripts). It clearly discourages uploading large datasets.

This is against the intended use-case of the ImageTagger: rendering the usage of large datasets + annotations easy to handle.

One could easily automatically split zip files by using 7zip, for example:

7z a -tzip dataset.zip -v1G *.png

This way, arbitrarily large datasets will get packed into split zip files, each of a size of exactly 1GB of data.

Unfortunately, the ImageTagger rejects files of extension:

dataset.zip.001
dataset.zip.002
dataset.zip....
dataset.zip.N

Accepting such file formats should be relatively easy to fix since most zip software allows for splitting zip files and supports extracting files out of split zip archives.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.