Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Corrupted snapshots for large datasets #113

Open
fbacall opened this issue Nov 27, 2019 · 0 comments
Assignees

Comments

@fbacall
Copy link
Contributor

@fbacall fbacall commented Nov 27, 2019

When creating a snapshot with large files (4+GB total), some files end up missing from the resultant Zip file.

It seems that the zip file is cleared whilst being written to:

finn@finn-p520c ~/workspace/seek $ zipinfo /tmp/20191127-17238-qn3pbq/ro-bundle.zip 
Archive:  /tmp/20191127-17238-qn3pbq/ro-bundle.zip
Zip file size: 2743 bytes, number of entries: 7
-rw-r--r--  5.2 unx       36 t- stor 19-Nov-27 14:37 mimetype
drwxr-xr-x  5.2 unx        0 t- stor 19-Nov-27 14:37 .ro/
-rw-r--r--  5.2 unx     1187 t- defN 19-Nov-27 14:37 .ro/manifest.json
-rw-------  5.2 unx     1719 t- defN 19-Nov-27 14:37 metadata.rdf
-rw-------  5.2 unx      642 t- defN 19-Nov-27 14:37 metadata.json
-rw-------  5.2 unx     1283 t- defN 19-Nov-27 14:37 data_files/733-a-third-large-file/metadata.rdf
-rw-------  5.2 unx      569 t- defN 19-Nov-27 14:37 data_files/733-a-third-large-file/metadata.json
7 files, 5436 bytes uncompressed, 1895 bytes compressed:  65.1%
finn@finn-p520c ~/workspace/seek $ zipinfo /tmp/20191127-17238-qn3pbq/ro-bundle.zip 
Archive:  /tmp/20191127-17238-qn3pbq/ro-bundle.zip
Zip file size: 1032 bytes, number of entries: 2
-rw-------  5.2 unx      566 t- defN 19-Nov-27 14:39 data_files/732-another-2gb-file/metadata.json
-rw-r--r--  5.2 unx     2833 t- defN 19-Nov-27 14:39 .ro/manifest.json
2 files, 3399 bytes uncompressed, 734 bytes compressed:  78.4%
@fbacall fbacall self-assigned this Nov 27, 2019
fbacall added a commit that referenced this issue Nov 28, 2019
fbacall added a commit that referenced this issue Nov 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.