CRC error with zip files downloaded from web UI

[kiv57]

Steps to reproduce

1. login to web UI
2. select a folder or multiple files, clic "Download"
3. unzip the downloaded zip with your desktop zip handler

Expected behaviour

Unzipped files should have correct CRC and be valid

Actual behaviour

• Errors about CRC and headers (message details depending on the unzip program used)
• Some files are invalid and can't be opened, depending on the file type

Server configuration

Operating system: Debian 8.6

Web server: Nginx 1.6.2

Database: MySQL 5.5.50-0+deb8u1

PHP version: 5.6.24

Nextcloud version: 10.0.1

Updated from an older Nextcloud/ownCloud or fresh install: OC 8.2.7 (debian repos) -> manual upgrade to NC 9.0.54 -> upgrade to NC 10.0.1

Where did you install Nextcloud from: https://download.nextcloud.com/server/releases/

Signing status: No errors have been found.

List of activated apps:

App list

Enabled:
  - activity: 2.3.2
  - admin_audit: 1.0.0
  - calendar: 1.4.1
  - comments: 1.0.0
  - contacts: 1.4.0.0
  - dav: 1.0.1
  - federatedfilesharing: 1.0.1
  - federation: 1.0.1
  - files: 1.5.2
  - files_mv: 0.8.2
  - files_pdfviewer: 0.8.1
  - files_sharing: 1.0.0
  - files_texteditor: 2.1
  - files_trashbin: 1.0.0
  - files_versions: 1.3.0
  - files_videoplayer: 0.9.8
  - firstrunwizard: 1.1
  - gallery: 15.0.0
  - notes: true
  - notifications: 0.3.0
  - password_policy: 1.0.0
  - passwords: 20.1
  - provisioning_api: 1.0.0
  - qownnotesapi: true
  - serverinfo: 1.1.1
  - survey_client: 0.1.5
  - systemtags: 1.0.2
  - tasks: true
  - templateeditor: 0.1
  - theming: 1.0.1
  - updatenotification: 1.0.1
  - workflowengine: 1.0.1
Disabled:
  - encryption
  - external
  - files_accesscontrol
  - files_automatedtagging
  - files_external
  - files_retention
  - user_external
  - user_ldap
  - user_saml

The content of config/config.php:

Config report

{
    "system": {
        "instanceid": "xxxxxxxxxxxx",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "cloud.domain.tld",
            "local.ip.behind.reverse.proxy"
        ],
        "datadirectory": "\/home\/www-data\/data",
        "overwrite.cli.url": "https:\/\/cloud.domain.tld",
        "default_language": "fr",
        "dbtype": "mysql",
        "version": "9.1.1.5",
        "dbname": "owncloud",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "mail_from_address": "me",
        "mail_smtpmode": "smtp",
        "mail_domain": "domain.tld",
        "loglevel": 2,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "theme": "",
        "maintenance": false,
        "remember_login_cookie_lifetime": 1296000,
        "session_lifetime": 86400,
        "session_keepalive": true,
        "updatechecker": false,
        "trashbin_retention_obligation": "auto",
        "mail_smtpsecure": "ssl",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtphost": "mail.xxx.yyy",
        "mail_smtpport": "465",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "appstore.experimental.enabled": true
    }
}

Are you using external storage, if yes which one: no

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: firefox 50.0

Operating system: Windows 7

Logs

Web server error log

Web server error log (dates don't fit with zip download dates)

2016/11/24 07:37:31 [error] 853#0: *7379 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught exception 'Exception' with message 'Session has been closed - no further changes to the session are allowed' in /var/www/owncloud/lib/private/Session/Internal.php:154
Stack trace:
#0 /var/www/owncloud/lib/private/Session/Internal.php(64): OC\Session\Internal->validateSession()
#1 /var/www/owncloud/lib/private/Session/CryptoSessionData.php(164): OC\Session\Internal->set('encrypted_sessi...', '43893e78d74e0a2...')
#2 /var/www/owncloud/lib/private/Session/CryptoSessionData.php(67): OC\Session\CryptoSessionData->close()
#3 [internal function]: OC\Session\CryptoSessionData->__destruct()
#4 {main}
  thrown in /var/www/owncloud/lib/private/Session/Internal.php on line 154" while reading upstream, client: 10.0.3.1, server: cloud.domain.tld, request: "GET /index.php/login HTTP/1.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "cloud.domain.tld"

2016/11/27 13:26:43 [error] 850#0: *46841 open() "/var/www/owncloud/favicon.ico" failed (2: No such file or directory), client: 10.0.3.1, server: cloud.domain.tld, request: "GET /favicon.ico HTTP/1.0", host: "cloud.domain.tld"

Nextcloud log (data/nextcloud.log)

Nextcloud log (just one repeated error I didn't investigate yet)

Error	core	Could not get application: cURL error 60: SSL certificate problem: unable to get local issuer certificate	2016-11-23T19:00:08+00:00

Browser log

Not sure it's useful in this case, if so please ask.

[nickvergessen]

Any information which files are affected? Because it seems to work just fine here with images and documents.

[kiv57]

The error messages always appear, with any file type :

• "Erreur en-têtes" with 7zip ("headers error" should be the translation ?)
• "There is a CRC mistake on the file (...) Continue ?" with power archiver (translated from french to).I have to clic "Yes" for each file in the zip.
  (test with the same .zip of course)

That said, some files are corrupted and can't be opened, some others seem valid and I can open them. The fun part is that it depends on the program which did the unzip :

• pdf and docx : valid with 7zip, corrupted with power archiver
• xlsx : valid with both
• CREO files (CAD) : always corrupted
  I must admit I didn't test every possible file type...
  Moreover, users of the download feature don't have always the choice of the unzip program.

Please feel free to ask for more tests if necessary.

[nickvergessen]

Do zips that only have english characters work?

[kiv57]

No : I already tested files without accents in file name, the problem persists.

[nerdissimo-de]

I can confirm the error. When trying to upload to Gmail it doesn't allow the upload because it thinks there is a virus (there definitely isn't). When trying to open the file with 7zip or Nemo Fileroller it's also not possible. Funny thing is: When I use the commandline unzip it works fine.

Edit: doesn't matter which filetypes are in the archives. Happens for images and documents here.

[JarbyDev]

I have the same problem Here using the microsoft default unzip software

and using 7zip, i was able to extracted the content, but after i tried to open the file i get the file is currupted. happen with pdf, excel, image files

[qwertygc]

I have the same problem with NC 11.

[jospoortvliet]

Same issue here, zip files are often not working in some unzip software but OK in others. On my desktop - Dolphin can't open them as folders but Ark is OK with 'em.

[Idlebawb]

I am having a similar Issue. I did not get any CRC Errors or completely unreadable files, but Headers Error on 7-zip. The Main issue with that is that there is a gateway / reverse proxy between the server and the clients and that doesn't let any of those zips through. So nobody can download folders or multiple files, which is a big problem.

See Errors in attached screenshots.

Server: SUSE Enterprise 12
NextCloud Version: tried 10.0.3 & 11.0.1
Client: Only tried on Windows 10 Pro x64 (1511) but I guess its not a Client OS issue.
Browser: tried FF 50.1, Chrome 56 & IE11 - no difference

What I did:
Select a Folder & Download it
Select multiple Files at once & Download

The Files were: a single line txt file and a new .xlsx with just random numbers in it.
No special characters in either folder or filenames.

I don't know how I would look at the headers in 7zip, the most information I can get out the zip file is from zipinfo -v, but the Version on SUSE is from 2009, so I don't know how helpful that will be. But here it is.

Output of zipinfo -v

Archive: /home/it-department/scripts/New folder.zip There is no zipfile comment.

End-of-central-directory record:

Zip archive file size: 8451 (0000000000002103h)
Actual end-cent-dir record offset: 8353 (00000000000020A1h)
Expected end-cent-dir record offset: 8353 (00000000000020A1h)
(based on the length of the central directory and its expected offset)

This zipfile constitutes the sole disk of a single-part archive; its
central directory contains 3 entries.
The central directory is 328 (0000000000000148h) bytes long,
and its (expected) offset in bytes from the beginning of the zipfile
is 8025 (0000000000001F59h).

Central directory entry #1:

New folder/

offset of local header from start of archive: 0
(0000000000000000h) bytes
file system or operating system of origin: Unix
version of encoding software: 4.5
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 4.5
compression method: none (stored)
file security status: not encrypted
extended local header: no
file last modified on (DOS date/time): 2017 Feb 15 08:50:18
32-bit CRC value (hex): 00000000
compressed size: 0 bytes
uncompressed size: 0 bytes
length of filename: 11 characters
length of extra field: 32 bytes
length of file comment: 0 characters
disk number on which file begins: disk 1
apparent file type: binary
Unix file attributes (040755 octal): drwxr-xr-x
MS-DOS file attributes (10 hex): dir

The central-directory extra field contains:

• A subfield with ID 0x0001 (PKWARE 64-bit sizes) and 28 data bytes. The first
  20 are: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00.

There is no file comment.

Central directory entry #2:

New folder/Microsoft Excel-Arbeitsblatt (neu).xlsx

offset of local header from start of archive: 73
(0000000000000049h) bytes
file system or operating system of origin: Unix
version of encoding software: 4.5
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 4.5
compression method: none (stored)
file security status: not encrypted
extended local header: yes
file last modified on (DOS date/time): 2017 Feb 15 08:50:18
32-bit CRC value (hex): 163d861c
compressed size: 7694 bytes
uncompressed size: 7694 bytes
length of filename: 50 characters
length of extra field: 32 bytes
length of file comment: 0 characters
disk number on which file begins: disk 1
apparent file type: binary
Unix file attributes (100644 octal): -rw-r--r--
MS-DOS file attributes (00 hex): none

The central-directory extra field contains:

• A subfield with ID 0x0001 (PKWARE 64-bit sizes) and 28 data bytes. The first
  20 are: 0e 1e 00 00 00 00 00 00 0e 1e 00 00 00 00 00 00 49 00 00 00.

There is no file comment.

Central directory entry #3:

There are an extra 20 bytes preceding this file.

New folder/Neues Textdokument.txt

offset of local header from start of archive: 7899
(0000000000001EDBh) bytes
file system or operating system of origin: Unix
version of encoding software: 4.5
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 4.5
compression method: none (stored)
file security status: not encrypted
extended local header: yes
file last modified on (DOS date/time): 2017 Feb 15 08:50:18
32-bit CRC value (hex): 4a17b156
compressed size: 11 bytes
uncompressed size: 11 bytes
length of filename: 33 characters
length of extra field: 32 bytes
length of file comment: 0 characters
disk number on which file begins: disk 1
apparent file type: binary
Unix file attributes (100644 octal): -rw-r--r--
MS-DOS file attributes (00 hex): none

The central-directory extra field contains:

• A subfield with ID 0x0001 (PKWARE 64-bit sizes) and 28 data bytes. The first
  20 are: 0b 00 00 00 00 00 00 00 0b 00 00 00 00 00 00 00 db 1e 00 00.

There is no file comment.

Edit: Just some additional info.
I tried to create some 64bit Zips of the same file with 7zip and dotnetzip.
7zip wont let me create any 64bit zips unless the source files are bigger than 4GB
dotnetzip with -64 option creates the same "Header Error", but always useses deflate, couldnt get it to "Store"
32Bit Zips were fine.

EDIT2:
Disabled zip64 through a change in Streamer.php line 45

$this->streamerInstance = new ZipStreamer(['zip64' => PHP_INT_SIZE !== 4]);

I just set straight up false for zip64. I guess I will get errors on bigger files, but I am willing to trade not working at all (for me) for possible occasional errors. Hope this will get a real fix in a future release.

[andyboeh]

Same issue for me (NC 11.0.2). I can also confirm that disabling Zip64 fixes the problem (I do not have any files larger than 4GB anyway) by setting
$this->streamerInstance = new ZipStreamer(['zip64' => false]);
in lib/private/Streamer.php

[McNetic]

Hi there. As some may already know, I'm the author of the ZipStreamer library responsible for creating zip files in OC and NC.
That said, I'm still using OC in production and can't seem to reproduce the problem in my installation. So, a few questions to those who can:

1. Is the problem reproducible with any file, or only with specific files? (if the latter, could you provide some of the problematic files?)
2. Are you running nextcloud on a 32 bit or a 64 bit machine?
3. Although the responsible code seems to still be the same: has anyone been able to reproduce the problem in Owncloud?

Thank you for your help.

[andyboeh]

Sure, thanks for the reply:

1. It seems to be easy to reproduce as it happens nearly every time I share something. I'll send you a problematic file directly.
2. I'm on a 64-bit Server, Apache 2.4, PHP 7.1.1, NextCloud 11.0.2

[kiv57]

Thank you for the reply.

1. Problem occurs on any file type, and any size, even without special characters
2. I run Nextcloud 10.0.3 on a 64 bit server under Debian 8.7, with PHP 5.6.24 and MySQL 5.5.50
3. I had this problem on every upgrade versions since my first installation, which was an ownCloud 8.*

Feel free to ask for more tests or info.

[nerdissimo-de]

Hey, thanks for the reply:

1. Happens every time with all filetypes I tested. Some apps are able to extract it, for example a simple "unzip" on the commandline in Archlinux (UnZip 6.00) works.
2. I run Nextcloud 11.0.2 on Ubuntu 16.04 LTS (64bit) with PHP 7.0.15 and MariaDB 10.0.29
3. I had the problem since installing Nextcloud (10.0.1 I think it was)

Best

[andyboeh]

For me, the issue seems to be a bit more complicated than I initially thought. I found the following cases:

• 64 bit ZIP that works on Windows (7z and Windows integrated ZIP), but fails on Linux (7z)
• 32 bit ZIP that fails on Windows (integrated ZIP, 7z reports it as correct), works on Linux (7z)

Unfortunately, I cannot share the failing 32 bit ZIP.

[Idlebawb]

I've been analyzing the 64bit Zips created by Nextcloud / Zipstreamer. And according to the Zip specification ( https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT ) they seem to be just fine.

The headers error I got with 7-Zip are likely incorrectly detected by 7-zip - see the post by the developer here:

https://sourceforge.net/p/sevenzip/discussion/45797/thread/cc8912f1/

The error i got referred to the file being a multi-part archive.
This was most likely caused by an incomplete or defect detection of zip64 in the gateway appliance we use.

I manually edited a test "broken" zip file created by nextcloud in a hex editor and replaced all the values in the "End of Central Directory Record" which are 0xFFFF with the corresponding values from the "Zip64 End of Central Record" and the zip went trough the gateway just fine.

So we opened a case with the manufaturer of our Web gateway appliance.

[KopfKrieg]

I'm running Nextcloud v11.0.3 via the official docker image behind an nginx reverse proxy (running on Ubuntu 16.04 LTS, all updates installed) and stumbled across the same issue.

I can't open any of the downloaded zip files (with and without special characters), but I'm able to unzip them via commandline. Disabling zip64 as suggested works.

Would be nice to have a solution for this :)

[qwertygc]

Yes :

$this->streamerInstance = new ZipStreamer(['zip64' => false]); in lib/private/Streamer.php

It's work

[McNetic]

It would be very nice to know if the problem persists when the reverse proxy is removed from the setup...

[qwertygc]

I write a PR #5112

[McNetic]

We should first try to identify what is causing the problems in the first place, before deliberately disabling features that appear to work for the vast majority of users (Zip64 is required for zip files and file entries bigger than 4 GB, and is standardized and documented since 2001).

I would love to fix any bug in ZipStreamer when it is found, but given that we already fixed a lot of the issues with the initial zip64 implementation, and given that we know various zip implementations out there do not correctly implement the standard (at least two are referenced in this thread alone), I suspect the problem(s) may lie in some of the other components involved.

So to narrow down the problem(s) and find it's cause, everyone having an issue should (1) make sure no third party software is involved. That means, no proxy, no deep packet inspecting and/or manipulating firewall, no antivirus or similar software. If the problem persists, you should exactly document which zip program in which version on which operating system exposed the problem.

[KopfKrieg]

I just created a sample folder with two sample files and downloaded it the following way:

• via proxy → gives an error if I try to open the zip file
• without proxy → still gives an error
• with modified Streamer.php via proxy → works fine

Nextcloud seems to generate the zip-files on-the-fly, everytime someone's downloading the files, using a non-deterministic algorithm (every zip file has another hashsum).

If you want to try it/inspect the generated zips, you can download them here (1,8kB all three files combined): [Link removed]

System is, as already said, Nextcloud 11.0.3 from nextcloud/docker, running on an up-to-date Ubuntu 16.04 LTS host with nginx 1.10.0 as proxy and let's encrypt for my certificates.

But maybe the zip files are fine (I can unzip them without problems via commandline [unzip-command]) and my archive manager (Nautilus archive manager, but I don't know what underlying library is used to unzip files) has a bug.

Does anybody know if the zip files can be verified? E.g. if they are compliant to the specification?

Update: $ 7z t Sample_via_Proxy.zip says:

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=de_DE.UTF-8,Utf16=on,HugeFiles=on,64 bits,4 CPUs Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz (306A9),ASM,AES-NI)

Scanning the drive for archives:
1 file, 709 bytes (1 KiB)

Testing archive: Sample_via_Proxy.zip

ERRORS:
Headers Error

--
Path = Sample_via_Proxy.zip
Type = zip
ERRORS:
Headers Error
Physical Size = 709
64-bit = +



Archives with Errors: 1

Open Errors: 1

Update: For me the issue persists with Nextcloud 12.0.0

[Bubu]

I can confirm the observations from @KopfKrieg. unzip works fine and tests the zip okay. p7zip gives the error posted above and can't extract the archive:

Type = zip
ERRORS:
Headers Error
Physical Size = 102989366
64-bit = +

Winrar (5,20, 2014) apparently also doesn't work. (Someone notified me that the downloads aren't working.)

Here are the program details:

UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with gcc 5.2.1 20151119 for Unix (Linux ELF).

p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,4 CPUs Intel(R) Core(TM) i5-6440HQ CPU @ 2.60GHz (506E3),ASM,AES-NI)

Using #5112 works around the problem.

[haraldkrischner]

For me this currently is the major issue when using nextcloud. Because this also affects people whom I want to share with not caring about the correct support of zip standards. If @nickvergessen 's fix fixes it release a fix? Perhaps?

[nickvergessen]

I will prepare a custom patch for our repository

[MorrisJobke]

Maybe it needs to be adjusted, because mac evolved in the last two years.

I just tested it (together with nextcloud/3rdparty#74) but I still get this error:

[es20490446e]

I have checked that gzip is supported by modern Android Phones and iPhones, but it's problematic with not so old models.

So for the time being I would simply use zip universally, and change to gzip once 90% of mobile phones support it.

Also I believe Windows cannot be taken into account in this regard. Windows is a clunky OS, and when you get it you assume you have to install other applications to make it work.

A Windows system without compression support is just very incapable. The same as not having office suite or multimedia codecs, they are just the basics.

[Keridos]

Windows can open and create zip files...

[es20490446e]

But that's the only compressed format it can open. Even when rar, 7zip, and gzip are widely used.

[zguithues]

many people are restricted from installing outside software on their computers, so they are left with only zip support.

I'm an insurance agent, i commonly have to send archives to my underwriter. He is not able/allowed to open .rar files, or any other alternative archive. I am required to send .zip files.

I don't like it, but we 100% DO have to take windows into account. it's a fact of life right now.

[es20490446e]

And the drawbacks of using ZIP are only superfluous.

[KopfKrieg]

Question: Why don't we let the user choose between zip and tar instead of assuming the user's preference based on his/her platform? (Maybe create a feature request?)

[es20490446e]

Because it doesn't matter:
(https://signalvnoise.com/archives2/it_just_doesnt_matter.php)

[KopfKrieg]

Are you serious? Of course it matters, that's why I'm asking.

[es20490446e]

We have said that it's somehow unpredictable if the target platform, the receiver of your files, will have support for gzip. So in practise having the option does nothing.

[nickvergessen]

While I see the point in "It just doesnt matter" and think its a valid argument in many cases,
this topic here is none of them.

Downloading files/archives is (apart from syncing files) the number two use case of Nextcloud. So that must work in as many situations as possible (by default). An option to download tar instead of zip (or vice versa) sounds like an option to me.

But fixing the broken zip download, which happens on some systems with some unzippers, should be the priority in first place.

[rullzer]

I just tested nextcloud/3rdparty#74 and now at least p7zip can extract the archive. It seems that OSX has a different bug. Lets get that in and we can see how to fix OSX in the next step.

[arnowelzel]

Looks like there are still some issues, see #8798

[AntoineMazuyer]

I have the same issue with nexctloud 13.0.1 :(

[pjoscha]

still the same isse in nextcoud 15.0.4

[alogoc]

Looks like this was never fixed?

[es20490446e]

Do you need me to test?

[arnowelzel]

Would it help to change to another Zip-Library? Maybe https://github.com/maennchen/ZipStream-PHP works better.

[osscombat]

So this is a known problem and it's still open for almost 3 years in production versions?

$this->streamerInstance = new ZipStreamer(['zip64' => false]); is already merged in lib/private/Streamer.php in the latest 15 and 16 versions, but zip's are still being corrupt.

[kesselb]

Mind to open a new issue if you still see this? #15871