[Bug]: File Download/Preview/Streaming Gets Truncated/Corrupted when Served from SMB/CIFS Share (Not Using NC smbclient) #31361
Comments
GuyPaddock
added
0. Needs triage
|
UPDATE: I tried rolling Icewind back to the version that was in Nextcloud 20.0.18, per comments in #26457, but the rollback had no effect on the behavior, so this is not looking like an Icewind issue per say. |
|
Another update: It appears that this issue is either related to a change in PHP or a change in Sabre/HTTP. It is documented elsewhere that mmap combined with SMB volumes can lead to corrupt file reads:
Now, it looks like Sabre HTTP 5.0.3 added enhancements to make better use of For more context, the MR referenced by the Sabre HTTP changelog is: And that version was pulled into Nextcloud 19.0.0 (coincidentally, we started seeing this issue upon upgrading from 18.0.14 to 19.0.13 as mentioned in the issue description): Prior to NC 19, NC was using version 4.2.4 of Sabre HTTP. Now, I thought I had a slam dunk here because this theory would have worked perfectly to explain the behavior that we are seeing, except that the code in sabre-io/http#119 doesn't actually call HOWEVER one last thing I tried was -- in addition to reverting 119 -- I modified if (null !== $contentLength) {
$output = fopen('php://output', 'wb');
if (is_resource($body) && 'stream' == get_resource_type($body)) {
if (PHP_INT_SIZE !== 4) {
// use the dedicated function on 64 Bit systems
stream_copy_to_stream($body, $output, (int) $contentLength);
} else {
// workaround for 32 Bit systems to avoid stream_copy_to_stream
while (!feof($body)) {
fwrite($output, fread($body, 8192));
}
}
} else {
fwrite($output, $body, (int) $contentLength);
}
} else {
file_put_contents('php://output', $body);
}It read as: if (null !== $contentLength) {
$output = fopen('php://output', 'wb');
if (is_resource($body) && 'stream' == get_resource_type($body)) {
// workaround for 32 Bit systems to avoid stream_copy_to_stream
while (!feof($body)) {
fwrite($output, fread($body, 8192));
}
} else {
fwrite($output, $body, (int) $contentLength);
}
} else {
file_put_contents('php://output', $body);
}... and that works. Attached is the resulting patch (for debugging only). This is obviously not a fix because it seems to have a significant performance hit, but it does seem to narrow down where the issue is. sabre-http-revert-5.0.2-and-mmap.patch.txt I can't take credit for this idea, as it was suggested by #26457 (comment). |
|
More surgical patch that just removes calls to |
|
It looks like our Nextcloud v18.0.14 image was running PHP 7.3.26 whereas the Nextcloud v19.0.13 image runs PHP 7.4.21. That's significant because, in addition to the Which... rolled out in PHP 7.4. My hypothesis is that files on an SMB share that are < 4 MB can be read with minimal or no side effects through |
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
See: nextcloud/server#31361 (comment) (cherry picked from commit ccd0dd8)
|
Hi, please update to 24.0.9 or better 25.0.3 and report back if it fixes the issue. Thank you! My goal is to add a label like e.g. 25-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort! If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+ |
Bug description
We recently upgraded an installation of Nextcloud from 18.0.14 to 19.0.13 and now are experiencing a serious problem where most files that are larger than about 2 MB do not download properly from Nextcloud when they are sourced from an SMB/CIFS share. Almost all of our files are hosted on Azure Files over SMB, so this is a big problem for us. (We can reproduce the issue with ALL VERSIONS OF NEXTCLOUD after 19 -- see below).
When working with image files that were previously uploaded (it does not matter if the image was uploaded before or after the upgrade), we will see one of the following happen:
The image file won't load at all when being previewed:
OR - The image loads, but is truncated or corrupt:
When we try to download the image, the download fails with a "Network error".
With video files, we find that the video either does not play or plays with artifacts and corruption throughout the image (the source video does NOT have this corruption).
Oddly, if we select multiple files and select "Download" (so that they download as a single ZIP), the resulting ZIP downloads without issue and the files inside are not corrupt, so this is only affecting the behavior of streaming/loading individual files from the web UI directly.
I saw #26457 and #26933 but our issue differs from those issues in significant ways:
When nginx is being used to host Nextcloud, it is significantly more strict about the issue and will terminate the connection if something isn't right, failing with the following error:
If we switch to Apache-based Docker containers, Apache does not report any errors but the file it serves-up has corruption throughout, which is arguably way worse (because users might not realize that the file is corrupt since they still get the right number of bytes that they asked for, but the bytes are corrupt).
Steps to reproduce
Setup Nextcloud to run in a Kubernetes-powered container (PHP-FPM or Apache; it does not matter).
Setup the container to mount a volume over an Azure Files or SMB to the path
/mnt/share/test.For example, a container with a volume mount like (the rest of the container boilerplate has been omitted):
Declared as:
With PVs and PVCs of (secrets omitted):
Configure Nextcloud with a local External storage that points to
/mnt/share/test:Turn image previews OFF for the share, so that the preview lightbox shows full-resolution images (makes this easier to reproduce).
Upload an image or video file that is larger than 5 MB to the share (for example, the 25 MB Rome, Italy image from https://effigis.com/en/solutions/satellite-images/satellite-image-samples/).
Attempt to open the image or video file in the preview lightbox. Repeat this step at least 5 times, clearing browser cache in between attempts.
Attempt to download the image or video file through the Nextcloud web UI. Repeat this step at least 5 times, clearing browser cache in between attempts.
Expected behavior
The image or video should load without error.
Installation method
Official Docker image
Operating system
Debian/Ubuntu
PHP engine version
PHP 7.4
Web server
Nginx
Database engine version
MariaDB
Is this bug present after an update or on a fresh install?
Updated to a major version (ex. 22.2.3 to 23.0.1)
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
What user-backends are you using?
Configuration report
{ "system": { "simpleSignUpLink.shown": false, "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "dev.cloudvault.inveniem.com" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "23.0.2.1", "overwrite.cli.url": "https:\/\/dev.cloudvault.inveniem.com", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "htaccess.RewriteBase": "\/", "maintenance": false, "appstoreenabled": false, "enable_previews": true, "preview_max_filesize_image": 50, "apps_paths": [ { "path": "\/var\/www\/html\/apps", "url": "\/apps", "writable": false }, { "path": "\/var\/www\/html\/custom_apps", "url": "\/custom_apps", "writable": false } ], "logfile": "\/var\/log\/nextcloud.log", "loglevel": 1, "filelocking.enabled": false, "memcache.local": "\\OC\\Memcache\\APCu", "memcache.distributed": "\\OC\\Memcache\\Redis", "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": "6379", "timeout": 1.5, "password": "***REMOVED SENSITIVE VALUE***" }, "dbdriveroptions": { "1009": "\/etc\/ssl\/certs\/Baltimore_CyberTrust_Root.pem" }, "config_is_read_only": true, "skeletondirectory": "", "mysql.utf8mb4": true, "preview_max_x": 1024, "preview_max_y": 1024, "trusted_proxies": "***REMOVED SENSITIVE VALUE***", "app_install_overwrite": [ "user_external", "user_saml" ] } }List of activated Apps
Nextcloud Signing status
Nextcloud Logs
Additional info
Configurations Attempted
This issue appears when using files > 2 MB over SMB with ALL of the following configurations:
sendfile offandsendfile on.sendfile onandsendfile_max_chunk 1m.cache=noneandcache=strict.The issue does NOT appear if we create a folder inside the container (e.g.,
mkdir -p /mnt/local/test) and then mount that as external storage. The issue also was not reproducible with Nextcloud 18.0.14.The text was updated successfully, but these errors were encountered: