Files on SMB-share corrupted while up- and downloading #26457
Comments
ShinjiLE
added
0. Needs triage
|
There's also a problem with PDFs, if you try to open them on a SMB-Share: |
|
Exact same problem here since the upgrade of nextcloud to 21.0.1.1 (Debian 10 buster). It used to work on nextcloud 20.0 |
|
Same problem here EDIT 1: I'm using Nextcloud official VM from hanssonit.se EDIT 2: |
|
Same problem after update 20.0.8 to 20.0.9. I had to restore 20.0.8. |
|
Same problem after update 20.0.8.1 to 21.0.1.1 @muchahagrande, good hint! |
|
Same issue here with a 20.0.8.1 to 21.0.1.1 upgrade, as released today. I am running on Ubuntu Server 20.04.2 LTS. The issue manifests as a whole series of massive buffer overflows in the fread() function when accessing External Storage via SMB: I have narrowed the issue down to only happening when I have another identical server running Nextcloud 20.0.8.1 with the same php7.4-smbclient_1.0.6-1+ubuntu20.04.1+deb.sury.org+1_amd64.deb installed. This does not exhibit any such issues. My workaround is to remove A quick patch would be much appreciated. Maybe @icewind1991 has already seen this? |
|
Could also be related with #26303. It is mentioned that the problem is with SFTP, although I experience the same issue with the SMB mounts as outlined here. Opening the files directly from the server, there is no corruption, but opening from withing the WebUI I experience the same as mentioned above; pictures open as seen in the first post or not at all and pdf files don't work. Audio files skip in the playback. After checking the logs, it is related with The problem also occurred after updating from 20.0.8 to 21.0.1 |
@Aejoh, I've just tried your workaround and I can confirm that is working again. Thank you. |
Same issue after upgrade 21.0.0 -> 21.0.1, same workaround (replace icewing with previous version). |
|
Same issue on a fresh 21.0.1.1 docker install (latest fpm) |
|
I have installed NC manually and can confirm the workaround as well. For me the path is:
|
|
I have the same issue after updating from 20.0.8 to 20.0.9. |
|
Same here, after upgrading from 20.0.8 to 21.0.1. |
|
Same for me. Replacing it with the older version works. |
|
Had the same issue after upgrading from 20.0.2 to 21.0.1. |
|
Had the same problem after upgrade from 20 to 21, the files were corrupted at 16KB |
|
Hello, i have the same problem, SMB doesn't work and i have this |
|
Confirming about restoring files from an older files_external version. This will affect a few of my installation if i'm upgrading them |
can you provide a way to do this pls ? |
|
able to reproduce, Debian 10, Apache 2.4, PHP 7.4 with php-smbclient workaround of copying old icewind directory works. |
|
@RisedSky, the steps to do it may differe a bit from system to system, but essentially: EDIT: |
Thanks !!!!!!! |
Dit this but had mutiples issues with my server after getting back the old icewind folder from my NC 20.8 :( |
|
Just for help @RisedSky . |
which errors, so that we can understand them |
|
After upload file in next cloud the image is 0kb the size . solve this |
have you done the update to the latest version of nextcloud ? |
|
I had that issues with external storages ( |
|
Guys. I am really so grateful I found this thread. Replacing the old icewind files fixed the issue for me. |
|
If you are having that kind of issue, why dont you consinder local smb
mount which you will expose over Local storage? In that case you are
avoiding that module.
…On Sat, 16 Apr 2022, 01:47 MarcS1975, ***@***.***> wrote:
Guys. I am really so grateful I found this thread. Replacing the old
icewind files fixed the issue for me.
After upgrading from NC 21 to 23.03, all SMB links failed to connect. I
tried everything and could not fix this issue.
So thank you thank you guys so much !!!
—
Reply to this email directly, view it on GitHub
<#26457 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA7DGPS6PLBPEGKWCQJL5A3VFH5YFANCNFSM42SOJOXQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Easy, permissions - if you have a 100 users connecting to a share each with different permissions to that share, which would you mount? |
|
Mount it with a global all permission service account and do your
permission on a Nextcloud side. Expose some of the folders to specific
parts of a users. Now nextcloud doesn't know your credentials to smb share,
but downside is that you can't show him only specific files.
sub, 16. tra 2022. u 10:20 Taomyn ***@***.***> napisao je:
… If you are having that kind of issue, why dont you consinder local smb
mount which you will expose over Local storage? In that case you are
avoiding that module.
… <#m_6298073299704870113_>
On Sat, 16 Apr 2022, 01:47 MarcS1975, *@*.*> wrote: Guys. I am really so
grateful I found this thread. Replacing the old icewind files fixed the
issue for me. After upgrading from NC 21 to 23.03, all SMB links failed to
connect. I tried everything and could not fix this issue. So thank you
thank you guys so much !!! — Reply to this email directly, view it on
GitHub <#26457 (comment)
<#26457 (comment)>>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7DGPS6PLBPEGKWCQJL5A3VFH5YFANCNFSM42SOJOXQ
<https://github.com/notifications/unsubscribe-auth/AA7DGPS6PLBPEGKWCQJL5A3VFH5YFANCNFSM42SOJOXQ>
. You are receiving this because you were mentioned.Message ID: @.*>
Easy, permissions - if you have a 100 users connecting to a share each
with different permissions to that share, which would you mount?
—
Reply to this email directly, view it on GitHub
<#26457 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA7DGPUMYTMI76EVF2JENH3VFJZ45ANCNFSM42SOJOXQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Hardly a solution, SMB should just work full stop. |
|
I agree. I always have trouble after a NC upgrade to get my SMB shares
working again. It's a shame because NC is supposed to be a cloud solution
and therefore SMB connectivity is a pretty basic feature. I have around 20
shares and fixing that after NC upgrades is painful.
…On Sat, 16 Apr 2022, 09:48 Taomyn, ***@***.***> wrote:
Hardly a solution, SMB should just work full stop.
—
Reply to this email directly, view it on GitHub
<#26457 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMNNHGRZHNX6ZBIQQKHIVQ3VFJ5D3ANCNFSM42SOJOXQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
Totally! I have 1500 users, 20 smb servers, 100 groups, and about 120 shares. Impossible to do otherwise. This has to work right out of the box. Even the importance of everything that needs to be done to make it work on that scale is not well documented and clarified. As I read this, I'm looking to get something up and running to index the files. Because in parallel I have people who use the nextcloud, and people who access the shares by other means. That breaks the sync and files created by other means are not seen via nextcloud. The documentation makes reference to leaving a connection open to detect file changes in shares. IMPOSSIBLE for that to work when you have more than 4 or 5 shares. Don't get me started on the fulltextsearch plugin. No way to make it work with that many shares/files. |
|
Strange inalso have people accessing SMB via maped drives directly and via
Nextcloud smb connection. File integrity is no problem. They always seem in
sync. For me the issue is that file access via NC is super slow. Opening a
200kb word doc takes 20 seconds in Nextcloud accessing an SMB share.
…On Sat, 16 Apr 2022, 15:47 glorenzutti, ***@***.***> wrote:
Totally! I have 1500 users, 20 smb servers, 100 groups, and about 120
shares. Impossible to do otherwise.
This has to work right out of the box.
Even the importance of everything that needs to be done to make it work on
that scale is not well documented and clarified.
As I read this, I'm looking to get something up and running to index the
files. Because in parallel I have people who use the nextcloud, and people
who access the shares by other means. That breaks the sync and files
created by other means are not seen via nextcloud.
The documentation makes reference to leaving a connection open to detect
file changes in shares. IMPOSSIBLE for that to work when you have more than
4 or 5 shares.
Don't get me started on the fulltextsearch plugin. No way to make it work
with that many shares/files.
—
Reply to this email directly, view it on GitHub
<#26457 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMNNHGU7WXN2XTET543JDITVFLHJDANCNFSM42SOJOXQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
The speed is good. I have nc 23.0.2 and i dont remember right now if I use smbclient or php-smbclient to browse the files. |
|
I don't use colobra office. I use the Nextcloud browser login and download
files or i use mobile app on Android and IPhone. Both are quite user
friendly but very slow when accessing files.
…On Sat, 16 Apr 2022, 15:54 glorenzutti, ***@***.***> wrote:
The speed is good. I have nc 23.0.2 and i dont remember right now if I use
smbclient or php-smbclient to browse the files.
Do you mean using the libreoffice, or to download the doc?
I have collabora office on another virtual machine and it works great.
—
Reply to this email directly, view it on GitHub
<#26457 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMNNHGTXPPK7A5OJANZMSBTVFLIBHANCNFSM42SOJOXQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
This works good for me. Without knowing how you have everything configured, I would suggest that you start with the performance of the webserver, the memory usage, having redis and the php cache, see the mysql. There are several things to check. The same happends when you access a file that isnt on the smb share? |
|
Also check filelocking on the nextcloud side, for me that took a huge
ammount of time.
…On Sat, 16 Apr 2022, 17:03 glorenzutti, ***@***.***> wrote:
This works good for me.
Without knowing how you have everything configured, I would suggest that
you start with the performance of the webserver, the memory usage, having
redis and the php cache, see the mysql. There are several things to check.
The same happends when you access a file that isnt on the smb share?
—
Reply to this email directly, view it on GitHub
<#26457 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA7DGPXP47XINVX2ZQLMP33VFLJFZANCNFSM42SOJOXQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
This is the bug about the sync on the share with multiples clients: |
|
Thanks .I have 33gb memory and it's a dedicated server. Php cache all fine.
Do not have redis unfortunately.
Yes it's also slow but not as bad for non smb files.
…On Sat, 16 Apr 2022, 16:04 glorenzutti, ***@***.***> wrote:
This works good for me.
Without knowing how you have everything configured, I would suggest that
you start with the performance of the webserver, the memory usage, having
redis and the php cache, see the mysql. There are several things to check.
The same happends when you access a file that isnt on the smb share?
—
Reply to this email directly, view it on GitHub
<#26457 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMNNHGVUYTSECSMNES4EOWTVFLJGBANCNFSM42SOJOXQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
Thanks. How do I check this please?
Btw I also found an incredibly fast alternative as a filesharing server
called Filerun. Functionality is a bit limited for the free version but I
can link the sane smb shares to Filelink and files open within half a
second.
…On Sat, 16 Apr 2022, 16:09 Dario Sindičić, ***@***.***> wrote:
Also check filelocking on the nextcloud side, for me that took a huge
ammount of time.
On Sat, 16 Apr 2022, 17:03 glorenzutti, ***@***.***> wrote:
> This works good for me.
>
> Without knowing how you have everything configured, I would suggest that
> you start with the performance of the webserver, the memory usage, having
> redis and the php cache, see the mysql. There are several things to
check.
>
> The same happends when you access a file that isnt on the smb share?
>
> —
> Reply to this email directly, view it on GitHub
> <
#26457 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AA7DGPXP47XINVX2ZQLMP33VFLJFZANCNFSM42SOJOXQ
>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#26457 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMNNHGQJ2WU6FZKHRXMLRCTVFLJ2RANCNFSM42SOJOXQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
Using redis its like day and night. it should help, by a lot. |
|
Unfortunately I run this server under Synology (Debian based) and although
I am able to install Redis and it runs well, I cannot get the phpredis
extention. So Nextcloud cannot utilise Redis.
…On Sat, 16 Apr 2022, 16:22 glorenzutti, ***@***.***> wrote:
Using redis its like day and night. it should help, by a lot.
—
Reply to this email directly, view it on GitHub
<#26457 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMNNHGTLE7BWDHHJCEE5UWDVFLLJNANCNFSM42SOJOXQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
I dont have access to a synology right now. But I do recall you have the tools to compile php modules. So you should be able to add it that way. |
|
Ok. If have installed it and updated the config php. It seems quicker
butnthis could be subjective.
Is there a way to check if
PhpCache and Redis are actually working and utilised by NC?
…On Sat, 16 Apr 2022, 17:24 glorenzutti, ***@***.***> wrote:
I dont have access to a synology right now. But I do recall you have the
tools to compile php modules. So you should be able to add it that way.
—
Reply to this email directly, view it on GitHub
<#26457 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMNNHGVAYFW2NAOF4Q6JDN3VFLSVHANCNFSM42SOJOXQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
I guess you could benchmark with some speed test plugins on your browser. |
|
I am getting these NC warnings. Strangely the NC server is still running
fine but I am concerned about these messages. Anyone seen this?
Current status: 1 (Interrupted)
Standard output/error:
PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone
away in
/volume1/web/nextcloud/3rdparty/doctrine/dbal/src/Driver/PDO/Statement.php:92
Stack trace:
#0
/volume1/web/nextcloud/3rdparty/doctrine/dbal/src/Driver/PDO/Statement.php(92):
PDOStatement->execute(NULL)
#1 /volume1/web/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1057):
Doctrine\DBAL\Driver\PDO\Statement->execute()
…On Sat, 16 Apr 2022, 17:24 glorenzutti, ***@***.***> wrote:
I dont have access to a synology right now. But I do recall you have the
tools to compile php modules. So you should be able to add it that way.
—
Reply to this email directly, view it on GitHub
<#26457 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMNNHGVAYFW2NAOF4Q6JDN3VFLSVHANCNFSM42SOJOXQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
Check your mysql logs, on the mysql server side. |
|
I found these entries. Does anyone know what the issue might be?
2022-04-18 16:00:25 0 [Note] /usr/local/mariadb10/bin/mysqld (initiated by:
unknown): Normal shutdown
2022-04-18 16:00:25 0 [Note] InnoDB: FTS optimize thread exiting.
2022-04-18 16:00:25 0 [Note] InnoDB: Starting shutdown...
2022-04-18 16:00:25 0 [Note] InnoDB: Dumping buffer pool(s) to
***@***.***/mariadb10/ib_buffer_pool
2022-04-18 16:00:25 0 [Note] InnoDB: Instance 0, restricted to 255 pages
due to innodb_buf_pool_dump_pct=25
2022-04-18 16:00:25 0 [Note] InnoDB: Buffer pool(s) dump completed at
220418 16:00:25
2022-04-18 16:00:26 0 [Note] InnoDB: Shutdown completed; log sequence
number 2258492082; transaction id 3739114
2022-04-18 16:00:26 0 [Note] InnoDB: Removed temporary tablespace data
file: "ibtmp1"
2022-04-18 16:00:26 0 [Note] /usr/local/mariadb10/bin/mysqld: Shutdown
complete
2022-04-18 16:00:27 0 [Note] InnoDB: Using Linux native AIO
2022-04-18 16:00:27 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic
builtins
2022-04-18 16:00:27 0 [Note] InnoDB: Uses event mutexes
2022-04-18 16:00:27 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2022-04-18 16:00:27 0 [Note] InnoDB: Number of pools: 1
2022-04-18 16:00:27 0 [Note] InnoDB: Using SSE2 crc32 instructions
2022-04-18 16:00:27 0 [Note] InnoDB: Initializing buffer pool, total size =
16M, instances = 1, chunk size = 16M
2022-04-18 16:00:27 0 [Note] InnoDB: Completed initialization of buffer pool
2022-04-18 16:00:27 0 [Note] InnoDB: If the mysqld execution user is
authorized, page cleaner thread priority can be changed. See the man pa
ge of setpriority().
2022-04-18 16:00:28 0 [Note] InnoDB: 128 out of 128 rollback segments are
active.
2022-04-18 16:00:28 0 [Note] InnoDB: Creating shared tablespace for
temporary tables
2022-04-18 16:00:28 0 [Note] InnoDB: Setting file
'/var/packages/MariaDB10/target/mysql/ibtmp1' size to 12 MB. Physically
writing the file f
ull; Please wait ...
2022-04-18 16:00:28 0 [Note] InnoDB: File
'/var/packages/MariaDB10/target/mysql/ibtmp1' size is now 12 MB.
2022-04-18 16:00:28 0 [Note] InnoDB: Waiting for purge to start
2022-04-18 16:00:28 0 [Note] InnoDB: 10.3.7 started; log sequence number
2258492082; transaction id 3739007
2022-04-18 16:00:28 0 [Note] InnoDB: Loading buffer pool(s) from
***@***.***/mariadb10/ib_buffer_pool
2022-04-18 16:00:28 0 [Note] InnoDB: Buffer pool(s) load completed at
220418 16:00:28
2022-04-18 16:00:28 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-04-18 16:00:28 0 [Note] Server socket created on IP: '0.0.0.0'.
2022-04-18 16:00:28 0 [Note] Reading of all Master_info entries succeded
2022-04-18 16:00:28 0 [Note] Added new Master_info '' to hash table
2022-04-18 16:00:28 0 [Note] /usr/local/mariadb10/bin/mysqld: ready for
connections.
…On Mon, Apr 18, 2022 at 5:18 PM glorenzutti ***@***.***> wrote:
Check your mysql logs, on the mysql server side.
—
Reply to this email directly, view it on GitHub
<#26457 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMNNHGR4RL5BYIBSHS2BND3VFWDLTANCNFSM42SOJOXQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
and others
Steps to reproduce
Using SMB Share
Expected behaviour
Files uploaded and downloaded without corruption
Actual behaviour
Datablocks are truncated to 8192 Bytes
Server configuration detail
Operating system: Linux 5.11.6-1-default #1 SMP Thu Mar 11 16:11:36 UTC 2021 (7358b30) x86_64
Webserver: Apache (fpm-fcgi)
Database: mysql 10.5.9
PHP version:
7.4.16
Modules loaded: Core, date, libxml, pcre, filter, hash, Reflection, SPL, session, SimpleXML, standard, xml, cgi-fcgi, mysqlnd, apcu, bcmath, bz2, ctype, curl, dom, enchant, mbstring, fileinfo, ftp, gd, gettext, gmp, ice, iconv, imagick, intl, json, lzf, exif, memcached, mysqli, openssl, pcntl, PDO, pdo_mysql, pdo_sqlite, zlib, posix, readline, redis, smbclient, sockets, sqlite3, tidy, tokenizer, uuid, xmlreader, xmlwriter, zip, Phar, libsmbclient, Zend OPcache
Nextcloud version: 21.0.1 RC1 - 21.0.1.0
Updated from an older Nextcloud/ownCloud or fresh install:
Where did you install Nextcloud from: unknown
Signing status
Array
(
)
List of activated apps
Configuration (config/config.php)
Are you using external storage, if yes which one: smb
Are you using encryption:
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
Client configuration
Browser: Mozilla/5.0 (X11; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0
Operating system:
Logs
Web server error log
Nextcloud log
Browser log
Insert your browser log here, this could for example include:
The text was updated successfully, but these errors were encountered: