Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: Sabre\DAV\Exception\BadRequest: Expected filesize of... #21382

Closed
nicrame opened this issue Jun 12, 2020 · 46 comments
Closed

Error: Sabre\DAV\Exception\BadRequest: Expected filesize of... #21382

nicrame opened this issue Jun 12, 2020 · 46 comments
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap bug feature: logging needs info

Comments

@nicrame
Copy link

nicrame commented Jun 12, 2020

I've got folder with about 31GB (116 633 files and 29 052 folders). I selected it to sync with folder inside Nextcloud app.
On maaaany files there is Sabre\DAV\Exception\BadRequest error. Files with different size, and different types.

There are also other problems i have that i think are not very problematic so i will give them on the end of this report.

Proxy in client is disabled. Connection between PC and Nextcloud box is 1Gb ethernet on both sides. The PC (win10 with desktop client) got 64GB RAM with i7 9gen CPU, the Nextcloud (CentOS 8) box is 8GB RAM, with J4105 CPU, and two HDD's in softraid (raid 1).

Tried so far without success: Disable SELinux, PHP downgrade to 7.3 (REMI's repo), MariaDB downgrade from 10.4.13 (MariaDB repo) to older (from CentOS official repo).
Change Linux distro into CentOS 7, with PHP7.3, nginx 1.16.1, older database and still no success!
Removing redis from config is some kind of solution but it slowdowns everything a lot.

  • All test above under fresh install.

My specific PHP/MariaDB configs that may have something to do.
/etc/php.ini:
memory_limit = 1024M
max_execution_time = 60
max_input_time = 120
post_max_size = 16384M
upload_max_filesize = 16384M
default_socket_timeout = 120
mysqli.cache_size = 2000

/etc/php-fpm.d/www.conf
pm.start_servers = 10
pm.max_children = 100

/etc/my.cnf.d/server.cnf
[server]
skip-name-resolve
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 32M
innodb_max_dirty_pages_pct = 90
query_cache_type = 1
query_cache_limit = 2M
query_cache_min_res_unit = 2k
query_cache_size = 64M
tmp_table_size= 64M
max_heap_table_size= 64M
slow-query-log = 1
slow-query-log-file = /var/log/mysql/slow.log
long_query_time = 1

[mysqld]
innodb_buffer_pool_size=1G
innodb_io_capacity=4000

Steps to reproduce

  1. Use desktop client for syncing from HDD to Nextcloud.
  2. Find errors in activity inside desktop (Windows x64 app) and inside Nextcloud thru website inside Settings \ Logging.

Expected behaviour

Find no errors while uploading files.

Actual behaviour

Getting Sabre\DAV\Exception\BadRequest errors.
Sabre\DAV\Exception\BadRequest: Expected filesize of 35358091 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 14827520 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side. /var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php - line 154:OCA\DAV\Connector\Sabre\File->put() /var/www/nextcloud/apps/dav/lib/Upload/UploadFolder.php - line 46:OCA\DAV\Connector\Sabre\Directory->createFile() /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 1104:OCA\DAV\Upload\UploadFolder->createFile()/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php - line 527:Sabre\DAV\Server->createFile() /var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php - line 89:Sabre\DAV\CorePlugin->httpPut() /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 474:Sabre\DAV\Server->emit() /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 251:Sabre\DAV\Server->invokeMethod() /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 319:Sabre\DAV\Server->start() /var/www/nextcloud/apps/dav/lib/Server.php - line 320:Sabre\DAV\Server->exec() /var/www/nextcloud/apps/dav/appinfo/v2/remote.php - line 35:OCA\DAV\Server->exec() /var/www/nextcloud/remote.php - line 167:require_once("/var/www/ne ... p")
Screenshots of Nextcloud:
screenshot00
screenshot01

Screenshots of Client:
screenshot03

Server configuration

Operating system:
cat /etc/redhat-release
CentOS Linux release 8.1.1911 (Core)
uname -a
Linux nextcloud 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Web server:'
nginx -V
nginx version: nginx/1.14.1
built by gcc 8.2.1 20180905 (Red Hat 8.2.1-3) (GCC)
built with OpenSSL 1.1.1 FIPS 11 Sep 2018 (running with OpenSSL 1.1.1c FIPS 28 May 2019)
TLS SNI support enabled

Database:
mariadb -V
mariadb Ver 15.1 Distrib 10.4.13-MariaDB, for Linux (x86_64) using readline 5.1

PHP version:
PHP-FPM connected with nginx thru socket
Version: 7.4.7
Memory Limit: 1 GB
Max Execution Time: 3600
Upload max size: 16 GB

Nextcloud version: (see Nextcloud admin page)
19.0.0.12

Updated from an older Nextcloud/ownCloud or fresh install:
Fresh install

Where did you install Nextcloud from:
nextcloud.com

Signing status:
No errors have been found.

List of activated apps:
sudo -u nginx php occ app:list
Enabled:

  • accessibility: 1.5.0
  • activity: 2.12.0
  • bruteforcesettings: 1.6.0
  • calendar: 2.0.3
  • cloud_federation_api: 1.2.0
  • comments: 1.9.0
  • contacts: 3.3.0
  • contactsinteraction: 1.0.0
  • dav: 1.15.0
  • federatedfilesharing: 1.9.0
  • federation: 1.9.0
  • files: 1.14.0
  • files_external: 1.10.0
  • files_pdfviewer: 1.8.0
  • files_rightclick: 0.16.0
  • files_sharing: 1.11.0
  • files_trashbin: 1.9.0
  • files_versions: 1.12.0
  • files_videoplayer: 1.8.0
  • firstrunwizard: 2.8.0
  • groupfolders: 6.0.6
  • logreader: 2.4.0
  • lookup_server_connector: 1.7.0
  • nextcloud_announcements: 1.8.0
  • notifications: 2.7.0
  • oauth2: 1.7.0
  • password_policy: 1.9.1
  • photos: 1.1.0
  • privacy: 1.3.0
  • provisioning_api: 1.9.0
  • recommendations: 0.7.0
  • serverinfo: 1.9.0
  • settings: 1.1.0
  • sharebymail: 1.9.0
  • support: 1.2.1
  • survey_client: 1.7.0
  • systemtags: 1.9.0
  • tasks: 0.13.1
  • text: 3.0.1
  • theming: 1.10.0
  • twofactor_backupcodes: 1.8.0
  • viewer: 1.3.0
  • workflowengine: 2.1.0
    Disabled:
  • admin_audit
  • encryption
  • updatenotification
  • user_ldap

Nextcloud configuration:
sudo -u nginx php occ config:list system
{
"system": {
"blacklisted_files": [],
"instanceid": "REMOVED SENSITIVE VALUE",
"passwordsalt": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE",
"trusted_domains": [
"localhost",
"nextcloud",
"127.0.0.1",
"192.168.50.2",
],
"datadirectory": "REMOVED SENSITIVE VALUE",
"dbtype": "mysql",
"version": "19.0.0.12",
"overwrite.cli.url": "http://nextcloud",
"dbname": "REMOVED SENSITIVE VALUE",
"dbhost": "REMOVED SENSITIVE VALUE",
"dbport": "",
"dbtableprefix": "1c_",
"mysql.utf8mb4": true,
"dbuser": "REMOVED SENSITIVE VALUE",
"dbpassword": "REMOVED SENSITIVE VALUE",
"installed": true,
"default_language": "pl",
"default_locale": "pl",
"memcache.local": "\OC\Memcache\Redis",
"filelocking.enabled": true,
"memcache.locking": "\OC\Memcache\Redis",
"memcache.distributed": "\OC\Memcache\Redis",
"redis": {
"host": "REMOVED SENSITIVE VALUE",
"port": 6379,
"dbindex": 0,
"timeout": 600
},
"skeletondirectory": "core/empty-default",
"maintenance": false
}
}

Are you using external storage, if yes which one: local/smb/sftp/...
There is smb plugin with attached storage, but it is not used for sync.

Are you using encryption:
no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
no

Client configuration

Browser:
Using Nextcloud desktop client.

Operating system:
Windows 10 Pro x64

Logs

Server respond "423 Locked" on "PUT http://192.168.50.2/remote.php/dav/files/USERNAME/dire...

Web server error log

nginx error.log is empty.

Nextcloud log (data/nextcloud.log)

{"reqId":"aG0WKf6Jk3WBKZzcfG5l","level":3,"time":"2020-06-12T10:10:42+00:00","remoteAddr":"192.168.50.5","user":"USERNAME","app":"no app in context","method":"PUT","url":"/remote.php/dav/uploads/USERNAME/2948563684/00000000","message":{"Exce
ption":"Sabre\DAV\Exception\BadRequest","Message":"Expected filesize of 35358091 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 14827520 bytes. Could either be a network problem on the sending side or a proble
m writing to the storage on the server side.","Code":0,"Trace":[{"file":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php","line":154,"function":"put","class":"OCA\DAV\Connector\Sabre\File","type":"->"},{"file":"/var/www
/nextcloud/apps/dav/lib/Upload/UploadFolder.php","line":46,"function":"createFile","class":"OCA\DAV\Connector\Sabre\Directory","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1104,"function":"cr
eateFile","class":"OCA\DAV\Upload\UploadFolder","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":527,"function":"createFile","class":"Sabre\DAV\Server","type":"->"},{"file":"/var/www/nextclo
ud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpPut","class":"Sabre\DAV\CorePlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":474,"function":"emit","class":"S
abre\DAV\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":251,"function":"invokeMethod","class":"Sabre\DAV\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Serv
er.php","line":319,"function":"start","class":"Sabre\DAV\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":320,"function":"exec","class":"Sabre\DAV\Server","type":"->"},{"file":"/var/www/nextcloud/apps/
dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\DAV\Server","type":"->"},{"file":"/var/www/nextcloud/remote.php","line":167,"args":["/var/www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"F
ile":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/File.php","Line":238,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Windows) mirall/2.6.4stable-Win64 (build 20200303) (Nextcloud)","version":"19.0.0.12"}
{"reqId":"aG0WKf6Jk3WBKZzcfG5l","level":4,"time":"2020-06-12T10:10:42+00:00","remoteAddr":"192.168.50.5","user":"USERNAME","app":"webdav","method":"PUT","url":"/remote.php/dav/uploads/USERNAME/2948563684/00000000","message":{"Exception":"Sab
re\DAV\Exception\BadRequest","Message":"Expected filesize of 35358091 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 14827520 bytes. Could either be a network problem on the sending side or a problem writing t
o the storage on the server side.","Code":0,"Trace":[{"file":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php","line":154,"function":"put","class":"OCA\DAV\Connector\Sabre\File","type":"->"},{"file":"/var/www/nextcloud/
apps/dav/lib/Upload/UploadFolder.php","line":46,"function":"createFile","class":"OCA\DAV\Connector\Sabre\Directory","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1104,"function":"createFile","
class":"OCA\DAV\Upload\UploadFolder","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":527,"function":"createFile","class":"Sabre\DAV\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty
/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpPut","class":"Sabre\DAV\CorePlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":474,"function":"emit","class":"Sabre\DAV\
Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":251,"function":"invokeMethod","class":"Sabre\DAV\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","li
ne":319,"function":"start","class":"Sabre\DAV\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":320,"function":"exec","class":"Sabre\DAV\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/appinfo
/v2/remote.php","line":35,"function":"exec","class":"OCA\DAV\Server","type":"->"},{"file":"/var/www/nextcloud/remote.php","line":167,"args":["/var/www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/
www/nextcloud/apps/dav/lib/Connector/Sabre/File.php","Line":238,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Windows) mirall/2.6.4stable-Win64 (build 20200303) (Nextcloud)","version":"19.0.0.12"}

Other Problems. Both are with some GUI elements. The bell on the top right with notifications have always red dot informing thet there are some notifications, but when i click on it there are no (then dot disappears). After refresh, or click on any link inside nextcloud red dot is appearing there.
screenshot04
screenshot05

Second problem is, when i start Nextcloud for first time, or click on my avatar -> About (About Nextcloud window). The first page is empty blue like on the screenshot.

screenshot06

Update1:
Solution for last problem with first run wizard is here: https://help.nextcloud.com/t/first-run-wizard-video-does-not-play-http-302/75281

Update2:
There is also a lot of errors:
session_start(): A session had already been started - ignoring at /var/www/nextcloud/lib/private/Session/Internal.php#209

But i find this is known problem for N19 and many people already reported that.

Update3:
it looks it may be problem with Redis cache. I'm making more tests,

Update4:
I changed 'memcache.local' => '\OC\Memcache\Redis', into 'memcache.local' => '\OC\Memcache\APCu'.
Deleting two lines in config.php fixed the problem:
'filelocking.enabled' => true,
'memcache.locking' => '\OC\Memcache\Redis',

I'll try to learn what is the problem, but i think for now i will leave it disabled so at least, it will work correctly.

Update5:
Fresh CentOS 7 install. Same problem.

@nicrame nicrame added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Jun 12, 2020
@KoffeinKaio
Copy link

KoffeinKaio commented Jul 6, 2020

Getting the same right now:

Chrome on Huawei P20 Pro Android 10 fails with below errors.
Chrome on Samsung S9 Plus with Android 10 and Chrome works like a charm
Chrome on Samsung S9 Plus with Android 10 and Firefox uploads one img of a batch at 0bytes

Fatal    webdav             Sabre\DAV\Exception\BadRequest: 2020-07-06T10:35:07+00:00 
                              Expected filesize of                             
                              2957473 bytes but read                           
                              (from Nextcloud                                  
                              client) and wrote (to                            
                              Nextcloud storage)                               
                              425984 bytes. Could                              
                              either be a network                              
                              problem on the sending                           
                              side or a problem                                
                              writing to the storage                           
                              on the server side. at                           
                              apps/dav/lib/Connector/Sabre/File.php                           
                              line 238                                         
                                                                               
                              0. .../Directory.php line 154                           
                                 OCA\DAV\Connector\Sabre\File->put(                           
                                   ... ll                                      
                                 )                                             
                              1. .../Server.php line 1104                           
                                 OCA\DAV\Connector\Sabre\Directory->createFile(                           
                                   ... g",                                     
                                   ... ll                                      
                                 )                                             
                              2. .../CorePlugin.php line 527                           
                                 Sabre\DAV\Server->createFile(                           
                                   "test\/IMG_ ... g",                           
                                   ... ll,                                     
                                   ... ll                                      
                                 )                                             
                              3. .../WildcardEmitterTrait.php line 89                           
                                 Sabre\DAV\CorePlugin->httpPut(                           
                                   ... {},                                     
                                   S ... {}                                    
                                 )                                             
                              4. .../Server.php line 474                           
                                 Sabre\DAV\Server->emit(                           
                                   ... T",                                     
                                   [Sabre\HTTP\Request {},Sabre\HTT ... }]                           
                                 )                                             
                              5. .../Server.php line 251                           
                                 Sabre\DAV\Server->invokeMethod(                           
                                   ... {},                                     
                                   ... {}                                      
                                 )                                             
                              6. .../Server.php line 319                           
                                 Sabre\DAV\Server->start(                           
                                                                               
                                 )                                             
                              7. .../publicwebdav.php line 112                           
                                 Sabre\DAV\Server->exec(                           
                                                                               
                                 )                                             
                              8. public.php line 81                            
                                 require_once(                                 
                                   "\/var\/www\/html\/apps\/dav\/appinfo\/v1\/publicwebdav. ... p"                           
                                 )                                             


Together with

  Error    PHP                Cannot modify header   2020-07-06T10:35:07+00:00 
                              information - headers                            
                              already sent by                                  
                              (output started at                               
                              /var/www/html/3rdparty/sabre/http/lib/Sapi.php:132)                           
                              at                                               
                              /var/www/html/apps/dav/lib/Connector/Sabre/File.php#680

Setup is on Nextcloud 19.0 on Docker with mysql/redis/nginx/php-fpm behind a Traefik 2.

I'll debug this a bit further and add informations as I find them.

@michel-thomas
Copy link

I also notice the same problem on only one instance of a shared server (several instances deployed).

  • Stack LAMP Debian 10 with mod_php (PHP 7.3)
  • all 5 Nextcloud are 18.0.6
  • the Nextcloud which have the problem has similar config.php than others
  • customers from faulty Nextcloud are all with nc-client 2.6.4 (Windows 10 and Mac)

Error is similar to described above:

[webdav] Fatal: Sabre\DAV\Exception\BadRequest: Expected filesize of 2657747 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 65536 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side. at <<closure>>

0. /home/cloud.my-domain.fr/www/apps/dav/lib/Connector/Sabre/Directory.php line 156
   OCA\DAV\Connector\Sabre\File->put(null)
1. /home/cloud.my-domain.fr/www/3rdparty/sabre/dav/lib/DAV/Server.php line 1096
   OCA\DAV\Connector\Sabre\Directory->createFile("Support_atelier_participatif_diag.pptx", null)
2. /home/cloud.my-domain.fr/www/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 525
   Sabre\DAV\Server->createFile("files/userx/A ... x", null, null)
3. <<closure>>
   Sabre\DAV\CorePlugin->httpPut(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
4. /home/cloud.my-domain.fr/www/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
   call_user_func_array([Sabre\DAV\CorePlugin {},"httpPut"], [Sabre\HTTP\Requ ... }])
5. /home/cloud.my-domain.fr/www/3rdparty/sabre/dav/lib/DAV/Server.php line 479
   Sabre\Event\EventEmitter->emit("method:PUT", [Sabre\HTTP\Requ ... }])
6. /home/cloud.my-domain.fr/www/3rdparty/sabre/dav/lib/DAV/Server.php line 254
   Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
7. /home/cloud.my-domain.fr/www/apps/dav/lib/Server.php line 319
   Sabre\DAV\Server->exec()
8. /home/cloud.my-domain.fr/www/apps/dav/appinfo/v2/remote.php line 35
   OCA\DAV\Server->exec()
9. /home/cloud.my-domain.fr/www/remote.php line 165
   require_once("/home/cloud.my- ... p")

PUT /remote.php/dav/files/userx/Commun%20Partage/06%20Production/01%20Dossiers%20clients/R%C3%A9alit%C3%A9s/03%20Missions/0%20Diagnostic/Support_atelier_participatif_diag.pptx
from 90.63.151.226 by userx at 2020-07-16T14:50:57+02:00

@jaalzateolaya
Copy link

Same here:

  • Private dockerized server:
    • DBMS: yobasystems/alpine-mariadb@latest
    • nextcloud@latest
  • Version Nextcloud 19.0.1

Error: [webdav] Fatal: Sabre\DAV\Exception\BadRequest: Expected filesize of [...]

PHP

Version: 7.4.8

Memory Limit: 512 MB

Max Execution Time: 3600

Upload max size: 2 MB

DB

Type: mysql

Version: 10.4.13

Size: 24,7 MB

System

Operating System: 	Linux 5.4.51-1-ARCH armv7l
CPU: 	ARMv7 Processor rev 3 (v7l) (4 cores)
Memory: 	3.75 GB
Server time: 	Thu Aug 20 00:56:51 UTC 2020
Uptime: 	6 days, 7 hours, 53 minutes, 33 seconds
Time Servers:

Client system: MS Windows 10

@pupboss
Copy link

pupboss commented Aug 20, 2020

@nicrame @KoffeinKaio @jaalzateolaya @esprit-libre

Try to place the source code and data repository on different disks, most possibly it was caused by IO limitation, even in some cases the website will not respond. Another solution is to turn on OPcache, also help

@michel-thomas
Copy link

On my side, OPCache is on. IO disk limitation... I will check: any idea to have relevant metrics? (iotop ?)

But strange thing is that problem occurs mainly (not 100% but nearly) with one instance, and not with the others on the same server. So I'm not confident that this is IO disk limitation.

@pupboss
Copy link

pupboss commented Aug 20, 2020

On my side, OPCache is on. IO disk limitation... I will check: any idea to have relevant metrics? (iotop ?)

But strange thing is that problem occurs mainly (not 100% but nearly) with one instance, and not with the others on the same server. So I'm not confident that this is IO disk limitation.

Aside of WebDAV syncing, I have another similar issue, php-fpm got stuck when I watch videos...But finally all were solved by placing source code and data into different disks.

In terms of disk utilisation, first you can check by top command, seeing the average load, if CPU & memory use is low, but a high load, most possibly it is caused by IO limitation (for example CPU use is 2% but the load is over 40). Secondly, you can use sudo iotop or iostat -x 2 (print the status every 2 seconds).

But if all look good, good luck to you haha, because I have no other ideas.

@nicrame
Copy link
Author

nicrame commented Aug 20, 2020

@nicrame @KoffeinKaio @jaalzateolaya @esprit-libre

Try to place the source code and data repository on different disks, most possibly it was caused by IO limitation, even in some cases the website will not respond. Another solution is to turn on OPcache, also help

From the beginning i got different HDD for OS, and different for Nextcloud with data directory. Also i got enabled OPCache and APCu. I've got only 4 users, and most of the time Nextcloud do nothing. Sometimes this error occurs, when some file is uploaded, but even then there is only that 1 file transfer :S

@Schroeffu
Copy link

I experience the same problem with NC 18.0.4 on mod-php 7.3 and Client 2.6.4 with SMB/CIFS Share External Storage.

Until now i do not know where the problem/disconnect while upload occurs. A Timeout/Blocker of PHP/Apache2 or something behind towards the External Storage... All the uploaders are using Notebooks with LTE connection, the broken uploads are often only 500kb.

It happens here very often and is going to be a real problem productively.

Example Error Message:

:{"Exception":"Sabre\\DAV\\Exception\\BadRequest","Message":"Expected filesize of 344294 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 114688 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.

@nunong21
Copy link

I'm having the same issue with the webclient.
Don't know what the hell is causing this, but everytime I try to copy like 22 RAW images, I get this error on at least 10 of them.

Also reported this issue on nextcloud foruns, but so far, no help, https://help.nextcloud.com/t/timeouts-while-uploading-multiple-files/90717

@fermulator
Copy link

fermulator commented Nov 14, 2020

"me too" ... more information/data :

SAMPLE:

[webdav] Fatal: Sabre\DAV\Exception\BadRequest: Expected filesize of 1927500 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 311296 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side. at <<closure>>

0. /srv/nextcloud/apps/dav/lib/Connector/Sabre/Directory.php line 154
   OCA\DAV\Connector\Sabre\File->put(null)
1. /srv/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 1104
   OCA\DAV\Connector\Sabre\Directory->createFile("2020-08-29_104237_2422.jpg", null)
2. /srv/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 527
   Sabre\DAV\Server->createFile("Family/Photos/0 ... g", null, null)
3. /srv/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php line 89
   Sabre\DAV\CorePlugin->httpPut(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})
4. /srv/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 474
   Sabre\DAV\Server->emit("method:PUT", [Sabre\HTTP\Requ ... }])
5. /srv/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 251
   Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})
6. /srv/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php line 319
   Sabre\DAV\Server->start()
7. /srv/nextcloud/apps/dav/appinfo/v1/webdav.php line 82
   Sabre\DAV\Server->exec()
8. /srv/nextcloud/remote.php line 167
   require_once("/srv/nextcloud/ ... p")

PUT /cloud/remote.php/webdav/Family/Photos/0%20_%20Incoming%20(Uploads)/foo/2020/08/2020-08-29_104237_2422.jpg
from 10.0.0.52 by fermulator at 2020-11-14T18:41:49+00:00
  • Using "Auto Upload" from iOS 14.2 iPhone
  • Large files (Camera Videos) >100MB nearly ALWAYS fail (probability)
  • Smaller files (Camera Photos) <5MB fail sporadically (even small <1MB files fail)
  • Hitting through WiFi strong signal, server is local (so routed internally)
  • Backend is an external drive (SMB/CIFS)

Version info:

  • Nextcloud 19.0.4, in a VM
Operating System: | Linux 4.15.0-99-generic x86_64
-- | --
CPU: | Westmere E56xx/L56xx/X56xx (Nehalem-C) (4 cores)
Memory: | 1.95 GB
apache2                                2.4.29-1ubuntu4.14 << nextcloud
apache2                               2.4.18-2ubuntu3.14  << proxy
  • php:
Version: 7.2.24
Memory Limit: 512 MB
Max Execution Time: 3600
Upload max size: 2 MB
  • DB:
Type: mysql
Version: 10.1.47
Size: 63.5 MB

config

root@nextcloud:/srv/nextcloud# cat config/config.php
<?php
$CONFIG = array (
  'instanceid' => 'SNIP',
  'passwordsalt' => 'SNIP',
  'secret' => 'SNIP',
  'trusted_domains' => 
  array (
    0 => 'nextcloud',
    1 => 'SNIP',
  ),
  'trusted_proxies' => 
  array (
    0 => '1.0.0.52',
  ),
  'overwritehost' => 'SNIP',
  'overwriteprotocol' => 'https',
  'overwritewebroot' => '/cloud',
  'overwritecondaddr' => '^1\\.0\\.0\\.52$',
  'datadirectory' => '/opt/nextcloud/',
  'dbtype' => 'mysql',
  'version' => '19.0.4.2',
  'overwrite.cli.url' => 'https://nextcloud',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'nextcloud_admin',
  'dbpassword' => 'SNIP',
  'installed' => true,
  'theme' => '',
  'loglevel' => 2,
  'maintenance' => false,
  'updater.release.channel' => 'stable',
  'mail_smtpmode' => 'smtp',
  'mail_smtphost' => 'SNIP',
  'mail_sendmailmode' => 'smtp',
  'mail_smtpport' => '465',
  'mail_smtpsecure' => 'ssl',
  'mail_from_address' => 'nextcloud',
  'mail_domain' => 'SNIP',
  'mail_smtpauthtype' => 'PLAIN',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'SNIP',
  'mail_smtppassword' => 'SNIP',
  'updater.secret' => 'SNIP',
  'app_install_overwrite' => 
  array (
    0 => 'breezedark',
  ),
);

Other Observations / Tests

  • am going through an Apache proxy : BUT: I monitored the logs on both proxy and nextcloud, and there are no "errors" correlated with these events
  • looks like parallel uploads are enabled (4x max)
  • iotop is decently quite, no major bottleneck here afaict (the WiFi is slower than disk capabilities)
  • CPU is busy ~75% over 4 cores, but not maxed; (apache2 & smbclient primarily)
  • memory is <30% utilized

EDIT/UPDATE:

on further observation, actually, the Apache2 proxy logs DO HAVE these:

[Sat Nov 14 14:25:10.266611 2020] [proxy_http:error] [pid 5859] (104)Connection reset by peer: [client 10.0.0.1:51291] AH02609: read request body failed to 10.0.0.55:443 (nextcloud) from 10.0.0.1 ()
[Sat Nov 14 14:25:10.266792 2020] [proxy_http:error] [pid 5859] [client 10.0.0.1:51291] AH01097: pass request body failed to 10.0.0.55:443 (nextcloud) from 10.0.0.1 ()

this seems to roughly correlate with the errors seen in Nextcloud backend; (they are not 1:1 or perfect correlation, BUT, the timestamps align)

perhaps in my case (and for others) if you too are using Apache2 reverse proxy for Nextcloud, ... if the proxy timeout itself is killing the connection / OR / nextcloud is killing/resetting the connection ... this might be the cause? (TBD on solution)

TESTING: nextcloud:~$ sudo a2dismod reqtimeout ... MONITORING WILL REPORT BACK

I dunno ... at first it seemed to help, but now it is uber slow (~ <500KB/sec), and when I restarted the "send full camera roll" the errors started happening again

@Schroeffu
Copy link

Schroeffu commented Nov 15, 2020

We are moving our production to seafile after weeks of troubleshooting with no result, due to this unresolveable bug, 30% moved already. I'll no longer help here with logs and configs -.-

Edit: fyi, now with seafile we are avoiding webdav+sabredav completely, the sync from linux seafile server to windows share is done by their original client prepared as a windows service.

@szaimen
Copy link
Contributor

szaimen commented Jun 17, 2021

Is this Issue still valid in NC21.0.2? If not, please close this issue. Thanks! :)

@matthewlswanson
Copy link

Is this Issue still valid in NC21.0.2? If not, please close this issue. Thanks! :)

Yes! I installed manually and also via nextcloudvm script and they both have this issue!

@szaimen szaimen added 1. to develop Accepted and waiting to be taken care of and removed 0. Needs triage Pending check for reproducibility or if it fits our roadmap needs info labels Jun 25, 2021
@dgedgedge
Copy link

dgedgedge commented Jul 24, 2021

FYI : I have the issue on my server :
Sabre\DAV\Exception\BadRequest: Expected filesize of 16992525208 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 4482760704 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.

  • Facts :

  • My Hypothesis :

    • Upload size could be limited by some parameters of Python/php ?
    • Limitation of computing duration (Instead of computing consumption ?
    • Bad magagement of file index while pushing the file to server ?
    • Error in the retrieval of the size of the file on one size ?
    • Error in the moment of the retrieval of the size of the file on one size ?
  • Server configuration detail

Distribution: Debian Buster

Operating system: Linux 4.19.0-17-amd64 #1 SMP Debian 4.19.194-2 (2021-06-21) x86_64

Webserver: Apache (apache2handler)

Database: mysql 10.3.29 (Mariadb)

PHP version: 7.3.29-1~deb10u1

Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, sodium, session, standard, apache2handler, mysqlnd, PDO, xml, apcu, bcmath, bz2, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, gmp, iconv, igbinary, imagick, intl, json, exif, msgpack, mysqli, pdo_mysql, apc, posix, readline, redis, shmop, SimpleXML, sockets, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, Phar, memcached, Zend OPcache

Nextcloud version: 21.0.3 - 21.0.3.1

  • Updated from an older Nextcloud/ownCloud or fresh install: Regular update each time there is a new stable release, since 2 years.

Where did you install Nextcloud from: Initialy = Tar + followup install process.

Signing status

Array ( )

List of activated apps

Enabled:

  • accessibility: 1.7.0
  • activity: 2.14.3
  • admin_audit: 1.11.0
  • analytics: 3.5.1
  • bruteforcesettings: 2.2.0
  • calendar: 2.3.1
  • carnet: 0.24.2
  • checksum: 1.1.2
  • cloud_federation_api: 1.4.0
  • comments: 1.11.0
  • contacts: 4.0.0
  • contactsinteraction: 1.2.0
  • cookbook: 0.8.4
  • dashboard: 7.1.0
  • dav: 1.17.1
  • deck: 1.4.3
  • documentserver_community: 0.1.11
  • drawio: 1.0.1
  • federatedfilesharing: 1.11.0
  • files: 1.16.0
  • files_external: 1.12.0
  • files_markdown: 2.3.4
  • files_mindmap: 0.0.24
  • files_pdfviewer: 2.1.0
  • files_rightclick: 1.0.0
  • files_sharing: 1.13.1
  • files_texteditor: 2.14.0
  • files_trashbin: 1.11.0
  • files_versions: 1.14.0
  • files_videoplayer: 1.10.0
  • firstrunwizard: 2.10.0
  • geoblocker: 0.5.1
  • keeweb: 0.6.5
  • logreader: 2.6.0
  • lookup_server_connector: 1.9.0
  • maps: 0.1.9
  • music: 1.2.1
  • nextcloud_announcements: 1.10.0
  • notes: 4.1.0
  • notifications: 2.9.0
  • oauth2: 1.9.0
  • onlyoffice: 7.0.2
  • password_policy: 1.11.0
  • photos: 1.3.0
  • polls: 3.0.0
  • privacy: 1.5.0
  • provisioning_api: 1.11.0
  • recommendations: 1.0.0
  • serverinfo: 1.11.0
  • settings: 1.3.0
  • sharebymail: 1.11.0
  • spreed: 11.3.1
  • support: 1.4.0
  • survey_client: 1.9.0
  • systemtags: 1.11.0
  • tasks: 0.14.0
  • text: 3.2.0
  • theming: 1.12.0
  • twofactor_backupcodes: 1.10.0
  • updatenotification: 1.11.0
  • user_status: 1.1.1
  • viewer: 1.5.0
  • weather: 1.7.5
  • weather_status: 1.1.0
  • workflowengine: 2.3.0
    Disabled:
  • encryption
  • federation
  • user_ldap
Configuration (config/config.php)

{
"instanceid": "REMOVED SENSITIVE VALUE",
"passwordsalt": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE",
"trusted_domains": [
"owncloud.fager.fr",
"192.168.0.110",
"89.156.37.141"
],
"datadirectory": "REMOVED SENSITIVE VALUE",
"overwrite.cli.url": "https://owncloud.fager.fr",
"dbtype": "mysql",
"version": "21.0.3.1",
"dbname": "REMOVED SENSITIVE VALUE",
"dbhost": "REMOVED SENSITIVE VALUE",
"dbport": "",
"dbtableprefix": "oc_",
"dbuser": "REMOVED SENSITIVE VALUE",
"dbpassword": "REMOVED SENSITIVE VALUE",
"loglevel": 2,
"logtimezone": "Europe/Paris",
"logfile": "/var/log/nextcloud.log",
"log_rotate_size": 0,
"installed": true,
"memcache.local": "\OC\Memcache\APCu",
"mail_from_address": "REMOVED SENSITIVE VALUE",
"mail_smtpmode": "smtp",
"mail_domain": "REMOVED SENSITIVE VALUE",
"mail_smtphost": "REMOVED SENSITIVE VALUE",
"maintenance": false,
"theme": "",
"default_phone_region": "FR",
"updater.release.channel": "stable",
"mail_smtpauth": 1,
"mail_sendmailmode": "smtp",
"mail_smtpauthtype": "LOGIN",
"mail_smtpname": "REMOVED SENSITIVE VALUE",
"mail_smtppassword": "REMOVED SENSITIVE VALUE",
"mysql.utf8mb4": true,
"trashbin_retention_obligation": "auto, 100",
"has_rebuilt_cache": true,
"app_install_overwrite": [
"files_reader"
]
}

External storages: yes

External storage configuration

No mounts configured

Encryption: no

User-backends:

OC\User\Database

Browser: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:90.0) Gecko/20100101 Firefox/90.0

@ggrzeczkowicz
Copy link

Does disabling HTTP/2 solve the problem (as proposed in #16138)?

@dgedgedge
Copy link

dgedgedge commented Sep 1, 2021

Does disabling HTTP/2 solve the problem (as proposed in #16138)?

The module http2 was not enabled on my configuration.



ls /etc/apache2/mods-enabled/
access_compat.load  auth_basic.load  authz_core.load  autoindex.conf  deflate.load  env.load      mime.conf         mpm_prefork.load  php7.3.conf      reqtimeout.load  setenvif.load       ssl.load
alias.conf          authn_core.load  authz_host.load  autoindex.load  dir.conf      filter.load   mime.load         negotiation.conf  php7.3.load      rewrite.load     socache_shmcb.load  status.conf
alias.load          authn_file.load  authz_user.load  deflate.conf    dir.load      headers.load  mpm_prefork.conf  negotiation.load  reqtimeout.conf  setenvif.conf    ssl.conf            status.load


$ sudo a2dismod http2
Module http2 already disabled

@nicrame
Copy link
Author

nicrame commented Sep 1, 2021

I also have disabled http2 on my Nginx.

@isaacolsen94
Copy link

isaacolsen94 commented Sep 28, 2021

Was there a fix to this? I also don't have http2 enabled and have this error

Edit: NC22.1.1 stable

@Wouter0100
Copy link

Wouter0100 commented Oct 7, 2021

I had the same issue, and I found the problem. In my case it was an unexpected place where the tmp files were written, which ran out of space.

PHP writes uploads to the tmp folder. I run docker, which means that by default it would get written to the /tmp directory within that container. The containers filesystem is present on my / mount, which is a 10GB disk just enough to run Docker with Nextcloud. In the start I tried to upload larger files (1GB+), but this would work - although now this drive is filling up a bit more due to automated updates with watchtower and logs. This would mean that the available space was just a couple of 100MB's, resulting in this error.

I fixed it by mapping the /tmp directory from within the container to my "data" mount (where all data is stored as wel and where I have enough storage available).

The only downside is in this case the tmp data is first written to the HDD's and then again written to the same HDD's. This could slow it down a bit, as we need to read AND write to the same HDD's. It may be useful just to expand the root mount, as this is SSD based.

If you don't have docker and have tmpfs on /tmp, it could also be that you're running out of RAM.

@julian70400
Copy link

julian70400 commented Dec 17, 2021

Same issue here, Nextcloud 21.0.7

Modifying chunks size ? sudo -u www-data php occ config:app:set files max_chunk_size --value 1048000

@Wouter0100
Copy link

Wouter0100 commented Jan 18, 2022

I honestly think this is a local configuration problem, just like I had. Did you guys check RAM usage and/or disk usage while uploading a large file?

@nicrame
Copy link
Author

nicrame commented Feb 28, 2022

After some long time, i still got those problems in my log, while many things changed.
Currently i'm using Nextcloud 22.2.5 and Client 3.4.2 (just installed 3.4.3).

I've changed motherboard with CPU (it is Ryzen 5 5600G now), there is fast Samsung SSD 870 EVO 4TB, with another HDD for backup (Seagate Barracuda 2.5 5400/ST4000LM024-2AN17V), 32GB DDR4 RAM. I've also changed ethernet device into Mellanox ConnectX-3 connected at 10Gb/s. Same time my laptop is also connected to the same switch with 10Gb/s connection.
Main OS have been changed into stable RHEL 8.5. I have APC cache enabled and Redis for locking files.
RAM usage is currently about 1,6GB. iotop shows total disk write about 100-700K/s (i'm just uploading files from computer far away, some files that are about 12MB on very slow internet connection), so IO is 0,00% sometimes grow up to 0.9%. So hardware or OS can't be problem here on any level.

Server configuration

Operating system:
cat /etc/redhat-release
Red Hat Enterprise Linux release 8.5 (Ootpa)
uname -a
Linux nextcloud 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Mon Jan 17 07:06:06 EST 2022 x86_64 x86_64 x86_64 GNU/Linux

Web server:'
nginx -V
nginx version: nginx/1.14.1
built by gcc 8.2.1 20180905 (Red Hat 8.2.1-3) (GCC)
built with OpenSSL 1.1.1 FIPS 11 Sep 2018 (running with OpenSSL 1.1.1k FIPS 25 Mar 2021)
TLS SNI support enabled

Database:
mariadb -V
mariadb Ver 15.1 Distrib 10.5.15-MariaDB, for Linux (x86_64) using readline 5.1

PHP version:
PHP-FPM connected with nginx thru socket
Version: 7.4.28
Memory Limit: 1 GB
Max Execution Time: 3600
Upload max size: 16,5 GB

Nextcloud version: (see Nextcloud admin page)
22.2.5.1

Updated from an older Nextcloud/ownCloud or fresh install:
Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from:
nextcloud.com

Signing status:
No errors have been found.

List of activated apps:
sudo -u nginx php occ app:list
Enabled:

  • accessibility: 1.8.0
  • activity: 2.15.0
  • bruteforcesettings: 2.3.0
  • calendar: 3.0.6
  • camerarawpreviews: 0.7.15
  • circles: 22.2.0
  • cloud_federation_api: 1.5.0
  • comments: 1.12.0
  • contacts: 4.0.8
  • contactsinteraction: 1.3.0
  • dashboard: 7.2.0
  • dav: 1.20.0
  • deck: 1.5.5
  • extract: 1.3.3
  • federatedfilesharing: 1.12.0
  • federation: 1.12.0
  • files: 1.17.0
  • files_external: 1.13.1
  • files_pdfviewer: 2.3.1
  • files_photospheres: 1.22.0
  • files_rightclick: 1.1.0
  • files_sharing: 1.14.0
  • files_trashbin: 1.12.0
  • files_versions: 1.15.0
  • files_videoplayer: 1.11.0
  • firstrunwizard: 2.11.0
  • groupfolders: 10.0.2
  • logreader: 2.7.0
  • lookup_server_connector: 1.10.0
  • mail: 1.11.7
  • metadata: 0.15.0
  • nextcloud_announcements: 1.11.0
  • notes: 4.3.1
  • notifications: 2.10.1
  • oauth2: 1.10.0
  • password_policy: 1.12.0
  • photos: 1.4.0
  • privacy: 1.6.0
  • provisioning_api: 1.12.0
  • recommendations: 1.1.0
  • serverinfo: 1.12.0
  • settings: 1.4.0
  • sharebymail: 1.12.0
  • spreed: 12.2.3
  • support: 1.5.0
  • survey_client: 1.10.0
  • systemtags: 1.12.0
  • tasks: 0.14.2
  • text: 3.3.0
  • theming: 1.13.0
  • twofactor_backupcodes: 1.11.0
  • updatenotification: 1.12.0
  • user_status: 1.2.0
  • viewer: 1.6.0
  • weather_status: 1.2.0
  • workflowengine: 2.4.0
    Disabled:
  • admin_audit
  • dicomviewer: 1.2.3
  • encryption: 2.9.0
  • forms: 2.4.0
  • maps: 0.1.10
  • ransomware_protection: 1.12.0
  • talk_matterbridge: 1.23.2
  • user_ldap

Nextcloud configuration:
sudo -u nginx php occ config:list system
{
"system": {
"blacklisted_files": [],
"instanceid": "REMOVED SENSITIVE VALUE",
"passwordsalt": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE",
"trusted_domains": [
"localhost",
"nextcloud",
"127.0.0.1",
"192.168.50.2",
],
"enable_previews": true,
"enabledPreviewProviders": [
"OC\Preview\TXT",
"OC\Preview\MarkDown",
"OC\Preview\PDF",
"OC\Preview\Image",
"OC\Preview\Photoshop",
"OC\Preview\TIFF",
"OC\Preview\SVG",
"OC\Preview\Font",
"OC\Preview\MP3",
"OC\Preview\Movie",
"OC\Preview\MKV",
"OC\Preview\MP4",
"OC\Preview\AVI"
],
"datadirectory": "REMOVED SENSITIVE VALUE",
"dbtype": "mysql",
"version": "22.2.5.1",
"overwrite.cli.url": "https://xxxxx.com",
"dbname": "REMOVED SENSITIVE VALUE",
"dbhost": "REMOVED SENSITIVE VALUE",
"dbport": "",
"dbtableprefix": "nas_",
"mysql.utf8mb4": true,
"dbuser": "REMOVED SENSITIVE VALUE",
"dbpassword": "REMOVED SENSITIVE VALUE",
"installed": true,
"default_language": "pl",
"default_locale": "pl",
"simpleSignUpLink.shown": false,
"memcache.local": "\OC\Memcache\APCu",
"memcache.locking": "\OC\Memcache\Redis",
"redis": {
"host": "REMOVED SENSITIVE VALUE",
"port": 0,
"timeout": 2.5
},
"skeletondirectory": "core/bones",
"maintenance": false,
"app_install_overwrite": [
"bruteforcesettings",
"dicomviewer",
"files_photospheres"
],
"mail_smtpmode": "smtp",
"mail_smtpsecure": "tls",
"mail_sendmailmode": "smtp",
"mail_domain": "REMOVED SENSITIVE VALUE",
"mail_smtpauthtype": "LOGIN",
"mail_smtpauth": 1,
"mail_smtphost": "REMOVED SENSITIVE VALUE",
"mail_smtpport": "25",
"mail_from_address": "REMOVED SENSITIVE VALUE",
"mail_smtpname": "REMOVED SENSITIVE VALUE",
"mail_smtppassword": "REMOVED SENSITIVE VALUE",
"default_phone_region": "PL",
"theme": "",
"loglevel": 2,
"updater.release.channel": "stable",
"preview_max_memory": "512"
}
}

Are you using external storage, if yes which one: local/smb/sftp/...
There is smb plugin with attached storage, but it is not used for sync in clients.

Are you using encryption:
no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
no

Client configuration

Browser:
Using Nextcloud desktop client.

Operating system:
Windows 10 Pro x64

Logs

[no app in context] Error: Expected filesize of 5335550 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 4341760 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.

Web server error log

nginx error.log is empty.

@firemdkfighter
Copy link

I met this issue too. I'm using S3, with the setting 'uploadPartSize' => '5242880'
Without this setting there is no issue.

@idl0r
Copy link

idl0r commented Jul 17, 2022

It really seems to be that it's using /tmp as temp dir which is, in my case, very limited and restricted.
Setting upload_tmp_dir via FPM worked.
It looks like it's storing the uploaded data into chunks of temp files and those chunks have exceeded my tmp directory space

Is it possible to configure the chunk size? In that case I could keep using my restricted and limited ramdisk /tmp

@solracsf solracsf changed the title Sabre\DAV\Exception\BadRequest on many files error NC19 Client 2.6.4Stable Error: Sabre\DAV\Exception\BadRequest: Expected filesize of... Oct 28, 2022
@Geexirooz
Copy link

Have you guys tried disabling IPv6 on your client? cause that worked for me.
I hope this helps someone

@nicrame I see you are so upset about this issue, please try this and see if it works.

@nicrame
Copy link
Author

nicrame commented Jan 6, 2023

Have you guys tried disabling IPv6 on your client? cause that worked for me. I hope this helps someone

@nicrame I see you are so upset about this issue, please try this and see if it works.

I'm not upset, i just like to give as much information as possible to fix things, especially when devs can't reproduce problems. But this error do not show for me anymore, but it may be related the OPCache that i got disabled in PHP, because it crashed whole PHP-FPM sometimes, and the reason was opcache library itself (confirmed by php devs). So there is possibility, that the whole problem was some php library problem.

@szaimen
Copy link
Contributor

szaimen commented Jan 9, 2023

Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!

@szaimen szaimen added needs info 0. Needs triage Pending check for reproducibility or if it fits our roadmap and removed 1. to develop Accepted and waiting to be taken care of labels Jan 9, 2023
@StefanSa
Copy link

Hi there,
unfortunately we also have the problem and do not understand why.
As a basis serves the well preconfigured VM from hanssonit. I think that we as most here, all optimization proposals be tested from the community or directly from nextcloud itself. At least none of these suggestions really help.

@szaimen Simon,
I just wonder how schools or universities that have significantly more users and traffic than most of us here manage to do that ?
For information, the problem also exists with the 25.03.
Simon, should you possibly want to debug the server, we can arrange that -> PM to me.

@szaimen
Copy link
Contributor

szaimen commented Jan 23, 2023

I think most likely is also the Nextcloud VM not following all things mentioned in https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/big_file_upload_configuration.html

@StefanSa
Copy link

StefanSa commented Jan 23, 2023

Simon, so far modified everything that we recommended and suggested.
It is "not" a problem of file size, i can upload a single file up to 16GB.
The problem seems to be multiple files and folders.
If we try to upload a project folder for example with about 30GB that has different file sizes, exactly this error happens.
We can upload the discarded / blocked packages individually afterwards without any problem.

@szaimen
Copy link
Contributor

szaimen commented Jan 23, 2023

So you modified the config manually and adjusted each of the values like

php_value upload_max_filesize 
php_value post_max_size 
php_value max_input_time 
php_value max_execution_time 
[LimitRequestBody](https://httpd.apache.org/docs/current/en/mod/core.html#limitrequestbody)
[SSLRenegBufferSize](https://httpd.apache.org/docs/current/mod/mod_ssl.html#sslrenegbuffersize)
[Timeout](https://httpd.apache.org/docs/current/mod/core.html#timeout)
[ProxyTimeout](https://httpd.apache.org/docs/current/mod/mod_proxy.html#proxytimeout)
upload_max_filesize
post_max_size
upload_tmp_dir

and it still does not work? Are you using a reverse proxy maybe? You will need to adjust the values in there as well. Also please make sure that the timeouts are high enough.

@StefanSa
Copy link

Simon,
This is the current configuration where this problem occurs. Is something missing here?
We also see the error message "server connection lost" sporadically, but rarely.

CPU 8 Core // 32GB RAM

php_value upload_max_filesize = 16G
php_value post_max_size = 16G
php_value max_input_time = 3600
php_value max_execution_time 3= 600
php_value max_file_uploads = 400
php_value output_buffering = 0
php_value upload_tmp_dir = > to 3TB NAS Storage
max_chunk_size = 0
[LimitRequestBody](https://httpd.apache.org/docs/current/en/mod/core.html#limitrequestbody) = LimitRequestBody 0
[SSLRenegBufferSize](https://httpd.apache.org/docs/current/mod/mod_ssl.html#sslrenegbuffersize) = SSLRenegBufferSize 201048600
[ProxyTimeout](https://httpd.apache.org/docs/current/mod/mod_proxy.html#proxytimeout) = RequestReadTimeout body=0

@szaimen
Copy link
Contributor

szaimen commented Jan 23, 2023

These look suspicious to me but I don't have the time to help you finding out the best values for your specific case.

php_value max_input_time = 3600
php_value max_execution_time 3= 600
php_value max_file_uploads = 400
php_value upload_tmp_dir = > to 3TB NAS Storage
max_chunk_size = 0

@StefanSa
Copy link

Simon this is rather suboptimal.
Don't you have any standard configuration that you could help us with in general for large files or similar scenarios?
Based on the age and length of the ticket, we are not alone in having this problem.
We are currently starting a pilot phase for nextcloud here in order to roll it out later on a larger scale.
It may be better to migrate to seafile, as a user here in the ticket has already suggested.....

thx for youre time and help here.

@szaimen
Copy link
Contributor

szaimen commented Jan 23, 2023

We are currently starting a pilot phase for nextcloud here in order to roll it out later on a larger scale.

Best is to contact https://nextcloud.com/de/contact/ then as they will have the time to help you and find out the best values for your specific case.

@szaimen
Copy link
Contributor

szaimen commented Jan 23, 2023

Apart from that there is this: https://github.com/nextcloud/all-in-one which should handle things correctly out of the box.

And https://help.nextcloud.com for help.

@chekesote
Copy link

Hope this help a little, I was looking how to fix this issue for 6 hours. my server is a 96GB RAM with 24 core CPU. my working pc is a windows 11 connected to a 100 MBps ethernet cable connected to a Cisco switch port.
My nextlcoud client was trying to syncronizing big files (around 10 GB ) and I realized that my ethernet connection drop for a few seconds until nextcloud client stop displaying a error and nextcloud log displayed "Sabre\DAV\Exception\BadRequest: Expected filesize of 35358091 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 14827520 bytes. Could either be a network problem on the sending side or a problem writing.... "

I tested hard disk and change many php and nginx settings but nothing works, So I change the atached port of my pc to a 1000 Mbps ports and then it works perfectly fine.. So I guess it's related to weird incompatibility with nextcloud client and ethernet card or even windows drivers?

@szaimen
Copy link
Contributor

szaimen commented Jan 27, 2023

So I change the atached port of my pc to a 1000 Mbps ports and then it works perfectly fine.. So I guess it's related to weird incompatibility with nextcloud client and ethernet card or even windows drivers?

This points indeed towards a bad ethernet card or windows drivers indeed.

@chrisv5179
Copy link

I also had this issue with a fresh deployed Nextcloud 25.0.3 and even small files.
These settings in php.ini fixed the problem:

  • sys_temp_dir : set to a directory with enough free space
  • open_basedir : has to contain the nextcloud home directory, data directory, /dev/urandom and the sys_temp_dir
  • output_buffering = Off
    (php.ini : depends on your linux flavour, on Debian based systems /etc/php/7.4/apache2/php.ini )

Hope this will help some of you.

@szaimen
Copy link
Contributor

szaimen commented Feb 10, 2023

Thanks @chrisv5179 for the update!

@szaimen szaimen closed this as completed Feb 10, 2023
@szaimen
Copy link
Contributor

szaimen commented Feb 10, 2023

As this seems to be a setup issue I would like to ask you to raise your question in the forums: https://help.nextcloud.com
If you wish support with setup issues from Nextcloud GmbH we offer this as part of the Nextcloud subscription. Learn more about this at https://nextcloud.com/enterprise/

@LiloBzH
Copy link

LiloBzH commented Mar 7, 2023

Hi

Same problem here but crazy thing. It's ok with CHROME but not with FIREFOX !

Sabre\DAV\Exception\BadRequest: Expected filesize of 20971520 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 15327232 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.

Debian 11
NextCloud 25.0.4

@steffen-kdab
Copy link

We have the same problem with the Nextcloud 25.0.4.1 Docker image.

@szaimen
Copy link
Contributor

szaimen commented Mar 21, 2023

Feel free to create a new issue here: https://github.com/nextcloud/docker/issues

@ZE3kr
Copy link
Contributor

ZE3kr commented Mar 30, 2023

I also encountered this error. It takes me hours to find out the problem. In my case, it is a problem for the client. I use Nextcloud for Mac. However, the default power management settings for MacBook will put the system to sleep when the display is off (even when powered on and the lid is open). When the Mac is sleeping, the network will disconnect periodically. If I prevent my mac from sleeping, this error never occurs again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap bug feature: logging needs info
Projects
None yet
Development

No branches or pull requests