[Bug]: S3 Multipart Retry counts bytes across attempts causing fails upload

[ching-kuo]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

Large uploads to an S3-compatible primary object store can fail after a retry even though multipart parts were uploaded successfully.

This does not look like the same issue as external storage timeouts. In this case the failure happens in lib/private/Files/ObjectStore/S3ObjectTrait.php after a retry.

writeMultiPart() retries once after S3MultipartUploadException, but $totalWritten is initialized outside the retry loop. If the first attempt uploaded parts before failing, the second attempt adds its bytes on top of the first attempt's bytes. Then before_complete throws:

Incomplete multi part upload, expected X bytes, wrote 2X

After that, Nextcloud aborts the multipart upload and the WebDAV request fails.

I reproduced this on Nextcloud 33.0.2 with S3 primary storage. A one-line local patch that resets $totalWritten at the start of each retry attempt fixed the problem.

Steps to reproduce

1. Configure Nextcloud with an S3-compatible backend as the primary object store
2. Upload a file large enough to trigger multipart upload over WebDAV
3. Let the first multipart attempt fail late enough that Nextcloud retries once
4. Observe that the retry runs
5. Observe that Nextcloud fails with Incomplete multi part upload, expected X bytes, wrote 2X

Expected behavior

If multipart upload is retried, the size check should validate only the current attempt.

Nextcloud should not accumulate byte counts across attempts.

Nextcloud Server version

32

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.4

Web server

Apache (supported)

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 31 to 32)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "nextcloud",
                "region": "RegionOne",
                "hostname": "REDACTED,
                "port": "6780",
                "storageClass": "STANDARD",
                "objectPrefix": "urn:oid:",
                "autocreate": false,
                "use_ssl": true,
                "use_path_style": true,
                "legacy_auth": false,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "concurrency": 2,
                "uploadPartSize": 524288000
            }
        },
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "100.96.*.*",
            "REDACTED"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "33.0.2.2",
        "overwrite.cli.url": "https:\/\/drive.igene.tw",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "5432",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "tempdirectory": "\/var\/php_tmp",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpsecure": "ssl",
        "mail_smtpauth": true,
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "loglevel": 0,
        "maintenance": false,
        "overwriteprotocol": "https",
        "memories.exiftool": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/exiftool-amd64-glibc",
        "memories.vod.path": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/go-vod-amd64",
        "enabledPreviewProviders": [
            "OC\\Preview\\Image",
            "OC\\Preview\\HEIC",
            "OC\\Preview\\TIFF"
        ],
        "app_install_overwrite": [
            "camerarawpreviews",
            "memories"
        ],
        "preview_max_memory": 1024,
        "default_phone_region": "JP",
        "memories.index.mode": "3",
        "memories.index.path": "\/Photos",
        "theme": "",
        "mail_sendmailmode": "smtp",
        "memories.db.triggers.fcu": true,
        "maintenance_window_start": 22,
        "overwritehost": "REDACTED",
        "upgrade.disable-web": true,
        "files.chunked_upload.max_size": 0,
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [],
        "openmetrics_allowed_clients": [
            "127.0.0.1",
            "10.42.0.0\/16",
            "10.43.0.0\/16"
        ]
    }
}

List of activated Apps

Enabled:
  - activity: 6.0.0
  - bruteforcesettings: 6.0.0
  - camerarawpreviews: 0.8.8
  - cloud_federation_api: 1.17.0
  - comments: 1.23.0
  - contactsinteraction: 1.14.1
  - dashboard: 7.13.0
  - dav: 1.36.0
  - federatedfilesharing: 1.23.0
  - federation: 1.23.0
  - files: 2.5.0
  - files_downloadlimit: 5.1.0
  - files_pdfviewer: 6.0.0
  - files_reminders: 1.6.0
  - files_sharing: 1.25.2
  - files_trashbin: 1.23.0
  - files_versions: 1.26.0
  - firstrunwizard: 6.0.0
  - logreader: 6.0.0
  - lookup_server_connector: 1.21.0
  - memories: 7.8.2
  - nextcloud_announcements: 5.0.0
  - notifications: 6.0.0
  - oauth2: 1.21.0
  - password_policy: 5.0.0
  - photos: 6.0.0
  - previewgenerator: 5.13.0
  - privacy: 5.0.0
  - profile: 1.2.0
  - provisioning_api: 1.23.0
  - recommendations: 6.0.0
  - related_resources: 4.0.0
  - serverinfo: 5.0.0
  - settings: 1.16.0
  - sharebymail: 1.23.0
  - support: 5.0.0
  - survey_client: 5.0.0
  - systemtags: 1.23.0
  - text: 7.0.0
  - theming: 2.8.0
  - twofactor_backupcodes: 1.22.0
  - twofactor_webauthn: 2.6.0
  - updatenotification: 1.23.0
  - user_status: 1.13.0
  - viewer: 6.0.0
  - weather_status: 1.13.0
  - webhook_listeners: 1.5.0
  - workflowengine: 2.15.0
Disabled:
  - admin_audit: 1.23.0
  - app_api: 33.0.0 (installed 32.0.0)
  - circles: 33.0.0 (installed 27.0.1)
  - encryption: 2.21.0
  - epubviewer: 1.8.1 (installed 1.8.1)
  - files_external: 1.25.1
  - files_rightclick: 0.15.1 (installed 1.6.0)
  - suspicious_login: 11.0.0
  - testing: 1.23.0
  - twofactor_nextcloud_notification: 7.0.0
  - twofactor_totp: 15.0.0 (installed 10.0.0-beta.2)
  - user_ldap: 1.24.0

Nextcloud Signing status

Technical information
=====================
The following list covers which files have failed the integrity check. Please read
the previous linked documentation to learn more about the errors and how to fix
them.

Results
=======
- core
	- INVALID_HASH
		- lib/private/Files/ObjectStore/S3ObjectTrait.php
	- EXTRA_FILE
		- lib/private/Files/ObjectStore/S3ObjectTrait.php.bak

Raw output
==========
Array
(
    [core] => Array
        (
            [INVALID_HASH] => Array
                (
                    [lib/private/Files/ObjectStore/S3ObjectTrait.php] => Array
                        (
                            [expected] => 03d0ad611f08126655854936eb3355531754b3c418e513e2563dbde4568bf6f0c62aee2a0b6593d20df340b373b1f7e40a2700e7284b74512da13fe5d424fe71
                            [current] => ad7988daaaeba83af35b7f4accc9e3589ce0b1e2b245f9978b5741a4d01c3766c388ba9f84b70a916149e4864aff9b6aaee34e6d893725138bbe98ea2dd6c07e
                        )

                )

            [EXTRA_FILE] => Array
                (
                    [lib/private/Files/ObjectStore/S3ObjectTrait.php.bak] => Array
                        (
                            [expected] => 
                            [current] => 03d0ad611f08126655854936eb3355531754b3c418e513e2563dbde4568bf6f0c62aee2a0b6593d20df340b373b1f7e40a2700e7284b74512da13fe5d424fe71
                        )

                )

        )

)

Nextcloud Logs

{
  "app": "objectstore",
  "method": "PUT",
  "message": "Could not create object urn:oid:<redacted> for files/<redacted>",
  "exception": {
    "Exception": "OCA\\DAV\\Connector\\Sabre\\Exception\\BadGateway",
    "Message": "Error while uploading to S3 bucket",
    "Previous": {
      "Exception": "Aws\\Exception\\MultipartUploadException",
      "Message": "An exception occurred while performing a multipart upload",
      "previous": {
        "class": "Exception",
        "message": "Incomplete multi part upload, expected 3646786296 bytes, wrote 7293572592"
      }
    }
  }
}



{
  "app": "webdav",
  "method": "PUT",
  "message": "Expected filesize of 3646786296 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) -1 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.",
  "exception": {
    "Exception": "Sabre\\DAV\\Exception\\BadRequest",
    "File": "/var/www/html/apps/dav/lib/Connector/Sabre/File.php",
    "Line": 261
  }
}

Additional info

The failed upload clearly retried:

• first multipart upload id
• second multipart upload id
• then abort of the retry upload id

The storage backend accepted multipart parts successfully. The final failure was caused by the size check after retry.

Local fix that resolved the issue:

while (!$uploaded && $attempts <= 1) {
    $totalWritten = 0;
    ...
}

I can provide a simple PR for this issue if the bug looks valid.
Thanks!