Can't update file with certain content on OneDrive Personal

[Kagami]

Category

• Question
• Documentation issue
• Bug

Expected or Desired Behavior

I've uploaded file on my OneDrive Personal and updating it with createUploadSession.

Observed Behavior

With some specific kind of content, I get the {"error":{"code":"invalidRequest","message":"Bad Argument"}} error on update.

> POST /v1.0/drives/__REDACTED__/items/__REDACTED__!19599:/1:/createUploadSession HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.64.0
Content-Length: 116
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

{"item":{"fileSystemInfo":{"createdDateTime":"2023-09-28T21:01:00Z","lastModifiedDateTime":"2023-09-28T21:01:00Z"}}}

< HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-store
Client-Request-Id: ff7f9fc4-bd3c-4bcc-8f3f-5475fa666e5a
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Date: Mon, 02 Oct 2023 07:20:40 GMT
Location: https://graph.microsoft.com
Odata-Version: 4.0
Request-Id: ff7f9fc4-bd3c-4bcc-8f3f-5475fa666e5a
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
X-Ms-Ags-Diagnostic: {"ServerInfo":{"DataCenter":"West Europe","Slice":"E","Ring":"5","ScaleUnit":"009","RoleInstance":"AM1PEPF00029CEA"}}

505
{"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#microsoft.graph.uploadSession","expirationDateTime":"2023-10-07T07:20:41.053Z","nextExpectedRanges":["0-"],"uploadUrl":"https://api.onedrive.com/rup/__REDACTED__/__REDACTED__"}
0

> PUT /rup/__REDACTED__/__REDACTED__ HTTP/1.1
Host: api.onedrive.com
User-Agent: rclone/v1.64.0
Content-Length: 5600
Authorization: XXXX
Content-Range: bytes 0-5599/5600
Accept-Encoding: gzip

<file_v2.bin>

< HTTP/2.0 400 Bad Request
Content-Type: application/json; charset=utf-8
Date: Mon, 02 Oct 2023 07:20:41 GMT
Ms-Cv: htFkvIC5t0yDhVmEehpkTw.0
P3p: CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
Strict-Transport-Security: max-age=31536000; includeSubDomains
Www-Authenticate: Bearer realm="OneDriveAPI", error="invalid_token", error_description="Invalid auth token"
X-Asmversion: UNKNOWN; 19.1248.908.2006
X-Cache: CONFIG_NOCACHE
X-Msedge-Ref: Ref A: 616CA86322874402B7266AB759416F3E Ref B: STOEDGE1608 Ref C: 2023-10-02T07:20:41Z
X-Msnserver: DB5PPF0EDE6A2ED
X-Qosstats: {"ApiId":0,"ResultType":2,"SourcePropertyId":0,"TargetPropertyId":42}
X-Throwsite: 7cfe.d208

{"error":{"code":"invalidRequest","message":"Bad Argument"}}

Steps to Reproduce

Attaching file_v1.bin which I'm updating with file_v2.bin at the same path. Note that modification time should be different to reproduce. I'm using touch -t 2309290000 file_v1.bin && touch -t 2309290001 file_v2.bin to ensure that.

I can provide more files like that if needed where I encounter this issue. They have similar structure: a lot of duplicating bytes (0xFF) and then updating with few different bytes.

file_v1.bin.gz
file_v2.bin.gz

Additional info

Initially reported at https://forum.rclone.org/t/can-t-update-certain-files-with-certain-modtime-on-onedrive/42077
At least one person can reproduce it on another OneDrive Personal account. But not on OneDrive Business.
cc @ncw

Thanks.

[ncw]

The important part to replicate is that we must be overwriting file_v1.bin with file_v2.bin. You can upload file_v1.bin and file_v2.bin on their own just fine, and also overwrite file_v2.bin with file_v1.bin just not the other way round.

Replication with rclone

First show we can upload file_v1.bin but overwriting it with file_v2.bin fails.

$ rclone delete onedrive:test/1
$ rclone copyto file_v1.bin onedrive:test/1 --retries 1
$ rclone copyto file_v2.bin onedrive:test/1 --retries 1
2023/10/01 12:10:38 ERROR : file_v2.bin: Failed to copy: invalidRequest: Bad Argument
2023/10/01 12:10:38 ERROR : Attempt 1/1 failed with 1 errors and: invalidRequest: Bad Argument
2023/10/01 12:10:38 Failed to copyto: invalidRequest: Bad Argument

Now show we can upload file_v2.bin and overwrite it with file_v1.bin just fine. But attempting to overwrite file_v1.bin with file_v2.bin fails.

$ rclone delete onedrive:test/1
$ rclone copyto file_v2.bin onedrive:test/1 --retries 1
$ rclone copyto file_v1.bin onedrive:test/1 --retries 1
$ rclone copyto file_v2.bin onedrive:test/1 --retries 1
2023/10/01 12:10:59 ERROR : file_v2.bin: Failed to copy: invalidRequest: Bad Argument
2023/10/01 12:10:59 ERROR : Attempt 1/1 failed with 1 errors and: invalidRequest: Bad Argument
2023/10/01 12:10:59 Failed to copyto: invalidRequest: Bad Argument

I thought maybe the files might have the same quickxor hash but they don't. They only differ by one bit, so I guess it makes sense that the quickxor hash only differs by one bit. I wonder if the structure of the quickxor hashes is causing a problem.

$ rclone hashsum quickxor . --include "*.bin"
000000000000000000000020e015000000000000  file_v1.bin
000000000000000000000000e015000000000000  file_v2.bin

Note that file_v1.bin and file_v2.bin are almost entirely 0xFF except for 1 bit difference.

$ diff -u <(hd -v file_v1.bin)  <(hd -v file_v2.bin )
--- /dev/fd/63	2023-10-02 11:44:22.792977880 +0100
+++ /dev/fd/62	2023-10-02 11:44:22.792977880 +0100
@@ -234,7 +234,7 @@
 00000e90  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
 00000ea0  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
 00000eb0  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
-00000ec0  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff f7 ff  |................|
+00000ec0  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
 00000ed0  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
 00000ee0  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
 00000ef0  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|

I tried to replicate this with onedrive business and it did not replicate. It worked just fine there!