BUG: v1.45-031 Dropbox-API-Arg": unknown field 'cursor'

[ajkis]

Version: rclone v1.45-031-ge7684b7e-beta

2018/12/10 11:44:12 ERROR : series/God Friended Me/Season 01/God Friended Me - S01E10 - WEBDL-1080p.mkv: Failed to copy: upload failed: Error in call to API function "files/upload_session/append": HTTP header "Dropbox-API-Arg": unknown field 'cursor'
2018/12/10 11:44:12 ERROR : series/God Friended Me/Season 01/God Friended Me - S01E10 - WEBDL-1080p.mkv: Not deleting source as copy failed: upload failed: Error in call to API function "files/upload_session/append": HTTP header "Dropbox-API-Arg": unknown field 'cursor'
2018/12/10 11:44:21 ERROR : series/Keeping Up with the Kardashians/Season 15/Keeping Up with the Kardashians - S15E16 - WEBDL-1080p.mkv: Failed to copy: upload failed: Error in call to API function "files/upload_session/append": HTTP header "Dropbox-API-Arg": unknown field 'cursor'
2018/12/10 11:44:21 ERROR : series/Keeping Up with the Kardashians/Season 15/Keeping Up with the Kardashians - S15E16 - WEBDL-1080p.mkv: Not deleting source as copy failed: upload failed: Error in call to API function "files/upload_session/append": HTTP header "Dropbox-API-Arg": unknown field 'cursor'
2018/12/10 11:44:23 ERROR : series/Madam Secretary/Season 05/Madam Secretary - S05E09 - WEBDL-1080p.mkv: Failed to copy: upload failed: Error in call to API function "files/upload_session/append": HTTP header "Dropbox-API-Arg": unknown field 'cursor'
2018/12/10 11:44:23 ERROR : series/Madam Secretary/Season 05/Madam Secretary - S05E09 - WEBDL-1080p.mkv: Not deleting source as copy failed: upload failed: Error in call to API function "files/upload_session/append": HTTP header "Dropbox-API-Arg": unknown field 'cursor'
2018/12/10 11:44:30 ERROR : series/The Flash (2014)/Season 05/The Flash (2014) - S05E09 - WEBDL-1080p.mkv: Failed to copy: upload failed: Error in call to API function "files/upload_session/append": HTTP header "Dropbox-API-Arg": unknown field 'cursor'
2018/12/10 11:44:30 ERROR : series/The Flash (2014)/Season 05/The Flash (2014) - S05E09 - WEBDL-1080p.mkv: Not deleting source as copy failed: upload failed: Error in call to API function "files/upload_session/append": HTTP header "Dropbox-API-Arg": unknown field 'cursor'

[ncw]

Err, yes it looks like 4138d5a or 8fb707e has broken the dropbox backend.

The integration tests are unhappy too: https://pub.rclone.org/integration-tests/current/

@mathieucarbou have you got time to look at this?

[mathieucarbou]

@ncw : This is not 8fb707e.

But this might be 4138d5a (the dropbox sdk upgrade).

@diwakergupta changed a lot of things and upgraded to a major version both SDK and api specs.
Look at the commit history here: https://github.com/dropbox/dropbox-sdk-go-unofficial/commits/master).

My original patch was just about updating some specific files inside the the vendor version you were pointing to but you preferred to link to the next Dropbox version and tag (which is, imo, better also). But it contains risks obviously of bringing new changes. I thought you would be OK with and you had plan on looking / reviewing / testing the dropbox integration carefully.

Sadly I do not have time to look right now at all the possible changes they did and that could break things.

@ajkis : to help us, are you able to re-run your command with --log-level DEBUG and give us more details about the error plus all the the rclone command and command-line args you are using ? (without any username / password)

Thanks!

UPDATE 1:

@ncw : BTW, looking at my PR that you have merged: #2622, the checks were successful...

Are you sure it is these commits then ?
Are the PR builds also running the integration tests (which they should at least) ?

UPDATE 2:

Opened 2 PRs:

• Revert #2622 #2830: Revert the changes in master
• Fixes #1788: Retry-After support for "sync" for Dropbox backend with old SDK #2831: My old fix (which only supports Retry-After) for some specific DP commands.

I strongly advice though to fix the integration tests... Because right now, knowing that Dropbox Team has upgraded its SDK to 5 with some changes, anyway rclone has no choice but to adapt to it.

[ncw]

Are the PR builds also running the integration tests (which they should at least) ?

No they don't - they are too expensive (time and money wise) to do that.

I strongly advice though to fix the integration tests... Because right now, knowing that Dropbox Team has upgraded its SDK to 5 with some changes, anyway rclone has no choice but to adapt to it.

Yes that is what I usually do. Have you got time to help with this?

[mathieucarbou]

Yes that is what I usually do. Have you got time to help with this?

Yes and no... I only have little time sporadically until January (moving in progress). I had a quick look yesterday and I do not understand the complain about the cursor being in the http header during a chunk upload. The cursor field is part of the dropbox api. I do not have enough Go knowledge also to be able to find a potential change to apply in the rclone code, or a potential bug in dropbox api...

So in conclusion, my help will be limited and not as valuable as you might expect.

[ncw]

Things are a bit hectic right now for me too, but I'm hoping to have a bit of time over the holiday season.

Thanks for your help @mathieucarbou :-)

[kernalkue]

I'm having big problems with this bug. Is there a work around?

[mathieucarbou]

@kernalkue: yes, do not use the latest rclone version. Also, I have sent days ago a PR (#2830) to revert the Dropbox upgrade. It is up to @ncw to merge or not.

If you want a working retry after support for now, then you have to build your own rclone version that includes PR #2830 and #2831 which is supporting retry-after header for some Dropbox API calls but not all.

@ncw: after our moving is finished and we are settled, I should have some time to look at this. But this will be mid/end January....

Also, I don't really understand why the code compiles but fail at runtime with this header, which is in the Dropbox API by the way...

[ncw]

Sorry everyone, I have a bit of a backlog!

I've figured out the problem with the dropbox SDK: dropbox/dropbox-sdk-go-unofficial#49 and I've sent a PR to fix dropbox/dropbox-sdk-go-unofficial#50

Here is a version of rclone with the SDK patch I submitted in. It now passes all the integration tests so I'll merge it after it has run through the CI.

@ajkis and @kernalkue can you give it a test to see if it fixes your problem?

https://beta.rclone.org/branch/v1.45-038-g3c624c6f-fix-2829-dropbox-sdk-beta/ (uploaded in 15-30 mins)

[ncw]

The PR has been applied to the dropbox repo and I've pulled it in and checked it isOK.

So the fix will be in in the latest beta in 15-30 mins.