Implement dropbox v2 api - v1 turns off on 28 September 2017 #349

ncw · 2016-02-10T17:42:12Z

The v2 api does allow setting modification times - the client_modified attribute: See upload docs

It doesn't allow setting it on an existing object so it would require #348 fully useful.

On 28th June 2017 API v1 will be turned off

Update deadline extended to 28 September 2017

NB: dropbox now have an unofficial sdk for go

See also: #969

The text was updated successfully, but these errors were encountered:

andreasotto · 2017-01-21T19:16:34Z

+1

paulraines68 · 2017-02-28T15:12:06Z

But even with v1 there is server_modfied right? So if you are syncing from local to Dropbox and the local mod time is newer than the server_modified time you can assume you need to upload the file even if the sizes are the same? Maybe there could be an option to rclone to enable this behavior? It could also take a datetime value like "--assume-newer-date " where rclone will only force a sync if the service_modified time is <= to to make it a bit safer. This could be used by backup scripts which would record the datetime they run and then pass the datetime of the last run to --assume-newer-date each time they run.

ncw · 2017-02-28T16:54:13Z

@paulraines68 --update is the option you want. http://rclone.org/docs/#u-update

paulraines68 · 2017-02-28T19:22:34Z

So --update does work with dropbox and uses the service_modified time?

ncw · 2017-02-28T19:52:32Z

@paulraines68

So --update does work with dropbox and uses the service_modified time?

Should do!

kami4ka · 2017-04-19T08:12:00Z

Would be nice to have it implemented.

vprokhurenko · 2017-04-19T08:13:49Z

@ncw can you please clarify the date when you are planning to integrate Dropbox v2 API into rclone? Would be helpful to know

fniephaus · 2017-04-20T19:39:08Z

Also, it'd be cool if rclone could support a resume mechanism based on Dropbox API v2 cursors at some point. This would dramatically speed up incremental syncs/copies from Dropbox.

ghost · 2017-05-07T18:59:07Z

+1

This is feature complete with the old version but now supports modification times.

ncw · 2017-05-30T15:53:58Z

Dropbox v2 API has just hit master!

Please try this beta

https://beta.rclone.org/v1.36-144-g178ff62d/ (uploaded in 15-30 mins)

ncw · 2017-05-30T21:34:33Z

Bit of trouble building: beta now here: https://beta.rclone.org/v1.36-146-g71028e0f/

alpha-rudy · 2017-06-11T04:02:52Z

Hi,

I tested "rclone-v1.36-146-g71028e0fβ-linux-amd64.zip", and found that rclone copy take a long time (used to be very quick).

$ du -sh .
2.9G	.

$ rclone --version
rclone v1.36-146-g71028e0fβ

<< change a small text file >>

$ time rclone copy . rudybox:Public/drops

real	3m39.140s
user	0m35.944s
sys	0m5.420s

<< do it again, not changing anything >>

$ time rclone copy . rudybox:Public/drops

real    3m6.348s
user    0m37.836s
sys    0m3.120s

Please check. Thank you! :)

ncw · 2017-06-26T14:53:42Z

@alpha-rudy which rclone are you comparing it with?

mchudoba · 2017-06-26T21:43:28Z

FYI, Dropbox extended the cutoff date of v1 to September 28, 2017.

https://blogs.dropbox.com/developers/2017/06/updated-api-v1-deprecation-timeline/

alpha-rudy · 2017-06-26T23:18:38Z

Hi @ncw, I compared it with rclone-v1.36-linux-amd64.

Thanks. :)

ncw · 2017-06-27T09:13:42Z

Current beta

$ rclone -v sync /tmp/1000files dropbox:1000files
2017/06/27 10:04:57 INFO  : Dropbox root '1000files': Modify window is 1s
2017/06/27 10:05:05 INFO  : Dropbox root '1000files': Waiting for checks to finish
2017/06/27 10:05:05 INFO  : Dropbox root '1000files': Waiting for transfers to finish
2017/06/27 10:05:05 INFO  : Waiting for deletions to finish
2017/06/27 10:05:05 INFO  : 
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:              1000
Transferred:            0
Elapsed time:        8.1s

v1.36

$ rclone-v1.36 -v sync /tmp/1000files TestDropboxOld:1000files
2017/06/27 10:05:52 INFO  : Dropbox root '1000files': Modify window not supported
2017/06/27 10:05:57 INFO  : Dropbox root '1000files': Waiting for checks to finish
2017/06/27 10:05:57 INFO  : Dropbox root '1000files': Waiting for transfers to finish
2017/06/27 10:05:57 INFO  : Waiting for deletions to finish
2017/06/27 10:05:57 INFO  : 
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:              1000
Transferred:            0
Elapsed time:        5.1s

Now log the transactions

$ rclone -vv sync /tmp/1000files dropbox:1000files --dump-bodies --log-file dropbox.new
$ rclone-v1.36 -vv sync /tmp/1000files TestDropboxOld:1000files --dump-bodies --log-file dropbox.old
$ ls -l dropbox.new dropbox.old
-rw-r----- 1 ncw ncw 940172 Jun 27 10:06 dropbox.new
-rw-r----- 1 ncw ncw 943699 Jun 27 10:07 dropbox.old
$ wc -l dropbox.new 
5516 dropbox.new
$ wc -l dropbox.old
5206 dropbox.old
$ grep -c 'HTTP REQUEST' dropbox.new dropbox.old
dropbox.new:103
dropbox.old:103
$

So there does appear to be a bit of a slow down with the v2 API from 5s to 8s in the test above. However it uses the same number of HTTP transactions of about the same size which means it is very likely the v2 API just is slower.

Does that agree with your tests @alpha-rudy

ncw · 2017-06-27T09:17:40Z

@mchudoba thanks for the heads-up!

alpha-rudy · 2017-07-01T07:30:55Z

Hi @ncw,

Thanks for your reply.

I'm sorry. I tried to follow your test but I cannot use rclone-v1.36 now. I think I will have same result as yours.

$ rclone-v1.36 -v sync . rudybox:Public/tests
2017/07/01 15:26:47 INFO  : Dropbox root 'Public/tests': Modify window not supported
2017/07/01 15:26:47 ERROR : Attempt 1/3 failed with 1 errors and: Invalid OAuth2 token.
2017/07/01 15:26:47 ERROR : Attempt 2/3 failed with 1 errors and: Invalid OAuth2 token.
2017/07/01 15:26:48 ERROR : Attempt 3/3 failed with 1 errors and: Invalid OAuth2 token.
2017/07/01 15:26:48 Failed to sync: Invalid OAuth2 token.

qopencc · 2017-07-05T16:02:28Z

There are 4000 files in the dropbox_crypt's test folder.

rclone copy d:\test "dropbox_crypt:test" -vv


2017/07/05 23:34:37 DEBUG : rclone: Version "v1.36-231-g575e779b" starting with parameters ["rclone" "copy" "d:\\test" "dropbox_crypt:test" "-vv"]
2017/07/05 23:34:38 INFO  : Encrypted drive 'dropbox_crypt:test': Modify window is 1s
2017/07/05 23:35:38 INFO  :
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      1m5.9s

2017/07/05 23:36:38 INFO  :
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      2m5.9s

2017/07/05 23:37:38 INFO  :
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      3m5.9s

2017/07/05 23:38:38 INFO  :
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      4m5.9s

2017/07/05 23:39:38 INFO  :
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      5m5.9s

2017/07/05 23:40:38 INFO  :
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      6m5.9s

2017/07/05 23:41:38 INFO  :
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      7m5.9s

2017/07/05 23:42:38 INFO  :
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      8m5.9s

2017/07/05 23:43:38 INFO  :
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:      9m5.8s

2017/07/05 23:44:38 INFO  :
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            0
Elapsed time:     10m5.8s

ncw · 2017-07-06T14:32:28Z

@alpha-rudy wrote

I'm sorry. I tried to follow your test but I cannot use rclone-v1.36 now. I think I will have same result as yours.

You'll need to make a new remote say dbold with the old version of rclone

@qopencc can you compare that with v1.36 too?

qopencc · 2017-07-06T17:55:13Z

@ncw
rclone-v1.36-137-ga2e3af05-windows-amd64 : There is not the situation.

rclone-v1.36-158-ga5cfdfd2-windows-amd64 ~~ rclone-v1.36-231-g575e779b-windows-amd64: There is the situation.

ncw · 2017-07-06T18:49:26Z

@qopencc what exactly is the problem?

qopencc · 2017-07-06T19:49:46Z

@ncw

rclone-v1.36-158-ga5cfdfd2-windows-amd64 ~~ rclone-v1.36-231-g575e779b-windows-amd64:

There are 4000 files in the dropbox_crypt's test folder.
rclone's copy and rclone's ls do not work.

There are 1000 files in the dropbox_crypt's test folder.
rclone's copy and rclone's ls work.

rclone-v1.36-137-ga2e3af05-windows-amd64 :

There are 4000 files in the dropbox_crypt's test folder.
rclone's copy and rclone's ls work.

There are 1000 files in the dropbox_crypt's test folder.
rclone's copy and rclone's ls work.

ncw · 2017-07-07T10:53:56Z

@qopencc I've done several tests and been unable to replicate this. One with 1000 files in a crypted directory and one with 2000 files in the same directory.

Can you attach a log file with -vv --dump-bodies please, eg

rclone ls d:\test "dropbox_crypt:test" -vv --dump-bodies

Does ls work in the underlying dropbox remote?

Which bits of rclone do work? How about rclone lsd dropbox_crypt:?

Can you check your network connectivity to api.dropboxapi.com?

I note this has both IPv4 and IPv6 addresses - which are you using?

qopencc · 2017-07-07T14:47:55Z

@ncw

rclone-v1.36-137-ga2e3af05-windows-amd64 :

There 3981 files in the same directory.

rclone's ls and lsd ,copy work in the directory.
rclone's ls and lsd ,copy work in other directory.

rclone-v1.36-158-ga5cfdfd2-windows-amd64 ~~ rclone-v1.36-231-g575e779b-windows-amd64:

There 3981 files in the same directory.

rclone's ls and lsd ,copy do not work in the directory.
rclone's ls and lsd ,copy work in other directory.

Can you attach a log file with -vv --dump-bodies please

https://drive.google.com/drive/folders/0Bz6Qqy38NFGNNEZyVjhhczVFVEk?usp=sharing

Does ls work in the underlying dropbox remote?

ls does not work in the underlying dropbox remote

2017/07/07 22:06:32 DEBUG : rclone: Version "v1.36-231-g575e779b" starting with parameters ["rclone" "ls" "office5678_db:(rclone_office5678_crypt_db)/esbldpslqhglbrmoumpr3ns12g/mbjh9d5icknf7q26e77sl47ptg/hb6g8mvkthf9gfi4adld8l6h6k/mbjh9d5icknf7q26e77sl47ptg/6n4lmm26dr5hqm6kque8vgaipc/32gomj1jiif66amgaumpjn1ba5g8en164irhhusu75qe6iepemo0" "-vv"]
2017/07/07 22:06:33 INFO : Dropbox root '(rclone_office5678_crypt_db)/esbldpslqhglbrmoumpr3ns12g/mbjh9d5icknf7q26e77sl47ptg/hb6g8mvkthf9gfi4adld8l6h6k/mbjh9d5icknf7q26e77sl47ptg/6n4lmm26dr5hqm6kque8vgaipc/32gomj1jiif66amgaumpjn1ba5g8en164irhhusu75qe6iepemo0': Modify window is 1s

Which bits of rclone do work? How about rclone lsd dropbox_crypt:?

lsd does not work

2017/07/07 22:17:53 DEBUG : rclone: Version "v1.36-231-g575e779b" starting with parameters ["rclone" "lsd" "office5678_crypt_db:(Program Files)\(portable)\(porable)\(portable)\(desktop)\(notepad++_backup)" "-vv"]
2017/07/07 22:17:55 INFO : Encrypted drive 'office5678_crypt_db:(Program Files)/(portable)/(porable)/(portable)/(desktop)/(notepad++_backup)': Modify window is 1s

Can you check your network connectivity to api.dropboxapi.com?

I chedked my network connectivity to api.dropboxapi.com.

I note this has both IPv4 and IPv6 addresses - which are you using?

IPv4

ncw · 2017-07-07T16:57:16Z

@qopencc I can see the problem with that log thanks - there is a problem with the directory paging... Will post a beta when I've worked out how to fix it.

ncw · 2017-07-07T20:22:26Z

@qopencc - I found the bug please try this beta!

https://beta.rclone.org/v1.36-242-gc3dac2e3/ (uploaded in 15-30 mins)

qopencc · 2017-07-08T01:49:18Z

@ncw

rclone-v1.36-242-gc3dac2e3β-linux-amd64.zip :

copy and ls ,lsd work.

There is not windows version in the v1.36-242-gc3dac2e3.

ncw · 2017-07-08T15:29:03Z

@qopencc thanks for testing.

I've fixed the windows build here

https://beta.rclone.org/v1.36-245-gda2e2544/ (uploaded in 15-30 mins)

qopencc · 2017-07-08T18:21:58Z

@ncw

rclone-v1.36-245-gda2e2544-windows-amd64 :

copy and ls ,lsd work.

Thank you.

ncw · 2017-07-10T10:13:52Z

I'm going to mark this as closed now - if anyone finds any more problems please open a new issue.

ncw added maintenance Remote: Dropbox labels Feb 10, 2016

ncw changed the title ~~Implement dropbox v2 api~~ Implement dropbox v2 api - v1 turns off on 28 June 2017 Jun 29, 2016

ncw added this to the Soon milestone Jun 29, 2016

ncw mentioned this issue Sep 7, 2016

Dropbox requiring lower case for rclone lsd #693

Closed

ncw mentioned this issue Dec 31, 2016

FIX Dropbox: HTTP/1.1 414 Request-URI Too Large (unexpected HTTP status code 414) #969

Closed

ncw modified the milestones: v1.38, Soon Mar 23, 2017

ncw mentioned this issue Mar 31, 2017

Feature Request: implement Content HASH for DROPBOX #1302

Closed

ncw modified the milestones: v1.37, v1.38 May 29, 2017

ncw added a commit that referenced this issue May 30, 2017

dropbox: Update dropbox to use the v2 API #349

8e214e8

This is feature complete with the old version but now supports modification times.

ncw changed the title ~~Implement dropbox v2 api - v1 turns off on 28 June 2017~~ Implement dropbox v2 api - v1 turns off on 28 September 2017 Jun 27, 2017

ncw closed this as completed Jul 10, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement dropbox v2 api - v1 turns off on 28 September 2017 #349

Implement dropbox v2 api - v1 turns off on 28 September 2017 #349

ncw commented Feb 10, 2016 •

edited

Loading

andreasotto commented Jan 21, 2017

paulraines68 commented Feb 28, 2017

ncw commented Feb 28, 2017

paulraines68 commented Feb 28, 2017

ncw commented Feb 28, 2017 •

edited

Loading

kami4ka commented Apr 19, 2017

vprokhurenko commented Apr 19, 2017

fniephaus commented Apr 20, 2017

ghost commented May 7, 2017

ncw commented May 30, 2017

ncw commented May 30, 2017

alpha-rudy commented Jun 11, 2017

ncw commented Jun 26, 2017

mchudoba commented Jun 26, 2017

alpha-rudy commented Jun 26, 2017

ncw commented Jun 27, 2017

ncw commented Jun 27, 2017

alpha-rudy commented Jul 1, 2017

qopencc commented Jul 5, 2017

ncw commented Jul 6, 2017

qopencc commented Jul 6, 2017 •

edited

Loading

ncw commented Jul 6, 2017

qopencc commented Jul 6, 2017

ncw commented Jul 7, 2017

qopencc commented Jul 7, 2017

ncw commented Jul 7, 2017

ncw commented Jul 7, 2017

qopencc commented Jul 8, 2017

ncw commented Jul 8, 2017

qopencc commented Jul 8, 2017

ncw commented Jul 10, 2017

Implement dropbox v2 api - v1 turns off on 28 September 2017 #349

Implement dropbox v2 api - v1 turns off on 28 September 2017 #349

Comments

ncw commented Feb 10, 2016 • edited Loading

andreasotto commented Jan 21, 2017

paulraines68 commented Feb 28, 2017

ncw commented Feb 28, 2017

paulraines68 commented Feb 28, 2017

ncw commented Feb 28, 2017 • edited Loading

kami4ka commented Apr 19, 2017

vprokhurenko commented Apr 19, 2017

fniephaus commented Apr 20, 2017

ghost commented May 7, 2017

ncw commented May 30, 2017

ncw commented May 30, 2017

alpha-rudy commented Jun 11, 2017

ncw commented Jun 26, 2017

mchudoba commented Jun 26, 2017

alpha-rudy commented Jun 26, 2017

ncw commented Jun 27, 2017

ncw commented Jun 27, 2017

alpha-rudy commented Jul 1, 2017

qopencc commented Jul 5, 2017

ncw commented Jul 6, 2017

qopencc commented Jul 6, 2017 • edited Loading

ncw commented Jul 6, 2017

qopencc commented Jul 6, 2017

ncw commented Jul 7, 2017

qopencc commented Jul 7, 2017

ncw commented Jul 7, 2017

ncw commented Jul 7, 2017

qopencc commented Jul 8, 2017

ncw commented Jul 8, 2017

qopencc commented Jul 8, 2017

ncw commented Jul 10, 2017

ncw commented Feb 10, 2016 •

edited

Loading

ncw commented Feb 28, 2017 •

edited

Loading

qopencc commented Jul 6, 2017 •

edited

Loading