Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

List large folder in Onedrive for business (od4b) #2580

Open
Bruceforce opened this Issue Jul 1, 2017 · 13 comments

Comments

Projects
None yet
5 participants
@Bruceforce
Copy link
Contributor

Bruceforce commented Jul 1, 2017

I have:

  • searched open and closed issues for duplicates

Version info

Duplicati Version: 2.0.1.69
Operating System: Debian 8.8 (Kernel 3.16.0-4-amd64)
Mono Version: Mono JIT compiler version 5.0.1.1 (2017-02/5077205 Thu May 25 09:19:47 UTC 2017)

Bug description

When I try to list the files of a failed Onedrive for business Backup with ~112 GB (50 MB per file) the command returns an empty list. If I run the same command on a different folder with just a single file in it's working and listing that single file.

It could be related to #2474 because I got a lot of these messages during backup and hat to restart over and over again until it finally can't list the files anymore.

I also tried to create a whole new Backup with the same result. After a few crashes and restarts with the error mentioned in #2474 it also couldn't list the files anymore.

If I look in the Onedrive for business webinterface the files are all there.

I was thinking about trying it again with a larger filesize. If that's the root of the error it may be added to #1788

Steps to reproduce

  • run mono Duplicati.CommandLine.BackendTool.exe LIST "od4b://... on a od4b backup with ~112 GB (50 MB per file)

Actual result:

Name Dir/File LastChange Size

(no files listed)

Expected result:

Name Dir/File LastChange Size
... ... ... ...
... ... ... ...

(files in that folder get listed)

Screenshots

debug log

If I try to run mono Duplicati.CommandLine.exe repair "od4b://... I get:

Listing remote folder ...
Duplicati.Library.Interface.UserInformationException: No files were found at the remote location, perhaps the target url is incorrect?
at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun (Duplicati.Library.Main.Database.LocalDatabase dbparent, System.Boolean updating, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x000dc] in <265b26535dcf483587ba8737e917cffa>:0
at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.Run (System.String path, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x00032] in <265b26535dcf483587ba8737e917cffa>:0
at Duplicati.Library.Main.Operation.RepairHandler.RunRepairLocal (Duplicati.Library.Utility.IFilter filter) [0x000b0] in <265b26535dcf483587ba8737e917cffa>:0
at Duplicati.Library.Main.Operation.RepairHandler.Run (Duplicati.Library.Utility.IFilter filter) [0x00138] in <265b26535dcf483587ba8737e917cffa>:0
at Duplicati.Library.Main.Controller+<>c__DisplayClass20_0.b__0 (Duplicati.Library.Main.RepairResults result) [0x0001c] in <265b26535dcf483587ba8737e917cffa>:0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action1[T] method) [0x0014b] in <265b26535dcf483587ba8737e917cffa>:0 at Duplicati.Library.Main.Controller.RunAction[T] (T result, Duplicati.Library.Utility.IFilter& filter, System.Action1[T] method) [0x00007] in <265b26535dcf483587ba8737e917cffa>:0
at Duplicati.Library.Main.Controller.Repair (Duplicati.Library.Utility.IFilter filter) [0x0001a] in <265b26535dcf483587ba8737e917cffa>:0
at Duplicati.CommandLine.Commands.Repair (System.IO.TextWriter outwriter, System.Action1[T] setup, System.Collections.Generic.List1[T] args, System.Collections.Generic.Dictionary2[TKey,TValue] options, Duplicati.Library.Utility.IFilter filter) [0x0002e] in <f566379fc0a94d07a41b0130b5bb2c13>:0 at (wrapper delegate-invoke) System.Func6[System.IO.TextWriter,System.Action1[Duplicati.Library.Main.Controller],System.Collections.Generic.List1[System.String],System.Collections.Generic.Dictionary2[System.String,System.String],Duplicati.Library.Utility.IFilter,System.Int32]:invoke_TResult_T1_T2_T3_T4_T5 (System.IO.TextWriter,System.Action1<Duplicati.Library.Main.Controller>,System.Collections.Generic.List1<string>,System.Collections.Generic.Dictionary2<string, string>,Duplicati.Library.Utility.IFilter)
at Duplicati.CommandLine.Program.RunCommandLine (System.IO.TextWriter outwriter, System.IO.TextWriter errwriter, System.Action`1[T] setup, System.String[] args) [0x003fb] in :0

@kenkendk

This comment has been minimized.

Copy link
Member

kenkendk commented Jul 11, 2017

Unfortunately, I do not have a OD4B account for testing, so I cannot debug this further.

@Bruceforce

This comment has been minimized.

Copy link
Contributor Author

Bruceforce commented Jul 11, 2017

Is there no way to get a debug log which I can send to you?

@sffetlio

This comment has been minimized.

Copy link
Contributor

sffetlio commented Sep 1, 2017

Version info

Duplicati Version: 2.0.2.1_beta_2017-08-01
Operating System: Windows 10 64bit 1703 build 15063.540
Backend: Local folder and then OneDrive for Business

Bug description

I believe I have the same issue so I'll put it here. I have a backup on OneDrive for Business. It contains 6758 files (volume size 100 MB) that are 329 GB total. See "Steps to reproduce" for my workflow. I am sure it is the correct path because I tried created a fresh backup with the same folder url and Duplicati had no issues and uploaded files to the correct folder.

Steps to reproduce

  • Create a backup to the local OneDrive for Business folder using 100 MB volumes.
  • Wait for the OneDrive for Business client to upload everything (I do it this way because I get way better upload speed: it uploads several files at a time saturating my 100 Mbps up link).
  • Create a new backup with destination the OneDrive for Business server using the files in the uploaded folder.
  • Recreate the database

Actual result: Recreating the database fails with the following error:

Error while running My Backup
No files were found at the remote location, perhaps the target url is incorrect?

and the log shows:

Sep 1, 2017 9:21 AM: list
[

]

Expected result: It should recreate the database and be able to continue backing up from now on.

debug log

Sep 1, 2017 9:21 AM: Failed while executing "Repair" with id: 12
Duplicati.Library.Interface.UserInformationException: No files were found at the remote location, perhaps the target url is incorrect?
   at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun(LocalDatabase dbparent, Boolean updating, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
   at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.Run(String path, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
   at Duplicati.Library.Main.Operation.RepairHandler.RunRepairLocal(IFilter filter)
   at Duplicati.Library.Main.Operation.RepairHandler.Run(IFilter filter)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   at Duplicati.Library.Main.Controller.Repair(IFilter filter)
   at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)
@Bruceforce

This comment has been minimized.

Copy link
Contributor Author

Bruceforce commented Sep 18, 2017

In the initial comment I wrote

I was thinking about trying it again with a larger filesize.

I tested the same backup job with 500 MB per file and it worked. So I think od4b can't list large folders. Of course this isn't a good workaround - with more revisions there will be more files inside the backup and the error will occur again.

@jma89

This comment has been minimized.

Copy link

jma89 commented Feb 8, 2018

@Bruceforce are you still seeing this with OD4B backup destinations?

I think I'm running into the same problem right now with one of my backup sets. (I have two working ones with 2959 files and 2810 files in them, and a broken one with 7039 files. All three use the default volume size of 50 MB.)

For what it's worth: I was trying to get counts through the Storage Metrics section of the site settings panel, but it would keep erroring out for the folder with 7000+ files. This may indicate that Sharepoint itself (what OD4B runs atop of) can't handle such large folders. (I got the final count by going page-of-30 at a time through the folder in the new interface.)

@kenkendk I'll try to get some captures of the interaction with Fiddler. Do you have the API documentation that was used to build the original implementation?

As a workaround we could map OD4B using WebDAV instead of the native handler. I'll do some tests and see if that works better for my largest backup set.

@jma89

This comment has been minimized.

Copy link

jma89 commented Feb 8, 2018

WebDAV is disabled on my company's account (and for good reason, really. Just a bummer for testing.) so I can't verify if that would have worked or not.

Digging around further: Turns out the old client had a limit of 5,000 files per folder (if I'm interpreting the limits correctly), whereas the new sync client bumps that to 100,000: https://support.office.com > SharePoint Online limits

I'm assuming there's a different API beneath the surface that enables this behavior.

@Bruceforce

This comment has been minimized.

Copy link
Contributor Author

Bruceforce commented Feb 8, 2018

@jma89 Yeah I just tested it with 2.0.2.17 and it's still the same error. Thought about using some third party sync client to try to sync my local backup with od4b. Unfortunately Microsoft doesn't care much about Linux users and they didn't even publish an official client.

@veljko

This comment has been minimized.

Copy link

veljko commented Apr 18, 2018

Same issue here with 6046 files (150MB dblock size). It is not only Linux they do not care about as this is the same problem on Windows :( I've spent two days trying to move some of the files (as I have 4 index files in the backup). In one moment repair started to work (Duplicati.CommandLine.exe repair od4b://...) but then I thought things will work for all the files and moved other files to the same folder. Now, I am back to square one where Duplicati.commandline.BackEndtool.exe" LIST od4b://... returns No filesets found on remote target :(

@Bruceforce

This comment has been minimized.

Copy link
Contributor Author

Bruceforce commented Apr 18, 2018

@veljko have you tried with the new onedrive v2 backend? I don't know if that can help in your situation but it's worth a try.

@veljko

This comment has been minimized.

Copy link

veljko commented Apr 18, 2018

@Bruceforce I am not sure what exactly you mean. How should I try it? Thanks

@Bruceforce

This comment has been minimized.

Copy link
Contributor Author

Bruceforce commented Apr 18, 2018

@veljko
Update to the newest duplicati version. Try to make a new backup job and select OneDrive v2 as Backend. Get an AuthID and set the destination folder as your corrupt path. Try the list command again with the new od4bv2 backend.

@veljko

This comment has been minimized.

Copy link

veljko commented Apr 19, 2018

Wow! Thanks a LOT for the hint. I couldn't wait for recreate to finish to thank you (and of course Duplicati team). Looking at the traffic and progress bar (~50% at the moment), things are working fine.
I will try list and then also try it from Debain (current restore running from Windows) and post results.

@veljko

This comment has been minimized.

Copy link

veljko commented Apr 19, 2018

All good on Debian too! Backup that I couldn't resume is happily working again with more consistent transfer rate. Nice touch with adding per file progress to UI! Thanks again for your help @Bruceforce

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.