New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't connect to SharePoint Online team sites such as https://orgname.sharepoint.com/sites/Site-Name #1770

Open
hi2u opened this Issue Oct 24, 2017 · 68 comments

Comments

Projects
None yet
@hi2u
Copy link

hi2u commented Oct 24, 2017

I’ve been able to successfully connect to the default https://orgname-my.sharepoint.com/ personal SharePoint Site...

$ rclone lsd sp3:
-1 2017-01-04 22:16:34         0 Attachments
-1 2015-01-23 11:13:10         0 Shared with Everyone

But I’m having difficultly figuring out how to connect to team sites on URLs such a: https://orgname.sharepoint.com/sites/Site-Name etc.

The “rclone config” guided process doesn’t let you set the resource_url when setting it up. So I’ve tried editing ~/.config/rclone.conf using a few different methods, changing the resource_url and then reauthorizing, I've tried a number of different addresses like...

For the main/default team site:

https://orgname.sharepoint.com/ 
https://orgname.sharepoint.com/Shared Documents

For separate team sites, or what Microsoft call "site collections":

https://orgname.sharepoint.com/sites/Site-Name
https://orgname.sharepoint.com/sites/Site-Name/
https://orgname.sharepoint.com/sites/Site-Name/Shared Documents
https://orgname.sharepoint.com/sites/Site-Name/Shared Documents/
https://orgname.sharepoint.com/sites/Site-Name/Shared%20Documents
https://orgname.sharepoint.com/sites/Site-Name/Shared%20Documents/

I'm not sure which address format I'm meant to use? (for either the main team site, or all the other ones under /sites/)

I always get the error:

$ rclone -vv lsd sp3:
2017/10/25 03:17:18 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2017/10/25 03:17:18 DEBUG : rclone: Version "v1.38" starting with parameters ["rclone" "-vv" "lsd" "sp3:"]
2017/10/25 03:17:19 Failed to create file system for "sp3:": failed to get root: 401 Unauthorized: 

(there's nothing after that last colon)

Does anyone know how I access team SharePoint sites?

My rclone version is:

rclone v1.38
- os/arch: linux/amd64
- go version: go1.9

...on Manjaro 64bit, installed from the distro's repos.

I'm choosing the "business" option when asked in rclone config.

@ncw

This comment has been minimized.

Copy link
Owner

ncw commented Oct 26, 2017

@olihey any ideas how we could make this work?

@olihey

This comment has been minimized.

Copy link
Contributor

olihey commented Oct 27, 2017

Changing the resource url in the rclone.conf will not work as the access token is only valid for ONE resource url.

But we use the discovery endpoint to get all available services the user has access to and then automatically choose the OneDrive drive. Maybe we could present the list to the user during config and let him choose?!?! Maybe pre-select the OneDrive?

@dienal-san

This comment has been minimized.

Copy link

dienal-san commented Oct 28, 2017

+1

@olihey

This comment has been minimized.

Copy link
Contributor

olihey commented Oct 28, 2017

I have created a pull request here #1780
This will ask the user for the correct resourceURL if the discovery service returns more than one resource URL.
BUT I couldn't test it as I only have a single OneDrive Business account.

@ncw

This comment has been minimized.

Copy link
Owner

ncw commented Nov 5, 2017

Find this fix in the latest beta - please can you have a test and report back.

@dienal-san

This comment has been minimized.

Copy link

dienal-san commented Nov 7, 2017

I tested but it does not work.
It is the same behavior than before. Output only the result_url https://orgname-my.sharepoint.com/.

@olihey

This comment has been minimized.

Copy link
Contributor

olihey commented Nov 7, 2017

Can you please try to setup the endpoint and use the "-v" when starting rclone?
You should see some debug messages and after "Got code" it should list all services Microsoft's discovery services returns for your user.
Mine looks like this:

Got code
2017/11/07 08:12:30 INFO  : Skipping API v1.0 endpoint https://XXXXXXX-my.sharepoint.com/_api/v1.0/me
2017/11/07 08:12:30 INFO  : Skipping API v2.0 endpoint https://XXXXXXX-my.sharepoint.com/_api/v2.0/me
--------------------
@maroc81

This comment has been minimized.

Copy link

maroc81 commented Nov 7, 2017

Here's what I got:

Waiting for code...
Got code
2017/11/07 08:35:02 INFO  : Skipping API v1.0 endpoint https://XXXXX-my.sharepoint.com/_api/v1.0/me
2017/11/07 08:35:02 INFO  : Skipping API v2.0 endpoint https://XXXXX-my.sharepoint.com/_api/v2.0/me

After that it dumped the config output for the remote site. There were no prompts for the correct resourceURL.

As far as I can tell, this remote is now pointing to my personal space in the business one drive.

@olihey

This comment has been minimized.

Copy link
Contributor

olihey commented Nov 7, 2017

Thanks for the feedback, unfortunately I only have a OneDrive Business account and NO additional Team sites so I can't test it.

I have created a commit in my repo which let a user type in a resource URL before getting the final access token. You can build that commit and see if typing in works for you.

@dienal-san

This comment has been minimized.

Copy link

dienal-san commented Nov 7, 2017

Yes, that is the same output that I get when I discover.
Got code
2017/11/07 09:16:36 INFO : Skipping API v1.0 endpoint https://xxxxx-my.sharepoint.com/_api/v1.0/me
2017/11/07 09:16:36 INFO : Skipping API v2.0 endpoint https://xxxxx-my.sharepoint.com/_api/v2.0/me

The sites from the organization that we are trying to access are not displayed there.
My organization does not have something like this:
https://orgname.sharepoint.com/
https://orgname.sharepoint.com/Shared Documents

All sites are team sites like this:
https://orgname.sharepoint.com/site/site-name

@monkeyx-net

This comment has been minimized.

Copy link

monkeyx-net commented Dec 6, 2017

Hi, could anyone please confirm if this was resolved. I downloaded the latest beta yesterday. I tried the webdav option to my sharepoint/teams files and also used the business one drive option and manually changed the resource_url and attempted to validate the token again.

I couldn't get either option to work. Happy to help test further if needed.

@olihey

This comment has been minimized.

Copy link
Contributor

olihey commented Dec 6, 2017

Hej, the latest beta only uses the discovery service for getting resource urls. Unfortunately the discovery service does not "see" the sharepoint sites/team sites.
As I said before, I only have one Business account that isn't connected to any sharepoint sites, so I can't test it. Changing the resource url will NOT work, as the refresh token in the rclone.conf is only valid for the resource_url specified in the config file. Changing the URL invalidates the refresh token.

I made a change in my forked repo where you get the option to input an arbitrary resource_url during setup of the remote.

Can you try to get this branch compiled and run?

@monkeyx-net

This comment has been minimized.

Copy link

monkeyx-net commented Dec 6, 2017

I used the following command to build your repo:-
go get -u -v github.com/olihey/rclone

It is not prompting me to enter a resource _url when creating a onedrive business remote though.

Is there something different I need to do to build via go?

@monkeyx-net

This comment has been minimized.

Copy link

monkeyx-net commented Dec 8, 2017

I have subsequently tried cloning your git fork and switch to the onedrive_resoure_select branch. I then used go build to check that branch and it created a new rclone file. But I must still be doing something wrong as I am pretty sure it is not using onedrive.go file from that branch.

I no very little about go projects, so not able to troubleshoot further

@olihey

This comment has been minimized.

Copy link
Contributor

olihey commented Dec 8, 2017

What OS are you on? Maybe I can create a build for you to test.

@monkeyx-net

This comment has been minimized.

Copy link

monkeyx-net commented Dec 8, 2017

Debian stretch 64 bit.

Is there something I can read up to build properly via go. Or is it the git part I might be getting wrong?

@olihey

This comment has been minimized.

Copy link
Contributor

olihey commented Dec 8, 2017

OKay, I build an executable on my Ubuntu 16.04 AMD64, you can get it here: https://drive.google.com/open?id=1IsNSE34QK_UO-qzgSv2wuedxwsiMqlCZ

Please be aware that this test build currently can't do "headless" authorization, you have to use a machine which can show a web browser

@monkeyx-net

This comment has been minimized.

Copy link

monkeyx-net commented Dec 8, 2017

Thanks. I have requested access to the file, as not a public share from Google Drive

@olihey

This comment has been minimized.

Copy link
Contributor

olihey commented Dec 8, 2017

Sorry, not that skilled with Google Drive. Try this: https://drive.google.com/file/d/1IsNSE34QK_UO-qzgSv2wuedxwsiMqlCZ/view?usp=sharing

@monkeyx-net

This comment has been minimized.

Copy link

monkeyx-net commented Dec 8, 2017

Did not prompt me output shown below

tim@somepc:~$ ./rcloneodb -V
rclone v1.38-DEV

  • os/arch: linux/amd64
  • go version: go1.9.2

tim@somepc:~$ ./rcloneodb -vv config
2017/12/08 11:26:07 DEBUG : Using config file from "/home/tim/.config/rclone/rclone.conf"
2017/12/08 11:26:07 DEBUG : rclone: Version "v1.38-DEV" starting with parameters ["./rcloneodb" "-vv" "config"]
Current remotes:

Name Type
==== ====
gDrive drive
oneTest onedrive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> oneAsk
Type of storage to configure.
Choose a number from below, or type in your own value
1 / Amazon Drive
\ "amazon cloud drive"
2 / Amazon S3 (also Dreamhost, Ceph, Minio)
\ "s3"
3 / Backblaze B2
\ "b2"
4 / Box
\ "box"
5 / Cache a remote
\ "cache"
6 / Dropbox
\ "dropbox"
7 / Encrypt/Decrypt a remote
\ "crypt"
8 / FTP Connection
\ "ftp"
9 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage"
10 / Google Drive
\ "drive"
11 / Hubic
\ "hubic"
12 / Local Disk
\ "local"
13 / Microsoft Azure Blob Storage
\ "azureblob"
14 / Microsoft OneDrive
\ "onedrive"
15 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
16 / Pcloud
\ "pcloud"
17 / QingClound Object Storage
\ "qingstor"
18 / SSH/SFTP Connection
\ "sftp"
19 / Webdav
\ "webdav"
20 / Yandex Disk
\ "yandex"
21 / http Connection
\ "http"
Storage> 14
Microsoft App Client Id - leave blank normally.
client_id>
Microsoft App Client Secret - leave blank normally.
client_secret>
Remote config
Choose OneDrive account type?

  • Say b for a OneDrive business account
  • Say p for a personal OneDrive account
    b) Business
    p) Personal
    b/p> b
    Use auto config?
  • Say Y if not sure
  • Say N if you are working on a remote or headless machine
    y) Yes
    n) No
    y/n> y
    2017/12/08 11:27:55 DEBUG : Starting auth server on 127.0.0.1:53682
    If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
    Log in and authorize rclone for access
    Waiting for code...
    2017/12/08 11:28:11 DEBUG : Received request on auth server
    2017/12/08 11:28:11 DEBUG : Successfully got code
    Got code
    2017/12/08 11:28:12 DEBUG : oneAsk: Added new token to config, still needs to be saved
    2017/12/08 11:28:12 DEBUG : Closing auth server
    2017/12/08 11:28:12 DEBUG : Closed auth server with error: http: Server closed
    2017/12/08 11:28:15 INFO : Skipping API v1.0 endpoint https://xyxyxy-my.sharepoint.com/_api/v1.0/me
    2017/12/08 11:28:15 INFO : Skipping API v2.0 endpoint https://xyxyxy-my.sharepoint.com/_api/v2.0/me
    2017/12/08 11:28:15 DEBUG : oneAsk: Added new token to config, still needs to be saved

[oneAsk]
client_id =
client_secret =
token = {"access_token":"my secret token","expiry":"2017-12-08T12:28:11.494924Z"}
resource_url = https://xyxyxy-my.sharepoint.com/

y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name Type
==== ====
gDrive drive
oneAsk onedrive
oneTest onedrive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config

@olihey

This comment has been minimized.

Copy link
Contributor

olihey commented Jan 24, 2018

Hej, sorry for the long "pause". Problem is, I only have a Office 365 account for testing. Meaning this is very hard for me to fix without having anyway to try it. Sorry.
Maybe someone with such an account can help?

@olihey

This comment has been minimized.

Copy link
Contributor

olihey commented Jan 24, 2018

Can someone please go to this page https://developer.microsoft.com/en-us/graph/graph-explorer, authenticate your account and run the URL: https://graph.microsoft.com/v1.0/me/drives left to the "Run Query" button (you have to click the button after entering the URL).

Do you see your sharepoint / team sites in the "Response Preview" section?

@maroc81

This comment has been minimized.

Copy link

maroc81 commented Jan 24, 2018

I only see my personal one drive account. The team site is not returned in the response.

@maroc81

This comment has been minimized.

Copy link

maroc81 commented Jan 24, 2018

Looking at that graph explorer, there are several examples that deal with sharepoint. So far I was able to find the shared site by using the request

https://graph.microsoft.com/v1.0/sites?search=my_keyword

Where my_keyword is part of the name for the shared site.

I'm still looking to see if there are ways to list shared sites (it wasn't in https://graph.microsoft.com/v1.0/sites/root/drives). However, maybe you could use the search and prompt the user for the keyword?

Thanks!

@maroc81

This comment has been minimized.

Copy link

maroc81 commented Jan 24, 2018

I'm sure you've seen this:

https://developer.microsoft.com/en-us/graph/docs/api-reference/v1.0/api/site_get

But I couldn't find anyway to list the sharepoint sites I have access to. Maybe GET /groups/{group-id}/sites/root would work but I don't have permission to lists the groups I am a member of (/me/memberOf).

So it looks search may be the only way to go.

@olihey

This comment has been minimized.

Copy link
Contributor

olihey commented Jan 25, 2018

I haven't found a way to properly query for available sites per user. I can see that I can access them by "id". Do you have any chance to see the ID for the drives you want to access?
One on mine looks like this: b!9Z8pPhaBr06tcI6zsoXXXXXXXXXXXXXXXXD2yjX1ayLDuZSTKTwRAieJlnL

Because then I could add an option during setting up the remote to type in the ID by yourself.

@maroc81

This comment has been minimized.

Copy link

maroc81 commented Jan 25, 2018

Yes there is an id returned for the site in the response to

https://graph.microsoft.com/v1.0/sites?search=my_keyword

However, the id is in the form:

"id": "mycompany.sharepoint.com,########-####-####-####-############,########-####-####-####-############","

Where # is a hex digit. I see the id in the form you list above from the "me/drives" query. It appears sharepoint uses a different form. Don't know if that means it's incompatible with the access by id method you mentioned.

@just-ak

This comment has been minimized.

Copy link

just-ak commented Apr 6, 2018

Testing the Beta (Ubuntu Version) against SharePoint Team Sites,

One SharePoint Account with multiple Document Libraries.

  • Selection of Libraries : OK, (Currently Selected and Using 2 Libraries out of 4)

First Library Sync A few files have errors ( rclonebeta sync -v )

  • Transferred 34212 files OK,

  • 45 files with "corrupted on transfer: sizes differ 228951 vs 212118"


2018/04/06 07:26:09 INFO  : Local file system at /SharePoint/xxx: Waiting for checks to finish
2018/04/06 07:26:09 INFO  : Local file system at /SharePoint/xxx: Waiting for transfers to finish
2018/04/06 07:26:09 ERROR : Local file system at /SharePoint/xxx: not deleting files as there were IO errors
2018/04/06 07:26:09 ERROR : Local file system at /SharePoint/xxx: not deleting directories as there were IO errors
2018/04/06 07:26:09 ERROR : Attempt 3/3 failed with 7 errors and: corrupted on transfer: sizes differ 340894 vs 336099
2018/04/06 07:26:09 Failed to sync: corrupted on transfer: sizes differ 340894 vs 336099

This is a great, how can help get this into the mainstream live version.

@ncw

This comment has been minimized.

Copy link
Owner

ncw commented Apr 6, 2018

Thanks for testing @just-ak :-) User testing like yours is essential before we merge this.

Next step is for @olihey to have some free time to do the merging if he is happy with the code.

@just-ak

This comment has been minimized.

Copy link

just-ak commented Apr 12, 2018

@ncw @olihey I'm new to using GO, but am happy to help out getting this into the Main Branch so we can progress. Can @olihey identify which branch/version the beta was built from his version of rclone so i can help.

@olihey

This comment has been minimized.

Copy link
Contributor

olihey commented Apr 12, 2018

@just-ak The branch with all the Microsoft Graph stuff is in my forked branch https://github.com/olihey/rclone/commits/onedrive_graph

Please be aware that this branch is quite behind the current rclone master and I am currently (due personal reasons) don't have any time for progressing on this.

@just-ak

This comment has been minimized.

Copy link

just-ak commented Apr 14, 2018

@olihey @ncw
I have attempted to merge the code from https://github.com/olihey/rclone/commits/onedrive_graph SharePoint selection into the latest rclone. https://github.com/just-ak/rclone/tree/SharePoint. Initial section of drive selection appears to work with my developer SharePoint site, but does not work with the sub-sites i have created.
I have not fully tested by any means, but wanted it visible so progress can be made.

@just-ak

This comment has been minimized.

Copy link

just-ak commented Apr 15, 2018

@ncw I have managed to get the files to upload into SharePoint, simplified drive selection to be first select "site" (Including Sub Sites) then "drive" because the graph uploaded requires the siteID as part of the reference.

It might be a good idea to have available Multiple API Keys that link to difference configuration requirements, allowing just ReadOnly, and another ReadWrite.

In my test case of just 6 files I repeatedly get a size error on one of the files.
2018/04/15 22:05:28 ERROR : Test333.docx: corrupted on transfer: sizes differ 18630 vs 18633
It is just a simple sample document.

Do you have any pointers as to why the file size errors occur.

@ncw

This comment has been minimized.

Copy link
Owner

ncw commented Apr 16, 2018

@just-ak wrote

In my test case of just 6 files I repeatedly get a size error on one of the files.
2018/04/15 22:05:28 ERROR : Test333.docx: corrupted on transfer: sizes differ 18630 vs 18633
It is just a simple sample document.

I think that should have been fixed by 2f63a9f - have you got that in your branch?

@just-ak

This comment has been minimized.

Copy link

just-ak commented Apr 16, 2018

@ncw
I had that change, which appears not to fire when going through the function uploadFragment , I've made a fix (I'm new to GO language so you can probably do the same in a lot neater code) 432d242

  • Read the OneDrive JSON Response,
  • If different, modify size to what was expected.
@just-ak

This comment has been minimized.

Copy link

just-ak commented Apr 17, 2018

@ncw
For SharePoint file sync to work, rclone needs to calculate the "quickXorHash" https://docs.microsoft.com/en-gb/onedrive/developer/code-snippets/quickxorhash as SharePoint does not use sha1hash or cr32Has, as per https://docs.microsoft.com/en-gb/onedrive/developer/rest-api/resources/hashes

@ncw

This comment has been minimized.

Copy link
Owner

ncw commented Apr 20, 2018

@just-ak - I recently merged support for quickXorHash :-) #2262

I think this is a minimal version of your just-ak@432d242

Can you see if that works for you?

diff --git a/backend/onedrive/onedrive.go b/backend/onedrive/onedrive.go
index 8834dbb8..c71c4aad 100644
--- a/backend/onedrive/onedrive.go
+++ b/backend/onedrive/onedrive.go
@@ -1180,6 +1180,10 @@ func (o *Object) uploadFragment(url string, start int64, totalSize int64, chunk
 		}
 		return retry, err
 	})
+	if info != nil {
+		// Overwrite size with actual size since size readings from Onedrive are unreliable.
+		info.Size = totalSize
+	}
 	return info, err
 }

I'm slightly uneasy about that patch though as it means if rclone does a partial upload then the size will be corrected by that code...

@just-ak

This comment has been minimized.

Copy link

just-ak commented Apr 21, 2018

@ncw Fantastic QuickXorHash, will give it a good test, I agree that the different sizes is a problem that can't be overlooked.

The size problems have been noted in a couple of articles

http://techtrainingnotes.blogspot.co.uk/2010/11/sharepoint-changes-files-as-they-are.html
https://www.infoworld.com/article/2607874/managed-services/microsoft-onedrive-for-business-changes-files.html

I think the upload file size check could be achieved by uploaded using a tmp filename which SharePoint does not modify, and then once upload is complete and file- sizes & QuickXorHash are checked ok, rename the file to the required office filename(This would at least get the file onto the server). The modified file would then have to be downloaded after again to stop the continual sync differences.

Word,Excel,PowerPoint Presentation, Publisher

  • Upload File to SharePoint First time - Document Modified (Size & Hash Change)
  • Download File from SharePoint, Modify Document Locally, Re-Upload to SharePoint (Size & Hash Correct)
@just-ak

This comment has been minimized.

Copy link

just-ak commented Apr 21, 2018

@ncw SharePoint working with QuickXorHash nicely.

SharePoint uses a different Microsoft graph URL structure to OneDrive & OneDrive Business (personal area). At the moment it is a little confusing in the code, and needs some clarity.

I have put in a warning about a Size Change when the upload file ends with a document type that Microsoft may change, saying that the document should be used in SharePoint first. Thus files with the wrong size will not be uploaded.

e0b949d
Sorry I got the commit including the full upstream merge. Files changed are just (onedrive.go)

@mkoegel

This comment has been minimized.

Copy link

mkoegel commented May 2, 2018

Version 1.41 should support SharePoint via webdav, but the docu is not updated yet. Did someone get this to run?
I use the URL https://.sharepoint.com/sites/<site#>/Shared%20Documents/ and my user/pw, but get a 403 couldn't list files.

@heerohawwah

This comment has been minimized.

Copy link

heerohawwah commented May 2, 2018

I can confirm partial reading/writing from rclone to my 365 sharepoint site using webdav. It did require the latest rclone beta with the updated QuickXorHash @ncw.

The URL syntax: https://.sharepoint.com/sites//Shared%20Documents

I am however getting lots of IO errors and object not found errors during a copy or sync.

2018/05/02 08:51:47 ERROR : : Failed to copy: object not found
2018/05/02 08:51:47 ERROR : : Failed to copy: object not found
2018/05/02 08:51:48 ERROR : webdav root '': not deleting files as there were IO errors
2018/05/02 08:51:48 ERROR : webdav root '': not deleting directories as there were IO errors
2018/05/02 08:51:48 ERROR : Attempt 1/3 failed with 6 errors and: object not found

Edit: However the 'failed to copy' files do actually get copied and are intact on the sharepoint site.

I was going to try using OneDrive in rclone instead of WebDav but don't know how to change the Resource URL in the rclone OneDrive config to target the sharepoint site rather than the personal drive...

@heerohawwah

This comment has been minimized.

Copy link

heerohawwah commented May 3, 2018

I ran a full rclone copy, few hundred gigs to sharepoint. Lots of failed copy errors even though it seemed to successfully copy all the files even the ones it reported as failed. I'm running a sync right now with -vv over the same data. Here are some of the errors.

Sizes differ (src 5787690 vs dst 5793191)
2018/05/03 15:25:20 DEBUG : Couldn't decode error response: EOF
2018/05/03 15:25:20 DEBUG : Couldn't decode error response: EOF
2018/05/03 15:25:20 DEBUG : Couldn't decode error response: EOF
Failed to copy: object not found

Sizes differ (src 1505348 vs dst 1511487)
2018/05/03 15:27:15 DEBUG : Couldn't decode error response: EOF
2018/05/03 15:27:16 DEBUG : Couldn't decode error response: EOF

Overall it does seem to work just some EOF error of some kind. I'm not a programmer so thanks for all the help and efforts.

@ncw

This comment has been minimized.

Copy link
Owner

ncw commented May 7, 2018

@mkoegel wrote:

Version 1.41 should support SharePoint via webdav, but the docu is not updated yet. Did someone get this to run?

You can find the docs here: https://rclone.org/webdav/#sharepoint

@heerohawwah
Those "DEBUG : Couldn't decode error response: EOF" messages mean that the remote end returned an HTTP error rather than an XML Webdav error most likely. You can ignore them - they aren't important.

I wonder if the "Failed to copy: object not found" errors are caused by the rclone uploading the file, then going to check it but finding it isn't there yet due to some eventual consistency stuff.

If you retry the sync, does it clean up any it missed OK?

@mkoegel

This comment has been minimized.

Copy link

mkoegel commented May 10, 2018

@ncw I have a corporate sharepoint site. The docu only shows the OneDrive for Business case.
My URL is something like https://.sharepoint.com/sites/<site#>/Shared%20Documents/
That was also the original topic for this issue by @hi2u. @dienal-san also asked about this above.

@just-ak

This comment has been minimized.

Copy link

just-ak commented May 10, 2018

I modified the API's started by @olihey , which would be good to get pulled into the main, if some more testing can be done by others. https://github.com/just-ak/rclone Don't forget when a microsoft document is first moved into a sharepoint area it's Hash will change as sharepoint does a little work on the file.

@ncw

This comment has been minimized.

Copy link
Owner

ncw commented May 11, 2018

@mkoegel apologies - I mis-understood (I've never used sharepoint).

@just-ak do you want to send a pull request? I can turn that into a branch that makes beta builds for people and we can stablilise it there.

@just-ak

This comment has been minimized.

Copy link

just-ak commented May 28, 2018

Apologies for the delay, I have now created a pull request.

@ffinger

This comment has been minimized.

Copy link

ffinger commented Jul 31, 2018

Hi,
I have a similar problem with a folder that was shared with me (it is in somebody else's personal drive). Thus I can't get access to it via rclone. I can see it in Graph Explorer using this query:

https://graph.microsoft.com/v1.0/me/drive/sharedWithMe

I can get it's id and url.

Could that be solved in a similar way?

Thanks

@just-ak

This comment has been minimized.

Copy link

just-ak commented Aug 4, 2018

Looking briefly at the Microsoft Graph section for the Personal Drive section. The graph path ( https://graph.microsoft.com/v1.0/me/drive/sharedWithMe ) appears that this shows individual files that have been shared to the user, as Folders are already included in the normal tree structure under https://graph.microsoft.com/v1.0/me/drive/root/children.

A decision would need to be made as to how rclone should work with these files, Individually or treating them all together as special folder. @ncw

@ncw

This comment has been minimized.

Copy link
Owner

ncw commented Aug 22, 2018

We are in the process of merging Microsoft Graph support in #2472 by @Cnly which may help - it allows queries.

@ncw ncw added this to the Help Wanted milestone Aug 30, 2018

@llevrel

This comment has been minimized.

Copy link

llevrel commented Oct 31, 2018

Hello,

I've had a hard time setting up a remote to my university sharepoint site. The default config procedure only offered my university-provided personal OneDrive, upecnumerique-my.sharepoint.com/personal/[mail address], but not the Sharepoint site, upecnumerique.sharepoint.com/sites/[name].
The Graph Explorer gave no result with keyword "upecnumerique".

I opened the Sharepoint site HTML source code and looked for IDs. Given the above comment #1770 (comment) and some trial-and-error, I found the ID is upecnumerique.sharepoint.com,[siteId],[webId] (I tested this by inserting the ID in request https://graph.microsoft.com/v1.0/sites/[ID]/drive/root/children as advised above).

The sharepoint site "contains" many folders. I thought rclone config would add the site as a remote, but instead each folder is seen as a drive and I have to choose only one among the 57 of them :-/ Fortunately I only need 4 of them, so I can cope with 4 separate remotes.

Thank you for the great software.

@Cnly

This comment has been minimized.

Copy link
Collaborator

Cnly commented Nov 5, 2018

@llevrel I'm a little lost in your words. Are you looking for some way to add the whole site as a drive? I doubt if it's possible because if you look at the response of https://graph.microsoft.com/v1.0/sites/[ID]/drives, you'll see those "folders" (if I get your point) are really different drives.

@llevrel

This comment has been minimized.

Copy link

llevrel commented Dec 18, 2018

@Cnly I think you got my point. I'm new to this so "sites", "drives", etc. are vague concepts for me. Anyway I understand my wish cannot be fulfilled. Never mind, as I need only 4 of them it's no big deal. Thank you for answering!

@True-Demon

This comment has been minimized.

Copy link

True-Demon commented Jan 23, 2019

@llevrel In my research of this problem, it seems like this is more of a shortcoming of the graph API and its permissions model rather than rclone. Unfortunately, I don't see any feasible way to support this.

Every resource that rclone would require in order to "search" for team sites or sub-sites within sharepoint requires some form of admin privilege, even for you to check your own drives. You can probably thank Microsoft's absurd permissions model which doesn't distinguish an individual's resources from a team's resources. In order to list sites/drives for team sites and sub-sites that you have been granted access to, you need the Sites.ReadAll and User.ReadAll permissions according to the documentation, which is admin-only. To add content, you would need to elevate to the ReadWrite permission respectively.

https://docs.microsoft.com/en-us/graph/api/user-list-joinedteams?view=graph-rest-1.0

Rclone would either need to be granted permission to an organization's entire root site's resources as an application (a massive security hole), or you would need to have your user added as an administrator to grant yourself permission to your own team site, which would also grant rclone permission to ALL team sites under your credentials (impractical, and yet another security hole).

This is the sad story that is the Microsoft Graph API, which isn't much better than Windows permissions 😞

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment