onedrive: Onedrive Business Support for organizations in an unamanged state (mostly universities)

[hensur]

This topic was discussed at the end of Issue #1577, but as @ncw requested I created a new one for this specific problem.

Many schools don't verify their Office 365 Education Account. This prevents students from authorizing application using the API. Authorization requests return this error:

What is your rclone version (eg output from rclone -V)

./rclone -V
rclone v1.39-025-g60afda00β
- os/arch: linux/amd64
- go version: go1.9.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

ArchLinux 4.14.13-1-ARCH #1 SMP PREEMPT Wed Jan 10 11:14:50 UTC 2018 x86_64 GNU/Linux

Which cloud storage system are you using? (eg Google Drive)

OneDrive (Office 365 Education)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

./rclone authorize onedrive
Choose OneDrive account type?
 * Say b for a OneDrive business account
 * Say p for a personal OneDrive account
b) Business
p) Personal
b/p> b
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
2018/01/14 15:50:08 Failed to configure token: failed to get code

Returned URL:

http://localhost:53682/?error=access_denied&error_description=AADSTS65005%3a+Using+application+%27rclone%27+is+currently+not+supported+for+your+organization+*********+because+it+is+in+an+unmanaged+state.+An+administrator+needs+to+claim+ownership+of+the+company+by+DNS+validation+of+***********+before+the+application+rclone+can+be+provisioned.

I already contacted my school, but they don't plan to verify their domain, because it would require them to handle password resets and other stuff by themselves.

However, there seem to be a few workarounds:
I managed to connect duplicati to my Office 365 education account.

I suppose it works because it uses a different onedrive for business/sharepoint implementation.
I had to put in my actual username and password and the server it should connect to (e.g. [universityname]-my.sharepoint.com).

Another way is using webdav, as mentioned by @putiyeb in the original issue:
"The OD4B of your account can be mounted by WebDav, using Cookie to authorize.
See detail:
https://shui.azurewebsites.net/2018/01/13/mount-onedrive-for-business-on-headless-linux-vps-through-webdav/"
Apparently this repo implements a way of getting an onedrive cookie in python and uses a bash script to mount the share using davfs2:
https://github.com/yulahuyed/test

[ncw]

@olihey Can you shed any light on this? I have to say I find the innumerable ways of authenticating to OneDrive very confusing!

[olihey]

I only have an Office365 account, so I have no chance to look into that.
I looked into duplicati but this is a C# and looking at this code:

if (useIntegratedAuthentication || (useUsername == null || usePassword == null))
            {
                // This might or might not work for on-premises SP. Maybe support if someone complains...
                m_userInfo = System.Net.CredentialCache.DefaultNetworkCredentials;
            }
            else
            {
                System.Security.SecureString securePwd = new System.Security.SecureString();
                usePassword.ToList().ForEach(c => securePwd.AppendChar(c));
                m_userInfo = new Microsoft.SharePoint.Client.SharePointOnlineCredentials(useUsername, securePwd);
                // Other options (also ADAL, see class remarks) might be supported on request.
                // Maybe go in deep then and also look at:
                // - Microsoft.SharePoint.Client.AppPrincipalCredential.CreateFromKeyGroup()
                // - ctx.AuthenticationMode = SP.ClientAuthenticationMode.FormsAuthentication;
                // - ctx.FormsAuthenticationLoginInfo = new SP.FormsAuthenticationLoginInfo(user, pwd);
            }

they are using some Windows libs to get access.

[olihey]

@hensur Can you please go to this page https://developer.microsoft.com/en-us/graph/graph-explorer, authenticate your account and run the URL: https://graph.microsoft.com/v1.0/me/drives left from the "Run Query" button. Do you see your Education account?

[hensur]

@olihey I can't even authenticate on the page, it shows the same error as rclone does in the URL:

https://developer.microsoft.com/en-us/graph/graph-explorer#error=invalid_client&error_description=AADSTS65005%3a+Using+application+%27Graph+explorer%27+is+currently+not+supported+for+your+organization+****+because+it+is+in+an+unmanaged+state.+An+administrator+needs+to+claim+ownership+of+the+company+by+DNS+validation+of+****+before+the+application+Graph+explorer+can+be+provisioned.

[olihey]

Thanks for trying, seems like you have to kick your IT department. Sorry

[hensur]

Hi, so in the last months I did some projects in go and now I thought about reviewing the other possibilities to authenticate. I implemented the cookie retrieval in go:
https://github.com/hensur/onedrive-cookie-test

I can get a cookie now and it is possible to mount the onedrive account using davfs2 and the retrieved cookie. It probably would be possible to setup a local rclone on this mountpoint, but that's not really my favorite way of solving this.

I looked into the rclone sources to find a method to integrate the cookie auth into, but I'm not quite sure where to start.

So I thought about here (https://github.com/ncw/rclone/blob/master/backend/webdav/webdav.go#L282) and added a SetCookie method to the rest module which basically does the same as the SetUserPass method (https://github.com/ncw/rclone/blob/master/lib/rest/rest.go#L96)

I got a 403 Forbidden response after hardcoding some values in there and trying it with a new webdav endpoint which points to my sharepoint URL.
The error came from here: https://github.com/ncw/rclone/blob/master/backend/webdav/webdav.go#L202
I guess it is because of api.CallXML, which calls api.callCodec and then api.Call?
The cookie seems to get lost here because a new request is created?!
https://github.com/ncw/rclone/blob/master/lib/rest/rest.go#L231

I'd be happy for some help to get started :)

[ncw]

@hensur that is great! I'm traveling at the moment so I'll be brief.

Did you try using --vv with --dump bodies? That will show you exactly what happens to the http requests.

Can you show code for what you've done so far?

Gotta go now plane doors shutting!

[ncw]

@hensur back now!

I would have thought setting your cookie with SetHeader would be the right approach.

The cookie seems to get lost here because a new request is created?!

I don't think a new request gets created... Did you try with -vv and --dump bodies to look at the HTTP transactions?

[hensur]

Hey @ncw! I hope you had a good flight :)
Thank you for the fast response. I had not much time last week, but I just looked into it again and found my mistake.
My SetCookie method just created a new request, added the cookie and returned that request. I just now noticed the api SetHeader method. So I changed my method to get the Cookie Header from the new request and saved it using api.SetHeader.

I just pushed my changes to my rclone fork: https://github.com/hensur/rclone
I also created a small package to fetch the required cookies: https://github.com/hensur/onedrive-cookie-test/tree/master/odrvcookie
Although it would probably be better to integrate this directly into rclone, it works quite good for testing the authentication.

In the current state I can use my fork to authenticate and somehow list the remote.
I'm getting this error for every folder in the remote using rclone lsd webdav:/

2018/04/09 10:11:55 ERROR : personal/MY_EMAIL_ADDRESS/Documents: Entry doesn't belong in directory "" (contains subdir) - ignoring
2018/04/09 10:11:55 ERROR : personal/MY_EMAIL_ADDRESS/Documents/example.file: Entry doesn't belong in directory "" (contains subdir) - ignoring
2018/04/09 10:11:55 ERROR : personal/MY_EMAIL_ADDRESS/Documents/Anlagen: Entry doesn't belong in directory "" (contains subdir) - ignoring
2018/04/09 10:11:55 ERROR : personal/MY_EMAIL_ADDRESS/Documents/Office Lens: Entry doesn't belong in directory "" (contains subdir) - ignoring
2018/04/09 10:11:55 ERROR : personal/MY_EMAIL_ADDRESS/Documents/Forms: Entry doesn't belong in directory "" (contains subdir) - ignoring
2018/04/09 10:11:55 ERROR : personal/MY_EMAIL_ADDRESS/Documents/backups: Entry doesn't belong in directory "" (contains subdir) - ignoring

[ncw]

I just pushed my changes to my rclone fork: https://github.com/hensur/rclone

I had a quick look through that - it is looking very nice :-)

I'm getting this error for every folder in the remote using rclone lsd webdav:/
2018/04/09 10:11:55 ERROR : personal/MY_EMAIL_ADDRESS/Documents: Entry doesn't belong in directory "" (contains subdir) - ignoring

What that means is that the directory itself appeared in the directory listing for the directory. This should be filtered out but that isn't working for some reason - I expect a bit of examination of the lsd with -vv --dump bodies and the code will tell you why!

Will this code work for any onedrive account? Eg my personal one?

Some thoughts about making this into a pull request

• needs docs (in docs/content/webdav.md)
• needs an extra OptionExample for the "onedrive" quirk...
• can you split changes to the rest module into a separate commit.
• if you want to keep your : https://github.com/hensur/onedrive-cookie-test/ library separate that is fine by me - we could then vendor it. Or you could include that file into the webdav remote.

Excellent progress - looking forward to seeing the next step :-)

[hensur]

Just a quick update, I did some testing and I noticed that the error above only happens if i execute rclone lsd webdav:/
After some trial and error I ran rclone lsd webdav: (without the slash) and the lsd worked perfectly fine.
I tried the same with my nextcloud and noticed this behavior there as well (even with the latest stable).
Is this something that should be fixed or some wanted behavior?

I'm currently working on the pull request. I can try to further examine the / issue...

A normal onedrive Account can be accessed over webdav as well, although the procedure is quite different: https://blogs.msdn.microsoft.com/robert_mcmurray/2014/09/30/using-the-webdav-redirector-with-onedrive-part-1-standard-security/
This method is password based, so there is no need to fetch any cookie. The quirk I added is therefore not needed for this to work.
I just tried it (rclone and davfs) and got a 401 Unauthorized response although I created an app password. The article is quite old, maybe something has changed. I will try it on Windows later today.

[ncw]

@hensur wrote:

Just a quick update, I did some testing and I noticed that the error above only happens if i execute rclone lsd webdav:/
After some trial and error I ran rclone lsd webdav: (without the slash) and the lsd worked perfectly fine.
I tried the same with my nextcloud and noticed this behavior there as well (even with the latest stable).
Is this something that should be fixed or some wanted behavior?

Rclone is supposed to be agnostic about / so webdav: and webdav:/ should be the same thing. So that is worth investigating (in another commit!).

Oh - I see you did that already in your PR :-) Excellent!

[ncw]

I think this issue is sorted at least for webdav.

In the light of #2325 I think there needs to be some documentation in docs/content/onedrive.md pointing people to the webdav solution. We should put the text of the error message in the docs.

I wonder if rclone can detect the error and print a more helpful error message...

[hensur]

ok, I'll add the docs.
I'm guessing by "more helpful message" you mean the webpage that is shown in the screenshot? I will look into that as well then.

[ncw]

Not necessarily the screenshot, but a pointer to what to do if your org turns out to be unmanaged and how you would recognise it.

[xiaolei0125]

@hensur , @ncw
This is a problem to copy file in webdav (sharepoint onedrive) , for example:
When copy or sync from a source directory to local, it work fine.
rclone.exe copy od1:song/ ./

When copy from a source file to local,, it has error:
rclone.exe copy od1:song/my.mp3 ./
ERROR : : error reading source directory: directory not found