Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

onedrive: Support for OneDrive for Business added #1577

Closed
wants to merge 4 commits into from
Closed

onedrive: Support for OneDrive for Business added #1577

wants to merge 4 commits into from

Conversation

olihey
Copy link
Contributor

@olihey olihey commented Aug 3, 2017

  • 2 test fail (MimeType and modification date when copying)
  • no headless setup
  • uses the credentials for the "rclonetest" app I have created

- 2 test fail (MimeType and modification date when copying)
- no headless setup
- uses the credentials for the "rclonetest" app I have created
@ncw
Copy link
Member

ncw commented Aug 4, 2017

Thanks for submitting this. I'll hopefully have time to test and review it tomorrow.

Copy link
Member

@ncw ncw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple of doc fixes!

After that rclone requires two authentications. First to authenicate your account
and second to get the final token to access your companies resources.

Headless authentication is not working at the moment.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should probably change this to "Headless authentication is not working at the moment with rclone authorize - you'll have to use the copy the config file method".

@@ -142,10 +165,6 @@ is 10MB.
Note that OneDrive is case insensitive so you can't have a
file called "Hello.doc" and one called "hello.doc".

Rclone only supports your default OneDrive, and doesn't work with One
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this allow access to the non default OneDrive? If not then then this needs amending slightly.

@ncw
Copy link
Member

ncw commented Aug 5, 2017

The two authentications is a little confusing! Perhaps we should tell the user that we will auth twice (via some Print statements).

2 test fail (MimeType and modification date when copying)

I ran the integration tests, but I'm getting a completely different error on copy - it does the test then I get an unauthorized on the redirect. Use go test -v -verbose -dump-bodies -remote yourremote: to see the HTTP transactions.

I then got a 500 error!

I've had trouble with copy being unreliable on non business...

The MimeType test passes for me though!

no headless setup

That probably needs documenting.

uses the credentials for the "rclonetest" app I have created

Does the user have to create new credentials for each onedrive for business account? That was what I understood when I read the docs, but I may have been confused about that. If so we need to document that process.

What would you recommend about the onedrive credentials?

@olihey
Copy link
Contributor Author

olihey commented Aug 8, 2017

Hej, sorry for not responding but I am busy with work ;)
I'll test the error 500 and see if I can re-produce it.

Regarding the second authentication, onedrivecmd actually does the second one silently but as I wrote sometime ago I haven't found a way to update the token using rclone.
https://github.com/cnbeining/onedrivecmd/blob/master/utils/session.py#L151

When the second auth can be done "silently" we could also offer the headless authentication.

@ncw
Copy link
Member

ncw commented Aug 9, 2017

Hej, sorry for not responding but I am busy with work ;)

Perfectly understood!

I'll test the error 500 and see if I can re-produce it.

I'm not sure I would bother. Copy seems to be unreliable on onedrive :-(

Regarding the second authentication, onedrivecmd actually does the second one silently but as I wrote sometime ago I haven't found a way to update the token using rclone.
https://github.com/cnbeining/onedrivecmd/blob/master/utils/session.py#L151

When the second auth can be done "silently" we could also offer the headless authentication.

Tell me what you need to do and I'll see if the oauth library can be co-erced into doing it!

@olihey
Copy link
Contributor Author

olihey commented Aug 15, 2017

To avoid the second auth via WebBrowser we can also do a POST to https://login.microsoftonline.com/common/oauth2/token with the resource we got from the service API and get the new access_token back. Doing the POST and getting the data is not the problem, but how do I update the access token for the Fs that's been set during the initial auth? The getToken and putToken from oauthutil.go aren't exported?

@olihey olihey mentioned this pull request Aug 29, 2017
@ncw
Copy link
Member

ncw commented Aug 31, 2017

Apologies for the delay in responding, I've been on vacation.

how do I update the access token for the Fs that's been set during the initial auth? The getToken and putToken from oauthutil.go aren't exported?

If you need them, then make them exported so rename them to GetToken and PutToken. I generally make stuff unexported by default until it is needed elsewhere to minimise the API surface.

@olihey
Copy link
Contributor Author

olihey commented Sep 6, 2017

Finally got some time to look into this PR. I made the GetToken/PutToken exported in my branch and do a direct call to the TokenURL to get the refresh token for the OneDrive resource for the user. Working fine.

One last issue I have is doing "headless" authorization.
<---End paste 2017/09/06 09:27:03 ERROR : Failed to get resource token: json: cannot unmarshal string into Go struct field tokenJSON.expires_in of type int32 2017/09/06 09:27:03 ERROR : Failed to save new token in config file: section '**temp-fs**' not found

The issue is that I use GetToken and in headless the token is not saved into the config file.
Is it okay to test if name matches "**temp-fs**" and the break at that point because I know the user is running headless mode? I would make the "**temp-fs**" a bit more official ;)

@ncw
Copy link
Member

ncw commented Sep 6, 2017

The issue is that I use GetToken and in headless the token is not saved into the config file.
Is it okay to test if name matches "temp-fs" and the break at that point because I know the user is running headless mode? I would make the "temp-fs" a bit more official ;)

A little untidy ;-)

Can you factor GetToken into two to avoid this? If not just get it working and I'll have a think about how to make it less untidy!

@olihey
Copy link
Contributor Author

olihey commented Sep 6, 2017

Found a better way to solve this problem: fs.ConfigFileGet(name, fs.ConfigAutomatic) != ""

PR is fixed, please review

@zenjabba
Copy link

zenjabba commented Sep 7, 2017

Built this version and am able to see my OneDrive Business account as Admin, but as a non-admin user I'm getting this error

screen shot 2017-09-06 at 10 06 25 pm

@olihey
Copy link
Contributor Author

olihey commented Sep 7, 2017

@zenjabba Thanks for the testing, really appreciated.
I did check back in my Azure Console and it actually had set one access right that required "Admin" access (see below). I have disabled this now.
Can you try again? I only have an admin user for testing
image

@ncw We should also think about the client_id/client_secret because currently this uses a test app "rclonetest" I registered at Azure. From what I have heard you don't need to be a paying customer at Azure to create one yourself. I can send you screenshots of my setup in the Azure console.

@ncw
Copy link
Member

ncw commented Sep 7, 2017

@olihey I tried following the new procedure to create an app ID and it led me to the page where I created the existing rclone onedrive app id. Will that work with onedrive for business now do you think? If not please send a screenshot, thanks!

@olihey
Copy link
Contributor Author

olihey commented Sep 7, 2017

Here are the screenshots: https://drive.google.com/open?id=0B4DtcHPk696Ndy03V2pYNGR6Skk

@zenjabba
Copy link

zenjabba commented Sep 7, 2017

Success, non-admin user able to use application.

@ncw
Copy link
Member

ncw commented Sep 7, 2017

@olihey thanks for the screenshots - we need something like this in the docs I think as every user will have to do this.

I need help with registering a new app id though - I think I need to create a converged app: https://apps.dev.microsoft.com/#/appList - in particular what do I add in the Microsoft Graph Permissions section?

image

@olihey
Copy link
Contributor Author

olihey commented Sep 7, 2017

I don't think users has to setup this themselves as @zenjabba was able to access his OneDrive Business account with my rclonetest app.
I am trying to re-create the steps to create such an app on Azure and provide you with a guide.

@zenjabba
Copy link

zenjabba commented Sep 7, 2017

Just a performance update, we are seeing 8Mb/s as the best performance upload so far. No sure if it's code problem or rate limit on onedrive business.

@zenjabba
Copy link

zenjabba commented Sep 7, 2017 via email

@olihey
Copy link
Contributor Author

olihey commented Sep 7, 2017

I just tried to re-create the setup with a non-paying Microsoft account and I could not.
Rclone needs to use the "Office 365 SharePoint Online" API which is not available for "free" users (it's not on the list of available APIs):
image

That means either someone with an Office365 subscription "sponsors" an client_id/secret or each user has to setup the app in azure by themselves.

Basically I would be willing to share my client_id/secret 😄 Not sure what's the worst that can happen!?!?

@zenjabba
Copy link

zenjabba commented Sep 7, 2017

Offered @ncw a free Office 365 business account so he can control it all.

@l3uddz
Copy link

l3uddz commented Sep 7, 2017

great job, just wanted to say have been testing it with bwlimit=85M which seems to limit it to 8.5 MB/s. seems to be a bug

@ncw
Copy link
Member

ncw commented Sep 8, 2017

@zenjabba thank you for the account - most helpful. Can you email me at nick@craig-wood.com so I have your email details too?

I succesfully attached @olihey 's client id to the storage which was good - the registration process works very well now :-)

I failed in making an rclone app in the account from @zenjabba though - I couldn't figure out which permissions to add (see my comment) and I couldn't figure out how to get to the screenshot in your comment!

Maybe what I should do is merge the code as it is then sort that stuff out later if necessary?

@olihey wrote

Basically I would be willing to share my client_id/secret  Not sure what's the worst that can happen!?!?

Not a lot! It is really only a way of identifying a client - all the access to data etc requires the user to login.

@ncw
Copy link
Member

ncw commented Sep 8, 2017

@l3uddz wrote

, just wanted to say have been testing it with bwlimit=85M which seems to limit it to 8.5 MB/s. seems to be a bug

bwlimit 85M is 85 MBytes/s, if you want Mbit/s then divide by 8 or 10. Rclone reports its usage in MByte/s normally too.

@l3uddz
Copy link

l3uddz commented Sep 8, 2017

@ncw Im aware, just saying that with bwlimit=85M it throttles it to 8.5 MBytes, instead of the expected 85 MBytes/s.

@olihey
Copy link
Contributor Author

olihey commented Sep 8, 2017

@ncw Feel free to merge this with my client_id/secret for now

@zenjabba
Copy link

zenjabba commented Sep 9, 2017

I can confirm this is running at full speed with --onedrive-chunk-size=30M if you dont have this, you run into API limits and it slows you down.

@ncw
Copy link
Member

ncw commented Sep 12, 2017

I have merged this now. Can anyone testing make comments on the original issue #254 please.

I'm going to close this PR now.

@olihey can you make new PRs for fixes, enhancements!

I'm hoping to stabilise this for the 1.38 release which I have planned for end of the month.

@ncw ncw closed this Sep 12, 2017
@ttestdock
Copy link

@ncw There are the permissions that a converged app needs.
default

@Grand-Grand
Copy link

Hey.
I am new to the rclone.
And I jumped into the same problem as mentioned above.
So I have a few OneDrive for business accounts would like to link with rclone; and turn out not working, due to the lack of admin permission I guess?

The situation I am facing is,
these accounts are free Office365 account which University provides to us.
Here is how the Subscription looks like:
image
When I add these account to rclone, it just results (http://localhost:53682/?error=access_denied&error_description=AADSTS65005%3a+Using+application+%27rclone%27+is+currently+not+supported+for+your+organization+sheffield.ac.uk+because+it+is+in+an+unmanaged+state.+An+administrator+needs+to+claim+ownership+of+the+company+by+DNS+validation+of+sheffield.ac.uk+before+the+application+rclone+can+be+provisioned.%0d%0aTrace+ID%3a+672c6641-32e7-415c-924b-12cbc7038600%0d%0aCorrelation+ID%3a+5ea5bfc9-4390-4a46-bae1-2015c813a11d%0d%0aTimestamp%3a+2017-10-17+15%3a07%3a26Z&state=f9c38dc6c155f395918844a5ce08d16d) as the return.

Problem is, I am only a student of the Uni, apparently, I do not have the access to the admin account. But besides, when I check the Site Permission, it shows only my account without the others, and I have the full control of the site.
image
However, I cannot access the OneDrive Admin.
image

I tried to use Office Admin, but this requests me own the Domain: sheffield.ac.uk; again, you can tell this is a University URL which doesn't belong to me...

Hence, is there any solution that I can bypass the admin requirement?

P.S. it seems like no one is admin this site though, I believe it's more likely Microsoft mis-understands my account?

@olihey
Copy link
Contributor Author

olihey commented Oct 17, 2017

image
As you can see, the premissions don't require Admin.
But looking at your plan, it does not include "Office 365 Sharepoint Online" because that's the API (Permission) rclone uses to access the files.
image

Also, if I look at the result from the registration it returns error AADSTS65005 which might indicate that there is something wrong with the permissions in Azure for the rclone app we use the credentials.
As I am not a Azure pro so there might be a setting we have forgotten.

You can try either creating your own app (clientID / client secret) on azure or try out if onedrivecmd works for you.

@zenjabba
Copy link

zenjabba commented Oct 17, 2017 via email

@Grand-Grand
Copy link

@olihey
@zenjabba

So in this case, what shall I do with the Sharepoint EDU or azure? Any suggestions?
I am pretty new in this thing. X_X

@olihey
Copy link
Contributor Author

olihey commented Oct 17, 2017

If I decode the URL I get:
http://localhost:53682/?error=access_denied&error_description=AADSTS65005: Using application rclone is currently not supported for your organization sheffield.ac.uk because it is in an unmanaged state. An administrator needs to claim ownership of the company by DNS validation of sheffield.ac.uk before the application rclone can be provisioned. Trace ID: 672c6641-32e7-415c-924b-12cbc7038600 Correlation ID: 5ea5bfc9-4390-4a46-bae1-2015c813a11d Timestamp: 2017-10-17 15:07:26Z&state=f9c38dc6c155f395918844a5ce08d16d

Please be aware that an organziation can control the access to the data which can prevent you from using rclone. One of the companies I work for only allows access to OneDrive Business from enrolled PCs and approved applications. No dice to use rclone there.

Try onedrivecmd and see if that works

@Grand-Grand
Copy link

@olihey
Thanks for replies. I will have a go with OneDriveCMD first.

(P.S. Actually I saw comments on OneDriveCMD said rclone is better; hence, I am here xD)

@putiyeb
Copy link

putiyeb commented Oct 18, 2017

@Grand-Grand
I write a detail tutorial. Follow it , you can create the api easily.
https://shui.azurewebsites.net/2017/07/17/onedrive-for-business-api/

@Grand-Grand
Copy link

@putiyeb
Unfortunately, I exactly followed your instruction, still failed to connect OneDrive Business with rclone. Here is the error URL: (https://login.microsoftonline.com/common/oauth2/authorize?access_type=offline&client_id=ab50ec69-f534-4ebd-9388-d782464bf8c5&redirect_uri=http%3A%2F%2Flocalhost%3A53682%2F&resource=https%3A%2F%2Fapi.office.com%2Fdiscovery%2F&response_type=code&state=e8888a091f5597f4acafcad29eba47cf)
This is another try (https://login.microsoftonline.com/common/oauth2/authorize?access_type=offline&client_id=ab50ec69-f534-4ebd-9388-d782464bf8c5&redirect_uri=http%3A%2F%2Flocalhost%3A53682%2F&resource=https%3A%2F%2Fapi.office.com%2Fdiscovery%2F&response_type=code&state=a38679c452b01fdc6aaab1ff12d50bf6).

Microsoft just shown: Sorry, but we’re having trouble with signing you in. We've received a bad request.
Not sure what problem cause the bad request.

BTW, on insrtruction, it said i have to put "http://localhost:53682/" as a reply URL, which doesn't work for me, I mannually put "127.0.0.1:53682" instead.

P.S. I guess you are Chinese? (I am Chinese too)

@putiyeb
Copy link

putiyeb commented Oct 18, 2017

@Grand-Grand
It looks like the domain of your account hasn't been verified. Without verifying, your account doesn't work with any API. Because the organization record cannot be found in AZURE.
Do you sign up the account through the website(https://products.office.com/en-us/student/office-in-education)?

PS:Yeah :)

@Grand-Grand
Copy link

@putiyeb
No, actually I am studying in UK. so I registered with this website (https://products.office.com/en-GB/student/office-in-education).

@hensur
Copy link
Contributor

hensur commented Dec 30, 2017

@Grand-Grand I actually got the same error for my university. I contacted them and they said that the validation would require them to manage password resets and other stuff by themselves, which is why they don't plan to validate the domain.
Yesterday I tried to use duplicati for some of my file backups and connected it to my Office 365 education account. I was surprised that it managed to upload to onedrive. I suppose it's because it uses a different onedrive for business/sharepoint implementation. I had to put in my actual username and password and the server it should connect to (e.g. [universityname]-my.sharepoint.com).
It would be really cool if rclone could use this as a second authentication method!

@ncw
Copy link
Member

ncw commented Jan 10, 2018

@hensur can you flesh this out in a new issue on github please?

@putiyeb
Copy link

putiyeb commented Jan 13, 2018

@hensur @Grand-Grand
The OD4B of your account can be mounted by WebDav, using Cookie to authorize.
See detail:
https://shui.azurewebsites.net/2018/01/13/mount-onedrive-for-business-on-headless-linux-vps-through-webdav/

@ncw
Is it possible to support cookie with the webdav of Rclone?

@hensur
Copy link
Contributor

hensur commented Jan 14, 2018

@putiyeb @Grand-Grand I just created issue #1975 for this, as requested by @ncw

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants