-
-
Notifications
You must be signed in to change notification settings - Fork 80
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add new storage backend: Dropbox (#103) #251
Conversation
Thanks for working on this. As you already said I also think working against live infrastructure in tests should be avoided. The nicest thing to have would be a mock server to work against in the tests (like it's done for Azure Blob Storage). If this does not exist, I'm not entirely sure what to do. There seems to be this: https://github.com/jgonera/fake_dropbox but it looks pretty outdated. I wonder if it would be worth searching GitHub for how others do it, as we can't be the only ones who want to test Dropbox integrations. E.g. what does |
One solution might be to write a small simple mock rest server which replies with the expected responses and errors (restricted to MkDir, DelFile, ListDir, UploadFile). |
I wonder if we could use something like this https://github.com/stomakun/DropboxSwagger to automatically generate a mock server? |
There also seems to be a Dropbox published spec for their API https://github.com/dropbox/dropbox-api-spec but it's using some esoteric homegrown format for defining it. Not sure how much tooling there is around it and what it would take to generate a mock server from the spec. |
Using this with go-swagger could get us a mock server rather quickly. I will give it a try, since the |
If you wanna give this a try and see if we can get a test server running, it'd be much appreciated. |
Stumbled upon an auth issue: The used auth technique via token is kinda bad. The token periodically (24h?) expires, so that's no good for an automatic backup solution. OAuth2 would open (Docs) a browser window to allow the connection... every time? That's no good for a docker setup either. Update: The user needs to get an auth-token once manually via a specific URL in the browser. After that, they need to send this to an API endpoint to retrieve a refresh-token. With that, we can request new short-lived tokens to use for authorization every time we need one. (Source) |
Is there a way this can be done outside of this tool, e.g. by using |
The access token (short-lived, only a couple hours) could be retrieved via |
I had a look at using the Swagger spec plus Swagger to generate a go-server that mocks Dropbox. Seems to me like implementing this for testing is more effort than the feature itself. I will see if I can find an alternative, unless you have an idea in the meanwhile. |
Another approach to use here would be to create a server that plays back a session you record from the live API once, but I'm not entirely sure how well thought out such an approach is when dealing with integration tests, instead of unit tests. A tool that does this for unit tests would be https://github.com/dnaeon/go-vcr for example |
Stumbled upon that too. For testing purposes, the Dropbox SDK exposes the http client to be overwritten. So it should be technically possible to implement this. I will have a further look soon. |
I was actually thinking about finding something that works on integration test level (as everything else would require setting up another test mechanism in here):
This way, the tests for this PR could follow the established setup. |
Found something that looks promising: Will test that later. |
I had a look at probably a dozen mock/replay/record server tools. Many were net suitable in regards to their deployment or features. The solution I came up with is now using Downside of all this is, that we can't really upload anything to the mock server and therefore cannot check, if the file was correctly uploaded. But that would mostly test the API and the Dropbox library anyway, which is not our concern. Let's see if I can get it running not only on my machine (where it worked fine) but also in the CI. |
Well, Docker Hub seems to be unstable today? |
…ocker-volume-backup into dropbox-storage-backend
This reverts commit e9b617b.
Well, after 3h of trying the oauth2 mock server seems to be still allergic to Github CI... locally everything worked from the beginning like a charm, but in the CI the error 'server misbehaving' coming from the internal Docker DNS server is really weird. |
Thanks again. Could you maybe try to describe what the CI only problem with the OAuth Server is, so I can start investigating? |
The |
I've recently seen this in another context where it was caused by the target service not being up and therefore unreachable, even though |
So I could not get the compose setup from the dropbox test case running locally and tried to figure out why: it occured to me the service would never start, thus its hostname would also never resolve. I fixed the configuration according to the documentation here: https://github.com/navikt/mock-oauth2-server and got the service starting / the tests passing. You can see what I have done over in this PR: #257 It seems you were passing the expected response as the config file instead of the configuration values. Do you still want to adjust this in here so we can merge the PR and cut a new release? |
Oh... well then I wonder how it could work on my machine. But I also saw in your commit/PR, that the file extension was wrong (I used yaml, but the contents were JSON). Maybe that led to that weird behavior. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great now, thank you very much, especially for going all these extra miles on the testing topic. I'll try to release this very soon, but I'll leave it unmerged for now as for not to promise anything in the docs that isn't there yet.
This is now released in v2.30.0 🎉 |
Implementation of the official unofficial Dropbox API (Repo).
Partially addresses #103.
Implementation notes
This is fully transparent for the user and the rest of DVB. Files of up to 350GB are supported by the API this way.
Usage notes
DROPBOX_REMOTE_PATH
to the app's subfolder, where the app has access to (scope)... unless when creating the app, one chooses to give the app full access to everything.DROPBOX_TOKEN
) must be generated manually in the Dropbox app console (website).To be discussed
Testing this automatically is kind of an issue... for webdav/s3 etc one can simply spin up a mock service, but afaik not for Dropbox. This would require a live Dropbox account, which is kind of messy and potentially insecure.
During manual testing I verified the functionality of the file upload as well as pruning.
--> PR as draft, until this issue is addressed/discussed.
Todo