Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request] Support backup targets like Box / Dropbox #103

Open
1 of 3 tasks
lonix1 opened this issue Jun 2, 2022 · 11 comments
Open
1 of 3 tasks

[Feature Request] Support backup targets like Box / Dropbox #103

lonix1 opened this issue Jun 2, 2022 · 11 comments
Labels
enhancement New feature or request pr welcome

Comments

@lonix1
Copy link

lonix1 commented Jun 2, 2022

  • I'm submitting a ...

    • bug report
    • feature request
    • support request

  • What is the current behavior?
    Backups can be made to a variety of S3-like targets, but not storage services like Dropbox, Box, etc.

  • What is the motivation / use case for changing the behavior?
    These are very common in business environments.

I'm not a go dev, but maybe someone else knows how to do this... Thanks!
@lonix1 lonix1 mentioned this issue Jun 2, 2022
Closed
@m90 m90 added enhancement New feature or request help wanted Extra attention is needed labels Jun 3, 2022
@m90
Copy link
Member

m90 commented Jun 3, 2022

If anyone wants to add this, there seems to be an official (published by Dropbox) "unofficial" Dropbox SDK for Go: https://github.com/dropbox/dropbox-sdk-go-unofficial which can do all we need to do here.

@lonix1
Copy link
Author

lonix1 commented Jun 3, 2022

Nice find.

For Box: has many dev tools and SDKs but not an official Go SDK; however it does have a CLI which (I assume) could be wrapped in go code.

Box is nice because their free tier is a massive 10GiB vs DropBox's 2GiB. Both are good though.

@MaxJa4 MaxJa4 mentioned this issue Jul 22, 2022
Merged
@m90 m90 added pr welcome and removed help wanted Extra attention is needed labels Sep 28, 2022
@MaxJa4 MaxJa4 mentioned this issue Aug 20, 2023
Merged
2 tasks
m90 pushed a commit that referenced this issue Aug 24, 2023
@MaxJa4
Add new storage backend: Dropbox (#103) (#251)
e08a330 
* Add new storage backend: Dropbox (#103)

* Remove duplicate check

* Add concurrency level for parallel upload to dropbox.

* Fixed some instabilites. Changed default concurrency to 6.

* Added some env config vars to readme. WIP

* Wrap errors for storage backend creation.

* Fixed token issue, added OAuth2 including recipe and docs.

* Readme typo fix

* Test for dropbox integration

* Update info and TOC

* Missed a file

* Docker-compose fix

* Fix endpoint connection

* Fix container names

* Fix log fetching

* Fix log fetching (again)

* Print command output to logs

* Addressing comments part 1

* Address comments part 2

* OpenAPI Mock spec path adjusted
* Dropbox FileMetadata reflection refactored
* NaturalNumber type added

* Add OAuth2 mock server for CI testing

* Fix env name of oauth2 endpoint

* Remove hostname

* Add forgotten change to commit...

* Fix oauth2 endpoint

"Worked on my machine"

* Try again

* Try suggested hostname again

* Fix docker internal DNS resolving issues (as suggested by oauth2 mock docs)

* Add docker network, remove hostname

* Network not external

* Last hostname try

* Add more delay, add oauth2 endpoint log

* Temp CI log output of command even when failing

* Try different config and method

* Add custom server-hostname. Rename test folder to accellerate debugging

* Try that fix again

* Adding quotes

* Port fix attempt

* Try localhost

* Try extra hosts

* Change network mode

* Undo some changes

* Use static IP

* Remove specific IP binding

* Change to default net driver

* Fix static IP

* Squash for revert

* Revert "Squash for revert"

This reverts commit e9b617b.

* Actual fix for CI testing from #257
@m90
Copy link
Member

m90 commented Aug 24, 2023

Dropbox is now supported as of v2.30.0

@MaxJa4
Copy link
Contributor

MaxJa4 commented Aug 24, 2023

Haven't heard of Box before. Is it widely used in some countries or am I the only one who never heard of them before?

On another note, if we include Dropbox, perhaps Google Drive make sense as an option too. There is an official Go SDK for it and it also uses OAuth2 like Dropbox.

@lonix1
Copy link
Author

lonix1 commented Aug 25, 2023 •
edited

Haven't heard of Box before. Is it widely used in some countries or am I the only one who never heard of them before?

The latter I'm afraid :-) Box is gigantic and in the same league as Dropbox - and like I said above very common in business environments.

@m90
Copy link
Member

m90 commented Aug 25, 2023

The latter I'm afraid :-) Box is gigantic and in the same league as Dropbox - and like I said above very common in business environments.

Gotta be honest, I also never came across Box outside of this issue.

On another note, if we include Dropbox, perhaps Google Drive make sense as an option too.

Google Drive is probably heavily used, which is good, but I am a bit worried about API stability here as Google is pretty notorious for just changing things when they feel like it. It's probably not possible to connect to Google Drive using some open standard, e.g. WebDAV, SSH, S3? Also: as we've just been bitten by this: is it possible to run a mock Google Drive in tests instead of having to resort to a live service?

@lonix1
Copy link
Author

lonix1 commented Aug 25, 2023

I also never came across Box outside of this issue.

Info: https://en.wikipedia.org/wiki/Box_(company) NYSE listed.

Just keep in mind you (probably?) won't find larger storage accounts elsewhere in a free tier.

@MaxJa4
Copy link
Contributor

MaxJa4 commented Aug 25, 2023 •
edited

Google Drive is probably heavily used, which is good, but I am a bit worried about API stability here as Google is pretty notorious for just changing things when they feel like it. It's probably not possible to connect to Google Drive using some open standard, e.g. WebDAV, SSH, S3? Also: as we've just been bitten by this: is it possible to run a mock Google Drive in tests instead of having to resort to a live service?

Didn't find anything about WebDAV etc. but there is also a OpenAPI spec for it, just like with Dropbox: Spec.

Edit: There is also a spec for Box.
Edit2: Box works similarly to Dropbox on the API level. Although using the CLI seems complicated (has to be installed via NPM), we could implement it by calling the REST API directly - we just need a few routes anyway. The docs are pretty good and the upload session architecture is similar to Dropbox's.

@MaxJa4
Copy link
Contributor

MaxJa4 commented Aug 30, 2023

Just saw that Box limits the total size of one file to 250 MB for the free tier... so the '10GB for free' is only for marketing purposes, unless you have many small files. For backups, I'd say backups are rarely below 250 MB and also, Box doesn't know anything about paths, it only uses IDs for their API - which doesn't exist for Go, so we'd need to call their REST API directly. Lastly, everything about Box is geared towards companies and business use, which doesn't make it easier to e.g. create a new app inside Box to try things out with test access tokens.
All that combined is making it quite complicated and let's me pause on that for now if I'm honest.

@lonix1
Copy link
Author

lonix1 commented Aug 31, 2023 •
edited

The 250Mib thing is surprising, maybe our backups are small.

EDIT:
oh just checked we create a series of gzip files of ~250 each. :)

Well, thanks for looking into it. After your research which provider do you recommend?

@MaxJa4
Copy link
Contributor

MaxJa4 commented Sep 3, 2023

Well if you don't need a long history of backups, maybe the 2GB of Dropbox is already enough.
OneDrive has a 5 GB free tier (not yet implemented here).
Google Drive even has 15 GB free tier (not yet implemented here).
Backblaze B2 has 10 GB free tier (egress will be free too starting at October). You can use this with the S3 backend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request pr welcome
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@m90 @lonix1 @MaxJa4