Initial support for BackBlaze B2 #8
Conversation
|
Looks promising, thanks for sharing! Are you looking for feedback already, or do you want to finish the implementation first? For the blackslash issue, I'd suggest to use a technique similar to the one implemented in the |
|
Thank you for suggest, i corrected the escaping functions. |
|
Note: Blackblaze seems to be open to add backend functionality to support s3ql better (see this comment). They could implement server-side copy if requested by opening an issue. |
|
@sylvainlehmann May I test your code? |
|
@dlight Thanks for the suggest, i will contact them. |
|
~/.s3ql/authinfo2 [blackbaze]
storage-url: b2://my-bucket/prefix_
backend-login: <Account ID>
backend-password: <Application key>
fs-passphrase: <high entropy>mkfs.s3ql b2://my-bucket/prefix_ and mount.s3ql b2://my-bucket/prefix_ /mnt work OK. Keep up the work. |
|
@sylvainlehmann May I begin migrating my clients to BB? |
|
Build S3QL + B2 in a Docker container - e.g. by starting an UpCloud instance |
Could that be the consequence that copying an empty dir with |
|
Hey @szepeviktor, thanks for your work. I have built s3ql with b2 using your docker and have got it to mount inside the docker, now my question is how do you get it working outside. Thanks. |
|
Although s3ql is installable by Also you may run the commands of that Dockerfile on your server. |
|
@szepeviktor OK so there is no way to export the s3ql mount from the docker to the host? |
|
Of course there is: |
|
@szepeviktor Hmm, I know of this and have tried it and for whatever reason I cannot see my mount from outside the docker but I can pass through files. I will try with different mount options. |
I think not. Creating folders should only concern database. And server-side copy is only used for metadata rolling backup (as far as i can see)
This implementation looks quite stable, i use it daily since a few monts with no problems, but i work on a quite low load and just for a few Tb of data. If you plan to use this implementation under heavy load and a very large amount of data, i suggest you to make extended tests before |
Could you convince @Nikratio to merge it? |
|
Right now we're using your fork and uploading about 20T of data. Everything is as stable as it can be. The plan is to use s3ql.b2 to stream multimedia. We will update as soon as we have some results |
|
I've been using this for the last 2 weeks, until recently the filesystem crashed: Coudn't find any logs (?). There was nothing in syslog... need to see if FUSE dump a core somewhere. |
|
@ToroNZ could you please open an issue on the fork (https://github.com/sylvainlehmann/s3ql) and past the content of ~/.s3ql/mount.log ? |
|
@sylvainlehmann Finally manage to log it! My apologies for taking so long, stuff gets in the way. |
|
Is this production ready yet? I tested this this earlier in the year and my results were pretty good. |
|
Sorry for the delay! I will take a look at this soon. |
|
At the moment this is very hard to review, since it mixes changes from in between S3QL releases with the actual changes from adding BackBlaze support. I suspect that something went wrong with the merges. Could you please squash everything into one commit, and rebase that onto current master? Thanks! |
.hgtags
Outdated
| @@ -86,3 +86,4 @@ cc9dfb6bc9ebb4290aaaa2e06e101c460d4126f5 release-2.15 | |||
| a7de95ab14ceb9f5be5eb8b44694a4c2a766fe89 release-2.18 | |||
| 18d9916a31218052107bcbd9739f504f9f536e24 release-2.19 | |||
| 6036e4dfee66dba05f2bbb1319b90ed7c9557688 release-2.20 | |||
| 5649b9e2d23e66d79b180107057515c73d621731 release-2.21 | |||
There was a problem hiding this comment.
This doesn't look like it has anything to do with adding BackBlaze support
Changes.txt
Outdated
| @@ -1,3 +1,10 @@ | |||
| 2016-10-28, S3QL 2.21 | |||
rst/installation.rst
Outdated
| @@ -59,7 +59,7 @@ that is not the case. | |||
| version between 1.0 (inclusive) and 2.0 (exclusive) | |||
| * `dugong <https://bitbucket.org/nikratio/python-dugong/>`_, any | |||
| version between 3.4 (inclusive) and 4.0 (exclusive) | |||
| * `pytest <http://pytest.org/>`_, version 2.3.3 or newer (optional, to run unit tests) | |||
There was a problem hiding this comment.
..nor this, stopping here
|
@sylvainlehmann Your code works at each night on my development server doing an almost complete backup (/usr, mysql, /home, /etc) Could you please update this PR? |
|
Or I can open a separate PR with your 4 commits. |
|
Thank you for review. I made a clean rebase. This should be OK now |
|
Thanks! One thing I can see right away is the lack of documentation. Could you please extend |
|
The documentation has been added |
|
@Nikratio Could you merge it for the next release? |
|
I will try. As an aside, when merging I will squash this into one commit (the current split into 5 commits doesn't seem to make sense), so in order to preserve correct attributions it would be nice to have the pull request just contain one commit too. But that's not a showstopper. |
|
@Backblaze @bwbeach May blog about it. |
There was a problem hiding this comment.
Alright, finally had the time to look at this. Apart from the issue with the expensive copy() operation I don't see any fundamential problems, but a lot of smaller things that need to be addressed. Do you think you would have time for that? Thanks a lot for your contribution!
| `BackBlaze B2 <https://www.backblaze.com/b2/cloud-storage.html>`_ is a low cost | ||
| storage provider, with their own API. To use this storage | ||
| you first need to sign up for an account. The account is free, you only pay for the | ||
| storage and the traffic you actually use. After account creation, you |
There was a problem hiding this comment.
Let's get rid of the "The account is free part". It sounds like advertisement, and may change in the future.
(There may be a similar paragraph for another backend, in that case feel free to remove that too)
| storage provider, with their own API. To use this storage | ||
| you first need to sign up for an account. The account is free, you only pay for the | ||
| storage and the traffic you actually use. After account creation, you | ||
| need to create a bucket to store data. |
There was a problem hiding this comment.
How is that done? Is there a command line tool, a web application, or do you need to somehow use the API directly?
| need to create a bucket to store data. | ||
| The storage URL for BackBlaze backend is :: | ||
|
|
||
| b2://<bucket>/[</prefix>] |
There was a problem hiding this comment.
Are you sure the the trailing slash is required if no prefix is specified? If so, is requiring that really a good idea?
|
|
||
| b2://<bucket>/[</prefix>] | ||
|
|
||
| Here *bucket* correspond to as existing bucket name, and *prefix* |
| b2://<bucket>/[</prefix>] | ||
|
|
||
| Here *bucket* correspond to as existing bucket name, and *prefix* | ||
| a folder where all s3ql files will be stored. |
There was a problem hiding this comment.
The b2 documentation says that there is no such thing as a folder in b2.
| if is_temp_network_error(exc): | ||
| # We probably can't use the connection anymore | ||
| self.conn.disconnect() | ||
| raise |
There was a problem hiding this comment.
Is this necessary? If yes, shouldn't the other backends be changed accordingly?
| self.conn_download.disconnect() | ||
|
|
||
|
|
||
| class ObjectR(object): |
There was a problem hiding this comment.
There seems to be a lot of code duplication from s3.ObjectR here. Can you just inherit from s3.ObjectR and change the base class so that the hash function and response header name can be easily customized?
| headers=headers, | ||
| body=self.fh, | ||
| auth_token=upload_auth_token, | ||
| body_size=self.obj_size) |
| upload_auth_token, upload_url = self.backend._get_upload_url() | ||
| upload_url = urllib.parse.urlparse(upload_url) | ||
|
|
||
| with HTTPConnection(upload_url.hostname, 443, ssl_context=self.backend.ssl_context) as conn_up: |
There was a problem hiding this comment.
This means every upload will require creating a fresh connection (maybe even SSL). Is there a way to re-use the connection object? Hopefully the hostname of the upload url will not change every time..
| if hit: | ||
| val = int(header) | ||
| else: | ||
| val = 1 |
There was a problem hiding this comment.
try:
val = int(header)
except ValueError:
int = 1
would be a lot clearer. However, just like s3c you should log an error if there's an invalid value returned by the server.
|
@sylvainlehmann Do you have the time to answer all these? |
|
@szepeviktor I do not have time to work on it theses days. I think i will start sending patches on the begining of december |
|
Thanks. |
|
@sylvainlehmann Could I help in any way? |
|
@sylvainlehmann Could you answer some of these questions? |
|
@sylvainlehmann @Nikratio I've contacted Backblaze several times to donate developer hours to this integration. |
|
Friendly ping. Do you think you'll have time to resolve the open issues in the near future? If not, I'll close this pull request for now. |
|
@Nikratio Hi. Sorry for the response delay. I have a very few time to work on it and does not use s3ql anymore. Feel free to close the issue, I will come back to you if i get some free time in the next months |
Hello,
Here's a prototype implementation of BackBlaze B2 Backend. It may need some extended tests, but respond correctly to backend's test feature. There is still some things left to do :