Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot connect to backblaze b2 S3 compatible storage #2356

Open
rogeliodh opened this issue Jul 16, 2020 · 13 comments
Open

Cannot connect to backblaze b2 S3 compatible storage #2356

rogeliodh opened this issue Jul 16, 2020 · 13 comments

Comments

@rogeliodh
Copy link

rogeliodh commented Jul 16, 2020

I'm trying to setup storage to use new B2's S3 compatible API but it is not working:

this is the releavant part of my seafile_storage_classes.json:

 {
     "storage_id": "b2_seafile_storage",
     "name": "B2 Seafile Storage",
     "is_default": true,
     "commits": {
         "backend": "s3",
         "bucket": "seafile-commit-objects",
         "key_id": "KEYID",
         "key": "KEY",
         "use_https": true,
         "host": "s3.us-west-001.backblazeb2.com",
         "aws_region": "us-west-001",
         "use_v4_signature": true
     },
     "fs": {
         "backend": "s3",
         "bucket": "seafile-fs-objects",
         "key_id": "KEYID",
         "key": "KEY",
         "use_https": true,
         "host": "s3.us-west-001.backblazeb2.com",
         "aws_region": "us-west-001",
         "use_v4_signature": true
     },
     "blocks": {
         "backend": "s3",
         "bucket": "seafile-block-objects",
         "key_id": "KEYID",
         "key": "KEY",
         "use_https": true,
         "host": "s3.us-west-001.backblazeb2.com",
         "aws_region": "us-west-001",
         "use_v4_signature": true
     }
 }]

but it fails at stratup:

[07/15/2020 08:10:19 AM] ../common/s3-client.c(940): S3 error status for HEAD: 403.
[07/15/2020 08:10:19 AM] ../common/s3-client.c(720): S3 error status for PUT: 400.
[07/15/2020 08:10:19 AM] ../common/s3-client.c(721): Request URL: https://seafile-commit-objects.s3.us-west-001.backblazeb2.com/e96d6e9e-70e1-2ebb-89ce-619d9016331f/688ca5aef509e7dc160d8a928ff55b17da7bde55
[07/15/2020 08:10:19 AM] ../common/s3-client.c(722): Request headers:
[07/15/2020 08:10:19 AM] ../common/s3-client.c(621): Date: Wed, 15 Jul 2020 13:10:19 +0000
[07/15/2020 08:10:19 AM] ../common/s3-client.c(621): Authorization: AWS4-HMAC-SHA256 Credential=KEY/20200715/us-west-001/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256,Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
[07/15/2020 08:10:19 AM] ../common/s3-client.c(621): x-amz-content-sha256: 5024b8f164f1e1f84c59fbea0d7e062f7480f10728b1aea58baeaff25070a685
[07/15/2020 08:10:19 AM] ../common/obj-backend-s3.c(348): Put object 688ca5aef509e7dc160d8a928ff55b17da7bde55 error 400. Response:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>InvalidRequest</Code>
    <Message>Error reading uploaded data: StreamCorruptedException(null)</Message>
</Error>

My credentials are ok, as tested with another tools.

It seems I'm not the only one with the same problem: https://forum.seafile.com/t/couldnt-connect-to-server-backblaze-b2/

@JokerQyou
Copy link

I thinks it's related to how seafile calculates the signature. I also suspect that any S3 compatible storage with V4 signature would not work with seafile.

@pixelsquared
Copy link

As a workaround I am using minio as a S3 gateway between Backblaze S3 and Seafile.

@killing
Copy link
Member

killing commented Sep 16, 2020

Currently we don't have time to look into the issue.

@asifbacchus
Copy link

This has been an issue for a while and it doesn't help that BackBlaze doesn't quite do things right with eTags, etc. I use B2 with my Seafile setup also and have to go through minio as a gateway. Save yourself a TON of time and frustration and use minio version RELEASE.2020-12-16T05-05-17Z. It is the latest version that actually works with B2. Anything newer will throw numerous random signature and checksum errors due to issues with how B2 implements S3 protocols. Also, do not use the minio cache if you are using memcached with Seafile -- also numerous errors. If you're behind a reverse proxy like NGINX, you can use that to cache and it will save you a bunch of $$$.

If you need any help getting set up, give me a shout and I'll share my configs and do my best to help. When you get it working, it's pretty great, but it's not easy at first since nothing is well documented on the Seafile, B2 or minio side of things.

@cbess
Copy link

cbess commented Jun 1, 2021

For me, an issue like this was resolved after I deleted the app key I was using and generated, then used, new credentials.

@asifbacchus
Copy link

Just to clarify -- this is a genuine issue between minio and BackBlaze B2 caused by B2 using a not fully compatible implementation of the S3 protocols (I'm simplifying the issue). Anything involving clearing keys and the like is a coincidental fix. If you try to use a newer minio container you will get random errors about missing blocks, corruption, etc. I know there are several requests with minio and BackBlaze regarding this issue but neither party seems keen on fixing things. Actually, the fix really does need to come from BackBlaze...

TL;DR: use RELEASE.2020-12-16T05-05-17Z and your problems should be resolved. If not, there is something else going on.

@kgnfth
Copy link

kgnfth commented Jun 13, 2021

i resolved my issue by moving to Pydio Cells
i suggest you guys to do the same 😊

@JokerQyou
Copy link

Pydio cells stores file as-is on S3, so it only support file-level deduplication. Meanwhile Seafile's deduplication works at a block level. So not everything is equivalent here. Also pydio cells requires gRPC traffic pass through to use sync client, which might cause trouble for some reverse proxies.

@asifbacchus
Copy link

@kgnfth I'm not sure how you'd think that moving to an entirely different platform and tech-stack resolves the problem. While Pydio Cells might be an alternative solution to Seafile, it certainly does not resolve the issue the the OP posted about. These are NOT complementary solutions, Pydio would be a replacement for Seafile and there is no indication that the OP was asking for alternatives.

@kemege
Copy link

kemege commented Aug 25, 2021

@killing Is it possible to tell us what specific features of S3 are required for Seafile to work, so that we can choose a "proper" S3 provider before deploying?
It may take a long time for people to realize this problem is not due to misconfiguration, but incompatibility between services.

@agowa
Copy link

agowa commented Nov 26, 2021

any update on this?
What exactly is missing or wrong with BackBlaze?
Is the issue solely on BackBlazes side or is something broken in seafile?
Is there a timeline for a fix?

@asifbacchus
Copy link

@agowa338 I'm not an expert but, from my understanding, this is not limited to Seafile. It is a Minio-B2 interaction issue. Seems that B2 does not handle eTags properly (more accurately: not the same as S3). I tried asking Backblaze about this but haven't heard anything back after like 4 months. Apparently others have asked also and they haven't heard back in an even longer period of time.

For now, the best option is to use Minio release RELEASE.2020-12-16T05-05-17Z since it does not have the problematic eTag handling. Backblaze really needs to step-up and fix this issue or provide a viable workaround since it is not a trivial issue and that December 2020 is getting really old.

As far as I've heard, Seafile is not prioritizing any form of compatibility with B2 (Backblaze) which is really unfortunate. The last response I found from Seafile was that they support S3 and it's up to any other providers to properly implement S3 compatibility -- not a great response but actually pretty reasonable. So, again, we're stuck using Minio as 'middle-ware' and end up facing the eTag issue.

It's a little bit of a worrying situation, but, if it makes you feel any better I've been using that older Minio release version with B2 since around June of this year and have not had a single problem. I use NGINX both as a reverse-proxy and as a cache to replace the Minio gateway cache and it keeps my costs very low.

I'm still looking for a better workaround and will update this thread if it's still open if I find something. Hope this helps.

@kemege
Copy link

kemege commented Nov 28, 2021

FYI, I ended up using s3proxy as the gateway between Seafile and B2, which might be another workaround in addition to Minio.
I did not test the performance of this setup, but it seem to work well most of the time.
The only problem I ever encountered is that syncing a large file repository in Seafile sometimes result in a timeout error, which can be resolved if I initialize the sync multiple times.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants