client,providers: add storj DCS as provider option

Signed-off-by: Stefan Benten <mail@stefan-benten.de>

.github/workflows/test.yaml

@@ -69,7 +69,7 @@ jobs:
       - name: Run unit tests
         env:
           # THANOS_TEST_OBJSTORE_SKIP: AZURE,COS,ALIYUNOSS,BOS
-          THANOS_TEST_OBJSTORE_SKIP: GCS,S3,SWIFT,AZURE,COS,ALIYUNOSS,BOS,OCI
+          THANOS_TEST_OBJSTORE_SKIP: GCS,S3,SWIFT,AZURE,COS,ALIYUNOSS,BOS,OCI,STORJ
           # Variables for Swift testing.
           OS_AUTH_URL: http://127.0.0.1:5000/v2.0
           OS_PASSWORD: s3cr3t

CHANGELOG.md

@@ -13,6 +13,7 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
 ### Fixed
 
 ### Added
+- [#17](https://github.com/thanos-io/objstore/pull/17) Add Storj DCS Object Storage Bucket support.
 - [#15](https://github.com/thanos-io/objstore/pull/15) Add Oracle Cloud Infrastructure Object Storage Bucket support.
 
 ### Changed

Makefile

@@ -5,7 +5,7 @@ MDOX_VALIDATE_CONFIG ?= .mdox.validate.yaml
 
 .PHONY: test-local
 test-local:
-	THANOS_TEST_OBJSTORE_SKIP=GCS,S3,AZURE,SWIFT,COS,ALIYUNOSS,BOS,OCI $(MAKE) test
+	THANOS_TEST_OBJSTORE_SKIP=GCS,S3,AZURE,SWIFT,COS,ALIYUNOSS,BOS,OCI,STORJ $(MAKE) test
 
 .PHONY: test
 test:

README.md

@@ -128,6 +128,7 @@ Current object storage client implementations:
 | [Baidu BOS](#baidu-bos)                                                                   | Beta               | Production Usage      | no                | ??                               |
 | [Local Filesystem](#filesystem)                                                           | Stable             | Testing and Demo only | yes               | @bwplotka                        |
 | [Oracle Cloud Infrastructure Object Storage](#oracle-cloud-infrastructure-object-storage) | Beta               | Production Usage      | yes               | @aarontams,@gaurav-05,@ericrrath |
+| [Storj DCS](#storj-dcs)                                                                   | Beta               | Production Usage      | yes               | @stefanbenten                    |
 
 **Missing support to some object storage?** Check out [how to add your client section](#how-to-add-a-new-client-to-thanos)
 
@@ -137,7 +138,7 @@ NOTE: Currently Thanos requires strong consistency (write-read) for object store
 
 Thanos uses the [minio client](https://github.com/minio/minio-go) library to upload Prometheus data into AWS S3.
 
-> NOTE: S3 client was designed for AWS S3, but it can be configured against other S3-compatible object storages e.g Ceph
+> NOTE: S3 client was designed for AWS S3, but it can be configured against other S3-compatible object storages e.g. Ceph.
 
 The S# object storage yaml configuration definition:
 
@@ -185,13 +186,13 @@ prefix: ""
 
 At a minimum, you will need to provide a value for the `bucket`, `endpoint`, `access_key`, and `secret_key` keys. The rest of the keys are optional.
 
-However if you set `aws_sdk_auth: true` Thanos will use the default authentication methods of the AWS SDK for go based on [known environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) (`AWS_PROFILE`, `AWS_WEB_IDENTITY_TOKEN_FILE` ... etc) and known AWS config files (~/.aws/config). If you turn this on, then the `bucket` and `endpoint` are the required config keys.
+However, if you set `aws_sdk_auth: true` Thanos will use the default authentication methods of the AWS SDK for go based on [known environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) (`AWS_PROFILE`, `AWS_WEB_IDENTITY_TOKEN_FILE` ... etc) and known AWS config files (~/.aws/config). If you turn this on, then the `bucket` and `endpoint` are the required config keys.
 
 The field `prefix` can be used to transparently use prefixes in your S3 bucket. This allows you to separate blocks coming from different sources into paths with different prefixes, making it easier to understand what's going on (i.e. you don't have to use Thanos tooling to know from where which blocks came).
 
 The AWS region to endpoint mapping can be found in this [link](https://docs.aws.amazon.com/general/latest/gr/s3.html).
 
-Make sure you use a correct signature version. Currently AWS requires signature v4, so it needs `signature_version2: false`. If you don't specify it, you will get an `Access Denied` error. On the other hand, several S3 compatible APIs use `signature_version2: true`.
+Make sure you use a correct signature version. Currently, AWS requires signature v4, so it needs `signature_version2: false`. If you don't specify it, you will get an `Access Denied` error. On the other hand, several S3 compatible APIs use `signature_version2: true`.
 
 You can configure the timeout settings for the HTTP client by setting the `http_config.idle_conn_timeout` and `http_config.response_header_timeout` keys. As a rule of thumb, if you are seeing errors like `timeout awaiting response headers` in your logs, you may want to increase the value of `http_config.response_header_timeout`.
 
@@ -247,7 +248,7 @@ You will also need to apply the following AWS IAM policy for the user to access
 
 ###### Credentials
 
-By default Thanos will try to retrieve credentials from the following sources:
+By default, Thanos will try to retrieve credentials from the following sources:
 
 1. From config file if BOTH `access_key` and `secret_key` are present.
 2. From the standard AWS environment variable - `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
@@ -329,7 +330,7 @@ Details about AWS policies: https://docs.aws.amazon.com/AmazonS3/latest/dev/usin
 
 ###### STS Endpoint
 
-If you want to use IAM credential retrieved from an instance profile, Thanos needs to authenticate through AWS STS. For this purposes you can specify your own STS Endpoint.
+If you want to use IAM credential retrieved from an instance profile, Thanos needs to authenticate through AWS STS. For this purpose you can specify your own STS Endpoint.
 
 By default Thanos will use endpoint: https://sts.amazonaws.com and AWS region corresponding endpoints.
 
@@ -637,6 +638,21 @@ config:
 
 You can also include any of the optional configuration just like the example in `Default Provider`.
 
+### Storj DCS
+
+In order to be able to configure [Storj DCS](https://storj.io/signup) as Thanos Object Store Backend you will need to 
+provide an access grant and a bucket name to be used to hold the objects. 
+You can choose an already existing bucket or a new name and the bucket will be created for you.
+
+The following snippet shows how to add these information to the configuration yaml.
+
+```yaml
+type: STORJ
+config:
+  access: ""
+  bucket: ""
+```
+
 #### How to add a new client to Thanos?
 
 Following checklist allows adding new Go code client to supported providers:

client/factory.go

@@ -23,6 +23,7 @@ import (
 	"github.com/thanos-io/objstore/providers/oci"
 	"github.com/thanos-io/objstore/providers/oss"
 	"github.com/thanos-io/objstore/providers/s3"
+	"github.com/thanos-io/objstore/providers/storj"
 	"github.com/thanos-io/objstore/providers/swift"
 )
 
@@ -38,6 +39,7 @@ const (
 	ALIYUNOSS  ObjProvider = "ALIYUNOSS"
 	BOS        ObjProvider = "BOS"
 	OCI        ObjProvider = "OCI"
+	STORJ      ObjProvider = "STORJ"
 )
 
 type BucketConfig struct {
@@ -80,6 +82,8 @@ func NewBucket(logger log.Logger, confContentYaml []byte, reg prometheus.Registe
 		bucket, err = bos.NewBucket(logger, config, component)
 	case string(OCI):
 		bucket, err = oci.NewBucket(logger, config)
+	case string(STORJ):
+		bucket, err = storj.NewBucket(logger, config, component)
 	default:
 		return nil, errors.Errorf("bucket with type %s is not supported", bucketConf.Type)
 	}

go.mod

@@ -31,6 +31,8 @@ require (
 	google.golang.org/api v0.80.0
 	gopkg.in/alecthomas/kingpin.v2 v2.2.6
 	gopkg.in/yaml.v2 v2.4.0
+	storj.io/common v0.0.0-20220802175255-aae0c09ec9d4
+	storj.io/uplink v1.9.0
 )
 
 require (
@@ -56,13 +58,15 @@ require (
 	github.com/aws/smithy-go v1.11.1 // indirect
 	github.com/baiyubin/aliyun-sts-go-sdk v0.0.0-20180326062324-cfa1a18b161f // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/calebcase/tmpfile v1.0.3 // indirect
 	github.com/cespare/xxhash/v2 v2.1.2 // indirect
 	github.com/clbanning/mxj v1.8.4 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/dimchansky/utfbom v1.1.1 // indirect
 	github.com/dustin/go-humanize v1.0.0 // indirect
 	github.com/go-logfmt/logfmt v0.5.1 // indirect
 	github.com/gofrs/flock v0.8.1 // indirect
+	github.com/gogo/protobuf v1.3.2 // indirect
 	github.com/golang-jwt/jwt/v4 v4.2.0 // indirect
 	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
 	github.com/golang/protobuf v1.5.2 // indirect
@@ -74,6 +78,7 @@ require (
 	github.com/jtolds/gls v4.20.0+incompatible // indirect
 	github.com/klauspost/compress v1.13.5 // indirect
 	github.com/klauspost/cpuid v1.3.1 // indirect
+	github.com/klauspost/cpuid/v2 v2.0.12 // indirect
 	github.com/mattn/go-ieproxy v0.0.1 // indirect
 	github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
 	github.com/minio/md5-simd v1.1.0 // indirect
@@ -91,17 +96,22 @@ require (
 	github.com/sirupsen/logrus v1.8.1 // indirect
 	github.com/smartystreets/assertions v1.2.1 // indirect
 	github.com/sony/gobreaker v0.5.0 // indirect
+	github.com/spacemonkeygo/monkit/v3 v3.0.18 // indirect
+	github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3 // indirect
+	github.com/zeebo/blake3 v0.2.3 // indirect
+	github.com/zeebo/errs v1.3.0 // indirect
 	go.opencensus.io v0.23.0 // indirect
 	go.uber.org/goleak v1.1.12 // indirect
-	golang.org/x/crypto v0.0.0-20220214200702-86341886e292 // indirect
-	golang.org/x/net v0.0.0-20220520000938-2e3eb7b945c2 // indirect
+	golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e // indirect
+	golang.org/x/net v0.0.0-20220624214902-1bab6f366d9e // indirect
 	golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a // indirect
 	golang.org/x/text v0.3.7 // indirect
 	golang.org/x/time v0.0.0-20220224211638-0e9765cccd65 // indirect
-	golang.org/x/xerrors v0.0.0-20220411194840-2f41105eb62f // indirect
+	golang.org/x/xerrors v0.0.0-20220517211312-f3a8303e98df // indirect
 	google.golang.org/appengine v1.6.7 // indirect
 	google.golang.org/genproto v0.0.0-20220524023933-508584e28198 // indirect
 	google.golang.org/grpc v1.46.2 // indirect
 	google.golang.org/protobuf v1.28.0 // indirect
 	gopkg.in/ini.v1 v1.57.0 // indirect
+	storj.io/drpc v0.0.32 // indirect
 )