using cassandra backup with a custom s3 endpoint

[puneetloya]

Does the cassandra backup support custom s3 endpoints?
For custom s3 providers like minio, will the AWS module for cassandra-backup work? it helps in validating backups in house without needing to use AWS s3.

[cscetbon]

It doesn't seem to be the case even if there is a key called endpoint in the secrets that can be set

apiVersion: v1
data:
  awsaccesskeyid: bWluaW8=
  awsendpoint: ....
  awsregion: dXMtZWFzdC0x
  awssecretaccesskey: bWluaW8xMjM=
kind: Secret
....

I can interact with that endpoint from a pod:

/ # export AWS_ACCESS_KEY_ID=minio AWS_SECRET_ACCESS_KEY=minio123 AWS_DEFAULT_REGION=us-east-1
/ # aws --endpoint-url http://minio.velero.svc:9000 s3 ls s3://cassandra-bucket/

However when trying to backup I get an error saying com.amazonaws.SdkClientException: Unable to execute HTTP request: cassandra-bucket.minio.velero.svc. Here is the complete log https://pastebin.com/raw/K21qb7gv. @smiklosovic you probably can take a look ?

[smiklosovic]

hi @puneetloya and @cscetbon

yes, we do support endpoints

https://github.com/instaclustr/cassandra-backup/blob/master/src/main/java/com/instaclustr/cassandra/backup/aws/S3Module.java#L67-L77

Based on what you are using this from, configuration is either fetched from Kubernetes config map or from environment variable:

https://github.com/instaclustr/cassandra-backup/blob/master/src/main/java/com/instaclustr/cassandra/backup/aws/S3Module.java#L127-L186

The error you got has nothing to do with this backup tool, see this:

Caused by: java.net.UnknownHostException: cassandra-bucket.minio.velero.svc
	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
	at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
	at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpCl

This seems like your container is not able to resolve IP of cassandra-bucket.minio.velero.svc, that is a different problem from supporting endpoint. You see from the stacktrace that it is AWS DNS resolver logic which is resolving this hostname stuff, not backup tool.

I would triple check that you can e.g. ping that service from inside your container / pod.

Your hostname has suffix svc and that is quite strong indication that it is a hostname of Kubernetes service but AWS SDK cant resolve it.

These people are getting same error like you: aws/aws-sdk-java#1273

[cscetbon]

I was able to reproduce the error. It comes from the fact that aws-sdk-java uses virtual hosts instead of path-style requests when querying s3 and minio (what I use on my cluster for s3) is not configured to support it by default. I can confirm that if I use path-style requests it works as expected. I also found that AWS S3 will no longer support path style requests after September 2020 https://forums.aws.amazon.com/ann.jspa?annID=6776 so better that I make the change on my end. thanks @smiklosovic