Switching a large blob store to Storj is problematic

[rjkroege]

I attempted to switch to Storj (Globally Distributed Cloud Object Storage) as a blobstore. This doesn't currently work very well. Two observations about this:

• kopia repository sync-to rclone seems to work but is very slow
• the Storj S3 compatibility interface works just fine for small blobstores but is not adequate for production use because the Storj S3 gateway limits s3_storage.go:/ListBlobs() to a thousand blobs (per documentation) total if using prefix matching.

I see two solutions:

• run a local Storj S3 gateway with much larger limits
• write a "native" Storj blob implementation in Kopia.

I would expect that the "native" implementation (i.e. Storj blob implementation) is preferable for its performance and robustness.

@jkowalski et al: are you open to PRs implementing such a feature? If so, I will attempt to write this.