[chore] use kv.KVStore also for S3 storage

[NyaaaWhatsUpDoc]

This should solve #1105

It replaces our current S3 storage driver with kv.KVStore using storage.S3 under the hood.

In time we can move to remove the now fairly-redundant storage.Driver interface.

Shoutout to @theSuess who was also working on this at the same time 😀

[NyaaaWhatsUpDoc]

Thank you @theSuess for the go-store fix!

[theSuess]

Got it working on my machine! One issue in this change needs to be addressed and I've sent you a second PR for go-store which fixes another issue

[theSuess]

I've rolled this out on my test instance and while it still allocates more memory than the uploaded file, it's less than previously.

[tsmethurst]

Good job nerds, will review now. Is this ready to merge if it lgtm?

[NyaaaWhatsUpDoc]

I've rolled this out on my test instance and while it still allocates more memory than the uploaded file, it's less than previously.

You see from the go-store code it only allocates that (5MiB as we have it set) write buffer once, and reuses it for each chunk. I wonder if this means minio is still heavy on memory allocation even when chunking 🤔

[NyaaaWhatsUpDoc]

Good job nerds, will review now. Is this ready to merge if it lgtm?

yep go for it!

[matthewp]

Should #1105 have been closed because of this?