Rethinking object store configuration

[nyurik]

Many project that use arrow-rs-object-store need to let their users to configure it. The number configuration options for each backend is fairly large, and most developers would not care to learn all of them, nor have a way to test them. I created apache/arrow-rs-object-store#507 to highlight the problem, but it would be good to figure out how Martin wants to approach this too, and how we can reuse it more generally than just for Martin.

[CommanderStorm]

I don't think that is currently a lot of code.

The code currently works quite well and most of what is there is just working around having had a previous implementation in place which needed migrating.

[nyurik]

we don't need to - this is a typical usage pattern for bucket-using k8s environment - where k8s provider automatically supplies authentication secrets to the AWS or other libraries, bypassing what Martin would handle explicitly. The other method, where the secret is given to Martin as a config or a Martin-handled env var, is considered deprecated by AWS and possibly others

[CommanderStorm]

Could you point me towards some docs how to do this "the good way"?

I never had professional experience with AWS, the closest I came was MinIO.

[nyurik]

There is this doc, but its a bit lengthy. In k8s, usually you would simply set multiple env vars including AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token (see docs) -- and that file will be dynamically supplied by the k8s's AWS EKS service. So when you run a pod, k8s will know which secret to give it, and your app "just works" - as it would have all the permissions needed to access all of AWS services including the S3 bucket.

[CommanderStorm]

Yea, that is a lot.
In this case, I don't think my original intention to "just document the good way" will work.

Seems to need more expert knowlege than I can quickly aquire+condense down.

[nyurik]

heh, that's normal - every domain knowledge is ... varied in size by the domain, and security domain is not small by any measure :)

[CommanderStorm]

The bigger issue that I see is maintaining our docs on this feature.
I primarily did not want document this due to this expectable churn.

[CommanderStorm]

Adding env vars, fine.

Removing all of that docs work (this was a lot of work, I don't want to have to redo it later) is not something I am super fond of.
Code wise, it should be fairly simple to support both env vars and normal vars.

[nyurik]

WOW, I forgot that we already have all this awesome docs! No, it's perfect, well done! And much more readable than searching through docs.rs. We should give a link to the docs.rs or similar though, as we are not the authoritative source.

[nyurik]

I mean -- add a link where we got that info, in case someone wants to read-up more about it. I am not certain what the best "source of truth" would be for each backend though.

[CommanderStorm]

problem is that the source for this is nessesarily to read the source code.
The docs.rs/object_store are less than ideal this way.

Maybe I can propose to improve this, but given how long below docs change has taken to date...

• docs: document missing config keys apache/arrow-rs-object-store#498

[nyurik]

So going back to the original thought - we are good to provide all these values, but we may also want to support env vars that are "standard" by each cloud provider, i.e. a set of env vars many folks are familiar with. I do not know which will be more popular though. Perhaps we should add use env var boolean to enable env var support, but possible to override it with the values from the config file.

Use case: secrets come from env var to allow access to buckets and possibly to PG(?), and the name of the bucket is given via config