Docs: show minimal permissions required for working with pre-signed URLs.

[ozkatz]

Currently the lakeFS docs show instructions on how to use lakeFS with minimal IAM permissions on AWS.

With the introduction of pre-signed URLs that also work from the client, we can also add a pre-signed mode: allow the lakeFS IAM role to Put/Get objects, but only from specific IP addresses or VPCs.

More info: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html

Relevant keys:

• aws:referer
• aws:SourceVpc
• aws:SourceIp

Perhaps there are more.

The definition of done should be a recommended IAM role that allows lakeFSFS to work with full functionality, and the CLI/UI with full functionality, without allowing a lakeFS instance on another AWS account (think lakeFS Cloud..) to access the data directly - but only to sign the URLs.

[Isan-Rivkin]

Deferring the D.O.D here to add documentation because currently its impossible to completely achieve block from Get/Put for lakeFS.
There are several bugs / features we need to figure out before doing that.

• Commits / Merge needs access to _lakeFS - Scope IAM Put/Get to */_lakefs/*
• Import: Needs access to _lakefs directory - Scope IAM Put/Get to */_lakefs/*
• UI: DuckDB doesn't work with pre-sign. - Out of scope for now (?)
• Create Repo: dummy file in the storage namespace root. IAM minimal permissions: dummy file dependency fails creating repository #6471
• UI: Get /Download Object [Bug]: UI ignores pre-sign URL configuration for GetObject requests  #6393
• UI: Markdown rendering [Bug]: UI Markdown Renderer not using presign URL to get images  #6412
• UI: [Bug]: UI not using presign for uploading objects [Bug]: UI not using presign for uploading objects #8158
• Sample Data when creating repository - will not work needs access to /data dir - solved by Isolate storage location (directory) for lakeFS Action files  #6472
• Actions: Needs access to /data dir to read the contents. - solved by Isolate storage location (directory) for lakeFS Action files  #6472
• UI: Bug when showing object diff in uncommitted changes - [Bug]: UI not using pre-sign when showing Uncommitted object changes #7693
• No managed GC
• S3 Gateway not supported

We will prioritize the following issues to achieve no get/post for lakeFS permissions.

(CC: @itaiad200 )

[itaiad200]

• Create Repo: dummy file in the storage namespace root.

Blocker: users can't create repositories.

• UI: Get /Download Object bug

Fixed with #6462?

• UI: Markdown rendering bug

Not a blocker but we should prioritize.

• UI: DuckDB doesn't work with pre-sign.

Tough one, I don't think it's a blocker but I could be convinced otherwise.

• Sample Data when creating repository - will not work needs access to /data dir.

Blocker - It's a UI thing, right? Can't we use presign for it?

• Actions: Needs access to /data dir to read the contents.

Blocker - Ouch, that's a pain. We need a smart, resilient workaround for this.

@ozkatz do you concur?

[Isan-Rivkin]

@itaiad200

Fixed with #6462?

nope, that's not related.

Tough one, I don't think it's a blocker but I could be convinced otherwise

Agreed- in terms of R.O.I. It seems to me that the root issue is that DuckDB does not support working with pre-sign URL's so its really one as you said., @ozkatz please correct me if Im wrong.

Blocker - It's a UI thing, right? Can't we use presign for it?

Nope, the feature is indeed a UI thing but, the root cause of it comes from Actions: Needs access to /data dir to read the contents.

And that's an issue indeed sure right now we use streaming approach to get/put those objects and ignoring pre-sign.
But, even if we use pre-sign lakeFS server will not be able to execute hooks before commits because the server itself is restricted by source vpc / account ID / IP for get requests.

Everything else - 💯

[Isan-Rivkin]

@ozkatz @itaiad200
Updated my original summary comment
List of all the related issues can be found here