feat: containerized ossfs auth enhanced #927

AlbeeSo · 2023-12-14T14:13:14Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

support RRSA & csi-secret-store authType for containerized ossfs.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

additional rbac: serviceaccount/csi-fuse-ossfs get

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

huww98

Is it possible to use OIDC in an non-ACK cluster? It is Open ID connect anyway.

pkg/mounter/ossfs.go

pkg/mounter/fuse_containerized_mounter.go

pkg/oss/controllerserver.go

pkg/oss/nodeserver.go

main.go

pkg/mounter/fuse_containerized_mounter.go

AlbeeSo · 2023-12-15T09:14:17Z

Is it possible to use OIDC in an non-ACK cluster? It is Open ID connect anyway.

IMO, it is too customized and will bring extra params. if users have no strong demand on RRSA, why not use IBMIAM auth with ossfs mount options directly instead?

huww98 · 2023-12-15T09:24:18Z

will bring extra params

I think we can just replace the clusterId parameter with full PROVIDER_ARN. We can still build this ARN for ACK user at installation.

AlbeeSo · 2024-01-19T04:04:03Z

will bring extra params

I think we can just replace the clusterId parameter with full PROVIDER_ARN. We can still build this ARN for ACK user at installation.

current version:

get aliuid from env first, retryGetMetadata if missing
get clusterid from env first, get cluster profile configmap if missing
get provider name from env first, used "ack-rrsa-clusterid" if missing

AlbeeSo · 2024-01-19T05:38:45Z

/retest-required

pkg/mounter/fuse_containerized_mounter.go

pkg/mounter/helper.go

pkg/mounter/ossfs.go

pkg/cloud/metadata/ecs.go

pkg/cloud/metadata/env.go

huww98 · 2024-02-22T06:36:46Z

pkg/mounter/helper.go

+
+// GetOIDCProvider get OIDC provider from env or ACK clusterId for RRSA
+func GetOIDCProvider() (provider string, err error) {
+	provider = os.Getenv("OIDC_PROVIDER")


I still would like this one to override the full ARN, and can be called "ALIBABA_CLOUD_OIDC_PROVIDER_ARN" for consistency. Can't a user potentially use a provider from others account?

And should this be a parameter in storage class? Maybe we can support multiple providers for different PV, or even different CSI drivers. Or CSI itself may gain OIDC capability in the future. Put it here can be ambiguous.

If role_arn or provider_arn does not conform to this fixed format, it means that the user is trying to access the OSS from other cloud vendor through RRSA. But ossfs itself does not have the ability to load external authentication libraries (for example, awscred-lib for IRSA). CSI can accept a full arn but it will still not work in ossfs.

And for multiple providers, I think supporting multiple roles is enough. @mowangdk PTAL

I mean the user may also want to override the account ID part of ARN. Consider that user A may want to access an OSS bucket from account B. User B then adds the ACK cluster in account A to his OIDC provider, then user A will need to specify the provider from account B.

sts will return error like "acs::ram::111111:oidc-provider/aaaa must be your AccountID, your AccountID is 2222"

I've tested that I can register ACK cluster in account A to OIDC provider in account B. Then everything else should only relate to account B. Which step (which OpenAPI) would fail?

could you please give a detailed test case and the AssumeRoleWithOIDC related logs of ossfs?

I don't know much about ossfs. But I've tried my proposed flow, it works and I got token from AssumeRoleWithOIDC.

Login account A, create an ACK cluster, enable RRSA, copy the provider URL

Create a Pod in above ACK cluster, mount this volume, and copy the token in it.
- name: rrsa-oidc-token projected: defaultMode: 0600 sources: - serviceAccountToken: audience: sts.aliyuncs.com expirationSeconds: 3600 path: token

Login account B, go to https://ram.console.aliyun.com/providers/create/OIDC and register an OIDC provider with URL from step 1, copy the provider ARN

create a RAM role for OIDC, copy the role ARN

go to https://next.api.aliyun.com/api/Sts/2015-04-01/AssumeRoleWithOIDC, fill in token from step 2, and ARNs from step 3 and 4, I get RAM sts token.

These steps should naturally apply to ossfs.

pkg/cloud/metadata/metadata.go

huww98

Overvall it is OK.

Although I reserve my opinion about the RoleARN/Name. I think API should be minimal. To use only RoleARN and not RoleName as parameter:

pros:

reduce the possibility of invalid input. simpler implementation, less surprise
fewer things for user to understand. Especially for those who use both default and non-default provider.
closer to our OIDC OpenAPI

cons:

harder for users who just want to use OSS without AK. They should find and copy the ARN from the web console, instead of just typing in the role name.
storage class yaml is a little harder to read if one don't know about role ARN.

mowangdk · 2024-03-06T08:02:18Z

/lgtm
/approve

mowangdk · 2024-03-06T08:06:32Z

/approve

k8s-ci-robot · 2024-03-06T08:07:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AlbeeSo, huww98, mowangdk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mowangdk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 14, 2023

k8s-ci-robot requested a review from mowangdk December 14, 2023 14:13

AlbeeSo force-pushed the feat/auth-enhanced branch from 40f2bf3 to 2537cf4 Compare December 15, 2023 02:16

huww98 suggested changes Dec 15, 2023

View reviewed changes

tydra-wang reviewed Dec 15, 2023

View reviewed changes

pkg/mounter/fuse_containerized_mounter.go Outdated Show resolved Hide resolved

AlbeeSo force-pushed the feat/auth-enhanced branch from afd58be to 62b7f22 Compare December 15, 2023 09:03

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 20, 2023

AlbeeSo marked this pull request as draft December 22, 2023 08:31

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 22, 2023

AlbeeSo force-pushed the feat/auth-enhanced branch from 62b7f22 to c71b526 Compare December 22, 2023 08:45

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 22, 2023

AlbeeSo force-pushed the feat/auth-enhanced branch from c71b526 to f259dde Compare December 22, 2023 09:11

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 11, 2024

AlbeeSo force-pushed the feat/auth-enhanced branch from f259dde to 1d1f7e8 Compare January 16, 2024 14:29

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 16, 2024

AlbeeSo force-pushed the feat/auth-enhanced branch from 1d1f7e8 to 4af1b0e Compare January 18, 2024 13:30

AlbeeSo marked this pull request as ready for review January 19, 2024 04:04

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 19, 2024

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 20, 2024

AlbeeSo force-pushed the feat/auth-enhanced branch from d817e61 to 77b1b8a Compare February 22, 2024 03:10

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 22, 2024

mowangdk reviewed Feb 22, 2024

View reviewed changes

pkg/mounter/fuse_containerized_mounter.go Outdated Show resolved Hide resolved

pkg/mounter/helper.go Outdated Show resolved Hide resolved

pkg/mounter/ossfs.go Outdated Show resolved Hide resolved

huww98 suggested changes Feb 22, 2024

View reviewed changes

huww98 reviewed Feb 22, 2024

View reviewed changes

pkg/cloud/metadata/metadata.go Outdated Show resolved Hide resolved

huww98 approved these changes Feb 26, 2024

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 4, 2024

AlbeeSo added 19 commits March 5, 2024 10:31

feat: support rrsa and csi_secret_store

1f2daee

csi-ossfs version

42b1b97

typo

585a535

do not create SA for RRSA if not found

8ef149c

perf: support get rrsa config from ENV

69d0e7e

mode from 420 to 0600

4b2aa48

reset GetRAMRoleOption func

254562b

get aliuid by metadata

2ee1080

rm const var

6dbc6e2

gofmt

f937219

fix test

008c8b8

format

97392f5

get uid from stsclient

bd1f099

move clusterid to metadata

6f356d6

format k8s metadata

dc8850b

allow create stsclient with token

4905685

add roleArn and oidcProviderArn parms in PV

90c16ff

support serviceaccountname param

5715aae

typos

2db75ec

AlbeeSo force-pushed the feat/auth-enhanced branch from 20ef114 to 2db75ec Compare March 5, 2024 02:50

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 5, 2024

k8s-ci-robot assigned mowangdk Mar 6, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 6, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 6, 2024

k8s-ci-robot merged commit 6479e99 into kubernetes-sigs:master Mar 6, 2024
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: containerized ossfs auth enhanced #927

feat: containerized ossfs auth enhanced #927

AlbeeSo commented Dec 14, 2023 •

edited

huww98 left a comment

AlbeeSo commented Dec 15, 2023

huww98 commented Dec 15, 2023

AlbeeSo commented Jan 19, 2024

AlbeeSo commented Jan 19, 2024

huww98 Feb 22, 2024

AlbeeSo Feb 22, 2024 •

edited

huww98 Feb 23, 2024

AlbeeSo Feb 23, 2024 •

edited

huww98 Feb 23, 2024

AlbeeSo Feb 23, 2024

huww98 Feb 23, 2024

huww98 left a comment •

edited

mowangdk commented Mar 6, 2024 •

edited

mowangdk commented Mar 6, 2024

k8s-ci-robot commented Mar 6, 2024

feat: containerized ossfs auth enhanced #927

feat: containerized ossfs auth enhanced #927

Conversation

AlbeeSo commented Dec 14, 2023 • edited

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

huww98 left a comment

Choose a reason for hiding this comment

AlbeeSo commented Dec 15, 2023

huww98 commented Dec 15, 2023

AlbeeSo commented Jan 19, 2024

AlbeeSo commented Jan 19, 2024

huww98 Feb 22, 2024

Choose a reason for hiding this comment

AlbeeSo Feb 22, 2024 • edited

Choose a reason for hiding this comment

huww98 Feb 23, 2024

Choose a reason for hiding this comment

AlbeeSo Feb 23, 2024 • edited

Choose a reason for hiding this comment

huww98 Feb 23, 2024

Choose a reason for hiding this comment

AlbeeSo Feb 23, 2024

Choose a reason for hiding this comment

huww98 Feb 23, 2024

Choose a reason for hiding this comment

huww98 left a comment • edited

Choose a reason for hiding this comment

mowangdk commented Mar 6, 2024 • edited

mowangdk commented Mar 6, 2024

k8s-ci-robot commented Mar 6, 2024

AlbeeSo commented Dec 14, 2023 •

edited

AlbeeSo Feb 22, 2024 •

edited

AlbeeSo Feb 23, 2024 •

edited

huww98 left a comment •

edited

mowangdk commented Mar 6, 2024 •

edited