Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for sub-folders when mounting S3 buckets #263

Merged
merged 13 commits into from
Jan 22, 2024

Conversation

srikumar003
Copy link
Collaborator

Enables datasets to be defined on subfolders in S3 buckets. A dataset such as:

apiVersion: com.ie.ibm.hpsys/v1alpha1
kind: Dataset
metadata:
name: test
spec:
   local:
   type: "COS"
   secret-name: "secret-name"
   endpoint: "http://s3.eu.cloud-object-storage.appdomain.cloud"
   bucket: "bucket/folder"

should have only the contents of bucket/folder available under /mnt/datasets/test when mounted into a pod.

However, if the dataset is used as a configmap this will not prevent the pod from accessing the contents of the bucket.

Signed-off-by: SRIKUMAR VENUGOPAL <srikumarv@ie.ibm.com>
@srikumar003 srikumar003 linked an issue Jun 25, 2023 that may be closed by this pull request
@vytautaskelmelissislt
Copy link

@Trainbow
Copy link

Trainbow commented Oct 15, 2023 via email

Signed-off-by: SRIKUMAR VENUGOPAL <srikumarv@ie.ibm.com>
Signed-off-by: SRIKUMAR VENUGOPAL <srikumarv@ie.ibm.com>
Signed-off-by: SRIKUMAR VENUGOPAL <srikumarv@ie.ibm.com>
Signed-off-by: SRIKUMAR VENUGOPAL <srikumarv@ie.ibm.com>
Signed-off-by: SRIKUMAR VENUGOPAL <srikumarv@ie.ibm.com>
@srikumar003 srikumar003 marked this pull request as ready for review January 22, 2024 00:00
Signed-off-by: SRIKUMAR VENUGOPAL <srikumarv@ie.ibm.com>
Signed-off-by: SRIKUMAR VENUGOPAL <srikumarv@ie.ibm.com>
Signed-off-by: SRIKUMAR VENUGOPAL <srikumarv@ie.ibm.com>
Signed-off-by: SRIKUMAR VENUGOPAL <srikumarv@ie.ibm.com>
@srikumar003 srikumar003 merged commit 24062e9 into datashim-io:master Jan 22, 2024
2 checks passed
@srikumar003 srikumar003 deleted the 200_support_s3_folders branch January 22, 2024 11:37
@Trainbow
Copy link

Trainbow commented Jan 22, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

mounting a folder in s3
3 participants