Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SYNPY-1337] Adding in a comprehensive example for migration to another S3 bucket #1015

Merged
merged 5 commits into from
Nov 21, 2023

Conversation

BryanFauble
Copy link
Contributor

Problem:
The examples we had did not mention that the same folder can be used when migrating data to another S3 bucket. We also did not have a full comprehensive example

Solution:
I wrote a full script from start -> finish

Testing:
I verified that I could migrate a folder to an S3 bucket I created through the service catalog

@BryanFauble BryanFauble requested a review from a team as a code owner November 20, 2023 23:54
docs/articles/data_storage.rst Outdated Show resolved Hide resolved
docs/articles/data_storage.rst Show resolved Hide resolved
docs/articles/data_storage.rst Show resolved Hide resolved
docs/articles/data_storage.rst Outdated Show resolved Hide resolved
Copy link
Member

@thomasyu888 thomasyu888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥 LGTM! It would be nice to have a tiny benchmark for how long this code takes to run on a 100 GB dataset so we can provide a super rough estimate for 10TB. I'm also happy to just provide the estimate as we start migrating.

@BryanFauble BryanFauble merged commit 3e9fc93 into develop Nov 21, 2023
22 of 27 checks passed
@BryanFauble BryanFauble deleted the SYNPY-1337-migration-example branch November 21, 2023 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants