Add Support for Archive Uploads to GitHub-Owned Storage with Multipart Upload Option #1261
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Unit Test Results0 tests 0 ✅ 0s ⏱️ Results for commit 25ea598. ♻️ This comment has been updated with latest results. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
hfishback01
commented
Sep 11, 2024
hfishback01
commented
Sep 11, 2024
src/OctoshiftCLI.Tests/bbs2gh/Commands/MigrateRepo/MigrateRepoCommandArgsTests.cs
Show resolved
Hide resolved
| @@ -198,6 +200,22 @@ private async Task<string> UploadArchiveToAws(string bucketName, string archiveP | |||
| return archiveBlobUrl; | |||
| } | |||
|
|
|||
| private async Task<string> UploadArchiveToGithub(string org, string archivePath) | |||
| { | |||
| #pragma warning disable IDE0063 | |||
There was a problem hiding this comment.
Is there a reason the warning was disabled? I think we can use using declaration here.
src/OctoshiftCLI.Tests/bbs2gh/Commands/MigrateRepo/MigrateRepoCommandHandlerTests.cs
Outdated
Show resolved
Hide resolved
src/OctoshiftCLI.Tests/gei/Commands/MigrateRepo/MigrateRepoCommandHandlerTests.cs
Outdated
Show resolved
Hide resolved
| { httpContent, "archive", archiveName } | ||
| }; | ||
|
|
||
| response = await _client.PostAsync(url, content); |
There was a problem hiding this comment.
A couple of observations:
- Even though this may work the
PostAsyncmethod will convert the data to JSON (basically base64 string) and will eventually createStringContentso I am not sure if this is the intended use here because we're now dealing with aStreamrather than string content. So We may want to refactor the SendAsync method to also support aMultiPartFormDataContentandStreamContent. It can simply check to see if the passed inbodyis either of those and not convert it to a string content. - When dealing with a stream we shouldn't log the entire body as we do here instead as I suggested in option 1, we can check the body and if it's a multipart form data or a stream content we can just say
BLOBorBinary datainstead of dumping the entire JSON encoded body!
begonaguereca
changed the title
|
6 tasks
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This PR introduces the ability for the GEI CLI to upload archives to GitHub-owned storage using multipart uploads. To enable this, a new CLI option, --use-github-storage, has been added. This option allows users to explicitly specify that their archives should be uploaded to GitHub's managed storage instead of their own.
Key Features:
Multipart Upload Support: Leveraging multipart upload for reliable and efficient large file uploads. This ensures the robustness of the uploads, minimizing the chances of failure during transmission. We're investigating libraries to handle multipart uploads, but for now, this feature is not rolled by hand unless necessary.
New Option: --use-github-storage:
This option is required to upload archives to GitHub-owned storage and will be hidden until GitHub-owned storage reaches general availability (GA).
By default, archives are still uploaded to user-specified storage locations unless the --use-github-storage flag is explicitly set.
Motivation:
As discussed during our EDR, we want to allow users to utilize GitHub-owned storage as an option without making it the default. This approach lets us roll out GitHub-managed storage in stages and gather feedback while providing an explicit way to opt-in to this feature.
Check in BBS MigrateRepoCommandArgs removed, relating to changes in this PR: #1057
Take it out for a 🚗 :
Migrating from GitHub Enterprise Server:
Successful migration_id:
RM_kgHaACQxMWNjY2RlMi1hNTAzLTQwMWItOTA1OS1kM2NkZjBiYjM4Y2QCloses: https://github.ghe.com/github/octoshift/issues/8969
ThirdPartyNotices.txt(if applicable)