[NIT-2589] init: define parts with manifest file #2376
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cla-bot
bot
added
the
s
Tristan-Wilson
previously approved these changes
Jun 14, 2024
amsanghi
previously approved these changes
Jun 17, 2024
[NIT-2467][Configuration Changes] Add option to enable the checksum validation
amsanghi
previously approved these changes
Jun 18, 2024
magicxyyz
approved these changes
Jun 18, 2024
cmd/nitro/init.go
Outdated
| // Download parts | ||
| for i, partName := range partNames { | ||
| log.Info("Downloading database part", "part", partName) | ||
| partUrl := url.JoinPath("..", partName).String() |
There was a problem hiding this comment.
why do we join ".." instead of "."?
There was a problem hiding this comment.
The .. removes the last element from the path. For instance, http://path/to/archive.tar becomes http://path/to/archive.tar/../archive.tar.part0, which then becomes http://path/to/archive.tar.part0.
I noticed the code is misleading because the variable url shadows the url module from the standard library. I will rename it to archiveUrl.
Tristan-Wilson
approved these changes
Jun 21, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In a previous PR, we added the capability of downloading the initialization snapshot in parts. This PR enhances this feature by requiring a manifest file with the parts' names and checksums. We use a manifest file to determine how many parts the archive was divided into. Thus, avoiding missing a part and loading a corrupted database.
When downloading a snapshot, the initialization code will look for the snapshot under
--init.url. Notice that this URL can be set automatically using the--init.latestoption, which is already merged.For instance, let's assume the downloaded snapshot URL is
https://snapshot.arbitrum.foundation/sepolia-rollup/2024/20/archive.tar. If this file is not found on the server (404 HTTP status code), nitro will look for the manifest file containing information about the parts. This manifest file is the archive URL with the suffix.manifest.txt. The format of the manifest file is the output of thesha256sumcommand.So, assuming we have the following file structure on the server:
The contents of the manifest file will be:
The init code will read the checksum and file names and download each part from the server. If the checksum validation option is enabled, the checksum will also be validated after downloading the file. Finally, the code will join all the parts into a single archive, extract it, and proceed to load the database.
This PR also includes the changes from the #2384 PR.