-
Notifications
You must be signed in to change notification settings - Fork 426
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[NIT-2589] init: define parts with manifest file #2376
Conversation
[NIT-2467][Configuration Changes] Add option to enable the checksum validation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, just one question about partUrl
cmd/nitro/init.go
Outdated
// Download parts | ||
for i, partName := range partNames { | ||
log.Info("Downloading database part", "part", partName) | ||
partUrl := url.JoinPath("..", partName).String() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do we join ".."
instead of "."
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The ..
removes the last element from the path. For instance, http://path/to/archive.tar
becomes http://path/to/archive.tar/../archive.tar.part0
, which then becomes http://path/to/archive.tar.part0
.
I noticed the code is misleading because the variable url
shadows the url
module from the standard library. I will rename it to archiveUrl
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
makes sense, thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
In a previous PR, we added the capability of downloading the initialization snapshot in parts. This PR enhances this feature by requiring a manifest file with the parts' names and checksums. We use a manifest file to determine how many parts the archive was divided into. Thus, avoiding missing a part and loading a corrupted database.
When downloading a snapshot, the initialization code will look for the snapshot under
--init.url
. Notice that this URL can be set automatically using the--init.latest
option, which is already merged.For instance, let's assume the downloaded snapshot URL is
https://snapshot.arbitrum.foundation/sepolia-rollup/2024/20/archive.tar
. If this file is not found on the server (404 HTTP status code), nitro will look for the manifest file containing information about the parts. This manifest file is the archive URL with the suffix.manifest.txt
. The format of the manifest file is the output of thesha256sum
command.So, assuming we have the following file structure on the server:
The contents of the manifest file will be:
The init code will read the checksum and file names and download each part from the server. If the checksum validation option is enabled, the checksum will also be validated after downloading the file. Finally, the code will join all the parts into a single archive, extract it, and proceed to load the database.
This PR also includes the changes from the #2384 PR.