-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
stellar-etl: implement export_ledgers command #67
stellar-etl: implement export_ledgers command #67
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is awesome!! Great work. The requested changes are on the smaller side.
logger.Fatal("could not create output file: ", err) | ||
} | ||
|
||
outFile, err := os.OpenFile(absolutePath, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0644) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd add a TODO to check the permissions on this file. Seems like a possible bug surface in the larger architecture.
cmd/export_ledgers_test.go
Outdated
} | ||
|
||
func getLastSeqNum() uint32 { | ||
backend, _ := ledgerbackend.NewHistoryArchiveBackendFromURL("http://history.stellar.org/prd/core-live/core_live_001") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's worth defining another function to define/initialize this backend. I suspect you'll be doing it a good amount across tests, it's an easy way to save some complexity. I'd also define the history archive URL as a variable.
type cliTest struct { | ||
name string | ||
args []string | ||
golden string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great use of golden files :)
cmd/export_ledgers.go
Outdated
outFile.WriteString("\n") | ||
} | ||
|
||
/* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you can delete these planning comments -- looks all implemented :)
ledgersCmd.Flags().Uint32P("limit", "l", 60, "Maximum number of ledgers to export") | ||
ledgersCmd.Flags().StringP("output-file", "o", "exported_ledgers.txt", "Filename of the output file") | ||
ledgersCmd.MarkFlagRequired("end-ledger") | ||
/* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The commented flags should be modified to account for what you've developed :)
PR Checklist
PR Structure
otherwise).
services/friendbot
, orall
ordoc
if the changes are broad or impact manypackages.
Thoroughness
.md
files, etc... affected by this change). Take a look in the
docs
folder for a given service,like this one.
What
This PR implements the export_ledgers command. It allows users to connect to the history archives, retrieve ledger information in a desired range, and export the JSON encodings to an output file.
Why
This command is part of the larger CLI tool. It will allow for updating the BigQuery table history_ledgers more efficiently.
Known limitations
The start-time and end-time flags are not implemented. The ingestion system uses sequence numbers instead of times. If it is expanded to use times, then we can add the flags. The serialize-method flag also is not implemented, since at the moment we only have json encodings for the structs.