Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Map out ledger input and output data structures for CLI #24

Closed
Isaiah-Turner opened this issue Jul 6, 2020 · 1 comment
Closed

Map out ledger input and output data structures for CLI #24

Isaiah-Turner opened this issue Jul 6, 2020 · 1 comment
Assignees

Comments

@Isaiah-Turner
Copy link
Contributor

What

We need to map out what information we get about ledgers from the new ingestion system. We also need to know what the output to BigQuery should look like. Part of #19.

Why

This issue helps us know what ledger information we can get from the ingestion system and what information may have to be reconstructed or omitted.

@Isaiah-Turner
Copy link
Contributor Author

In order to get Ledger info, we can call GetLedger() with the desired sequence number (call it meta). Then we can get the actual ledger by doing ledger = meta.V0.
In order to export the data to BigQuery, we need a struct with the following fields:

sequence: integer - given from command line
ledger_hash: string - get from ledger.LedgerHeader.Hash
previous_ledger_hash: string - get from ledger.LedgerHeader.Header.PreviousLedgerHash
transaction_count: integer - get from len(ledger.TxSet.Txs); verify that the length is equal to len(ledger.TxProcessing)
operation_count: integer - iterate through transactions using NewLedgerTransactionReader() and count operations
closed_at: timestamp - get from ledger.LedgerHeader.ScpValue.CloseTime
created_at: timestamp - TODO: figure this out (relates to creation time in BigQuery)
updated_at: timestamp - TODO: figure this out (relates to table update time in BigQuery)
importer_version: integer - version of ingestion system; TODO: figure out how to access
total_coins: integer - get from ledger.LedgerHeader.Header.TotalCoins
fee_pool: integer - get from ledger.LedgerHeader.Header.FeePool
base_fee: integer - get from ledger.LedgerHeader.Header.BaseFee
base_reserve:integer - get from ledger.LedgerHeader.Header.BaseReserve
max_tx_set_size: integer - get from ledger.LedgerHeader.Header.MaxTxSetSize
protocol_version: integer - get from ledger.LedgerHeader.LedgerVersion
ledger_header: bytes - get from xdr.MarshalBase64(ledger.LedgerHeader.Header)
successful_transaction_count: integer - iterate through transactions using NewLedgerTransactionReader() and count operations for successful transactions
failed_transaction_count: integer - iterate through transactions using NewLedgerTransactionReader() and count operations for failed transactions

In order to transform the xdr.Hash into a hex string, convert the Hash to a byte array and use
the EncodeToString function in the encoding/hex package.

Created and updated at times need to be converted to Unix timestamps, right now they are integers.
Use Unix function from time package to get a Time object; format as a UTC time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant