Fetch (and store) all XRPL transactions (to a custom datastore)
Switch branches/tags
Clone or download
Latest commit ca03137 Oct 2, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore Add Ledger headers Oct 1, 2018
LICENSE Add LICENSE Aug 15, 2018
README.md Google BigQuery done (?) Sep 27, 2018
applySchema.js Google BigQuery done (?) Sep 27, 2018
index.js >200 = large Oct 2, 2018
ledgerInfo.js Remove TX, join Oct 1, 2018
package-lock.json Google BigQuery done (?) Sep 27, 2018
package.json Google BigQuery done (?) Sep 27, 2018
schema.js Google BigQuery done (?) Sep 27, 2018

README.md

Fetch all transactions from the XRP ledger

This code allows you to fetc all transactions from the XRP ledger and insert them into Google BigQuery.

The data is already READY TO USE 🎉!

The data is already available in a PUBLIC dataset at Google BigQuery in:

xrpledgerdata.fullhistory.transactions

So a working sample query with some stats would be:

SELECT 
  COUNT(1) as TxCount,
  MIN(LedgerIndex) as MinLedger,
  MAX(LedgerIndex) as MaxLedger,
  COUNT(DISTINCT LedgerIndex) as LedgersWithTxCount
FROM 
  xrpledgerdata.fullhistory.transactions

Starting Sept. 27 2018 the dataset will be backfilled from a full history rippled node. Once up to date, I'll run a service that will add all new transactions to the dataset as well.

Run and insert into your own Google BigQuery project

Setup

If you want to insert the data in your own BigQuery project, download your credentials (JSON) from the Google Admin Console, and export the path to your credentials file:

OSX

export GOOGLE_APPLICATION_CREDENTIALS="[PATH]"

Windows

Powershell:

$env:GOOGLE_APPLICATION_CREDENTIALS="[PATH]"

Command Prompt:

set GOOGLE_APPLICATION_CREDENTIALS=[PATH]

OSX Sample

export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/[FILE_NAME].json"

More information: https://cloud.google.com/docs/authentication/getting-started

Now modify schema.js (line 1-3) to point to your own projectId, dataset and table.

Create schema

Run node applySchema.js

WARNING! If an existing table exists, the table, schema and data will be REMOVED!

Insert data

You can invoke the script from a node enabled environment by setting these environment variables:

  • NODE: the rippled node (wss://...) to connect to, default: wss://s2.ripple.com
  • LEDGER: the ledger index to start fetching transactions from, default: 32570