Stock Market Data Generator for CockroachDB

This is a simple Go program to generate and load into CockroachDB some stock market oriented data. It was built in response to a question about how many rows of this type of data could be loaded within a specified time window.

Build

$ go get github.com/google/uuid
$ go get github.com/jackc/pgx/v4
$ go build

Run

• Edit the environment file, setting the various DB connection parameters as well as:

  • BATCH_SIZE: the number of rows copied into the table at a time (defaults to 128)
  • N_THREADS: the number of parallel goroutines used to parallelize the work (defaults to 4). NOTE: each of these consumes one DB connection

• Dump the table DDL and use the output to create the table to hold this generated data: $ ./gen_data --dump-ddl

• Generate and load the data: $ ./gen_data 100000

Notes

• The data directory contains some data files used as reference here since it seemed like "stock market data".