Multi-process test data files generator
Failed to load latest commit information.
bin Initial commit. Apr 27, 2012
conf Add example gen commands. Jul 31, 2013
examples Add datagen#generate, simplify worker handling. Jun 15, 2012
lib Remove optional worker-farm injection, use proxyquire instead. Nov 6, 2014
screenshots Fix screenshot, command was previously cropped. Oct 16, 2013
test-integration Use params on cmdt integration test. Mar 21, 2014
test Update path assertion to handle non-project-name workspace. Jan 30, 2016
.bob.json Replace gh-pages-cli site tool with buildbranch. Jun 21, 2015
.gitignore Change bob dir. Aug 23, 2012
.jshintrc Move jshint config to .jshintrc file. Jul 31, 2013
.npmignore Add screenshots. Oct 15, 2013
.travis.yml Update Travis config to latest. Update node versions to 4.x and 6.x . Aug 10, 2016 Bump version up to v1.0.2 . Aug 10, 2016
LICENSE Update (c) year. Feb 19, 2013 Fix build report links following site tool change to buildbranch. Jun 21, 2015
package.json Bump version up to v1.0.2 . Aug 10, 2016


Build Status Dependencies Status Coverage Status Published Version
npm Badge


DataGen is a multi-process test data files generator.

This is handy when you want to generate large test data files (e.g. XMLs, JSONs, CSVs, etc), over multiple processes, utilising available CPU cores on your machine. It's also very easy to generate random numbers, dates, and strings as test data. You only need to create template files, no scripting involved.


npm install -g datagen 


Create example header, segment, and footer template files:

datagen init

Generate 8 data files containing 1 million segments in each file, written to data1 ... data8 output files, running over 8 processes, each process is used to generate 1 file:

datagen gen -s 1000000 -w 8 -o data

Limit the number of worker processes that can run concurrently to just 3:

datagen gen -s 1000000 -w 8 -m 3


DataGen uses three template files: header, segment, and footer. These templates are simple text files which are used to construct a data file in this format:

segment 1
segment 2
segment N (number of segments)

Example header:

<?xml version="1.0" encoding="UTF-8"?>

Example segment:

  <name>{first_name()} {last_name()}</name>

Example footer:


The above templates will generate an XML like this:

<?xml version="1.0" encoding="UTF-8"?>
  <name>Niels Bryant</name>
  <name>John Bohr</name>

Curly brackets in templates (e.g. JSON format) need to be escaped with double brackets:

{{ "id": "{gen_id}-{worker_id}-{segment_id}", "name": "{first_name()} {last_name()}", "dob": "{date('dd-mm-yyyy')}" }}

Templates can contain the following parameters:

{gen_id}Unique to each datagen execution. Default value is master process PID, can be overridden via -i flag.
{worker_id}Unique to each worker. Value from 1 to number of workers.
{segment_id}Unique to each segment within the generated data file, repeated in each file. Value from 1 to number of segments. Not available in header and footer templates.
{integer()}Random integer.
{integer(100, 200)}Random integer between 100 and 200.
{float()}random float.
{float(1.23, 5.67)}Random float between 1.23 and 5.67 .
{date()}random date between 1970 and 2020 in ISO format.
{date('yyyy-mm-dd')}Random date between 1970 and 2020 in yyyy-mm-dd format.
Check out felixge/node-dateformat for more date formats.
{date(2000, 2010)}Random date between 2000 and 2010 in ISO format.
{date('yyyy-mm-dd', 2000, 2010)}Random date between 2000 and 2010 in yyyy-mm-dd format.
{select('apple', 'orange', ..., 'kiwi')}Select one item from the arguments. You can have as many arguments as you want.
{word()}Random word from Lorem Ipsum.
{word(3)}Random 3 words from Lorem Ipsum.
{first_name()}Random first name.
{last_name()}Random last name.
{email()}Random email address.
{phone()}Random phone number. Default format #### ####.
{phone('(###) ####-####')}Random phone number with a custom format. Each '#' will be replaced by a random number.


Command init Command gen with flags


Developer's Guide

Build reports: