Skip to content
Switch branches/tags
This branch is up to date with master.

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Unified New York City Taxi and Uber data (now in Clojure!)

Forked from, but instead of Bash and PostgreSQL this project uses Clojure and Elasticsearch. At this point only supports Yellow and Green datasets as they have the most trips, have detailed pickup/dropoff coordinates

Setting up Elasticsearch + Kibana docker images and ZFS (works on my machine...)

Starting Elasticsearch 5 with 16 GB of RAM, storing files to main EXT4 SSD:

cd ~/projects
git clone
cd docker-scripts
./ elasticsearch5 kibana5
./ 5 16 --data /data0/es5
./ 5

Starting Elasticsearch 5 with 16 GB of RAM, storing files to mirrored SSDs on ZFS on Linux:

zfs zpool create data1 /dev/disk/by-id/ata-Samsung_SSD_750_EVO_500GB_XXX /dev/disk/by-id/ata-Samsung_SSD_750_EVO_500GB_YYY

# I had terrible 40 MB/s write troughput without this hack... Not sure why :( Root partition is EXT4
truncate -s 8g /data1_zil.dat && zpool add data1 log /data1_zil.dat

zfs create /data1/volume1

# These might give us a performance boost
zfs set atime=off data1/volume1
zfs set recordsize=8K data1/volume1

./ 5 16 --data /data1/volume1/es5

ETL commands

I cannot guarantee these instructions will be up to date, but this is how this project works at the moment. Note that this was written before I cherry-picked the commit which brought 2016 July - December rows to raw_data_urls.txt, so these counts are missing 68.8 million items.

$ git clone
$ cd nyc-taxi-data

# Downloading raw CSVs
$ ./

# Checking what we've got, apparently 148.5 GB of CSVs (879.2 million rows) compressed to 31 GB
$ cd data
$ du -hc * | tail -n1
31G	total

$ ./
ans =
     0.87915   148.52923

# Building the JAR
$ cd ../taxi-rides-clj
$ lein uberjar

# You need Java 8 or newer to run this project as dates are parsed by java.time
$ java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

# Do less parallel work if you run out of memory or want to use the computer for other work as well.
$ N_PARALLEL=`nproc`
$ JAR=target/taxi-rides-clj-0.0.1-SNAPSHOT-standalone.jar

# Parsing, removing duplicates, merging with weather data and writing to local Elasticsearch.
# Destination can be overridden by ES_SERVER= env variable if needed.
# I could index on average about 20k docs / second on Core i7 6700K, resulting in 873.3
# million docs taking 331.3 GB of storage. _all was disabled but _source was not.
$ time ls ../data/*.gz | shuf | xargs java -Xms16g -Xmx30g -jar $JAR insert $N_PARALLEL

# Parsing, removing duplicates, merging with weather data and writing out to .csv.gz files.
# On Core i7 6700K this took 143 core-hours! There might be room for optimization, but then again
# it produced 189.4 gigabytes of raw CSV and compressed it down to 45.4 gigabytes. It should be
# easy to bulk-insert to other database systems such as Redshift or just MS SQL Server.
$ mkdir data_out
$ time ls ../data/*.gz | shuf | xargs java -Xms16g -Xmx30g  -jar $JAR extract $N_PARALLEL


Parse and unify public NYC taxi trip data in Clojure, index into Elasticsearch







No releases published


No packages published