Count the number of times certain words were said in a particular neighborhood. Performed as a basic MapReduce job against 25M tweets. Implemented with different programming languages as a educational exercise.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
csharp
elixir
erlang
fixtures
golang
haskell
mathematica
nim
nodejs
ocaml
perl
php
python
ruby
rust
scala
shell
.gitignore
README.md
fetch_tweets
reference_output
run_csharp.ps1
run_elixir
run_erlang
run_go
run_haskell
run_mathematica
run_nim
run_nodejs
run_ocaml
run_perl
run_php
run_python
run_ruby
run_ruby_parallel
run_rust
run_scala
run_sh

README.md

Update

Please see the following blog posts for the latests updates:

  1. ETL Language Showdown - Sept. 2014
  2. ETL Language Showdown Part 2 - Now with Python - May. 2015
  3. ETL Language Showdown Part 3 - 10 Languages and growing - Nov. 2015

Wins

Analyses and discussions done here have led to the following language pull requests:

  1. Add BIF binary:split/2,3 to Erlang
  2. Improve case insensitive regex to Golang

ETL Language Showdown

This repo implements the same map reduce ETL (Extract-Transform-Load) task in multiple languages in an effort to compare language productivity, terseness and readability. The performance comparisons should not be taken seriously. If anything, it is a bigger indication of my skillset in that language rather than their performance capabilities.

The Task

Count the number of tweets that mention 'knicks' in their message and bucket based on the neighborhood of origin. The ~1GB dataset for this task, sampled below, contains a tweet's message and its NYC neighborhood.

Simply run fetch_tweets in the repo directory or downloaded here.

91	west-brighton	Brooklyn	Uhhh
121	turtle-bay-east-midtown	Manhattan	Say anything
175	morningside-heights	Manhattan	It feels half-cheating half-fulfilling to cite myself.

Initial Assumption

  • These tasks are not run on Hadoop but do run concurrently. Performance numbers are moot since the CPU mostly sits idle waiting on Disk IO.
  • **UPDATE: Boy was the IO bound assumption wrong.

The Languages

Below you will find the languages run. Note that frameworks also play a big role, for example the Scala implementation compares the parallel collection to futures and the Akka framework. Click through on each language to read more.

LanguageOwner
Ruby
Golangmatttproud
Scala
Nim
Node
PHP
Erlang
Elixirjosevalim
Rust
Python
C#mganss
shellmganss
perlsitaramc