Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Scripts for locally hosting and searching a compressed StackOverflow data dump
C Shell PHP Python
Branch: master

Compression explained

Some posts are thrown away, but they're too slow to find anyway.
latest commit bb7499e781
@peterburk authored


StackOverflow Local
Peter Burkimsher

This collection of scripts allows the StackOverflow data dump to be rapidly accessed while offline. 

-- About --
The data dump is compressed to only 2.09 GB, plus 230.5 MB for titles. 
Search time is about 8 seconds. 
Post reading time is about 6 seconds. 

-- Installing --
I recommend not running ./ as normal! It will be very slow. 
Instead, open three Terminal windows and run the commands inside in parallel. 

-- Requirements --
You'll need cc to compile isQaId for setup. 
At runtime, you'll need a web server, PHP, Python, and bzip2recover. 

-- Developers --
You can run the scripts without the web server.
1. ./ "your search here"
(returns the IDs and titles, separated by a > character)
2. ./ 1234567
(returns the post, in compressed form)

The compressed format of the database is delimited by < and > characters as follows:
<<Post ID<>Answer Id><Body>>

For reference, the original StackOverflow database is 4.6 GB. 
It's compressed by only keeping the accepted answer for each question, and cutting out a lot of the XML. 
Searching for all answers to a question takes 30 minutes of search time per question, which is too slow. 

Using dd instead of is possible, but post reading time increases to 30 seconds. 
Something went wrong with that request. Please try again.