This is an experiment in text search/retrieval. It doesn't work (yet). We're making this available in the spirit of doing development out in the open, but nothing here has been cleaned up for public consumption. The documentation is non-existent and the code is in an incomplete state.
If you really want to try things out, we have some instructions on how to index Wikipedia, and how to play with our query parser. However, the system doesn't do anything useful to an end user -- these instructions are intended to allow contributors to build and run part of a system for the purpose of understand or debugging the system. If you're looking to play with an incomplete and likely buggy system, go ahead! If you want a working search index that you can use in production, check back later.
If you want a description of one of the core ideas, see this StrangeLoop talk transcript. See our engineering diary for descriptions of other parts of the system. We seem to be adding new descriptions at least once a week, and we (sometimes) go back and update old descriptions as things change.
In order to build BitFunnel you will need CMake (2.8.11+), and a modern C++ compiler (gcc 5+, clang 3.5+, or VC 2015+). You can run CMake directly to generate the appropriate build setup for your platform. Alternately, we have some scripts that have the defaults that we use available.
For *nix platforms (including OS X),
./Configure_Make.sh cd build-make make make test
Note that while these instructions are for a
make build, it's also possible to build using
ninja by changing the
cmake command to create
ninja files instead of
Makefiles. These aren't listed in the instructions because
ninja requires installing an extra dependency for some developers, but if you want to use
ninja it's available via
brew, etc., and is susbtantially faster than
If you're on Ubuntu 15+, you can install dependencies with:
sudo apt-get install clang cmake
On Ubuntu 14 and below, you'll need to install a newer version of CMake. To
install a new-enough CMake, see this link.
If you're using gcc, you'll also need to make sure you have gcc-5 (
sudo apt-get install g++-5).
To override the default compiler, set the
CC environment variables.
For example, if you have clang-3.8 installed as
clang-3.8 and are using bash:
export CXX="clang++-3.8" export CC="clang-3.8"
Install XCode and then run the following command to install required packages using Homebrew (http://brew.sh/):
brew install cmake
BitFunnel can be built on OS X using either standard *nix makefiles or XCode.
In order to generate and build makefiles, in the root
BitFunnel directory run:
If you want to create an Xcode project instead of using Makefiles, run:
If you use XCode, you'll have to either re-run
Configure_XCode or run the
ZERO_CHECK target when the
CMakeLists changes, e.g., when source files are added or removed.
Install the following tools:
- Visual Studio 2015 with C++ compiler
- CMake (http://www.cmake.org/download/)
You can get the free version of Visual Studio here. Note that if you're installing Visual Studio for the first time and select the default install options, you won't get a C++ compiler. To force the install of the C++ compiler, you need to either create a new C++ project or open an existing C++ project.
In order to configure solution for Visual Studio 2015 run the following
commands from the root
From now on you can use the generated solution
build-msvc\BitFunnel.sln from Visual Studio
or build from command line using