C++ Java CMake Cuda Yacc C Other
Clone or download
Permalink
Failed to load latest commit information.
Analyzer Added support for TINYINT May 15, 2018
Archive Fix race from concurrent streaming log messages to LOG(INFO) Apr 19, 2018
Calcite calcite: mark server as unavailable after calling shutdown Jul 16, 2018
Catalog Add GRANT/REVOKE ALL ON DATABASE command Jul 16, 2018
Chunk ALTER TABLE only ADD COLUMN with lock/spi Jul 10, 2018
CudaMgr Using shared memory for single column keyless group by operation with… Jul 16, 2018
DataMgr ALTER TABLE only ADD COLUMN with lock/spi Jul 10, 2018
Distributed Better manages the code split between OS/EE/CE editions. Nov 22, 2017
Fragmenter ALTER TABLE only ADD COLUMN with lock/spi Jul 10, 2018
Import ALTER TABLE only ADD COLUMN with lock/spi Jul 10, 2018
Parser Add GRANT/REVOKE ALL ON DATABASE command Jul 16, 2018
Planner Add Apache License, Version 2.0 headers May 8, 2017
QueryEngine Add timing for fetchchunk time, will include disk time for first runs Jul 16, 2018
QueryRunner Return EXIT_SUCCESS for SIGTERM, actual code for other signals Jul 16, 2018
SQLFrontend remove const variable Jun 4, 2018
SampleCode Use appropriate shared_ptr impl based on Thrift version Mar 21, 2018
SampleData Dictionary encoded strings support 1/? Feb 18, 2015
Shared Add TINYINT to typed data accessors Jul 16, 2018
SqliteConnector deps: move sqlite, linenoise to ThirdParty May 15, 2017
StringDictionary resolve name conflict in OS X builds Jul 10, 2018
SystemProfiler Add Apache License, Version 2.0 headers May 8, 2017
Tests Return EXIT_SUCCESS for SIGTERM, actual code for other signals Jul 16, 2018
ThirdParty git: additional file formats for gitattributes Jun 26, 2018
ThriftHandler Fix null handling for TINYINT Jul 16, 2018
Utils GeoSql: Add 'mapd_geo_bounds' physical column to LINESTRING/POLYGON/M… May 15, 2018
cmake/Modules cmake: arrow: temporary workaround for 0.8.0+ build support Jun 12, 2018
config Add TSAN suppressions file. Ignore lock inversion warning in gdal. May 15, 2018
docker docker: change base image to CUDA 9.0 Jun 26, 2018
docs docs: update link to external docs, remove internal files Aug 28, 2017
java ALTER TABLE only ADD COLUMN with lock/spi Jul 10, 2018
scripts deps: bump Boost to 1.67.0 Jul 7, 2018
systemd systemd: web: bump nofile limit to 65536 Jun 12, 2018
.clang-format Initial LLVM 3.8 support Nov 18, 2016
.gitattributes git: remove odbc/*.xml from gitattributes Jun 27, 2018
.gitignore java: use maven-surefire-plugin for tests Jul 8, 2018
CLA.txt Reformat license, eula, cla text May 7, 2017
CMakeLists.txt cmake: use c++14 for everything except CUDA kernels Jul 10, 2018
CMakePackaging.txt cmake: add install paths for rpm, deb Mar 31, 2017
Doxyfile.in Moved TableDescriptor and ColumnDescriptor to seperate .h files Dec 12, 2014
EULA-CE.txt eula: update EULAs, increase trial to 30 days Jan 16, 2018
LICENSE.md Expose Calcite's SqlAdvisor as a Thrift endpoint Nov 25, 2017
MapDServer.cpp Return EXIT_SUCCESS for SIGTERM, actual code for other signals Jul 16, 2018
MapDServer.h Add S3 data detection Mar 17, 2018
MapDWebServer.go Add http->https redirect handler to mapd_web_server Jul 16, 2018
README.md readme: add instructions for generating packages Jul 10, 2018
ROADMAP.md Roadmap typos/minor style changes Jul 10, 2018
completion_hints.thrift Expose Calcite's SqlAdvisor as a Thrift endpoint Nov 25, 2017
initdb.cpp Import sample geo tables automatically with initdb May 15, 2018
insert_sample_data insert_sample_data: fix cli option parsing Sep 8, 2017
mapd.conf.sample web: deprecate --quiet, add --verbose Apr 20, 2017
mapd.thrift Adds approximations for median & quantile vega transforms. Jun 29, 2018
run_sanity_tests import: add initial test for type detection Apr 20, 2017
startmapd startmapd: don't pass --config option if file does not exist Feb 26, 2018

README.md

MapD Core

MapD Core is an in-memory, column store, SQL relational database that was designed from the ground up to run on GPUs.

Table of Contents

Links

License

This project is licensed under the Apache License, Version 2.0.

The repository includes a number of third party packages provided under separate licenses. Details about these packages and their respective licenses is at ThirdParty/licenses/index.md.

The standard build process for this project downloads the Community Edition of the MapD Immerse visual analytics client. This version of MapD Immerse is governed by a separate license agreement, included in the file EULA-CE.txt, and may only be used for non-commercial purposes.

Contributing

In order to clarify the intellectual property license granted with Contributions from any person or entity, MapD must have a Contributor License Agreement ("CLA") on file that has been signed by each Contributor, indicating agreement to the Contributor License Agreement. After making a pull request, a bot will notify you if a signed CLA is required and provide instructions for how to sign it. Please read the agreement carefully before signing and keep a copy for your records.

Building

If this is your first time building MapD Core, install the dependencies mentioned in the Dependencies section below.

MapD uses CMake for its build system.

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=debug ..
make -j 4

The following cmake/ccmake options can enable/disable different features:

  • -DCMAKE_BUILD_TYPE=release build type and compiler options to use. Options: Debug, Release, RelWithDebInfo, MinSizeRel, and unset.
  • -DENABLE_CUDA=off disable CUDA. Default on.
  • -DMAPD_IMMERSE_DOWNLOAD=on download the latest master build of Immerse / mapd2-frontend. Default on.
  • -DMAPD_DOCS_DOWNLOAD=on download the latest master build of the documentation / docs.mapd.com. Default off. Note: this is a >50MB download.
  • -DPREFER_STATIC_LIBS=on static link dependencies, if available. Default off.
  • -DENABLE_AWS_S3=on enable AWS S3 support, if available. Default on.
  • -DENABLE_TESTS=on build unit tests. Default on.

Testing

MapD Core uses Google Test as its main testing framework. Tests reside under the Tests directory.

The sanity_tests target runs the most common tests. If using Makefiles to build, the tests may be run using:

make sanity_tests

AddressSanitizer

AddressSanitizer can be activated by setting the ENABLE_ASAN CMake flag in a fresh build directory. At this time CUDA must also be disabled. In an empty build directory run CMake and compile:

mkdir build && cd build
cmake -DENABLE_ASAN=on -DENABLE_CUDA=off ..
make -j 4

Finally run the tests:

export ASAN_OPTIONS=alloc_dealloc_mismatch=0:handle_segv=0
make sanity_tests

ThreadSanitizer

ThreadSanitizer can be activated by setting the ENABLE_TSAN CMake flag in a fresh build directory. At this time CUDA must also be disabled. In an empty build directory run CMake and compile:

mkdir build && cd build
cmake -DENABLE_TSAN=on -DENABLE_CUDA=off ..
make -j 4

We use a TSAN suppressions file to ignore warnings in third party libraries. Source the suppressions file by adding it to your TSAN_OPTIONS env:

export TSAN_OPTIONS="suppressions=/path/to/mapd/config/tsan.suppressions"

Finally run the tests:

make sanity_tests

Generating Packages

MapD Core uses CPack to generate packages for distribution. Packages generated on CentOS with static linking enabled can be used on most other recent Linux distributions.

To generate packages on CentOS (assuming starting from top level of the mapd-core repository):

mkdir build-package && cd build-package
cmake -DPREFER_STATIC_LIBS=on -DCMAKE_BUILD_TYPE=release ..
make -j 4
cpack -G TGZ

The first command creates a fresh build directory, to ensure there is nothing left over from a previous build.

The second command configures the build to prefer linking to the dependencies' static libraries instead of the (default) shared libraries, and to build using CMake's release configuration (enables compiler optimizations). Linking to the static versions of the libraries libraries reduces the number of dependencies that must be installed on target systems.

The last command generates a .tar.gz package. The TGZ can be replaced with, for example, RPM or DEB to generate a .rpm or .deb, respectively.

Using

The startmapd wrapper script may be used to start MapD Core in a testing environment. This script performs the following tasks:

  • initializes the data storage directory via initdb, if required
  • starts the main MapD Core server, mapd_server
  • starts the MapD Core web server, mapd_web_server, for serving MapD Immerse
  • offers to download and import a sample dataset, using the insert_sample_data script
  • attempts to open MapD Immerse in your web browser

Assuming you are in the build directory, and it is a subdirectory of the mapd-core repository, startmapd may be run by:

../startmapd

Starting Manually

It is assumed that the following commands are run from inside the build directory.

Initialize the data storage directory. This command only needs to be run once.

mkdir data && ./bin/initdb data

Start the MapD Core server:

./bin/mapd_server

In a new terminal, start the MapD Core web server:

./bin/mapd_web_server

If desired, insert a sample dataset by running the insert_sample_data script in a new terminal:

../insert_sample_data

You can now start using the database. The mapdql utility may be used to interact with the database from the command line:

./bin/mapdql -p HyperInteractive

where HyperInteractive is the default password. The default user mapd is assumed if not provided.

You can also interact with the database using the web-based MapD Immerse frontend by visiting the web server's default port of 9092:

http://localhost:9092

Note: usage of MapD Immerse is governed by a separate license agreement, provided under EULA-CE.txt. The version bundled with this project may only be used for non-commercial purposes.

Code Style

A .clang-format style configuration, based on the Chromium style guide, is provided at the top level of the repository. Please format your code using a recent version (6.0+ preferred) of ClangFormat before submitting.

To use:

clang-format -i File.cpp

Contributed code should compile without generating warnings by recent compilers on most Linux distributions. Changes to the code should follow the C++ Core Guidelines.

Dependencies

MapD has the following dependencies:

Package Min Version Required
CMake 3.3 yes
LLVM 3.8-4.0, 6.0 yes
GCC 5.1 no, if building with clang
Go 1.6 yes
Boost 1.57.0 yes
OpenJDK 1.7 yes
CUDA 8.0 yes, if compiling with GPU support
gperftools yes
gdal yes
Arrow 0.7.0 yes

Dependencies for mapd_web_server and other Go utils are in ThirdParty/go. See ThirdParty/go/src/mapd/vendor/README.md for instructions on how to add new deps.

CentOS 7

MapD Core requires a number of dependencies which are not provided in the common CentOS/RHEL package repositories. The script scripts/mapd-deps-centos.sh is provided to automatically build and install these dependencies. A prebuilt package containing these dependencies is also provided for CentOS 7 (x86_64).

First install the basic build tools:

sudo yum groupinstall -y "Development Tools"
sudo yum install -y \
    zlib-devel \
    epel-release \
    libssh \
    openssl-devel \
    ncurses-devel \
    git \
    maven \
    java-1.8.0-openjdk-devel \
    java-1.8.0-openjdk-headless \
    gperftools \
    gperftools-devel \
    gperftools-libs \
    python-devel \
    wget \
    curl \
    environment-modules

Next download and install the prebuilt dependencies:

curl -OJ https://internal-dependencies.mapd.com/mapd-deps/deploy.sh
sudo bash deploy.sh

These dependencies will be installed to a directory under /usr/local/mapd-deps. The deploy.sh script also installs Environment Modules in order to simplify managing the required environment variables. Log out and log back in after running the deploy.sh script in order to active Environment Modules command, module.

The mapd-deps environment module is disabled by default. To activate for your current session, run:

module load mapd-deps

To disable the mapd-deps module:

module unload mapd-deps

WARNING: The mapd-deps package contains newer versions of packages such as GCC and ncurses which might not be compatible with the rest of your environment. Make sure to disable the mapd-deps module before compiling other packages.

Instructions for installing CUDA are below.

CUDA

It is preferred, but not necessary, to install CUDA and the NVIDIA drivers using the .rpm using the instructions provided by NVIDIA. The rpm (network) method (preferred) will ensure you always have the latest stable drivers, while the rpm (local) method allows you to install does not require Internet access.

The .rpm method requires DKMS to be installed, which is available from the Extra Packages for Enterprise Linux repository:

sudo yum install epel-release

Be sure to reboot after installing in order to activate the NVIDIA drivers.

Environment Variables

scripts/mapd-deps-centos.sh generates two files with the appropriate environment variables: mapd-deps-<date>.sh (for sourcing from your shell config) and mapd-deps-<date>.modulefile (for use with Environment Modules, yum package environment-modules). These files are placed in mapd-deps install directory, usually /usr/local/mapd-deps/<date>. Either of these may be used to configure your environment: the .sh may be sourced in your shell config; the .modulefile needs to be moved to the modulespath.

macOS

scripts/mapd-deps-osx.sh is provided that will automatically install and/or update Homebrew and use that to install all dependencies. Please make sure macOS is completely update to date and Xcode is installed before running. Xcode can be installed from the App Store.

CUDA

mapd-deps-osx.sh will automatically install CUDA via Homebrew and add the correct environment variables to ~/.bash_profile.

Java

mapd-deps-osx.sh will automatically install Java and Maven via Homebrew and add the correct environment variables to ~/.bash_profile.

Ubuntu 16.04 - 18.04

Most build dependencies required by MapD Core are available via APT. Certain dependencies such as Thrift, Blosc, and Folly must be built as they either do not exist in the default repositories or have outdated versions. The provided scripts/mapd-deps-ubuntu.sh script will install all required dependencies (except CUDA) and build the dependencies which require it. The built dependencies will be installed to /usr/local/mapd-deps/ by default; see the Environment Variables section below for how to add these dependencies to your environment.

Ubuntu 18.04

Some installs of Ubuntu 18.04 may fail while building with a message similar to:

java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

This is a known issue in 18.04 which will be resolved in Ubuntu 18.04.1. To resolve on 18.04:

sudo rm /etc/ssl/certs/java/cacerts
sudo update-ca-certificates -f

Environment Variables

The CUDA and mapd-deps lib directories need to be added to LD_LIBRARY_PATH; the CUDA and mapd-deps bin directories need to be added to PATH. The mapd-deps-ubuntu.sh script above will generate a script named mapd-deps.sh containing the environment variables which need to be set. Simply source this file in your current session (or symlink it to /etc/profile.d/mapd-deps.sh) in order to activate it:

source /usr/local/mapd-deps/mapd-deps.sh

CUDA

Recent versions of Ubuntu provide the NVIDIA CUDA Toolkit and drivers in the standard repositories. To install:

sudo apt install -y \
    nvidia-cuda-toolkit

Be sure to reboot after installing in order to activate the NVIDIA drivers.

Arch

The following uses yaourt to install packages from the Arch User Repository.

yaourt -S \
    git \
    cmake \
    boost \
    google-glog \
    extra/jdk8-openjdk \
    clang \
    llvm \
    thrift \
    go \
    gdal \
    maven

VERS=1.21-45
wget --continue https://github.com/jarro2783/bisonpp/archive/$VERS.tar.gz
tar xvf $VERS.tar.gz
pushd bisonpp-$VERS
./configure
make -j $(nproc)
sudo make install
popd

CUDA

CUDA and the NVIDIA drivers may be installed using the following.

yaourt -S \
    linux-headers \
    cuda \
    nvidia

Be sure to reboot after installing in order to activate the NVIDIA drivers.

Environment Variables

The CUDA bin directories need to be added to PATH. The easiest way to do so is by creating a new file named /etc/profile.d/mapd-deps.sh containing the following:

PATH=/opt/cuda/bin:$PATH
export PATH