- Install the dependencies mentioned in the Dependencies page.
- Download and Install the package.
- Read up about Usage and Configuration
- Initialize servers using a wrapper or manually
- Load sample data. Sample sets can be found in the Data Set Library.
- Perform a Sample Query
Developing OmniSciDB: Table of Contents
- Downloads and Installation Instructions
- Working With Data
- Code Style
- Developing With Specific Programming Languages
- APIS, Interfaces & Projects
- Need Help?
Downloads and Installation Instructions
OmniSci provides pre-built binaries for Linux for stable releases of the project:
|Distro||Package type||CPU/GPU||Download Link||Installation Guide|
|Ubuntu||DEB||CPU||deb https://releases.omnisci.com/os/apt/ stable cpu||https://www.omnisci.com/docs/latest/4_ubuntu-apt-cpu-os-recipe.html|
|Ubuntu||DEB||GPU||deb https://releases.omnisci.com/os/apt/ stable cuda||https://www.omnisci.com/docs/latest/4_ubuntu-apt-gpu-os-recipe.html|
- Installing OmniSci GPU Open Source for Ubuntu Using Apt
- Installing OmniSci GPU Open Source for Ubuntu With a Tarball
- Installing OmniSci GPU Open Source for CentOS Using Yum
- Installing OmniSci GPU Open Source for CentOS With a Tarball
Learn more by visiting the Setup page.
If this is your first time building OmniSciDB, check out the Quickstart above or visit the Setup page for a setup checklist and additional resources. Additionally, there is a curated list of known issues & limitations of this project.
Reminder: Before you begin building, install the appropriate dependencies.
Once you have completed the installation guide listed in the Downloads and Installation Instructions, you are ready to build.
OmniSciDB uses CMake for its build system.
mkdir build cd build cmake -DCMAKE_BUILD_TYPE=debug .. make -j 4
ccmake options can enable/disable different features:
-DCMAKE_BUILD_TYPE=release- Build type and compiler options to use. Options are
MinSizeRel, and unset.
-DENABLE_ASAN=off- Enable address sanitizer. Default is
-DENABLE_AWS_S3=on- Enable AWS S3 support, if available. Default is
-DENABLE_CALCITE_DELETE_PATH=on- Enable Calcite Delete Path. Default is
-DENABLE_CALCITE_UPDATE_PATH=on- Enable Calcite Update Path. Default is
-DENABLE_CUDA=off- Disable CUDA. Default is
-DENABLE_CUDA_KERNEL_DEBUG=off- Enable debugging symbols for CUDA kernels. Will dramatically reduce kernel performance. Default is
-DENABLE_DECODERS_BOUNDS_CHECKING=off- Enable bounds checking for column decoding. Default is
-DENABLE_FOLLY=on- Use Folly. Default is
-DENABLE_IWYU=off- Enable include-what-you-use. Default is
-DENABLE_JIT_DEBUG=off- Enable debugging symbols for the JIT. Default is
-DENABLE_PROFILER=off- Enable google perftools. Default is
-DENABLE_STANDALONE_CALCITE=off- Require standalone Calcite server. Default is
-DENABLE_TESTS=on- Build unit tests. Default is
-DENABLE_TSAN=off- Enable thread sanitizer. Default is
-DENABLE_CODE_COVERAGE=off- Enable code coverage symbols (clang only). Default is
-DENALBE_JAVA_REMOTE_DEBUG=on- Enable Java Remote Debug. Default is
-DMAPD_DOCS_DOWNLOAD=on- Download the latest master build of the documentation /
docs.mapd.com. Default is
off. Note: this is a >50MB download.
-DPREFER_STATIC_LIBS=off- Static link dependencies, if available. Default is
sanity_tests target runs the most common tests. If using Makefiles to build, the tests may be run using:
AddressSanitizer can be activated by setting the
ENABLE_ASAN CMake flag in a fresh build directory. At this time CUDA must also be disabled. In an empty build directory run CMake and compile:
mkdir build && cd build cmake -DENABLE_ASAN=on -DENABLE_CUDA=off .. make -j 4
Finally run the tests:
export ASAN_OPTIONS=alloc_dealloc_mismatch=0:handle_segv=0 make sanity_tests
Additional testing and test scripts can be found by visiting Tests folder.
ThreadSanitizer can be activated by setting the
ENABLE_TSAN CMake flag in a fresh build directory. At this time CUDA must also be disabled. In an empty build directory run CMake and compile:
mkdir build && cd build cmake -DENABLE_TSAN=on -DENABLE_CUDA=off .. make -j 4
We use a TSAN suppressions file to ignore warnings in third party libraries. Source the suppressions file by adding it to your
Finally run the tests:
OmniSciDB uses CPack to generate packages for distribution. Packages generated on CentOS with static linking enabled can be used on most other recent Linux distributions.
To generate packages on CentOS (assuming starting from top level of the omniscidb repository):
mkdir build-package && cd build-package cmake -DPREFER_STATIC_LIBS=on -DCMAKE_BUILD_TYPE=release .. make -j 4 cpack -G TGZ
The first command creates a fresh build directory, to ensure there is nothing left over from a previous build.
The second command configures the build to prefer linking to the dependencies' static libraries instead of the (default) shared libraries, and to build using CMake's
release configuration (enables compiler optimizations). Linking to the static versions of the libraries libraries reduces the number of dependencies that must be installed on target systems.
The last command generates a
.tar.gz package. The
TGZ can be replaced with, for example,
DEB to generate a
Starting Using The StartOmniSci Wrapper
startomnisci wrapper script may be used to start OmniSciDB in a testing environment. This script performs the following tasks:
- initializes the
datastorage directory via
initdb, if required
- starts the main OmniSciDB server,
- offers to download and import a sample dataset, using the
Assuming you are in the
build directory, and it is a subdirectory of the
startomnisci may be run by:
It is assumed that the following commands are run from inside the
data storage directory. This command only needs to be run once.
mkdir data && ./bin/initdb data
Start the OmniSciDB server:
You can now start using the database. The
omnisql utility may be used to interact with the database from the command line:
./bin/omnisql -p HyperInteractive
HyperInteractive is the default password. The default user
admin is assumed if not provided.
Working With Data
Now that you're ready to start working with data, you can insert a sample dataset by running the
insert_sample_data script in a new terminal:
Additional data sets can be found in the Data Set Library.
There are several possible ways to load data to OmniSci using SQL commands.
- COPY FROM
- Importing AWS S3 Files
- HDFS with Sqoop
If you need a complete list and more detailed instructions, please visit the OmniSci documentation.
COPY ( <SELECT statement> ) TO '<file path>' [WITH (<property> = value, ...)];
must be a path on the server. This command exports the results of any SELECT statement to the file. There is a special mode when is empty. In that case, the server automatically generates a file in /omnisci_export that is the client session id with the suffix .txt.
Access a table of available properties in the optional WITH clause and more detailed instructions on the OmniSci documentation.
Learn more about how to add a logger to OmniSciDB by using instructions in Logger.md.
Contributed code should compile without generating warnings by recent compilers on most Linux distributions. Changes to the code should follow the C++ Core Guidelines.
.clang-format style configuration, based on the Chromium style guide, is provided at the top level of the repository. Please format your code using a recent version (8.0+ preferred) of ClangFormat before submitting.
clang-format -i File.cpp
.clang-tidy configuration is provided at the top level of the repository. Please lint your code using a recent version (6.0+ preferred) of clang-tidy before submitting.
clang-tidy requires all generated files to exist before running. The easiest way to accomplish this is to simply run a full build before running
clang-tidy. A build target which runs
clang-tidy is provided. To use:
clang-tidy may make invalid or overly verbose changes to the source code. It is recommended to first commit your changes, then run
clang-tidy and review its recommended changes before amending them to your commit.
clang-tidy target uses the
run-clang-tidy.py script provided with LLVM, which may depend on
PyYAML. The target also depends on
jq, which is used to filter portions of the
Specific Programming Languages
For more resources, please visit the primary resources for Java.
APIS, Interfaces & Projects
APIS & Interfaces
- omnisql: omnisql is the client-side SQL console that displays query results for SQL statements you submit to the OmniSciDB Server.
Thrift: OmniSciDB uses Apache Thrift to generate client-side interfaces. Interface definitions can be found in the
- JDBC:OmniSciDB supports Java Database Connectivity (JDBC) connections.
- ODBC: Open Database Connectivity (ODBC) is a standard application programming interface (API) for accessing database management systems (DBMS). OmniSciDB supports ODBC connections.
- Vega: Vega is a powerful, data-driven tool for specifying visualizations. The JSON Vega specification describes your data source and visualization properties.
- RJDBC: OmniSciDB supports the R language using RJDBC. RJDBC is a package implementing DBI in R on the basis of JDBC. This allows the use of any database management system (DBMS) in R through the JDBC interface.
- PyMapD: Python client for OmniSci GPU-accelerated SQL engine and analytics platform
- OmniSci.jl: Julia client for OmniSci GPU-accelerated SQL engine and analytics platform.
Developers are encouraged to contribute to this open source project. Check out the Contributing page to learn more.
If you have questions and would like to connect with the maintainers of this open source project, please visit the official online forum and community. or access additional resources on our Community page.
This project is licensed under the Apache License, Version 2.0.
The repository includes a number of third party packages provided under separate licenses. Details about these packages and their respective licenses is at ThirdParty/licenses/index.md.