Mirror of Apache Quickstep Incubator
C++ CMake Other
Switch branches/tags
Nothing to show
Clone or download
Permalink
Failed to load latest commit information.
build QUICKSTEP-40: Fix Copyright notice to confirm to Apache. Aug 7, 2016
catalog Fix and rearrange all CMakeLists.txt so that they are ready to be pro… Apr 27, 2018
cli Fixed the command execution bug in the distributed version. May 21, 2018
cmake Refactored out proto GenerateCPP function Apr 6, 2017
compression Fix and rearrange all CMakeLists.txt so that they are ready to be pro… Apr 27, 2018
expressions QUICKSTEP-121: Added the self-join support. May 9, 2018
parser Minor bug fixes and refactors. May 9, 2018
query_execution Fix the inclusion guard of ForemanSingleNode.hpp file Jun 22, 2018
query_optimizer Fix a bug of filter side in HashJoinOperator Jun 5, 2018
relational_operators Fix a bug of filter side in HashJoinOperator Jun 5, 2018
release Added script for release license audit Mar 17, 2017
storage Fix a bug of filter side in HashJoinOperator Jun 5, 2018
third_party Fixed the build issues regarding tmb benchmark. Jun 19, 2018
threading Fix and rearrange all CMakeLists.txt so that they are ready to be pro… Apr 27, 2018
transaction Fix and rearrange all CMakeLists.txt so that they are ready to be pro… Apr 27, 2018
types Fix and rearrange all CMakeLists.txt so that they are ready to be pro… Apr 27, 2018
utility Minor bug fixes and refactors. May 9, 2018
yarn Fix and rearrange all CMakeLists.txt so that they are ready to be pro… Apr 27, 2018
.gitattributes Init release scripts Mar 1, 2017
.gitignore Make the third party directory leaner. Jan 28, 2017
.gitmodules Make the third party directory leaner. Jan 28, 2017
.travis.yml Add a flag to allow disabling of Comparison inline expansion to enabl… Feb 2, 2018
BUILDING.md Refactor building.md Feb 9, 2017
CMakeLists.txt Data Provider thread added. Jun 4, 2018
CREDITS.md QUICKSTEP-40: Fix Copyright notice to confirm to Apache. Aug 7, 2016
DEV_README.md Added header to scripts and yaml files, updated NOTICE Mar 8, 2017
DISCLAIMER QUICKSTEP-1: Add incubation disclaimer to README.md. Jun 8, 2016
Doxyfile Initial commit of Quickstep Jan 16, 2016
GENERATOR_FUNCTIONS.md QUICKSTEP-40: Fix Copyright notice to confirm to Apache. Aug 7, 2016
HISTORY.md Added header to scripts and yaml files, updated NOTICE Mar 8, 2017
LICENSE Update license file to summarize 3rd party Mar 17, 2017
NOTICE Update license file to summarize 3rd party Mar 17, 2017
README.md Remove double can. Thanks @zuyu for catching this. Apr 29, 2017
WORKING_WITH_AN_IDE.md IDE Documentation fixes Jan 11, 2018
autofix_cmakelists.py Fix a bug of filter side in HashJoinOperator Jun 5, 2018
cyclic_dependency.py Fixed the root path check in the cyclic_dependency.py. Oct 3, 2017
empty_src.cpp QUICKSTEP-40: Fix Copyright notice to confirm to Apache. Aug 7, 2016
lint_everything.py Upgrade cpplint Dec 21, 2017
validate_cmakelists.py Fixed the root path check in the validate_cmakelists script. Sep 29, 2017

README.md

Apache Quickstep (Incubating)

Travis Widget

What is Quickstep?

Apache Quickstep is high-performance database engine designed to exploit the full potential of hardware that is packed in modern computing boxes (servers and laptops). The initial version (available now!) targets single-node in-memory environments. If your data spills overs the memory limit Quickstep will still work, so you don't have to obsessively worry about the in-memory part. Also, if your working set fits in memory then Quickstep will transparently and automatically figure that out, and cache that hot set to deliver in-memory performance.

Distributed execution is the next big feature for Quickstep.

Quickstep began life in 2011 as a research project at the University of Wisconsin and entered incubation at the Apache Software Foundation in April, 2016.

Why Quickstep?

Did you know that the hardware that you have in your laptop was spread across a small cluster just a decade ago? (PS: Hopefully you are not using a very old laptop!) If you look at a high-end server box, then that packs compute and storage power that was a full rack about 5 years ago! And, the way hardware technology is going, that box is going to become even more powerful in the future. In fact, it is likely that the computing power in each box is going to grow faster than other hardware components (e.g. networking) in data centers. So, if you care about performance and/or total operating costs, paying attention to single box performance is likely to be super important in the long run.

In other words there is a small data center in an individual compute boxes today! Quickstep aims to allow you to fully exploit the potential of that data center that is hidden in each individual box today. We call this the scaling-in approach, and it complements a scaling-out approach. But without scaling-in, you are overpaying (by a lot!) when you run your data service.

What are the key ingredients?

Modern computing boxes contain a large number of computing cores and large main memory configuration. Quickstep allows you to fully exploit these hardware resources using novel data processing, data storage, and query processing methods that include:

  1. A unique decoupling of data-flow from control-flow for query execution that allows for unlimited intra and inter-query parallelism. Thus, using all the processing core effectively.

  2. A template meta-programming framework that provides fast vectorized query execution. Thus, using each processor cycle very efficiently.

  3. A hybrid data storage architecture that includes columnar and row-store. Yes, this may surprise some of you, but sometimes a row-store beats a column-store!

And, it is open source!

Giving it a spin

  1. Checkout the code: git clone https://git-wip-us.apache.org/repos/asf/incubator-quickstep.git quickstep
  2. Then, go to the code directory: cd quickstep
  3. Initialize the dependencies: git submodule init
  4. Checkout the dependencies: git submodule update
  5. Download additional third-party dependencies and apply patches:
    cd third_party && ./download_and_patch_prerequisites.sh && cd ../
  6. Go into the build directory: cd build
  7. Create the Makefile: cmake -D CMAKE_BUILD_TYPE=Release ..
  8. Build: make -j4. Note you may replace the 4 with the number of cores on your machine.
  9. Start quickstep: ./quickstep_cli_shell --initialize_db=true. You can now fire SQL queries. To quit, you can type in quit; Your data is stored in the directory qsstor. Note the next time you start Quickstep, you can omit the --initialize_db flag (as the database has already been initialized), and simply start Quickstep as: ./quickstep_cli_shell. There are also a number of optional flags that you can specify, and to see the full list, you can type in: ./quickstep_cli_shell --help
  10. Next let us load some data and fire some queries. A few points to note: The SQL surface of Quickstep is small (it will grow over time). The traditional SQL CREATE TABLE and SELECT statements work. The data types that are supported include INTEGER, FLOAT, DOUBLE, VARCHAR, CHAR, DATE, and DATETIME. Quickstep also does not have support for NULLS or keys (yet). Let us create two tables by typing into the Quickstep shell (which you opened in the step above), the following SQL command:
CREATE TABLE Weather (cid INTEGER, recordDate DATE, highTemperature FLOAT, lowTemperature FLOAT);

and then,

CREATE TABLE City (cid Integer, name VARCHAR(80), state CHAR(2));
  1. Next, let us insert some tuples in these two tables.

    INSERT INTO City VALUES (1, 'Madison', 'WI');
    INSERT INTO City VALUES (2, 'Palo Alto', 'CA');
    INSERT INTO Weather VALUES (1, '2015-11-1', 50, 30);
    INSERT INTO Weather VALUES (1, '2015-11-2', 51, 32);
    INSERT INTO Weather VALUES (2, '2015-11-1', 60, 50);
    
  2. We can now issue SQL queries such as: a. Find all weather records for California:

SELECT * FROM WEATHER W, City C WHERE C.cid = W.cid AND C.state = 'CA';

b. Find the min and max temperature for each city, printing the cid:

SELECT cid, MIN(lowTemperature), MAX(highTemperature) FROM Weather GROUP BY cid;

c. Find the min and max temperature for each city using a nested query, and printing thie city name:

SELECT * FROM City C, (SELECT cid, MIN(lowTemperature), MAX(highTemperature) FROM Weather GROUP BY cid) AS T WHERE C.cid = T.cid;
  1. Quickstep also supports a COPY TABLE command. If you want to try that, then from a separate shell file type in the following:

    echo "3|2015-11-3|49|29" > /tmp/tmp.tbl
    echo "3|2015-11-4|48|28" >> /tmp/tmp.tbl
    echo "3|2015-11-5|47|27" >> /tmp/tmp.tbl
    

    Then, load this new data by typing the following SQL in the Quickstep shell:

    COPY Weather FROM '/tmp/tmp.tbl' WITH (DELIMITER '|');
    

    Now, you have loaded three more tuples into the Weather table, and you can fire the SQL queries above again against this modified database.

    Remember, to quit Quickstep, you can type in quit; into the Quickstep shell.

Additional pointers

  1. For other build options, see the more comprehensive build guide.
  2. To get started as a developer, you should start with the code organization guide.

Licensing

Quickstep is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

Disclaimer

Apache Quickstep is an effort undergoing incubation at the Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC.

Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects.

While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.