The goal is to achieve a system of high concurrency, real-time write and analytical query optimization, with mixed workloads
-
Simple:
- Multi-Modal APIs for Rust/Python/Java/R/SQL
- Cloud native serverless On S3 or local
-
Feature-rich:
- Extensive SQL Plus support
- Transactions, persistence
- External Direct Parquet & CSV & Json querying
-
Fast
- Vectorized engine in SIMD
- Parallel query processing
- Optimized for analytics
- High-performance RowwiseRowBlock and RowwiseColumnarBlock format store: lstore and cstore
The foundational technologies in Hetu are:
- Apache Arrow memory model and compute kernels for efficient processing of data.
- DataFusion Query Engine for query execution
- Apache Arrow Flight Protocol for efficient data transfer between processes.
- Google Protocol Buffers for serializing query plans
Hetu can be deployed as a standalone cluster and also supports Kubernetes. In either case, the scheduler can be configured to use etcd as a backing store to (eventually) provide redundancy in the case of a scheduler failing.
Hetu is currently a proof-of-concept based on datafusion and provides batch execution of SQL queries. Although it is already capable of executing complex queries, it is not yet scalable or robust.
There is an excellent discussion in #1 about the future of the project and we encourage you to participate and add your feedback there if you are interested in using or contributing to Hetu.
The current initiatives being considered are:
- Supports reliable raft-based meta storage
- Supports lightweight storage based on lsm-tree real-time writes and reads
- Continue to improve the current batch-based execution
- Add support for low-latency query execution based on a streaming model
The easiest way to get started is to run one of the standalone or distributed examples. After that, refer to the Getting Started Guide.
$ mysql -h 127.0.0.1 -P 3307 -u root -p123
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 5.7.25-HetuDB-v0.1.0-alpha HetuDB Server (Apache License 2.0)
Community Edition, MySQL 5.7 compatiible
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
mysql> select 'hello, hetu';
+---------------------+
| Utf8("hello, hetu") |
+---------------------+
| hello, hetu |
+---------------------+
1 row in set (0.24 sec)
mysql> explain select 'hello, hetu' limit 10;
+---------------+----------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+----------------------------------------------------------------------------------------------------------------------------+
| logical_plan | Limit: 10
Projection: Utf8("hello, hetu")
EmptyRelation |
| physical_plan | GlobalLimitExec: limit=10
ProjectionExec: expr=[hello, hetu as Utf8("hello, hetu")]
EmptyExec: produce_one_row=true |
+---------------+----------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.23 sec)
- Refer to the developer documentation for the Architecture Overview
Please see Contribution Guide for information about contributing to HetuDB.