OpenMLDB v0.5 Roadmap

[lumianph]

This is the roadmap for the OpenMLDB v0.5 release. As a community-driven project, we love your feedback. Please follow this thread to join our discussion:

• You may ask us about the detail of features on this roadmap.
• You are welcome to contribute to any items (even they are not on the list).
• Please reply this topic if you are planning to implement any features that are not on the list, and we will add them here gradually.

Release Date (Tentative)

v0.5.0: 25th Apr 2022

Online Core Engine

• Pre-aggregation optimization for performance enhancement
• The new RocksDB based storage engine

Offline Core Engine

• The new blocking APIs to execute offline queries
• Performance enhancement

CLI and SQL

• Support long timestamp type for csv import
• The new SQL syntax CREATE DATABASE IF NOT EXISTS
• Supporting blocking execution for async SQLs in CLI
• Supporting show jobs with a specific order
• Supporting offline queries to display results instead of job info only
• Supporting C UDF
• Supporting to set global system variable
• Supporting feature encoding: one-hot and label encoding
• Supporting SQL deployment for non-empty tables
• Supporting blocking execution for offline jobs in CLI, and showing returned results in CLI as well
• Long window aggregator support based on the pre-aggregation optimization: sum, count, min, max, avg (OpenMLDB RFC-010 : Long Time Window)

Java SDK

• The new API SqlClusterExecutor::executeSQL(sql)
• Supporting blocking execution for async SQLs
• Supporting the standalone version

Python SDK

• The new API openmldb.dbapi.connect().cursor().executemany(): batch insertion
• The new API openmldb.dbapi.connect().cursor().batch_row_request(): executing batch requests
• The new API openmldb.dbapi.connect().cursor().in_online_mode(): checking whether in the online mode
• The new API openmldb.dbapi.connect().cursor().get_tables(db): getting all tables of a given database
• The new API openmldb.dbapi.connect().cursor().get_all_tables(): getting all tables of all databases
• The new API openmldb.dbapi.connect().cursor().get_databases(): getting all databases
• The new API openmldb.dbapi.connect().cursor().executeRequest(SQL, parameter): executing a request
• The new API openmldb.dbapi.connect().cursor().get_resultset_schema(): getting the schema of the result set
• Supporting blocking execution for async SQLs
• Supporting the standalone version

Monitoring

• Enhancement for the online monitoring (OpenMLDB RFC-011 : Online Monitoring)

Ecosystem

• [Online data sources] Apache Kafka connector
• [Online data sources] Apache Pulsar connector
• [Deployment] Airflow provider package

Usability

• Migrating the Chinese documentation server to https://openmldb.ai/docs/zh
• English documentation translation and be accessible at: https://openmldb.ai/docs/en
• Performance benchmarking for online queries
• Building the use case of OpenMLDB + XGBoost
• Building the use case of OpenMLDB + Kafka
• Building the use case of OpenMLDB + Pulsar
• Building the use case of OpenMLDB + Airflow

[Shouren]

Hi,
Any details of the feature of "Supporting C UDF" in the section of "CLI and SQL" ? Does it support dynamic loading?

[lumianph]

Hi, Any details of the feature of "Supporting C UDF" in the section of "CLI and SQL" ? Does it support dynamic loading?

C UDF should be able to support dynamic loading. This feature is almost done. @dl239 may show some examples later.

[lumianph]

We are very pleased to announce that we have released our v0.5.0.

Highlights

• We have introduced an important performance optimization technique of pre-aggregation, which can significantly improve the performance for a query with time windows containing massive amount of rows, e.g., a few millions. (feat: add runner for RequestAggUnion and Reduce #1532 fix: support pre-aggregate in standalone #1573 feat: more aggregator op #1583 feat: aggregator recovery #1622 feat: add avg, count, min and max ops in runner #1627 fix: aggregator reset and empty aggr table #1672 # 1712 @zhanghaohit @nautaa)
• We have added a new storage engine that supports persistent storage (such as HDD and SSD) for the online SQL engine. Such a storage engine is helpful when a user wants to reduce the cost with acceptable performance degradation. (feat: add disktable to tablet and nameserver #1483 @Leowner)
• We have supported C/C++ based User-Defined Functions (UDFs) with dynamic registration to enhance the development experience. (feat: add external c++ udf #1509 feat: update external function proto for offline #1733 feat: support register external function for offline #1700 @dl239 @tobegit3hub)

Other Features

• Enhance the OpenMLDB Prometheus exporter ( feat(monitoring): openmldb prometheus exporter for v0.5.0 #1584, feat(monitoring): add mixin for grafana dashboard #1645, fix(grafana dashboard): datasource.uid should point to ${datasource} #1754 @aceforeverd )
• Support collecting statistics of query response time for online queries ( feat: create INFORMATION_SCHEMA.DEPLOY_RESPONSE_TIME on start #1497, feat: deploy response time collector #1521 @aceforeverd )
• Support new SQL commands: SHOW COMPONENTS, SHOW TABLE STATUS (feat: impl show components #1380 feat: support SHOW TABLE STATUS #1431 fix: refine error handling for SHOW COMPONENTS #1704 @aceforeverd)
• Support setting global variables (feat: add GLOBAL_VARIABLE table and support ShowGlobalVariables #1310 feat: support set global variable #1359 feat(planner): suport SET GLOBAL|SESSIOIN statement #1364 @keyu813 @aceforeverd)
• Support reading Spark configuration files from the CLI (feat: support reading spark conf file from cli #1600 @tobegit3hub)
• Support using multiple threads for the Spark local mode (feat: support spark local mode with multiple threads #1675 @tobegit3hub)
• Enhance the performance of join by using the Spark's native expression (feat: optimize join condition expr #1502 tobegit3hub)
• Support the validation for TaskManager configuration (feat: add check for taskmanager config #1262 @tobegit3hub)
• Support tracking unfinished jobs in the TaskManager (feat: add tracker service to track unfinished jobs #1474 @tobegit3hub)
• Other minor features (fix: add rows cnt in cmd #1601 @dl239; feat: add KeepAliveTime option in taskmanager  #1574 @vagetablechicken; feat: update default partition number and replica num in cluster mode #1546 @keyu813; feat: jdbc improvements for kafka #1729 @vagetablechicken; feat: support create table if not exists #1460 @tobegit3hub)

Bug Fixes

• Incorrect results when the order of conditions specified in where is different from that of the index (fix: 1356 combine index query #1709 @aceforeverd)
• Incorrect results of lag/at/lead under certain circumstances (fix: 1554 lag correctness #1605 fix(#1587): pure history rows range window iterator initialize position wrongly #1739 @aceforeverd)
• Memory leakage in zk_client (fix: memroy leak in zk_client.cc #1660 @wuxiaobai24)
• Catalog update failed if the role of a tablet is changed (fix: update catalog when change role #1655 @dl239)
• Related bugs about UnsafeRow for the offline engine (feat: fix unsaferow for window #1298, feat: support window union with unsaferow opt #1312, fix: fix string columns as null for UnsafeRowOpt #1326, feat: support unsaferow format for codec #1362, fix: fix unsaferow for window append slice #1637, feat: refactor row format with slice formant and single/multi slice row format #1381, feat: fix gflags of UnsafeRowOpt when running in yarn-cluster #1731 @tobegit3hub)
• Incorrect results after adding a new index in the standalone mode (fix: query result is incorrect after adding a new index in standalone version #1721 @keyu813)
• Incorrect results of SHOW JOBS under certain circumstances (feat: use sql for show jobs and show job command #1453 @tobegit3hub)
• Incorrect results of the date columns with UnsafeRowOpt(feat: support date udf with new date format for UnsafeRowOpt #1469 @tobegit3hub)
• Other minor bug fixes (fix(openmldb-import): add utf-8 source encoding in pom.xml, solve pro… #1698 @kfiring; fix: fix message when icu package not found #1651 @kutlayacar; feat: select into null exception #1621 @KaidoWang; feat: fix git plugin and return version of openmldb-batch #1150, feat: add print table in python sdk and show in notebook magic function #1243 @tobegit3hub; )

Code Refactoring

#1616 @dl239; #1743 @zjx1319

Acknowledgement

We appreciate the contribution to this release from all of our contributors, especially those from the community, including @nautaa @Leowner @keyu813 @wuxiaobai24 @kfiring @kutlayacar @KaidoWang @zjx1319

We are looking forward to your contribution!