Release Notes 0.12.0

[chaoyli]

Release Notes 0.12.0

New Features

Segment V2

We have re-implemented the underlying data storage format, using Page-Based logical concepts to organize different data types or index types. We call this storage format Segment V2.
Based on the new storage format, we have introduced a series of new features, including dictionary compression and bitmap indexing, to improve Doris's read and write performance.

[#1305]

Bitmap Index

On the basis of Segment V2, Doris implements the function of adding bitmap indexes to the specified columns. This feature significantly enhances Doris's query performance for any combination of multi-dimensional scenarios.

[#2487] [#2050] [#2319]

Spark on Doris

With spark-doris-connector, users can now query data stored in Doris directly through Spark. Better meet users' federal query needs.

[#2228]

Support materialized view on duplicate mode

Supports the creation of Materialized Views tables based on the Duplicate data model, which not only meets the needs of users directly using detailed table analysis, but also meets the efficient query of some specific dimension analysis.

[#2101]

Support INTERSECT and EXCEPT operators

[#2834]

Support Loading ORC format file

Supports reading and loading ORC format files stored on HDFS.

[#2554]

Create partition automatically

[Dynamic Partition] Support for automatically adding partitions

Doris can now automatically create partitions through scheduled tasks, reducing user maintenance costs for partition operations.

[#2262]

Grouping Set SQL syntax

Supports Grouping Set SQL syntax. Can further reduce the complexity of writing SQL for data analysts.

[#2293]

Enhancement

Doris on ES Performance Enhancement

Significantly enhanced query performance for Doris on Es. Doris on ES is probably the best open source "SQL on ES" solution.

[#2237] [#2575]

Bitmap aggregate type support BIGINT

The integer type supported by the Bitmap aggregation method has been extended from 32 bits to 64 bits. Further strengthen the scope of application of Doris's precise deduplication function.

[#2772]

Supports conversion of more column types

Now Doris support conversion of more column types by altering table.

Float -> Double
Datetime <-> Date
Int -> Date
Varchar -> TINTINT/SMALLINT/INT/BIGINT/LARGEINT/FLOAT/DOUBLE/DATE

[#2310] [#2393] [#2481] [#2489] [#2501]

NIO-based MySQL protocol connection layer

A new MySQL protocol connection layer based on NIO is implemented, which can support highly concurrent connection requests.

[Experimental, enable it by setting FE config mysql_service_nio_enabled to true]

[#2603]

More convenient information viewing

Now we support adding filter conditions for following stmts:

• SHOW ALTER TABLE [add where/orderby/limit after “SHOW ALTER TABLE COLUMN“ syntax #2380]
• SHOW PARTITIONS [Add filter conditions for 'show partitions from table' syntax #2553]
• SHOW FUNCTIONS [Change replace bzip and boost sources #2880]

New Built-in Functions

• bitmap_or [ADD BE UDF bitmap_and and bitmap_or #2649]
• bitmap_and [ADD BE UDF bitmap_and and bitmap_or #2649]
• bitamp_to_string [Add bitamp_to_string function #2731]
• bitmap_from_string [Add bitamp_to_string function #2731]
• bitmap_contains [Add be bitmap udf  #2752]
• bitmap_has_any [Add be bitmap udf  #2752]
• bitmap_union_count [Add bitmap_union_count function #2425]
• bitmap_hash [Add bitmap_hash function #2439]
• intersect_count [Add intersect_count UDAF #2418]
• ends_with [Support ends_with function #2746]
• curdate [Add curdate function in Apache Doris #2521]

Other

• Support aggregation type of REPLACE_IF_NOT_NULL [Support aggregation type of REPLACE_IF_NOT_NULL #2127]
• Introduce GSON library to serialize the meta data in FE [[Tag System] Add classes related to "tag". #2343]
• Support correlated non-scalar subquery [Support correlated non-scalar subquery #2468]
• Support ALTER VIEW stmt [Alter view stmt #2522]
• Support replication_num setting for table level [Support replication_num setting for table level #2737]
• SupportBroker reading config from hdfs-site.xml [Support hdfs broker reading config from hdfs-site.xml #2148]
• Support date_add function for partition pruning [Add date_add/sub in fe to support partition prune #2154]
• Optimize compaction strategy of tablet on BE [Optimize compaction strategy of tablet on BE #2473]
• Support decompressing csv file with deflate format in hdfs broker load [Support decompressing csv file with deflate format in hdfs broker load #2583]
• Support sql mode [Support sql mode #2083]
• Modify the authorization checking logic [Modify the password and privilege checking logic #2372]
• Improve SkipList memory usage tracking [Improve SkipList memory usage tracking #2359]
• Create table with nullable column for default [Create table with nullable column for default #2256]
• Enable decimalV2 cast to different precision [Decimal(a,b) cast to Decimal(b,c) is forbidden #2135]

Thirdparty

• Bump Apache Maven version to 3.6.3 in Docker dev image [Dockerfile: update maven to 3.6.3 #2395]
• Bump JMockit version to 1.48 [Upgrade JMockit from version 1.13 to 1.48 #2423]
• Bump Apache Arrow version to 0.15.1 [Adapt arrow 0.15 API #2657] [update arrow's version to 0.15.1 and shaded it in spark-doris-connector #2769]

API Change

Default Table Storage Model

Change default storage model from AGGREGATE KEY to DUPLICATE KEY

[#2318]

Forbid Cluster Related Operations

Forbid cluster related operations:

• create/drop cluster
• add free backend/add backend to cluster/decommission cluster balance
• change the backends num of cluster
• link/migration db

[#2429]

Credits

Thanks to everyone who contributed to this release!

@EmmyMiao87
@HangyuanLiu
@HappenLee
@IanWatsons
@JustinTsui
@Lishi-hub
@MLikeWater
@Seaven
@WingsGo
@wsyzxxn
@Youngwb
@blackfox1983
@caiconghui
@chaoyli
@firetree01
@frwrdt
@gaodayue
@imay
@infearOnTheWay
@kangkaisen
@kangpinghuang
@landon-dai
@lingbin
@liutang123
@lshmouse
@marising
@morningman
@shengyunyao
@vagetablechicken
@vinson0526
@wangbo
@wkhappy1
@wubiaoi
@wutiangan
@wuyunfeng
@xhg-tech
@xy720
@yangzhg
@yuanfeng0905
@zhouhaibing089

[kangkaisen]

API change should include #2256

[gaodayue]

The ongoing work to support complex data type may introduce incompatible changes to V2 format, so the release should pause until we finalized V2 format.

[morningman]

The ongoing work to support complex data type may introduce incompatible changes to V2 format, so the release should pause until we finalized V2 format.

What do you mean incompatible ? Why there is an incompatible feature on going?

[gaodayue]

What do you mean incompatible ? Why there is an incompatible feature on going?

It means incompatible with the segment_v2 file format in master branch. It's OK because segment_v2 is not released yet.

[chaoyli]

The incompatibility has been solved. Restart release 0.12.0.

[EmmyMiao87]

@chaoyli Support materialized view on duplicate mode instead of Support aggregated materialized view framework

[EmmyMiao87]

@chaoyli Please include the new feature of query in tpc-ds. The feature list is https://github.com/apache/incubator-doris/projects/6#card-34831609

The grouping set, sql mode, intersect, except, non-scalar subquery

[chaoyli]

@chaoyli Please include the new feature of query in tpc-ds. The feature list is https://github.com/apache/incubator-doris/projects/6#card-34831609

The grouping set, sql mode, intersect, except, non-scalar subquery

all have added

[gaodayue]

I just notice that the default result ordering of SHOW PARTITIONS statements is changed from ascending PartitionId in 0.11 to descending PartitionId. This causes some of our existing tools to break because they relies on the ascending behavior of 0.11.

Is it a bug or a feature? If it's not a bug, we'd better include this in the API Change of release note.

[morningman]

Is it a bug or a feature? If it's not a bug, we'd better include this in the API Change of release note.

SHOW PARTITIONS is now supporting ORDER BY clause.
But I think this should be changed back to the origin to avoid your problem.
Could you submit a PR for this?

[gaodayue]

But I think this should be changed back to the origin to avoid your problem.
Could you submit a PR for this?

Sure, I submitted #3184 for it