Skip to content
Permalink
Browse files
weekly 20220130
  • Loading branch information
@morningman-cmy
morningman-cmy committed Jan 31, 2022
1 parent fdd77cd commit 915277ff6420ff7487f8ffe3b8481681fdf52870
Showing 2 changed files with 282 additions and 0 deletions.
127 blogs/en/weekly-20220130.md
View file
@@ -0,0 +1,127 @@
---
{
"title": "[Doris Weekly] 20220103-20220116",
"description": "Doris Weekly",
"date": "2022-01-23",
"metaTitle": "[Doris Weekly] 20220103-20220116",
"isArticle": true,
"language": "en",
"author": "morningman",
"layout": "Article",
"sidebar": false
}
---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

## Statistics

A total of 28 authors have submitted 54 Commits. Thanks to the following authors for their contributions (Github IDs, in alphabetical order).

924060929, adonis0147, Aiden-Dong, caiconghui, chovy-3012, eyesmoons, GoGoWen, Henry2SS, hf200012, jackwener, littleeleventhwolf, liutang123, lovingfeel, luzhijing, morningman, qidaye, qzsee, tianhui5, Userwhite, vinson0526, wayofeng, weajun, WindyGao, xuzifu666, xy720, yangzhg, zenoyang, zuochunwei

In the last 2 weeks, 4420 new lines of code were added and 1998 lines of code were deleted.

## Main progress

### New features

* https://github.com/apache/incubator-doris/pull/7403

SQL block rule adds restrictions on partition num, tablet num and cardinality. You can set these parameters to block SQL queries that do not match the rule.

* https://github.com/apache/incubator-doris/pull/7673

Added WEEKDAY function.

### Feature improvements

* https://github.com/apache/incubator-doris/pull/7423

Add `repair_slow_replica` parameter to automatically remove replicas with too many versions.

* https://github.com/apache/incubator-doris/pull/7240

Optimize the return result of the load command. When load data is empty, Success is returned instead of `all partitions have no load data`.

* https://github.com/apache/incubator-doris/pull/7574

Runtime Filter supports determining whether to use IN or Bloom Filter type at runtime.

* https://github.com/apache/incubator-doris/pull/7568

Optimize the DataX doriswriter plugin to use json format by default to avoid problems with conflicting column separators.

* https://github.com/apache/incubator-doris/pull/7514
* https://github.com/apache/incubator-doris/pull/7531

Optimized error handling logic on the Spark Connector write side.

* https://github.com/apache/incubator-doris/pull/7664

Support more hint syntax to enhance mysql compatibility.

* https://github.com/apache/incubator-doris/pull/7630

Routine Load support for older versions of kafka (< 0.10)

* https://github.com/apache/incubator-doris/pull/7543

Optimized locking logic for BE-side RPC client cache.

* https://github.com/apache/incubator-doris/pull/7754

Optimize the execution logic of Routine Load to reduce the frequency of RPC timeout during Routine Load execution.

### Bug Fixes

* https://github.com/apache/incubator-doris/pull/7563

Fix a problem where Decommission Backend may be blocked due to `no proper tag is chosen for tablet` error in some cases.

* https://github.com/apache/incubator-doris/pull/7607

Fix an issue where the Audit Log plugin would exit under certain exceptions.

* https://github.com/apache/incubator-doris/pull/7605

Fix a concurrent locking issue that may be caused by creating error hubs in some cases.

* https://github.com/apache/incubator-doris/pull/7691

Fix the issue that using S3 SDK cannot access AliCloud object storage.

* https://github.com/apache/incubator-doris/pull/7721

Fix bugs related to Lateral View.

* https://github.com/apache/incubator-doris/pull/7526

Fix a bug that FE Metadata Checkpoint operation may rewrite the current metadata value incorrectly in some cases.

### Other

* https://github.com/apache/incubator-doris/pull/7616

Added process support for pushing Spark/Flink Connector to Maven repositories.

* https://github.com/apache/incubator-doris/pull/7638

Optimized the content of Pull Request Template.
155 blogs/zh-CN/weekly-20220130.md
View file
@@ -0,0 +1,155 @@
---
{
"title": "[Doris 周报] 20220117-20220130（虎年大吉）",
"description": "Doris 周报 20220117-20220130",
"date": "2022-01-30",
"metaTitle": "[Doris 周报] 20220117-20220130",
"isArticle": true,
"language": "zh-CN",
"author": "morningman",
"layout": "Article",
"sidebar": false
}
---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

这是中国农历辛丑年牛年的最后一篇周报。在此先祝各位虎年大吉，万事如意！

## 统计数据

共 46 位作者提交了 67 个 Commit 。感谢以下作者的贡献（Github ID，按字母序）：

924060929, adonis0147, amosbird, anneji-dev, awakeljw, bingzxy, BiteTheDDDDt, blackstar-baba, caiconghui, Cui Kaifeng, emmymiao87, EmmyMiao87, freemandealer, Gabriel, HappenLee, Henry2SS, hf200012, htyoung, jackwener, JNSimba, luozenglin, luzhijing, morningman, obobj, Pxl, qidaye, qzsee, stdpain, steadyBoy, taberylyang, thinker, ucasfl, wangbo, WangShuo, weizuo93, xinghuayu007, yangzhg, yiguolei, zbtzbtzbt, zenoyang, zhangstar333, ZhengguoYang, zhengshengjun, zhengshiJ, zhoubintao, zuochunwei

最近 2 周，共修改新增代码行 339581 ，删除代码行 264318。

## 主要进展

### 新增功能

* https://github.com/apache/incubator-doris/pull/7785

向量化引擎合入主干分支。感谢19位开发者在长达半年多的时间里，以开源协作的方式共同完成了如此庞大的工程。后续该功能会继续在主干分支进行完善和优化。

* https://github.com/apache/incubator-doris/pull/7521

新增 ADMIN COMPACT 语句支持触发指定分区的compaction任务。

* https://github.com/apache/incubator-doris/pull/7457

Flink Connector 增加对 delete flag 的支持，方便通过 Flink CDC 同步上游数据库的更新和删除操作。

* https://github.com/apache/incubator-doris/pull/7457

新增 `show proc "/current_query_stmts"` 语句已查看当前正在进行的 SQL。

### Work-in-Progess

* https://github.com/apache/incubator-doris/pull/7391

新增对 Iceberg 外表的支持。

### 功能改进

* https://github.com/apache/incubator-doris/pull/7656

新增 `colocate_group_relocate_delay_second` 来延迟因BE短暂停机而可能触发的colocation 表副本重分布操作。

* https://github.com/apache/incubator-doris/pull/6319

优化导入逻辑，在错误容忍度为0的情况下，当遇到错误数据行时快速取消导入任务，以方便快速定位问题。

* https://github.com/apache/incubator-doris/pull/7674

增强导入作业的错误容忍度，在少数节点写入失败的情况下，依然可以完成导入任务。

* https://github.com/apache/incubator-doris/pull/7821

`show proc "/statistic"` 新增查看 compaction 较慢的副本以及数据量过大的副本。

* https://github.com/apache/incubator-doris/pull/7876

优化了 Datetime 类型的改写逻辑，使其和 MySQL 更加兼容。

* https://github.com/apache/incubator-doris/pull/7851

通过 EXPLAIN VERBOSE 语句查看 SQL 执行计划，可以查看隐式 cast 的改写情况。

* https://github.com/apache/incubator-doris/pull/7630

Routine Load 支持老版本的 kafka(< 0.10)

* https://github.com/apache/incubator-doris/pull/7543

优化了 BE 端 RPC client cache 的锁逻辑。

* https://github.com/apache/incubator-doris/pull/7754

优化Routine Load 的执行逻辑，以降低 Routine Load执行过程中出现 RPC timeout 的频率。

* https://github.com/apache/incubator-doris/pull/7546

优化 Runtime Filter 逻辑，使其可以自适应的使用 IN 或 BloomFilter 类型。

### Bug 修复

* https://github.com/apache/incubator-doris/pull/7772
* https://github.com/apache/incubator-doris/pull/7865

修复Lateral View 可能导致BE宕机或查询结果错误的bug。

* https://github.com/apache/incubator-doris/pull/7788

修复某些情况下，bitmap索引可能导致查询结果不正确的问题。

* https://github.com/apache/incubator-doris/pull/7808

修复某些情况下，schema change 会返回 -102 错误的问题。

* https://github.com/apache/incubator-doris/pull/7813

修复某些情况下，使用 export 语句导出 unique key 表时报错的问题。

* https://github.com/apache/incubator-doris/pull/7888

修复 digital_masking 函数数值溢出的bug。

* https://github.com/apache/incubator-doris/pull/7804

修复 fe audit plugin 会因为某些异常情况而彻底退出的问题。

* https://github.com/apache/incubator-doris/pull/7800

修复 grouping set 的相关错误。

* https://github.com/apache/incubator-doris/pull/7931

修复部分情况下，使用窗口函数并返回 empty set 时，会导致 BE 宕机的问题。

* https://github.com/apache/incubator-doris/pull/7927

修复部分情况下，使用 OUTFILE 功能可能返回 Empty Partition Info 错误的问题。

### 其他

* https://github.com/apache/incubator-doris/pull/7712

编译方式更新为使用 ldb-toolchain 工具编译，以屏蔽在不同系统下的编译组件差异。

0 comments on commit 915277f

Please sign in to comment.