fix(tianmu): To fixup the instance crashed if the result of aggregate is out of boundary #1856
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for the contribution! Please review the labels and make any necessary changes. |
haitaoguan
approved these changes
Jun 6, 2023
… function goes out of bounds To caculate `sum(length())` will lead a corruption in destructor of `ValueOrNull`, frees an array of char. Before the array deletes, the validity of that should be checked, and after that the pointer of that arrary should also be set to nullptr, which make sure it's a safe code piece.
|
Merge Mannually |
3 tasks
RingsC
added a commit
that referenced
this pull request
Jun 29, 2023
* docs(downlaod)：update the docs of download(#1453) * fix(mtr):Resolve the nightly run error problem(#1458) * fix(workflow): fix can not run lcov in workflow * feat(tianmu):New configuration parameters: "tianmu_mandatory" and "tianmu_no_key_error (#1462)" In version 1.0.4, we will discard "MANDATORY_TIANMU" and "NO_KEY_ERROR" in (sql_mode) Specifies whether to enable mandatory Tianmu engine in table. if yes ,set tianmu_mandatory to ON, otherwise set the variable to OFF. Specifies whether to to directly skip DDL statements that are not supported by the SQL layer, instead of reporting errors. if yes ,set tianmu_no_key_error to ON, otherwise set the variable to OFF. * feat(tianmu):Discard "MANDATORY_TIANMU" and "NO_KEY_ERROR" in (sql_mode) * feat(tianmu):Resolve the issue of warning messages when setting new parameters * fix(tianmu):(Primary/Secondary)Error 1032 occasionally occurs during primary/secondary synchronization if UUIDs are used as the primary key(#1464) Cause of the problem: When performing a primary key scan under the master slave, "ha_tianmu:: position()" is called first to obtain the primary key value from the "record", However, in this scenario, after calling the "key_copy()" function, the "record" will be cleared, resulting in the subsequent "GetKeys()" obtaining a null primary key value. Solution: Because the value "handler->ref" is not used in the future, you can simply delete the call to "key_copy()". * test(mtr): add integer/unsigned/in subquery/create temporary testcases and update escape.test(#1196) * docs(v1.0.3): update the docs for v1.0.3 * fix(dcoker): FixDocker deployment commands(#1499) * feat: add Baidu statistics script(#1498) * fix(website): fix Baidu statistics script(#1502) * test(mtr): Optimize parallel scheduling to execute more innodb engine testcases, add date type and std func testcase(#1196) * fix crash when the aggregated element was decimal (#1402) 1. Fix the crash first 2. then redesign the entire aggregated data stream * build(deps): bump nth-check and unist-util-select in /website Bumps [nth-check](https://github.com/fb55/nth-check) to 2.1.1 and updates ancestor dependency [unist-util-select](https://github.com/syntax-tree/unist-util-select). These dependencies need to be updated together. Updates `nth-check` from 1.0.2 to 2.1.1 - [Release notes](https://github.com/fb55/nth-check/releases) - [Commits](fb55/nth-check@v1.0.2...v2.1.1) Updates `unist-util-select` from 2.0.2 to 4.0.1 - [Release notes](https://github.com/syntax-tree/unist-util-select/releases) - [Commits](syntax-tree/unist-util-select@2.0.2...4.0.1) --- updated-dependencies: - dependency-name: nth-check dependency-type: indirect - dependency-name: unist-util-select dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * docs(deploy): Update the document and fix the link * update the copyright * update the readme(#1324) * Create FUNDING.yml update the Sponsor button * Update FUNDING.yml fix the link of opencollective * remove dup lines in git workflow * feat(tianmu): mv func from public to protected(#1501) * style change * strip spaces * remove dup lines * change to uint * fix(tianmu):Resolve DDL and insert inot select Possible space inflation issues(#366) Cause of the problem: 1. For multiple versions, Tianmu needs to first copy an original pack when performing DML operations, Modify the copied package and use append write or overwrite write after modification (If there is invalid space in the DATA file that can be written to the current pack, use overwrite write, otherwise use append write) to write to the file, After the latest package is written to a file, the latest version chain will point to the address that was last written. There is a problem with the current (TianmuAttr:: LoadData) logic. Every time you call (TianmuAttr:: LoadData), Will write data to disk, If there are multiple rows written in a transaction, there will be multiple copies of data, "Because the current transaction has not been committed, the space for previous repeated writes has not been released, so the logic of overwriting writes will not be reached.", "I only follow the logic of additional writing, which is the fundamental reason for the skyrocketing space.". If you encounter a particularly large multiline write transaction, it will lead to a space explosion. Moreover, disk IO is performed once per load line, which can also lead to degraded insert performance. Solution: To optimize the logic of (TianmuAttr:: LoadData), it is necessary to determine whether the data in the pack is full before saving changes, Is whether to reach 65536 lines, and if so, write again, If it cannot be reached, it is necessary to write again in the commit phase. * feat(tianmu):Reconstruct direct insert into parallel execution, improving direct insert performance. * feat(tianmu):Add code comments for easy understanding * fix(tianmu):The myloader cann't work if the autocommit is closed. (#1510) Currently, TIANMU does not support manual transactions and only supports automatic commit. However, it does determine whether to commit the transaction based on the automatic commit parameters of MySQL. If automatic commit is turned off, automatic commit will not be performed on the transaction * fix(core): fix bug: The instance occasionally crashes if both fields specified for an equi-join are of the string data types (#1476) * fix: page hover font style * feat(mtr): To fix the mtr usage The disabeld mtr use cases should follow the mtr conventional rules. Adds a disabled.def to indeciate which case is disabled or not. * fix(tianmu):The mysqld is crashed when you are starting replication.(#1523) It is possible to update null values with null values in the delta layer, For example: update t set name="xiaohua" where id=1; update t set name=null where id=1; So when encountering this situation, directly return * feat(tianmu):Add delta layer mtr * bug 1538:The instance occasionally crashes when the parallel degree is enabled of the right table #1538 * feat(tianmu): To support vocalno framwork The PRs to support vocalno framwork include a serials of PR. Part1: Refine framework of code to make it clean and clear to read. * docs(quickstart): add stonedb-8.0 compiling guide #1449 * fix(website): fix website error #1449 * feat(tianmu): support volcano framewrok (#1546) * feat(tianmu): To support volcano framework This is a serial of PRs, part2: To remove global var `ha_kvstore`, which should be in an engine. To support volcano framework. and In order to do that, at secondly, ha_kvstore_, a global variable removed to the engine. That follows innodb conventional rules. * fix(tianmu): revert code, mv ret value from try block back to catch block [summary] The logic of this modification is as follows: Previously, set ret action has been moved to try block, which is not efficient because every time we do truncate success, ret will be setted to 1. This time we will move the set ret action back to catch block, which will only trigged when truncate failed. * feat(tiamnu): hard code in defs.h (#1481) change magic number to readable const * docs:update the compile guides #1562 * test(mtr): add more innodb testcases and tianmu range testcase(#1196) * fix(tianmu):Remove excess log printing and add some code comments(#1545) * fix(tianmu): fix mysqld crash when exec query with AggregateRough, assert failed on i < m_idx.size() at tianmu_attr.h:387, msg: [bad dpn index 0/0] (#1580) * fix(website): fix the download link of 5.7(#1518) * feat(website): update the latest content(#1587) * feat(tianmu): support volcano framework (#1554) Part3: To remove the `ha_tianmu_engine`, and gets it from hton's data. This makes it behavior just like innodb. MySQL gets innodb handler instance from table->s->file. and it will make the code logic more concise. * fix: max-width navbar search style * feat(website): upgrade the docusaurus version(#1604) fix #1604 * fix(website): fix Roadmap module location(#1597) * website(community): update the content * feat(website): update the logo of XinChuang(#1590) * fix(tianmu):The instance occasionally crashes when the memory leak. (#1549) * fix(tianmu):Modify merge_ The assignment method of ID, delaying the assignment to ensure that the final value is correct * fix(tianmu):Fix bug in delta layer initialization * fix(tianmu):Resolve the issue of assertion failure caused by memory allocation bugs in (pack_int) * fix(tianmu):Code format adjustment * fix(website): fix the wrong QR code(#1624) * feat(tianmu):Add delta layer information output and table name output * fix(tianmu):Perfect atomic operations for delta_table * feat(tianmu):Optimize delta layer merge operations to remove useless logic * fix(tianmu):assert failed on ptr == buff.get() + data_.sum_len at pack_str.cpp:584(#1620) * fix(tianmu):assert failed on oldv <= dpn_->max_i at pack_int.cpp:337 (#1610) * feat(tianmu):Increase assertion printing information and optimize code logic(#1617) * fix(tianmu): fix mysqld crash when query where JOIN::propagate_dependencies (#1628) * fix(tianmu): fix MySQL server has gone away when exec query (#1641 #1640) * fix(tianmu):Support insert ignore syntax (#1637) * fix(tianmu): fix query input variables wrong result (#1647) * fix(tianmu): fix result of the query using the subquery derived table is incorrect (#1662) * fix(tianmu): fix results of two queries using a derived table and a custom face change are incorrect (#1696) * feat(tianmu): Test cases that supplement custom variables (#1703) * fix(tianmu): fix mysqld crash when assigning return values using both custom variables and derived tables (#1707) * fix(tianmu): Insert ignore can insert duplicate values.(#1699) * fix(tianmu): fix error occurred in the union all query result (#1599) * remove unused code block * fix bug and change test case exptected result * add stonedb-8.0 compiling guide for CentOS 7.x * docs(quickstart): add stonedb-8.0 compiling guide(Chinese) for CentOS 7.x * fix(tianmu):Even if a primary key is defined, duplicate data may be imported.(#1648) * add delete/drop into tianmu log stat * open log for all cmds * fit format * fix(tianmu): fix Error result set of the IN subquery with semi join (#1764) * doc(develop-guide): modify method for complie stonedb using docker 1. change the version of the stonedb from 5.6 to 5.7 in docs. 2. list both manual install and automatic install in docs. 3. update the reference in zh-doc to a valid one: 可以参考：[StoneDB快速部署手册](https://stonedb.io/zh/docs/getting-started/quick-deployment) 可以参考：[StoneDB快速部署手册](https://stonedb.io/zh/docs/quick-deployment) * docs:add docker compile guide of stonedb8.0.(#1780) * feature: remove DBUG_OFF and repalce DEBUG_ASSERT with assert * automatically formatting * fix: fix storage of DT type * fix incorrect result of TIME type by distinguishing the processing of TIME type from other time types * fix(tinmu): fix tianmu crash when set varchar to num when order by * docs(website): update the documentation for Compile StoneDB 8.0 in Docker(#1823) * fix(tianmu): fix up the incompatible type 1) In result value setup phase, it only deals with num, but in some case, some non-num types involved, therefore, we should also deal with these types. 2) To fixup the boundary of the error codes. * fix(tianmu): Fix up the unknown exception after instance killed randomly (#1841) If the instance was killed by `kill -9 'pid'` randomly, the new inserted data will not be written into `DATA` file under the param of `tianmu_insert_delayed` was set to 0. Under this configuration, the data will write to memory not a `DATA` file. Therefore, when the instance was killed and restarted, the data will be lost, and the `DATA` can not be found, and an TianmuError exception will be thrown. changed the writing behavior of instance from writting data into memory to writting data into `DATA` file immediately as that of `tianmu_insert_delayed=1`. * fix(tianmu): fix up the incorrect meta-info leads unexpected behavior (#1840) In `ColumnShare::scan_dpn`, it will throw an exception to identify the in-consistent meta-data, which the offset violates the rules. Now, the deleted DPNs should not be added to `segs`. And, some auxiliary functions are added for helping to identify the status of files. * fix(workflow): nightly build failed #1830 [summary] Currently we disable tar pkg in ci/cd. * feat(tianmu): revert assert() --> debug_assert() #1551 [summary] To avoid some cases of some assert failed in release mode. * feat(tianmu): fixup the default delimeter for load data (#1843) Tianmu default delimiter is ';' not '\t'. In order to follow the mysql convention, we change the default delimeter to `\t`. * fix(tianmu): revert PR #1841. (#1850) Due to the disk space flattion, this PR is used to revert PR #1841. But this will lead to data in-consistent after the instance killed at randomly because the new data inserted into memory, and will lost. The root cause pls ref to discussion in #1621. * fix(tianmu): fix up mtr test case for delim of load data command (#1854) To fix up some MTR test cases because the delim of load data is changed from ; to \t. * fix(tianmu): fix up the `group_concat` function in tianmu (#1852) To allow `group_concat` function executed in tianmu, and Changes `SI` to `SpecialInstruction`. Some exceptions are catched. * fix(tianmu): To fixup the instance crashed if the result of aggregate function goes out of bounds (#1856) To caculate `sum(length())` will lead a corruption in destructor of `ValueOrNull`, frees an array of char. Before the array deletes, the validity of that should be checked, and after that the pointer of that arrary should also be set to nullptr, which make sure it's a safe code piece. * docs(developer-guide): update the compiling guide of stonedb 8.0 for centos7.x(#1817) * fix(tianmu): Fixup the mem leakage of aggregation function 1: Fixup the memory leakage of aggregation function, which may lead to failed of malloc. 2: Re-impl the operator= of `ValueOrNull`. 3: Fixup the assertion of `dow_cast` in ` Query::ClearSubselectTransformation`, `Item_func_trig_cond`. * fix(tianmu): fix UNION of non-matching columns (column no 0) * test(tianmu): add order by sentence in the mtr case various_join.test * test(mtr): add more test cases for tianmu(#1196) [summary] case_when.test drop_restric.test empty_string_not_null.test left_right_func.test like_not_like.test multi_join.test order_by.test ssb_small.test union_case.test * test(mtr): add order by sentence in the mtr case various_join.test * ci(codecov): update the config * fix(tianmu): To suuport ignore option for update statement To support `update ignore` statement. The logic of uniqueness check is re-implemented. * ci(codecov): update the codecov congfig * docs(intro): update the support for 8.0 * wokflow(codecov): Filter out excess code files * workflow(coverage): Update the lcov running logic * fix(tianmu): default value of the field take unaffect in load #1865 Cause: in the function ParsingStrategy::ParseResult ParsingStrategy::GetOneRow field->val_str(str) cannot distinguish 0 and NULL value. Solution: Check whether field's default value is NULL. * fix(tianmu): To support union(all) the statement which is without from clause 1: To fixup unsupport union or union all a sql statement which is without from clause. 2: Re-format some codes and functions. * fix(tianmu): To remove unnessary optimization in tianmu 1:Removes the unnessary optimization in stage of compiliation of tianmu. It doesnot have any helps for us. and may introuduce unexepected behaviors. 2:Refine MTR: issue848, issue1865, alter_table1, issue1523 * fix(tianmu): hotfix corruption in ValueOrNull under multi-thread In multi-thread aggregation, ExpressionColumn will occur double free due to without protection. Thread A will do ValueOrNull::operator ==, but in thread B, it will try to free it. Therefore, it leads to instance crash. * fix(tianmu): incorrect result when using where expr and args > bigint_max #1564 [summary] 1. static_cast<int64_t>(18446744073709551601) = -15 2. Item will set 18446744073709551601 with unsigned flag, but in tianmu transform to ValueOrNot, the value will be set to `-15`. 3. add `unsigned flag` in value_or_null & TianmuNum & tianmu expr. * fix(tianmu): add TIME_to_ulonglong_time_round process and fix up precision loss problem (#1173) When converting TIME/DATETIME to ulonglong numeric, tianmu engine does not take the TIME_to_ulonglong_time_round process. This causes the results different from innodb. Furthermore, when we close the tianmu_insert_delayed parameter and execute an insert SQL, the TIME/DATETIME/TIMESTAMP type's data will loss precision due to incomplete attribute copying. PR Close #1173 * fix(tianmu): fix format using clang-format #792 * feat: rm files after rebase leftover #1217 files deleted: storage/tianmu/core/rc_attr_typeinfo.h storage/tianmu/handler/tianmu_handler.cpp storage/tianmu/handler/tianmu_handler_com.cpp storage/tianmu/types/rc_data_types.cpp storage/tianmu/types/rc_num.cpp storage/tianmu/types/rc_num.h storage/tianmu/types/rc_value_object.cpp --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: LiMK <slibre@qq.com> Co-authored-by: lihongjian <lihongjian@stoneatom.com> Co-authored-by: shizhao <shizhao@stoneatom.com> Co-authored-by: adofsauron <adofsauron@163.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: StoneAtom <98451811+stoneatomadmin@users.noreply.github.com> Co-authored-by: zzzz-vincent <wenzishen.vincent@hotmail.com> Co-authored-by: wisehead <alex.chenhui@gmail.com> Co-authored-by: Agility6 <agility6@163.com> Co-authored-by: Double0101 <double0101@foxmail.com> Co-authored-by: dysprosium0626 <1119493091@qq.com> Co-authored-by: augety <chenshengjiang@stoneatom.com> Co-authored-by: xuxinqiang <xxq2933@163.com> Co-authored-by: Jinrong Duan <duanjr02@gmail.com> Co-authored-by: unknown <adofsauron> Co-authored-by: hustjieke <gaoriyao1@gmail.com>
RingsC
added a commit
that referenced
this pull request
Jun 30, 2023
* docs(downlaod)：update the docs of download(#1453) * fix(mtr):Resolve the nightly run error problem(#1458) * fix(workflow): fix can not run lcov in workflow * feat(tianmu):New configuration parameters: "tianmu_mandatory" and "tianmu_no_key_error (#1462)" In version 1.0.4, we will discard "MANDATORY_TIANMU" and "NO_KEY_ERROR" in (sql_mode) Specifies whether to enable mandatory Tianmu engine in table. if yes ,set tianmu_mandatory to ON, otherwise set the variable to OFF. Specifies whether to to directly skip DDL statements that are not supported by the SQL layer, instead of reporting errors. if yes ,set tianmu_no_key_error to ON, otherwise set the variable to OFF. * feat(tianmu):Discard "MANDATORY_TIANMU" and "NO_KEY_ERROR" in (sql_mode) * feat(tianmu):Resolve the issue of warning messages when setting new parameters * fix(tianmu):(Primary/Secondary)Error 1032 occasionally occurs during primary/secondary synchronization if UUIDs are used as the primary key(#1464) Cause of the problem: When performing a primary key scan under the master slave, "ha_tianmu:: position()" is called first to obtain the primary key value from the "record", However, in this scenario, after calling the "key_copy()" function, the "record" will be cleared, resulting in the subsequent "GetKeys()" obtaining a null primary key value. Solution: Because the value "handler->ref" is not used in the future, you can simply delete the call to "key_copy()". * test(mtr): add integer/unsigned/in subquery/create temporary testcases and update escape.test(#1196) * docs(v1.0.3): update the docs for v1.0.3 * fix(dcoker): FixDocker deployment commands(#1499) * feat: add Baidu statistics script(#1498) * fix(website): fix Baidu statistics script(#1502) * test(mtr): Optimize parallel scheduling to execute more innodb engine testcases, add date type and std func testcase(#1196) * fix crash when the aggregated element was decimal (#1402) 1. Fix the crash first 2. then redesign the entire aggregated data stream * build(deps): bump nth-check and unist-util-select in /website Bumps [nth-check](https://github.com/fb55/nth-check) to 2.1.1 and updates ancestor dependency [unist-util-select](https://github.com/syntax-tree/unist-util-select). These dependencies need to be updated together. Updates `nth-check` from 1.0.2 to 2.1.1 - [Release notes](https://github.com/fb55/nth-check/releases) - [Commits](fb55/nth-check@v1.0.2...v2.1.1) Updates `unist-util-select` from 2.0.2 to 4.0.1 - [Release notes](https://github.com/syntax-tree/unist-util-select/releases) - [Commits](syntax-tree/unist-util-select@2.0.2...4.0.1) --- updated-dependencies: - dependency-name: nth-check dependency-type: indirect - dependency-name: unist-util-select dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * docs(deploy): Update the document and fix the link * update the copyright * update the readme(#1324) * Create FUNDING.yml update the Sponsor button * Update FUNDING.yml fix the link of opencollective * remove dup lines in git workflow * feat(tianmu): mv func from public to protected(#1501) * style change * strip spaces * remove dup lines * change to uint * fix(tianmu):Resolve DDL and insert inot select Possible space inflation issues(#366) Cause of the problem: 1. For multiple versions, Tianmu needs to first copy an original pack when performing DML operations, Modify the copied package and use append write or overwrite write after modification (If there is invalid space in the DATA file that can be written to the current pack, use overwrite write, otherwise use append write) to write to the file, After the latest package is written to a file, the latest version chain will point to the address that was last written. There is a problem with the current (TianmuAttr:: LoadData) logic. Every time you call (TianmuAttr:: LoadData), Will write data to disk, If there are multiple rows written in a transaction, there will be multiple copies of data, "Because the current transaction has not been committed, the space for previous repeated writes has not been released, so the logic of overwriting writes will not be reached.", "I only follow the logic of additional writing, which is the fundamental reason for the skyrocketing space.". If you encounter a particularly large multiline write transaction, it will lead to a space explosion. Moreover, disk IO is performed once per load line, which can also lead to degraded insert performance. Solution: To optimize the logic of (TianmuAttr:: LoadData), it is necessary to determine whether the data in the pack is full before saving changes, Is whether to reach 65536 lines, and if so, write again, If it cannot be reached, it is necessary to write again in the commit phase. * feat(tianmu):Reconstruct direct insert into parallel execution, improving direct insert performance. * feat(tianmu):Add code comments for easy understanding * fix(tianmu):The myloader cann't work if the autocommit is closed. (#1510) Currently, TIANMU does not support manual transactions and only supports automatic commit. However, it does determine whether to commit the transaction based on the automatic commit parameters of MySQL. If automatic commit is turned off, automatic commit will not be performed on the transaction * fix(core): fix bug: The instance occasionally crashes if both fields specified for an equi-join are of the string data types (#1476) * fix: page hover font style * feat(mtr): To fix the mtr usage The disabeld mtr use cases should follow the mtr conventional rules. Adds a disabled.def to indeciate which case is disabled or not. * fix(tianmu):The mysqld is crashed when you are starting replication.(#1523) It is possible to update null values with null values in the delta layer, For example: update t set name="xiaohua" where id=1; update t set name=null where id=1; So when encountering this situation, directly return * feat(tianmu):Add delta layer mtr * bug 1538:The instance occasionally crashes when the parallel degree is enabled of the right table #1538 * feat(tianmu): To support vocalno framwork The PRs to support vocalno framwork include a serials of PR. Part1: Refine framework of code to make it clean and clear to read. * docs(quickstart): add stonedb-8.0 compiling guide #1449 * fix(website): fix website error #1449 * feat(tianmu): support volcano framewrok (#1546) * feat(tianmu): To support volcano framework This is a serial of PRs, part2: To remove global var `ha_kvstore`, which should be in an engine. To support volcano framework. and In order to do that, at secondly, ha_kvstore_, a global variable removed to the engine. That follows innodb conventional rules. * fix(tianmu): revert code, mv ret value from try block back to catch block [summary] The logic of this modification is as follows: Previously, set ret action has been moved to try block, which is not efficient because every time we do truncate success, ret will be setted to 1. This time we will move the set ret action back to catch block, which will only trigged when truncate failed. * feat(tiamnu): hard code in defs.h (#1481) change magic number to readable const * docs:update the compile guides #1562 * test(mtr): add more innodb testcases and tianmu range testcase(#1196) * fix(tianmu):Remove excess log printing and add some code comments(#1545) * fix(tianmu): fix mysqld crash when exec query with AggregateRough, assert failed on i < m_idx.size() at tianmu_attr.h:387, msg: [bad dpn index 0/0] (#1580) * fix(website): fix the download link of 5.7(#1518) * feat(website): update the latest content(#1587) * feat(tianmu): support volcano framework (#1554) Part3: To remove the `ha_tianmu_engine`, and gets it from hton's data. This makes it behavior just like innodb. MySQL gets innodb handler instance from table->s->file. and it will make the code logic more concise. * fix: max-width navbar search style * feat(website): upgrade the docusaurus version(#1604) fix #1604 * fix(website): fix Roadmap module location(#1597) * website(community): update the content * feat(website): update the logo of XinChuang(#1590) * fix(tianmu):The instance occasionally crashes when the memory leak. (#1549) * fix(tianmu):Modify merge_ The assignment method of ID, delaying the assignment to ensure that the final value is correct * fix(tianmu):Fix bug in delta layer initialization * fix(tianmu):Resolve the issue of assertion failure caused by memory allocation bugs in (pack_int) * fix(tianmu):Code format adjustment * fix(website): fix the wrong QR code(#1624) * feat(tianmu):Add delta layer information output and table name output * fix(tianmu):Perfect atomic operations for delta_table * feat(tianmu):Optimize delta layer merge operations to remove useless logic * fix(tianmu):assert failed on ptr == buff.get() + data_.sum_len at pack_str.cpp:584(#1620) * fix(tianmu):assert failed on oldv <= dpn_->max_i at pack_int.cpp:337 (#1610) * feat(tianmu):Increase assertion printing information and optimize code logic(#1617) * fix(tianmu): fix mysqld crash when query where JOIN::propagate_dependencies (#1628) * fix(tianmu): fix MySQL server has gone away when exec query (#1641 #1640) * fix(tianmu):Support insert ignore syntax (#1637) * fix(tianmu): fix query input variables wrong result (#1647) * fix(tianmu): fix result of the query using the subquery derived table is incorrect (#1662) * fix(tianmu): fix results of two queries using a derived table and a custom face change are incorrect (#1696) * feat(tianmu): Test cases that supplement custom variables (#1703) * fix(tianmu): fix mysqld crash when assigning return values using both custom variables and derived tables (#1707) * fix(tianmu): Insert ignore can insert duplicate values.(#1699) * fix(tianmu): fix error occurred in the union all query result (#1599) * remove unused code block * fix bug and change test case exptected result * add stonedb-8.0 compiling guide for CentOS 7.x * docs(quickstart): add stonedb-8.0 compiling guide(Chinese) for CentOS 7.x * fix(tianmu):Even if a primary key is defined, duplicate data may be imported.(#1648) * add delete/drop into tianmu log stat * open log for all cmds * fit format * fix(tianmu): fix Error result set of the IN subquery with semi join (#1764) * doc(develop-guide): modify method for complie stonedb using docker 1. change the version of the stonedb from 5.6 to 5.7 in docs. 2. list both manual install and automatic install in docs. 3. update the reference in zh-doc to a valid one: 可以参考：[StoneDB快速部署手册](https://stonedb.io/zh/docs/getting-started/quick-deployment) 可以参考：[StoneDB快速部署手册](https://stonedb.io/zh/docs/quick-deployment) * docs:add docker compile guide of stonedb8.0.(#1780) * feature: remove DBUG_OFF and repalce DEBUG_ASSERT with assert * automatically formatting * fix: fix storage of DT type * fix incorrect result of TIME type by distinguishing the processing of TIME type from other time types * fix(tinmu): fix tianmu crash when set varchar to num when order by * docs(website): update the documentation for Compile StoneDB 8.0 in Docker(#1823) * fix(tianmu): fix up the incompatible type 1) In result value setup phase, it only deals with num, but in some case, some non-num types involved, therefore, we should also deal with these types. 2) To fixup the boundary of the error codes. * fix(tianmu): Fix up the unknown exception after instance killed randomly (#1841) If the instance was killed by `kill -9 'pid'` randomly, the new inserted data will not be written into `DATA` file under the param of `tianmu_insert_delayed` was set to 0. Under this configuration, the data will write to memory not a `DATA` file. Therefore, when the instance was killed and restarted, the data will be lost, and the `DATA` can not be found, and an TianmuError exception will be thrown. changed the writing behavior of instance from writting data into memory to writting data into `DATA` file immediately as that of `tianmu_insert_delayed=1`. * fix(tianmu): fix up the incorrect meta-info leads unexpected behavior (#1840) In `ColumnShare::scan_dpn`, it will throw an exception to identify the in-consistent meta-data, which the offset violates the rules. Now, the deleted DPNs should not be added to `segs`. And, some auxiliary functions are added for helping to identify the status of files. * fix(workflow): nightly build failed #1830 [summary] Currently we disable tar pkg in ci/cd. * feat(tianmu): revert assert() --> debug_assert() #1551 [summary] To avoid some cases of some assert failed in release mode. * feat(tianmu): fixup the default delimeter for load data (#1843) Tianmu default delimiter is ';' not '\t'. In order to follow the mysql convention, we change the default delimeter to `\t`. * fix(tianmu): revert PR #1841. (#1850) Due to the disk space flattion, this PR is used to revert PR #1841. But this will lead to data in-consistent after the instance killed at randomly because the new data inserted into memory, and will lost. The root cause pls ref to discussion in #1621. * fix(tianmu): fix up mtr test case for delim of load data command (#1854) To fix up some MTR test cases because the delim of load data is changed from ; to \t. * fix(tianmu): fix up the `group_concat` function in tianmu (#1852) To allow `group_concat` function executed in tianmu, and Changes `SI` to `SpecialInstruction`. Some exceptions are catched. * fix(tianmu): To fixup the instance crashed if the result of aggregate function goes out of bounds (#1856) To caculate `sum(length())` will lead a corruption in destructor of `ValueOrNull`, frees an array of char. Before the array deletes, the validity of that should be checked, and after that the pointer of that arrary should also be set to nullptr, which make sure it's a safe code piece. * docs(developer-guide): update the compiling guide of stonedb 8.0 for centos7.x(#1817) * fix(tianmu): Fixup the mem leakage of aggregation function 1: Fixup the memory leakage of aggregation function, which may lead to failed of malloc. 2: Re-impl the operator= of `ValueOrNull`. 3: Fixup the assertion of `dow_cast` in ` Query::ClearSubselectTransformation`, `Item_func_trig_cond`. * fix(tianmu): fix UNION of non-matching columns (column no 0) * test(tianmu): add order by sentence in the mtr case various_join.test * test(mtr): add more test cases for tianmu(#1196) [summary] case_when.test drop_restric.test empty_string_not_null.test left_right_func.test like_not_like.test multi_join.test order_by.test ssb_small.test union_case.test * test(mtr): add order by sentence in the mtr case various_join.test * ci(codecov): update the config * fix(tianmu): To suuport ignore option for update statement To support `update ignore` statement. The logic of uniqueness check is re-implemented. * ci(codecov): update the codecov congfig * docs(intro): update the support for 8.0 * wokflow(codecov): Filter out excess code files * workflow(coverage): Update the lcov running logic * fix(tianmu): default value of the field take unaffect in load #1865 Cause: in the function ParsingStrategy::ParseResult ParsingStrategy::GetOneRow field->val_str(str) cannot distinguish 0 and NULL value. Solution: Check whether field's default value is NULL. * fix(tianmu): To support union(all) the statement which is without from clause 1: To fixup unsupport union or union all a sql statement which is without from clause. 2: Re-format some codes and functions. * fix(tianmu): To remove unnessary optimization in tianmu 1:Removes the unnessary optimization in stage of compiliation of tianmu. It doesnot have any helps for us. and may introuduce unexepected behaviors. 2:Refine MTR: issue848, issue1865, alter_table1, issue1523 * fix(tianmu): hotfix corruption in ValueOrNull under multi-thread In multi-thread aggregation, ExpressionColumn will occur double free due to without protection. Thread A will do ValueOrNull::operator ==, but in thread B, it will try to free it. Therefore, it leads to instance crash. * fix(tianmu): incorrect result when using where expr and args > bigint_max #1564 [summary] 1. static_cast<int64_t>(18446744073709551601) = -15 2. Item will set 18446744073709551601 with unsigned flag, but in tianmu transform to ValueOrNot, the value will be set to `-15`. 3. add `unsigned flag` in value_or_null & TianmuNum & tianmu expr. * fix(tianmu): add TIME_to_ulonglong_time_round process and fix up precision loss problem (#1173) When converting TIME/DATETIME to ulonglong numeric, tianmu engine does not take the TIME_to_ulonglong_time_round process. This causes the results different from innodb. Furthermore, when we close the tianmu_insert_delayed parameter and execute an insert SQL, the TIME/DATETIME/TIMESTAMP type's data will loss precision due to incomplete attribute copying. PR Close #1173 * fix(tianmu): fix format using clang-format #792 * feat: rm files after rebase leftover #1217 files deleted: storage/tianmu/core/rc_attr_typeinfo.h storage/tianmu/handler/tianmu_handler.cpp storage/tianmu/handler/tianmu_handler_com.cpp storage/tianmu/types/rc_data_types.cpp storage/tianmu/types/rc_num.cpp storage/tianmu/types/rc_num.h storage/tianmu/types/rc_value_object.cpp * fix(sql,tianmu):fix when binlog format is row, the load data statement cannot be recorded(#1876) 1. Actually, tianmu uses its own code to handle load which lacks support of row format of binlog 2. When tianmu parsing rows, write table map event first 3. Once tianmu constructs a row, just add it to the rows log event, when parsing is done, the rows log event will also be ready, then write it to the binlog --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: LiMK <slibre@qq.com> Co-authored-by: lihongjian <lihongjian@stoneatom.com> Co-authored-by: shizhao <shizhao@stoneatom.com> Co-authored-by: adofsauron <adofsauron@163.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: StoneAtom <98451811+stoneatomadmin@users.noreply.github.com> Co-authored-by: zzzz-vincent <wenzishen.vincent@hotmail.com> Co-authored-by: wisehead <alex.chenhui@gmail.com> Co-authored-by: Agility6 <agility6@163.com> Co-authored-by: Double0101 <double0101@foxmail.com> Co-authored-by: dysprosium0626 <1119493091@qq.com> Co-authored-by: augety <chenshengjiang@stoneatom.com> Co-authored-by: xuxinqiang <xxq2933@163.com> Co-authored-by: Jinrong Duan <duanjr02@gmail.com> Co-authored-by: unknown <adofsauron> Co-authored-by: hustjieke <gaoriyao1@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary about this PR
To fixup the instance crashed if the result of aggregate is out of boundary function goes out of bounds
To caculate
sum(length())will lead a corruption in destructor ofValueOrNull, frees an array of char. Before the array deletes, the validity of that should be checked, and after that the pointer of that arrary should also be set to nullptr, which make sure it's a safe code piece.Issue Number: close #issue_number_you_created
Tests Check List
Changelog
Documentation