Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: Incremental data causing instance crash. #1805

Closed
3 tasks done
haitaoguan opened this issue May 19, 2023 · 5 comments
Closed
3 tasks done

bug: Incremental data causing instance crash. #1805

haitaoguan opened this issue May 19, 2023 · 5 comments
Assignees
@chenshengjiang
Labels
A-bug Something isn't working prio: normal Medium priority
Milestone
StoneDB_5.7_v1.0.4

Comments

@haitaoguan
Copy link
Collaborator

Have you read the Contributing Guidelines on issues?

Please confirm if bug report does NOT exists already ?

  • I confirm there is no existing issue for this

Describe the problem

[2023-05-19 14:58:56.869758] [7591] [INFO] [engine.cpp:293] MSG: table name: lxjxc_production.sms_accounts_lxjxc, delta table id: 1260 delta_size: 0, current load id: 0, merge id: 0, current row_id: 0
[2023-05-19 14:58:56.869765] [7591] [INFO] [engine.cpp:293] MSG: table name: lxjxc_production.sms_settings, delta table id: 1272 delta_size: 0, current load id: 0, merge id: 0, current row_id: 0
[2023-05-19 14:58:56.869771] [7591] [INFO] [engine.cpp:293] MSG: table name: lxjxc_production.sms_signs, delta table id: 1276 delta_size: 0, current load id: 0, merge id: 0, current row_id: 0
[2023-05-19 14:58:56.869778] [7591] [INFO] [engine.cpp:293] MSG: table name: lxjxc_production.sms_template_params, delta table id: 1278 delta_size: 0, current load id: 0, merge id: 0, current row_id: 0
[2023-05-19 14:59:06.774058] [7828] [WARN] [exception.cpp:42] MSG: Exception: Data format error. Row: 0 Column: 29.

STACK TRACE BEGIN
        /opt/stonedb57/install/bin/mysqld(Tianmu::common::FormatException::FormatException(unsigned long long, unsigned int)+0x102) [0x162efa2]
        /opt/stonedb57/install/bin/mysqld() [0x98a6b7]
        /opt/stonedb57/install/bin/mysqld(Tianmu::loader::ParsingStrategy::GetOneRow(char const*, unsigned long, std::vector<Tianmu::loader::ValueCache, std::allocator<Tianmu::loader::ValueCache> >&, unsigned int&, int&, bool)+0xe13) [0x1665a73]
        /opt/stonedb57/install/bin/mysqld(Tianmu::loader::LoadParser::MakeRow(std::vector<Tianmu::loader::ValueCache, std::allocator<Tianmu::loader::ValueCache> >&)+0x68) [0x165fd08]
        /opt/stonedb57/install/bin/mysqld(Tianmu::loader::LoadParser::GetPackrow(unsigned int, std::vector<Tianmu::loader::ValueCache, std::allocator<Tianmu::loader::ValueCache> >&)+0x17d) [0x16608ed]
        /opt/stonedb57/install/bin/mysqld(Tianmu::core::TianmuTable::ProceedNormal(Tianmu::system::IOParameters&)+0x144) [0x16451e4]
        /opt/stonedb57/install/bin/mysqld(Tianmu::core::TianmuTable::LoadDataInfile(Tianmu::system::IOParameters&)+0x21f) [0x164860f]
        /opt/stonedb57/install/bin/mysqld(Tianmu::core::Engine::RunLoader(THD*, sql_exchange*, TABLE_LIST*, void*)+0x15c) [0x15e18bc]
        /opt/stonedb57/install/bin/mysqld(Tianmu::DBHandler::tianmu_load_impl(THD*, sql_exchange*, TABLE_LIST*, void*, char*, int, int&)+0x79) [0x17f0099]
        /opt/stonedb57/install/bin/mysqld(Tianmu::DBHandler::ha_my_tianmu_load(THD*, sql_exchange*, TABLE_LIST*, void*)+0x93) [0x17f02c3]
        /opt/stonedb57/install/bin/mysqld(mysql_load(THD*, sql_exchange*, TABLE_LIST*, List<Item>&, List<Item>&, List<Item>&, enum_duplicates, bool)+0x769) [0x11425c9]
        /opt/stonedb57/install/bin/mysqld(mysql_execute_command(THD*, bool)+0x3b32) [0xfc7aa2]
        /opt/stonedb57/install/bin/mysqld(mysql_parse(THD*, Parser_state*)+0x425) [0xfcad05]
        /opt/stonedb57/install/bin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x1599) [0xfcc309]
        /opt/stonedb57/install/bin/mysqld(do_command(THD*)+0x207) [0xfcd027]
        /opt/stonedb57/install/bin/mysqld(handle_connection+0x2b8) [0x1085a28]
        /opt/stonedb57/install/bin/mysqld(pfs_spawn_thread+0x164) [0x155cf64]
        /lib64/libpthread.so.0(+0x7ea5) [0x7fb59ef04ea5]
        /lib64/libc.so.6(clone+0x6d) [0x7fb59c23eb0d]
STACK TRACE END

[2023-05-19 14:59:06.778120] [7828] [ERROR] [engine.cpp:970] MSG: Roll back query '/* DTS-full-fzmo5bn119x85ok */LOAD DATA CONCURRENT LOCAL INFILE 'a.csv' ignore INTO TABLE `lxjxc_production`.`sales` CHARACTER SET utf8mb4 FIELDS TERMINATED BY '\t' ENCLOSED BY '"' ESCAPED BY '\\' (@`note`,@`contact_phone`,`remaining_invoice_amount`,`discount`,`total_amount_with_tax`,`created_at`,`invoiced_amount`,`contact_id`,@`number`,`updated_at`,`io_at`,`total_quantity`,`total_tax_amount`,`invoice_type`,@`id`,@`contact_address`,`seller_id`,@`customer_address`,`amount`,@`contact_name`,`deleted_at`,`approved_level`,`sale_order_id`,`total_base_quantity`,`deduction`,`total_amount`,`org_id`,`parent_id`,`creator_id`,`total_fee_amount`,`total_deduction`,`fifo_total_amount`,@`contact_mobile`,`customer_id`,`category`,`warehouse_id`,`status`,`invoice_status`) SET `note` = UNHEX(@`note`),`contact_phone` = UNHEX(@`contact_phone`),`number` = UNHEX(@`number`),`id` = @`id`,`contact_address` = UNHEX(@`contact_address`),`customer_address` = UNHEX(@`customer_address`),`contact_name` = UNHEX(@`contact_name`),`contact_mobile` = UNHEX(@`contact_mobile`)'
[2023-05-19 14:59:06.778161] [7828] [INFO] [transaction.cpp:90] MSG: rollback transaction 64670e7c00002950.
[2023-05-19 14:59:06.778171] [7828] [ERROR] [transaction.cpp:98] MSG: Tianmu storage engine has encountered an unexpected error. The current transaction has been rolled back.
[2023-05-19 14:59:06.778204] [7828] [INFO] [tianmu_table.cpp:689] MSG: roll back table lxjxc_production../lxjxc_production/sales.tianmu
[2023-05-19 14:59:06.778328] [7828] [ERROR] [ha_my_tianmu.cpp:130] MSG: RunLoader Error: Data format error. Row: 0 Column: 29
[2023-05-19 14:59:06.785237] [7828] [ERROR] [engine.cpp:970] MSG: Roll back query 'rollback'
[2023-05-19 14:59:08.815205] [7823] [WARN] [exception.cpp:42] MSG: Exception: Data format error. Row: 0 Column: 29.

STACK TRACE BEGIN
        /opt/stonedb57/install/bin/mysqld(Tianmu::common::FormatException::FormatException(unsigned long long, unsigned int)+0x102) [0x162efa2]
        /opt/stonedb57/install/bin/mysqld() [0x98a6b7]
        /opt/stonedb57/install/bin/mysqld(Tianmu::loader::ParsingStrategy::GetOneRow(char const*, unsigned long, std::vector<Tianmu::loader::ValueCache, std::allocator<Tianmu::loader::ValueCache> >&, unsigned int&, int&, bool)+0xe13) [0x1665a73]
        /opt/stonedb57/install/bin/mysqld(Tianmu::loader::LoadParser::MakeRow(std::vector<Tianmu::loader::ValueCache, std::allocator<Tianmu::loader::ValueCache> >&)+0x68) [0x165fd08]
        /opt/stonedb57/install/bin/mysqld(Tianmu::loader::LoadParser::GetPackrow(unsigned int, std::vector<Tianmu::loader::ValueCache, std::allocator<Tianmu::loader::ValueCache> >&)+0x17d) [0x16608ed]
        /opt/stonedb57/install/bin/mysqld(Tianmu::core::TianmuTable::ProceedNormal(Tianmu::system::IOParameters&)+0x144) [0x16451e4]
        /opt/stonedb57/install/bin/mysqld(Tianmu::core::TianmuTable::LoadDataInfile(Tianmu::system::IOParameters&)+0x21f) [0x164860f]
        /opt/stonedb57/install/bin/mysqld(Tianmu::core::Engine::RunLoader(THD*, sql_exchange*, TABLE_LIST*, void*)+0x15c) [0x15e18bc]
        /opt/stonedb57/install/bin/mysqld(Tianmu::DBHandler::tianmu_load_impl(THD*, sql_exchange*, TABLE_LIST*, void*, char*, int, int&)+0x79) [0x17f0099]
        /opt/stonedb57/install/bin/mysqld(Tianmu::DBHandler::ha_my_tianmu_load(THD*, sql_exchange*, TABLE_LIST*, void*)+0x93) [0x17f02c3]
        /opt/stonedb57/install/bin/mysqld(mysql_load(THD*, sql_exchange*, TABLE_LIST*, List<Item>&, List<Item>&, List<Item>&, enum_duplicates, bool)+0x769) [0x11425c9]
        /opt/stonedb57/install/bin/mysqld(mysql_execute_command(THD*, bool)+0x3b32) [0xfc7aa2]
        /opt/stonedb57/install/bin/mysqld(mysql_parse(THD*, Parser_state*)+0x425) [0xfcad05]
        /opt/stonedb57/install/bin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x1599) [0xfcc309]
        /opt/stonedb57/install/bin/mysqld(do_command(THD*)+0x207) [0xfcd027]
        /opt/stonedb57/install/bin/mysqld(handle_connection+0x2b8) [0x1085a28]
        /opt/stonedb57/install/bin/mysqld(pfs_spawn_thread+0x164) [0x155cf64]
        /lib64/libpthread.so.0(+0x7ea5) [0x7fb59ef04ea5]
        /lib64/libc.so.6(clone+0x6d) [0x7fb59c23eb0d]
STACK TRACE END

Expected behavior

No response

How To Reproduce

No response

Environment

No response

Are you interested in submitting a PR to solve the problem?

  • Yes, I will!
@haitaoguan haitaoguan added the A-bug Something isn't working label May 19, 2023
@RingsC
Copy link
Contributor

RingsC commented May 19, 2023 •
edited
Loading

Row format error, can not dealing with unexpected row format properly. The query statment is listed here,

[2023-05-19 14:59:06.778120] [7828] [ERROR] [engine.cpp:970] MSG: Roll back query '/* DTS-full-fzmo5bn119x85ok */

LOAD DATA CONCURRENT LOCAL INFILE 'a.csv' ignore 
INTO TABLE `lxjxc_production`.`sales` CHARACTER SET utf8mb4 FIELDS TERMINATED BY '\t' 
ENCLOSED BY '"' ESCAPED BY '\\' (@`note`,@`contact_phone`,`remaining_invoice_amount`,`discount`,`total_amount_with_tax`,`created_at`,`invoiced_amount`,`contact_id`,@`number`,`updated_at`,`io_at`,`total_quantity`,`total_tax_amount`,`invoice_type`,@`id`,@`contact_address`,`seller_id`,@`customer_address`,`amount`,@`contact_name`,`deleted_at`,`approved_level`,`sale_order_id`,`total_base_quantity`,`deduction`,`total_amount`,`org_id`,`parent_id`,`creator_id`,`total_fee_amount`,`total_deduction`,`fifo_total_amount`,@`contact_mobile`,`customer_id`,`category`,`warehouse_id`,`status`,`invoice_status`) SET `note` = UNHEX(@`note`),`contact_phone` = UNHEX(@`contact_phone`),`number` = UNHEX(@`number`),`id` = @`id`,`contact_address` = UNHEX(@`contact_address`),`customer_address` = UNHEX(@`customer_address`),`contact_name` = UNHEX(@`contact_name`),`contact_mobile` = UNHEX(@`contact_mobile`)'

Enn, Maybe, Pay more attention to UNHEX function here. and charset should also be watched.

@haitaoguan
Copy link
Collaborator Author

haitaoguan commented May 22, 2023 •
edited
Loading

Because the value of the data type decimal is more than 18.

mysql> show create table xxx\G
*************************** 1. row ***************************
       Table: xxx
Create Table: CREATE TABLE `xxx` (
  `id` int(11) DEFAULT NULL,
  `c1` decimal(18,6) DEFAULT NULL
) ENGINE=TIANMU DEFAULT CHARSET=utf8mb4
1 row in set (0.00 sec)

mysql> insert into xxx values(1,100000000000.00000);
Query OK, 1 row affected (0.00 sec)

mysql> select * from xxx;
+------+---------------------+
| id   | c1                  |
+------+---------------------+
|    1 | 100000000000.000000 |
+------+---------------------+
1 row in set (0.00 sec)

mysql> load data infile '/tmp/xxx.csv' into table xxx FIELDS TERMINATED BY ','  LINES TERMINATED BY '\n';
ERROR 1264 (22003): Out of range value for column 'c1' at row 4

# more /tmp/xxx.csv 
1,100000000000.000000
2,100000000001.000000
2,100000000001.000000
3,10000000000000.000000
3,10000000000000.000000
4,10000000000001.000000
4,10000000000001.000000

@hustjieke hustjieke added this to the StoneDB_5.7_v1.0.4 milestone May 29, 2023
@hustjieke hustjieke added the prio: high High priority label May 29, 2023
@RingsC RingsC added prio: normal Medium priority and removed prio: high High priority labels May 31, 2023
@RingsC
Copy link
Contributor

RingsC commented May 31, 2023 •
edited
Loading

In ParsingStrategy::GetValue function, it will throw common::FormatException.
From the code and exception message, we know that:

The exception message:

Data format error. Row: 0 Column: 29.

and the code:

  } else {
    types::BString tmp_string(value_ptr, value_size);
    // reaching here, the parsing function should not be null
    auto function = types::ValueParserForText::GetParsingFuntion(ati);
    if (function(tmp_string, *reinterpret_cast<int64_t *>(buffer.Prepare(sizeof(int64_t)))) ==
        common::ErrorCode::FAILED)
      throw common::FormatException(0,
                                    col);  // TODO: throw appropriate exception
    buffer.ExpectedSize(sizeof(int64_t));
  }

@RingsC
Copy link
Contributor

RingsC commented May 31, 2023

After decimal type supported fully,
in this function, should add a case to deal with decimal type.

auto GetParsingFuntion(const core::AttributeTypeInfo &at) {

}

and the data type of output params, out, of this function. Should be a decimal type not int64_t . enn, int128_t may be a better choice.

common::ErrorCode ValueParserForText::ParseDecimal(BString const &tianmu_s, int64_t &out, short precision,
                                                   short scale) {

@RingsC
Copy link
Contributor

RingsC commented May 31, 2023

@haitaoguan , pls use the latest version to verify it again, thx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chenshengjiang chenshengjiang

Labels
A-bug Something isn't working prio: normal Medium priority
Projects
StoneDB for MySQL 5.7
Status: Done
Milestone
StoneDB_5.7_v1.0.4
Development

No branches or pull requests
4 participants
@RingsC @hustjieke @haitaoguan @chenshengjiang