Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]If the key column of the table is a double / float type, it will crash when checksum #3735

Closed
marising opened this issue Jun 1, 2020 · 0 comments
Assignees
Labels
area/storage Issues or PRs related to storage engine kind/fix Categorizes issue or PR as related to a bug.

Comments

@marising
Copy link
Contributor

marising commented Jun 1, 2020

Describe the bug
If the key column of the table is double/float type,it will crash when checksum.

  1. be/src/olap/task/engine_checksum_task.cpp
    return_columns do not contain column of double/float type
    // ignore float and double type considering to precision lose
    for (size_t i = 0; i < tablet->tablet_schema().num_columns(); ++i) {
        FieldType type = tablet->tablet_schema().column(i).type();
        if (type == OLAP_FIELD_TYPE_FLOAT || type == OLAP_FIELD_TYPE_DOUBLE) {
            continue;
        }

        reader_params.return_columns.push_back(i);
    }
  1. be/src/olap/row.h Use the all columns
    use num_key_columns from Schema,num_key_columns include the column of double/float type
template<typename LhsRowType, typename RhsRowType>
int compare_row(const LhsRowType& lhs, const RhsRowType& rhs) {
    for (uint32_t cid = 0; cid < lhs.schema()->num_key_columns(); ++cid) {
        auto res = lhs.schema()->column(cid)->compare_cell(lhs.cell(cid), rhs.cell(cid));
        if (res != 0) {
            return res;
        }
    }
    return 0;
}

// Used to compare row with input scan key. Scan key only contains key columns,
// row contains all key columns, which is superset of key columns.
// So we should compare the common prefix columns of lhs and rhs.
//
// NOTE: if you are not sure if you can use it, please don't use this function.
template<typename LhsRowType, typename RhsRowType>
int compare_row_key(const LhsRowType& lhs, const RhsRowType& rhs) {
    auto cmp_cids = std::min(lhs.schema()->num_column_ids(), rhs.schema()->num_column_ids());
    for (uint32_t cid = 0; cid < cmp_cids; ++cid) {
        auto res = lhs.schema()->column(cid)->compare_cell(lhs.cell(cid), rhs.cell(cid));
        if (res != 0) {
            return res;
        }
    }
    return 0;
}
  1. Core file

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/export/home/lihaibo/deploy_jd/be/lib/palo_be'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000000d6b838 in compare_cell<doris::RowCursorCell, doris::RowCursorCell> (this=, rhs=..., lhs=...) at /root/map_doris/be/src/olap/field.h:138
138 /root/map_doris/be/src/olap/field.h: No such file or directory.
Missing separate debuginfos, use: debuginfo-install libgcc-4.8.5-28.el7.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 zlib-1.2.7-15.el7.x86_64
(gdb) bt
#0 0x0000000000d6b838 in compare_cell<doris::RowCursorCell, doris::RowCursorCell> (this=, rhs=..., lhs=...) at /root/map_doris/be/src/olap/field.h:138
#1 compare_row<doris::RowCursor, doris::RowCursor> (rhs=..., lhs=...) at /root/map_doris/be/src/olap/row.h:62
#2 operator() (this=, b=0x58dee40, a=0x5df3b00) at /root/map_doris/be/src/olap/reader.cpp:269
#3 operator()<__gnu_cxx::__normal_iterator<doris::CollectIterator::ChildCtx**, std::vectordoris::CollectIterator::ChildCtx* >, doris::CollectIterator::ChildCtx*> (__val=, __it=..., this=)
at /usr/include/c++/7.3.0/bits/predefined_ops.h:177
#4 std::__push_heap<__gnu_cxx::__normal_iterator<doris::CollectIterator::ChildCtx**, std::vector<doris::CollectIterator::ChildCtx*, std::allocatordoris::CollectIterator::ChildCtx* > >, long, doris::CollectIterator::ChildCtx*, __gnu_cxx::__ops::_Iter_comp_valdoris::CollectIterator::ChildCtxComparator > (__first=..., __holeIndex=1, __topIndex=0, __value=0x58dee40, __comp=...) at /usr/include/c++/7.3.0/bits/stl_heap.h:133
#5 0x0000000000d66cd1 in push_heap<__gnu_cxx::__normal_iterator<doris::CollectIterator::ChildCtx**, std::vectordoris::CollectIterator::ChildCtx* >, doris::CollectIterator::ChildCtxComparator> (__last=..., __first=..., __comp=...)
at /usr/include/c++/7.3.0/bits/stl_heap.h:207
#6 push (__x=@0x58df878: 0x58dee40, this=0x58df888) at /usr/include/c++/7.3.0/bits/stl_queue.h:598
#7 _merge_next (delete_flag=, row=, this=0x58df860) at /root/map_doris/be/src/olap/reader.cpp:227
#8 next (delete_flag=, row=, this=0x58df860) at /root/map_doris/be/src/olap/reader.cpp:68
#9 doris::Reader::_agg_key_next_row (this=0x7ffe9d5a1920, row_cursor=0x7ffe9d5a1800, mem_pool=0x5db2fc0, agg_pool=, eof=) at /root/map_doris/be/src/olap/reader.cpp:371
#10 0x0000000000d2c617 in next_row_with_aggregation (eof=0x7ffe9d5a1770, agg_pool=, mem_pool=, row_cursor=0x7ffe9d5a1800, this=0x7ffe9d5a1920) at /root/map_doris/be/src/olap/reader.h:130
#11 doris::EngineChecksumTask::_compute_checksum (this=this@entry=0x7ffe9d5a45d0) at /root/map_doris/be/src/olap/task/engine_checksum_task.cpp:120
#12 0x0000000000d2d2ba in doris::EngineChecksumTask::execute (this=this@entry=0x7ffe9d5a45d0) at /root/map_doris/be/src/olap/task/engine_checksum_task.cpp:36
#13 0x0000000000aa462e in main (argc=1, argv=0x7ffe9d5a4708) at /root/map_doris/be/src/service/doris_main.cpp:183

@marising marising changed the title If the key column of the table is a double / float type, it will crash when checksum [Bug]If the key column of the table is a double / float type, it will crash when checksum Jun 1, 2020
@morningman morningman added area/storage Issues or PRs related to storage engine kind/fix Categorizes issue or PR as related to a bug. labels Jun 1, 2020
morningman pushed a commit that referenced this issue Jun 10, 2020
1. the table include key column of double/float type
2. when run checksum task, will use all of key columns to compare
3. schema.column(idx) of double/float type is NULL

#3735
morningman pushed a commit to morningman/doris that referenced this issue Jun 22, 2020
1. the table include key column of double/float type
2. when run checksum task, will use all of key columns to compare
3. schema.column(idx) of double/float type is NULL

apache#3735
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/storage Issues or PRs related to storage engine kind/fix Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

2 participants