Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](HDFS) Correctly set hdfs cache allocator's offset #33960

Merged
merged 1 commit into from
Apr 22, 2024

Conversation

ByteYue
Copy link
Contributor

@ByteYue ByteYue commented Apr 22, 2024

Proposed changes

Issue Number: close #xxx

We should refresh the bytes_appended field to correctly calculate the cache's offset.

F20240421 22:51:00.731791 2191196 block_file_cache.cpp:600] Check failed: !file_blocks.empty() 
*** Check failure stack trace: ***
    @     0x55f954283bc6  google::LogMessage::SendToLog()
    @     0x55f954280610  google::LogMessage::Flush()
    @     0x55f954284409  google::LogMessageFatal::~LogMessageFatal()
    @     0x55f9204cc65d  doris::io::BlockFileCache::get_or_set()
    @     0x55f92054afd8  doris::io::FileCacheAllocatorBuilder::allocate_cache_holder()
    @     0x55f920599fc0  doris::io::HdfsFileWriter::_write_into_local_file_cache()
    @     0x55f9205999ae  doris::io::HdfsFileWriter::_flush_buffer()
    @     0x55f92059be82  doris::io::HdfsFileWriter::_append()
    @     0x55f92059c3d3  doris::io::HdfsFileWriter::appendv()
    @     0x55f9229259f5  doris::segment_v2::PageIO::write_page()
    @     0x55f922a86be0  doris::segment_v2::PageIO::compress_and_write_page()
    @     0x55f922c06541  doris::segment_v2::ScalarColumnWriter::write_data()
    @     0x55f922d3c2c2  doris::segment_v2::VerticalSegmentWriter::write_batch()
    @     0x55f92254e2cf  doris::SegmentFlusher::_add_rows()
    @     0x55f92254992a  doris::SegmentFlusher::flush_single_block()
    @     0x55f922554348  doris::SegmentCreator::flush_single_block()
    @     0x55f922491396  doris::BaseBetaRowsetWriter::flush_memtable()
    @     0x55f92215d768  doris::FlushToken::_do_flush_memtable()
    @     0x55f92215f183  doris::FlushToken::_flush_memtable()
    @     0x55f92216e752  doris::MemtableFlushTask::run()
    @     0x55f9246b4a5f  doris::ThreadPool::dispatch_thread()
    @     0x55f9246e03d6  std::__invoke_impl<>()
    @     0x55f9246e027f  std::__invoke<>()
    @     0x55f9246e01f7  _ZNSt5_BindIFMN5doris10ThreadPoolEFvvEPS1_EE6__callIvJEJLm0EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
    @     0x55f9246e0090  std::_Bind<>::operator()<>()
    @     0x55f9246dff97  std::__invoke_impl<>()
    @     0x55f9246dff09  _ZSt10__invoke_rIvRSt5_BindIFMN5doris10ThreadPoolEFvvEPS2_EEJEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESA_E4typeEOSB_DpOSC_
    @     0x55f9246dfacf  std::_Function_handler<>::_M_invoke()
    @     0x55f920250d77  std::function<>::operator()()
    @     0x55f924679214  doris::Thread::supervise_thread()
    @     0x7f18418d8609  start_thread
    @     0x7f1841b85133  clone

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 22, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@ByteYue
Copy link
Contributor Author

ByteYue commented Apr 22, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.37% (8916/25210)
Line Coverage: 27.10% (73326/270601)
Region Coverage: 26.23% (37869/144371)
Branch Coverage: 23.04% (19285/83698)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6f163cb77943bbdf172a134ba1e72a8b6a1ee242_6f163cb77943bbdf172a134ba1e72a8b6a1ee242/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40503 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f163cb77943bbdf172a134ba1e72a8b6a1ee242, data reload: false

------ Round 1 ----------------------------------
q1	17718	4759	4819	4759
q2	2000	185	186	185
q3	10485	1172	1274	1172
q4	10217	817	822	817
q5	7521	2797	2744	2744
q6	238	135	138	135
q7	1067	637	588	588
q8	9210	2269	2223	2223
q9	7582	6839	6849	6839
q10	8717	3625	3635	3625
q11	458	235	235	235
q12	499	231	227	227
q13	17797	2969	2977	2969
q14	290	242	235	235
q15	538	490	477	477
q16	518	398	403	398
q17	1002	744	737	737
q18	7497	6882	6785	6785
q19	7471	1685	1683	1683
q20	673	316	315	315
q21	3817	3034	3130	3034
q22	382	322	321	321
Total cold run time: 115697 ms
Total hot run time: 40503 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4730	4586	4559	4559
q2	387	262	256	256
q3	3058	2784	2781	2781
q4	1935	1660	1616	1616
q5	5447	5458	5364	5364
q6	230	127	131	127
q7	2278	1820	1870	1820
q8	3427	3582	3578	3578
q9	8694	8685	8710	8685
q10	4084	3831	3809	3809
q11	600	486	480	480
q12	790	600	582	582
q13	17574	2956	2959	2956
q14	304	279	269	269
q15	539	486	507	486
q16	529	467	449	449
q17	1927	1627	1579	1579
q18	8096	8220	7997	7997
q19	1928	1771	1736	1736
q20	2126	1873	1860	1860
q21	11547	5164	5123	5123
q22	565	512	472	472
Total cold run time: 80795 ms
Total hot run time: 56584 ms

@dataroaring dataroaring merged commit 8e5369c into apache:master Apr 22, 2024
25 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants