Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

core: use 64 bit head+tail pointers in fair_queue #10

Merged
merged 1 commit into from Sep 1, 2021
Merged

core: use 64 bit head+tail pointers in fair_queue #10

merged 1 commit into from Sep 1, 2021

Conversation

jcsp
Copy link

@jcsp jcsp commented Aug 31, 2021

This is the near-term fix for hanging IOs under heavy load. The switch from uint32_t to uint64_t should be cheap on modern x86_64 processors, but may be more expensive on other archs (I'm not sure what the deal is with the various ARM archs and 8 byte atomics).

When uint32_t offsets were used here, the frequent
integer wrapping caused queues to get stuck. The _pending
tail values would compare as greater than the head values
forever if there weren't always new requests coming through
to wrap the head/tail pointers again.

Signed-off-by: John Spray jcs@vectorized.io

@jcsp
core: use 64 bit head+tail pointers in fair_queue
d831320 
When uint32_t offsets were used here, the frequent
integer wrapping caused queues to get stuck.  The _pending
tail values would compare as greater than the head values
forever if there weren't always new requests coming through
to wrap the head/tail pointers again.

Signed-off-by: John Spray <jcs@vectorized.io>
@CLAassistant
Copy link

CLAassistant commented Aug 31, 2021 •
edited

CLA assistant check
All committers have signed the CLA.

dotnwat
dotnwat approved these changes Aug 31, 2021
View reviewed changes
src/core/fair_queue.cc
return fair_queue_ticket(std::max<int32_t>(_weight - other._weight, 0),
std::max<int32_t>(_size - other._size, 0));

uint64_t weight_d = std::clamp<int64_t>(_weight - other._weight, 0, std::numeric_limits<int32_t>::max());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is _weight - other._weight unsigned? if so, doesn't that mean it will never compare less than 0 and this entire expression is equivalent to min(int32::max, weight-other.weight)?

edit: ohhh, does it cast to that signed int64_t template parameter before any of the calculations? wild interface

@dotnwat dotnwat merged commit 48f7ad9 into redpanda-data:master Sep 1, 2021
@jcsp jcsp deleted the wip-fairqueue-1 branch September 1, 2021 08:44
mmaslankaprv pushed a commit to mmaslankaprv/seastar that referenced this pull request Sep 7, 2021
@bhalevy @avikivity
test: closeable: allocate variables accessed by continuations using d…
4d69526 
…o_with

Fixes failures in debug mode:
```
$ build/debug/tests/unit/closeable_test -l all -t deferred_close_test
WARNING: debug mode. Not for benchmarking or production
random-seed=3064133628
Running 1 test case...
Entering test module "../../tests/unit/closeable_test.cc"
../../tests/unit/closeable_test.cc(0): Entering test case "deferred_close_test"
../../src/testing/seastar_test.cc(43): info: check true has passed
==9449==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
terminate called after throwing an instance of 'seastar::broken_promise'
  what():  broken promise
==9449==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fbf1f49f000; bottom 0x7fbf40971000; size: 0xffffffffdeb2e000 (-558702592)
False positive error reports may follow
For details see google/sanitizers#189
=================================================================
==9449==AddressSanitizer CHECK failed: ../../../../libsanitizer/asan/asan_thread.cpp:356 "((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0)
    #0 0x7fbf45f39d0b  (/lib64/libasan.so.6+0xb3d0b)
    redpanda-data#1 0x7fbf45f57d4e  (/lib64/libasan.so.6+0xd1d4e)
    redpanda-data#2 0x7fbf45f3e724  (/lib64/libasan.so.6+0xb8724)
    redpanda-data#3 0x7fbf45eb3e5b  (/lib64/libasan.so.6+0x2de5b)
    redpanda-data#4 0x7fbf45eb51e8  (/lib64/libasan.so.6+0x2f1e8)
    redpanda-data#5 0x7fbf45eb7694  (/lib64/libasan.so.6+0x31694)
    redpanda-data#6 0x7fbf45f39398  (/lib64/libasan.so.6+0xb3398)
    redpanda-data#7 0x7fbf45f3a00b in __asan_report_load8 (/lib64/libasan.so.6+0xb400b)
    redpanda-data#8 0xfe6d52 in bool __gnu_cxx::operator!=<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > >(__gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&, __gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&) /usr/include/c++/10/bits/stl_iterator.h:1116
    redpanda-data#9 0xfe615c in dl_iterate_phdr ../../src/core/exception_hacks.cc:121
    redpanda-data#10 0x7fbf44bd1810 in _Unwind_Find_FDE (/lib64/libgcc_s.so.1+0x13810)
    redpanda-data#11 0x7fbf44bcd897  (/lib64/libgcc_s.so.1+0xf897)
    redpanda-data#12 0x7fbf44bcea5f  (/lib64/libgcc_s.so.1+0x10a5f)
    redpanda-data#13 0x7fbf44bcefd8 in _Unwind_RaiseException (/lib64/libgcc_s.so.1+0x10fd8)
    redpanda-data#14 0xfe6281 in _Unwind_RaiseException ../../src/core/exception_hacks.cc:148
    redpanda-data#15 0x7fbf457364bb in __cxa_throw (/lib64/libstdc++.so.6+0xaa4bb)
    redpanda-data#16 0x7fbf45e10a21  (/lib64/libboost_unit_test_framework.so.1.73.0+0x1aa21)
    redpanda-data#17 0x7fbf45e20fe0 in boost::execution_monitor::execute(boost::function<int ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2afe0)
    redpanda-data#18 0x7fbf45e21094 in boost::execution_monitor::vexecute(boost::function<void ()> const&) (/lib64/libboost_unit_test_framework.so.1.73.0+0x2b094)
    redpanda-data#19 0x7fbf45e43921 in boost::unit_test::unit_test_monitor_t::execute_and_translate(boost::function<void ()> const&, unsigned long) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d921)
    redpanda-data#20 0x7fbf45e5eae1  (/lib64/libboost_unit_test_framework.so.1.73.0+0x68ae1)
    redpanda-data#21 0x7fbf45e5ed31  (/lib64/libboost_unit_test_framework.so.1.73.0+0x68d31)
    redpanda-data#22 0x7fbf45e2e547 in boost::unit_test::framework::run(unsigned long, bool) (/lib64/libboost_unit_test_framework.so.1.73.0+0x38547)
    redpanda-data#23 0x7fbf45e43618 in boost::unit_test::unit_test_main(bool (*)(), int, char**) (/lib64/libboost_unit_test_framework.so.1.73.0+0x4d618)
    redpanda-data#24 0x44798d in seastar::testing::entry_point(int, char**) ../../src/testing/entry_point.cc:77
    redpanda-data#25 0x4134b5 in main ../../include/seastar/testing/seastar_test.hh:65
    redpanda-data#26 0x7fbf44a1b1e1 in __libc_start_main (/lib64/libc.so.6+0x281e1)
    redpanda-data#27 0x4133dd in _start (/home/bhalevy/dev/seastar/build/debug/tests/unit/closeable_test+0x4133dd)
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20210406100911.12278-1-bhalevy@scylladb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dotnwat dotnwat dotnwat approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@jcsp @CLAassistant @dotnwat