Exceptional future ignored: seastar::gate_closed_exception in sstable_compaction_test.simple_backlog_controller_test #15211
Assignees
Labels
area/compaction
P1
Urgent
symptom/ci stability
Issues that failed in ScyllaDB CI - tests and framework
Milestone
Comments
bhalevy
added
area/compaction
symptom/ci stability
|
dup of #15162? |
|
Yes, though I like the information in this report better :) |
|
Pick the best one from your perspective |
Indeed, the original is not always better :-) Will close that one. |
|
I produced a coredump with: diff --git a/src/core/future.cc b/src/core/future.cc
index c0373654..1d19285b 100644
--- a/src/core/future.cc
+++ b/src/core/future.cc
@@ -213,7 +213,7 @@ void future_state_base::rethrow_exception() const& {
void report_failed_future(const std::exception_ptr& eptr) noexcept {
++engine()._abandoned_failed_futures;
- seastar_logger.warn("Exceptional future ignored: {}, backtrace: {}", eptr, current_backtrace());
+ on_fatal_internal_error(seastar_logger, format("Exceptional future ignored: {}", eptr));
}
void report_failed_future(const future_state_base& state) noexcept { |
|
This change seems to be required and sufficient: diff --git a/test/boost/sstable_compaction_test.cc b/test/boost/sstable_compaction_test.cc
index eb8911fbba..64f62b6edf 100644
--- a/test/boost/sstable_compaction_test.cc
+++ b/test/boost/sstable_compaction_test.cc
@@ -4454,12 +4454,14 @@ SEASTAR_TEST_CASE(simple_backlog_controller_test) {
auto as = abort_source();
auto task_manager = tasks::task_manager({}, as);
+ auto stop_task_manager = deferred_stop(task_manager);
compaction_manager::config cfg = {
.compaction_sched_group = { default_scheduling_group() },
.maintenance_sched_group = { default_scheduling_group() },
.available_memory = available_memory,
};
auto manager = compaction_manager(std::move(cfg), as, task_manager);
+ auto stop_manager = deferred_stop(manager);
auto add_sstable = [&env] (table_for_tests& t, uint64_t data_size, int level) {
auto sst = env.make_sstable(t.schema()); |
|
Nope, still happens. |
|
Looks like the gate_close_exception originates in scylladb/tasks/task_manager.cc Line 186 in 5afc242 and scylladb/tasks/task_manager.cc Line 198 in 5afc242 in the catch clause. |
bhalevy
added a commit
to bhalevy/scylla
that referenced
this issue
Aug 29, 2023
Passing the gate_closed_exception to the task promise ends up with abandoned exception since no-one is waiting for it. Instead, let the exception be thrown from start(). Fixes scylladb#15211 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy
added a commit
to bhalevy/scylla
that referenced
this issue
Aug 30, 2023
Passing the gate_closed_exception to the task promise ends up with abandoned exception since no-one is waiting for it. Instead, let the exception be thrown from start(). Fixes scylladb#15211 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy
added a commit
to bhalevy/scylla
that referenced
this issue
Aug 30, 2023
Passing the gate_closed_exception to the task promise in start() ends up with abandoned exception since no-one is waiting for it. Instead, enter the gate when the task is made so it will fail make_task if the gate is already closed. Fixes scylladb#15211 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy
added a commit
to bhalevy/scylla
that referenced
this issue
Aug 31, 2023
Passing the gate_closed_exception to the task promise in start() ends up with abandoned exception since no-one is waiting for it. Instead, enter the gate when the task is made so it will fail make_task if the gate is already closed. Fixes scylladb#15211 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy
added a commit
to bhalevy/scylla
that referenced
this issue
Sep 4, 2023
Passing the gate_closed_exception to the task promise in start() ends up with abandoned exception since no-one is waiting for it. Instead, enter the gate when the task is made so it will fail make_task if the gate is already closed. Fixes scylladb#15211 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
nyh
added a commit
that referenced
this issue
Sep 6, 2023
…reated' from Benny Halevy Passing the gate_closed_exception to the task promise ends up with abandoned exception since no-one is waiting for it. Instead, enter the gate when the task is made so it will fail make_task if the gate is already closed. Fixes #15211 In addition, this series adds a private abort_source for each task_manager module (chained to the main task_manager::abort_source) and abort is requested on task_manager::module::stop(). gate holding in compaction_manager is hardened and makes sure to stop compaction_manager and task_manager in sstable_compaction_test cases. Closes #15213 * github.com:scylladb/scylladb: compaction_manager: stop: close compaction_state:s gates compaction_manager: gracefully handle gate close task_manager: task: start: fixup indentation task_manager: module: make_task: enter gate when the task is created task_manaer: module: stop: request abort task_manager: task::impl: subscribe to module about_source test: compaction_manager_stop_and_drain_race_test: stop compaction and task managers test: simple_backlog_controller_test: stop compaction and task managers
|
@raphaelsc / @Deexie: which branches require backport of the fix? |
All containing task manager, that is from 5.2 (I don't know if those should be considered but from the PR introducting bug: scylla-5.3.0-rc0 scylla-5.2.9 scylla-5.2.8 scylla-5.2.7 scylla-5.2.6 scylla-5.2.5 scylla-5.2.4 scylla-5.2.3 scylla-5.2.2 scylla-5.2.1 scylla-5.2.0 scylla-5.2.0-rc5 scylla-5.2.0-rc4 scylla-5.2.0-rc3 scylla-5.2.0-rc2 scylla-5.2.0-rc1 scylla-5.2.0-rc0) |
|
@Deexie it's only necessary to name the branch (5.2), not patch releases on that branch. |
avikivity
pushed a commit
that referenced
this issue
Nov 30, 2023
Passing the gate_closed_exception to the task promise in start() ends up with abandoned exception since no-one is waiting for it. Instead, enter the gate when the task is made so it will fail make_task if the gate is already closed. Fixes #15211 Signed-off-by: Benny Halevy <bhalevy@scylladb.com> (cherry picked from commit f9a7635)
|
Backported to 5.2. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Seen in https://jenkins.scylladb.com/view/master/job/scylla-master/job/next/6449/artifact/testlog/x86_64/release/boost.sstable_compaction_test.simple_backlog_controller_test.1.log
This is reproducible locally for me with high repeat count.
Decoded:
I suspect this might have to do with the use of shared_future for compaction tasks.
Cc @Deexie
The text was updated successfully, but these errors were encountered: