[runtime] Fix possible race in terminate handler #4194

knebekaizer · 2020-05-28T07:46:47Z

No description provided.

projedi

Incidentally, is it possible to write a test that triggers concurrent termination?

runtime/src/main/cpp/Exceptions.cpp

projedi · 2020-06-05T11:34:12Z

runtime/src/main/cpp/Exceptions.cpp

-  RuntimeCheck(oldTerminateHandler != nullptr, "Underlying exception handler is not set.");
-  oldTerminateHandler();
-}
+  typedef __attribute__((noreturn)) void (*QH)();


Nit: What about using using instead of typedef? The former would look like:

using QH = __attribute__((noreturn)) void(*)();

Which I personally find more readable (the name of a thing comes before the thing definition). But if you prefer typedefs then it's fine.

Also, maybe use RUNTIME_NORETURN instead of __attribute__((noreturn)) to keep consistency?

I would agree, but macro RUNTIME_NORETURN has its own issue: it hides the fact it is language extension, not the standard [[noreturn]] as one may expect (or as it may change some day). However, in that particular case [[noreturn]] will not work (function pointer), so I have to stay with gcc extension here, or use explicit cast. So at this point I want to may the choice clear (gcc extension vs. standard language feature). Moreover, I believe the macro RUNTIME_NORETURN is a sort of abuse: it reinvents language feature, hides important aspects and does not add value.

runtime/src/main/cpp/Exceptions.cpp

projedi · 2020-06-05T11:47:59Z

runtime/src/main/cpp/Exceptions.cpp

+    : queuedHandler_((QH)std::set_terminate(kotlinHandler)) {}
+
+  static TerminateHandler *instance() {
+    static TerminateHandler *singleton = new TerminateHandler();


By the way, it's possible to use clang's no_destroy here:

static TerminateHandler& instance() { [[clang::no_destroy]] static TerminateHandler singleton; return singleton; }

It'll avoid heap allocation, but that's about it.

Is this attribute standard or compiler-specific?

Is this attribute standard or compiler-specific?

I think it's clang only for now (there's a proposal but I can't find it's current status). However, I don't think it's a huge problem: we are tied to LLVM, and so unlikely to use any compiler but clang in the foreseeable future.

Well, we are generally free to compile runtime with any C++ compiler.
There are indeed number of blockers, but none of them are fundamental.
So I would prefer moving to compiler-agnostic code, not the opposite.

At the moment it's clang extension and WG21 proposal (P1247) for C++ standard which probably will be accepted. A think it is safe anyway, as this is an optimization technique only and may be ignored if not supported.

knebekaizer · 2020-06-09T06:48:25Z

Incidentally, is it possible to write a test that triggers concurrent termination?

@projedi Yes I wrote a standalone test, i.e. c++ app which does a sort of concurrent stress test for the same implementation. Not related to kotlin per se but sufficient to test the logic. I haven't found a simple way to include such test into our auto test environment.
Here is the gist: https://gist.github.com/knebekaizer/a9fd742d092fa1ba7c37d127b9c14907
You can see there two variants of implementation, both seems to be correct but I choose TerminateHandler2 for better encapsulation.

projedi · 2020-06-09T07:45:06Z

runtime/src/main/cpp/Exceptions.cpp

+    : queuedHandler_((QH)std::set_terminate(kotlinHandler)) {}
+
+  static TerminateHandler* instance() {
+    static TerminateHandler* singleton [[clang::no_destroy]] = new TerminateHandler();


Are pointers necessary here still? Because I don't think no_destroy means anything for primitive types.

projedi · 2020-06-09T07:46:39Z

runtime/src/main/cpp/Exceptions.cpp


-static SimpleMutex konanTerminateHandlerInitializationMutex;
+	// Copy, move and assign would be safe, but not much useful, so let's delete all (rule of 5)
+	TerminateHandler(const TerminateHandler&) = delete;


Formatting: tab indent here, but 2-space indent in the code just above.

projedi · 2020-06-09T07:47:34Z

runtime/src/main/cpp/Exceptions.cpp

 }

+


Formatting: a stray new line?

projedi · 2020-06-09T08:16:17Z

I haven't found a simple way to include such test into our auto test environment.
Here is the gist: https://gist.github.com/knebekaizer/a9fd742d092fa1ba7c37d127b9c14907

@knebekaizer What about something like this: interop test with objc?

create two threads from objc;
initialize kotlin runtime in both;
install an unhandled exception handler (Kotlin's setUnhandledExceptionHook) that prints some string x and just loops indefinitely;
do @try { throw [NSException …] } @catch (...) { objc_terminate() } trick in both threads.

We expect this to print x and terminate at some point. Also, if there'd be diagnostic messages for (1) concurrent termination detected, and (2) force exiting by timeout while waiting for the concurrent termination; we could expect them in output also.

runtime/src/main/cpp/Exceptions.cpp

projedi · 2020-06-11T06:15:03Z

runtime/src/main/cpp/Exceptions.cpp

+  // will not reconstruct handler anyway, so let's keep dtor deleted to avoid confusion.
+  ~TerminateHandler() = delete;
+public:
+  /// First call will do the job, all consecuent will do nothing.


Nit: consequent. Sorry for not spotting this earlier.

projedi · 2020-06-11T06:15:15Z

runtime/src/main/cpp/Exceptions.cpp


+// Use one public funuction to limit access to the class declaration


Nit: function

projedi · 2020-06-11T06:28:05Z

runtime/src/main/cpp/Exceptions.cpp

+        sleep(timeoutSec);
+        // We come here when another terminate handler hangs for 5 sec, that looks fatally broken. Go to forced exit now.
+      }
+      _Exit(EXIT_FAILURE); // force exit


I'm afraid one of my questions got lost. What about logging to stderr facts (1) that there's a concurrent termination attempt and (2) that one of them got tired of waiting and is force quitting?

This may be dangerous, as output itself is a sort of conspicuous regarding races, etc. I consider hanging termination (which already happens here) as extremely emergence case, something is awfully wrong - it may be heap corruption or whatever that affects any printing. So I'd prefer to minimize any actions.

knebekaizer · 2020-06-19T08:08:11Z

I haven't found a simple way to include such test into our auto test environment.
Here is the gist: https://gist.github.com/knebekaizer/a9fd742d092fa1ba7c37d127b9c14907

@knebekaizer What about something like this: interop test with objc?
1. create two threads from objc;

2. initialize kotlin runtime in both;

3. install an unhandled exception handler (Kotlin's `setUnhandledExceptionHook`) that prints some string `x` and just loops indefinitely;

4. do `@try { throw [NSException …] } @catch (...) { objc_terminate() }` trick in both threads.
We expect this to print x and terminate at some point. Also, if there'd be diagnostic messages for (1) concurrent termination detected, and (2) force exiting by timeout while waiting for the concurrent termination; we could expect them in output also.

I suggest stress-test (a number of threads) as a separate commit

SvyatoslavScherbina · 2020-06-19T15:56:04Z

runtime/src/main/cpp/Exceptions.cpp

+  public:
+    template <class Fun> RUNTIME_NORETURN void operator()(Fun block) {
+      if (!compareAndSet(&terminatingFlag, 0, 1)) {
+        block();


Why does it invoke the block if compareAndSet failed?

SvyatoslavScherbina · 2020-06-19T16:04:12Z

runtime/src/main/cpp/Exceptions.cpp


+void reportUnhandledException(KRef throwable) {


This file now has two function with names different only by case of first letters. Doesn't seem ok.

SvyatoslavScherbina · 2020-06-22T07:46:27Z

Incidentally, is it possible to write a test that triggers concurrent termination?

I guess we need at least simple non-concurrent tests. Because nothing else prevented us from having the barely noticeable but fatal typo.

SvyatoslavScherbina · 2020-09-25T06:43:03Z

backend.native/tests/interop/concurrentTerminate/concurrentTerminate.def

+
+---
+
+int test_ConcurrentTerminate();


Suggested change

int test_ConcurrentTerminate();

int test_ConcurrentTerminate(void);

(probably in async.h too)

projedi · 2020-09-25T09:34:20Z

backend.native/tests/interop/concurrentTerminate/async.cpp

+    for (size_t i = 0; i < 100; ++i) {
+        futures.emplace_back(std::async(std::launch::async,
+                [](size_t param) {
+                    std::this_thread::sleep_for(std::chrono::milliseconds(param));


Just for the record (i.e. not doubting your approach), I used the following pattern to induce races:

int launchedThreads = 0; bool threadsCanContinue = false; for (int i = 0; i < numberOfThreads; ++i) { startAThread([]() { launchedThreads += 1; // increment atomically while (!threadsCanContinue) {} // read atomically // Do a racy thing here. }); } while (launchedThreads < numberOfThreads) {} // read atomically threadsCanContinue = true; // write atomically

On my machine this pattern turned out to be quite reliable.

Thanks! Indeedfuture seems to be a bit "indirect" way, in comparison with startAtThread.
My snippet is derived from more complicate test involving exception propagation with promise and set_exception.

I think std::async with futures would work just fine either way. My point was in using spinlocks for synchronisation as opposed to sleeping.

knebekaizer requested a review from SvyatoslavScherbina May 28, 2020 07:46

knebekaizer force-pushed the vi/fix_termination_race branch 3 times, most recently from 7065123 to d3c5515 Compare June 4, 2020 14:49

SvyatoslavScherbina requested a review from projedi June 5, 2020 08:37

projedi reviewed Jun 5, 2020

View reviewed changes

knebekaizer force-pushed the vi/fix_termination_race branch from 68b5bc3 to 587cc56 Compare June 9, 2020 07:15

projedi reviewed Jun 9, 2020

View reviewed changes

runtime/src/main/cpp/Exceptions.cpp Outdated

}

Copy link

Member

projedi Jun 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Formatting: a stray new line?

projedi reviewed Jun 9, 2020

View reviewed changes

runtime/src/main/cpp/Exceptions.cpp Show resolved Hide resolved

projedi reviewed Jun 11, 2020

View reviewed changes

projedi self-requested a review June 19, 2020 07:48

projedi approved these changes Jun 19, 2020

View reviewed changes

knebekaizer force-pushed the vi/fix_termination_race branch from 978c385 to 27cb454 Compare June 19, 2020 10:32

SvyatoslavScherbina reviewed Jun 19, 2020

View reviewed changes

knebekaizer force-pushed the vi/fix_termination_race branch from 2bfbb2e to 7c90d7a Compare September 23, 2020 21:31

SvyatoslavScherbina reviewed Sep 25, 2020

View reviewed changes

SvyatoslavScherbina approved these changes Sep 25, 2020

View reviewed changes

projedi reviewed Sep 25, 2020

View reviewed changes

Vladimir Ivanov added 5 commits September 25, 2020 19:04

[runtime] Fix possible race in terminate handler

2745ab9

Add test (direct interop)

f1a2510

Add test (reverse interop)

90f781c

Test: downgrade from C++14 to 11

3862795

[test] fix and workaround for windows and linux

deb3cbc

knebekaizer force-pushed the vi/fix_termination_race branch from 7c90d7a to deb3cbc Compare September 25, 2020 16:18

knebekaizer merged commit 0058928 into master Sep 28, 2020

knebekaizer deleted the vi/fix_termination_race branch September 28, 2020 12:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[runtime] Fix possible race in terminate handler #4194

[runtime] Fix possible race in terminate handler #4194

knebekaizer commented May 28, 2020

projedi left a comment

projedi Jun 5, 2020

projedi Jun 5, 2020

knebekaizer Jun 8, 2020 •

edited

projedi Jun 5, 2020

SvyatoslavScherbina Jun 9, 2020

projedi Jun 9, 2020

SvyatoslavScherbina Jun 9, 2020

knebekaizer Jun 19, 2020 •

edited

knebekaizer commented Jun 9, 2020 •

edited

projedi Jun 9, 2020

projedi Jun 9, 2020

projedi Jun 9, 2020

projedi commented Jun 9, 2020

projedi Jun 11, 2020

projedi Jun 11, 2020

projedi Jun 11, 2020

knebekaizer Jun 19, 2020

knebekaizer commented Jun 19, 2020

SvyatoslavScherbina Jun 19, 2020

SvyatoslavScherbina Jun 19, 2020

SvyatoslavScherbina commented Jun 22, 2020

SvyatoslavScherbina Sep 25, 2020

projedi Sep 25, 2020

knebekaizer Sep 25, 2020

projedi Sep 25, 2020 •

edited


		// Use one public funuction to limit access to the class declaration

	int test_ConcurrentTerminate();
	int test_ConcurrentTerminate(void);

[runtime] Fix possible race in terminate handler #4194

[runtime] Fix possible race in terminate handler #4194

Conversation

knebekaizer commented May 28, 2020

projedi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knebekaizer Jun 8, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knebekaizer Jun 19, 2020 • edited

Choose a reason for hiding this comment

knebekaizer commented Jun 9, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

projedi commented Jun 9, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knebekaizer commented Jun 19, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SvyatoslavScherbina commented Jun 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

projedi Sep 25, 2020 • edited

Choose a reason for hiding this comment

knebekaizer Jun 8, 2020 •

edited

knebekaizer Jun 19, 2020 •

edited

knebekaizer commented Jun 9, 2020 •

edited

projedi Sep 25, 2020 •

edited