Graceful shutdown exit before shutdown timeout expiration #9411

nshy · 2023-11-27T07:51:29Z

Tarantool version is 3.0.0-beta1-11-g3ee68d8ba.

Repro script:

local fiber = require('fiber')

box.ctl.on_shutdown(function()
    fiber.sleep(0.2)
    print('shutdown callback finished')
end, nil)
box.ctl.set_on_shutdown_timeout(1)

fiber.create(function()
    os.exit(0)
end)

fiber.sleep(0.1)

The script is expected to print shutdown callback finished but it does not.

The text was updated successfully, but these errors were encountered:

Graceful shutdown is done in a special fiber which is started for example on SIGTERM. So it can run concurrently with fiber executing Tarantool init script. On init fiber exit we break event loop to pass control back to the Tarantool initialization code. But we fail to run event loop a bit more to finish graceful shutdown. The test is a bit contrived. A more real world case is when Tarantool is termintated during lingering box.cfg(). Close tarantool#9411 NO_DOC=bugfix

Graceful shutdown is done in a special fiber which is started for example on SIGTERM. So it can run concurrently with fiber executing Tarantool init script. On init fiber exit we break event loop to pass control back to the Tarantool initialization code. But we fail to run event loop a bit more to finish graceful shutdown. The test is a bit contrived. A more real world case is when Tarantool is termintated during lingering box.cfg(). Close #9411 NO_DOC=bugfix

Graceful shutdown is done in a special fiber which is started for example on SIGTERM. So it can run concurrently with fiber executing Tarantool init script. On init fiber exit we break event loop to pass control back to the Tarantool initialization code. But we fail to run event loop a bit more to finish graceful shutdown. The test is a bit contrived. A more real world case is when Tarantool is termintated during lingering box.cfg(). Close #9411 NO_DOC=bugfix (cherry picked from commit 786eb2a)

I got four fails on the given tests in a row on debug-asan job in CI for tarantool-ee. It seems, tarantool-ee is more sensitive to small timeouts, when the address sanitizer slows down the execution. Or I'm just lucky. Anyway, the given tests don't really need small timeouts: increasing it doesn't break any test logic, doesn't increase duration of the test in a successful case and doesn't increase it in case of a failure. The tests are more stable after the change: I verified it locally by running each of the tests in parallel many times on tarantool built with enabled address sanitizer. See the following commits for details about the given test cases and the problems behind. * commit 1fcfb8c ("app: start init script event loop explicitly") * commit 786eb2a ("main: don't break graceful shutdown on init script exit") Follows up tarantool#9266 Follows up tarantool#9411 NO_DOC=test adjustment NO_CHANGELOG=see NO_DOC

I got four fails on the given tests in a row on debug-asan job in CI for tarantool-ee. It seems, tarantool-ee is more sensitive to small timeouts, when the address sanitizer slows down the execution. Or I'm just lucky. Anyway, the given tests don't really need small timeouts: increasing it doesn't break any test logic, doesn't increase duration of the test in a successful case and doesn't increase it in case of a failure. The tests are more stable after the change: I verified it locally by running each of the tests in parallel many times on tarantool built with enabled address sanitizer. See the following commits for details about the given test cases and the problems behind. * commit 1fcfb8c ("app: start init script event loop explicitly") * commit 786eb2a ("main: don't break graceful shutdown on init script exit") Follows up #9266 Follows up #9411 NO_DOC=test adjustment NO_CHANGELOG=see NO_DOC

I got four fails on the given tests in a row on debug-asan job in CI for tarantool-ee. It seems, tarantool-ee is more sensitive to small timeouts, when the address sanitizer slows down the execution. Or I'm just lucky. Anyway, the given tests don't really need small timeouts: increasing it doesn't break any test logic, doesn't increase duration of the test in a successful case and doesn't increase it in case of a failure. The tests are more stable after the change: I verified it locally by running each of the tests in parallel many times on tarantool built with enabled address sanitizer. See the following commits for details about the given test cases and the problems behind. * commit 1fcfb8c ("app: start init script event loop explicitly") * commit 786eb2a ("main: don't break graceful shutdown on init script exit") Follows up tarantool#9266 Follows up tarantool#9411 NO_DOC=test adjustment NO_CHANGELOG=see NO_DOC (cherry picked from commit dfca3c6)

I got four fails on the given tests in a row on debug-asan job in CI for tarantool-ee. It seems, tarantool-ee is more sensitive to small timeouts, when the address sanitizer slows down the execution. Or I'm just lucky. Anyway, the given tests don't really need small timeouts: increasing it doesn't break any test logic, doesn't increase duration of the test in a successful case and doesn't increase it in case of a failure. The tests are more stable after the change: I verified it locally by running each of the tests in parallel many times on tarantool built with enabled address sanitizer. See the following commits for details about the given test cases and the problems behind. * commit 1fcfb8c ("app: start init script event loop explicitly") * commit 786eb2a ("main: don't break graceful shutdown on init script exit") Follows up #9266 Follows up #9411 NO_DOC=test adjustment NO_CHANGELOG=see NO_DOC (cherry picked from commit dfca3c6)

nshy added the bug Something isn't working label Nov 27, 2023

nshy self-assigned this Nov 27, 2023

nshy mentioned this issue Nov 27, 2023

main: don't break graceful shutdown on init script exit #9415

Merged

locker closed this as completed in #9415 Nov 28, 2023

locker added the 2.11 Target is 2.11 and all newer release/master branches label Nov 28, 2023

Totktonada mentioned this issue Dec 13, 2023

test: increase timeouts in on_shutdown tests #9481

Merged

sergepetrenko mentioned this issue Feb 19, 2024

[backport 2.11] test: increase timeouts in on_shutdown tests #9712

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Graceful shutdown exit before shutdown timeout expiration #9411

Graceful shutdown exit before shutdown timeout expiration #9411

nshy commented Nov 27, 2023

Graceful shutdown exit before shutdown timeout expiration #9411

Graceful shutdown exit before shutdown timeout expiration #9411

Comments

nshy commented Nov 27, 2023