Error in net.box on_disconnect trigger hangs server

[drewdzzz]

Consider simple test:

local server = require('luatest.server')
local t = require('luatest')
local net = require('net.box')

local g = t.group()

g.test_net_box_hang = function()
    g.server = server:new()
    g.server:start()

    local conn = net.connect(g.server.net_box_uri, {wait_connected = true})
    local errmsg = 'net.box on_disconnect error'
    conn:on_disconnect(function() error(errmsg) end)
    
    -- Call connection close to receive an error
    t.assert_error_msg_content_equals(errmsg, conn.close, conn)

    g.server:drop()
end

This test fails with such error: Timed out to wait for "process is terminated" condition for server (alias: server, workdir: server-CKeEWskegH-t, pid: 39893) within 60s - it is reported from line g.server:drop(). If I comment out the line that closes the connection, it will work fine.

I suspect that luatest causes the problem because the same scenario using bare Tarantool with closing server by SIGTERM (ctrl + C) does not hang.

My specs: MacOS 11, Darwin Kernel Version 20.6.0
Tarantool is built from sources at this commit, test-run and luatest commits can be found there.
The test was run using test-run utility from Tarantool.

[CuriousGeorgiy]

I get the following error messages in the logs, when reproducing this issue on two bare servers (one with box, one with net.box):

-- server
box.cfg{listen = 3301}

require('console').start()

-- client

c = require('net.box').connect(3301)
c:on_disconnect(function() error(777) end)
pcall(function() c:close() end)
require('console').start()

2024-03-12 14:43:59.173 [87145] main/115/iproto.shutdown I> tx_binary: stopped
2024-03-12 14:44:02.173 [87145] main/103/on_shutdown fiber.c:804 E> TimedOut: timed out
2024-03-12 14:44:02.173 [87145] main/103/on_shutdown main.cc:156 E> on_shutdown triggers failed
2024-03-12 14:44:02.173 [87145] main/103/on_shutdown on_shutdown.c:273 E> TimedOut: timed out
tarantool> 2024-03-12 14:44:02.176 [87145] main I> fiber `sched' has been cancelled
2024-03-12 14:44:02.176 [87145] main I> fiber `sched': exiting
2024-03-12 14:44:02.176 [87145] main F> fatal error, exiting the event loop

If I add the following option to the server: box.ctl.set_on_shutdown_timeout(600) — the server hangs. Apparently, this is a Tarantool bug.