test: box/func_reload flaky fails on hard loaded hosts

[avtikhon]

Tarantool version:
master

OS version:
Linux

Bug description:
Failed:

[004] --- box/func_reload.result	Mon Aug  5 08:08:20 2019
[004] +++ box/func_reload.reject	Wed Aug 14 09:39:01 2019
[004] @@ -217,12 +217,12 @@
[004]  ...
[004]  ch:get()
[004]  ---
[004] +- [[2]]
[004] +...
[004] +ch:get()
[004] +---
[004]  - [[1]]
[004]  ...
[004] -ch:get()
[004] ----
[004] -- [[2]]
[004] -...
[004]  s:drop()
[004]  ---
[004]  ...

Steps to reproduce:

l=0 ; while ./test-run.py -j20 `for r in {1..100} ; do echo box/func_reload.test.lua ; done` 2>/dev/null ; do l=$(($l+1)) ; echo ======== $l ============= ; done

Optional (but very desirable):

• coredump
• backtrace
• netstat

[avtikhon]

Reproduced on 2.4.0-16-gcdf502c66

[avtikhon]

Results from the channel swaps and it is the issue.

To reproduce on dev1 host use:
l=0 ; while ./test-run.py -j100 for r in {1..200} ; do echo box/func_reload_gh-4425.test.lua ; done 2>/dev/null ; do l=$(($l+1)) ; echo ======== $l ============= ; done

box/func_reload_gh-4425.test.lua

fio  = require('fio')
net = require('net.box')
fiber = require('fiber')

ext = (jit.os == "OSX" and "dylib" or "so")
build_path = os.getenv("BUILDDIR")
reload1_path = build_path..'/test/box/reload1.'..ext
reload2_path = build_path..'/test/box/reload2.'..ext
reload_path = "reload."..ext
_ = fio.unlink(reload_path)
fio.symlink(reload1_path, reload_path)

c = net.connect(os.getenv("LISTEN"))
box.schema.func.create('reload.test_reload', {language = "C"})
box.schema.user.grant('guest', 'execute', 'function', 'reload.test_reload')
s = box.schema.space.create('test_reload')
_ = s:create_index('pk')
box.schema.user.grant('guest', 'read,write', 'space', 'test_reload')
ch = fiber.channel(2)
-- call first time to load function
c:call("reload.test_reload")
s:delete({1})
_ = fio.unlink(reload_path)
fio.symlink(reload2_path, reload_path)

_ = fiber.create(function() ch:put(c:call("reload.test_reload")) end)
while s:get({1}) == nil do fiber.yield(0.0001) end
box.schema.func.reload("reload")
_ = fiber.create(function() ch:put(c:call("reload.test_reload")) end)
ch:get()
ch:get()
s:drop()

box.schema.func.drop("reload.test_reload")
_ = fio.unlink(reload_path)