New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

travis-ci: box/on_shutdown.test.lua test flaky fails under highload #4134

Closed

avtikhon opened this issue Apr 10, 2019 · 3 comments

Assignees

Labels

Milestone

Contributor

avtikhon commented Apr 10, 2019 •

edited

Tarantool version:
master

OS version image:
packpack/packpack:debian-stretch

Bug description:
The following failed even after the grep_log was changed to wait_log with 30 secs timeout

[079] --- box/misc.result Wed Apr 10 15:13:37 2019
[079] +++ box/misc.reject Wed Apr 10 18:12:13 2019
[079] @@ -1406,7 +1406,6 @@
[079] ...
[079] test_run:wait_log('test', 'on_shutdown 5', nil, 30, {noreset=true})
[079] ---
[079] -- on_shutdown 5
[079] ...
[079] -- make sure we exited because of os.exit(), not a signal.
[079] test_run:wait_log('test', 'signal', nil, 10, {noreset=true})
[079]

Steps to reproduce:
./test-run.py -j 50 box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua box/misc.test.lua

Optional (but very desirable):

coredump
backtrace
netstat

avtikhon added the flaky test label

avtikhon assigned sergepetrenko

sergepetrenko added a commit that referenced this issue


          test: fix box/misc failure in parallel mode

d0e8a9f

Rewrite the test so that it doesn't depend on timeouts, which can be
exceeded under high load, and simplify it a little.

Closes #4134

avtikhon pushed a commit that referenced this issue


          test: fix box/misc failure in parallel mode

5c9d104

Rewrite the test so that it doesn't depend on timeouts, which can be
exceeded under high load, and simplify it a little.

Closes #4134

sergepetrenko added a commit that referenced this issue


          test: rework box/misc test part regarding on_shutdown triggers

70b5c7c

Rewrite the test so that it doesn't depend on timeouts, which can be
exceeded under high load, too much and simplify it a little.

Also extract the whole part regarding on_shutdown triggers into a
separate test, so that its sporadic failures would be easier to
investigate.

Part of #4134

sergepetrenko added a commit that referenced this issue


          test: extract on_shutdown tests from box/misc

ce79f3b

This part of the test is flaky when tests are run in parallel, besides,
it is quite big on its own, so extract it into a separate file to add
more flexibility in running tests and to make finding problems easier.

Part of #4134

sergepetrenko added a commit that referenced this issue


          test: rework box/on_shutdown test

96a2be9

The test is flaky under high load (e.g. when is run in parallel with a
lot of workers). Make it less dependent on arbitrary timeouts to improve
stability.

Part of #4134

locker pushed a commit that referenced this issue


          test: extract on_shutdown tests from box/misc

70ea999

This part of the test is flaky when tests are run in parallel, besides,
it is quite big on its own, so extract it into a separate file to add
more flexibility in running tests and to make finding problems easier.

Part of #4134

locker pushed a commit that referenced this issue


          test: rework box/on_shutdown test

c026052

The test is flaky under high load (e.g. when is run in parallel with a
lot of workers). Make it less dependent on arbitrary timeouts to improve
stability.

Part of #4134

locker pushed a commit that referenced this issue


          test: extract on_shutdown tests from box/misc

773462c

This part of the test is flaky when tests are run in parallel, besides,
it is quite big on its own, so extract it into a separate file to add
more flexibility in running tests and to make finding problems easier.

Part of #4134

(cherry picked from commit 70ea999)

locker pushed a commit that referenced this issue


          test: rework box/on_shutdown test

The test is flaky under high load (e.g. when is run in parallel with a
lot of workers). Make it less dependent on arbitrary timeouts to improve
stability.

Part of #4134

(cherry picked from commit c026052)

avtikhon pushed a commit that referenced this issue


          test: extract on_shutdown tests from box/misc

2a39a2c

This part of the test is flaky when tests are run in parallel, besides,
it is quite big on its own, so extract it into a separate file to add
more flexibility in running tests and to make finding problems easier.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: rework box/on_shutdown test

5eb7156

The test is flaky under high load (e.g. when is run in parallel with a
lot of workers). Make it less dependent on arbitrary timeouts to improve
stability.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: extract on_shutdown tests from box/misc

cec613d

This part of the test is flaky when tests are run in parallel, besides,
it is quite big on its own, so extract it into a separate file to add
more flexibility in running tests and to make finding problems easier.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: rework box/on_shutdown test

7da1970

The test is flaky under high load (e.g. when is run in parallel with a
lot of workers). Make it less dependent on arbitrary timeouts to improve
stability.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: extract on_shutdown tests from box/misc

41a7d08

This part of the test is flaky when tests are run in parallel, besides,
it is quite big on its own, so extract it into a separate file to add
more flexibility in running tests and to make finding problems easier.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: rework box/on_shutdown test

fee5682

The test is flaky under high load (e.g. when is run in parallel with a
lot of workers). Make it less dependent on arbitrary timeouts to improve
stability.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: extract on_shutdown tests from box/misc

d1809b2

This part of the test is flaky when tests are run in parallel, besides,
it is quite big on its own, so extract it into a separate file to add
more flexibility in running tests and to make finding problems easier.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: rework box/on_shutdown test

c10018e

The test is flaky under high load (e.g. when is run in parallel with a
lot of workers). Make it less dependent on arbitrary timeouts to improve
stability.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: extract on_shutdown tests from box/misc

d426c32

This part of the test is flaky when tests are run in parallel, besides,
it is quite big on its own, so extract it into a separate file to add
more flexibility in running tests and to make finding problems easier.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: rework box/on_shutdown test

bfcd406

The test is flaky under high load (e.g. when is run in parallel with a
lot of workers). Make it less dependent on arbitrary timeouts to improve
stability.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: extract on_shutdown tests from box/misc

83a1928

This part of the test is flaky when tests are run in parallel, besides,
it is quite big on its own, so extract it into a separate file to add
more flexibility in running tests and to make finding problems easier.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: rework box/on_shutdown test

22f0e4c

The test is flaky under high load (e.g. when is run in parallel with a
lot of workers). Make it less dependent on arbitrary timeouts to improve
stability.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: extract on_shutdown tests from box/misc

84bd5b6

This part of the test is flaky when tests are run in parallel, besides,
it is quite big on its own, so extract it into a separate file to add
more flexibility in running tests and to make finding problems easier.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: rework box/on_shutdown test

4b196ed

The test is flaky under high load (e.g. when is run in parallel with a
lot of workers). Make it less dependent on arbitrary timeouts to improve
stability.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: extract on_shutdown tests from box/misc

5e725de

This part of the test is flaky when tests are run in parallel, besides,
it is quite big on its own, so extract it into a separate file to add
more flexibility in running tests and to make finding problems easier.

Part of #4134

avtikhon pushed a commit that referenced this issue


          test: rework box/on_shutdown test

a7609d6

The test is flaky under high load (e.g. when is run in parallel with a
lot of workers). Make it less dependent on arbitrary timeouts to improve
stability.

Part of #4134

kyukhin added the qa label

kyukhin added this to the 2.1.3 milestone

Collaborator

sergepetrenko commented Jun 7, 2019

Doesn't reproduce on my side anymore. (Mac OS X 10.14.5)

./test-run.py -j128 box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown. box/on_shutdown.
...
Statistics:

pass: 128

sergepetrenko added the needs feedback label

Contributor Author

avtikhon commented Jun 17, 2019

Still fails:
https://travis-ci.org/tarantool/tarantool/jobs/546634475

Member

Totktonada commented Jun 20, 2019

Note: The test case is now extracted into box/on_shutdown.test.lua.

Totktonada pushed a commit that referenced this issue


          test: temporary disable on_shutdown.test.lua test

142232c

The test is flaky and often fails in parallel testing. We want to enable
parallel testing within scope of #4156 (enabling GitLab CI).

It should be enabled back in the scope of #4134.

Needed for #4156.

avtikhon added a commit that referenced this issue


          test: temporary disable on_shutdown.test.lua test

15e303a

The test is flaky and often fails in parallel testing. We want to enable
parallel testing within scope of #4156 (enabling GitLab CI).

It should be enabled back in the scope of #4134.

Needed for #4156.

Totktonada changed the title ~~travis-ci: box/misc test flaky fails under highload~~ travis-ci: box/on_shutdown.test.lua test flaky fails under highload

Totktonada pushed a commit that referenced this issue


          test: temporary disable on_shutdown.test.lua test

96e6d5f

The test is flaky and often fails in parallel testing. We want to enable
parallel testing within scope of #4156 (enabling GitLab CI).

It should be enabled back in the scope of #4134.

Needed for #4156.

sergepetrenko added a commit that referenced this issue


          test: fix box/on_shutdown flakiness

8e7c0a7

Replace prints that indicate on_shutdown trigger execution with
log.warn, which is more reliable. This eliminates occasional test
failures. Also instead of waiting for the server to start and executing
grep_log, wait for the desired log entries to appear with wait_log.

Closes #4134

avtikhon added a commit that referenced this issue


          test: temporary disable on_shutdown.test.lua test

15ec132

The test is flaky and often fails in parallel testing. We want to enable
parallel testing within scope of #4156 (enabling GitLab CI).

It should be enabled back in the scope of #4134.

Needed for #4156.

sergepetrenko added ready for review and removed needs feedback labels

avtikhon added a commit that referenced this issue


          test: temporary disable on_shutdown.test.lua test

85d50c0

The test is flaky and often fails in parallel testing. We want to enable
parallel testing within scope of #4156 (enabling GitLab CI).

It should be enabled back in the scope of #4134.

Needed for #4156.

Totktonada closed this as completed in

Totktonada pushed a commit that referenced this issue


          test: fix box/on_shutdown flakiness

ea3e1ad

Replace prints that indicate on_shutdown trigger execution with
log.warn, which is more reliable. This eliminates occasional test
failures. Also instead of waiting for the server to start and executing
grep_log, wait for the desired log entries to appear with wait_log.

Closes #4134

(cherry picked from commit 5046069)

Totktonada removed the ready for review label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment