Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flaky test] Lingering job in bgw_db_scheduler_fixed #4859

Closed
mkindahl opened this issue Oct 20, 2022 · 8 comments · Fixed by #4970
Closed

[Flaky test] Lingering job in bgw_db_scheduler_fixed #4859

mkindahl opened this issue Oct 20, 2022 · 8 comments · Fixed by #4970
Assignees
@konskov
Labels
flaky-test Issue about a flaky test

Comments

@mkindahl
Copy link
Contributor

mkindahl commented Oct 20, 2022 •
edited by konskov

Which test is flaky?

bgw_db_scheduler_fixed

Since when has the test been flaky?

October 20, 2022

Link to the failed test run

https://github.com/timescale/timescaledb/actions/runs/3287871530/jobs/5417535202
https://github.com/timescale/timescaledb/actions/runs/3289398600/jobs/5420879404
https://github.com/timescale/timescaledb/actions/runs/3295874767/jobs/5434885747

Log output

diff -u /__w/timescaledb/timescaledb/tsl/test/expected/bgw_db_scheduler_fixed.out /__w/timescaledb/timescaledb/build/tsl/test/results/bgw_db_scheduler_fixed.out
--- /__w/timescaledb/timescaledb/tsl/test/expected/bgw_db_scheduler_fixed.out	2022-10-20 08:09:14.342490485 +0000
+++ /__w/timescaledb/timescaledb/build/tsl/test/results/bgw_db_scheduler_fixed.out	2022-10-20 08:12:32.282002979 +0000
@@ -252,9 +252,10 @@
 
 RESET client_min_messages;
 SELECT application_name FROM pg_stat_activity WHERE application_name LIKE 'User-Defined Action%';
- application_name 
-------------------
-(0 rows)
+      application_name      
+----------------------------
+ User-Defined Action [1001]
+(1 row)
 
 --
 -- Test running a normal job

Reason for flakiness

No response
@mkindahl mkindahl added the flaky-test Issue about a flaky test label Oct 20, 2022
@konskov konskov self-assigned this Oct 20, 2022
@mkindahl
Copy link
Contributor Author

https://github.com/timescale/timescaledb/actions/runs/3289466699/jobs/5421034736

diff -u /__w/timescaledb/timescaledb/tsl/test/expected/bgw_db_scheduler_fixed.out /__w/timescaledb/timescaledb/build/tsl/test/results/bgw_db_scheduler_fixed.out
--- /__w/timescaledb/timescaledb/tsl/test/expected/bgw_db_scheduler_fixed.out	2022-10-20 12:13:57.417784366 +0000
+++ /__w/timescaledb/timescaledb/build/tsl/test/results/bgw_db_scheduler_fixed.out	2022-10-20 12:17:17.251884475 +0000
@@ -1754,7 +1754,7 @@
    1027 | Fri Dec 31 16:00:00 1999 PST     | Fri Dec 31 16:00:00 1999 PST     | Sat Jan 15 16:00:00 2000 PST     | Fri Dec 31 16:00:00 1999 PST
    1028 | Fri Dec 31 16:00:00.005 1999 PST | Fri Dec 31 16:00:00.005 1999 PST | Fri Jan 21 16:00:00.005 2000 PST | Fri Dec 31 16:00:00.005 1999 PST
    1029 | Fri Dec 31 16:00:00.025 1999 PST | Fri Dec 31 16:00:00.025 1999 PST | Mon Jul 31 16:00:00.01 2000 PDT  | Fri Dec 31 16:00:00.025 1999 PST
-   1030 | Fri Dec 31 16:00:00.025 1999 PST | Fri Dec 31 16:00:00.025 1999 PST | Mon Jul 31 16:00:00.01 2000 PDT  | Fri Dec 31 16:00:00.025 1999 PST
+   1030 | Fri Dec 31 16:00:00.025 1999 PST | Fri Dec 31 16:00:00.05 1999 PST  | Mon Jul 31 16:00:00.01 2000 PDT  | Fri Dec 31 16:00:00.05 1999 PST
    1031 | Fri Dec 31 16:00:00.025 1999 PST | Fri Dec 31 16:00:00.025 1999 PST | Sat Jan 15 16:00:00.015 2000 PST | Fri Dec 31 16:00:00.025 1999 PST
 (6 rows)

@konskov
Copy link
Contributor

konskov commented Oct 21, 2022 •
edited

it seems to me that this only fails on 32-bit Linux. I also haven't been able to reproduce it locally so far.
Edit: the lingering job diff does not only occur on 32-bit: https://github.com/timescale/timescaledb/actions/runs/3296144429/jobs/5435466029

@konskov
Copy link
Contributor

konskov commented Nov 2, 2022

note, 20cdd9c has a fix for this flaky test?

@mkindahl
Copy link
Contributor Author

mkindahl commented Nov 4, 2022

note, 20cdd9c has a fix for this flaky test?

Seems like that. Closing. Will reopen if it surfaces again.

@mkindahl mkindahl closed this as completed Nov 4, 2022
@mkindahl
Copy link
Contributor Author

mkindahl commented Nov 7, 2022

Bah. Still failing: https://github.com/timescale/timescaledb/actions/runs/3410736906/jobs/5674063710

@mkindahl mkindahl reopened this Nov 7, 2022
@mkindahl
Copy link
Contributor Author

mkindahl commented Nov 9, 2022

Another failure: https://github.com/timescale/timescaledb/actions/runs/3426036399/jobs/5707418509

@konskov
Copy link
Contributor

konskov commented Nov 10, 2022

another one: https://github.com/timescale/timescaledb/actions/runs/3431413972/jobs/5719521375

@konskov
Copy link
Contributor

konskov commented Nov 11, 2022

looks like this happens daily now: https://github.com/timescale/timescaledb/actions/runs/3440015152/jobs/5737978936

mkindahl added a commit to mkindahl/timescaledb that referenced this issue Nov 14, 2022
@mkindahl
Fix race in bgw_db_scheduler_fixed
cf92dd8 
When deleting a job in the test, the job does not necessarily terminate
immediately, so wait for a log entries from the job before checking the
jobs table.

Fixed timescale#4859
@mkindahl mkindahl mentioned this issue Nov 14, 2022
Merged
mkindahl added a commit to mkindahl/timescaledb that referenced this issue Nov 14, 2022
@mkindahl
Fix race in bgw_db_scheduler_fixed
1573175 
When deleting a job in the test, the job does not necessarily terminate
immediately, so wait for a log entries from the job before checking the
jobs table.

Fixed timescale#4859
mkindahl added a commit that referenced this issue Nov 14, 2022
@mkindahl
Fix race in bgw_db_scheduler_fixed
141e114 
When deleting a job in the test, the job does not necessarily terminate
immediately, so wait for a log entries from the job before checking the
jobs table.

Fixed #4859
SachinSetiya pushed a commit that referenced this issue Nov 28, 2022
@mkindahl @SachinSetiya
Fix race in bgw_db_scheduler_fixed
64330f1 
When deleting a job in the test, the job does not necessarily terminate
immediately, so wait for a log entries from the job before checking the
jobs table.

Fixed #4859
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@konskov konskov

Labels
flaky-test Issue about a flaky test
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix race in bgw_db_scheduler_fixed mkindahl/timescaledb
2 participants
@mkindahl @konskov