Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

org.yb.pgsql.TestPgRegressFeature.testPgRegressFeature , org.yb.pgsql.TestPgRegressTypesNumeric.testPgRegressTypes are leaving stray postgres processes behind #946

Open
mbautin opened this issue Mar 7, 2019 · 3 comments
Open

org.yb.pgsql.TestPgRegressFeature.testPgRegressFeature , org.yb.pgsql.TestPgRegressTypesNumeric.testPgRegressTypes are leaving stray postgres processes behind #946

mbautin opened this issue Mar 7, 2019 · 3 comments
Labels
kind/bug This issue is a bug kind/failing-test Tests and testing infra
Projects
CI/CD YSQL

Comments

@mbautin
Copy link
Collaborator

mbautin commented Mar 7, 2019

One postgres backend process is failing to terminate. It looks like there is a connection leak or a refcount leak.

Example output:

84003 [process_tree_supervisor.py:133] 2019-03-07 01:59:12,607 INFO: YB_STRAY_PROCESS: Found a stray process, killing: pid: 13573, command line: ['$BUILD_ROOT/postgres/bin/postgres', '-D', '/tmp/ybtest-jenkins-1551923640617-1176848888/ts-127.204.35.125-17749-1551923641610/pg_data', '-p', '24000', '-h', '127.204.35.125', '-k', '']
84004 [process_tree_supervisor.py:133] 2019-03-07 01:59:12,607 INFO: YB_STRAY_PROCESS: Found a stray process, killing: pid: 13635, command line: ['postgres: checkpointer ', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']
84005 [process_tree_supervisor.py:133] 2019-03-07 01:59:12,608 INFO: YB_STRAY_PROCESS: Found a stray process, killing: pid: 13636, command line: ['postgres: stats collector ', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']
84006 [process_tree_supervisor.py:133] 2019-03-07 01:59:12,608 INFO: YB_STRAY_PROCESS: Found a stray process, killing: pid: 15477, command line: ['postgres: postgres postgres 127.0.0.1(55940) idle', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']
84007 [process_tree_supervisor.py:157] 2019-03-07 01:59:12,609 WARNING: Stray process still running: 13573, cmd line: []
84008 [process_tree_supervisor.py:157] 2019-03-07 01:59:12,609 WARNING: Stray process still running: 13635, cmd line: []
84009 [process_tree_supervisor.py:157] 2019-03-07 01:59:12,609 WARNING: Stray process still running: 15477, cmd line: []
84010 [process_tree_supervisor.py:221] 2019-03-07 01:59:12,609 INFO: Supervisor of pid 13317's process tree (this script's pid: 13333) is terminating (caught signal 10 / SIGUSR1
84011 [process_tree_supervisor.py:231] 2019-03-07 01:59:12,609 INFO: Supervisor process 13333 is exiting with code 0
@mbautin
Copy link
Collaborator Author

mbautin commented Mar 7, 2019

Narrowed it down to the yb_numeric test:

# src/test/regress/yb_pg_type_serial_schedule
#
####################################################################################################
# Postgres Testsuites: Porting from PostgreSQL original tests.
# This suite includes all tests on numeric datatypes that are enabled for YugaByte.
####################################################################################################
#test: yb_oid
#test: yb_int2
#test: yb_int4
#test: yb_int8
#test: yb_float4
#test: yb_float8
test: yb_numeric
#test: yb_numeric_big
#test: yb_money

Then yb_build.sh release --java-test org.yb.pgsql.TestPgRegressTypesNumeric#testPgRegressTypes finishes with:

[process_tree_supervisor.py:126] 2019-03-07 08:30:05,128 INFO: YB_STRAY_PROCESS: Found a stray process, killing: pid: 23544, command line: ['~/code/yugabyte/build/release-gcc-dynamic-enterprise-ninja/postgres/bin/postgres', '-D', '/tmp/ybtest-mbautin-1551947146415-1282002766/ts-127.141.249.147-29563-1551947147427/pg_data', '-p', '17975', '-h', '127.141.249.147', '-k', '']
[process_tree_supervisor.py:126] 2019-03-07 08:30:05,129 INFO: YB_STRAY_PROCESS: Found a stray process, killing: pid: 23580, command line: ['postgres: stats collector process ', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']
[process_tree_supervisor.py:126] 2019-03-07 08:30:05,129 INFO: YB_STRAY_PROCESS: Found a stray process, killing: pid: 23579, command line: ['postgres: checkpointer process ', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']
[process_tree_supervisor.py:126] 2019-03-07 08:30:05,130 INFO: YB_STRAY_PROCESS: Found a stray process, killing: pid: 23895, command line: ['postgres: postgres postgres 127.0.0.1(33344) idle', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '']
[process_tree_supervisor.py:150] 2019-03-07 08:30:05,130 WARNING: Stray process still running: 23544, cmd line: []
[process_tree_supervisor.py:150] 2019-03-07 08:30:05,131 WARNING: Stray process still running: 23579, cmd line: []
[process_tree_supervisor.py:206] 2019-03-07 08:30:05,131 INFO: Supervisor of pid 23308's process tree (this script's pid: 23323) is terminating (caught signal 10 / SIGUSR1
[process_tree_supervisor.py:216] 2019-03-07 08:30:05,131 INFO: Supervisor process 23323 is exiting with code 0
[dev-server-mbautin-4cpu-15gb-1 2019-03-07T08:30:05 run-test.sh:71 cleanup] Stray processes reported in /tmp/yb_process_supervisor_for_pid_23308__29137.log, considering the test failed.

@rkarthik007 rkarthik007 added the kind/bug This issue is a bug label Mar 8, 2019
@rkarthik007 rkarthik007 added this to To do in CI/CD via automation Mar 8, 2019
@rkarthik007 rkarthik007 added this to To do in YSQL via automation Mar 8, 2019
@bmatican bmatican added the kind/failing-test Tests and testing infra label Mar 20, 2019
@ndeodhar ndeodhar moved this from To do to Backlog in YSQL Aug 5, 2020
@m-iancu
Copy link
Contributor

m-iancu commented Mar 13, 2022

Couldn't find this error in the latest test runs. Assuming it was fixed since unless the issue appears again.

@m-iancu m-iancu closed this as completed Mar 13, 2022
CI/CD automation moved this from To do to Done Mar 13, 2022
YSQL automation moved this from Backlog to Done Mar 13, 2022
@deeps1991
Copy link
Contributor

deeps1991 commented Feb 6, 2023 •
edited

This issue is not actually fixed, looks like we just masked the failure in common-test-env.sh as its not a serious issue:

# TODO: Re-enable the logic below for treating stray processes as test failures, when the
    # following tests are fixed:
    # - org.yb.pgsql.TestPgRegressFeature.testPgRegressFeature
    # - org.yb.pgsql.TestPgRegressTypesNumeric.testPgRegressTypes
    # See https://github.com/YugaByte/yugabyte-db/issues/946 for details.
    if false && grep -q YB_STRAY_PROCESS "$process_supervisor_log_path"; then
      log "Stray processes reported in $process_supervisor_log_path, considering the test failed."
      log "The JUnit-compatible XML file will be updated to reflect this error."
      process_supervisor_success=false

@deeps1991 deeps1991 reopened this Feb 6, 2023
CI/CD automation moved this from Done to In progress Feb 6, 2023
YSQL automation moved this from Done to In progress Feb 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug This issue is a bug kind/failing-test Tests and testing infra
Projects
CI/CD
  
In progress
YQL-beta
Status: Done
YSQL
  
In progress
Milestone
No milestone
Development

No branches or pull requests
5 participants
@bmatican @mbautin @m-iancu @rkarthik007 @deeps1991