Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: short-living non-stopped monitors on ungraceful shutdown #124848

Closed
cockroach-teamcity opened this issue May 30, 2024 · 1 comment · Fixed by #125004
Closed

sql: short-living non-stopped monitors on ungraceful shutdown #124848

cockroach-teamcity opened this issue May 30, 2024 · 1 comment · Fixed by #125004
Assignees
@yuzefovich
Labels
branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-queries SQL Queries Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 30, 2024 •
edited by cockroach-jira-scripts
Loading

pkg/sql/logictest/tests/fakedist/fakedist_test.TestLogic_tenant failed on master @ 99c0d277c26344a08903e1372ab62a034336d178:

Fatal error:

panic: found 1 short-living non-stopped monitors in in-mem temp storage
in-mem temp storage 0 B (512 MiB / 0 B)
    flow-disk-monitor 0 B
 [recovered]
	panic: found 1 short-living non-stopped monitors in in-mem temp storage
in-mem temp storage 0 B (512 MiB / 0 B)
    flow-disk-monitor 0 B
 [recovered]
	panic: found 1 short-living non-stopped monitors in in-mem temp storage
in-mem temp storage 0 B (512 MiB / 0 B)
    flow-disk-monitor 0 B

Stack:

goroutine 94566 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x475de5?, {0x71799b8, 0xc0185ee720})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x65
panic({0x550c2e0?, 0xc00c1e6a20?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x10db58a?, {0x71799b8, 0xc0185ee720})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x65
panic({0x550c2e0?, 0xc00c1e6a20?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop(0xc015ddf0e0, {0x71799b8, 0xc0070794a0}, 0x0)
	github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:657 +0x653
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).EmergencyStop(...)
	github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:593
github.com/cockroachdb/cockroach/pkg/server.newSQLServer.func6()
	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:862 +0x2c
github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close(0xc016207ac0?)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:108 +0x12
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc00ad65a70, {0x71799b8, 0xc0185ee720})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:550 +0x131
github.com/cockroachdb/cockroach/pkg/server.(*channelOrchestrator).propagateUngracefulStopAsync.func1({0x71799b8, 0xc0185ee720})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_channel_orchestrator.go:489 +0x15c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 94565
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
Log preceding fatal error 

=== RUN   TestLogic_tenant
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestLogic_tenant2576607349
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
[05:51:57] setting distsql_workmem='54447B';
=== RUN   TestLogic_tenant/drop_tenant
=== RUN   TestLogic_tenant/read_only
=== RUN   TestLogic_tenant/reclaim_name
=== RUN   TestLogic_tenant/regression_105115
=== RUN   TestLogic_tenant/restrict_system_access
[05:51:59] --- done: /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/tenant with config fakedist: 117 tests, 0 failures

Parameters:

  • attempt=1
  • run=18
  • shard=45
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

Jira issue: CRDB-39107
@cockroach-teamcity cockroach-teamcity added branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team labels May 30, 2024
@github-actions GitHub Actions
Copy link

pkg/sql/logictest/tests/fakedist/fakedist_test.TestLogic_tenant failed on master @ afbd978f872cddd04fd316918d5799da40fad87c:

Fatal error:

panic: found 1 short-living non-stopped monitors in in-mem temp storage
in-mem temp storage 0 B (512 MiB / 0 B)
    flow-disk-monitor 0 B
 [recovered]
	panic: found 1 short-living non-stopped monitors in in-mem temp storage
in-mem temp storage 0 B (512 MiB / 0 B)
    flow-disk-monitor 0 B
 [recovered]
	panic: found 1 short-living non-stopped monitors in in-mem temp storage
in-mem temp storage 0 B (512 MiB / 0 B)
    flow-disk-monitor 0 B

Stack:

goroutine 95398 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x475de5?, {0x71993f8, 0xc013074ab0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x65
panic({0x55199a0?, 0xc00e034f10?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x10db58a?, {0x71993f8, 0xc013074ab0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:231 +0x65
panic({0x55199a0?, 0xc00e034f10?})
	GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).doStop(0xc015ed61e0, {0x71993f8, 0xc0139096b0}, 0x0)
	github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:657 +0x653
github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).EmergencyStop(...)
	github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:593
github.com/cockroachdb/cockroach/pkg/server.newSQLServer.func6()
	github.com/cockroachdb/cockroach/pkg/server/server_sql.go:862 +0x2c
github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close(0xc00e213760?)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:108 +0x12
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc003b206c0, {0x71993f8, 0xc013074ab0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:550 +0x131
github.com/cockroachdb/cockroach/pkg/server.(*channelOrchestrator).propagateUngracefulStopAsync.func1({0x71993f8, 0xc013074ab0})
	github.com/cockroachdb/cockroach/pkg/server/server_controller_channel_orchestrator.go:489 +0x15c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 95397
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
Log preceding fatal error 

=== RUN   TestLogic_tenant
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestLogic_tenant2929405586
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
[11:37:05] setting distsql_workmem='13058B';
=== RUN   TestLogic_tenant/drop_tenant
=== RUN   TestLogic_tenant/read_only
=== RUN   TestLogic_tenant/reclaim_name
=== RUN   TestLogic_tenant/regression_105115
=== RUN   TestLogic_tenant/restrict_system_access
[11:37:08] --- done: /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/tenant with config fakedist: 117 tests, 0 failures

Parameters:

  • attempt=1
  • run=1
  • shard=45
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@yuzefovich yuzefovich self-assigned this May 30, 2024
@yuzefovich yuzefovich removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label May 30, 2024
This was referenced May 30, 2024
Closed
Closed
Merged
Closed
Closed
@yuzefovich yuzefovich changed the title pkg/sql/logictest/tests/fakedist/fakedist_test: TestLogic_tenant failed sql: short-living non-stopped monitors on ungraceful shutdown May 30, 2024
craig bot pushed a commit that referenced this issue May 30, 2024
@yuzefovich
Merge #124870
55b4bb6 
124870: util/mon: disable recently added assertion for now r=yuzefovich a=yuzefovich

There are some more issues to address (probably around the server shutdown), so let's disable the recently added assertion that all short-living monitors are stopped when the parent is stopped.

Informs: #124848
Informs: #124849
Informs: #124850
Epic: None

Release note: None

Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
craig bot pushed a commit that referenced this issue May 30, 2024
@yuzefovich
Merge #124870
c5168ac 
124870: util/mon: disable recently added assertion for now r=yuzefovich a=yuzefovich

There are some more issues to address (probably around the server shutdown), so let's disable the recently added assertion that all short-living monitors are stopped when the parent is stopped.

Informs: #124848
Informs: #124849
Informs: #124850
Epic: None

Release note: None

Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
@yuzefovich yuzefovich mentioned this issue Jun 3, 2024
Merged
@craig craig bot closed this as completed in e4c1bbf Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yuzefovich yuzefovich

Labels
branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-queries SQL Queries Team
Projects
SQL Queries
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

util/mon: re-enable assertion about short-living monitors yuzefovich/cockroach
2 participants
@yuzefovich @cockroach-teamcity