ccl/backupccl: TestDataDriven failed #93492

cockroach-teamcity · 2022-12-13T09:14:43Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ 2a1bed15513905dd4172f66bfc2421326775fd09:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x1e7ad03]

Stack:

goroutine 552125834 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x6bcd468?, {0x6bccb38, 0xc0e778eb10})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:229 +0x6a
panic({0x4e97dc0, 0x988a4b0})
	GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).poisonInflightLatches(0xc0063e0000, {0x6b90780, 0xc0ed9f36c0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1350 +0x1e3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaCircuitBreaker).asyncProbe.func1({0x6bccb38, 0xc0e778eb10})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_circuit_breaker.go:200 +0xaf
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x43b

Log preceding fatal error

        defaultdb root eu-north-1  {eu-north-1} zone
        eu_central_db root eu-central-1  {eu-central-1} zone
        no_region_db root eu-north-1  {eu-north-1} zone
        no_region_db_2 root eu-north-1  {eu-north-1} zone
        postgres root eu-north-1  {eu-north-1} zone
        system node <nil> <nil> {} <nil>
*
* ERROR: a panic has occurred!
* runtime error: invalid memory address or nil pointer dereference
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:884
*   | runtime.panicmem
*   | 	GOROOT/src/runtime/panic.go:260
*   | runtime.sigpanic
*   | 	GOROOT/src/runtime/signal_unix.go:835
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).poisonInflightLatches
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1350
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaCircuitBreaker).asyncProbe.func1
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_circuit_breaker.go:200
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
*   | runtime.goexit
*   | 	GOROOT/src/runtime/asm_amd64.s:1594
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*
*
* ERROR: a panic has occurred!
* runtime error: invalid memory address or nil pointer dereference
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:884
*   | runtime.panicmem
*   | 	GOROOT/src/runtime/panic.go:260
*   | runtime.sigpanic
*   | 	GOROOT/src/runtime/signal_unix.go:835
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).poisonInflightLatches
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1350
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaCircuitBreaker).asyncProbe.func1
*   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_circuit_breaker.go:200
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
*   | runtime.goexit
*   | 	GOROOT/src/runtime/asm_amd64.s:1594
* Wraps: (2) runtime error: invalid memory address or nil pointer dereference
* Error types: (1) *withstack.withStack (2) runtime.errorString
*

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

ccl/backupccl: TestDataDriven failed #92886 ccl/backupccl: TestDataDriven failed [C-test-failure O-robot T-disaster-recovery branch-release-22.2.0]

/cc @cockroachdb/disaster-recovery _{This test on roachdash | Improve this report!

Jira issue: CRDB-22363}

The text was updated successfully, but these errors were encountered:

renatolabs · 2022-12-13T16:36:52Z

This has failed on master again, but with a different error:

    datadriven_test.go:410: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/7733/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/multiregion:111:
         RESTORE DATABASE no_region_db FROM LATEST IN 'nodelocal://1/no_region_database_backup/';
        expected:
        NOTICE: setting the PRIMARY REGION as eu-central-1 on database no_region_db
        HINT: to change the default primary region, use SET CLUSTER SETTING sql.defaults.primary_region = 'region' or use RESET CLUSTER SETTING sql.defaults.primary_region to disable this behavior
        
        found:
        pq: region "non-existent-region" does not exist
        HINT: valid regions: eu-central-1, eu-north-1

Build: https://teamcity.cockroachdb.com/viewLog.html?buildId=7944353&buildTypeId=Cockroach_PostMerge_MergedArm64Tests

cockroach-teamcity · 2022-12-16T06:52:28Z

ccl/backupccl.TestDataDriven failed with artifacts on master @ dcdf599e9eaee8010137167f419509ccb7627406:

        query-sql [0 args]
        SELECT database_name FROM [SHOW DATABASES] WHERE database_name = 'b2';
        ----
        b2
    datadriven_test.go:410: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/in-progress-restores:149:
        query-sql [0 args]
        SELECT * FROM b2.me.baz;
        ----
        1 howdy
        2 hello
        3 heyyy
    datadriven_test.go:410: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/in-progress-restores:157:
        query-sql [0 args]
        SELECT * FROM [SHOW INDEX FROM b2.me.baz] WHERE index_name = 'greeting_idx';
        ----
        baz greeting_idx true 1 y ASC false false true
        baz greeting_idx true 2 rowid ASC false true true
    datadriven_test.go:410: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/in-progress-restores:169:
        exec-sql [0 args]
        USE d;
        DROP SCHEMA IF EXISTS me CASCADE;
        ----
    datadriven_test.go:410: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/testdata/backup-restore/in-progress-restores:175:
        exec-sql [0 args]
        SET CLUSTER SETTING jobs.debug.pausepoints = restore.before_publishing_descriptors;
        ----
    datadriven_test.go:443: 
        	Error Trace:	/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/datadriven_test.go:443
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/datadriven_test.go:725
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/datadriven.go:320
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/datadriven.go:325
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/datadriven.go:194
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/datadriven.go:167
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/datadriven.go:135
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/datadriven_test.go:410
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/datadriven.go:412
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3563/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/backupccl/backupccl_test_/backupccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/backupccl/datadriven.go:425
        	Error:      	Expect "pq: referenced schema ID 109: descriptor is a *dbdesc.immutable: unexpected descriptor type" to match "pause point .* hit$"
        	Test:       	TestDataDriven/in-progress-restores
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:199: no Invalid Descriptors
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    testutils.go:181: Warning: Could not check for invalid descriptors: sql: database is closed
    --- FAIL: TestDataDriven/in-progress-restores (6.08s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

ccl/backupccl: TestDataDriven failed #93773 ccl/backupccl: TestDataDriven failed [C-test-failure O-robot T-disaster-recovery branch-release-22.1]
ccl/backupccl: TestDataDriven failed #92886 ccl/backupccl: TestDataDriven failed [C-test-failure O-robot T-disaster-recovery branch-release-22.2.0]

_{This test on roachdash | Improve this report!}

msbutler · 2022-12-19T17:39:35Z

Closing as each error has been addressed or has a separate issue:

First error Issue kvserver: v22.1.4: nil pointer in poisonInflightLatches #86547
Second fixed by backupccl: deflake multinode datadriven tests that set cluster settings #92911
Third discussed in slack https://cockroachlabs.slack.com/archives/C03JCUUSCD6/p1671216425942199

cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Dec 13, 2022

cockroach-teamcity added this to the 23.1 milestone Dec 13, 2022

blathers-crl bot added the T-disaster-recovery label Dec 13, 2022

cockroach-teamcity mentioned this issue Dec 16, 2022

ccl/backupccl: TestDataDriven failed #93773

Closed

msbutler closed this as completed Dec 19, 2022

This was referenced Dec 19, 2022

backupccl: TestDataDriven/multiregion fails under stress during 4th test cluster startup #93941

Closed

kvserver: v22.1.4: nil pointer in poisonInflightLatches #86547

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ccl/backupccl: TestDataDriven failed #93492

ccl/backupccl: TestDataDriven failed #93492

cockroach-teamcity commented Dec 13, 2022 •

edited by cockroach-jira-scripts

Loading

renatolabs commented Dec 13, 2022

cockroach-teamcity commented Dec 16, 2022

msbutler commented Dec 19, 2022

ccl/backupccl: TestDataDriven failed #93492

ccl/backupccl: TestDataDriven failed #93492

Comments

cockroach-teamcity commented Dec 13, 2022 • edited by cockroach-jira-scripts Loading

renatolabs commented Dec 13, 2022

cockroach-teamcity commented Dec 16, 2022

msbutler commented Dec 19, 2022

cockroach-teamcity commented Dec 13, 2022 •

edited by cockroach-jira-scripts

Loading