Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/backupccl: TestTenantBackupNemesis failed #120469

Closed
cockroach-teamcity opened this issue Mar 14, 2024 · 2 comments
Closed

ccl/backupccl: TestTenantBackupNemesis failed #120469

cockroach-teamcity opened this issue Mar 14, 2024 · 2 comments
Labels
branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Projects
Disaster Recovery Backlog
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 14, 2024 •
edited by cockroach-jira-scripts
Loading

ccl/backupccl.TestTenantBackupNemesis failed on master @ 455b16592df7d8efd121b3ba1256fb477e227564:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantBackupNemesis (14m33s)

Stack:

goroutine 886027 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394865509055946.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394865702852747.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394865964198725.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394866313022958.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394866538745518.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394866846173146.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:594: backup-nemesis: CREATE UNIQUE INDEX (will fail) finished
    tenant_backup_nemesis_test.go:589: backup-nemesis: CREATE TENANT started
    tenant_backup_nemesis_test.go:594: backup-nemesis: CREATE TENANT finished
    tenant_backup_nemesis_test.go:589: backup-nemesis: CANCELED IMPORT INTO started
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394867147740581.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394867511039569.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394867820152181.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394868128645714.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394868461097529.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:594: backup-nemesis: CANCELED IMPORT INTO finished
    tenant_backup_nemesis_test.go:589: backup-nemesis: CREATE INDEX started
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394868801894100.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394869103706989.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394869342116435.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394869676994485.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394870038853688.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:547: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394870417644578.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:594: backup-nemesis: CREATE INDEX finished
    tenant_backup_nemesis_test.go:589: backup-nemesis: REPLICATE TENANT started
    tenant_backup_nemesis_test.go:594: backup-nemesis: REPLICATE TENANT finished
    tenant_backup_nemesis_test.go:589: backup-nemesis: CREATE INDEX started
    tenant_backup_nemesis_test.go:551: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:594: backup-nemesis: CREATE INDEX finished
    tenant_backup_nemesis_test.go:254: backup-nemesis: restoring tenant 10 into restored-tenant-10: RESTORE TENANT 10 FROM LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1710394870417644578.0000000000 WITH virtual_cluster = '11', virtual_cluster_name = 'restored-tenant-10'
    tenant_backup_nemesis_test.go:285: backup-nemesis: checking table bank.bank at 1710394870417644578.0000000000
    tenant_backup_nemesis_test.go:285: backup-nemesis: checking table bank."import_into_0edae2b5-ee0d-4e6b-b0cd-f0d5f4855c34" at 1710394870417644578.0000000000
    tenant_backup_nemesis_test.go:285: backup-nemesis: checking table bank."create_index_2ac6dfa4-af08-4c3b-b6d9-0cc3eb23500e" at 1710394870417644578.0000000000

Parameters:

  • attempt=1
  • run=9
  • shard=4
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-36704
@cockroach-teamcity cockroach-teamcity added branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery labels Mar 14, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Mar 14, 2024
@pav-kv
Copy link
Collaborator

pav-kv commented Mar 15, 2024

Another failure in CI.

@msbutler
Copy link
Collaborator

@stevendanna we can close this out now that #120414 has merged?

Disaster Recovery Backlog automation moved this from Backlog to Done Mar 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-disaster-recovery
Projects
Disaster Recovery Backlog
Done
Milestone
24.1
Development

No branches or pull requests
4 participants
@stevendanna @pav-kv @cockroach-teamcity @msbutler