Skip to content

kv/kvserver/storeliveness: TestStoreLivenessRestart failed #148566

@cockroach-teamcity

Description

@cockroach-teamcity
Member

kv/kvserver/storeliveness.TestStoreLivenessRestart failed on release-25.2.2-rc @ 0780d511bd527bfd9d02f4c7ca5ecc19115d9f5e:

=== RUN   TestStoreLivenessRestart
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestStoreLivenessRestart2469995663
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:154: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.

Parameters:

  • attempt=1
  • race=true
  • run=2
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-51675

Activity

added
C-test-failureBroken test (automatically or manually discovered).
O-robotOriginated from a bot.
release-blockerIndicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
on Jun 19, 2025
added
C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
A-testingTesting tools and infrastructure
and removed
release-blockerIndicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
on Jun 20, 2025
tbg

tbg commented on Jun 20, 2025

@tbg
Member
IsMetamorphicBuild==false
^V=== RUN   TestStoreLivenessAllToAllSupport
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestStoreLivenessAllToAllSupport853018984
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:154: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
    multi_store_test.go:175: -- test log scope end --
^V--- PASS: TestStoreLivenessAllToAllSupport (121.65s)
^V=== NAME
^V=== RUN   TestStoreLivenessRestart
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestStoreLivenessRestart2469995663
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:154: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
I250619 10:56:57.671395 1 (gostd) testmain.go:125  [-] 1  Test //pkg/kv/kvserver/storeliveness:storeliveness_test exited with error code -1

so exited 255, which I'm assuming is an OOM.

The last mem rss output line shows 3.7GiB

storelivenesstest-healthI250619 10:56:53.806689 34953 2@util/log/event_log.go:39 ⋮ [T1,Vsystem,n1] 854 ={"Timestamp":1750330613806672382,"EventType":"runtime_stats","MemRSSBytes":3763564544,"GoroutineCount":1865,"MemStackSysBytes":30015488,"GoAllocBytes":993658096,"GoTotalBytes":1213666888,"HeapFragmentBytes":24001296,"HeapReservedBytes":145293312,"HeapReleasedBytes":90062848,"CGoAllocBytes":401536,"CGoTotalBytes":2842624,"CGoCallRate":0.29712048,"CPUUserPercent":88.24478,"CPUSysPercent":7.9232135,"GCPausePercent":0.00049187307,"GCRunCount":102,"NetHostRecvBytes":563129,"NetHostSendBytes":563129}

which is just for one of the nodes. So I buy that.
The cluster seems to hang around in an unavailable state, likely due to the duress build.

added a commit that references this issue on Jun 23, 2025
blathers-crl

blathers-crl commented on Jun 23, 2025

@blathers-crl

Based on the specified backports for linked PR #148595, I applied the following new label(s) to this issue: branch-release-25.2. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

10 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      Participants

      @tbg@cockroach-teamcity

      Issue actions

        kv/kvserver/storeliveness: TestStoreLivenessRestart failed · Issue #148566 · cockroachdb/cockroach