Skip to content

cnservice bootstrap panics on context.Canceled instead of returning error #24064

@XuPeng-SH

Description

@XuPeng-SH

Description

cnservice.(*service).bootstrap() at pkg/cnservice/server.go:944-945 panics on ANY bootstrap error, including context.Canceled and context.DeadlineExceeded. This causes unrecoverable crashes instead of graceful error propagation when the bootstrap timeout expires or the service context is canceled during shutdown.

Root Cause

The current code:

if err := s.bootstrapService.Bootstrap(ctx); err != nil {
    panic(moerr.AttachCause(ctx, err))
}

Panics on all errors, but context cancellation/timeout is NOT a bootstrap failure — it's an external signal that the operation should stop.

Inconsistency

The same file already handles context.Canceled gracefully for BootstrapUpgrade (line 955):

if err != context.Canceled {
    // only log non-cancel errors
}

But initial Bootstrap does not have this guard.

Impact

  • All embedded cluster tests with multiple CNs can fail with panic: context canceled when bootstrap takes longer than 5 minutes
  • Tests affected: TestBasicCluster, Test_TxnExecutorExec, TestPartitionBasedShardCanBeCreated, TestDeleteAndSelect, TestCreateAndDropPitr, Test_UpgradeEntry, TestAffectedRows, TestCompile, and more
  • In production, this could crash a CN process during graceful shutdown

Additional Issue

pkg/embed/cluster.go:doStartLocked() also panics on CN Start() errors instead of propagating them, preventing callers from handling startup failures gracefully.

Proposed Fix

  1. Check for context cancellation/timeout before panicking in bootstrap
  2. Return error instead of panicking for context-related failures
  3. Change doStartLocked() to propagate errors instead of panicking

Metadata

Metadata

Assignees

Labels

kind/bugSomething isn't workingseverity/s0Extreme impact: Cause the application to break down and seriously affect the use

Type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions