Sentry: pebble.go:1440: log.Dev.Fatal: local corruption detected: pebble: backing file 000048 error: file 000048 (type blob) unknown to the objstorage provider: file does not exist (1) attached stack ...

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/7588239357/?referrer=webhooks_plugin

Panic Message:

pebble.go:1440: log.Dev.Fatal: local corruption detected: pebble: backing file 000048 error: file 000048 (type blob) unknown to the objstorage provider: file does not exist
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func2
  | 	pkg/storage/pebble.go:1440
  | github.com/cockroachdb/pebble.TeeEventListener.func6
  | 	external/com_github_cockroachdb_pebble/event.go:1283
  | github.com/cockroachdb/pebble.(*DB).reportCorruption
  | 	external/com_github_cockroachdb_pebble/event.go:1471
  | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateBlob
  | 	external/com_github_cockroachdb_pebble/file_cache.go:285
  | github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties-range1
  | 	external/com_github_cockroachdb_pebble/table_stats.go:319
  | github.com/cockroachdb/pebble/internal/manifest.(*btree[...]).All.func1
  | 	external/com_github_cockroachdb_pebble/internal/manifest/btree.go:792
  | github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties
  | 	external/com_github_cockroachdb_pebble/table_stats.go:308
  | github.com/cockroachdb/pebble.(*DB).collectTableStats
  | 	external/com_github_cockroachdb_pebble/table_stats.go:121
  | runtime.goexit
  | 	src/runtime/asm_arm64.s:1268
Wraps: (2) secondary error attachment
  | pebble: backing file 000048 error: file 000048 (type blob) unknown to the objstorage provider: file does not exist
  | (1) ×
  | Wraps: (2) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/pebble.NewFileCache.func1
  |   | 	external/com_github_cockroachdb_pebble/file_cache.go:499
  |   | [...repeated from below...]
  | Wraps: (3) pebble: backing file 000048 error
  | Wraps: (4) forced error mark
  |   | ×
  |   | github.com/cockroachdb/errors/withstack/*withstack.withStack::
  | Wraps: (5) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).Lookup
  |   | 	external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:480
  |   | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).OpenForReading
  |   | 	external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:273
  |   | github.com/cockroachdb/pebble.(*fileCacheHandle).openFile
  |   | 	external/com_github_cockroachdb_pebble/file_cache.go:210
  |   | github.com/cockroachdb/pebble.NewFileCache.func1
  |   | 	external/com_github_cockroachdb_pebble/file_cache.go:497
  |   | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
  |   | 	external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:234
  |   | github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
  |   | 	external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:92
  |   | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateBlob
  |   | 	external/com_github_cockroachdb_pebble/file_cache.go:283
  |   | github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties-range1
  |   | 	external/com_github_cockroachdb_pebble/table_stats.go:319
  |   | github.com/cockroachdb/pebble/internal/manifest.(*btree[...]).All.func1
  |   | 	external/com_github_cockroachdb_pebble/internal/manifest/btree.go:792
  |   | github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties
  |   | 	external/com_github_cockroachdb_pebble/table_stats.go:308
  |   | github.com/cockroachdb/pebble.(*DB).collectTableStats
  |   | 	external/com_github_cockroachdb_pebble/table_stats.go:121
  |   | runtime.goexit
  |   | 	src/runtime/asm_arm64.s:1268
  | Wraps: (6) file 000048 (type blob) unknown to the objstorage provider
  | Wraps: (7) file does not exist
  | Error types: (1) *hintdetail.withHint (2) *withstack.withStack (3) *errutil.withPrefix (4) *markers.withMark (5) *withstack.withStack (6) *errutil.withPrefix (7) *errors.errorString
Wraps: (3) log.Dev.Fatal: local corruption detected: pebble: backing file 000048 error: file 000048 (type blob) unknown to the objstorage provider: file does not exist
  | (1) ×
  | Wraps: (2) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/pebble.NewFileCache.func1
  |   | 	external/com_github_cockroachdb_pebble/file_cache.go:499
  |   | [...repeated from below...]
  | Wraps: (3) pebble: backing file 000048 error
  | Wraps: (4) forced error mark
  |   | ×
  |   | github.com/cockroachdb/errors/withstack/*withstack.withStack::
  | Wraps: (5) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).Lookup
  |   | 	external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:480
  |   | github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).OpenForReading
  |   | 	external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:273
  |   | github.com/cockroachdb/pebble.(*fileCacheHandle).openFile
  |   | 	external/com_github_cockroachdb_pebble/file_cache.go:210
  |   | github.com/cockroachdb/pebble.NewFileCache.func1
  |   | 	external/com_github_cockroachdb_pebble/file_cache.go:497
  |   | github.com/cockroachdb/pebble/internal/genericcache.(*shard[...]).findOrCreateValue
  |   | 	external/com_github_cockroachdb_pebble/internal/genericcache/shard.go:234
  |   | github.com/cockroachdb/pebble/internal/genericcache.(*Cache[...]).FindOrCreate
  |   | 	external/com_github_cockroachdb_pebble/internal/genericcache/cache.go:92
  |   | github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateBlob
  |   | 	external/com_github_cockroachdb_pebble/file_cache.go:283
  |   | github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties-range1
  |   | 	external/com_github_cockroachdb_pebble/table_stats.go:319
  |   | github.com/cockroachdb/pebble/internal/manifest.(*btree[...]).All.func1
  |   | 	external/com_github_cockroachdb_pebble/internal/manifest/btree.go:792
  |   | github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties
  |   | 	external/com_github_cockroachdb_pebble/table_stats.go:308
  |   | github.com/cockroachdb/pebble.(*DB).collectTableStats
  |   | 	external/com_github_cockroachdb_pebble/table_stats.go:121
  |   | runtime.goexit
  |   | 	src/runtime/asm_arm64.s:1268
  | Wraps: (6) file 000048 (type blob) unknown to the objstorage provider
  | Wraps: (7) file does not exist
  | Error types: (1) *hintdetail.withHint (2) *withstack.withStack (3) *errutil.withPrefix (4) *markers.withMark (5) *withstack.withStack (6) *errutil.withPrefix (7) *errors.errorString
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Dev.Fatal: local corruption detected: pebble: backing file 000048 error: file 000048 (type blob) unknown to the objstorage provider: file does not exist
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
pebble.go:1440: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

src/runtime/asm_arm64.s#L1267-L1269
external/com_github_cockroachdb_pebble/table_stats.go#L120-L122
external/com_github_cockroachdb_pebble/table_stats.go#L307-L309
external/com_github_cockroachdb_pebble/internal/manifest/btree.go#L791-L793
external/com_github_cockroachdb_pebble/table_stats.go#L318-L320
external/com_github_cockroachdb_pebble/file_cache.go#L284-L286
external/com_github_cockroachdb_pebble/event.go#L1470-L1472
external/com_github_cockroachdb_pebble/event.go#L1282-L1284
pkg/storage/pebble.go#L1439-L1441

src/runtime/asm_arm64.s in runtime.goexit at line 1268
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).collectTableStats at line 121
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties at line 308
external/com_github_cockroachdb_pebble/internal/manifest/btree.go in github.com/cockroachdb/pebble/internal/manifest.(*btree[...]).All.func1 at line 792
external/com_github_cockroachdb_pebble/table_stats.go in github.com/cockroachdb/pebble.(*DB).scanBlobFileProperties-range1 at line 319
external/com_github_cockroachdb_pebble/file_cache.go in github.com/cockroachdb/pebble.(*fileCacheHandle).findOrCreateBlob at line 285
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.(*DB).reportCorruption at line 1471
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func6 at line 1283
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func2 at line 1440

Tags

Tag	Value
Command	start-single-node
Environment	v26.2.3
Go Version	go1.25.5
Platform	linux arm64
Distribution	CCL
Cockroach Release	v26.2.3
Cockroach SHA	90d3b60
# of CPUs	13
# of Goroutines	71

Jira issue: CRDB-65342

[blathers-crl]

CC'ing via the CODEOWNERS-based sentry heuristic:

• @cockroachdb/storage

Sentry issue cause: pkg/storage/pebble.go

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}