Sentry: db.go:812: log.Fatal: pebble: fatal commit error: write ×: disk quota exceeded (1) attached stack trace -- stack trace: | github.com/cockroachdb/pebble.(*DB).applyInternal | github.com... #105265

cockroach-sentry · 2023-06-21T09:59:01Z

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4265053338/?referrer=webhooks_plugin

Panic Message:

db.go:812: log.Fatal: pebble: fatal commit error: write ×: disk quota exceeded
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/pebble.(*DB).applyInternal
  | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:812
  | github.com/cockroachdb/pebble.(*DB).Apply
  | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:747
  | github.com/cockroachdb/pebble.(*DB).Set
  | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:600
  | github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).put
  | 	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1608
  | github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).PutUnversioned
  | 	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1585
  | github.com/cockroachdb/cockroach/pkg/storage.(*putBuffer).putInlineMeta
  | 	github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1393
  | github.com/cockroachdb/cockroach/pkg/storage.mvccPutInternal
  | 	github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1810
  | github.com/cockroachdb/cockroach/pkg/storage.mvccPutUsingIter
  | 	github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1548
  | github.com/cockroachdb/cockroach/pkg/storage.MVCCPut
  | 	github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1465
  | github.com/cockroachdb/cockroach/pkg/storage.MVCCPutProto
  | 	github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:847
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).setLastReplicaGCTimestamp
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica.go:1380
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaGCQueue).process
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_gc_queue.go:264
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:1012
  | github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout
  | 	github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:91
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:971
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2.1
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:882
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) secondary error attachment
  | write ×: disk quota exceeded
  | (1) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/pebble/vfs.(*syncingFile).Write
  |   | 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/syncing_file.go:67
  |   | github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock
  |   | 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:551
  |   | github.com/cockroachdb/pebble/record.(*LogWriter).flushPending
  |   | 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:520
  |   | github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop
  |   | 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:466
  |   | runtime/pprof.Do
  |   | 	GOROOT/src/runtime/pprof/runtime.go:40
  |   | github.com/cockroachdb/pebble/record.NewLogWriter.func2
  |   | 	github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:351
  |   | runtime.goexit
  |   | 	GOROOT/src/runtime/asm_amd64.s:1594
  | Wraps: (2) write ×
  | Wraps: (3) disk quota exceeded
  | Error types: (1) *withstack.withStack (2) *fs.PathError (3) syscall.Errno
Wraps: (3) log.Fatal: pebble: fatal commit error: write ×: disk quota exceeded
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: pebble: fatal commit error: write ×: disk quota exceeded
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
db.go:812: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/kv/kvserver/pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).processLoop.func2.1 at line 882
pkg/kv/kvserver/pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).processReplica at line 971
pkg/util/contextutil/context.go in pkg/util/contextutil.RunWithTimeout at line 91
pkg/kv/kvserver/pkg/kv/kvserver/queue.go in pkg/kv/kvserver.(*baseQueue).processReplica.func1 at line 1012
pkg/kv/kvserver/pkg/kv/kvserver/replica_gc_queue.go in pkg/kv/kvserver.(*replicaGCQueue).process at line 264
pkg/kv/kvserver/pkg/kv/kvserver/replica.go in pkg/kv/kvserver.(*Replica).setLastReplicaGCTimestamp at line 1380
pkg/storage/mvcc.go in pkg/storage.MVCCPutProto at line 847
pkg/storage/mvcc.go in pkg/storage.MVCCPut at line 1465
pkg/storage/mvcc.go in pkg/storage.mvccPutUsingIter at line 1548
pkg/storage/mvcc.go in pkg/storage.mvccPutInternal at line 1810
pkg/storage/mvcc.go in pkg/storage.(*putBuffer).putInlineMeta at line 1393
pkg/storage/pebble.go in pkg/storage.(*Pebble).PutUnversioned at line 1585
pkg/storage/pebble.go in pkg/storage.(*Pebble).put at line 1608
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).Set at line 600
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).Apply at line 747
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).applyInternal at line 812

Tags

Tag	Value
Command	server
Environment	v23.1.3
Go Version	go1.19.4
Platform	linux amd64
Distribution	CCL
Cockroach Release	v23.1.3
Cockroach SHA	`c663f84`
# of CPUs	12
# of Goroutines	485

Jira issue: CRDB-28952

The text was updated successfully, but these errors were encountered:

blathers-crl · 2023-06-21T09:59:08Z

Hello, I am Blathers. I am here to help you get the issue triaged.

It looks like you have not filled out the issue in the format of any of our templates. To best assist you, we advise you to use one of these templates.

I have CC'd a few people who may be able to assist you:

@tbg (commented on Merge {marshal,unmarshal}TableValue into marshalValue. #1393, author of Test failure in CI build 1242 #600, author of TestRangeLookupWithOpenTransaction timeout #747, author of Test failure in CI build 4362 #1594, commented on Do not block Store.Start while waiting for the first gossip. #971, author of sql: SELECT ("*") parse oddities #1810, commented on How to get Key Prev for Reverse Scan #1585, author of Test failure in CI build 2392 #1012, author of reverse raft log & tweak Entries #812)
@bdarnell (commented on Test failure in CI build 1242 #600, commented on Test failure in CI build 4362 #1594, author of Do not block Store.Start while waiting for the first gossip. #971, assigned to Reorganized all local keys into two sections/range: by ID & by Key. #264, commented on Change MarshalResponse and UnmarshalRequest to gracefully handle a proto type check error #470)
@BramGruneir (author of Status Server get Nodes. #882)
@spencerkimball (commented on Status Server get Nodes. #882, author of Need to differentiate between client and nodes in server/node RPC gateway #847, commented on Do not block Store.Start while waiting for the first gossip. #971, author of Reorganized all local keys into two sections/range: by ID & by Key. #264)
@petermattis (author of Remove proto.Value.Integer. #1380, assigned to Need to differentiate between client and nodes in server/node RPC gateway #847, author of Merge {marshal,unmarshal}TableValue into marshalValue. #1393, assigned to sql: SELECT ("*") parse oddities #1810, commented on How to get Key Prev for Reverse Scan #1585, commented on Split native code into storage/engine/rocksdb #1608)
@cockroachdb/cdc (found keywords: webhook)
@cockroachdb/kv (found keywords: kv,mvcc)
@cockroachdb/storage (found keywords: pebble)

If we have not gotten back to your issue within a few business days, you can try the following:

Join our community slack channel and ask on #cockroachdb.
Try find someone from here if you know they worked closely on the area and CC them.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

blathers-crl · 2023-06-21T09:59:12Z

cc @cockroachdb/cdc

amruss · 2023-06-28T17:30:16Z

Removing from our backlog, this seems like a storage issue and does not involve the cdc webhook sink

erikgrinaker · 2023-06-28T18:29:10Z

There isn't much to be done here. The node ran out of disk (it hit the quota allocated by the operator).

blathers-crl bot added A-cdc Change Data Capture O-community Originated from the community X-blathers-triaged blathers was able to find an owner T-cdc labels Jun 21, 2023

blathers-crl bot added the T-storage Storage Team label Jun 28, 2023

amruss removed A-cdc Change Data Capture T-cdc labels Jun 28, 2023

erikgrinaker closed this as completed Jun 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sentry: db.go:812: log.Fatal: pebble: fatal commit error: write ×: disk quota exceeded (1) attached stack trace -- stack trace: | github.com/cockroachdb/pebble.(*DB).applyInternal | github.com... #105265

Sentry: db.go:812: log.Fatal: pebble: fatal commit error: write ×: disk quota exceeded (1) attached stack trace -- stack trace: | github.com/cockroachdb/pebble.(*DB).applyInternal | github.com... #105265

cockroach-sentry commented Jun 21, 2023 •

edited by cockroach-jira-scripts

Loading

blathers-crl bot commented Jun 21, 2023

blathers-crl bot commented Jun 21, 2023

amruss commented Jun 28, 2023

erikgrinaker commented Jun 28, 2023

Sentry: db.go:812: log.Fatal: pebble: fatal commit error: write ×: disk quota exceeded (1) attached stack trace -- stack trace: | github.com/cockroachdb/pebble.(*DB).applyInternal | github.com... #105265

Sentry: db.go:812: log.Fatal: pebble: fatal commit error: write ×: disk quota exceeded (1) attached stack trace -- stack trace: | github.com/cockroachdb/pebble.(*DB).applyInternal | github.com... #105265

Comments

cockroach-sentry commented Jun 21, 2023 • edited by cockroach-jira-scripts Loading

Tags

blathers-crl bot commented Jun 21, 2023

blathers-crl bot commented Jun 21, 2023

amruss commented Jun 28, 2023

erikgrinaker commented Jun 28, 2023

cockroach-sentry commented Jun 21, 2023 •

edited by cockroach-jira-scripts

Loading