Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stats: multiple ANALYZE runs can fail with out of memory error #67113

Closed
rytaft opened this issue Jul 1, 2021 · 1 comment · Fixed by #69483
Closed

stats: multiple ANALYZE runs can fail with out of memory error #67113

rytaft opened this issue Jul 1, 2021 · 1 comment · Fixed by #69483
Assignees
@rytaft
Labels
A-sql-table-stats Table statistics (and their automatic refresh). C-investigation Further steps needed to qualify. C-label will change. O-postmortem Originated from a Postmortem action item. T-sql-queries SQL Queries Team
Projects
SQL Queries

Comments

@rytaft
Copy link
Collaborator

rytaft commented Jul 1, 2021

Multiple back-to-back manual ANALYZE runs can fail with an out of memory error (just a regular error, not an OOM crash). We should investigate whether we are not releasing all the memory accounted for by the stats processors.
@rytaft rytaft added C-investigation Further steps needed to qualify. C-label will change. A-sql-table-stats Table statistics (and their automatic refresh). O-postmortem Originated from a Postmortem action item. T-sql-queries SQL Queries Team labels Jul 1, 2021
@rytaft rytaft added this to Triage in SQL Queries via automation Jul 1, 2021
@rytaft
Copy link
Collaborator Author

rytaft commented Jul 1, 2021

Here's an example stack trace (v20.2.9):

  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/util/mon.memoryResource.NewBudgetExceededError
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/resource.go:37
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).increaseBudget
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:682
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).reserveBytes
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:615
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BoundAccount).Grow
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:570
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).increaseBudget
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:690
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).reserveBytes
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:615
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BoundAccount).Grow
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:570
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).increaseBudget
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:690
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).reserveBytes
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:615
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BoundAccount).Grow
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:570
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BoundAccount).Resize
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:550
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BoundAccount).ResizeTo
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:563
  | github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:345
  | github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:454
  | github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).NextKV
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_fetcher.go:118
  | github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).NextKey
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/fetcher.go:671
  | github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).StartScanFrom
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/fetcher.go:661
  | github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).StartScan
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/fetcher.go:553
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*tableReader).Start
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/tablereader.go:191
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*samplerProcessor).Run
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/sampler.go:221
  | github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).startInternal.func1
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:340

@rytaft rytaft self-assigned this Jul 13, 2021
@rytaft rytaft moved this from Triage to 21.2 High Likelihood (90%) in SQL Queries Jul 13, 2021
@yuzefovich yuzefovich mentioned this issue Aug 27, 2021
Merged
@craig craig bot closed this as completed in 45ffbd0 Aug 28, 2021
SQL Queries automation moved this from 21.2 High Likelihood (90%) to Done Aug 28, 2021
This was referenced Aug 30, 2021
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rytaft rytaft

Labels
A-sql-table-stats Table statistics (and their automatic refresh). C-investigation Further steps needed to qualify. C-label will change. O-postmortem Originated from a Postmortem action item. T-sql-queries SQL Queries Team
Projects
SQL Queries
Archived in project
SQL Queries
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

distsql: restore EvalCtx.Mon on the flow cleanup yuzefovich/cockroach
1 participant
@rytaft