FAQ / SEO-optimized page for OOM errors in CockroachDB

[otan]

Oliver Tan (otan) commented:

When one googles "CockroachDB OOM" and "CockroachDB out of memory", you get hit with a swathe of people posting issues with that title on the crdb forums or on our GitHub, and it's hard to navigate what one should actually do in this instance.

It would be nice if instead there a single page that contained tuning parameters for memory issues:

• --cache -- for RocksDB
• --max-sql-memory -- for maximum sql memory
• sql.distsql-temp_storage.workmem -- for disk spilling

and the tradeoffs which setting each of those higher or lower.

Ideally this would be the first result on Google too!

Related issues:

• Memory management and best practices #6087
• cli: guidance on the setting --max-sql-memory flag #5923
• wip: cli: guidance on the setting of --cache flag #5924

Jira Issue: DOC-498

[awoods187]

We need to prioritize this work and move it up. It is a frequent request.

[otan]

cc @jseldess do you mind finding someone who could take this on?

[ericharmeling]

I think there are several existing pages where we could add some guidance on tuning parameters for memory-related issues.

Some pages I have in mind:

• https://www.cockroachlabs.com/docs/stable/cluster-setup-troubleshooting.html#memory-issues
• https://www.cockroachlabs.com/docs/stable/recommended-production-settings.html#cache-and-sql-memory-size
• https://www.cockroachlabs.com/docs/stable/common-errors.html
• https://www.cockroachlabs.com/docs/stable/query-behavior-troubleshooting.html
• https://www.cockroachlabs.com/docs/stable/sql-tuning-with-explain.html

We could just add a section and some guidance + an example or something. Then it would be just a matter of counting on the SEO to direct searchers to our docs.

@rmloveland

This issue seems to be storage-related. Is there an existing page to which you would recommend adding some docs on tuning parameters for memory issues?

[otan]

it can be a sql execution layer too -- most of my problems have been there.

[rmloveland]

This issue seems to be storage-related. Is there an existing page to which you would recommend adding some docs on tuning parameters for memory issues?

I'm not sure if there's an existing page I can think of, beyond the things that have been suggested above (--cache in startup params, etc.). As Oliver says, it pops up in multiple layers, I don't think it is confined to one layer (based on seeing things people type in Slack, etc. - I have not been lucky enough to OOM one yet)

[awoods187]

This is SQL Execution, KV, and Storage related. However, it usually presents in SQL and the controls available mostly affect SQL. I'd work with @jordanlewis and @asubiotto to make a dedicated page on how to handle this. Its also a good place to talk about various improvements like vectorized execution the reduce OOMs.

[ianjevans]

@taroface would this issue qualify for some of the troubleshooting knowledge-base docs you worked on?

[taroface]

Those docs were more about edge cases. There's one related to schema changes that applies here:

• https://cockroachlabs.atlassian.net/wiki/spaces/TS/pages/1171784024/Memory+leak+from+rapid+schema+changes

However, that issue was fixed in v20.1.10.

[taroface]

I agree with @ericharmeling that https://www.cockroachlabs.com/docs/stable/recommended-production-settings.html#cache-and-sql-memory-size can be expanded according to the three issues linked in the description. cc @mikeCRL who is updating that doc.

(That covers cache and max-sql-memory; not sure about sql.distsql-temp_storage.workmem.)

[jseldess]

@mikeCRL, @taroface, @ericharmeling, what's the current state of this?