Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
[YSQL] Creating a table with "IF NOT EXISTS" leads to tables being created over and over again until OOM #2001
I had a cluster of 1.3.1 nodes go very sideways in a Jepsen test last night. During table creation, (which uses
... over and over. We've seen this error before, during testing, but it's usually transient. This time it went on for hours before the box OOMed, and presumably killed a bunch of processes, but left one tserver (on node n3) spinning at 93-97% CPU use, on a 48-way (with HT) box.
Some debugging data:
Unfortunately I wasn't able to get a corefile--attaching GDB to the process made it crash immediately.
This happened with Jepsen af7285b96952258f3e3cb22cb18796ddbf37c56f, running
@aphyr I'm assuming you don't have the logs for this (or they were too massive). Dug a bit into the sampled data you were able to collect and found this:
So my assumption of what happened was that we generated 19961 tablets worth of tables until we could not do any more... cc @m-iancu @ndeodhar I'm guessing this is still txn DDL related, despite