[YSQL] Creating and deleting tables can result in unusable tables

[aphyr]

Jira Link: [DB-406](https://yugabyte.atlassian.net/browse/DB-406)
In 1.3.1.0, performing table creates and deletes concurrently (or perhaps just in rapid succession) can render a table completely unusable; perhaps indefinitely. Inserts fail because of constraint violations (even though every column has a default set!), reads fail because the table doesn't exist, create if not exists apparently succeeds, but the table can't be deleted, because it's already being deleted.

20190905T143634.000-0400.zip

Deletes:

2019-09-05 14:42:44,392{GMT}    INFO    [jepsen worker 3] jepsen.util: 11048    :info    :drop-table    nil    [:psql-exception "ERROR: Not found: The object was deleted: Started deleting at 2019-09-05 18:38:29 UTC"]

Reads:

INFO [2019-09-05 14:40:02,017] jepsen worker 0 - jepsen.util 3475    :info    :read    nil    [:psql-exception "ERROR: Not found: Error loading table with oid 16386 in database with oid 12453: The table '000030a5000030008000000000000000.foo' does not exist"]

Inserts:

INFO [2019-09-05 14:41:22,649] jepsen worker 4 - jepsen.util 7579    :info    :insert    nil    [:psql-exception "ERROR: null value in column \"k\" violates not-null constraint\n  Detail: Failing row contains (null, null)."]

You can reproduce this with Jepsen 60d3417: see the default-values test.

lein run test --os debian --version 1.3.1.0 -w ysql/default-value --time-limit 300

[ndeodhar]

Related to #1383

[Miaourt]

Hello !
What's the current state of this issue ?

[IS-Josh]

Hello, i've just encountered this issue with the latest version, is this on the cards to resolve soon?

[aphyr]

@IS-Josh, just out of curiosity, what kind of circumstances led you to hit this problem? This one seemed like a bug that people might be less likely to encounter in production, and I always love to hear Weird Database Stories. :)

[IS-Josh]

@aphyr Yes I believe it's a symptom of development. I left out a column and since there wasn't any data in the table and I preferred the column to be in a particular position, I dropped the table instead of altering it with a new column. I created the new table too soon after deleting it and ended up with this issue. I did find an ugly hack that allowed me to continue to use the table name i wanted. I was able to rename the broken table and re-use the table_name, however now i'm stuck with a dead table i cant remove.

[frozenspider]

Confirmed that this still happens in 9f4448d (included in v2.1.3), although I only managed to catch a read error, not insert/delete

[qvad]

Can't reproduce this issue in currently active releases 2.6 and latest master branch after 200+ iterations.