Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[YSQL] Creating a table can cause a constraint violation on pg_class_oid_index. #1962

Closed
aphyr opened this issue Aug 2, 2019 · 3 comments
Closed

Comments

@aphyr
Copy link

@aphyr aphyr commented Aug 2, 2019

With version 1.3.1.0, I've hit occasional (~1:10 runs) issues creating YSQL tables during Jepsen tests. For instance, the append test creates a single table like so:

  (setup-cluster! [this test c conn-wrapper]
    (c/execute! c (j/create-table-ddl table
                                      [[:k :int]
                                       [:v :text]
                                       ["PRIMARY KEY" "(k)"]])))

... which is executed in a global lock, so only one client should be acting at a time. Sometimes, but not always, this can throw:

org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pg_class_oid_index"

which suggests to me that there might be some sort of double-allocation of OIDs possible during table creation--Postgres mailing lists suggest this might be due to OID overflow or filesystem corruption, but I doubt either of those is at play here. Since we create a fresh cluster every time, I wonder if there might be some sort of table initialization code that's running concurrently with our table creation, and the two can occasionally race? Not sure!

For full logs, see 20190802T132514.000-0400.zip.

To reproduce this, try Jepsen 23dbb5e78725318c35d07dcc6d727ebb3ab7a3e7, and run something like

lein run test --os debian --version 1.3.1.0 --time-limit 5 -w ysql/append --test-count 30

I've only hit this once, after a handful of runs, so I'm not sure how frequent a problem it is yet. I'll post back if I see it again!

@frozenspider frozenspider added this to To do in Jepsen Testing via automation Aug 2, 2019
@frozenspider frozenspider added this to To do in YSQL via automation Aug 2, 2019
@frozenspider frozenspider changed the title Creating a table can cause a constraint violation on pg_class_oid_index. [YSQL] Creating a table can cause a constraint violation on pg_class_oid_index. Aug 2, 2019
@aphyr
Copy link
Author

@aphyr aphyr commented Aug 2, 2019

Hit a second case, looks to be roughly 1:20 runs so far. 20190802T144108.000-0400.zip

Loading

@ndeodhar
Copy link
Contributor

@ndeodhar ndeodhar commented Aug 2, 2019

This happens because our SQL DDL statements are not transactional. I see this in the logs:

I0802 18:41:39.796936 48903 pg_ddl.cc:141] PgCreateTable: creating a transactional table: postgres.append
I0802 18:41:44.952004 48903 table_creator.cc:286] Created table postgres.append of type PGSQL_TABLE_TYPE
I0802 18:41:49.305517 48913 meta_cache.cc:588] Got resp error 7, code=OK                             
W0802 18:41:49.306098 48913 meta_cache.cc:602] Leader Master has changed, re-trying... 

So postgres system tables were updated but there was a master leader change and so create table never completed successfully. When client retries request on a new leader, we see the duplicate key error.

Related issue about transactional DDLs: #1383

Loading

@frozenspider
Copy link
Contributor

@frozenspider frozenspider commented Jun 5, 2020

This has been resolved and is no longer reproducible. This was confirmed by verifying the following:

  • These issues still reproduce when test is run against the original version v1.3.1.0
  • These issues no longer reproduce when test is run against v2.1.3 (as of commit 9f4448d)

Note that in order to be safe, the testing was done using 600 iterations of the Jepsen test with --test-count 30 and 10 more with --test-count 300 (using Jepsen commit 23dbb5e).

Loading

Jepsen Testing automation moved this from To do to Done Jun 5, 2020
YSQL automation moved this from To do to Done Jun 5, 2020
@m-iancu m-iancu moved this from To do to Done in DDL improvements Apr 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
YSQL
  
Done
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants