[YSQL] DDL changes may not be visible on other connections for up to one heartbeat period

[srhickma]

Jira Link: DB-2093
Summary: If SELECT * statements are made on one node while an ALTER TABLE is being performed on another (and not before), then until the next tserver heartbeat, the selects will continue to be processed using the old schema without error.

Probable Cause: When the first select is performed, the YB client (from pggate) must go to the master to get the metadata for the table. Since the alter has finished internally (in DocDB), the YB client gets the new schema version from the master, which now matches the version known by docdb. When following selects are performed (until the next heartbeat updates the ysql_catalog_version_) both the schema version (from YB client cache and DocDB) and ysql catalog versions (from YSQL cache and DocDB) are matching (and so no error is generated) despite the fact that the ysql catalog version is actually behind the schema version.

Test Case:

try (Statement statement1 = createConnection(0).createStatement()) {
  statement1.execute("CREATE TABLE test_table(id int, PRIMARY KEY (id))");

  final Lock lock = new ReentrantLock();
  final Condition connectedCondition  = lock.newCondition();
  final AtomicBoolean connected = new AtomicBoolean(false);

  new Thread(() -> {
    try (Statement statement2 = createConnection(1).createStatement()) {
      // Ready the alter statement.
      lock.lock();
      connected.set(true);
      connectedCondition.signal();
      lock.unlock();

      // Allow the alter to start.
      Thread.sleep(10);

      for (int i = 0; i < 250; i ++) {
          statement2.execute("SELECT * FROM test_table");
          ResultSet resultSet = statement2.getResultSet();
          // Print result set here (omitted)
      }
    } catch (Exception ignored) { }
  }).start();

  // Wait for signal to start alter
  lock.lock();
  while (!connected.get()) {
    connectedCondition.await();
  }
  lock.unlock();

  System.out.println("<- ALTER START");

  statement1.execute("ALTER TABLE test_table ADD COLUMN b int");

  System.out.println("<- ALTER DONE");

  statement1.execute("INSERT INTO test_table(id, b) VALUES (1, 2)");

  System.out.println("<- INSERT DONE");
}

Example Output:

<- ALTER START
selected 0 rows
 ...
<- ALTER DONE
selected 0 rows
 ...
<- INSERT DONE
selected 1 row: (1)
 ...
selected 1 row: (1)
selected 1 row: (1, 2)
 ...

Expected Output: It is expected that after the insert completes, and select should either return both columns, or a "Catalog Version Mismatch" error, but instead there is a period (until the next t-server heartbeat) where the table can be read using the out-of-date catalog.

Note: This issue is most prevalent when the second statement is created from the thread in which it is used. If the second statement is created from the main thread, the issue can still be seen, however it is less frequent (likely due to some locking done internally by JDBC).