[4.x] Schema queries with configurable server-side timeouts #312

Bouncheck · 2024-07-01T08:40:25Z

Adds USING TIMEOUT clause to schema queries when applicable.
Uses the same timeout as driver-side timeouts defined by METADATA_SCHEMA_REQUEST_TIMEOUT ("advanced.metadata.schema.request-timeout").
Fixes #298

core/src/main/java/com/datastax/oss/driver/api/core/config/DefaultDriverOption.java

core/src/main/java/com/datastax/oss/driver/api/core/metadata/Node.java

core/src/main/java/com/datastax/oss/driver/internal/core/metadata/DefaultTopologyMonitor.java

mykaul · 2024-07-01T09:04:48Z

I don't fully understand the whole Scylla version issue here. 5.0/2022.1 are ancient version. If someone is using such an old version, what happens when we try to fetch the schema with timeout?

Bouncheck · 2024-07-01T09:11:02Z

I don't fully understand the whole Scylla version issue here. 5.0/2022.1 are ancient version. If someone is using such an old version, what happens when we try to fetch the schema with timeout?

I couldn't figure out a clean way to get Scylla version without querying system.versions which was introduced in those versions. In this change if we cannot determine Scylla version we fall back to relying on reported Cassandra version. If the Scylla version is too old (or undetermined) for using timeout we just don't use it.

I probably should've marked this as draft for now since I'm still cleaning this up. I may switch to either checking for shard awareness (as a proxy check for "is it scylla cluster") or running trial query with using timeout instead of checking exact Scylla version

mykaul · 2024-07-01T09:57:48Z

I don't fully understand the whole Scylla version issue here. 5.0/2022.1 are ancient version. If someone is using such an old version, what happens when we try to fetch the schema with timeout?

I couldn't figure out a clean way to get Scylla version without querying system.versions which was introduced in those versions. In this change if we cannot determine Scylla version we fall back to relying on reported Cassandra version. If the Scylla version is too old (or undetermined) for using timeout we just don't use it.

What do you mean too old? What happen when you do try to use timeout, and the scylla version is too old? If the timeout is ignored - fine, go with it. Doesn't matter. If it crashes, I'm more worried. If it errs the CQL query - then I think you should fallback to not using timeout.

I probably should've marked this as draft for now since I'm still cleaning this up. I may switch to either checking for shard awareness (as a proxy check for "is it scylla cluster") or running trial query with using timeout instead of checking exact Scylla version

Why? What would happen when you use it against a Cassandra cluster? Same as the above, IMHO (just as we try initially with high DSE versions that are not supported against Scylla!)

Bouncheck · 2024-07-01T10:10:42Z

Cassandra does not have USING TIMEOUT. Select queries do not allow USING altogether. Both will result in SyntaxException. I have to check how older versions of Scylla react, I've just picked the oldest supporting according to commits (4.4 was first with using timeout cql extension, 5.0 was for system.versions table containing scylla version).

mykaul · 2024-07-01T10:14:15Z

Cassandra does not have USING TIMEOUT. Select queries do not allow USING altogether. Both will result in SyntaxException. I have to check how older versions of Scylla react, I've just picked the oldest supporting according to commits (4.4 was first with using timeout cql extension, 5.0 was for system.versions table containing scylla version).

I'd catch the exception and re-run the query without 'USING TIMEOUT' . Probably a more optimistic approach (happy path) then the whole additional query we have. Of course, we may use the Scylla version or whatnot elsewhere, later, etc, so it might have value by itself.

Bouncheck · 2024-07-01T14:29:36Z

I've switched to running small trial select with USING TIMEOUT clause. If it completes without exceptions then the following schema queries will use this clause with the same configurable timeout as it is for driver side timeout. I've also removed unnecessary new additions.

Commit with querying for Scylla version remains as it was. While it is no longer necessary for this change I also believe it may be useful later and its impact should be minimal.

Waiting to see CI results.

core/src/main/java/com/datastax/oss/driver/api/core/metadata/Node.java

core/src/main/java/com/datastax/oss/driver/internal/core/metadata/DefaultTopologyMonitor.java

core/src/main/java/com/datastax/oss/driver/internal/core/metadata/NodeInfo.java

core/src/main/java/com/datastax/oss/driver/api/core/metadata/Node.java

...src/test/java/com/datastax/oss/driver/internal/core/metadata/DefaultTopologyMonitorTest.java

Bouncheck · 2024-07-01T16:00:29Z

I've removed commit with getScyllaVersion() change for now. There are more test adjustments needed for it, besides missed license headers, etc. and it's blocking the main purpose of this PR. I'll save that one for another time.

Bouncheck · 2024-07-01T16:15:23Z

I don't understand what's happening with gh actions. Unit tests fail due to cancellation but I have no idea what is triggering it.

Bouncheck · 2024-07-01T16:16:57Z

Oh, it's just timeout. I guess it became too small at some point.
edit: or not, since everything seems to hang on a test related to the change

...va/com/datastax/oss/driver/internal/core/metadata/schema/queries/CassandraSchemaQueries.java

Bouncheck · 2024-07-02T09:04:47Z

Unit tests should pass now

Bouncheck · 2024-07-02T14:47:58Z

I couldn't understand why my test query was causing integration tests to take too long, so instead I look at sharding info of the control connection channel. If it's non-null I am assuming I'm dealing with Scylla and will use the timeouts. I don't check the Scylla version.
I see that some integration tests failed already but this time it's the flakiness of other stuff.

mykaul · 2024-07-02T15:08:50Z

This looks much better - please resolve all comments.

Lorak-mmk

Commit message says USING TIMESTAMP instead of USING TIMEOUT
Apart from that looks good to me

The driver will now use existing configurable option `METADATA_SCHEMA_REQUEST_TIMEOUT` in a `USING TIMEOUT` clause with each schema query (if applicable). This will make it a server side timeout in addition to driver side. The driver will apply the `USING` clause if the sharding info of control connection channel is not null. This is used as a proxy check to differentiate between Scylla and Cassandra. The behaviour should remain unchanged for Cassandra clusters. USING TIMEOUT is a ScyllaDB CQL extension and it is available in versions 4.4/2022.1.0 or newer.

Bouncheck · 2024-07-03T16:21:33Z

I've fixed the typo @Lorak-mmk

Lorak-mmk

LGTM. One more thing: shouldn't we do the same with topology queries?

mykaul · 2024-07-04T06:58:34Z

LGTM. One more thing: shouldn't we do the same with topology queries?

Is topology response large? I doubt it, but we can certainly test it with a (mockup) large setup.

Lorak-mmk · 2024-07-04T08:36:00Z

LGTM. One more thing: shouldn't we do the same with topology queries?

Is topology response large? I doubt it, but we can certainly test it with a (mockup) large setup.

There are as many rows as nodes in the cluster. What is the biggest cluster that we know of?