PHOENIX-6883 : Phoenix Metadata Caching Redesign

[palashc]

JIRA - https://issues.apache.org/jira/browse/PHOENIX-6883

1. Introduce a region server endpoint for phoenix and server side metadata caching on region servers.
2. Server side cache kept consistent by invalidating cache entries synchronously during a DDL operation.
3. Client sets UPDATE_CACHE_FREQUENCY=never and uses LAST_DDL_TIMESTAMP to validate metadata before query/upsert.
4. Client refreshes its cache using the getTable RPC only when it receives a stale cache exception from the server.

[palashc]

This came up in discussion today and the recommendation is to create a separate thread pool for executing cache invalidations. I can create a subtask for the same and add the changes to this PR.
@shahrs87 @tkhurana @kadirozde

[kadirozde]

Assuming that these new methods are only called at the server side, I think we should move this code to a separate or existing server side class.

[palashc]

Since the cache invalidation code is in CQSI and we would want to avoid creating a new thread pool for every DDL operation, how do you suggest we save the executor? We can have an executor in CQSI but we would not want client side CQSI to create an executor unnecessarily - or is that okay?

[palashc]

@kadirozde This code did start out in a separate class but I remember @shahrs87 and @jpisaac had some reasons to move it to CQSI. I could not find which PR that was. @shahrs87 Can you recall those details?

[palashc]

This was the PR - #1748
@kadirozde Please see comments here #1748 (comment)

[tkhurana]

What is the difference between getCurrentTime() and getTimeStamp() ?

[palashc]

@tkhurana Currently both of those methods return the same value i.e. upperBoundTimestamp which is either -1 or whatever is provided to the TableRef constructor. Current code:

// if UPDATE_CACHE_FREQUENCY is set, always let the server set timestamps
this.upperBoundTimeStamp = table.getUpdateCacheFrequency()!=0 ? QueryConstants.UNSET_TIMESTAMP : upperBoundTimeStamp;
this.currentTime = this.upperBoundTimeStamp;

public long getTimeStamp() {
        return this.upperBoundTimeStamp;
}
public long getCurrentTime() {
        return this.currentTime;
 }

Once we set UPDATE_CACHE_FREQ to NEVER, currentTime was being set to -1 and features using currentTime were breaking (like QueryOptimizer deciding whether a disabled index is under its usability threshold or asyncCreatedDate during CreateIndex). The following change in TableRef keeps currentTime to whatever is provided to the TableRef constructor so that features using currentTime can continue to use it. There is effectively no change in UpsertCompiler - it will still be using the same value i.e. TableRef.upperBoundTimeStamp.

this.currentTime = upperBoundTimeStamp;

[tkhurana]

Where is this being used ?

[palashc]

These are the client-side/server-side flags we would use to enable the feature.

DEFAULT_LAST_DDL_TIMESTAMP_VALIDATION_ENABLED used in helper methods in ValidateLastDDLTimestampUtil which client uses to decide whether to validate timestamps and DEFAULT_PHOENIX_METADATA_INVALIDATE_CACHE_ENABLED in CQSI.invalidateServerMetadataCache to decide whether to invalidate cache on server side.

[tkhurana]

Nice comment

[tkhurana]

When validating timestamps we validate the timestamps of all the ancestors and the current table as well as indexes. When the server returns StaleMetadataCacheException are we also re-populating the client cache for all the objects that we are validating ? Are we also re-populating the indexes of the current table ?

[palashc]

Yes. Currently, we are re-populating everything since we use MetaDataClient.updateCache workflow - it resolves ancestors and goes through the process of adding inherited columns, indexes from them, it also re-populates the indexes into the client cache.

[tkhurana]

Why didn't we consider using the ServerSideRpcControllerFactory instead of creating a new factory ?

[palashc]

I am not sure about this one