HIVE-29035 : new version of cache by henrib · Pull Request #6441 · apache/hive

henrib · 2026-04-17T18:13:20Z

This checks the table location from HMS DB to ensure no stale table object is returned;

…e DB to ensure no stale table object is returned;

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

- improved check; - quiesce console logs due to internal throws in servlet;

henrib · 2026-04-28T16:35:15Z

@ayushtkn if you have some spare time to review, thanks

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 9 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ayushtkn · 2026-05-07T16:39:35Z

@henrib there are some warnings from Sonar & I feel few could be fixed
https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6441&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true

okumin · 2026-05-10T06:30:39Z

+    String tableName = baseTableIdentifier.name();
+    try {
+      List<Table> tables = clients.run(
+          client -> client.getTables(catName, database, Collections.singletonList(tableName), PARAM_SPEC)


TODO: It is possible to use IMetaStoreClient#getTable, but #getTables with PARAM_SPEC could be more lightweight. Which is better?

okumin · 2026-05-10T06:34:44Z

+      LOGGER.debug("Table {} not found: {}", baseTableIdentifier, e.getMessage());
+      throw e;
+    } catch (NoSuchObjectException e) {
+      throw new NoSuchTableException("Table %s not found: %s", baseTableIdentifier, e.getMessage());


This might also not happen since we obtain the list of tables. Also, this method can either return null or throw NoSuchTableException. We may use only either.

okumin · 2026-05-10T06:37:45Z

+      // A RESTException is thrown by HMSCatalogAdapter.execute() after the error handler has
+      // already written the correct HTTP status and body to the response (e.g. 404, 403).
+      // It is not an unexpected server failure, so log at DEBUG to avoid flooding the console.
+      LOG.debug("REST request resulted in a client error (already handled): {}", e.getMessage());


We may not need this one now

okumin · 2026-05-10T09:25:51Z

Please also address issues reported by SonarQube.
https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6441&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true

…cache performance metrics; - remove end point to access cache performance metrics; - enhanced tests to check L1 cache; - simplified MetadataLocator exception handling;

henrib · 2026-05-12T08:25:34Z

Please also address issues reported by SonarQube. https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6441&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true

Can't remove all of them in particular usage of clientPool().

okumin · 2026-05-12T12:18:12Z

+    // so the first post-reset load is a genuine cold miss rather than an L1/L2 hit.
+    // NOTE: catalog.invalidateTable() only clears the REST *client* state and does not
+    // reach the server-side cache.
+    serverCachingCatalog.invalidateTable(tableId);


I feel we should write this not as an integration test but as a unit test. If we do so, we can remove an escape hatch, i.e., getLatestCache

okumin · 2026-05-12T12:23:09Z

+      MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
+      String sanitized = catalogName == null || catalogName.isEmpty()
+        ? "default"
+        : catalogName.replaceAll("[^a-zA-Z0-9.\\\\-]", "_");


Should we use CATALOG_DEFAULT in MetastoreConf?

okumin · 2026-05-12T12:23:55Z

+      this.jmxObjectName = name;
+      LOG.info("Registered JMX MBean: {}", name);
+    } catch (JMException e) {
+      LOG.warn("Failed to register JMX MBean for HMSCachingCatalog", e);


This is probably eligible for error

Suggested change

LOG.warn("Failed to register JMX MBean for HMSCachingCatalog", e);

LOG.error("Failed to register JMX MBean for HMSCachingCatalog", e);

okumin · 2026-05-12T12:27:30Z

+  private final int l1CacheSize;
+
+  // Metrics counters.
+  private final AtomicLong cacheHitCount = new AtomicLong(0);


LongAdder could be an option

okumin · 2026-05-12T12:38:22Z

+    this(catalog, expirationMs, /*caseSensitive*/ true);
+  }
+
+  public HMSCachingCatalog(HiveCatalog catalog, long expirationMs, boolean caseSensitive) {


Can we use this constructor in the near future? If no, we may always use the default sensitivity.

okumin · 2026-05-12T12:39:30Z

+   *
+   * @param tid the table identifier
+   */
+  protected void onCacheMetaLoad(TableIdentifier tid) {


Suggested change

protected void onCacheMetaLoad(TableIdentifier tid) {

private void onCacheMetaLoad(TableIdentifier tid) {

okumin · 2026-05-12T12:39:45Z

+   *
+   * @param tid the table identifier
+   */
+  protected void onL1CacheHit(TableIdentifier tid) {


Suggested change

protected void onL1CacheHit(TableIdentifier tid) {

private void onL1CacheHit(TableIdentifier tid) {

okumin · 2026-05-12T12:39:56Z

+   *
+   * @param tid the table identifier
+   */
+  protected void onL1CacheMiss(TableIdentifier tid) {


Suggested change

protected void onL1CacheMiss(TableIdentifier tid) {

private void onL1CacheMiss(TableIdentifier tid) {

okumin · 2026-05-12T12:51:43Z

+      // If the table is no longer in L1 cache, we need to check the location.
+      final String location = metadataLocator.getLocation(canonicalized);
+      if (location == null) {
+        LOG.debug("Table {} has no location, returning cached table without location", canonicalized);


I think this case needs to throw NoSuchTableException because it does mean the table metadata is deleted, which means manifest lists are highly likely to be deleted or already invalid

okumin · 2026-05-12T12:56:50Z

+          LOG.debug("Table {} is in L1 cache, returning cached table", canonicalized);
+          onL1CacheHit(canonicalized);
+          onCacheHit(canonicalized);
+          return cachedTable;


Can we make L1 stuff out of scope from this PR? As CachingCatalog is designed for client-side caching, there is no additional authorization against cached objects. It means if Alice successfully loads abc table into the cache and then Bob fetches it, the security boundary can be breached. It never happens on the client-side since it is always used by Alice if it is initialized by Alice. In the case of server-side (HMS), it is not always true. I'm not 100% confident that the current implementation is safe enough.
If we could split a PR, I could merge the majority of this PR quickly and then closely review the security model for the advanced feature in the second PR.

okumin · 2026-05-12T13:15:30Z

+    final TableIdentifier baseTableIdentifier;
+    if (!catalog.isValidIdentifier(identifier)) {
+      if (!isValidMetadataIdentifier(identifier)) {
+        return null;


If we follow the semantics of loadTable, NoSuchTableException

okumin · 2026-05-12T13:16:27Z

+          clients.run(client -> client.getTables(catName, database, Collections.singletonList(tableName), PARAM_SPEC));
+      if (tables != null && !tables.isEmpty()) {
+        Table table = tables.getFirst();
+        if (table != null) {


I guess this is not a view

Suggested change

if (table != null) {

if (table != null) {

HiveOperationsBase.validateIcebergViewNotLoadedAsIcebergTable(table, fullName);

okumin · 2026-05-12T13:17:29Z

+          return table.getParameters().get(BaseMetastoreTableOperations.METADATA_LOCATION_PROP);
+        }
+      }
+      return null;


Probably, NoSuchTableException

okumin · 2026-05-12T13:18:31Z

+   *
+   * @param tid the table identifier to invalidate
+   */
+  protected void onCacheInvalidate(TableIdentifier tid) {


Or we may not need a method

sonarqubecloud · 2026-05-13T06:59:14Z

Quality Gate passed

Issues
33 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

HIVE-29035 : new version of cache that checks table location from Hiv…

abd759f

…e DB to ensure no stale table object is returned;

Copilot AI review requested due to automatic review settings April 17, 2026 18:13

asf-ci-hive added the tests pending label Apr 17, 2026

Copilot started reviewing on behalf of henrib April 17, 2026 18:18 View session

Copilot AI reviewed Apr 17, 2026

View reviewed changes

asf-ci-hive added tests unstable and removed tests pending labels Apr 17, 2026

HIVE-29035 : added the forgotten test;

19481e6

- improved check; - quiesce console logs due to internal throws in servlet;

asf-ci-hive added tests pending tests unstable and removed tests unstable tests pending labels Apr 17, 2026

Merge branch 'apache:master' into HIVE-29035

6713373

asf-ci-hive added tests pending tests passed and removed tests unstable tests pending labels Apr 18, 2026

Merge branch 'apache:master' into HIVE-29035

ae3fce6

asf-ci-hive added tests pending and removed tests passed labels Apr 28, 2026

HIVE-29035 : rebased and updated review nits;

7a9f12b

henrib requested a review from Copilot April 28, 2026 16:34

Copilot AI reviewed Apr 28, 2026

View reviewed changes

Copilot started reviewing on behalf of henrib April 28, 2026 16:59 View session

ayushtkn requested a review from okumin April 28, 2026 17:03

asf-ci-hive added tests passed tests pending and removed tests pending tests passed labels Apr 28, 2026

asf-ci-hive added the tests pending label May 6, 2026

HIVE-29035 : fix merge conflict;

cec47cf

asf-ci-hive added tests failed tests pending tests passed and removed tests pending tests failed labels May 6, 2026

okumin reviewed May 10, 2026

View reviewed changes

HIVE-29035: add JMX MXBean interface for HMSCachingCatalog to expose …

806e028

…cache performance metrics; - remove end point to access cache performance metrics; - enhanced tests to check L1 cache; - simplified MetadataLocator exception handling;

asf-ci-hive added tests pending and removed tests passed labels May 12, 2026

Merge branch 'apache:master' into HIVE-29035

9460452

asf-ci-hive added tests failed and removed tests pending labels May 12, 2026

okumin reviewed May 12, 2026

View reviewed changes

asf-ci-hive added tests pending and removed tests failed labels May 12, 2026

okumin reviewed May 12, 2026

View reviewed changes

asf-ci-hive added tests passed and removed tests pending labels May 12, 2026

Merge branch 'apache:master' into HIVE-29035

5b9add2

asf-ci-hive added tests pending and removed tests passed labels May 13, 2026

asf-ci-hive added tests unstable and removed tests pending labels May 13, 2026

	LOG.warn("Failed to register JMX MBean for HMSCachingCatalog", e);
	LOG.error("Failed to register JMX MBean for HMSCachingCatalog", e);

	protected void onCacheMetaLoad(TableIdentifier tid) {
	private void onCacheMetaLoad(TableIdentifier tid) {

	protected void onL1CacheHit(TableIdentifier tid) {
	private void onL1CacheHit(TableIdentifier tid) {

	protected void onL1CacheMiss(TableIdentifier tid) {
	private void onL1CacheMiss(TableIdentifier tid) {

	if (table != null) {
	if (table != null) {
	HiveOperationsBase.validateIcebergViewNotLoadedAsIcebergTable(table, fullName);

Conversation

henrib commented Apr 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

henrib commented Apr 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ayushtkn commented May 7, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

okumin commented May 10, 2026

Uh oh!

henrib commented May 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

okumin May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented May 13, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

okumin May 12, 2026 •

edited

Loading