[hive] Fix hive catalog lock may encounter deadlock.

[zhuangchong]

Purpose

Linked issue: close #6782

There are many possible causes for “hive lock may encounter deadlock”, for example:
1.The first table task is still running, and subsequent tasks cannot acquire the Hive table lock, leading to a timeout.
2.Delays in acquiring the Hive metastore lock, which also cause timeouts.
And so on.

In my latest changes:

1.Added detailed logs to show whether a lock acquisition failure was caused by a timeout or by another lock state.

paimon/paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalogLock.java

Lines 116 to 120 in 3342ef9

	String msg =
	String.format(
	"for table %s.%s (lockId=%d) after %dms. Final lock state: %s",
	database, table, lockId, duration, lockState);
	LOG.info("Acquire lock {}", msg);

2.Fixed an issue where lockResponse = clients.run(client -> client.checkLock(lockId)); would throw an exception and the lock would not be released, preventing subsequent tasks from acquiring the lock.

paimon/paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalogLock.java

Lines 95 to 112 in 3342ef9

	try {
	while (lockResponse.getState() == LockState.WAITING) {
	long elapsed = System.currentTimeMillis() - startMs;
	if (elapsed >= acquireTimeout) {
	break;
	}
	nextSleep = Math.min(nextSleep * 2, checkMaxSleep);
	Thread.sleep(nextSleep);
	lockResponse = clients.run(client -> client.checkLock(lockId));
	}
	} finally {
	if (lockResponse.getState() != LockState.ACQUIRED) {
	// unlock if not acquired
	unlock(lockId);
	}
	}

Tests

API and Format

Documentation

[JingsongLi]

Just for runWithLock method:
I cannot get it why modify here. What is difference?

[zhuangchong]

There are many possible causes for “hive lock may encounter deadlock”, for example:

The first table task is still running, and subsequent tasks cannot acquire the Hive table lock, leading to a timeout.

Delays in acquiring the Hive metastore lock, which also cause timeouts.
And so on.

In my latest changes:

Added detailed logs to show whether a lock acquisition failure was caused by a timeout or by another lock state.

Fixed an issue where lockResponse = clients.run(client -> client.checkLock(lockId)); would throw an exception and the lock would not be released, preventing subsequent tasks from acquiring the lock.

[JingsongLi]

Can you update the Purpose in the description? I cannot get information in try lock method.

[zhuangchong]

Can you update the Purpose in the description? I cannot get information in try lock method.

done.

[JingsongLi]

Thanks @zhuangchong +1