[FLINK-25173][table][hive] Introduce CatalogLock and implement HiveCatalogLock

[JingsongLi]

What is the purpose of the change

Introduce CatalogLock for connectors, source & sink can use this lock to do some global sync.

/**
 * An interface that allows source and sink to use global lock to some transaction-related things.
 */
@Internal
public interface CatalogLock extends Closeable {
 
    /** Run with catalog lock. The caller should tell catalog the database and table name. */
    <T> T runWithLock(String database, String table, Callable<T> callable) throws Exception;
 
    /** Factory to create {@link CatalogLock}. */
    interface Factory extends Serializable {
        CatalogLock create();
    }
} 

And we need a interface to set lock to source&sink by catalog:

/**
 * Source and sink implement this interface if they require {@link CatalogLock}.
 */
@Internal
public interface RequireCatalogLock {
 
    void setLockFactory(CatalogLock.Factory lockFactory);
} 

Brief change log

• Introduce CatalogLock and RequireCatalogLock
• Implement HiveCatalogLock

Verifying this change

HiveRunnerITCase.testCatalogLock

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (yes / no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
• The serializers: (yes / no / don't know)
• The runtime per-record code paths (performance sensitive): (yes / no / don't know)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
• The S3 file system connector: (yes / no / don't know)

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[flinkbot]

CI report:

• ad59747 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit d00c16d (Wed Dec 15 09:14:35 UTC 2021)

Warnings:

• No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[JingsongLi]

@flinkbot run azure

[RocMarshal]

@JingsongLi Thanks for the efforts. Shall we add a description section into the relevant documentation page based on HiveConfOptions.java?

[luoyuxia]

Should we change the maximum to The maximum to make it consistent with the following options?

[luoyuxia]

Otherwise, lgtm.

[JingsongLi]

@JingsongLi Thanks for the efforts. Shall we add a description section into the relevant documentation page based on HiveConfOptions.java?

I am currently not ready to add, these options need a user scenario to expose, directly exposed in the documentation users can not use.

[JingsongLi]

@flinkbot run azure

[shouweikun]

I have a question about this. Shall we also add String catalogName as a parameter too? Maybe only databaseName and TableName can not avoid clash? eg: catalog_a.db.tb and catalog_b.db.tb

[JingsongLi]

No, This lock is provided by catalog, and catalog itself already knows its name.

[shouweikun]

For now we have to check and call RequireCatalogLock#setLockFactory manually in every DynamicTableFactory if necessary. Shall we lift this to Planner?

[JingsongLi]

Now this is an internal interface. If we need lock to be more general, we can put lock factory into {@link DynamicTableFactory.Context}.

[shouweikun]

Hi @JingsongLi , still I have some questions. Would you mind explaining it a little bit more? thx~