[FLINK-13090][hive] Test Hive connector with hive runner

[lirui-apache]

What is the purpose of the change

To use hive runner in our hive connector tests.

Brief change log

• Added a JUnit runner FlinkStandaloneHiveRunner which launches a standalone HMS and provides an embedded Hive shell. The standalone HMS is closer to real world deployment, and the Hive shell makes it easier for us to write test cases.
• Changed HiveTableSinkTest to use FlinkStandaloneHiveRunner, so that we can use the Hive shell to verify data instead of reading from files ourselves. This will give us more confidence that the written data is hive-compatible.

Verifying this change

Adapted existing test cases.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? NA

[lirui-apache]

cc @xuefuz @bowenli86 @zjuwangg
Please note although the PR comes with a shim for hive runner, I haven't verified it against Hive-1.2.1 since the build is currently broken. The shim is simply ported from our internal Blink.

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit c0939e0 (Wed Aug 07 08:14:55 UTC 2019)

Warnings:

• 1 pom.xml files were touched: Check for build and licensing issues.
• No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❗ 3. Needs [attention] from.
  • Needs attention by @zjuwangg
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[xuefuz]

Thanks for the contribution. PR looks good to me. One thing that I'm not sure of is about the license. Do we need to update the license file as we are introducing some external libraries (though for test only). Please verify.

@zjuwangg Since you're mostly familiar to HiveRunner, please also take a look. Thanks.

[lirui-apache]

@xuefuz I don't think we need to worry about licenses because all the newly introduced deps are in test scope and only intended to be used during testing.

[xuefuz]

Sounds good then. +1 for the PR to be merged.

[zentol]

CI report for commit c9a63b5: SUCCESS Travis

[bowenli86]

Thanks for the contribution. I only have time to briefly look at the PR, and most part LGTM. @zjuwangg can you help to review this in detail?

[flinkbot]

CI report for commit b3e5599: FAILURE Build

[bowenli86]

@flinkbot attention @zjuwangg

[bowenli86]

I spent time and took a closer look at this PR. LGTM

Can you rebase to master? @lirui-apache

[lirui-apache]

Rebased and addressed comments. @bowenli86 please take another look, thanks.

[flinkbot]

CI report for commit c0939e0: SUCCESS Build

[bowenli86]

@lirui-apache LGTM, thanks for your contribution.

Merging