Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-8861] [table] Add support for batch queries in SQL Client #5660

Closed
wants to merge 1 commit into from

Conversation

xccui
Copy link
Member

@xccui xccui commented Mar 7, 2018

What is the purpose of the change

This PR added support for batch queries in SQL Client.

Brief change log

  • Added a BatchResult, which also implemented DynamicResult, for the batch queries.
  • Added a program deployer for jobs since the results in batch mode are acquired by JobExecutionResult.
  • Added the logic for retrieving batch query results referring to Dataset.collect().
  • Adapted the viewing logic for batch results to a "two-phase" (snapshotted or not) table result view.
  • Added a retrieveTableResult() method in LocalExecutorITCase.
  • Replaced some default values with "" in Execution.java.

Verifying this change

This change can be verified by the added test case testBatchQueryExecution().

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (yes)
  • If yes, how is the feature documented? (JavaDocs)

@xccui
Copy link
Member Author

xccui commented Mar 14, 2018

Hi @twalthr and @fhueske, the PR has been rebased. I wonder if you could help review it when you are convenient. Thanks~

@twalthr
Copy link
Contributor

twalthr commented Mar 15, 2018

Hi @xccui, I will review this as soon as possible. Right now the priority is the Flink 1.5 release with FLIP-6 support.

@xccui
Copy link
Member Author

xccui commented Mar 15, 2018

I see, @twalthr. Sorry for my impatience.

Copy link
Contributor

@twalthr twalthr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @xccui. I had a quick look at the changes and added some comments. I will do a proper review once the PR is rebased on the FLIP-6 changes. Could you update it?

@@ -159,6 +160,9 @@ protected void evaluate(ResultTableOperation operation, String binding) {
case PREV:
gotoPreviousPage();
break;
case FIRST:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't need a FIRST. GOTO allows for the same functionality. LAST is a special feature to stay at the last page in streaming mode.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the function's point of view, that's true. I added it just for symmetry 😃.

clusterClient.shutdown();
} catch (Exception e) {
// ignore
List<Row> resultTable = SerializedListAccumulator.deserializeList(accResult, batchResult.getSerializer());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move the deserialization logic into the result class? The executor should not need to handle internals of a result.

if (result == null) {
throw new SqlExecutionException("Could not find a result with result identifier '" + resultId + "'.");
}
if (!resultStore.isStatic(resultId)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need the distinction between dynamic and state result here? The executor should actually not matter. It should just kill whatever Flink job is running.

@xccui
Copy link
Member Author

xccui commented May 17, 2018

Sure @twalthr, I'll rebase the PR soon.

@xccui xccui force-pushed the FLINK-8861 branch 2 times, most recently from 2fa49a3 to 03fc9ec Compare June 10, 2018 09:18
@xccui
Copy link
Member Author

xccui commented Jun 10, 2018

Hi @twalthr, the PR has been reworked. Take a look when you are convenient. Thanks!

twalthr pushed a commit to twalthr/flink that referenced this pull request Jun 11, 2018
@twalthr
Copy link
Contributor

twalthr commented Jun 11, 2018

Thank you @xccui. I will test your code and merge this...

@asfgit asfgit closed this in e8e74a6 Jun 11, 2018
@xccui
Copy link
Member Author

xccui commented Jun 11, 2018

Thanks for the improvements, @twalthr.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants