Skip to content

[FLINK-37536] [TABLE-SQL] Improve SQL Gateway documentation #26378

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 12, 2025

Conversation

ammu20-dev
Copy link
Contributor

What is the purpose of the change

This pull request if for improving the SQL gateway documentation

Brief change log

Updated the doc with details on submitting a job via gateway to a remote host.

Verifying this change

Please make sure both new and modified tests in this PR follow the conventions for tests defined in our code quality guide.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no) - no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no) - no
  • The serializers: (yes / no / don't know) - no
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know) - no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - no
  • The S3 file system connector: (yes / no / don't know) - no

Documentation

  • Does this pull request introduce a new feature? (yes / no) -no
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@flinkbot
Copy link
Collaborator

flinkbot commented Mar 31, 2025

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@@ -84,6 +84,14 @@ $ curl --request POST http://localhost:8083/v1/sessions/${sessionHandle}/stateme

The `operationHandle` in the return results is used by the SQL Gateway to uniquely identify the submitted SQL.

The Flink SQL Gateway allows clients to specify which Flink cluster to submit jobs to, enabling remote execution of SQL statements and facilitating easier interaction with Flink clusters through a REST API. Enrich the POST request body with `executionConfig` variable to set the Flink cluster address. For example:
Copy link
Contributor

@davidradl davidradl Mar 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the executionConfig this if so shouldn't this parameter be documented there? Can the other parts of the executionConfig be specified on the rest call? Or is this a different executionConfig? We should document what the defaults are for the ExecutionContext and its parameters?

Also we should mention in the text why/when you would want to specify a non-default jm endpoint for the gateway.

I think the parameters in the example should be parameterized i.e. {port} and then explain what port is. As is I support port will fail as it is not a number.
@ammu20-dev

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The request body part of Gateway REST /sessions/:session_handle/statements api contains an executionConfig variable which can be used to override the default configuration. I have mentioned the same here in the doc.

Also we should mention in the text why/when you would want to specify a non-default jm endpoint for the gateway.

The statement The Flink SQL Gateway allows clients to specify which Flink cluster to submit jobs to, enabling remote execution of SQL statements and facilitating easier interaction with Flink clusters through a REST API. mentions about the possibility of using the gateway to submit jobs to a remote cluster. I hope this would be sufficient to convey the why/when part.

I think the parameters in the example should be parameterized i.e. {port} and then explain what port is. As is I support port will fail as it is not a number.

Done through 70b67d6

You can also override the default job manager configuration in SQL client by explicitly providing the cluster address while initialising the client as below:

```bash
./bin/sql-client.sh gateway --endpoint <gateway address> -Drest.address=<flink cluster host> -Drest.port=<port>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not look very consistent.

  • endpoint . - I assume this will include the port so it should be
  • then the other endpoint for the JM is defined as an address and a port.

Is this an opportunity to make them more consistent?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SQL Client documentation here mentions The <gateway address> can be provided in two formats: as a host:port combination or as a full URL. So endpoint is the gateway address to which the SQL Client should connect to and rest.address and rest.port are the JM host and port respectively.

Copy link
Contributor

@davidradl davidradl Apr 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest ,without changing the code we change gateway address to gateway endpoint. As address implies IP address that does not include the port.

Copy link
Contributor Author

@ammu20-dev ammu20-dev Apr 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think gateway address is more suitable. As address indicates the location of the gateway, whether it's provided as a host:port combination or a full URL.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with what you say. IMHO in this context it is right next to rest.address which is a host. So the word address is used to mean 2 different things on the same line. I do not feel that strongly about this, so will approve anyway.

Copy link
Contributor

@davidradl davidradl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have left comments requesting changes. Also the Chinese docs should be updated with the english ones.

@ammu20-dev
Copy link
Contributor Author

I have left comments requesting changes. Also the Chinese docs should be updated with the english ones.

Thanks @davidradl for the review. Could you please help me to work on the Chinese translation.

@davidradl
Copy link
Contributor

I have left comments requesting changes. Also the Chinese docs should be updated with the english ones.

Thanks @davidradl for the review. Could you please help me to work on the Chinese translation.

Normally, if thee is a new section then you should add the text in English into the Chinese version.

@@ -84,6 +84,14 @@ $ curl --request POST http://localhost:8083/v1/sessions/${sessionHandle}/stateme

The `operationHandle` in the return results is used by the SQL Gateway to uniquely identify the submitted SQL.

The Flink SQL Gateway allows clients to specify which Flink cluster to submit jobs to, enabling remote execution of SQL statements and facilitating easier interaction with Flink clusters through a REST API. Enrich the POST request body with [rest.address](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/deployment/config/#rest-address) and [rest.port](https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/deployment/config/#rest-port) inside the `executionConfig` variable to set the Flink cluster address. For example:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as the change it to master - it should refer to the master docs , not the the v2.0 docs. They based on this url https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql-gateway/overview/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done through 9d84d54

@ammu20-dev
Copy link
Contributor Author

@davidradl Thanks. Added the doc changes in English in the translation folder through 9d84d54

Copy link
Member

@fsk119 fsk119 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Sorry for the late response. Merging...

@fsk119 fsk119 merged commit dda2acb into apache:master Jun 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants