[FLINK-6749] [table] Table API / SQL Docs: SQL Page #4046

haohui · 2017-06-01T22:47:54Z

This first iteration of the documentation. Comments are appreciated.

alpinegizmo

This is mostly quite clear. Just a few suggestions.

alpinegizmo · 2017-06-02T16:04:20Z

docs/dev/table/sql.md

 -->

-SQL queries are specified using the `sql()` method of the `TableEnvironment`. The method returns the result of the SQL query as a `Table` which can be converted into a `DataSet` or `DataStream`, used in subsequent Table API queries, or written to a `TableSink` (see [Writing Tables to External Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can seamlessly mixed and are holistically optimized and translated into a single DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries using the `sql()` method of the `TableEnvironment`. The method returns the result of the SQL query as a `Table`. A `Table` can be used in the subsequent SQL / Table API queries, be converted into a `DataSet` or `DataStream`, used in subsequent Table API queries or written to a `TableSink` (see [Writing Tables to External Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can seamlessly mixed and are holistically optimized and translated into a single program.


A Table can be used in subsequent SQL / Table API queries, be converted into a DataSet or DataStream, or written to a TableSink ...

(drop "the" and the redundant "used in subsequent Table API queries")

alpinegizmo · 2017-06-02T16:05:53Z

docs/dev/table/sql.md

+To access the data in the SQL queries, users must register data sources, including `Table`, `DataSet`, `DataStream` or external `TableSource`, in the `TableEnvironment` (see [Registering Tables](common.html#register-a-table-in-the-catalog)). Alternatively, users can also register external catalogs in the `TableEnvironment` to specify the location of the data sources.

-*Note: Flink's SQL support is not feature complete, yet. Queries that include unsupported SQL features will cause a `TableException`. The limitations of SQL on batch and streaming tables are listed in the following sections.*
+For convenience `Table.toString()` will automatically register an unique table name under the `Table`'s `TableEnvironment` and return the table name. So it allows to call SQL directly on tables in a string concatenation (see examples below).


... a unique table name ...

alpinegizmo · 2017-06-02T16:07:45Z

docs/dev/table/sql.md

+To access the data in the SQL queries, users must register data sources, including `Table`, `DataSet`, `DataStream` or external `TableSource`, in the `TableEnvironment` (see [Registering Tables](common.html#register-a-table-in-the-catalog)). Alternatively, users can also register external catalogs in the `TableEnvironment` to specify the location of the data sources.

-*Note: Flink's SQL support is not feature complete, yet. Queries that include unsupported SQL features will cause a `TableException`. The limitations of SQL on batch and streaming tables are listed in the following sections.*
+For convenience `Table.toString()` will automatically register an unique table name under the `Table`'s `TableEnvironment` and return the table name. So it allows to call SQL directly on tables in a string concatenation (see examples below).


This allows SQL to be called directly on tables in a string concatenation (see examples below).

alpinegizmo · 2017-06-02T16:08:19Z

docs/dev/table/sql.md

+For convenience `Table.toString()` will automatically register an unique table name under the `Table`'s `TableEnvironment` and return the table name. So it allows to call SQL directly on tables in a string concatenation (see examples below).

-**TODO: Rework intro. Move some parts below. **
+*Note: Flink's SQL support is not feature complete, yet. Queries that include unsupported SQL features will cause a `TableException`. The limitations of SQL on batch and streaming tables are listed in the following sections.*


Flink's SQL support is not yet feature complete.

alpinegizmo · 2017-06-02T16:08:44Z

docs/dev/table/sql.md

 Specifying a Query
 ---------------

+Here are a few examples on how to specify a DataStream / DataSet program using SQL:


examples of how to

alpinegizmo · 2017-06-02T16:10:21Z

docs/dev/table/sql.md

 ----------------

-Flink uses [Apache Calcite](https://calcite.apache.org/docs/reference.html) for SQL parsing. Currently, Flink SQL only supports query-related SQL syntax and only a subset of the comprehensive SQL standard. The following BNF-grammar describes the supported SQL features:
+Flink parses SQL using [Apache Calcite](https://calcite.apache.org/docs/reference.html). Flink supports the standard ANSI SQL but it provides no supports for DML and DDL. The following BNF-grammar describes the supported SQL features:


Flink supports standard ANSI SQL, but it provides no support for DML or DDL.

alpinegizmo · 2017-06-02T16:19:51Z

docs/dev/table/sql.md

+Flink supports specifying DataStream or DataSet programs with SQL queries using the `sql()` method of the `TableEnvironment`. The method returns the result of the SQL query as a `Table`. A `Table` can be used in the subsequent SQL / Table API queries, be converted into a `DataSet` or `DataStream`, used in subsequent Table API queries or written to a `TableSink` (see [Writing Tables to External Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can seamlessly mixed and are holistically optimized and translated into a single program.

-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be registered in the `TableEnvironment` in order to be accessible by a SQL query (see [Registering Tables](#registering-tables)). For convenience `Table.toString()` will automatically register an unique table name under the `Table`'s `TableEnvironment` and return the table name. So it allows to call SQL directly on tables in a string concatenation (see examples below).
+To access the data in the SQL queries, users must register data sources, including `Table`, `DataSet`, `DataStream` or external `TableSource`, in the `TableEnvironment` (see [Registering Tables](common.html#register-a-table-in-the-catalog)). Alternatively, users can also register external catalogs in the `TableEnvironment` to specify the location of the data sources.


This reads a bit awkwardly. How about this?

Before using data in a SQL query, the data source(s) must first be registered in the TableEnvironment (see Registering Tables). Possible data sources include Tables, DataSets, DataStreams, and external TableSources. Alternatively, ...

fhueske

Thanks for the PR @haohui! I left a few comments and suggestions.

It would also be good, if you could check if the Built-in Function limitations are still up to date. Some of the functions might have been added in the mean time.

What do you think?
Thanks, Fabian

fhueske · 2017-06-07T21:29:11Z

docs/dev/table/sql.md

 -->

-SQL queries are specified using the `sql()` method of the `TableEnvironment`. The method returns the result of the SQL query as a `Table` which can be converted into a `DataSet` or `DataStream`, used in subsequent Table API queries, or written to a `TableSink` (see [Writing Tables to External Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can seamlessly mixed and are holistically optimized and translated into a single DataStream or DataSet program.
+Flink supports specifying DataStream or DataSet programs with SQL queries using the `sql()` method of the `TableEnvironment`. The method returns the result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / Table API queries, be converted into a `DataSet` or `DataStream`, or written to a `TableSink` (see [Writing Tables to External Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can seamlessly mixed and are holistically optimized and translated into a single program.


Add links to the relevant sections of the Overview page:

http://localhost:4000/dev/table/common.html#mixing-table-api-and-sql

http://localhost:4000/dev/table/common.html#integration-with-datastream-and-dataset-api

http://localhost:4000/dev/table/common.html#emit-a-table

I would not mention the translation into DataStream and DataSet programs (see another comment below).

fhueske · 2017-06-07T21:30:05Z

docs/dev/table/sql.md

+Flink supports specifying DataStream or DataSet programs with SQL queries using the `sql()` method of the `TableEnvironment`. The method returns the result of the SQL query as a `Table`. A `Table` can be used in subsequent SQL / Table API queries, be converted into a `DataSet` or `DataStream`, or written to a `TableSink` (see [Writing Tables to External Sinks](common.html#emit-to-a-tablesink)). SQL and Table API queries can seamlessly mixed and are holistically optimized and translated into a single program.

-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be registered in the `TableEnvironment` in order to be accessible by a SQL query (see [Registering Tables](#registering-tables)). For convenience `Table.toString()` will automatically register an unique table name under the `Table`'s `TableEnvironment` and return the table name. So it allows to call SQL directly on tables in a string concatenation (see examples below).
+Before using data in a SQL query, the data source(s) must first be registered in the `TableEnvironment` (see see [Registering Tables](common.html#register-a-table-in-the-catalog)). Possible data sources include Tables, DataSets, DataStreams, and external TableSources. Alternatively, users can also register external catalogs in the `TableEnvironment` to specify the location of the data sources.


Add a link to http://localhost:4000/dev/table/common.html#register-an-external-catalog

fhueske · 2017-06-07T21:31:32Z

docs/dev/table/sql.md

+Before using data in a SQL query, the data source(s) must first be registered in the `TableEnvironment` (see see [Registering Tables](common.html#register-a-table-in-the-catalog)). Possible data sources include Tables, DataSets, DataStreams, and external TableSources. Alternatively, users can also register external catalogs in the `TableEnvironment` to specify the location of the data sources.

-*Note: Flink's SQL support is not feature complete, yet. Queries that include unsupported SQL features will cause a `TableException`. The limitations of SQL on batch and streaming tables are listed in the following sections.*
+For convenience `Table.toString()` will automatically register a unique table name under the `Table`'s `TableEnvironment` and return the table name. This allows SQL to be called directly on tables in a string concatenation (see examples below).


... will automatically register the table under a unique name in its TableEnvironment and ...?

fhueske · 2017-06-07T21:34:47Z

docs/dev/table/sql.md

 Specifying a Query
 ---------------

+Here are a few examples on specifying a DataStream / DataSet program using SQL:


I would not mention that SQL queries are translated into DataStream / DataSet programs. This is useful information for some users and it should be mentioned in an internals section. I would assume that most users do not care about this and might be even confused.

What do you think @haohui ?

fhueske · 2017-06-07T21:42:04Z

docs/dev/table/sql.md

 ----------------

-Flink uses [Apache Calcite](https://calcite.apache.org/docs/reference.html) for SQL parsing. Currently, Flink SQL only supports query-related SQL syntax and only a subset of the comprehensive SQL standard. The following BNF-grammar describes the supported SQL features:
+Flink parses SQL using [Apache Calcite](https://calcite.apache.org/docs/reference.html). Flink supports standard ANSI SQL, but it provides no supports for DML and DDL. The following BNF-grammar describes the supported SQL features:


We should mention that not all features are supported by batch and streaming. The Operations section will show which features are supported by batch and streaming.

fhueske · 2017-06-07T21:52:21Z

docs/dev/table/sql.md

+  </thead>
+  <tbody>
+  	<tr>
+      <td><strong>Inner Equi-join / Outer Equi-join</strong>(Batch only)</td>


Add a comment that join must have at least one conjunctive equality predicate. CROSS or Theta joins are not supported.

Add a comment that Flink does not optimize join order yet and joins in the same order as specified in the query.

fhueske · 2017-06-07T21:53:59Z

docs/dev/table/sql.md

+    <tr>
+    	<td><strong>User Defined Table Function (UDTF)</strong></td>
+    	<td>
+      <p>SQL queries can refer to UDTFs to expand a value into a relation provided that they are registered in the <pre>TableEnvironment</pre>.</p>


<pre>TableFunction</pre> is not nicely rendered in my setup. Can you check on yours?

TableFunctions do also accept multiple values. Add a link to the UDF docs page.

fhueske · 2017-06-07T21:57:11Z

docs/dev/table/sql.md

 {% top %}

-### GroupBy Windows
+### Aggregations


Add a line for User defined aggregation function?

fhueske · 2017-06-07T21:59:42Z

docs/dev/table/sql.md

 </table>

-For SQL queries on streaming tables, the `time_attr` argument of the group window function must be one of the `rowtime()` or `proctime()` time-indicators, which distinguish between event or processing time, respectively. For SQL on batch tables, the `time_attr` argument of the group window function must be an attribute of type `TIMESTAMP`. 
+For SQL queries on streaming tables, the `time_attr` argument of the group window function must refer to the virtual column that specifies the processing time or the event time. For SQL on batch tables, the `time_attr` argument of the group window function must be an attribute of type `TIMESTAMP`.


Add a link to the time attribute section on the Table API Streaming doc page.

fhueske · 2017-06-07T22:04:13Z

docs/dev/table/sql.md

-
-**TODO: Integrate this with the examples**
-
 #### Batch


I would remove the Batch and Streaming limitations sections.
Instead, I would integrate the relevant information with the Operations sections, e.g., add a comments to UNNEST that WITH ORDINALITY is not supported, etc.

fhueske · 2017-06-08T12:30:02Z

I've seen several questions on the mailing list and Stack Overflow related to breaking changes in the 1.3 release (mostly about the time indicator). That's why I would like to update the documentation ASAP.

I will integrate my comments and merge this PR.

fhueske · 2017-06-08T14:40:32Z

Hi @haohui, I merged the PR to the tableDocs branch.
Can you close it? Thanks!

[FLINK-6749] [table] Table API / SQL Docs: SQL Page.

15a5654

alpinegizmo reviewed Jun 2, 2017

View reviewed changes

Address comments.

db66592

fhueske reviewed Jun 7, 2017

View reviewed changes

haohui closed this Jun 8, 2017

rmetzger added component=API/TableSQL component=Documentation labels Mar 14, 2019

flinkbot added component=TableSQL/API and removed component=API/TableSQL labels Mar 17, 2022

[FLINK-6749] [table] Table API / SQL Docs: SQL Page #4046

[FLINK-6749] [table] Table API / SQL Docs: SQL Page #4046

Uh oh!

Conversation

haohui commented Jun 1, 2017

Uh oh!

alpinegizmo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fhueske left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fhueske commented Jun 8, 2017

Uh oh!

fhueske commented Jun 8, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants