[SPARK-28801][DOC] Document SELECT statement in SQL Reference (Main page) #27216

dilipbiswal · 2020-01-15T08:25:20Z

What changes were proposed in this pull request?

Document SELECT statement in SQL Reference Guide. In this PR includes the main
entry page for SELECT. I will open follow-up PRs for different clauses.

Why are the changes needed?

Currently Spark lacks documentation on the supported SQL constructs causing
confusion among users who sometimes have to look at the code to understand the
usage. This is aimed at addressing this issue.

Does this PR introduce any user-facing change?

Yes.

Before:
There was no documentation for this.

After.

How was this patch tested?

Tested using jykyll build --serve

SparkQA · 2020-01-15T08:40:13Z

Test build #116770 has finished for PR 27216 at commit de627aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2020-01-16T14:43:57Z

docs/sql-ref-syntax-qry-select-window.md

+  limitations under the License.
+---
+
+**This page is under construction**


Is there value in adding these placeholders, vs just adding/linking them when available?

@srowen Thanks. Actually, thats the approach we followed when we started this work a few months back. I guess we didn't want to have a link broken at any point. Also, makes it easier for others to contribute without having to rebase ?

Yeah I know, I'm not sure that was great as not all are filled out. Unless it would be hard to link them everywhere later, I wonder why they need to be here now? I don't see a difference w.r.t merging. I don't feel strongly but think we are just going to end up with more dummy pages that nobody fills out

@srowen OK. Let me remove the links and push.

dilipbiswal · 2020-01-16T17:41:01Z

docs/sql-ref-syntax-qry-select.md

+
+### Related clauses
+- [FROM clause](sql-ref-syntax-qry-select-from.html)
+- [WHERE clause](sql-ref-syntax-qry-select-where.html)


@srowen Are you okay to keep these or we want to remove the related clauses section add add as we go ? Please let me know.

maropu · 2020-01-17T05:29:37Z

docs/sql-ref-syntax-qry-select.md

+  <dd>
+    Hints can be specified to help spark optimizer make better planning decisions. Currently spark supports hints
+    that influence selection of join strategies and repartitioning of the data. For a detailed explanation, please
+    refer to. 


refer to where?

@maropu Thanks a lot for reviewing and catching this. I am going to remove that sentence for now. I wanted to refer to the Hints page from here. I will add it when i have it ready.

maropu · 2020-01-17T05:49:41Z

docs/sql-ref-syntax-qry-select.md

+  </dd>
+  <dt><code><em>ORDER BY</em></code></dt>
+  <dd>
+    Specifies an ordering of the rows of the complete result set of the query. The output rows are ordered


How about writing the default behaviour here (e.g., direction and null order)?

@maropu I have a page written up for ORDER BY where i have explained the sort direction and null order in detail with examples. In this page, i wanted to just briefly introduce the params. what do you think ?

maropu · 2020-01-17T05:52:25Z

docs/sql-ref-syntax-qry-select.md

@@ -18,8 +18,132 @@ license: |
  See the License for the specific language governing permissions and
  limitations under the License.
 ---
+Spark supports `SELECT` statement and conforms to ANSI SQL standard. Queries are


SELECT statement => a SELECT statement?

ANSI SQL standard -> the ANSI SQL standard?

maropu · 2020-01-17T05:52:52Z

docs/sql-ref-syntax-qry-select.md

@@ -18,8 +18,132 @@ license: |
  See the License for the specific language governing permissions and
  limitations under the License.
 ---
+Spark supports `SELECT` statement and conforms to ANSI SQL standard. Queries are
+used to retrieve result sets from one or more table. The following section 


more table -> more tables?

maropu · 2020-01-17T05:55:40Z

docs/sql-ref-syntax-qry-select.md

+  </dd>
+  <dt><code><em>boolean_expression</em></code></dt>
+  <dd>
+    Specifies a expression with a return type of boolean.


a expression -> an expression

maropu · 2020-01-17T05:57:31Z

docs/sql-ref-syntax-qry-select.md

+  <dt><code><em>LIMIT</em></code></dt>
+  <dd>
+    Specifies the maximum number of rows that can be returned by a statement or subquery. This clause 
+    is mostly used in the conjunction with <code>ORDER BY</code> to produce deterministic result. 


deterministic result -> a deterministic result?

maropu · 2020-01-17T06:00:42Z

docs/sql-ref-syntax-qry-select.md

+  <dt><code><em>SORT BY</em></code></dt>
+  <dd>
+    Specifies an ordering by which the rows are ordered within each partition. This parameter is mutually
+    exclusive with <code>ORDER BY</code>, <code>CLUSTER BY</code> and can not be specified together.


exclusive with <code>ORDER BY</code>, <code>CLUSTER BY</code> and => exclusive with <code>ORDER BY</code> or <code>CLUSTER BY</code>, and?

@maropu yeah.. sounds good. perhaps say "and" as opposed to "or" ?

yea, it looks fine to me. (but, I'm not a good English writer, so better to follow the others, hahaha

@maropu Same here. Hopefully @srowen will keep us honest :-)

…age)

dilipbiswal · 2020-01-17T06:38:11Z

@maropu @srowen I have incorporated the comments. Also i have removed the "Related Sections" and the links.

SparkQA · 2020-01-17T06:58:33Z

Test build #116903 has finished for PR 27216 at commit bd4c0ae.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2020-01-19T04:08:32Z

docs/sql-ref-syntax-qry-select.md

+    Specifies a set of expressions that is used to repartition and sort the rows. Using this clause has
+    the same effect of using <code>DISTRIBUTE BY</code> and <code>SORT BY</code> together. 
+  </dd>
+  <dt><code><em>DISTRIBUTE BY</em></code></dt>


https://cwiki.apache.org/confluence/display/Hive/LanguageManual+SortBy I think we also need a dedicated file for these clauses [SORT BY, CLUSTER BY and DISTRIBUTE BY].

These three clauses are very special. It is from Hive. Could we have a simple SELECT and then a full SELECT?

@gatorsmile the way i have at the moment is to have separate links for each of the clauses each having its syntax, parameters and examples. So yes, i will have separate links for the clauses you have mentioned. What do you think ?

Could you file jiras (sub-tickets) for planned dedicated files having these syntaxes?

@maropu Sure.

maropu · 2020-01-20T02:12:06Z

Also, can you update the screenshot in the PR description? That looks stale.

huaxingao · 2020-01-20T05:10:11Z

docs/sql-ref-syntax-qry-select.md

+  <dd>
+    Specifies the common table expressions (CTEs) before the main <code>SELECT</code> query block.
+    These table expressions are allowed to be referenced later in the main query. This is useful to abstract
+    out repeated sub query blocks in the main query and improves readability of the query.


I guess it is OK to use either sub query or subquery, but it might be better to pick one and keep consistent.

huaxingao · 2020-01-20T05:10:36Z

docs/sql-ref-syntax-qry-select.md

+  </dd>
+  <dt><code><em>from_item</em></code></dt>
+  <dd>
+    Specifies a source of input for the query. It can be one of the following.


: instead of . in the end of the sentence.

huaxingao · 2020-01-20T05:10:40Z

docs/sql-ref-syntax-qry-select.md

+  <dt><code><em>GROUP BY</em></code></dt>
+  <dd>
+    Specifies the expressions that are used to group the rows. This is used in conjunction with aggregate functions
+    (MIN, MAX, COUNT, SUM, AVG) to group rows bsed on the grouping expressions.


bsed typo

huaxingao · 2020-01-20T05:10:52Z

docs/sql-ref-syntax-qry-select.md

+  <dt><code><em>HAVING</em></code></dt>
+  <dd>
+    Specifies the predicates by which the rows produced by GROUP BY are filtered. The HAVING clause is used to
+    filter rows after the grouping is performed


. in the end of the sentence.

huaxingao · 2020-01-20T05:20:54Z

docs/sql-ref-syntax-qry-select.md

-along with usage examples when applicable.
+### Syntax
+{% highlight sql %}
+[WITH with_query [, ...]]


I guess always have a space between symbol and text, and between symbol and symbol?

[ WITH with_query [ , ... ] ]

SparkQA · 2020-01-20T07:35:12Z

Test build #117091 has finished for PR 27216 at commit 7a643d7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

guykhazma · 2020-01-20T08:53:56Z

docs/sql-ref-syntax-qry-select.md

+  </dd>
+  <dt><code><em>named_expression</em></code></dt>
+  <dd>
+    A expression with an assigned name. In general, it denotes a column expression.<br><br>


minor: An expression

SparkQA · 2020-01-20T21:44:05Z

Test build #117131 has finished for PR 27216 at commit 54dd2da.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2020-01-23T00:46:35Z

Merged to master

srowen reviewed Jan 16, 2020

View reviewed changes

dilipbiswal commented Jan 16, 2020

View reviewed changes

maropu reviewed Jan 17, 2020

View reviewed changes

[SPARK-28588][DOC] Document SELECT statement in SQL Reference (Main p…

bd4c0ae

…age)

dilipbiswal force-pushed the sql_ref_select_hook branch from de627aa to bd4c0ae Compare January 17, 2020 06:37

gatorsmile changed the title ~~[SPARK-28588][DOC] Document SELECT statement in SQL Reference (Main page)~~ [SPARK-28801][DOC] Document SELECT statement in SQL Reference (Main page) Jan 19, 2020

gatorsmile reviewed Jan 19, 2020

View reviewed changes

huaxingao reviewed Jan 20, 2020

View reviewed changes

review

7a643d7

guykhazma reviewed Jan 20, 2020

View reviewed changes

review

54dd2da

srowen approved these changes Jan 22, 2020

View reviewed changes

srowen closed this in 38f4e59 Jan 23, 2020

dongjoon-hyun added the SQL label Feb 5, 2020

[SPARK-28801][DOC] Document SELECT statement in SQL Reference (Main page) #27216

[SPARK-28801][DOC] Document SELECT statement in SQL Reference (Main page) #27216

Conversation

dilipbiswal commented Jan 15, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Jan 15, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dilipbiswal Jan 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dilipbiswal commented Jan 17, 2020

SparkQA commented Jan 17, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dilipbiswal Jan 19, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maropu commented Jan 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jan 20, 2020

Choose a reason for hiding this comment

SparkQA commented Jan 20, 2020

srowen commented Jan 23, 2020

dilipbiswal commented Jan 15, 2020 •

edited

Loading

dilipbiswal Jan 17, 2020 •

edited

Loading

dilipbiswal Jan 19, 2020 •

edited

Loading