[SPARK-20425][SQL] Support a vertical display mode for Dataset.show #17733

maropu · 2017-04-23T07:57:08Z

What changes were proposed in this pull request?

This pr added a new display mode for Dataset.show to print output rows vertically (one line per column value). In the current master, when printing Dataset with many columns, the readability is low like;

scala> val df = spark.range(100).selectExpr((0 until 100).map(i => s"rand() AS c$i"): _*)
scala> df.show(3, 0)
+------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+------------------+------------------+-------------------+------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+--------------------+-------------------+------------------+-------------------+--------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+--------------------+--------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+--------------------+-------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+-------------------+------------------+-------------------+------------------+------------------+-----------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+------------------+-------------------+-------------------+------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+
|c0                |c1                |c2                |c3                 |c4                |c5                |c6                 |c7                |c8                |c9                |c10               |c11                |c12               |c13               |c14               |c15                |c16                |c17                |c18               |c19               |c20                |c21               |c22                |c23               |c24                |c25                |c26                |c27                 |c28                |c29               |c30                |c31                 |c32               |c33               |c34                |c35                |c36                |c37               |c38               |c39                |c40               |c41               |c42                |c43                |c44                |c45               |c46                 |c47                 |c48                |c49                |c50                |c51                |c52                |c53                |c54                 |c55                |c56                |c57                |c58                |c59               |c60               |c61                |c62                |c63               |c64                |c65               |c66               |c67              |c68                |c69                |c70               |c71                |c72               |c73                |c74                |c75                |c76               |c77                |c78               |c79                |c80                |c81                |c82                |c83                |c84                |c85                |c86                |c87               |c88                |c89                |c90               |c91               |c92               |c93                |c94               |c95                |c96               |c97                |c98                |c99                |
+------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+------------------+------------------+-------------------+------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+--------------------+-------------------+------------------+-------------------+--------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+--------------------+--------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+--------------------+-------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+-------------------+------------------+-------------------+------------------+------------------+-----------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+------------------+-------------------+-------------------+------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+
|0.6306087152476858|0.9174349686288383|0.5511324165035159|0.3320844128641819 |0.7738486877101489|0.2154915886962553|0.4754997600674299 |0.922780639280355 |0.7136894772661909|0.2277580838165979|0.5926874459847249|0.40311408392226633|0.467830264333843 |0.8330466896984213|0.1893258482389527|0.6320849515511165 |0.7530911056912044 |0.06700254871955424|0.370528597355559 |0.2755437445193154|0.23704391110980128|0.8067400174905822|0.13597793616251852|0.1708888820162453|0.01672725007605702|0.983118121881555  |0.25040195628629924|0.060537253723083384|0.20000530582637488|0.3400572407133511|0.9375689433322597 |0.057039316954370256|0.8053269714347623|0.5247817572228813|0.28419308820527944|0.9798908885194533 |0.31805988175678146|0.7034448027077574|0.5400575751346084|0.25336322371116216|0.9361634546853429|0.6118681368289798|0.6295081549153907 |0.13417468943957422|0.41617137072255794|0.7267230869252035|0.023792726137561115|0.5776157058356362  |0.04884204913195467|0.26728716103441275|0.646680370807925  |0.9782712690657244 |0.16434031314818154|0.20985522381321275|0.24739842475440077 |0.26335189682977334|0.19604841662422068|0.10742950487300651|0.20283136488091502|0.3100312319723688|0.886959006630645 |0.25157102269776244|0.34428775168410786|0.3500506818575777|0.3781142441912052 |0.8560316444386715|0.4737104888956839|0.735903101602148|0.02236617130529006|0.8769074095835873 |0.2001426662503153|0.5534032319238532 |0.7289496620397098|0.41955191309992157|0.9337700133660436 |0.34059094378451005|0.6419144759403556|0.08167496930341167|0.9947099478497635|0.48010888605366586|0.22314796858167918|0.17786598882331306|0.7351521162297135 |0.5422057170020095 |0.9521927872726792 |0.7459825486368227 |0.40907708791990627|0.8903819313311575|0.7251413746923618 |0.2977174938745204 |0.9515209660203555|0.9375968604766713|0.5087851740042524|0.4255237544908751 |0.8023768698664653|0.48003189618006703|0.1775841829745185|0.09050775629268382|0.6743909291138167 |0.2498415755876865 |
|0.6866473844170801|0.4774360641212433|0.631696201340726 |0.33979113021468343|0.5663049010847052|0.7280190472258865|0.41370958502324806|0.9977433873622218|0.7671957338989901|0.2788708556233931|0.3355106391656496|0.88478952319287   |0.0333974166999893|0.6061744715862606|0.9617779139652359|0.22484954822341863|0.12770906021550898|0.5577789629508672 |0.2877649024640704|0.5566577406549361|0.9334933255278052 |0.9166720585157266|0.9689249324600591 |0.6367502457478598|0.7993572745928459 |0.23213222324218108|0.11928284054154137|0.6173493362456599  |0.0505122058694798 |0.9050228629552983|0.17112767911121707|0.47395598348370005 |0.5820498657823081|0.6241124650645072|0.18587258258036776|0.14987593554122225|0.3079446253653946 |0.9414228822867968|0.8362276265462365|0.9155655305576353 |0.5121559807153562|0.8963362656525707|0.22765970274318037|0.8177039187132797 |0.8190326635933787 |0.5256005177032199|0.8167598457269669  |0.030936807130934496|0.6733006585281015 |0.4208049626816347 |0.24603085738518538|0.22719198954208153|0.1622280557565281 |0.22217325159218038|0.014684419513742553|0.08987111517447499|0.2157764759142622 |0.8223414104088321 |0.4868624404491777 |0.4016191733088167|0.6169281906889263|0.15603611040433385|0.18289285085714913|0.9538408988218972|0.15037154865295121|0.5364516961987454|0.8077254873163031|0.712600478545675|0.7277477241003857 |0.19822912960348305|0.8305051199208777|0.18631911396566114|0.8909532487898342|0.3470409226992506 |0.35306974180587636|0.9107058868891469 |0.3321327206004986|0.48952332459050607|0.3630403307479373|0.5400046826340376 |0.5387377194310529 |0.42860539421837585|0.23214101630985995|0.21438968839794847|0.15370603160082352|0.04355605642700022|0.6096006707067466 |0.6933354157094292|0.06302172470859002|0.03174631856164001|0.664243581650643 |0.7833239547446621|0.696884598352864 |0.34626385933237736|0.9263495598791336|0.404818892816584  |0.2085585394755507|0.6150004897990109 |0.05391193524302473|0.28188484028329097|
+------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+------------------+------------------+-------------------+------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+--------------------+-------------------+------------------+-------------------+--------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+------------------+------------------+-------------------+-------------------+-------------------+------------------+--------------------+--------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+--------------------+-------------------+-------------------+-------------------+-------------------+------------------+------------------+-------------------+-------------------+------------------+-------------------+------------------+------------------+-----------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------+------------------+-------------------+-------------------+------------------+------------------+------------------+-------------------+------------------+-------------------+------------------+-------------------+-------------------+-------------------+
only showing top 2 rows

psql, CLI for PostgreSQL, supports a vertical display mode for this case like:
http://stackoverflow.com/questions/9604723/alternate-output-format-for-psql

-RECORD 0-------------------
 c0  | 0.6306087152476858
 c1  | 0.9174349686288383
 c2  | 0.5511324165035159
...
 c98 | 0.05391193524302473  
 c99 | 0.28188484028329097  
-RECORD 1-------------------
 c0  | 0.6866473844170801
 c1  | 0.4774360641212433
 c2  | 0.631696201340726
...
 c98 | 0.05391193524302473
 c99 | 0.28188484028329097
only showing top 2 rows

How was this patch tested?

Added tests in DataFrameSuite.

SparkQA · 2017-04-23T08:04:18Z

Test build #76077 has finished for PR 17733 at commit 901bcf8.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-04-23T10:28:32Z

Test build #76078 has finished for PR 17733 at commit 4551e77.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-04-23T13:30:19Z

Test build #76083 has finished for PR 17733 at commit b418900.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-04-23T14:57:07Z

Test build #76084 has finished for PR 17733 at commit 0df6eca.

This patch fails R style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-04-24T02:33:23Z

Test build #76087 has finished for PR 17733 at commit ca8bfbd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-04-25T19:09:41Z

R/pkg/R/DataFrame.R

@@ -194,6 +194,8 @@ setMethod("isLocal",
 #'                 20 characters will be truncated. However, if set greater than zero,
 #'                 truncates strings longer than \code{truncate} characters and all cells
 #'                 will be aligned right.
+#' @param extendedMode enable expanded table formatting mode to print a column data


extendedMode -> vertical?

yea, STGM. I'll update. Thanks!

gatorsmile · 2017-04-25T19:17:42Z

sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

+   *  c2  | 0.8508820250344251
+   *  c3  | 0.4593368817024575
+   *  c4  | 0.2216918145613194
+   *  c5  | 0.3756882647319614


Could you use the same example, as shown above?

gatorsmile · 2017-04-25T19:18:52Z

sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

+   * @param numRows Number of rows to show
+   * @param truncate If set to more than 0, truncates strings to `truncate` characters and
+   *                    all cells will be aligned right.
+   * @param extendedMode Enable expanded table formatting mode to print a column data per line.


Maybe we can follow what MySQL documents.

@param vertical Displays the result vertically.

This one? https://dev.mysql.com/doc/refman/5.7/en/mysql-command-options.html
Print query output rows vertically (one line per column value)?

SparkQA · 2017-04-26T04:19:23Z

Test build #76166 has finished for PR 17733 at commit f696d35.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-04-26T05:57:58Z

sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

+        }.getOrElse(0))
+      }
+
+      dataRows.zipWithIndex.foreach { case (row, i) =>


Like the non-vertical output, when no row exists, we at least need to output the column names?

df.limit(0).show(20, 0, true)

I checked and found that both pg and mysql output no column name in the mode;

// pg postgres=# create table t(a INT, b TEXT); CREATE TABLE postgres=# select * from t; a | b ---+--- (0 rows) postgres=# \x Expanded display is on. postgres=# select * from t; (0 rows) // mysql mysql -u root --vertical mysql> create table t(a INT, b TEXT); Query OK, 0 rows affected (0.04 sec) mysql> select * from t; Empty set (0.00 sec)

Now, in this PR, we output nothing in this case. Postgres and MySQL at least output the message to indicate the result set is empty.

cc @cloud-fan @sameeragarwal @hvanhovell @rxin Any suggestion here?

Aha, I see. I'll update. Thanks!

+1 for indicating that result set is empty. About displaying column names if the output is empty, perhaps it'd be best to stick with the postgres/mysql semantics.

SparkQA · 2017-04-26T16:45:09Z

Test build #76189 has finished for PR 17733 at commit 23feb21.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2017-04-26T22:54:59Z

Jenkins, retest this please.

SparkQA · 2017-04-27T01:05:05Z

Test build #76201 has finished for PR 17733 at commit 23feb21.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2017-04-27T01:53:04Z

@gatorsmile Could you check again? Thanks!

gatorsmile · 2017-04-27T05:17:05Z

LGTM

maropu force-pushed the SPARK-20425 branch from 901bcf8 to 4551e77 Compare April 23, 2017 08:45

maropu force-pushed the SPARK-20425 branch from 4551e77 to b418900 Compare April 23, 2017 11:24

maropu force-pushed the SPARK-20425 branch from b418900 to 0df6eca Compare April 23, 2017 14:50

maropu force-pushed the SPARK-20425 branch from 0df6eca to c3f48b8 Compare April 24, 2017 00:19

Support an extended display mode for Dataset.show

ca8bfbd

maropu force-pushed the SPARK-20425 branch from c3f48b8 to ca8bfbd Compare April 24, 2017 00:19

gatorsmile reviewed Apr 25, 2017

View reviewed changes

Apply comments

f696d35

felixcheung approved these changes Apr 26, 2017

View reviewed changes

maropu changed the title ~~[SPARK-20425][SQL] Support an extended display mode for Dataset.show~~ [SPARK-20425][SQL] Support an vertical display mode for Dataset.show Apr 26, 2017

maropu changed the title ~~[SPARK-20425][SQL] Support an vertical display mode for Dataset.show~~ [SPARK-20425][SQL] Support a vertical display mode for Dataset.show Apr 26, 2017

gatorsmile reviewed Apr 26, 2017

View reviewed changes

In a vertical mode, print an empty set

23feb21

asfgit closed this in b4724db Apr 27, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-20425][SQL] Support a vertical display mode for Dataset.show #17733

[SPARK-20425][SQL] Support a vertical display mode for Dataset.show #17733

maropu commented Apr 23, 2017 •

edited

SparkQA commented Apr 23, 2017

SparkQA commented Apr 23, 2017

SparkQA commented Apr 23, 2017

SparkQA commented Apr 23, 2017

SparkQA commented Apr 24, 2017

gatorsmile Apr 25, 2017

maropu Apr 26, 2017

gatorsmile Apr 25, 2017

maropu Apr 26, 2017

gatorsmile Apr 25, 2017

maropu Apr 26, 2017

gatorsmile Apr 26, 2017

SparkQA commented Apr 26, 2017

gatorsmile Apr 26, 2017 •

edited

maropu Apr 26, 2017 •

edited

gatorsmile Apr 26, 2017

maropu Apr 26, 2017

sameeragarwal Apr 26, 2017

SparkQA commented Apr 26, 2017

maropu commented Apr 26, 2017

SparkQA commented Apr 27, 2017

maropu commented Apr 27, 2017

gatorsmile commented Apr 27, 2017

[SPARK-20425][SQL] Support a vertical display mode for Dataset.show #17733

[SPARK-20425][SQL] Support a vertical display mode for Dataset.show #17733

Conversation

maropu commented Apr 23, 2017 • edited

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Apr 23, 2017

SparkQA commented Apr 23, 2017

SparkQA commented Apr 23, 2017

SparkQA commented Apr 23, 2017

SparkQA commented Apr 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Apr 26, 2017

gatorsmile Apr 26, 2017 • edited

Choose a reason for hiding this comment

maropu Apr 26, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Apr 26, 2017

maropu commented Apr 26, 2017

SparkQA commented Apr 27, 2017

maropu commented Apr 27, 2017

gatorsmile commented Apr 27, 2017

maropu commented Apr 23, 2017 •

edited

gatorsmile Apr 26, 2017 •

edited

maropu Apr 26, 2017 •

edited