[CARBONDATA-3087] Improve DESC FORMATTED output #2908

jackylk · 2018-11-08T12:13:01Z

Change output of DESC FORMATTED to:

+----------------------------+-------------------------------------------------------------------------------------+-------+
|col_name                    |data_type                                                                            |comment|
+----------------------------+-------------------------------------------------------------------------------------+-------+
|shortfield                  |smallint                                                                             |null   |
|intfield                    |int                                                                                  |null   |
|bigintfield                 |bigint                                                                               |null   |
|doublefield                 |double                                                                               |null   |
|stringfield                 |string                                                                               |null   |
|timestampfield              |timestamp                                                                            |null   |
|decimalfield                |decimal(18,2)                                                                        |null   |
|datefield                   |date                                                                                 |null   |
|charfield                   |string                                                                               |null   |
|floatfield                  |double                                                                               |null   |
|                            |                                                                                     |       |
|## Table Basic Information  |                                                                                     |       |
|Comment                     |                                                                                     |       |
|Path                        |/Users/jacky/code/carbondata/examples/spark2/target/store/default/carbonsession_table|       |
|Table Block Size            |1024 MB                                                                              |       |
|Table Blocklet Size         |64 MB                                                                                |       |
|Streaming                   |false                                                                                |       |
|Flat Folder                 |false                                                                                |       |
|Bad Record Path             |                                                                                     |       |
|Min Input Per Node          |0.0B                                                                                 |       |
|                            |                                                                                     |       |
|## Index Information        |                                                                                     |       |
|Sort Scope                  |LOCAL_SORT                                                                           |       |
|Sort Columns                |stringfield,timestampfield,datefield,charfield                                       |       |
|Index Cache Level           |BLOCK                                                                                |       |
|Cached Index Columns        |All columns                                                                          |       |
|                            |                                                                                     |       |
|## Encoding Information     |                                                                                     |       |
|Local Dictionary Enabled    |true                                                                                 |       |
|Local Dictionary Threshold  |10000                                                                                |       |
|Local Dictionary Include    |stringfield                                                                          |       |
|                            |                                                                                     |       |
|## Compaction Information   |                                                                                     |       |
|MAJOR_COMPACTION_SIZE       |1024                                                                                 |       |
|AUTO_LOAD_MERGE             |false                                                                                |       |
|COMPACTION_LEVEL_THRESHOLD  |4,3                                                                                  |       |
|COMPACTION_PRESERVE_SEGMENTS|0                                                                                    |       |
|ALLOWED_COMPACTION_DAYS     |0                                                                                    |       |
|                            |                                                                                     |       |
|## Dynamic Information      |                                                                                     |       |
|Table Data Size             |2.93KB                                                                               |       |
|Table Index Size            |1.53KB                                                                               |       |
|Last Update Time            |Wed Nov 07 17:16:01 CST 2018                                                         |       |
+----------------------------+-------------------------------------------------------------------------------------+-------+

Any interfaces changed?
Any backward compatibility impacted?
Document update required?
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

CarbonDataQA · 2018-11-08T12:20:33Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1331/

CarbonDataQA · 2018-11-08T12:25:58Z

Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9590/

CarbonDataQA · 2018-11-08T12:32:38Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1542/

CarbonDataQA · 2018-11-08T13:34:53Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1333/

CarbonDataQA · 2018-11-08T14:34:12Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1544/

CarbonDataQA · 2018-11-08T14:34:37Z

Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9592/

zzcclp · 2018-11-08T14:54:06Z

Hi @jackylk :
I have some questions shown below:

How to show external info if it's an external table?
It need to show the partition info if it has partition columns;
missing property: DICTIONARY_INCLUDE;
doesn't it show database and table name?
will remove NO_INVERTED_INDEX ? right?
doesn't it show default value for each properties? and as our discussion, we need to save the default value to schema info even user doesn't set the value, is this done?

jackylk · 2018-11-09T01:27:19Z

@zzcclp

I think we can show the table path
The current DESC already has the logic to show partition info, this part is not changed
DICTIONARY_INCLUDE is added in encoding section
Since the DESC syntax includes database name and table name, so I think it is not required to show them again.
yes, INVERTED_INDEX is added in [CARBONDATA-3065]make inverted index false by default #2886
It is not done, and in this PR the most important table property is shown, it will show default value is user did not specify the property when creating table.

Will fix according to these comments and push again

CarbonDataQA · 2018-11-09T03:09:45Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1338/

jackylk · 2018-11-09T03:13:53Z

@zzcclp currently, it will show

+-------------------------------+------------------------------------------------------------------------------------------------+----------+--+
|           col_name            |                                           data_type                                            | comment  |
+-------------------------------+------------------------------------------------------------------------------------------------+----------+--+
| l_partkey                     | int                                                                                            | null     |
| l_suppkey                     | string                                                                                         | null     |
| l_linenumber                  | int                                                                                            | null     |
| l_quantity                    | double                                                                                         | null     |
| l_extendedprice               | double                                                                                         | null     |
| l_discount                    | double                                                                                         | null     |
| l_tax                         | double                                                                                         | null     |
| l_returnflag                  | string                                                                                         | null     |
| l_linestatus                  | string                                                                                         | null     |
| l_shipdate                    | date                                                                                           | null     |
| l_commitdate                  | date                                                                                           | null     |
| l_receiptdate                 | date                                                                                           | null     |
| l_shipinstruct                | string                                                                                         | null     |
| l_shipmode                    | string                                                                                         | null     |
| l_comment                     | string                                                                                         | null     |
| l_orderkey                    | int                                                                                            | null     |
|                               |                                                                                                |          |
| ## Table Basic Information    |                                                                                                |          |
| Path                          | /Users/jacky/code/spark-2.2.1-bin-hadoop2.7/carbonstore/tpchcarbon_base/lineitem_hash_no_sort  |          |
| External                      | true                                                                                           |          |
| Transactional                 | true                                                                                           |          |
| Flat Folder                   | false                                                                                          |          |
| Streaming                     | false                                                                                          |          |
| Table Block Size              | 128 MB                                                                                         |          |
| Table Blocklet Size           | 64 MB                                                                                          |          |
| Comment                       |                                                                                                |          |
| Bad Record Path               |                                                                                                |          |
| Min Input Per Node            | 0.0B                                                                                           |          |
|                               |                                                                                                |          |
| ## Index Information          |                                                                                                |          |
| Sort Scope                    | no_sort                                                                                        |          |
| Sort Columns                  | l_orderkey                                                                                     |          |
| Index Cache Level             | BLOCK                                                                                          |          |
| Cached Index Columns          | All columns                                                                                    |          |
|                               |                                                                                                |          |
| ## Encoding Information       |                                                                                                |          |
| Local Dictionary Enabled      | true                                                                                           |          |
| Local Dictionary Threshold    | 10000                                                                                          |          |
| Local Dictionary Include      | l_suppkey,l_returnflag,l_linestatus,l_shipinstruct,l_shipmode,l_comment                        |          |
| Global Dictionary             |                                                                                                |          |
|                               |                                                                                                |          |
| ## Compaction Information     |                                                                                                |          |
| MAJOR_COMPACTION_SIZE         | 1024                                                                                           |          |
| AUTO_LOAD_MERGE               | false                                                                                          |          |
| COMPACTION_LEVEL_THRESHOLD    | 4,3                                                                                            |          |
| COMPACTION_PRESERVE_SEGMENTS  | 0                                                                                              |          |
| ALLOWED_COMPACTION_DAYS       | 0                                                                                              |          |
|                               |                                                                                                |          |
| ## Partition Information      |                                                                                                |          |
| Partition Type                | HASH                                                                                           |          |
| Partition Columns             | l_orderkey:INT                                                                                 |          |
| Number of Partitions          | 6                                                                                              |          |
| Partitions Ids                | 0,1,2,3,4,5                                                                                    |          |
|                               |                                                                                                |          |
| ## Dynamic Information        |                                                                                                |          |
| Table Data Size               | 226.26MB                                                                                       |          |
| Table Index Size              | 13.71KB                                                                                        |          |
| Last Update Time              | Thu Nov 08 19:31:30 CST 2018                                                                   |          |
+-------------------------------+------------------------------------------------------------------------------------------------+----------+--+

CarbonDataQA · 2018-11-09T04:09:46Z

Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9597/

CarbonDataQA · 2018-11-09T04:10:20Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1549/

CarbonDataQA · 2018-11-09T09:10:19Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1347/

CarbonDataQA · 2018-11-09T09:48:04Z

Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9606/

CarbonDataQA · 2018-11-09T09:48:21Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1558/

CarbonDataQA · 2018-11-09T09:51:11Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1348/

CarbonDataQA · 2018-11-09T13:20:01Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1350/

CarbonDataQA · 2018-11-09T14:19:35Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1560/

CarbonDataQA · 2018-11-09T14:19:50Z

Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9608/

zzcclp · 2018-11-09T17:05:33Z

@jackylk it now looks good to me, you added database and table name in the latest commit , right?

CarbonDataQA · 2018-11-09T17:49:52Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1352/

CarbonDataQA · 2018-11-09T18:48:55Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1562/

CarbonDataQA · 2018-11-09T18:49:44Z

Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9610/

CarbonDataQA · 2018-11-13T15:02:29Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1385/

fix compile fix comment fix comment fix test fix test fix test wip fix test

CarbonDataQA · 2018-11-13T15:49:05Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1386/

CarbonDataQA · 2018-11-13T16:49:23Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1596/

CarbonDataQA · 2018-11-13T17:02:38Z

Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9644/

CarbonDataQA · 2018-11-14T02:14:35Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1391/

CarbonDataQA · 2018-11-14T03:26:10Z

Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9649/

CarbonDataQA · 2018-11-14T03:39:14Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1601/

CarbonDataQA · 2018-11-14T07:24:51Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1401/

CarbonDataQA · 2018-11-14T08:27:11Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1611/

CarbonDataQA · 2018-11-14T08:28:01Z

Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9659/

xuchuanyin · 2018-11-15T08:37:40Z

LGTM

ravipesala · 2018-11-15T09:05:56Z

core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java

        }
+        return cachedColsList;
+      } else {
+        return new LinkedList<>();


I think it is not required to create LinkedList here. we can return cachedColsList directly

Change output of DESC FORMATTED This closes #2908

jackylk force-pushed the desc branch from 15dd83e to 0047842 Compare November 9, 2018 02:58

jackylk force-pushed the desc branch from a97472d to 1ecd2d7 Compare November 9, 2018 09:45

jackylk force-pushed the desc branch from 1ecd2d7 to 1ea2bae Compare November 9, 2018 13:08

improve desc

7267413

fix compile fix comment fix comment fix test fix test fix test wip fix test

jackylk force-pushed the desc branch from 818d624 to 7267413 Compare November 13, 2018 15:38

support range partition info

4de2059

fix test

12e36e6

asfgit closed this in 851dd2c Nov 15, 2018

ravipesala reviewed Nov 15, 2018

View reviewed changes

asfgit pushed a commit that referenced this pull request Nov 21, 2018

[CARBONDATA-3087] Improve DESC FORMATTED output

ab1070b

Change output of DESC FORMATTED This closes #2908

[CARBONDATA-3087] Improve DESC FORMATTED output #2908

[CARBONDATA-3087] Improve DESC FORMATTED output #2908

Conversation

jackylk commented Nov 8, 2018

CarbonDataQA commented Nov 8, 2018

CarbonDataQA commented Nov 8, 2018

CarbonDataQA commented Nov 8, 2018

CarbonDataQA commented Nov 8, 2018

CarbonDataQA commented Nov 8, 2018

CarbonDataQA commented Nov 8, 2018

zzcclp commented Nov 8, 2018

jackylk commented Nov 9, 2018

CarbonDataQA commented Nov 9, 2018

jackylk commented Nov 9, 2018

CarbonDataQA commented Nov 9, 2018

CarbonDataQA commented Nov 9, 2018

CarbonDataQA commented Nov 9, 2018

CarbonDataQA commented Nov 9, 2018

CarbonDataQA commented Nov 9, 2018

CarbonDataQA commented Nov 9, 2018

CarbonDataQA commented Nov 9, 2018

CarbonDataQA commented Nov 9, 2018

CarbonDataQA commented Nov 9, 2018

zzcclp commented Nov 9, 2018

CarbonDataQA commented Nov 9, 2018

CarbonDataQA commented Nov 9, 2018

CarbonDataQA commented Nov 9, 2018

CarbonDataQA commented Nov 13, 2018

CarbonDataQA commented Nov 13, 2018

CarbonDataQA commented Nov 13, 2018

CarbonDataQA commented Nov 13, 2018

CarbonDataQA commented Nov 14, 2018

CarbonDataQA commented Nov 14, 2018

CarbonDataQA commented Nov 14, 2018

CarbonDataQA commented Nov 14, 2018

CarbonDataQA commented Nov 14, 2018

CarbonDataQA commented Nov 14, 2018

xuchuanyin commented Nov 15, 2018

ravipesala Nov 15, 2018

Choose a reason for hiding this comment