-
Notifications
You must be signed in to change notification settings - Fork 1
/
improvmentsFromPartitioning.out
98 lines (84 loc) · 7.37 KB
/
improvmentsFromPartitioning.out
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
[marty@gromit statsParquetPartition]$ impala-shell -i $DATANODE
Starting Impala Shell without Kerberos authentication
Connected to gromit.lurie.biz:21000
Server version: impalad version 2.12.0-cdh5.15.0 RELEASE (build 23f574543323301846b41fa5433690df32efe085)
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.12.0-cdh5.15.0 (23f5745) built on Thu May 24 04:07:31 PDT 2018)
Want to know what version of Impala you're connected to? Run the VERSION command to
find out!
***********************************************************************************
[gromit.lurie.biz:21000] > explain select myonein10k from partitionparquet where mydayofyear=365 limit 5;
Query: explain select myonein10k from partitionparquet where mydayofyear=365 limit 5
+------------------------------------------------------------------------------------+
| Explain String |
+------------------------------------------------------------------------------------+
| Max Per-Host Resource Reservation: Memory=0B |
| Per-Host Resource Estimates: Memory=32.00MB |
| WARNING: The following tables are missing relevant table and/or column statistics. |
| default.partitionparquet |
| |
| PLAN-ROOT SINK |
| | |
| 01:EXCHANGE [UNPARTITIONED] |
| | limit: 5 |
| | |
| 00:SCAN HDFS [default.partitionparquet] |
| partitions=84/84 files=84 size=8.17MB |
| predicates: mydayofyear = 365 |
| limit: 5 |
+------------------------------------------------------------------------------------+
Fetched 14 row(s) in 0.03s
[gromit.lurie.biz:21000] > explain select myonein10k from partitionparquet where myweekday=3 limit 5;
Query: explain select myonein10k from partitionparquet where myweekday=3 limit 5
+------------------------------------------------------------------------------------+
| Explain String |
+------------------------------------------------------------------------------------+
| Max Per-Host Resource Reservation: Memory=0B |
| Per-Host Resource Estimates: Memory=16.00MB |
| Codegen disabled by planner |
| WARNING: The following tables are missing relevant table and/or column statistics. |
| default.partitionparquet |
| |
| PLAN-ROOT SINK |
| | |
| 00:SCAN HDFS [default.partitionparquet] |
| partitions=12/84 files=12 size=1.17MB |
| limit: 5 |
+------------------------------------------------------------------------------------+
Fetched 11 row(s) in 0.01s
[gromit.lurie.biz:21000] > explain select myonein10k from partitionparquet where mymonth=5 and myweekday=3 limit 5;
Query: explain select myonein10k from partitionparquet where mymonth=5 and myweekday=3 limit 5
+------------------------------------------------------------------------------------+
| Explain String |
+------------------------------------------------------------------------------------+
| Max Per-Host Resource Reservation: Memory=0B |
| Per-Host Resource Estimates: Memory=16.00MB |
| Codegen disabled by planner |
| WARNING: The following tables are missing relevant table and/or column statistics. |
| default.partitionparquet |
| |
| PLAN-ROOT SINK |
| | |
| 00:SCAN HDFS [default.partitionparquet] |
| partitions=1/84 files=1 size=99.59KB |
| limit: 5 |
+------------------------------------------------------------------------------------+
Fetched 11 row(s) in 0.01s
Sample directory structure:
[marty@gromit ~]$ hadoop fs -ls -R /user/hive/warehouse/partitionparquet
... lines deleted....
drwxr-xr-x - impala hive 0 2018-08-12 16:04 /user/hive/warehouse/partitionparquet/mymonth=8/myweekday=5
-rw-r--r-- 3 impala hive 101938 2018-08-12 16:04 /user/hive/warehouse/partitionparquet/mymonth=8/myweekday=5/7542477d2955a40b-c74624c600000000_2143280810_data.0.parq
drwxr-xr-x - impala hive 0 2018-08-12 16:04 /user/hive/warehouse/partitionparquet/mymonth=8/myweekday=6
-rw-r--r-- 3 impala hive 101977 2018-08-12 16:04 /user/hive/warehouse/partitionparquet/mymonth=8/myweekday=6/7542477d2955a40b-c74624c600000000_396942592_data.0.parq
drwxr-xr-x - impala hive 0 2018-08-12 16:04 /user/hive/warehouse/partitionparquet/mymonth=8/myweekday=7
-rw-r--r-- 3 impala hive 101977 2018-08-12 16:04 /user/hive/warehouse/partitionparquet/mymonth=8/myweekday=7/7542477d2955a40b-c74624c600000000_76907610_data.0.parq
drwxr-xr-x - impala hive 0 2018-08-12 16:04 /user/hive/warehouse/partitionparquet/mymonth=9
drwxr-xr-x - impala hive 0 2018-08-12 16:04 /user/hive/warehouse/partitionparquet/mymonth=9/myweekday=1
-rw-r--r-- 3 impala hive 101977 2018-08-12 16:04 /user/hive/warehouse/partitionparquet/mymonth=9/myweekday=1/7542477d2955a40b-c74624c600000000_839998333_data.0.parq
drwxr-xr-x - impala hive 0 2018-08-12 16:04 /user/hive/warehouse/partitionparquet/mymonth=9/myweekday=2
-rw-r--r-- 3 impala hive 101977 2018-08-12 16:04 /user/hive/warehouse/partitionparquet/mymonth=9/myweekday=2/7542477d2955a40b-c74624c600000000_1718551971_data.0.parq
drwxr-xr-x - impala hive 0 2018-08-12 16:04 /user/hive/warehouse/partitionparquet/mymonth=9/myweekday=3
-rw-r--r-- 3 impala hive 101977 2018-08-12 16:04 /user/hive/warehouse/partitionparquet/mymonth=9/myweekday=3/7542477d2955a40b-c74624c600000000_91939897_data.0.parq
drwxr-xr-x - impala hive 0 2018-08-12 16:04 /user/hive/warehouse/partitionparquet/mymonth=9/myweekday=4