Skip to content

Commit

Permalink
[HOTFIX] Fix documentation errors.Add examples for pre-aggregate usage
Browse files Browse the repository at this point in the history
  • Loading branch information
sraghunandan committed Feb 7, 2018
1 parent 3d57d13 commit 3cb6bff
Show file tree
Hide file tree
Showing 2 changed files with 31 additions and 24 deletions.
31 changes: 7 additions & 24 deletions docs/data-management-on-carbondata.md
Original file line number Diff line number Diff line change
Expand Up @@ -910,18 +910,13 @@ pre-aggregate tables satisfy the query condition, the plan is transformed automa
pre-aggregate table to fetch the data

##### Compacting pre-aggregate tables
Compaction is an optional operation for pre-aggregate table. If compaction is performed on main
table but not performed on pre-aggregate table, all queries still can benefit from pre-aggregate
table.To further improve performance on pre-aggregate table, compaction can be triggered on
pre-aggregate tables directly, it will merge the segments inside pre-aggregation table.
To do that, use ALTER TABLE COMPACT command on the pre-aggregate table just like the main table
Compaction command (ALTER TABLE COMPACT) need to be run separately on each pre-aggregate table.
Running Compaction command on main table will **not automatically** compact the pre-aggregate
tables.Compaction is an optional operation for pre-aggregate table. If compaction is performed on
main table but not performed on pre-aggregate table, all queries still can benefit from
pre-aggregate tables.To further improve performance on pre-aggregate tables, compaction can be
triggered on pre-aggregate tables directly, it will merge the segments inside pre-aggregate table.

NOTE:
* If the aggregate function used in the pre-aggregate table creation included distinct-count,
during compaction, the pre-aggregate table values are recomputed.This would a costly
operation as compared to the compaction of pre-aggregate tables containing other aggregate
functions alone

##### Update/Delete Operations on pre-aggregate tables
This functionality is not supported.

Expand Down Expand Up @@ -996,16 +991,6 @@ roll-up for the queries on these hierarchies.
SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
avg(price) FROM sales GROUP BY order_time, country, sex
CREATE DATAMAP agg_minute
ON TABLE sales
USING "timeseries"
DMPROPERTIES (
'event_time’=’order_time’,
'minute_granualrity’=’1’,
) AS
SELECT order_time, country, sex, sum(quantity), max(quantity), count(user_id), sum(price),
avg(price) FROM sales GROUP BY order_time, country, sex
CREATE DATAMAP agg_minute
ON TABLE sales
USING "timeseries"
Expand All @@ -1028,9 +1013,7 @@ roll-up for the queries on these hierarchies.
```

It is **not necessary** to create pre-aggregate tables for each granularity unless required for
query
.Carbondata
can roll-up the data and fetch it
query.Carbondata can roll-up the data and fetch it.

For Example: For main table **sales** , If pre-aggregate tables were created as

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,30 @@ object PreAggregateTableExample {
println("time for query on table without pre-aggregate table:" + time_without_aggTable.toString)
// scalastyle:on

// 3. if avg function is defined for a column, sum also can be used on that;but not other way
// round
val time_without_aggTable_sum = time {
spark.sql(
s"""
| SELECT id, sum(age)
| FROM personTableWithoutAgg group by id
""".stripMargin).count()
}

val time_with_aggTable_sum = time {
spark.sql(
s"""
| SELECT id, sum(age)
| FROM personTable group by id
""".stripMargin).count()
}
// scalastyle:off
println("time for query with function sum on table with pre-aggregate table:" +
time_with_aggTable_sum.toString)
println("time for query with function sum on table without pre-aggregate table:" +
time_without_aggTable_sum.toString)
// scalastyle:on

spark.sql("DROP TABLE IF EXISTS mainTable")
spark.sql("DROP TABLE IF EXISTS personTable")
spark.sql("DROP TABLE IF EXISTS personTableWithoutAgg")
Expand Down

0 comments on commit 3cb6bff

Please sign in to comment.