Skip to content

Commit

Permalink
Fix OOM when data loading with the new Hive SNAPSHOT.
Browse files Browse the repository at this point in the history
This patch is required for upgrading thirdparty.

After upgrading thirdparty, Hive OOMs in data loading TPCDS.
This patch fixes the OOM, but I have no idea why exactly it works.
I had tried several other approaches like increasing the
mapred and JVM heap memory to no avail.

Change-Id: I3c69e4c3bf0f24e0c3b6272c946f71c17e312e01
Reviewed-on: http://gerrit.cloudera.org:8080/2145
Reviewed-by: Alex Behm <alex.behm@cloudera.com>
Tested-by: Internal Jenkins
  • Loading branch information
Alex Behm authored and Internal Jenkins committed Feb 14, 2016
1 parent 6e41426 commit 53cc3a3
Show file tree
Hide file tree
Showing 2 changed files with 1 addition and 5 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -1694,7 +1694,7 @@ functional
---- BASE_TABLE_NAME
avro_unicode_nulls
---- CREATE_HIVE
create external table {db_name}{db_suffix}.{table_name} like {db_name}.liketbl stored as avro LOCATION '/test-warehouse/avro_null_char';
create external table if not exists {db_name}{db_suffix}.{table_name} like {db_name}.liketbl stored as avro LOCATION '/test-warehouse/avro_null_char';
---- LOAD
`hdfs dfs -mkdir -p /test-warehouse/avro_null_char && \
hdfs dfs -put -f ${IMPALA_HOME}/testdata/avro_null_char/000000_0 /test-warehouse/avro_null_char/
Expand Down
4 changes: 0 additions & 4 deletions testdata/datasets/tpcds/tpcds_schema_template.sql
Original file line number Diff line number Diff line change
Expand Up @@ -463,8 +463,6 @@ WHERE 2452184 <= ss_sold_date_sk;
---- LOAD
USE {db_name};

-- Disable auto.convert.join due to HIVE-5068
set hive.auto.convert.join=false;
set hive.exec.max.dynamic.partitions.pernode=10000;
set hive.exec.max.dynamic.partitions=10000;
set hive.exec.dynamic.partition.mode=nonstrict;
Expand Down Expand Up @@ -499,8 +497,6 @@ distribute by ss_sold_date_sk;
---- LOAD_LOCAL
USE {db_name};

-- Disable auto.convert.join due to HIVE-5068
set hive.auto.convert.join=false;
set hive.exec.max.dynamic.partitions.pernode=10000;
set hive.exec.max.dynamic.partitions=10000;
set hive.exec.dynamic.partition.mode=nonstrict;
Expand Down

0 comments on commit 53cc3a3

Please sign in to comment.