Skip to content

Commit

Permalink
print Hadoop profile.
Browse files Browse the repository at this point in the history
  • Loading branch information
wangyum committed May 10, 2019
1 parent aaf2952 commit 994b5a7
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 4 deletions.
6 changes: 3 additions & 3 deletions dev/sparktestsupport/modules.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,13 @@
import re
import os

all_modules = []

if os.environ.get("AMPLAB_JENKINS"):
hadoop_version = os.environ.get("AMPLAB_JENKINS_BUILD_PROFILE", "hadoop2.7")
else:
hadoop_version = os.environ.get("HADOOP_PROFILE", "hadoop2.7")
print "[info] Choosing supported modules with Hadoop profile", hadoop_version

all_modules = []

@total_ordering
class Module(object):
Expand Down Expand Up @@ -79,7 +79,7 @@ def __init__(self, name, dependencies, source_file_regexes, build_profile_flags=
for dep in dependencies:
dep.dependent_modules.add(self)
if name == "hive-thriftserver" and hadoop_version == "hadoop3.2":
print("[info] Skip unsupported module: ", name)
print "[info] Skip unsupported module:", name
else:
all_modules.append(self)

Expand Down
2 changes: 1 addition & 1 deletion docs/sql-migration-guide-upgrade.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ license: |

## Upgrading From Spark SQL 2.4 to 3.0

- Since Spark 3.0, we upgraded the built-in Hive to 2.3 when enabling the Hadoop 3.2+ profile. This upgrade fixes some Hive issues:
- Since Spark 3.0, we upgraded the built-in Hive to 2.3 when enabling the Hadoop 3.2+ profile. This upgrade fixes the following issues:

- [HIVE-6727](https://issues.apache.org/jira/browse/HIVE-6727): Table level stats for external tables are set incorrectly.

Expand Down

0 comments on commit 994b5a7

Please sign in to comment.