Skip to content

Commit

Permalink
[MINOR] docs: update document and build script for Hadoop-3.2 (#912)
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

- Update document, including the supported mapReduce version and build command.
- Update build_distribution.sh. We can't recognize the hadoop version from final package name for now. We should update the final package name.
  • Loading branch information
zhengchenyu committed May 30, 2023
1 parent b08efa3 commit 16f39f5
Show file tree
Hide file tree
Showing 2 changed files with 25 additions and 4 deletions.
14 changes: 13 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ Currently supports Spark 2.3.x, Spark 2.4.x, Spark 3.0.x, Spark 3.1.x, Spark 3.2
Note: To support dynamic allocation, the patch(which is included in patch/spark folder) should be applied to Spark

## Supported MapReduce Version
Currently supports the MapReduce framework of Hadoop 2.8.5
Currently supports the MapReduce framework of Hadoop 2.8.5, Hadoop 3.2.1

## Building Uniffle
> note: currently Uniffle requires JDK 1.8 to build, adding later JDK support is on our roadmap.
Expand All @@ -96,6 +96,14 @@ Build against Spark 3.2.0

mvn -DskipTests clean package -Pspark3.2.0

Build against Hadoop 2.8.5

mvn -DskipTests clean package -Pmr,hadoop2.8

Build against Hadoop 3.2.1

mvn -DskipTests clean package -Pmr,hadoop3.2

To package the Uniffle, run:

./build_distribution.sh
Expand All @@ -108,6 +116,10 @@ Package against Spark 3.2.0, run:

./build_distribution.sh --spark3-profile 'spark3.2.0'

Package will build against Hadoop 2.8.5 in default. If you want to build package against Hadoop 3.2.1, run:

./build_distribution.sh --hadoop-profile 'hadoop3.2'

rss-xxx.tgz will be generated for deployment

## Deploy
Expand Down
15 changes: 12 additions & 3 deletions build_distribution.sh
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ SPARK2_MVN_OPTS=""
SPARK3_PROFILE_ID="spark3"
SPARK3_MVN_OPTS=""
HADOOP_PROFILE_ID="hadoop2.8"
NAME=none
while (( "$#" )); do
case $1 in
--spark2-profile)
Expand All @@ -70,6 +71,10 @@ while (( "$#" )); do
HADOOP_PROFILE_ID=$2
shift
;;
--name)
NAME="$2"
shift
;;
--help)
exit_with_usage
;;
Expand Down Expand Up @@ -122,6 +127,10 @@ SPARK3_VERSION=$("$MVN" help:evaluate -Dexpression=spark.version -P$SPARK3_PROFI
| grep -v "WARNING"\
| tail -n 1)

if [ "$NAME" == "none" ]; then
NAME=$HADOOP_PROFILE_ID
fi

echo "RSS version is $VERSION"

export MAVEN_OPTS="${MAVEN_OPTS:--Xmx2g -XX:ReservedCodeCacheSize=1g}"
Expand All @@ -139,7 +148,7 @@ echo -e "\$ ${BUILD_COMMAND[@]}\n"


# Make directories
DISTDIR="rss-$VERSION"
DISTDIR="rss-$VERSION-$NAME"
rm -rf "$DISTDIR"
mkdir -p "${DISTDIR}/jars"
echo "RSS ${VERSION}${GITREVSTRING} built for Hadoop ${HADOOP_VERSION} Spark2 ${SPARK2_VERSION} Spark3 ${SPARK3_VERSION}" >"${DISTDIR}/RELEASE"
Expand Down Expand Up @@ -210,6 +219,6 @@ cp $MR_CLIENT_JAR $MR_CLIENT_JAR_DIR
cp -r bin $DISTDIR
cp -r conf $DISTDIR

rm -rf "rss-$VERSION.tgz"
tar czf "rss-$VERSION.tgz" $DISTDIR
rm -rf "rss-$VERSION-$NAME.tgz"
tar czf "rss-$VERSION-$NAME.tgz" $DISTDIR
rm -rf $DISTDIR

0 comments on commit 16f39f5

Please sign in to comment.