Skip to content
Permalink
Browse files
HBASE-23003 [HBCK2/hbase-operator-tools] Release-making scripts
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
  • Loading branch information
saintstack committed Sep 13, 2019
1 parent 1c175f8 commit abc881a4e4552cbaca9899d78577cf7e41081c8c
Showing 9 changed files with 1,196 additions and 0 deletions.
@@ -0,0 +1,61 @@
Entrance script is _do-release-docker.sh_. We run the release
creation inside a Docker container to ensure consistent
build environment (See _hbase_rm_ dir for the _Dockerfile_
used). The script depends on their being a local docker;
for example, on mac os x, a _Docker for Desktop_ installed
and running. It does all steps building a release. See
head of _do-release-docker.sh_ for list of what this all
entails.

_do-release-docker.sh_ calls _do-release.sh_ which is the core
absent docker machinations that runs the release-making process.
Use it to avoid running release creation in a container.

If you just want to generate tarballs locally, run release-build.sh.
It will ask for passwords to use building or you can supply as
environment variables. For example:

$ ASF_USERNAME="USERNAME" ASF_PASSWORD="PASSWORD" GPG_PASSPHRASE="PASSWORD" GPG_KEY="stack@duboce.net" GIT_REF=e3eb8ce83075cdfee7f7a9be3cb9de7686c4905f ./release-build.sh build

Ditto for running _do-release*.sh_.

Before starting the RC build, run a reconciliation of what is in
JIRA with what is in the commit log. Make sure they align and that
anomalies are explained up in JIRA (TODO, a script that can
do this: HBASE-22853).

See http://hbase.apache.org/book.html#maven.release

Running a build on GCE is easy enough. Here are some notes that
may help.

Create an instance. 4CPU/15G/10G disk seems to work well enough.
Once up, run the below to make your machine fit for RC building:

# Presuming debian-compatible OS
$ sudo apt-get install -y git openjdk-8-jdk maven gnupg gnupg-agent
# Install docker
$ sudo apt-get install -y \
apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common
$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
$ sudo add-apt-repository -y \
"deb [arch=amd64] https://download.docker.com/linux/debian \
$(lsb_release -cs) \
stable"
$ sudo apt-get update
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
$ sudo usermod -a -G docker $USERID
# LOGOUT and then LOGIN again so $USERID shows as part of docker group
# Copy up private key for $USERID export from laptop and import on gce.
$ gpg --import stack.duboce.net.asc
$ export GPG_TTY=$(tty) # https://github.com/keybase/keybase-issues/issues/2798
$ eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400)
$ git clone https://github.com/apache/hbase-operator-tools.git
$ cd hbase-operator-tools
$ mkdir ~/build
$ ./dev-resources/create-release/do-release-docker.sh -d ~/build
# etc.
@@ -0,0 +1,167 @@
#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#
# Creates a HBase release candidate. The script will update versions, tag the branch,
# build HBase binary packages and documentation, and upload maven artifacts to a staging
# repository. There is also a dry run mode where only local builds are performed, and
# nothing is uploaded to the ASF repos.
#
# Run with "-h" for options. For example, running below will do all
# steps above using the 'rm' dir under Downloads as workspace:
#
# $ ./do-release-docker.sh -d ~/Downloads/rm
#
# The scripts in this directory came originally from spark [1]. They were then
# modified to suite the hbase context. These scripts supercedes the old
# ../make_rc.sh script for making release candidates because what is here is more
# comprehensive doing more steps of the RM process as well as running in a
# container so the RM build environment can be a constant.
#
# It:
# * Tags release
# * Sets version to the release version
# * Sets version to next SNAPSHOT version.
# * Builds, signs, and hashes all artifacts.
# * Pushes release tgzs to the dev dir in a apache dist.
# * Pushes to repository.apache.org staging.
#
# The entry point is here, in the do-release-docker.sh script.
#
# 1. https://github.com/apache/spark/tree/master/dev/create-release
#
set -e

export PROJECT="hbase-operator-tools"

SELF=$(cd $(dirname "$0") && pwd)
. "$SELF/release-util.sh"

function usage {
local NAME
NAME="$(basename "$0")"
cat <<EOF
Usage: $NAME [options]
This script runs the release scripts inside a docker image.
Options:
-d [path] required. working directory. output will be written to "output" in here.
-n dry run mode. Checks and local builds, but does not upload anything.
-t [tag] tag for the hbase-rm docker image to use for building (default: "latest").
-j [path] path to local JDK installation to use building. By default the script will
use openjdk8 installed in the docker image.
-s [step] runs a single step of the process; valid steps are: tag, build, publish. if
none specified, runs tag, then build, and then publish.
EOF
}

WORKDIR=
IMGTAG=latest
JAVA=
RELEASE_STEP=
while getopts "d:hj:ns:t:" opt; do
case $opt in
d) WORKDIR="$OPTARG" ;;
n) DRY_RUN=1 ;;
t) IMGTAG="$OPTARG" ;;
j) JAVA="$OPTARG" ;;
s) RELEASE_STEP="$OPTARG" ;;
h) usage ;;
?) error "Invalid option. Run with -h for help." ;;
esac
done

if [ -z "$WORKDIR" ] || [ ! -d "$WORKDIR" ]; then
error "Work directory (-d) must be defined and exist. Run with -h for help."
fi

if [ -d "$WORKDIR/output" ]; then
read -r -p "Output directory already exists. Overwrite and continue? [y/n] " ANSWER
if [ "$ANSWER" != "y" ]; then
error "Exiting."
fi
fi

cd "$WORKDIR"
rm -rf "$WORKDIR/output"
mkdir "$WORKDIR/output"

get_release_info

# Place all RM scripts and necessary data in a local directory that must be defined in the command
# line. This directory is mounted into the image. Its WORKDIR, the arg passed with -d.
for f in "$SELF"/*; do
if [ -f "$f" ]; then
cp "$f" "$WORKDIR"
fi
done

GPG_KEY_FILE="$WORKDIR/gpg.key"
fcreate_secure "$GPG_KEY_FILE"
"$GPG" --passphrase "$GPG_PASSPHRASE" --export-secret-key --armor "$GPG_KEY" > "$GPG_KEY_FILE"

run_silent "Building hbase-rm image with tag $IMGTAG..." "docker-build.log" \
docker build -t "hbase-rm:$IMGTAG" --build-arg UID=$UID "$SELF/hbase-rm"

# Write the release information to a file with environment variables to be used when running the
# image.
ENVFILE="$WORKDIR/env.list"
fcreate_secure "$ENVFILE"

function cleanup {
rm -f "$ENVFILE"
rm -f "$GPG_KEY_FILE"
}

trap cleanup EXIT

cat > "$ENVFILE" <<EOF
DRY_RUN=$DRY_RUN
SKIP_TAG=$SKIP_TAG
RUNNING_IN_DOCKER=1
GIT_BRANCH=$GIT_BRANCH
NEXT_VERSION=$NEXT_VERSION
RELEASE_VERSION=$RELEASE_VERSION
RELEASE_TAG=$RELEASE_TAG
GIT_REF=$GIT_REF
PACKAGE_VERSION=$PACKAGE_VERSION
ASF_USERNAME=$ASF_USERNAME
GIT_NAME=$GIT_NAME
GIT_EMAIL=$GIT_EMAIL
GPG_KEY=$GPG_KEY
ASF_PASSWORD=$ASF_PASSWORD
GPG_PASSPHRASE=$GPG_PASSPHRASE
RELEASE_STEP=$RELEASE_STEP
RELEASE_STEP=$RELEASE_STEP
API_DIFF_TAG=$API_DIFF_TAG
EOF

JAVA_VOL=
if [ -n "$JAVA" ]; then
echo "JAVA_HOME=/opt/hbase-java" >> "$ENVFILE"
JAVA_VOL="--volume $JAVA:/opt/hbase-java"
fi

echo "Building $RELEASE_TAG; output will be at $WORKDIR/output"
docker run -ti \
--env-file "$ENVFILE" \
--volume "$WORKDIR:/opt/hbase-rm" \
"$JAVA_VOL" \
"hbase-rm:$IMGTAG"
@@ -0,0 +1,85 @@
#!/usr/bin/env bash

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# Use the adjacent do-release-docker.sh instead, if you can.
# Otherwise, this runs core of the release creation.
# Will ask you questions on what to build and for logins
# and passwords to use building.
export PROJECT="${PROJECT:-hbase-operator-tools}"

SELF=$(cd $(dirname $0) && pwd)
. "$SELF/release-util.sh"

while getopts "bn" opt; do
case $opt in
b) GIT_BRANCH=$OPTARG ;;
n) DRY_RUN=1 ;;
?) error "Invalid option: $OPTARG" ;;
esac
done

# If running in docker, import and then cache keys.
if [ "$RUNNING_IN_DOCKER" = "1" ]; then
# Run gpg agent.
eval $(gpg-agent --disable-scdaemon --daemon --no-grab --allow-preset-passphrase --default-cache-ttl=86400 --max-cache-ttl=86400)
echo "GPG Version: `gpg --version`"
# Inside docker, need to import the GPG key stored in the current directory.
echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --import "$SELF/gpg.key"

# We may need to adjust the path since JAVA_HOME may be overridden by the driver script.
if [ -n "$JAVA_HOME" ]; then
export PATH="$JAVA_HOME/bin:$PATH"
else
# JAVA_HOME for the openjdk package.
export JAVA_HOME=/usr
fi
else
# Outside docker, need to ask for information about the release.
get_release_info
fi
export GPG_TTY=$(tty)

function should_build {
local WHAT=$1
[ -z "$RELEASE_STEP" ] || [ "$WHAT" = "$RELEASE_STEP" ]
}

if should_build "tag" && [ $SKIP_TAG = 0 ]; then
run_silent "Creating release tag $RELEASE_TAG..." "tag.log" \
"$SELF/release-tag.sh"
echo "It may take some time for the tag to be synchronized to github."
echo "Press enter when you've verified that the new tag ($RELEASE_TAG) is available."
read
else
echo "Skipping tag creation for $RELEASE_TAG."
fi

if should_build "build"; then
run_silent "Building ${PROJECT}..." "build.log" \
"$SELF/release-build.sh" build
else
echo "Skipping build step."
fi

if should_build "publish"; then
run_silent "Publishing release" "publish.log" \
"$SELF/release-build.sh" publish-release
else
echo "Skipping publish step."
fi
@@ -0,0 +1,47 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# Image for building HBase releases. Based on Ubuntu 16.04.
#
# Includes:
# * Java 8
FROM ubuntu:18.04

# These arguments are just for reuse and not really meant to be customized.
ARG APT_INSTALL="apt-get install --no-install-recommends -y"

# Install extra needed repos and refresh.
#
# This is all in a single "RUN" command so that if anything changes, "apt update" is run to fetch
# the most current package versions (instead of potentially using old versions cached by docker).
RUN apt-get clean && \
apt-get update && \
# Install openjdk 8.
$APT_INSTALL openjdk-8-jdk && \
update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java && \
# Install build / source control tools
$APT_INSTALL curl gnupg python-pip wget git maven subversion lsof \
libcurl4-openssl-dev libxml2-dev && \
pip install python-dateutil

WORKDIR /opt/hbase-rm/output

ARG UID
RUN useradd -m -s /bin/bash -p hbase-rm -u $UID hbase-rm
USER hbase-rm:hbase-rm

ENTRYPOINT [ "/opt/hbase-rm/do-release.sh" ]

0 comments on commit abc881a

Please sign in to comment.