Skip to content
Permalink
Browse files
Change how we build and sign release artifacts
These simple scripts automate creation and signing of the release
artifacts. They help guarantee that the source artifacts published match
the Git tag.

Updated the How To Release documentation to explain how to use the
scripts.

This closes #98

Reviewers: Denes Arvay, Bessenyei Balázs Donát
  • Loading branch information
mpercy committed Jan 29, 2017
1 parent a844914 commit 18453d3ef182e92785d9fb74ff4fbdc244a4bc88
Show file tree
Hide file tree
Showing 5 changed files with 231 additions and 34 deletions.
@@ -229,60 +229,70 @@ If an rc2, rc3 etc is needed, simply create a new rc tag:
git push origin release-X.Y.Z-rc2


### Performing sanity check
### Preparing to sign the artifacts

All artifacts must be signed and checksummed. In order to sign a release
you will need a PGP key. You should get your key signed by a few other
people. You will also need to recv their keys from a public key server.
See the [Apache release signing](https://www.apache.org/dev/release-signing)
page for more details.

1\. Add your key to the
[KEYS](https://dist.apache.org/repos/dist/release/flume/KEYS) file:


(gpg --list-sigs <your-email> && gpg --armor --export <your-email>) >> KEYS


And commit the changes.

1\. Check out the candidate tag

### Generating and signing the source artifacts

git checkout release-X.Y.Z-rc1
There is a script in the Flume source tree for generating and signing the Flume
source artifacts. Once the release candidate is tagged, generate the source
release with the following steps.

1\. From the top of the Flume source tree, create a directory for the artifacts
and then generate them:

2\. Generate a tarball

mkdir ./source-artifacts
./dev-support/generate-source-release.sh X.Y.Z release-X.Y.Z-rc1 ./source-artifacts/

mvn clean install -DskipTests

The artifacts will be placed in the directory you specify (in this case,
`./source-artifacts`)

3\. Unpack the source tarball

### Testing the source tarball

1\. Unpack the source tarball


cd flume-ng-dist/target
rm -rf ./apache-flume-X.Y.Z-src/
tar xzvf apache-flume-X.Y.Z-src.tar.gz


4\. Do another full build inside the source tarball. This time, allow all
unit tests & integration tests to run and also include the docs
2\. Do a full build inside the source tarball. Allow all unit tests &
integration tests to run and also include the docs.


cd apache-flume-X.Y.Z-src
export LC_ALL=C.UTF-8 # Required to build the javadocs on some platforms and in some locales
mvn clean install -Psite -DskipTests


5\. Verify that the HTML docs that should have been generated inside the
3\. Verify that the HTML docs that should have been generated inside the
binary artifact under /docs are there and do not have rendering errors.

### Signatures and Checksums

All artifacts must be signed and checksummed. In order to sign a release
you will need a PGP key. You should get your key signed by a few other
people. You will also need to recv their keys from a public key server.
See the [Apache release
signing](https://www.apache.org/dev/release-signing)
page for more details.
### Generating, signing, and deploying the binary artifacts

1\. Add your key to the
[KEYS](https://dist.apache.org/repos/dist/release/flume/KEYS)
file:
Maven is configured to generate, sign, and deploy the binary artifacts
automatically. Use the following steps to do that:


(gpg --list-sigs <your-email> && gpg --armor --export <your-email>) >> KEYS


And commit the changes.

2\. Create and sign the artifacts, including site docs. This pushes the
1\. Create and sign the artifacts, including site docs. This pushes the
signed artifacts to the ASF staging repository.

In order to do this, you will need a settings.xml file with your
@@ -301,14 +311,14 @@ placed in \~/.m2/settings.xml and might look something like this:
</settings>


Once your settings.xml file is correct, you run the following from the
flume root directory to generate and deploy the artifacts:
2\. Once your settings.xml file is correct, run the following from the
Flume source directory to generate and deploy the artifacts:


mvn clean deploy -Psite -Psign -DskipTests


This will sign, hash, and upload each artifact to Nexus.
This will sign, checksum, and upload each artifact to Nexus.

Note: the checksum files will not be mirrored; They should be downloaded
from the main apache dist site.
@@ -329,16 +339,17 @@ OK, and then click Close using "Apache Flume X.Y.Z" as the description
to allow others to see the repository. Note that the staging repository
will have a numeric id associated with it that will be used later

4\. Copy artifacts to people.apache.org
4\. Copy the source artifacts you built locally to people.apache.org

$ rsync -e ssh -av source-artifacts/apache-flume-X.Y.Z-src.tar.gz* people.apache.org:public_html/apache-flume-X-Y.Z-rcN/

Copy the apache-flume-X.Y.Z-{bin,src}.tar.gz{,.{asc,md5,sha1}} files to
people.apache.org.
5\. Copy the binary artifacts you deployed via Maven to people.apache.org

$ ssh people.apache.org
$ cd public_html
$ mkdir apache-flume-X.Y.Z-rcN
$ cd apache-flume-X.Y.Z-rcN
$ wget --no-check-certificate https://repository.apache.org/content/repositories/orgapacheflume-XXXX/org/apache/flume/flume-ng-dist/X.Y.Z/flume-ng-dist-X.Y.Z-{src,bin}.tar.gz{,.{asc,md5,sha1}}
$ wget --no-check-certificate https://repository.apache.org/content/repositories/orgapacheflume-XXXX/org/apache/flume/flume-ng-dist/X.Y.Z/flume-ng-dist-X.Y.Z-bin.tar.gz{,.{asc,md5,sha1}}
$ for file in flume-ng-dist-*; do mv $file $(echo $file | sed -e "s/flume-ng-dist/apache-flume/g");done


@@ -0,0 +1,51 @@
#!/bin/bash -e
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
################################################################################
# Script to generate a source release tarball.
# The contract of this script is that it outputs the location of the generated
# tarball to stdout upon successful completion.
################################################################################
DEV_SUPPORT=$(cd $(dirname $0); pwd)
source "$DEV_SUPPORT/includes.sh"

VERSION_NUMBER=$1
GIT_TAG=$2
OUTPUT_DIR=$3

if [[ -z "$VERSION_NUMBER" || -z "$GIT_TAG" || -z "$OUTPUT_DIR" ]]; then
echo "Usage: $0 VERSION_NUMBER GIT_TAG OUTPUT_DIR" 1>&2
echo "Example: $0 1.7.0 release-1.7.0-rc1 target" 1>&2
exit 1
fi

[ ! -d "$OUTPUT_DIR" ] && error "Output directory $OUTPUT_DIR does not exist."
ABS_OUTPUT_DIR=$(cd $OUTPUT_DIR; pwd)

EXT=tar.gz
ARTIFACT_NAME=apache-flume-${VERSION_NUMBER}-src
ARTIFACT_PATH=$ABS_OUTPUT_DIR/$ARTIFACT_NAME.$EXT

# Need to call git archive from the root of the tree.
cd $DEV_SUPPORT/..

echo git archive --prefix=$ARTIFACT_NAME/ --output=$ARTIFACT_PATH --format "$EXT" "$GIT_TAG" 1>&2
git archive --prefix=$ARTIFACT_NAME/ --output=$ARTIFACT_PATH --format "$EXT" "$GIT_TAG"

echo $ARTIFACT_PATH
exit 0
@@ -0,0 +1,49 @@
#!/bin/bash -e
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
################################################################################
# Script to generate and sign a source release.
################################################################################
DEV_SUPPORT=$(cd $(dirname $0); pwd)
source "$DEV_SUPPORT/includes.sh"

VERSION_NUMBER=$1
GIT_TAG=$2
OUTPUT_DIR=$3

if [[ -z "$VERSION_NUMBER" || -z "$GIT_TAG" || -z "$OUTPUT_DIR" ]]; then
echo "Usage: $0 VERSION_NUMBER GIT_TAG OUTPUT_DIR"
echo "Example: $0 1.7.0 release-1.7.0-rc1 target"
exit 1
fi

# Generate the source artifact.
echo "Creating source archive..."
CREATE_ARCHIVE=$DEV_SUPPORT/create-source-archive.sh
ARCHIVE_PATH=$($CREATE_ARCHIVE "$VERSION_NUMBER" "$GIT_TAG" "$OUTPUT_DIR")
[ $? != 0 ] && error "Failed to generate source archive. $CREATE_ARCHIVE returned $?"
[ ! -r $ARCHIVE_PATH ] && error "Failed to generate source archive. Unknown error."

# Sign and checksum the source artifact.
echo "Signing source artifact..."
SIGN_ARTIFACT=$DEV_SUPPORT/sign-checksum-artifact.sh
$SIGN_ARTIFACT "$ARCHIVE_PATH"

echo "Release artifacts generated in $OUTPUT_DIR"
exit 0

@@ -0,0 +1,40 @@
#!/bin/bash -e
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
################################################################################
# Utilities for dev-support scripts.
################################################################################

# Print an error message and exit.
error() {
echo $1 1>&2
exit 1
}

# Searches the PATH for each command name passed, and returns the path of the
# first one found.
find_in_path() {
for COMMAND in "$@"; do
FOUND=$(which $COMMAND)
if [ -n "$FOUND" ]; then
echo "$FOUND"
return
fi
done
error "Cannot find $1. Please install $1 to continue."
}
@@ -0,0 +1,46 @@
#!/bin/bash -e
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
################################################################################
# Sign and checksum release artifacts.
################################################################################
DEV_SUPPORT=$(cd $(dirname $0); pwd)
source "$DEV_SUPPORT/includes.sh"

usage() {
echo "Usage: $0 RELEASE_ARTIFACT" 1>&2
echo "Example: $0 ./apache-flume-1.7.0-src.tar.gz" 1>&2
exit 1
}

ARTIFACT=$1
if [ ! -r "$ARTIFACT" ]; then
echo "The artifact at $ARTIFACT does not exist or is not readable." 1>&2
usage
fi

# The tools we need.
GPG=$(find_in_path gpg)
MD5=$(find_in_path md5sum md5)
SHA1=$(find_in_path sha1sum shasum)

# Now sign and checksum the artifact.
set -x
$GPG --sign $ARTIFACT
$MD5 < $ARTIFACT > $ARTIFACT.md5
$SHA1 < $ARTIFACT > $ARTIFACT.sha1

0 comments on commit 18453d3

Please sign in to comment.