Skip to content

Commit

Permalink
DRILL-8175: Update Drill release script after 1.20 (apache#2503)
Browse files Browse the repository at this point in the history 
* Set DRILL_PID_DIR in Dockerfile to writable location for distributed mode.

Some users of the images built from this Dockerfile customise
them so that they launch Drill in distributed mode instead of
embedded mode.  This change saves them from having to set
DRILL_PID_DIR themselves in order to succeed.

* Update release script and instructions after the release of 1.20.

- Add support for specifying a build profile such as "hadoop-2".
- Update instuctions for the Drill web site.
- Update instructions for uploading RCs (no more home.apache)
- Some fixes.
  • Loading branch information
@jnturton
jnturton committed Jul 11, 2022
1 parent 52a3ba2 commit d7bb171
Show file tree
Hide file tree
Showing 3 changed files with 84 additions and 61 deletions.
1 change: 1 addition & 0 deletions Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ ENTRYPOINT $DRILL_HOME/bin/drill-embedded
ENV DRILL_HOME=/opt/drill
ENV DRILL_USER=drilluser
ENV DRILL_USER_HOME=/var/lib/drill
ENV DRILL_PID_DIR=$DRILL_USER_HOME
ENV DRILL_LOG_DIR=$DRILL_USER_HOME/log
ENV DATA_VOL=/data

Expand Down
73 changes: 42 additions & 31 deletions docs/dev/Release.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,9 @@
1. ### SVN
1. Install subversion client (see instructions on http://subversion.apache.org/packages.html#osx for
installing svn on different systems).
2. Check that svn works:
2. Check that svn works and obtain a working copy of dist/dev/drill:
```
svn co https://dist.apache.org/repos/dist/release/drill ~/src/release/drill-dist
svn co https://dist.apache.org/repos/dist/dev/drill ~/src/release/drill-dist-dev
```
You also need writable access to Apache SVN. (You need to be a PMC member for this).
2. ### GPG key:
Expand All @@ -58,7 +58,7 @@
gpg --allow-secret-key-import --import mygpgkey_sec.gpg
```
5. Have another committer signed your key (add to the trust chain).
Apache advises to do it at
Apache advises to do it at
[key signing parties](https://www.apache.org/dev/release-signing.html#key-signing-party).
6. Make sure the default key is the key generated using the Apache email.
7. Publish your public key to a public server (e.g. http://pgp.surfnet.nl or http://pgp.mit.edu)
Expand All @@ -77,7 +77,7 @@
3. Note that you can add more than one SSH key corresponding to multiple machines.
4. Enter your Apache password and submit the changes.
5. Verify that you can do an sftp to the Apache server by running the following: `sftp <username>@home.apache.org`.
4. ### Setup Maven
4. ### Set up Maven
1. Apache's Maven repository access is documented here:
http://www.apache.org/dev/publishing-maven-artifacts.html
http://www.apache.org/dev/publishing-maven-artifacts.html#dev-env.
Expand Down Expand Up @@ -130,16 +130,16 @@
```
9. Do the release preparation:
```
mvn -X release:prepare -Papache-release -DpushChanges=false -DskipTests -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests=true -Dmaven.javadoc.skip=false" -DreleaseVersion=1.17.0 -DdevelopmentVersion=1.18.0-SNAPSHOT -Dtag=drill-1.17.0
mvn -X release:prepare -Papache-release -DpushChanges=false -DskipTests -DreleaseVersion=1.17.0 -DdevelopmentVersion=1.18.0-SNAPSHOT -Dtag=drill-1.17.0 -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests=true -Dmaven.javadoc.skip=false [-Pfoo_profile]"
```
10. Make sure to change Drill version to the proper one.
11. Check that `target` folder contains the following files (with the correct version number):
```
apache-drill-1.17.0-src.tar.gz
apache-drill-1.17.0-src.tar.gz
apache-drill-1.17.0-src.tar.gz.asc
apache-drill-1.17.0-src.tar.gz.sha512
apache-drill-1.17.0-src.zip
apache-drill-1.17.0-src.zip.asc
apache-drill-1.17.0-src.zip.asc
apache-drill-1.17.0-src.zip.sha512
```
12. Verify signature, ensure that GPG key for Apache was used (see details at
Expand Down Expand Up @@ -167,29 +167,31 @@
```
If you want to additionally check resulting archives and jars, add `-Dmaven.deploy.skip=true` flag to avoid deploying jars to the Nexus repository:
```
mvn release:perform -DconnectionUrl=scm:git:git@github.com:vvysotskyi/drill.git -DskipTests -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests=true -DconnectionUrl=scm:git:git@github.com:vvysotskyi/drill.git -Dmaven.deploy.skip=true"
mvn release:perform -DconnectionUrl=scm:git:git@github.com:vvysotskyi/drill.git -DskipTests -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests=true -DconnectionUrl=scm:git:git@github.com:vvysotskyi/drill.git -Dmaven.deploy.skip=true [-Pfoo_profile]"
```
After checks are performed, run this command without the flag.
15. Deploy the release commit:
```
git checkout drill-1.17.0
mvn deploy -Papache-release -DskipTests -Dgpg.passphrase=${GPG_PASSPHRASE}
```
16. Copy release files to a local release staging directory:
16. Copy release files to your svn working copy of dist.apache.org/repos/dist/dev/drill:
```
cp ~/src/release/drill/target/target/checkout/apache-drill-1.17.0-src.tar.gz* ~/release/1.17.0-rc0/ && \
cp ~/src/release/drill/target/target/checkout/apache-drill-1.17.0.zip* ~/release/1.17.0-rc0/ \
cp ~/src/release/drill/target/checkout/distribution/target/apache-drill-1.17.0.tar.gz* ~/release/1.17.0-rc0/ \
cp ~/src/release/drill/target/target/checkout/apache-drill-1.17.0-src.tar.gz* ~/src/release/drill-dist-dev/1.17.0-rc0/ && \
cp ~/src/release/drill/target/target/checkout/apache-drill-1.17.0.zip* ~/src/release/drill-dist-dev/1.17.0-rc0/ \
cp ~/src/release/drill/target/checkout/distribution/target/apache-drill-1.17.0.tar.gz* ~/src/release/drill-dist-dev/1.17.0-rc0/ \
```
17. Check if the artifacts are signed properly:
```
./tools/release-scripts/checksum.sh ~/release/1.17.0-rc0/apache-drill-1.17.0-src.tar.gz
./tools/release-scripts/checksum.sh ~/release/1.17.0-rc0/apache-drill-1.17.0-src.zip
./tools/release-scripts/checksum.sh ~/release/1.17.0-rc0/apache-drill-1.17.0.tar.gz
./tools/release-scripts/checksum.sh ~/src/release/drill-dist-dev/1.17.0-rc0/apache-drill-1.17.0-src.tar.gz
./tools/release-scripts/checksum.sh ~/src/release/drill-dist-dev/1.17.0-rc0/apache-drill-1.17.0-src.zip
./tools/release-scripts/checksum.sh ~/src/release/drill-dist-dev/1.17.0-rc0/apache-drill-1.17.0.tar.gz
```
18. Copy release files to a directory on `home.apache.org` for voting:
18. Commit the release files and browse to https://dist.apache.org/repos/dist/dev/drill/ to check that they are accessible. This can only done by a PMC member:
```
scp ~/release/1.17.0-rc0/* <username>@home.apache.org:~/public_html/drill/releases/1.17.0/rc0
cd ~/src/release/drill-dist-dev
svn add 1.17.0-rc0
svn commit
```

4. ## Automated release process
Expand All @@ -198,9 +200,18 @@
tools/release-scripts/release.sh
```
The release script will push the maven artifacts to the Maven staging repo.

5. ## Multiple builds
Currently, releasing multiple builds is done by performing consecutive releases. E.g. to add
an Hadoop 2 build of Drill, loop back to the top of the instructions now and start again with
a build profile of 'hadoop-2' and a release version of '1.17.0-hadoop2'. Note that it is not
necessary to close the jar release in the Maven repo first since the artifacts from your next
release will cause no identifier collisions due to their different version suffix. This means
that a single Maven repo can hold the artifacts for both 1.17.0 and 1.17.0-hadoop2.

5. ## Publish release candidate and vote
1. Go to the [Apache Maven staging repo](https://repository.apache.org/) and close the new jar release.
This step is done in the Maven GUI. For detailed instructions on sonatype GUI please refer to
This step is done in the Maven GUI. For detailed instructions on sonatype GUI please refer to
https://central.sonatype.org/pages/releasing-the-deployment.html#locate-and-examine-your-staging-repository.
2. Start vote (vote should last at least 72 hours).

Expand Down Expand Up @@ -236,7 +247,7 @@
[1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820&version=12341087
[2] http://home.apache.org/~arina/drill/releases/1.12.0/rc0/
[2] http://home.apache.org/~arina/drill/releases/1.12.0-rc0/
[3] https://repository.apache.org/content/repositories/orgapachedrill-1043/
[4] https://github.com/arina-ielchiieva/drill/commits/drill-1.12.0
```
Expand Down Expand Up @@ -280,25 +291,25 @@
3x +1 (binding): Arina, Aman, Parth
5x +1 (non-binding): Vitalii, Holger, Prasad, Vova, Charles
5x +1 (non-binding): Vitalii, Holger, Prasad, Vova, Charles
No 0s or -1s.
I'll start process for pushing the release artifacts and send an announcement once propagated.
Kind regards
```
2. Add the release to the [dist.apache.org](https://dist.apache.org/repos/dist/release/drill/) and delete the old version, keeping two most recent.
2. Move the release files to the Drill release directory with an entirely remote `svn move` operation and delete the old version, keeping two most recent.
This can only done by a PMC member:
```
svn co https://dist.apache.org/repos/dist/release/drill ~/src/release/drill-dist
cd ~/src/release/drill-dist
mkdir drill-1.17.0
cp -r ~/release/1.17.0-rc0 drill-1.17.0
svn add drill-1.17.0
svn commit --message "Upload Apache Drill 1.17.0 release."
svn delete 1.15.0
svn commit --message "Deleting drill-1.15.0 to keep only last two versions"
svn move \
-m "Upload Apache Drill 1.17.0 release." \
https://dist.apache.org/repos/dist/dev/drill/drill-1.17.0-rc0 \
https://dist.apache.org/repos/dist/release/drill/drill-1.17.0
svn delete \
-m "Deleting drill-1.15.0 to keep only last two versions" \
https://dist.apache.org/repos/dist/release/drill/drill-1.15.0
```
3. Go to the [Apache Maven staging repo](https://repository.apache.org/) and promote the release to the production.
4. Create branch and tag for this release and update Drill version in master (if used automated scripts, tag will be like this `drill-1.11.0`. Branch should be named as `1.11.0`).
Expand All @@ -309,7 +320,7 @@
6. Post release:
1. "What's New" for the new release.
2. Update Apache JIRA and add release date for this release. Add a new release tag if not already there.
3. Update Drill Web site:
3. Update Drill Web site through its source repo, https://github.com/apache/drill-site.
1. Generate release notes for Drill: https://confluence.atlassian.com/jira/creating-release-notes-185729647.html
and create a MarkDown file for the release notes - post to the site the day of the release.
2. Write the blog post and push it out to the Apache Drill website so it can be referenced in the announcement.
Expand Down Expand Up @@ -346,7 +357,7 @@
```
6. Commit changes with the commit message `Publish JavaDocs for the Apache Drill 1.17.0`
5. Instructions how to build and deploy Web site may be found here:
https://github.com/apache/drill/blob/gh-pages/README.md
https://github.com/apache/drill-site
3. Post the announcement about new release on [Apache Drill Twitter](https://twitter.com/apachedrill]).
4. A PMC member needs to update the release date for new release here:
https://reporter.apache.org/addrelease.html?drill
Expand Down
71 changes: 41 additions & 30 deletions tools/release-scripts/release.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ function runCmd(){
echo " ----------------- $1 " >> ${DRILL_RELEASE_OUTFILE}
echo " ----------------- " >> ${DRILL_RELEASE_OUTFILE}
shift
echo "Will execute $@"
# run the command, send output to out file
"$@" >> ${DRILL_RELEASE_OUTFILE} 2>&1
if [ $? -ne 0 ]; then
Expand All @@ -42,11 +43,12 @@ function runCmd(){
}

function copyFiles(){
rm -rf ${LOCAL_RELEASE_STAGING_DIR}
mkdir -p ${LOCAL_RELEASE_STAGING_DIR}/${DRILL_RELEASE_VERSION}
cp ${DRILL_SRC}/target/apache-drill-${DRILL_RELEASE_VERSION}-src.tar.gz* ${LOCAL_RELEASE_STAGING_DIR}/${DRILL_RELEASE_VERSION}/ && \
cp ${DRILL_SRC}/target/apache-drill-${DRILL_RELEASE_VERSION}-src.zip* ${LOCAL_RELEASE_STAGING_DIR}/${DRILL_RELEASE_VERSION}/ && \
cp ${DRILL_SRC}/distribution/target/apache-drill-${DRILL_RELEASE_VERSION}.tar.gz* ${LOCAL_RELEASE_STAGING_DIR}/${DRILL_RELEASE_VERSION}/
target_dir=${APACHE_DIST_WORKING_COPY}/${DRILL_RELEASE_VERSION}-rc${RELEASE_ATTEMPT}
rm -rf $target_dir
mkdir -p $target_dir
cp ${DRILL_SRC}/target/apache-drill-${DRILL_RELEASE_VERSION}-src.tar.gz* $target_dir/ && \
cp ${DRILL_SRC}/target/apache-drill-${DRILL_RELEASE_VERSION}-src.zip* $target_dir/ && \
cp ${DRILL_SRC}/distribution/target/apache-drill-${DRILL_RELEASE_VERSION}.tar.gz* $target_dir/

}

Expand All @@ -69,12 +71,17 @@ function createDirectoryIfAbsent() {

function readInputAndSetup(){

read -p "JAVA_HOME of the JDK 8 to use for the release : " JAVA_HOME
export JAVA_HOME

read -p "Drill Working Directory : " WORK_DIR
createDirectoryIfAbsent "${WORK_DIR}"

read -p "Drill Release Version (eg. 1.4.0) : " DRILL_RELEASE_VERSION
read -p "Build profile (e.g. hadoop-2, blank for default) : " BUILD_PROFILE

read -p "Drill Release Version (e.g. 1.4.0, 1.20.0-hadoop2) : " DRILL_RELEASE_VERSION

read -p "Drill Development Version (eg. 1.5.0-SNAPSHOT) : " DRILL_DEV_VERSION
read -p "Drill Development Version (e.g. 1.5.0-SNAPSHOT) : " DRILL_DEV_VERSION

read -p "Release Commit SHA : " RELEASE_COMMIT_SHA

Expand All @@ -83,8 +90,7 @@ function readInputAndSetup(){

read -p "Staging (personal) repo : " MY_REPO

read -p "Local release staging directory : " LOCAL_RELEASE_STAGING_DIR
createDirectoryIfAbsent "${LOCAL_RELEASE_STAGING_DIR}"
read -p "Svn working copy of dist.apache.org/repos/dist/dev/drill : " APACHE_DIST_WORKING_COPY

read -p "Apache login : " APACHE_LOGIN

Expand All @@ -94,17 +100,20 @@ function readInputAndSetup(){

DRILL_RELEASE_OUTFILE="${DRILL_RELEASE_OUTDIR}/drill_release.out.txt"
DRILL_SRC=${WORK_DIR}/drill-release
[ -z "$BUILD_PROFILE" ] || BUILD_PROFILE="-P$BUILD_PROFILE"

echo ""
echo "-----------------"
echo "JAVA_HOME : " ${JAVA_HOME}
echo "Drill Working Directory : " ${WORK_DIR}
echo "Drill Src Directory : " ${DRILL_SRC}
echo "Build profile mvn arg: " ${BUILD_PROFILE}
echo "Drill Release Version : " ${DRILL_RELEASE_VERSION}
echo "Drill Development Version : " ${DRILL_DEV_VERSION}
echo "Release Commit SHA : " ${RELEASE_COMMIT_SHA}
echo "Write output to : " ${DRILL_RELEASE_OUTFILE}
echo "Staging (personal) repo : " ${MY_REPO}
echo "Local release staging dir : " ${LOCAL_RELEASE_STAGING_DIR}
echo "Svn working copy of dist.apache.org/repos/dist dir : " ${APACHE_DIST_WORKING_COPY}

touch ${DRILL_RELEASE_OUTFILE}
}
Expand Down Expand Up @@ -142,43 +151,45 @@ runCmd "Cloning the repo" cloneRepo
runCmd "Checking the build" mvn install -DskipTests

export MAVEN_OPTS=-Xmx2g
runCmd "Clearing release history" mvn release:clean -Papache-release -DpushChanges=false -DskipTests
runCmd "Clearing release history" mvn release:clean \
-Papache-release \
-DpushChanges=false \
-DskipTests

export MAVEN_OPTS='-Xmx4g -XX:MaxPermSize=512m'
runCmd "Preparing the release " mvn -X release:prepare -Papache-release -DpushChanges=false -DskipTests -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests=true -Dmaven.javadoc.skip=false" -DreleaseVersion=${DRILL_RELEASE_VERSION} -DdevelopmentVersion=${DRILL_DEV_VERSION} -Dtag=drill-${DRILL_RELEASE_VERSION}
runCmd "Preparing the release " mvn -X release:prepare \
-Papache-release \
-DpushChanges=false \
-DdevelopmentVersion=${DRILL_DEV_VERSION} \
-DreleaseVersion=${DRILL_RELEASE_VERSION} \
-Dtag=drill-${DRILL_RELEASE_VERSION} \
-Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests -Dmaven.javadoc.skip=false ${BUILD_PROFILE}"

runCmd "Pushing to private repo ${MY_REPO}" git push ${MY_REPO} drill-${DRILL_RELEASE_VERSION}

runCmd "Performing the release to ${MY_REPO}" mvn release:perform -DconnectionUrl=scm:git:${MY_REPO} -DskipTests -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests=true -DconnectionUrl=scm:git:${MY_REPO}"
runCmd "Performing the release to ${MY_REPO}" mvn release:perform \
-DconnectionUrl=scm:git:${MY_REPO} \
-DlocalCheckout=true \
-Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests ${BUILD_PROFILE}"

runCmd "Checking out release commit" git checkout drill-${DRILL_RELEASE_VERSION}

# Remove surrounding quotes
tempGPG_PASSPHRASE="${GPG_PASSPHRASE%\"}"
tempGPG_PASSPHRASE="${tempGPG_PASSPHRASE#\"}"
runCmd "Deploying ..." mvn deploy -Papache-release -DskipTests -Dgpg.passphrase="${tempGPG_PASSPHRASE}"

runCmd "Copying" copyFiles
runCmd "Deploying ..." mvn deploy \
-Papache-release \
-DskipTests \
-Dgpg.passphrase="${tempGPG_PASSPHRASE}"

runCmd "Verifying artifacts are signed correctly" ${CHKSMDIR}/checksum.sh ${DRILL_SRC}/distribution/target/apache-drill-${DRILL_RELEASE_VERSION}.tar.gz
pause

runCmd "Copy release files to home.apache.org" sftp ${APACHE_LOGIN}@home.apache.org <<EOF
mkdir public_html
cd public_html
mkdir drill
cd drill
mkdir releases
cd releases
mkdir ${DRILL_RELEASE_VERSION}
cd ${DRILL_RELEASE_VERSION}
mkdir rc${RELEASE_ATTEMPT}
cd rc${RELEASE_ATTEMPT}
put ${LOCAL_RELEASE_STAGING_DIR}/${DRILL_RELEASE_VERSION}/* .
bye
EOF
runCmd "Copying release files to your working copy of dist.apache.org/repos/dist/dev/drill" copyFiles

echo "Go to the Apache maven staging repo and close the new jar release"
echo "and go to ${APACHE_DIST_WORKING_COPY} and svn add/commit the new"
echo "release candidate after checking the pending changes there."
pause

echo "Start the vote \(good luck\)\n"

0 comments on commit d7bb171

Please sign in to comment.