Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Browse files

Script to generate and distribute tarballs with one command. See comm…

…it msg for usage.

This is a script that directs the generation of tarballs on a remote server
and the remote server also hosts it on a web server.

You need to have fab installed locally (not remotely).  The remote build
directory need to be prepopulated with some checkout of the Nimbus git
repository (not the most recent one, just something).

It will pull to the latest in a tracked branch but you can provide a different
refspec.

The result will print, like:

http://www.nimbusproject.org/downloads/2010-07-08__20-24__4004a4d4-d45a-4547-81ff-73e4edd69b74/nimbus-2.5RC1.9-src.tar.gz
http://www.nimbusproject.org/downloads/2010-07-08__20-24__4004a4d4-d45a-4547-81ff-73e4edd69b74/nimbus-controls-2.5RC1.9.tar.gz

Everything besides the git <refspec> is defined at the top of the script.

Or you can override those setting with the environment variables described.

For example, I have a local, non-commited script ~/bin/nimbus-tarball.sh that
is invoked like:

  nimbus-tarball.sh HEAD

The contents of that script make up the rest of this commit message:

export MAKE_DIST_REMOTE_BUILDDIR="/scratch/nimbus"
export MAKE_DIST_REMOTE_WEBDIR="/mcs/ee.mcs.anl.gov/nimbus/downloads"
export MAKE_DIST_REMOTE_USERNAME="tfreeman"
export MAKE_DIST_REMOTE_HOSTNAME="login3.mcs.anl.gov"
export MAKE_DIST_SSHKEY="/home/tim/.ssh/id_rsa_xyz"
export MAKE_DIST_PRINT_URL_BASE="http://www.nimbusproject.org/downloads/"

/home/tim/code/nimbus/scripts/make-dist-remote.sh $@
  • Loading branch information...
commit 6abab3d4cbf4d6c8692d380439da493449dba10c 1 parent 02079d3
@timf timf authored
Showing with 141 additions and 0 deletions.
  1. +78 −0 scripts/lib/web_ball.py
  2. +63 −0 scripts/make-dist-remote.sh
View
78 scripts/lib/web_ball.py
@@ -0,0 +1,78 @@
+from fabric.api import cd,run
+from datetime import datetime
+import os
+import uuid
+
+def newball(builddir=None, webdir=None, printbase=None, gitref=None):
+
+ if not builddir:
+ raise Exception("builddir required")
+ if not webdir:
+ raise Exception("webdir required")
+ if not printbase:
+ raise Exception("printbase required")
+ if not gitref:
+ raise Exception("gitref required")
+
+ _checknode(builddir)
+ _pull_and_reset(builddir, gitref)
+ result_dir = _make_dist(builddir)
+ filelist = _make_available(result_dir, webdir)
+
+ print "\n\n"
+ for f in filelist:
+ print "%s%s/%s" % (printbase, os.path.basename(result_dir), f)
+ print "\n\n"
+
+
+# ---------------------------------------------------------------------
+
+def _checknode(builddir):
+ print "\n* Remote node status"
+ with cd(builddir):
+ run("git log | head -n 6")
+
+def _pull_and_reset(builddir, gitref):
+ print "\n* Resetting git against '%s'" % gitref
+ with cd(builddir):
+
+ # first make sure nothing is tainting it, these two commands will
+ # return non-zero if there is something uncommitted in the directory
+ # for some weird reason
+ run("git diff --quiet --exit-code")
+ run("git diff --quiet --exit-code --cached")
+
+ # use pull ant not just fetch so that HEAD becomes relevant
+ run("git pull")
+ run("git reset --hard %s" % gitref)
+
+def _make_dist(builddir):
+ print "\n* Running make-dist"
+
+ # make a unique directory for all of the tarballs, easier to reason
+ # about the result this way
+ now = datetime.now()
+ basename = "%.4d-%.2d-%.2d__%.2d-%.2d__%s" % (now.year, now.month, now.day, now.hour, now.minute, str(uuid.uuid4()))
+ result_dir = os.path.join("/tmp/", basename)
+
+ with cd(builddir):
+ run("mkdir %s" % result_dir)
+ run("./scripts/make-dist.sh %s %s" % (result_dir, builddir))
+
+ return result_dir
+
+def _make_available(result_dir, webdir):
+
+ target_dir = os.path.join(webdir, os.path.basename(result_dir))
+
+ # fail if target directory exists:
+ run("[ ! -e %s ] || /bin/false" % target_dir)
+
+ run("mv %s %s" % (result_dir, webdir))
+
+ with cd(target_dir):
+ files = run("ls")
+ filelist = files.split("\n")
+
+ return filelist
+
View
63 scripts/make-dist-remote.sh
@@ -0,0 +1,63 @@
+#!/bin/bash
+
+# You can call this with the following environment variables to override,
+# make a script elsewhere to wrap this so you don't always need to edit:
+
+# MAKE_DIST_REMOTE_BUILDDIR
+# MAKE_DIST_REMOTE_WEBDIR
+# MAKE_DIST_REMOTE_USERNAME
+# MAKE_DIST_REMOTE_HOSTNAME
+# MAKE_DIST_SSHKEY
+# MAKE_DIST_PRINT_URL_BASE
+
+
+REMOTE_BUILDDIR=${MAKE_DIST_REMOTE_BUILDDIR-/scratch/nimbus}
+REMOTE_WEBDIR=${MAKE_DIST_REMOTE_WEBDIR-/mcs/ee.mcs.anl.gov/nimbus/downloads}
+USERNAME=${MAKE_DIST_REMOTE_USERNAME-XXX}
+HOSTNAME=${MAKE_DIST_REMOTE_HOSTNAME-login.mcs.anl.gov}
+SSHKEY=${MAKE_DIST_SSHKEY-$HOME/.ssh/id_rsa}
+PRINT_URL_BASE=${MAKE_DIST_PRINT_URL_BASE-http://www.nimbusproject.org/downloads/}
+
+EXPLANATION="
+
+This is a script that directs the generation of tarballs on a remote server
+and the remote server also hosts it on a web server.
+
+You need to have fab installed locally (not remotely). The remote build
+directory need to be prepopulated with some checkout of the Nimbus git
+repository (not the most recent one, just something).
+
+It will pull to the latest in a tracked branch but you can provide a different
+refspec.
+
+The result will print, like:
+
+http://www.nimbusproject.org/downloads/2010-07-08__20-24__4004a4d4-d45a-4547-81ff-73e4edd69b74/nimbus-2.5RC1.9-src.tar.gz
+http://www.nimbusproject.org/downloads/2010-07-08__20-24__4004a4d4-d45a-4547-81ff-73e4edd69b74/nimbus-controls-2.5RC1.9.tar.gz
+
+
+Everything besides the git <refspec> is defined at the top of the script.
+
+Or you can override those setting with the environment variables described.
+"
+
+if ([ "X$1" == "X--help" ] || [ "X$1" == "X-h" ]); then
+ echo "Usage:"
+ echo "$0 <refspec>"
+ echo "$EXPLANATION"
+ exit 0
+fi
+
+if [ $# -ne 1 ]; then
+ echo "Usage:"
+ echo "$0 <refspec>"
+ echo "$EXPLANATION"
+ exit 1
+fi
+
+NH_REL="`dirname $0`/.."
+NH=`cd $NH_REL; pwd`
+FAB_SCRIPT=$NH/scripts/lib/web_ball.py
+
+fab -f $FAB_SCRIPT -i $SSHKEY -u $USERNAME -H $HOSTNAME newball:builddir=$REMOTE_BUILDDIR,webdir=$REMOTE_WEBDIR,printbase=$PRINT_URL_BASE,gitref=$1
+
Please sign in to comment.
Something went wrong with that request. Please try again.