Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JBDS-2741 publish to builds/staging/CI/${JOB_NAME}/${BUILD_ID}-B${BUILD_NUMBER} instead of builds/staging{,.previous}/${JOB_NAME} #63

Closed
wants to merge 3 commits into from
Closed

JBDS-2741 publish to builds/staging/CI/${JOB_NAME}/${BUILD_ID}-B${BUILD_NUMBER} instead of builds/staging{,.previous}/${JOB_NAME} #63

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
74ad31c
JBDS-2741 publish to builds/staging/CI//-B instead of builds/staging{…
nickboldt Sep 11, 2013
51f909d
JBDS-2741 check if jbosstools-cleanup.sh exists locally before fetchi…
nickboldt Sep 11, 2013
20361eb
add TODO marker for updating the examples; will also need to update j…
nickboldt Sep 11, 2013
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions publish/promote.sh
View file
Open in desktop
Expand Up @@ -17,6 +17,7 @@ SOURCE_ZIP=
tmpfile=`mktemp`
tmpdir=`mktemp -d`

# TODO: update examples to include both the old style of publishing - {JOB_NAME}/all/repo - and the new style - {JOB_NAME}/{BUILD_ID}/all/repo
if [[ $# -lt 1 ]]; then
echo "Usage : $0 [-DESTINATION destination] [-RELEASE_TYPE release_type] -BUILD_TYPE build_type -TARGET_PLATFORM target_platform -PROJECT_NAME project_name -TARGET_FOLDER target_folder -SOURCE_PATH source_path"
# push to http://download.jboss.org/jbosstools/updates/integration/kepler/core/base/4.1.0.Alpha1/
Expand Down
111 changes: 14 additions & 97 deletions publish/publish.sh
View file
Open in desktop
Expand Up @@ -187,7 +187,11 @@ if [[ ${JOB_NAME/devstudio} != ${JOB_NAME} ]]; then # devstudio build
fi
# TODO JBDS-2627: switch this to pulling GIT_REV instead of SVN_REV
fi
<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes
# get name of upstream project (eg., for devstudio.product_70 want jbosstools-build-sites.aggregate.site_41)
wget ${wgetParams} -O $tmpdir/upstreamProject.name.xml "http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/${JOB_NAME}/api/xml?xpath=%28//upstreamProject/name%29[1]"
UPSTREAM_JOB_NAME=`sed -e "s#<name>\(.\+\)</name>#\1#g" $tmpdir/upstreamProject.name.xml`
Expand Down Expand Up @@ -519,111 +523,22 @@ if [[ $ec == "0" ]] && [[ $fc == "0" ]]; then
date; rsync -arzq --protocol=28 --delete ${STAGINGDIR}/* $DESTINATION/builds/nightly/${JOBNAMEREDUX}/${BUILD_ID}-B${BUILD_NUMBER}/
fi
rm -f $tmpdir/latestBuild.html
#else
# COMMENTED OUT as this uses too much disk space
# if a release build, create a named dir
#if [[ ${RELEASE} == "Yes" ]]; then
# date; rsync -arzq --protocol=28 --delete ${STAGINGDIR}/* $DESTINATION/builds/staging/${JOB_NAME}-${ZIPSUFFIX}/
#fi
fi

# and create/replace a snapshot dir w/ static URL
date; rsync -arzq --protocol=28 --delete ${STAGINGDIR}/* $DESTINATION/builds/staging/${JOB_NAME}.next

# 1. To recursively purge contents of .../staging.previous/foobar/ folder:
# mkdir -p $tmpdir/foobar;
# rsync -aPrz --delete $tmpdir/foobar tools@filemgmt.jboss.org:/downloads_htdocs/tools/builds/staging.previous/
# 2. To then remove entire .../staging.previous/foobar/ folder:
# echo -e "rmdir foobar" | sftp tools@filemgmt.jboss.org:/downloads_htdocs/tools/builds/staging.previous/
# rmdir $tmpdir/foobar

# JBIDE-8667 move current to previous; move next to current
# JBDS-2741 - publish to a new unique build folder for each spin, then later use jbosstools-cleanup.sh to purge old builds
if [[ ${DESTINATION##*@*:*} == "" ]]; then # user@server, do remote op
# create folders if not already there (could be empty)
echo -e "mkdir ${JOB_NAME}" | sftp $DESTINATION/builds/staging/
echo -e "mkdir ${JOB_NAME}" | sftp $DESTINATION/builds/staging.previous/
#echo -e "mkdir ${JOB_NAME}.2" | sftp $DESTINATION/builds/staging.previous/

# IF using .2 folders, purge contents of /builds/staging.previous/${JOB_NAME}.2 and remove empty dir
# NOTE: comment out next section - should only purge one staging.previous/* folder
#mkdir -p $tmpdir/${JOB_NAME}.2
#rsync -arzq --delete --protocol=28 $tmpdir/${JOB_NAME}.2 $DESTINATION/builds/staging.previous/
#echo -e "rmdir ${JOB_NAME}.2" | sftp $DESTINATION/builds/staging.previous/
#rmdir $tmpdir/${JOB_NAME}.2

# OR, purge contents of /builds/staging.previous/${JOB_NAME} and remove empty dir
mkdir -p $tmpdir/${JOB_NAME}
rsync -arzq --protocol=28 --delete $tmpdir/${JOB_NAME} $DESTINATION/builds/staging.previous/
echo -e "rmdir ${JOB_NAME}" | sftp $DESTINATION/builds/staging.previous/
rmdir $tmpdir/${JOB_NAME}

# move contents of /builds/staging.previous/${JOB_NAME} into /builds/staging.previous/${JOB_NAME}.2
#echo -e "rename ${JOB_NAME} ${JOB_NAME}.2" | sftp $DESTINATION/builds/staging.previous/

# move contents of /builds/staging/${JOB_NAME} into /builds/staging.previous/${JOB_NAME}
echo -e "rename ${JOB_NAME} ../staging.previous/${JOB_NAME}" | sftp $DESTINATION/builds/staging/

# move contents of /builds/staging/${JOB_NAME}.next into /builds/staging/${JOB_NAME}
echo -e "rename ${JOB_NAME}.next ${JOB_NAME}" | sftp $DESTINATION/builds/staging/
echo -e "mkdir CI" | sftp $DESTINATION/builds/staging/
echo -e "mkdir ${JOB_NAME}" | sftp $DESTINATION/builds/staging/CI/
else # work locally
# create folders if not already there (could be empty)
mkdir -p $DESTINATION/builds/staging/${JOB_NAME}
mkdir -p $DESTINATION/builds/staging.previous/${JOB_NAME}
#mkdir -p $DESTINATION/builds/staging.previous/${JOB_NAME}.2

# purge contents of /builds/staging.previous/${JOB_NAME}.2 and remove empty dir
# NOTE: comment out next section - should only purge one staging.previous/* folder
#rm -fr $DESTINATION/builds/staging.previous/${JOB_NAME}.2/

# OR, purge contents of /builds/staging.previous/${JOB_NAME} and remove empty dir
rm -fr $DESTINATION/builds/staging.previous/${JOB_NAME}/

# move contents of /builds/staging.previous/${JOB_NAME} into /builds/staging.previous/${JOB_NAME}.2
#mv $DESTINATION/builds/staging.previous/${JOB_NAME} $DESTINATION/builds/staging.previous/${JOB_NAME}.2

# move contents of /builds/staging/${JOB_NAME} into /builds/staging.previous/${JOB_NAME}
mv $DESTINATION/builds/staging/${JOB_NAME} $DESTINATION/builds/staging.previous/${JOB_NAME}

# move contents of /builds/staging/${JOB_NAME}.next into /builds/staging/${JOB_NAME}
mv $DESTINATION/builds/staging/${JOB_NAME}.next $DESTINATION/builds/staging/${JOB_NAME}
mkdir -p $DESTINATION/builds/staging/CI/${JOB_NAME}/
fi

# generate 2 ${STAGINGDIR}/all/composite*.xml files which will point at:
# /builds/staging/${JOB_NAME}/all/repo/
# /builds/staging.previous/${JOB_NAME}/all/repo/
# /builds/staging.previous/${JOB_NAME}.2/all/repo/
now=$(date +%s000)
mkdir -p ${STAGINGDIR}/all
echo "<?xml version='1.0' encoding='UTF-8'?>
<?compositeMetadataRepository version='1.0.0'?>
<repository name='JBoss Tools Staging - ${JOB_NAME} Composite' type='org.eclipse.equinox.internal.p2.metadata.repository.CompositeMetadataRepository' version='1.0.0'>
" > ${STAGINGDIR}/all/compositeContent.xml
echo "<?xml version='1.0' encoding='UTF-8'?>
<?compositeArtifactRepository version='1.0.0'?>
<repository name='JBoss Tools Staging - ${JOB_NAME} Composite' type='org.eclipse.equinox.internal.p2.artifact.repository.CompositeArtifactRepository' version='1.0.0'> " > ${STAGINGDIR}/all/compositeArtifacts.xml
metadata="<properties size='2'><property name='p2.compressed' value='true'/><property name='p2.timestamp' value='"${now}"'/></properties>
<children size='2'>
<child location='../../../staging/${JOB_NAME}/all/repo/'/>
<child location='../../../staging.previous/${JOB_NAME}/all/repo/'/>
</children>
</repository>
"
echo $metadata >> ${STAGINGDIR}/all/compositeContent.xml
echo $metadata >> ${STAGINGDIR}/all/compositeArtifacts.xml
date; rsync -arzq --protocol=28 ${STAGINGDIR}/all/composite*.xml $DESTINATION/builds/staging/${JOB_NAME}/all/
date; rsync -arzq --protocol=28 --delete ${STAGINGDIR}/* $DESTINATION/builds/staging/CI/${JOB_NAME}/${BUILD_ID}-B${BUILD_NUMBER}/

# create a snapshot dir outside Hudson which is file:// accessible
mkdir -p $INTRNALDEST/builds/staging/${JOB_NAME}.next/
date; rsync -arzq --delete ${STAGINGDIR}/* $INTRNALDEST/builds/staging/${JOB_NAME}.next/

# cycle internal copy of ${JOB_NAME} in staging and staging.previous
mkdir -p $INTRNALDEST/builds/staging/${JOB_NAME}/
# purge contents of /builds/staging.previous/${JOB_NAME} and remove empty dir
rm -fr $INTRNALDEST/builds/staging.previous/${JOB_NAME}/
# move contents of /builds/staging/${JOB_NAME} into /builds/staging.previous/${JOB_NAME}
mv $INTRNALDEST/builds/staging/${JOB_NAME} $INTRNALDEST/builds/staging.previous/${JOB_NAME}
# move contents of /builds/staging/${JOB_NAME}.next into /builds/staging/${JOB_NAME}
mv $INTRNALDEST/builds/staging/${JOB_NAME}.next $INTRNALDEST/builds/staging/${JOB_NAME}
mkdir -p $INTRNALDEST/builds/staging/CI/${JOB_NAME}/
date; rsync -arzq --delete ${STAGINGDIR}/* $INTRNALDEST/builds/staging/CI/${JOB_NAME}/${BUILD_ID}-B${BUILD_NUMBER}/
fi

# extra publish step for aggregate update sites ONLY
Expand Down Expand Up @@ -660,7 +575,9 @@ rm -fr $tmpdir

if [[ ${JOB_NAME/.aggregate} != ${JOB_NAME} ]]; then
# regenerate http://download.jboss.org/jbosstools/builds/nightly/*/*/composite*.xml files for up to 5 builds, cleaning anything older than 5 days old
wget -q --no-check-certificate -N https://raw.github.com/jbosstools/jbosstools-build-ci/master/util/cleanup/jbosstools-cleanup.sh
if [[ ! -f jbosstools-cleanup.sh ]]; then
wget -q --no-check-certificate -N https://raw.github.com/jbosstools/jbosstools-build-ci/master/util/cleanup/jbosstools-cleanup.sh
fi
chmod +x jbosstools-cleanup.sh
./jbosstools-cleanup.sh --keep 5 --age-to-delete 5 --childFolderSuffix /all/repo/
rm -f jbosstools-cleanup.sh
Expand Down
2 changes: 1 addition & 1 deletion util/cleanup/jbosstools-cleanup.sh
View file
Open in desktop
Expand Up @@ -19,7 +19,7 @@ echo "" | tee -a $log
#defauls
numbuildstokeep=1000 # keep X builds per branch
threshholdwhendelete=365 # purge builds more than X days old
dirsToScan="builds/nightly/core builds/nightly/coretests builds/nightly/soa-tooling builds/nightly/soatests builds/nightly/webtools builds/nightly/integrationtests"
dirsToScan="builds/staging/CI builds/nightly/core builds/nightly/coretests builds/nightly/soa-tooling builds/nightly/soatests builds/nightly/webtools builds/nightly/integrationtests"
delete=1 # if 1, files will be deleted. if 0, files will be listed for delete but not actually removed
checkTimeStamps=1 # if 1, check for timestamped folders, eg., 2012-09-30_04-01-36-H5622 and deduce the age from name. if 0, skip name-to-age parsing and delete nothing
childFolderSuffix="/" # for component update sites, set to "/"; for aggregate builds (not update sites) use "/all/repo/"
Expand Down