Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Web build #19

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
.*un~
home/.*
home/*.log
home/cache/
home/fingerprints/
home/identity.key.enc
home/init.groovy.d/
Expand Down
14 changes: 14 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,25 @@ slave:
- JAVA_HOME=/etc/alternatives/java_sdk
- SERVICE_NAME=SPACENAME-omero
- DOCKER_FIX=' '
command: -labels java -labels slave -name slave -executors 4

web:
build: web
environment:
- JENKINS_LABEL=/etc/alternatives/java_sdk
- JENKINS_MODE=exclusive
- SERVICE_NAME=SPACENAME-omero
links:
- jenkins
volumes:
- ./web:/home/omero
command: -labels web -name web -executors 4
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need newer swarm plugin jenkinsci/swarm-plugin#33, otherwise this is never going to work because of jenkinsci/swarm-plugin#26

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was using the label here to tie jobs to the correct location rather than the name. But as long as the newer plugin works with the rest of the stack, 👍

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Newer plugin is needed to disableClientsUniqueId otherwise jenkins adds unique id to the name


nginx:
image: nginx
links:
- slave
- web
volumes:
- ./slave:/home/jenkins-slave
environment:
Expand Down
2 changes: 1 addition & 1 deletion home/hudson.plugins.git.GitSCM.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@
<globalConfigName></globalConfigName>
<globalConfigEmail></globalConfigEmail>
<createAccountBasedOnEmail>false</createAccountBasedOnEmail>
</hudson.plugins.git.GitSCM_-DescriptorImpl>
</hudson.plugins.git.GitSCM_-DescriptorImpl>
2 changes: 1 addition & 1 deletion home/hudson.plugins.git.GitTool.xml
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@
<properties/>
</hudson.plugins.git.GitTool>
</installations>
</hudson.plugins.git.GitTool_-DescriptorImpl>
</hudson.plugins.git.GitTool_-DescriptorImpl>
4 changes: 3 additions & 1 deletion home/hudson.tasks.Shell.xml
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
<?xml version='1.0' encoding='UTF-8'?>
<hudson.tasks.Shell_-DescriptorImpl/>
<hudson.tasks.Shell_-DescriptorImpl>
<shell>/bin/bash</shell>
</hudson.tasks.Shell_-DescriptorImpl>
23 changes: 21 additions & 2 deletions home/jobs/BIOFORMATS-maven/config.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,25 @@
<actions/>
<description></description>
<keepDependencies>false</keepDependencies>
<properties/>
<properties>
<jenkins.plugins.slack.SlackNotifier_-SlackJobProperty plugin="slack@1.8.1">
<teamDomain></teamDomain>
<token></token>
<room></room>
<startNotification>false</startNotification>
<notifySuccess>false</notifySuccess>
<notifyAborted>false</notifyAborted>
<notifyNotBuilt>false</notifyNotBuilt>
<notifyUnstable>false</notifyUnstable>
<notifyFailure>false</notifyFailure>
<notifyBackToNormal>false</notifyBackToNormal>
<notifyRepeatedFailure>false</notifyRepeatedFailure>
<includeTestSummary>false</includeTestSummary>
<showCommitList>false</showCommitList>
<includeCustomMessage>false</includeCustomMessage>
<customMessage></customMessage>
</jenkins.plugins.slack.SlackNotifier_-SlackJobProperty>
</properties>
<scm class="hudson.plugins.git.GitSCM" plugin="git@2.4.0">
<configVersion>2</configVersion>
<userRemoteConfigs>
Expand All @@ -20,7 +38,8 @@
<submoduleCfg class="list"/>
<extensions/>
</scm>
<canRoam>true</canRoam>
<assignedNode>slave</assignedNode>
<canRoam>false</canRoam>
<disabled>false</disabled>
<blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
<blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
Expand Down
22 changes: 20 additions & 2 deletions home/jobs/BIOFORMATS-push/config.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,23 @@
<description></description>
<keepDependencies>false</keepDependencies>
<properties>
<jenkins.plugins.slack.SlackNotifier_-SlackJobProperty plugin="slack@1.8.1">
<teamDomain></teamDomain>
<token></token>
<room></room>
<startNotification>false</startNotification>
<notifySuccess>false</notifySuccess>
<notifyAborted>false</notifyAborted>
<notifyNotBuilt>false</notifyNotBuilt>
<notifyUnstable>false</notifyUnstable>
<notifyFailure>false</notifyFailure>
<notifyBackToNormal>false</notifyBackToNormal>
<notifyRepeatedFailure>false</notifyRepeatedFailure>
<includeTestSummary>false</includeTestSummary>
<showCommitList>false</showCommitList>
<includeCustomMessage>false</includeCustomMessage>
<customMessage></customMessage>
</jenkins.plugins.slack.SlackNotifier_-SlackJobProperty>
<hudson.model.ParametersDefinitionProperty>
<parameterDefinitions>
<hudson.model.StringParameterDefinition>
Expand All @@ -16,9 +33,9 @@
<description></description>
<choices class="java.util.Arrays$ArrayList">
<a class="string-array">
<string>none</string>
<string>success-only</string>
<string>no-error</string>
<string>none</string>
</a>
</choices>
</hudson.model.ChoiceParameterDefinition>
Expand Down Expand Up @@ -57,7 +74,8 @@
</hudson.plugins.git.extensions.impl.CloneOption>
</extensions>
</scm>
<canRoam>true</canRoam>
<assignedNode>slave</assignedNode>
<canRoam>false</canRoam>
<disabled>false</disabled>
<blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
<blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
Expand Down
4 changes: 3 additions & 1 deletion home/jobs/OMERO-build/config.xml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@
<hudson.plugins.git.extensions.impl.CleanCheckout/>
</extensions>
</scm>
<canRoam>true</canRoam>
<assignedNode>slave</assignedNode>
<canRoam>false</canRoam>
<disabled>false</disabled>
<blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
<blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
Expand All @@ -54,6 +55,7 @@ source docs/hudson/OMERO.sh
<onlyIfSuccessful>false</onlyIfSuccessful>
<fingerprint>false</fingerprint>
<defaultExcludes>true</defaultExcludes>
<caseSensitive>true</caseSensitive>
</hudson.tasks.ArtifactArchiver>
</publishers>
<buildWrappers/>
Expand Down
22 changes: 20 additions & 2 deletions home/jobs/OMERO-push/config.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,23 @@
<description></description>
<keepDependencies>false</keepDependencies>
<properties>
<jenkins.plugins.slack.SlackNotifier_-SlackJobProperty plugin="slack@1.8.1">
<teamDomain></teamDomain>
<token></token>
<room></room>
<startNotification>false</startNotification>
<notifySuccess>false</notifySuccess>
<notifyAborted>false</notifyAborted>
<notifyNotBuilt>false</notifyNotBuilt>
<notifyUnstable>false</notifyUnstable>
<notifyFailure>false</notifyFailure>
<notifyBackToNormal>false</notifyBackToNormal>
<notifyRepeatedFailure>false</notifyRepeatedFailure>
<includeTestSummary>false</includeTestSummary>
<showCommitList>false</showCommitList>
<includeCustomMessage>false</includeCustomMessage>
<customMessage></customMessage>
</jenkins.plugins.slack.SlackNotifier_-SlackJobProperty>
<hudson.model.ParametersDefinitionProperty>
<parameterDefinitions>
<hudson.model.StringParameterDefinition>
Expand Down Expand Up @@ -53,7 +70,8 @@
</hudson.plugins.git.extensions.impl.RelativeTargetDirectory>
</extensions>
</scm>
<canRoam>true</canRoam>
<assignedNode>slave</assignedNode>
<canRoam>false</canRoam>
<disabled>false</disabled>
<blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
<blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
Expand All @@ -68,4 +86,4 @@ $HOME/.local/bin/scc $MERGE_COMMAND -S $STATUS --push $PUSH_BRANCH</command>
</builders>
<publishers/>
<buildWrappers/>
</project>
</project>
3 changes: 2 additions & 1 deletion home/jobs/OMERO-start/config.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@
<keepDependencies>false</keepDependencies>
<properties/>
<scm class="hudson.scm.NullSCM"/>
<canRoam>true</canRoam>
<assignedNode>slave</assignedNode>
<canRoam>false</canRoam>
<disabled>false</disabled>
<blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
<blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
Expand Down
91 changes: 91 additions & 0 deletions home/jobs/OMERO-web/config.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
<?xml version='1.0' encoding='UTF-8'?>
<project>
<actions/>
<description></description>
<keepDependencies>false</keepDependencies>
<properties>
<jenkins.plugins.slack.SlackNotifier_-SlackJobProperty plugin="slack@1.8.1">
<teamDomain></teamDomain>
<token></token>
<room></room>
<startNotification>true</startNotification>
<notifySuccess>true</notifySuccess>
<notifyAborted>true</notifyAborted>
<notifyNotBuilt>true</notifyNotBuilt>
<notifyUnstable>true</notifyUnstable>
<notifyFailure>true</notifyFailure>
<notifyBackToNormal>true</notifyBackToNormal>
<notifyRepeatedFailure>false</notifyRepeatedFailure>
<includeTestSummary>false</includeTestSummary>
<showCommitList>false</showCommitList>
<includeCustomMessage>false</includeCustomMessage>
<customMessage></customMessage>
</jenkins.plugins.slack.SlackNotifier_-SlackJobProperty>
</properties>
<scm class="hudson.scm.NullSCM"/>
<assignedNode>web</assignedNode>
<canRoam>false</canRoam>
<disabled>false</disabled>
<blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
<blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
<triggers/>
<concurrentBuild>false</concurrentBuild>
<builders>
<hudson.tasks.Shell>
<command>OMERO_DIST=$HOME/OMERO.server
if [ -e $OMERO_DIST ]; then

if [ -e $WORKSPACE/omeropy-virtualenv ]; then
source $WORKSPACE/omeropy-virtualenv/bin/activate
eval $(bash /opt/multi-config.sh ice3.5)
$OMERO_DIST/bin/omero web stop
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is one thing that is still not clear to me how that meant to work. I thought web docker container is created every time jenkins job is run, that would result in job just passing appropriate OMERO.py artifact to the docker build. This will reduce maintenance of scripts

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not yet. At the moment, the docker-compose creates a single web container which is re-used for multiple runs. We'll need to look into new jenkins plugins in order to enable the "spin up a fresh container each time" workflow. Definitely for it, but I think we should make the migration to omero-install atomically and then look at the bigger step to a many-container system.

sleep 5
$OMERO_DIST/bin/omero web status
deactivate
rm -rf $WORKSPACE/omeropy-virtualenv
fi
rm -rf $OMERO_DIST
fi

</command>
</hudson.tasks.Shell>
<hudson.plugins.copyartifact.CopyArtifact plugin="copyartifact@1.37">
<project>OMERO-build</project>
<filter>src/target/OMERO.py*zip</filter>
<target></target>
<excludes></excludes>
<selector class="hudson.plugins.copyartifact.StatusBuildSelector"/>
<flatten>true</flatten>
<doNotFingerprintArtifacts>false</doNotFingerprintArtifacts>
</hudson.plugins.copyartifact.CopyArtifact>
<hudson.tasks.Shell>
<command>OMERO_DIST=$HOME/OMERO.server
OMERO_INSTALL=/tmp/omero-install/linux

ZIP=$(ls OMERO.py*.zip)
DIST=${ZIP%.zip}
unzip $ZIP
rm -f $ZIP
mv $WORKSPACE/$DIST $OMERO_DIST

virtualenv $WORKSPACE/omeropy-virtualenv --system-site-packages
source $WORKSPACE/omeropy-virtualenv/bin/activate

cd $OMERO_INSTALL
source settings.env

# From step04
$OMERO_DIST/bin/omero web config nginx --http &quot;$OMERO_WEB_PORT&quot; &gt; $OMERO_DIST/nginx.conf.tmp

sudo -u root bash step05_centos7_nginx.sh
sudo -u root bash step06_centos7_daemon.sh

eval $(bash /opt/multi-config.sh ice3.5)
BUILD_ID=DONT_KILL_ME $OMERO_DIST/bin/omero web start

$OMERO_DIST/bin/omero admin diagnostics</command>
</hudson.tasks.Shell>
</builders>
<publishers/>
<buildWrappers/>
</project>
2 changes: 1 addition & 1 deletion home/org.jenkinsci.plugins.gitclient.JGitTool.xml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<?xml version='1.0' encoding='UTF-8'?>
<org.jenkinsci.plugins.gitclient.JGitTool_-DescriptorImpl plugin="git-client@1.19.0">
<installations class="hudson.plugins.git.GitTool-array"/>
</org.jenkinsci.plugins.gitclient.JGitTool_-DescriptorImpl>
</org.jenkinsci.plugins.gitclient.JGitTool_-DescriptorImpl>
16 changes: 16 additions & 0 deletions ubuntu1204/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
FROM ubuntu:12.04

MAINTAINER OME

ENV JENKINS_SWARM_VERSION 1.24
ENV LANG en_US.UTF-8

RUN adduser -u 11521 slave
RUN apt-get update && apt-get install -y curl

USER slave
WORKDIR /tmp
RUN curl --create-dirs -sSLo /tmp/swarm-client-$JENKINS_SWARM_VERSION-jar-with-dependencies.jar http://maven.jenkins-ci.org/content/repositories/releases/org/jenkins-ci/plugins/swarm-client/$JENKINS_SWARM_VERSION/swarm-client-$JENKINS_SWARM_VERSION-jar-with-dependencies.jar

COPY jenkins-slave.sh /tmp/jenkins-slave.sh
ENTRYPOINT ["/tmp/jenkins-slave.sh"]
21 changes: 21 additions & 0 deletions ubuntu1204/jenkins-slave.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/bin/bash
# https://raw.githubusercontent.com/carlossg/jenkins-swarm-slave-docker/master/jenkins-slave.sh

# if `docker run` first argument start with `-` the user is passing jenkins swarm launcher arguments
if [[ $# -lt 1 ]] || [[ "$1" == "-"* ]]; then

# jenkins swarm slave
JAR=`ls -1 /tmp/swarm-client-*.jar | tail -n 1`

# if -master is not provided and using --link jenkins:jenkins
if [[ "$@" != *"-master "* ]] && [ ! -z "$JENKINS_PORT_8080_TCP_ADDR" ]; then
PARAMS="-master http://$JENKINS_PORT_8080_TCP_ADDR:$JENKINS_PORT_8080_TCP_PORT"
fi

echo Running java $JAVA_OPTS -jar $JAR -fsroot $HOME $PARAMS "$@"
exec java $JAVA_OPTS -jar $JAR -fsroot $HOME $PARAMS "$@"
fi

# As argument is not jenkins, assume user want to run his own process, for sample a `bash` shell to explore this image
exec "$@"

26 changes: 26 additions & 0 deletions web/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
FROM openmicroscopy/devslave-c7:latest
MAINTAINER ome-devel@lists.openmicroscopy.org.uk

USER root

# Likely remove
#RUN sed -i.bak -re 's/( default_server.*)/; #\1/' /etc/nginx/nginx.conf
#ADD ./omeroweb.sh /home/omero/omeroweb.sh
#RUN chown omero:omero /home/omero/omeroweb.sh
#RUN chmod +x /home/omero/omeroweb.sh
#RUN chmod a+X /home/omero

EXPOSE 80

# Configuration for a general slave
# Note: a JDK MUST be installed above.
RUN usermod -u 11521 omero

# Ola new commit
ADD ./omeroweb.service /etc/systemd/system/omeroweb.service
RUN ln -s /etc/systemd/system/omeroweb.service /etc/systemd/system/multi-user.target.wants/omeroweb.service

# AS OMERO user
USER omero
ENV HOME /home/omero
WORKDIR /home/omero
16 changes: 16 additions & 0 deletions web/omeroweb.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
[Unit]
Description=OMERO.web
Requires=nginx.service

[Service]
User=omero
Type=forking
PIDFile=/home/omero/OMERO.py/var/django.pid
Restart=on-failure
RestartSec=10
Environment="VENVDIR=/home/omero/omeropy-virtualenv" "BINDIR=/home/omero/OMERO.py/bin"
ExecStart=/usr/bin/bash -c "source $VENVDIR/bin/activate; $BINDIR/omero web start"
ExecStop=/usr/bin/bash -c "source $VENVDIR/bin/activate; $BINDIR/omero web stop"

[Install]
WantedBy=multi-user.target