jenkins master/slave cleanup #11844

gabemontero · 2016-11-08T21:35:49Z

gabemontero · 2016-11-08T21:49:56Z

Note - still have to rework the kubernetes extended test to no longer use the old master/slave templates. But I wanted to get review of the doc going.

I should be able to simplify it and match it to the new master/slave example ... create a sample job similar to the pipeline plugin test that restricts the job to the nodejs slave.

bparees · 2016-11-08T22:37:16Z

examples/jenkins/master-slave/README.md

-plugin and define the sample job used by the tutorial.
+The sample builds upon the [sample job](https://github.com/openshift/origin/blob/master/examples/jenkins/README.md)
+described in this file's parent directory, and leverages the OpenShift Jenkins slave image for NodeJS to launch the sample
+job in a Jenkins slave provisioned as Kubernetes Pod on OpenShift.


is the job actually reliant on using the nodejs slave image as opposed to the base slave image? (it's not doing anything nodejs specific, is it?)

(I guess the base image isn't available as a slave in the default jenkins image configuration. still it would be worth noting why we're using the nodejs slave)

The example/jenkins/application-template.json is nodejs based....the src git repo in the build config is https://github.com/openshift/nodejs-ex.git and and the source strategy from clause is an ImageStreamTag for nodejs-010-centos7:latest. Are you saying to just point out those two details?

oh ok. for some reason i thought the example was ruby based. so nevermind. (though it's still true that the jenkins job itself doesn't need local nodejs tools on the slave pod, because it's just triggering a build in openshift, it's not running any nodejs commands itself)

So there's probably nothing additional to say... it's still a bit misleading that we use a nodejs slave image when we don't actually leverage the fact that it has nodejs tooling in it, but probably not worth discussing here. It's at least less confusing than what i thought we were doing, which was using the nodejs slave image with a ruby application.

bparees · 2016-11-08T22:39:09Z

examples/jenkins/master-slave/plugins.txt

-matrix-project:1.6
-scm-api:1.0
-script-security:1.13
-junit:1.2


that this exists and would need to be maintained makes me confident we are making the right choice in dumping this :)

OK ... merging into the top level README and completely removing master-slave has been pushed

bparees · 2016-11-08T22:40:28Z

examples/jenkins/master-slave/README.md

@@ -1,97 +1,37 @@
 Jenkins Master->Slave Example


this is going to be the only file left in this directory, right? maybe we should just append this content to the jenkins/README.md as a secondary tutorial/extended example?

Yes it is the only file. I'm good with moving the content into the base README and nuking the master-slave subdir. I'll go with that approach (thought it may be an hour or so until I get to it, in case you want to ruminate on it some more and end up deciding against this).

so far i still like it :)

bparees · 2016-11-09T20:46:05Z

@gabemontero doc changes look good. sounds like you've already looked into the extended tests but please do a general grep of the codebase for master-slave to see anywhere else we might have been referencing this. Also the official docs.

gabemontero · 2016-11-09T20:50:48Z

Good call - turns out ./test/integration/build_admission_test.go leverages the jenkins master template. Will have to update that one as well.

the openshift-docs grep came up clean.

sferich888 · 2016-11-10T19:52:02Z

examples/jenkins/README.md

+for Centos and RHEL produced by the [OpenShift Jenkins repository](https://github.com/openshift/jenkins).  The OpenShift
+Jenkins repository also produces a [base Jenkins slave image](https://github.com/openshift/jenkins/tree/master/slave-base),
+as well as sample Jenkins slave images for [Maven](https://github.com/openshift/jenkins/tree/master/slave-maven) and
+[NodeJS](https://github.com/openshift/jenkins/tree/master/slave-nodejs) which extend that base Jenkins slave image.


I don't think we should refer to these as "sample" images. We (Red Hat) ship these images, and as such they should be supported.

I also feel that telling customer that they should use these images as "samples" to build other images, portrays a very unpolished look and feel to the product. We should ship images for all of the supported SCL images, if we are going to promote this route.

@sferich888 per the exchange between @bparees and I below, is "reference" adequate for you, or simply dropping the word "sample" from the referenced phrase?

Dropping the word sample is better IMO, however it may lead to question about when other slave images will be provided and supported.

For now dropping "sample" is the best solution, and we can deal with the RFEs should we get them.

OK "sample" remove in ref to slave images pushed

gabemontero · 2016-11-10T20:42:14Z

On Thu, Nov 10, 2016 at 2:52 PM, Eric Rich notifications@github.com wrote:

@sferich888 commented on this pull request.

In examples/jenkins/README.md
#11844 (review)
:

@@ -73,6 +73,43 @@ Steps
     $ oc get services -n test | grep frontend
+Advanced
+-----
+
+You can also work with this sample and demonstrate the use of the kubernetes-plugin to manage
+Jenkins slaves that run as on-demand Pods. The kubenetes-plugin is pre-installed into the OpenShift Jenkins Images
+for Centos and RHEL produced by the OpenShift Jenkins repository. The OpenShift
+Jenkins repository also produces a base Jenkins slave image,
+as well as sample Jenkins slave images for Maven and
+NodeJS which extend that base Jenkins slave image.

I don't think we should refer to these as "sample" images. We (Red Hat)
ship these images, and as such they should be supported.

Would "reference" be more amenable than "sample" to everyone? Or
"supported"? ... or just "...as well as Jenkins slave images for ..."

I also feel that telling customer that they should use these images as
"samples" to build other images, portrays a very unpolished look and feel
to the product. We should ship images for all of the supported SCL images,
if we are going to promote this route.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#11844 (review),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADbadAKemlSwtaIcHhQPqiDCkujYiQ2jks5q83XpgaJpZM4Ks75Q
.

bparees · 2016-11-10T20:43:26Z

reference or just dropping sample is fine with me.

gabemontero · 2016-11-11T20:40:12Z

Pushed the ext/integ test updates for nuking master-slave.

Also fixed per our brief changed the plugin extended test job configs for the ramifications from the jenkins example app changing its ict to auto=false

gabemontero · 2016-11-11T20:40:35Z

and removed the WIP/DO NOT MERGE moniker

gabemontero · 2016-11-11T21:19:16Z

This PR will need the jenkin image update with v1.0.33 of openshift pipeline from openshift/jenkins#194 for the plugin extended tests to pass, since some of the test jobs start with a scale frontend 0 call.

bparees · 2016-11-11T21:27:19Z

test/integration/build_admission_test.go

 	// pipeline buildconfig's successfully, so we're not using the standard jenkins template.
 	// but we do need a template that creates a service named jenkins.
-	template, err := testutil.GetTemplateFixture("../../examples/jenkins/master-slave/jenkins-master-template.json")
+	template, err := testutil.GetTemplateFixture("../testdata/jenkins-template.json")


i'd like to see this template stripped down to the bare minimum (just creates a service named jenkins).

template pruned

bparees · 2016-11-11T22:54:48Z

[testextended][extended:core(jenkins)]

gabemontero · 2016-11-13T16:23:49Z

travis failure was a govet issue which make verify did not report locally for me ... none of the modules reported were ones I changed ... smells like a flake - will explore on Monday to confirm.

on the extended tests, Haoran Wang's new pipeline build test failed. The failure looks orthogonal to this PRs changes. The diagnositic dump included this:

** Build Logs:
Nov 11 19:58:32.789: INFO: Running 'oc logs --config=/tmp/extended-test-jenkins-pipeline-zwh0s-pgm6s-user.kubeconfig --namespace=extended-test-jenkins-pipeline-zwh0s-pgm6s -f build/sample-pipeline-1 --timestamps'
Nov 11 19:58:43.047: INFO: Error running &{/data/src/github.com/openshift/origin/_output/local/bin/linux/amd64/oc [oc logs --config=/tmp/extended-test-jenkins-pipeline-zwh0s-pgm6s-user.kubeconfig --namespace=extended-test-jenkins-pipeline-zwh0s-pgm6s -f build/sample-pipeline-1 --timestamps] []   Error from server: Timeout: timed out waiting for build sample-pipeline-1 to start after 10s
 Error from server: Timeout: timed out waiting for build sample-pipeline-1 to start after 10s
 [] <nil> 0xc820c3ef40 exit status 1 <nil> true [0xc8204c6338 0xc8204c6370 0xc8204c6370] [0xc8204c6338 0xc8204c6370] [0xc8204c6340 0xc8204c6368] [0xb14000 0xb14160] 0xc82153b2c0}:
Error from server: Timeout: timed out waiting for build sample-pipeline-1 to start after 10s
Error during log retrieval: Error retieving logs for util.BuildResult{BuildPath:"build/sample-pipeline-1", StartBuildStdErr:"", StartBuildStdOut:"build/sample-pipeline-1", StartBuildErr:error(nil), BuildConfigName:"", Build:(*api.Build)(0xc8207d4580), BuildAttempt:true, BuildSuccess:false, BuildFailure:false, BuildTimeout:true, oc:(*util.CLI)(0xc820cc2e10)}: exit status 1

The Build appears to have been created:

*****************************************
Dumping Build Result: util.BuildResult{BuildPath:"build/sample-pipeline-1", StartBuildStdErr:"", StartBuildStdOut:"build/sample-pipeline-1", StartBuildErr:error(nil), BuildConfigName:"", Build:(*api.Build)(0xc8207d4580), BuildAttempt:true, BuildSuccess:false, BuildFailure:false, BuildTimeout:true, oc:(*util.CLI)(0xc820cc2e10)}
Nov 11 19:58:32.531: INFO: Running 'oc describe --config=/tmp/extended-test-jenkins-pipeline-zwh0s-pgm6s-user.kubeconfig --namespace=extended-test-jenkins-pipeline-zwh0s-pgm6s build/sample-pipeline-1'

** Build Description:
Name:       sample-pipeline-1
Namespace:  extended-test-jenkins-pipeline-zwh0s-pgm6s
Created:    About an hour ago
Labels:     app=jenkins-pipeline-example
        buildconfig=sample-pipeline
        name=sample-pipeline
        openshift.io/build-config.name=sample-pipeline
        openshift.io/build.start-policy=Serial
        template=jenkins-pipeline-test
Annotations:    openshift.io/build-config.name=sample-pipeline
        openshift.io/build.number=1

Status:     New
Duration:   waiting for 1h0m3s
Build Config:   sample-pipeline
Build Pod:  sample-pipeline-1-build

Strategy:   JenkinsPipeline
Jenkinsfile contents:
  node{
  stage 'build'
  openshiftBuild(buildConfig: 'ruby-sample-build', showBuildLogs: 'true')
  }
Empty Source:   no input source provided

Build trigger cause:    Manually triggered

No events.

I don't see any pod events related to the build. If we need say build controller logs to see why the build that was created was never started, the current debug in the extended tests do not included that.

Thoughts on how to diagnose the pipeline failure @bparees or @csrwng ?

Otherwise, the extended test changes related to the PR did pass. One could argue this could be merged, and we diagnose the pipeline failure (possibly add additional debug for these type of failures) with a separate PR.

gabemontero · 2016-11-14T14:21:00Z

[testextended][extended:core(jenkins)]

openshift-bot · 2016-11-14T14:21:15Z

Evaluated for origin testextended up to 8a3f0b6

bparees · 2016-11-14T15:45:19Z

lgtm for a post 3.4 merge, but i'd definitely feel better if the pipeline test passes (Though i agree it doesn't seem related)

openshift-bot · 2016-11-14T15:53:21Z

continuous-integration/openshift-jenkins/testextended SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin_extended/794/) (Base Commit: 9c4f9ee) (Extended Tests: core(jenkins))

gabemontero · 2016-11-14T15:56:06Z

On Mon, Nov 14, 2016 at 10:45 AM, Ben Parees notifications@github.com
wrote:

lgtm for a post 3.4 merge, but i'd definitely feel better if the pipeline
test passes (Though i agree it doesn't seem related)

Cool deal - and the extended test passed this time. So the prior failure
is intermittent.

If we wanted to add debug to capture why a build that was created timedout
/ never started, would the build controller logs
but the right place to look .... i.e. would we want to dump that
pod's/container's logs ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#11844 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADbadDwnDtWm9yuXPX5VYCLbTiQVL_jsks5q-IITgaJpZM4Ks75Q
.

bparees · 2016-11-14T15:57:49Z

@gabemontero the build that never started is a pipeline strategy, so there's no pod associated. we'd have to dump the jenkins pod logs and hope the sync plugin contained some useful debug. This could be the timing window that i think @jim-minter noticed: openshift/jenkins-sync-plugin#106

gabemontero · 2016-11-14T16:01:53Z

On Mon, Nov 14, 2016 at 10:57 AM, Ben Parees notifications@github.com
wrote:

@gabemontero https://github.com/gabemontero the build that never
started is a pipeline strategy, so there's no pod associated. we'd have to
dump the jenkins pod logs and hope the sync plugin contained some useful
debug. This could be the timing window that i think @jim-minter
https://github.com/jim-minter noticed: fabric8io/openshift-jenkins-
sync-plugin#106
openshift/jenkins-sync-plugin#106

I see ... yeah, we are already dumping the jenkins pod logs. My
examination of them at least showed some exceptions but nothing that
directly pointed me to this behavior. That said, the discussion in
openshift/jenkins-sync-plugin#106
sounds spot wrt
to explaining what could have happened. I feel comfortable with
attributing the failure to that issue. Thanks.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#11844 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADbadHZFmmrgN3hnAd_RcZsjGZxtC-fnks5q-IUBgaJpZM4Ks75Q
.

gabemontero · 2016-11-14T16:02:43Z

So otherwise, I'll bump when 3.4 ships and the gates open for 3.5. Thanks.

bparees · 2016-11-14T16:07:51Z

sgtm

bparees · 2016-11-16T22:36:34Z

[merge]

openshift-bot · 2016-11-16T22:37:29Z

Evaluated for origin merge up to 8a3f0b6

openshift-bot · 2016-11-16T22:45:21Z

[Test]ing while waiting on the merge queue

openshift-bot · 2016-11-16T22:53:47Z

Evaluated for origin test up to 8a3f0b6

openshift-bot · 2016-11-17T00:21:17Z

continuous-integration/openshift-jenkins/test FAILURE (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/11515/) (Base Commit: a4b7101)

openshift-bot · 2016-11-17T01:05:30Z

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/11518/) (Base Commit: 3970bca) (Image: devenv-rhel7_5372)

gabemontero changed the title ~~jenkins master/slave cleanup~~ WIP / DO NOT MERGE: jenkins master/slave cleanup Nov 8, 2016

bparees reviewed Nov 8, 2016

View reviewed changes

gabemontero force-pushed the redoMasterSlave branch from 4403dbd to 61ce991 Compare November 9, 2016 20:31

sferich888 reviewed Nov 10, 2016

View reviewed changes

gabemontero force-pushed the redoMasterSlave branch 2 times, most recently from 3deb30f to 44f7d61 Compare November 11, 2016 20:27

gabemontero changed the title ~~WIP / DO NOT MERGE: jenkins master/slave cleanup~~ jenkins master/slave cleanup Nov 11, 2016

bparees requested changes Nov 11, 2016

View reviewed changes

bparees self-assigned this Nov 11, 2016

jenkins master/slave cleanup; fix job xml now that ict is auto==false

8a3f0b6

gabemontero force-pushed the redoMasterSlave branch from 44f7d61 to 8a3f0b6 Compare November 11, 2016 22:13

bparees added this to the 1.5.0 milestone Nov 11, 2016

gabemontero mentioned this pull request Nov 16, 2016

Fix extended tests #11922

Merged

bparees approved these changes Nov 16, 2016

View reviewed changes

openshift-bot merged commit 9a21f9b into openshift:master Nov 17, 2016

gabemontero deleted the redoMasterSlave branch November 17, 2016 03:52

gabemontero mentioned this pull request Nov 21, 2016

Multiple test fails #11903

Closed

nak3 mentioned this pull request Nov 22, 2016

remove dead link to the Jenkins template openshift/openshift-docs#3266

Closed

jenkins master/slave cleanup #11844

jenkins master/slave cleanup #11844

Uh oh!

Conversation

gabemontero commented Nov 8, 2016

Uh oh!

gabemontero commented Nov 8, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bparees commented Nov 9, 2016

Uh oh!

gabemontero commented Nov 9, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gabemontero Nov 10, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gabemontero commented Nov 10, 2016

@sferich888 commented on this pull request.

Uh oh!

bparees commented Nov 10, 2016

Uh oh!

gabemontero commented Nov 11, 2016

Uh oh!

gabemontero commented Nov 11, 2016

Uh oh!

gabemontero commented Nov 11, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bparees commented Nov 11, 2016

Uh oh!

gabemontero commented Nov 13, 2016

Uh oh!

gabemontero commented Nov 14, 2016

Uh oh!

openshift-bot commented Nov 14, 2016

Uh oh!

bparees commented Nov 14, 2016

Uh oh!

openshift-bot commented Nov 14, 2016

Uh oh!

gabemontero commented Nov 14, 2016

Uh oh!

bparees commented Nov 14, 2016

Uh oh!

gabemontero commented Nov 14, 2016

Uh oh!

gabemontero commented Nov 14, 2016

Uh oh!

bparees commented Nov 14, 2016

Uh oh!

bparees commented Nov 16, 2016

gabemontero Nov 10, 2016 •

edited

Loading

openshift-bot commented Nov 17, 2016 •

edited

Loading