Fix build controller performance issues #12623

csrwng · 2017-01-23T18:26:56Z

Calls policy completed build processing as soon as it's marked Complete
instead of doing it when the build is handled by the BuildController.
Uses a build.openshift.io/accepted annotation to bump a build so the
BuildController can see it in its queue before the next Resync.

csrwng · 2017-01-23T18:27:06Z

@bparees ptal

bparees

naming nits. Also we're going to want to run the full extended builds/image_ecosystem suite on this.

bparees · 2017-01-23T18:28:17Z

pkg/build/api/types.go

+	BuildCompletedAnnotation = "openshift.io/build.completed"
+	// BuildAcceptedAnnotation is an annotation used to update a build that has been
+	// created so it can be seen build controller queue before a resync.
+	BuildAcceptedAnnotation = "openshift.io/build.accepted"


the new annotation format(sigh) is
BuildSourceSecretMatchURIAnnotationPrefix = "build.openshift.io/source-secret-match-uri-"

please align these new annotations with that.

bparees · 2017-01-23T18:34:15Z

pkg/build/api/types.go

+	// will prevent the build controller from processing it further.
+	BuildCompletedAnnotation = "openshift.io/build.completed"
+	// BuildAcceptedAnnotation is an annotation used to update a build that has been
+	// created so it can be seen build controller queue before a resync.


BuildAcceptedAnnotation is an annotation used to update a build that can now be run based on the RunPolicy(e.g. Serial). Updating the build with this annotation forces the build to be processed by the build controller queue without waiting for a resync.

bparees · 2017-01-23T18:35:07Z

and technically this needs @openshift/api-review

csrwng · 2017-01-23T18:39:03Z

comments addressed [test]

deads2k · 2017-01-23T18:49:40Z

I'm a little surprised these are annotations. Why wouldn't we have conditions for these on BuildStatus?

smarterclayton · 2017-01-23T18:53:37Z

Um. Why?

smarterclayton · 2017-01-23T18:54:01Z

pkg/build/controller/controller.go

@@ -102,9 +102,13 @@ func (bc *BuildController) HandleBuild(build *buildapi.Build) error {
 	}

 	if buildutil.IsBuildComplete(build) {
+		if _, ok := build.Annotations[buildapi.BuildCompletedAnnotation]; ok {
+			return nil
+		}


So you have complete... and really complete?

How is this safe? Why aren't you just creating a cache of "really complete" builds in memory and consulting that?

:-) I couldn't think of a better name ... but basically it means that we shouldn't run the runPolicy.OnComplete command on it. Maybe ran-policy-on-complete ?

@smarterclayton how is building a cache better ? It really doesn't matter that we run the policy.OnComplete again, but we'd rather avoid it if we can.

@smarterclayton how is building a cache better ? It really doesn't matter that we run the policy.OnComplete again, but we'd rather avoid it if we can.

Adding post-processing annotations instead of just using the straight status fields we have (you can index on them if you want) is effectively adding multiple (potentially conflicting) sources of truth. Adding them to annotations in particular gets weird quick, since you've created a "status" annotation you could set via a spec update. In addition, you don't even get label selection benefit as an annotation and you aren't using them for restricting selection from the server, so they aren't buying much in terms of performance over checking the actual status.

also memory caches die. we need this persisted so we don't have to run through all the builds again on a restart of the controller.

The distinction is a complete build means the build is complete. this annotation means "and we don't care about it anymore" (we still care about complete builds, the first time we see them enter the complete state)

@deads2k so you're proposing to modify BuildStatus to add this state ("really completed") to it?

bparees · 2017-01-23T19:06:48Z

Um. Why?

can't tell if that's directed at @deads2k or the PR itself.

deads2k · 2017-01-23T19:10:51Z

@deads2k so you're proposing to modify BuildStatus to add this state ("really completed") to it?

If it isn't actually computable from current status, yes. If it is computable from current status, do that and produce your own index (other controllers do this for various resources). Having it as annotation doesn't seem like a good result.

smarterclayton · 2017-01-23T19:19:07Z

Yeah, I'm -1 on modifying the object. Your controller should cache the negatives (ignore this because we know it's calculated) and you should bound that check if you're concerned with memory.

smarterclayton · 2017-01-23T19:19:38Z

Also, you cannot prevent racing controllers or severely delayed updates, so you will have to run this policy periodically, not just once.

bparees · 2017-01-23T19:22:54Z

Also, you cannot prevent racing controllers or severely delayed updates, so you will have to run this policy periodically, not just once.

resync logic will always figure out if a build needs to be run eventually. this logic is related to an optimization path around "whenever a build completes, see if there is another build queued that we should immediately run".

deads2k · 2017-01-23T19:23:37Z

Yeah, I'm -1 on modifying the object. Your controller should cache the negatives (ignore this because we know it's calculated) and you should bound that check if you're concerned with memory.

Speaking in person, this sounds like a build condition that is just missing from status today and the phase,condition tuple controls behavior (which is why we moved to conditions, away from phases).

The starttime one is just supposed to be a poke, so a secondary path to kicking the controller logic locally ought to work.

bparees · 2017-01-23T19:23:43Z

(and the problem is that we are currently doing that check for every build, every time we resync. we want to only do it for builds that really did just complete)

csrwng · 2017-01-23T23:07:11Z

[test]

bparees · 2017-01-24T00:19:38Z

i think it lgtm.

smarterclayton · 2017-01-24T00:40:28Z

pkg/build/controller/policy/policy.go

 	for _, build := range nextBuilds {
-		build.Status.StartTimestamp = &now
+		build.Annotations[buildapi.BuildAcceptedAnnotation] = uuid.NewRandom().String()


Add a TODO for this block - "replace with informer notification requeueing in the future"

Calls policy completed build processing as soon as it's marked Complete instead of doing it when the build is handled by the BuildController. Uses a build.openshift.io/accepted annotation to bump a build so the BuildController can see it in its queue before the next Resync.

csrwng · 2017-01-24T02:52:27Z

[testextended][extended:core(builds)]

smarterclayton · 2017-01-24T02:53:10Z

Lgtm as well

openshift-bot · 2017-01-24T02:53:22Z

Evaluated for origin test up to f3d26a3

openshift-bot · 2017-01-24T02:53:31Z

Evaluated for origin testextended up to f3d26a3

openshift-bot · 2017-01-24T04:13:21Z

continuous-integration/openshift-jenkins/test SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/13216/) (Base Commit: b0f2c58)

openshift-bot · 2017-01-24T04:13:38Z

continuous-integration/openshift-jenkins/test SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/13216/) (Base Commit: b0f2c58)

openshift-bot · 2017-01-24T04:45:46Z

continuous-integration/openshift-jenkins/testextended FAILURE (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin_extended/1022/) (Base Commit: b0f2c58) (Extended Tests: core(builds))

csrwng · 2017-01-24T11:54:46Z

1 extended test failed with a flake
[merge]

openshift-bot · 2017-01-24T12:05:27Z

Evaluated for origin merge up to f3d26a3

openshift-bot · 2017-01-24T21:21:52Z

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/13267/) (Base Commit: 2ea8719) (Image: devenv-rhel7_5766)

bparees requested changes Jan 23, 2017

View reviewed changes

bparees added the needs-api-review label Jan 23, 2017

bparees self-assigned this Jan 23, 2017

csrwng force-pushed the fix-build-controller branch from 8d16772 to e1df6bd Compare January 23, 2017 18:38

csrwng force-pushed the fix-build-controller branch from e1df6bd to 28625e6 Compare January 23, 2017 18:39

smarterclayton reviewed Jan 23, 2017

View reviewed changes

csrwng force-pushed the fix-build-controller branch from 28625e6 to bd0fc2a Compare January 23, 2017 23:06

smarterclayton reviewed Jan 24, 2017

View reviewed changes

smarterclayton added api-approved and removed needs-api-review labels Jan 24, 2017

csrwng force-pushed the fix-build-controller branch from bd0fc2a to f3d26a3 Compare January 24, 2017 02:49

openshift-bot merged commit ed6d1ad into openshift:master Jan 24, 2017

Fix build controller performance issues #12623

Fix build controller performance issues #12623

Conversation

csrwng commented Jan 23, 2017 • edited

csrwng commented Jan 23, 2017

bparees left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bparees commented Jan 23, 2017

csrwng commented Jan 23, 2017

deads2k commented Jan 23, 2017 • edited

smarterclayton commented Jan 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bparees commented Jan 23, 2017

deads2k commented Jan 23, 2017

smarterclayton commented Jan 23, 2017

smarterclayton commented Jan 23, 2017

bparees commented Jan 23, 2017

deads2k commented Jan 23, 2017

bparees commented Jan 23, 2017

csrwng commented Jan 23, 2017

bparees commented Jan 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csrwng commented Jan 24, 2017

smarterclayton commented Jan 24, 2017

openshift-bot commented Jan 24, 2017

openshift-bot commented Jan 24, 2017

openshift-bot commented Jan 24, 2017

openshift-bot commented Jan 24, 2017

openshift-bot commented Jan 24, 2017

csrwng commented Jan 24, 2017

openshift-bot commented Jan 24, 2017

openshift-bot commented Jan 24, 2017 • edited

csrwng commented Jan 23, 2017 •

edited

deads2k commented Jan 23, 2017 •

edited

openshift-bot commented Jan 24, 2017 •

edited