fix(controller): calculate satisfied with && instead of ||

[GuoHaiqing]

If calculating satisfied with '||', it only guarantee some replicaTypes not all replicaTypes are satisfied. There will be some problems. So it should be changed to '&&'.

---

This change is 

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[k8s-ci-robot]

Hi @GuoHaiqing. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[coveralls]

Coverage remained the same at 96.512% when pulling d7855fd on GuoHaiqing:fix/controller-satisfied into 4b67180 on kubeflow:master.

[TravisBuddy]

Hey @GuoHaiqing,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: f80653a0-261e-11ea-8748-5b98fb6aa2de

[GuoHaiqing]

@googlebot I signed it!

[TravisBuddy]

Hey @GuoHaiqing,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 1c08a900-2620-11ea-8748-5b98fb6aa2de

[TravisBuddy]

Hey @GuoHaiqing,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: 6b1245b0-2620-11ea-8748-5b98fb6aa2de

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[GuoHaiqing]

/assign @richardsliu

[TravisBuddy]

Hey @GuoHaiqing,
Your changes look good to me!

View build log

TravisBuddy Request Identifier: fd6cb3f0-2620-11ea-8748-5b98fb6aa2de

[gaocegege]

lating satisfied with '||', it only guarantee some replicaTypes not all replicaTypes are satisfied. There will be some problems. So it should be changed to '&&'.

Could you please explain more about why it may cause problems? Thanks

[GuoHaiqing]

reconcileTFJobs reconcile all replicaTypes and satisfiedExpectations determines whether there should begin a new reconciliation or not. I think it should begin a new reconciliation only when all replicaTypes are satisfied.

tfjobNeedsSync := tc.satisfiedExpectations(tfjob)
// Set default for the new tfjob.
scheme.Scheme.Default(tfjob)

var reconcileTFJobsErr error
if tfjobNeedsSync && tfjob.DeletionTimestamp == nil {
	reconcileTFJobsErr = tc.reconcileTFJobs(tfjob)
}

// Diff current active pods/services with replicas.
for rtype, spec := range tfjob.Spec.TFReplicaSpecs {
	err = tc.reconcilePods(tfjob, pods, rtype, spec, replicasStatus)
	if err != nil {
		logger.Warnf("reconcilePods error %v", err)
		return err
	}

	err = tc.reconcileServices(tfjob, services, rtype, spec)

	if err != nil {
		logger.Warnf("reconcileServices error %v", err)
		return err
	}
}

[gaocegege]

Then, the problem is about performance?

[GuoHaiqing]

In one reconciliation, controller creates one master and one ps. Then controller observes one master pod creation. At this moment, satisfiedExpectations returns true. It will begin a new reconciliation. In this reconciliation, ps pods from client-go will be empty and controller will create a new ps pod.

ControllerExpectations is for handling stale client-go cache. At this situation, I think ControllerExpectations doesn't play a corresponding role.

[gaocegege]

Gotcha. Reasonable. Let me have a look.

[gaocegege]

/lgtm

[gaocegege]

/ok-to-test

[richardsliu]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: richardsliu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [richardsliu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[richardsliu]

/retest

[richardsliu]

@GuoHaiqing Can you rebase your PR? Sorry it has been many months and I forgot about this.

[gaocegege]

/lgtm

[gaocegege]

I think there are some new problems introduced by this PR if the controller deletes some pods when the job is succeeded and the cleanpodpolicy is running.

The pod is deleted, we will get one observation here jc.Expectations.DeletionObserved(expectationPodsKey). But we do not sync the TFJob because of the PR. Then we cannot reach here. https://github.com/GuoHaiqing/tf-operator/blob/d7855fdee562594830f3b0fa4d3bbb60be0c014a/pkg/controller.v1/tensorflow/controller.go#L373

WDYT @GuoHaiqing @SimonCqk