Extended.deploymentconfigs with revision history limits [Conformance] should never persist more old deployments than acceptable after being observed by the controller

[sosiouxme]

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/6409/consoleFull had a conformance test flake:

• Failure [288.339 seconds]
deploymentconfigs
/data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:767
  with revision history limits
  /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:716
    should never persist more old deployments than acceptable after being observed by the controller [Conformance] [It]
    /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:715

    Expected an error to have occurred.  Got:
        <nil>: nil

    /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:696

Additional Information

This is from #11045 where I revise how the asset bundle for the console is built. I don't see how it could be related to deployments.

[0xmichalis]

@sosiouxme is this failure from 4 days ago? If so, it should be fixed on master by #11066

[mfojtik]

@sosiouxme right, this should be fixed by now, if you see this again in more recent test run feel free to re-open this.

[sosiouxme]

sorry for the false alarm.

[soltysh]

This looks like a more recent run: https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/6719/consoleFull although I'm wondering if that didn't fail in what I'm tracking in #11016 (comment)

[soltysh]

My bad, wrongly interpreted logs.

[0xmichalis]

@enj can you post the job that failed?

[mfojtik]

@Kargakis https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/6910/

[enj]

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_conformance/6910/consoleFull

• Failure [486.240 seconds]
deploymentconfigs
/data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:767
  with revision history limits
  /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:716
    should never persist more old deployments than acceptable after being observed by the controller [Conformance] [It]
    /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:715

    Expected error:
        <*errors.errorString | 0xc8200f9600>: {
            s: "timed out waiting for the condition",
        }
        timed out waiting for the condition
    not to have occurred

    /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:696

[soltysh]

I've been hitting similar, but wrt to PodDisruptionbudget in #11187:

• Failure [80.527 seconds]
[k8s.io] DisruptionController [Feature:PodDisruptionbudget]
/data/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:793
  should update PodDisruptionBudget status [It]
  /data/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/disruption.go:102

  Expected error:
      <*errors.errorString | 0xc8200dd780>: {
          s: "timed out waiting for the condition",
      }
      timed out waiting for the condition
  not to have occurred

  /data/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/disruption.go:100

It all goes to #11016, I need to go back to that topic next week.

[bparees]

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin_future/693/testReport/junit/(root)/Extended/deploymentconfigs_with_revision_history_limits__Conformance__should_never_persist_more_old_deployments_than_acceptable_after_being_observed_by_the_controller/

[mfojtik]

@stevekuznetsov just saw this on GCE

[bparees]

and non-GCE
https://ci.openshift.redhat.com/jenkins/job/merge_pull_request_origin/489/

[stevekuznetsov]

@bparees Don't think that is the same flake -- openshift-eng/aos-cd-jobs#199 should have fixed the ETCD issue. There is no mention of credentials in the flake you linked.

[stevekuznetsov]

@smarterclayton are we running etcd data in a tmpfs in the GCE job? If not, we will continue to see etcd issues with fsync there

[0xmichalis]

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_gce/4883/

[mrogers950]

Failure [78.051 seconds]
deploymentconfigs
/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:1121
  with revision history limits [Conformance]
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:892
    should never persist more old deployments than acceptable after being observed by the controller [It]
    /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:891

    the current deployment needs to have finished before attempting to trigger a new deployment through configuration change
    Expected error:
        <*errors.errorString | 0xc421861bf0>: {
            s: "found multiple running deployments: [history-limit-2 history-limit-3]",
        }
        found multiple running deployments: [history-limit-2 history-limit-3]
    not to have occurred

    /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/deployments/deployments.go:851

https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_install_update/3885/

[smarterclayton]

https://ci.openshift.redhat.com/jenkins/job/zz_origin_gce_image/503/testReport/junit/(root)/Extended/_Feature_DeploymentConfig__deploymentconfigs_with_revision_history_limits__Conformance__should_never_persist_more_old_deployments_than_acceptable_after_being_observed_by_the_controller__Suite_openshift_conformance_parallel_/

[tnozicka]

Feels like this issue is tracking many different failures.

• I guess the etcd issue is fixed now?
• found multiple running deployments was already fixed
• this last failure remains to be investigated

[0xmichalis]

Different failure mode:

expected len of old deployments: 2 to equal dc revisionHistoryLimit: 3

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17104/test_pull_request_origin_extended_conformance_gce/11266/

[jim-minter]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17666/test_pull_request_origin_extended_conformance_install/3727/

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[stevekuznetsov]

/close