Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hack/build: Pin to RHCOS 47.330 and quay.io/openshift-release-dev/ocp-release:4.0.0-0.5 #1271

Closed
wants to merge 2 commits into from
Closed

hack/build: Pin to RHCOS 47.330 and quay.io/openshift-release-dev/ocp-release:4.0.0-0.5 #1271

wants to merge 2 commits into from

Conversation

wking
Copy link
Member

@wking wking commented Feb 19, 2019

DO NOT MERGE! This PR is just for CI coverage (like #1192 and similar before it).

The bump from RHCOS 47.297 to 47.318 brings in, among other things, Podman 1.0.1.

@smarterclayton pushed 4.0.0-0.nightly-2019-02-17-024922 to quay.io/openshift-release-dev/ocp-release:4.0.0-0.4.

/hold

@openshift-ci-robot openshift-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 19, 2019
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 19, 2019
@wking
Copy link
Member Author

wking commented Feb 19, 2019

e2e-aws:

level=fatal msg="failed to initialize the cluster: Cluster operator machine-config is reporting a failure: Failed to resync 4.0.0-0.175.0.0-dirty because: error syncing: [Get https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterroles/machine-config-server: dial tcp 172.30.0.1:443: connect: connection refused, Get https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterroles/machine-config-daemon: dial tcp 172.30.0.1:443: connect: connection refused]"

/retest

@abhinavdahiya
Copy link
Contributor

The e2e errors were fixed by openshift/machine-config-operator#442

@crawford
Copy link
Contributor

/retest

@abhinavdahiya
Copy link
Contributor

The release image being used was generated on

$ oc adm release info quay.io/openshift-release-dev/ocp-release:4.0.0-0.4
Name:      4.0.0-0.4
Digest:    sha256:9f5f1d4514124f7406129629d8afd587f6beddd8b384b9ed09c5b4532f9f5631
Created:   2019-02-16 22:47:59 -0800 PST
OS/Arch:   linux/amd64
Manifests: 219

Release Metadata:
  Version:  4.0.0-0.4
  Upgrades: 4.0.0-0.3

And it has installer 1ae7d5c8633f94ba220260aac6aaab55a5ef6b7d

$ oc adm release info quay.io/openshift-release-dev/ocp-release:4.0.0-0.4 --commits | grep installer
  installer                                     https://github.com/openshift/installer                                     1ae7d5c8633f94ba220260aac6aaab55a5ef6b7d

And quite a few changes in the chanegelog were merged after the commit 1ae7d5c8633f94ba220260aac6aaab55a5ef6b7d

$ git log --pretty=oneline 1ae7d5c8633f94ba220260aac6aaab55a5ef6b7d..
ec2d3bbc7c26cd63af621318b25396718c22d743 (HEAD -> master, upstream/master) Merge pull request #1266 from mrogers950/mco_bootstrap_kube_ca
820ff4c0cb625b57e600ddee4f451a2716dc6a7d Merge pull request #1179 from mrogers950/ca_roots
2d62d2b5589e6845a8fdcb6cb5c53571d32b2953 bootkube: pass --kube-ca to MCO
a4d952ecde40dd2dbd815654c133458512e60d0c pkg/asset/kubeconfig: use kube-ca in kubelet kubeconfig
4ccb47cc6f050bb676dc42ac5e654e8202a7a1c1 pkg/asset/kubeconfig: use kube-ca in admin kubeconfig
ec2580ca2c0abd692c3a462ccd784191b1c72d82 pkg/asset/tls: don't append parent to kube-apiserver cert
5d2abe5c6df1a20d035b68eaae1569003cf6455f pkg/asset/tls: self-sign kube-ca
3e8831844df49cfca0c3ebcccabc8b537ab87971 (validate_network) Merge pull request #1126 from flaper87/switch-yaml-libs
54506be98cff773ed6ef60b18db0e26d9f7f618f Merge pull request #1259 from crawford/channel
03c258babd53842d49ae8105e9e9c42b49cd46d6 Merge pull request #1262 from smarterclayton/depend_util_linux
71d199c55e0e897801765d4283049486562beb92 image: Take explicit dependency on util-linux for uuidgen and gzip
5af1819660a96e61c3485909c6f238cb2f1cca50 Merge pull request #1245 from Fedosin/update_machines_namespace
b4c027ff2ebfbb98d6d46e20e7576d5d3a32294b Merge pull request #1169 from abhinavdahiya/cluster_domain
e382f907f18c5dd9766733f3e1f853f86b2825e2 Merge pull request #1253 from mfojtik/strip-openshift-prefix
0ab9b19f1e10c3b3976f9b5d030ba64f7a94e54b data/manifests: update the default update channel
a0a77add3ddf259f7c0dbcfabc00a61af5cc9575 (origin/cluster_domain, cluster_domain) openstack: switch from base_domain to cluster_domain
bab60857dbfe19bd12686a0d1c766254b2d056f8 destroy/aws: update the public zone discovery
2a0f9b3415520c3e7594fc0e95d58b0971e75975 assets: update assets to use cluster_domain, new api URL
1ab1cd3fb04f2ed0b4ff831c1a436ddfd9171ae0 types: add ClusterDomain helper for InstallConfig
65380308c19b3b4d597dbb58c4e7b3fc21adf72e aws: switch from base_domain to cluster_domain
3f7f0c94f3c8e7c4be31a9bebef5c3e9f3bb473a libvirt: switch from base_domain to cluster_domain
92986dfc962dcbc32d4937ba75e5ff04f1d3130b bootstrap: add new names for kube apiserver and controller manager pods
425ac3dee7ef557bc5f229eca7c831007bcb857c Update machines namespace for OpenStack
03d8058734bf2562b2d71c82f0074339856b6059 openstack: remove use of yaml.v2
bd3a025d2b7224eefb747512ec47758bb4b82cab vendor: update gophercloud/utils

@wking
Copy link
Member Author

wking commented Feb 19, 2019

The e2e errors were fixed by openshift/machine-config-operator#442

Did we want to push a more-recent update payload to quay.io to pull that in? Or are we ok rolling with 4.0.0-0.nightly-2019-02-17-024922?

And quite a few changes in the chanegelog were merged after the commit 1ae7d5c

That's fine, as long as we don't land incompatible installer changes. Are there some you think are incompatible?

@abhinavdahiya
Copy link
Contributor

The e2e errors were fixed by openshift/machine-config-operator#442

Did we want to push a more-recent update payload to quay.io to pull that in? Or are we ok rolling with 4.0.0-0.nightly-2019-02-17-024922?

And quite a few changes in the chanegelog were merged after the commit 1ae7d5c

That's fine, as long as we don't land incompatible installer changes. Are there some you think are incompatible?

from the commits all the DNS domain switches happened after the release image was built

@smarterclayton
Copy link
Contributor

So if we pick up a new payload we will pick up bootstrap pivot which means we have other challenges.

Do we need those changes?

@abhinavdahiya
Copy link
Contributor

So if we pick up a new payload we will pick up bootstrap pivot which means we have other challenges.

Do we need those changes?

These are being left out... which I think we would like. there were changes merged to operators for those.

a0a77add3ddf259f7c0dbcfabc00a61af5cc9575 (origin/cluster_domain, cluster_domain) openstack: switch from base_domain to cluster_domain
bab60857dbfe19bd12686a0d1c766254b2d056f8 destroy/aws: update the public zone discovery
2a0f9b3415520c3e7594fc0e95d58b0971e75975 assets: update assets to use cluster_domain, new api URL
1ab1cd3fb04f2ed0b4ff831c1a436ddfd9171ae0 types: add ClusterDomain helper for InstallConfig
65380308c19b3b4d597dbb58c4e7b3fc21adf72e aws: switch from base_domain to cluster_domain
3f7f0c94f3c8e7c4be31a9bebef5c3e9f3bb473a libvirt: switch from base_domain to cluster_domain

So either we can release installer at the 1ae7d5c8633f94ba220260aac6aaab55a5ef6b7d so that we don't have mis-match...

@crawford
Copy link
Contributor

I think we could also use 4.0.0-0.alpha-2019-02-18-042039 (it's not a nightly though). That will have all of the DNS changes, but won't include the OS pivot change.

@smarterclayton
Copy link
Contributor

Why don't we release 1ae and if necessary we can recut? I haven't seen any pivot issues yet, so I'm not terribly worried. But I'm trying to keep other operator changes out because there are destabilizing things already this week.

@smarterclayton
Copy link
Contributor

We probably need to check when monitoring regressed (whether that was before this nightly or after) due to cert rotation.

@wking
Copy link
Member Author

wking commented Feb 19, 2019

We probably need to check when monitoring regressed (whether that was before this nightly or after) due to cert rotation.

Is that openshift/cluster-monitoring-operator#250? openshift/cluster-kube-apiserver-operator#247? openshift/cluster-monitoring-operator#256? Checking the monitoring operator:

$ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.0.0-0.4 | grep monitoring
  cluster-monitoring-operator                   https://github.com/openshift/cluster-monitoring-operator                   ace45df5318630b643a605a441465b9732914d8a
$ git log --first-parent -7 --format='%ai %h %s'
2019-02-19 19:01:48 +0100 b86dd7c Merge pull request #256 from squat/kubelet-ca
2019-02-18 22:49:05 +0100 49fe2fd Merge pull request #251 from squat/reconfig_telemeter
2019-02-18 16:13:51 +0100 63a886d Merge pull request #240 from abhinavdahiya/fix_waits
2019-02-18 14:46:38 +0100 0ad103c Merge pull request #252 from njhale/fix-operatorgroup
2019-02-18 14:46:33 +0100 0adaf09 Merge pull request #245 from mxinden/travis-gobindata
2019-02-18 11:50:45 +0100 47f30f2 Merge pull request #250 from squat/kubelet-ca
2019-02-15 10:02:14 +0100 ace45df Merge pull request #249 from s-urbaniak/mon-219

So we're missing those monitoring changes.

@openshift-ci-robot openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 23, 2019
@wking wking mentioned this pull request Feb 26, 2019
Closed
@wking
CHANGELOG: Document changes since 8f08508
4caac53 
Through 0208204 (Merge pull request openshift#1305 from
deads2k/admin-kubeconfig-2, 2019-02-25).
@openshift-ci-robot openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 27, 2019
@wking wking changed the title hack/build: Pin to RHCOS 47.318 and quay.io/openshift-release-dev/ocp-release:4.0.0-0.4 hack/build: Pin to RHCOS 47.330 and quay.io/openshift-release-dev/ocp-release:4.0.0-0.5 Feb 27, 2019
@wking
Copy link
Member Author

wking commented Feb 27, 2019

Rebased onto #1316 with c246237 -> 7b42fa7, which also bumps the release image to quay.io/openshift-release-dev/ocp-release:4.0.0-0.5 and RHCOS to 47.330 (as referenced from the release image).

@wking
hack/build: Pin to RHCOS 47.330 and quay.io/openshift-release-dev/ocp…
e02b35a 
…-release:4.0.0-0.5

Clayton pushed 4.0.0-0.nightly-2019-02-26-125216 to
quay.io/openshift-release-dev/ocp-release:4.0.0-0.5.  Extracting the
associated RHCOS build:

  $ oc adm release info --pullspecs quay.io/openshift-release-dev/ocp-release:4.0.0-0.5 | grep machine-os-content
    machine-os-content                            registry.svc.ci.openshift.org/ocp/4.0-art-latest-2019-02-26-125216@sha256:1262533e31a427917f94babeef2774c98373409897863ae742ff04120f32f79b
  $ oc image info registry.svc.ci.openshift.org/ocp/4.0-art-latest-2019-02-26-125216@sha256:1262533e31a427917f94babeef2774c98373409897863ae742ff04120f32f79b | grep version
              version=47.330

The bump from RHCOS 47.297 to 47.330 brings in, among other things,
Podman 1.0.1.

Renaming OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE gets us CI testing
of the pinned release despite openshift/release@60007df2 (Use
RELEASE_IMAGE_LATEST for CVO payload, 2018-10-03,
openshift/release#1793).

Also comment out regions which this particular RHCOS build wasn't
pushed to, leaving only:

  $ curl -s https://releases-rhcos.svc.ci.openshift.org/storage/releases/maipo/47.330/meta.json | jq -r '.amis[] | .name'
  ap-northeast-1
  ap-northeast-2
  ap-south-1
  ap-southeast-1
  ap-southeast-2
  ca-central-1
  eu-central-1
  eu-west-1
  eu-west-2
  eu-west-3
  sa-east-1
  us-east-1
  us-east-2
  us-west-1
  us-west-2

I'd initially expected to export the pinning environment variables in
release.sh, but I've put them in build.sh here because our continuous
integration tests use build.sh directly and don't go through
release.sh.

[1]: containers/podman#2106
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Feb 27, 2019 •
edited
Loading

@wking: The following test failed for commit e02b35a, say /retest to rerun them:

Test name Details Rerun command
ci/prow/e2e-aws link /test e2e-aws

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@crawford
Copy link
Contributor

LGTM

@wking
Copy link
Member Author

wking commented Feb 27, 2019

images:

2019/02/27 02:18:51 Ran for 3m22s
error: could not run steps: test "release-latest" failed: pod release-latest was already deleted

/retest

@wking
Copy link
Member Author

wking commented Feb 27, 2019

e2e-aws:

Flaky tests:

[Conformance][Area:Networking][Feature:Router] The HAProxy router should serve a route that points to two services and respect weights [Suite:openshift/conformance/parallel/minimal]
[sig-cli] Kubectl Port forwarding [k8s.io] With a server listening on 0.0.0.0 [k8s.io] that expects a client request should support a client that connects, sends DATA, and disconnects [Suite:openshift/conformance/parallel] [Suite:k8s]

Failing tests:

[sig-api-machinery] CustomResourcePublishOpenAPI [Feature:CustomResourcePublishOpenAPI] works for CRD with validation schema [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-api-machinery] CustomResourcePublishOpenAPI [Feature:CustomResourcePublishOpenAPI] works for CRD without validation schema [Suite:openshift/conformance/parallel] [Suite:k8s]

I dunno if those two failures are real or flakes, but they don't look important enough to be worth holding up the tag. Re-testing anyway to see if they pass while I push the tag and binaries:

/retest

@wking
Copy link
Member Author

wking commented Feb 27, 2019

This is now v0.13.0.

@wking
Copy link
Member Author

wking commented Feb 27, 2019

e2e-aws:

Flaky tests:

[Feature:Builds][Conformance] s2i build with a quota  Building from a template should create an s2i build with a quota and run it [Suite:openshift/conformance/parallel/minimal]

Failing tests:

[sig-api-machinery] CustomResourcePublishOpenAPI [Feature:CustomResourcePublishOpenAPI] works for CRD with validation schema [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-api-machinery] CustomResourcePublishOpenAPI [Feature:CustomResourcePublishOpenAPI] works for CRD without validation schema [Suite:openshift/conformance/parallel] [Suite:k8s]

So maybe real. But oh, well, 0.14.0 will be out soon :).

/close

@openshift-ci-robot
Copy link
Contributor

@wking: Closed this PR.

In response to this:

e2e-aws:

Flaky tests:

[Feature:Builds][Conformance] s2i build with a quota  Building from a template should create an s2i build with a quota and run it [Suite:openshift/conformance/parallel/minimal]

Failing tests:

[sig-api-machinery] CustomResourcePublishOpenAPI [Feature:CustomResourcePublishOpenAPI] works for CRD with validation schema [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-api-machinery] CustomResourcePublishOpenAPI [Feature:CustomResourcePublishOpenAPI] works for CRD without validation schema [Suite:openshift/conformance/parallel] [Suite:k8s]

So maybe real. But oh, well, 0.14.0 will be out soon :).

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@wking wking deleted the version-0.13.0-pins branch February 27, 2019 06:26
This was referenced Feb 28, 2019
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

@staebler staebler Awaiting requested review from staebler

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@wking @openshift-ci-robot @abhinavdahiya @crawford @smarterclayton