Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OCPBUGS-29003: move azure storage blobs from docker back into /docker #998

Merged
merged 3 commits into from Feb 16, 2024

Conversation

flavianmissi
Copy link
Member

@flavianmissi flavianmissi commented Feb 9, 2024

This undoes the effects caused by bug OCPBUGS-29003.
It needs to land in tandem with openshift/image-registry#392.


Testing

On a 4.13.z cluster, push a few images to the internal registry.
(Optional) Start some deployments from the pushed images.
Upgrade to 4.14.z (NOTE! use a 4.14.z that does not contain the bug fix).
Pulling will no longer work for images pushed on 4.13.z.
Push new images. To test more parts of the code, ensure that you push both images that were not pushed to 4.13.z as well as SOME images that were already pushed to 4.13.z (NOTE! make sure you still have images pushed to 4.13.z that cannot be pulled at this point).
Update the cluster to a 4.14.z version that contains the fix. I think if you use ClusterBot to create a release build, then you can upgrade to that build using oc adm upgrade --to-image= (but I have not tried this).

While the migration job is running, the operator will change to "Progressing: true". Once the migration job successfully finishes running, the operator should become available again.

Test the above steps, but using with a cluster wide proxy.
If you know other configurations for the registry that could affect the migration pod job, test them too.

Test deploying a cluster containing the fix to a different cloud. The migration job azure-path-fix should not be created and the operator should become available as it normally would.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 9, 2024
@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Feb 9, 2024
@openshift-ci-robot
Copy link
Contributor

@flavianmissi: This pull request references Jira Issue OCPBUGS-29003, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.16.0) matches configured target version for branch (4.16.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @wewang58

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This undoes the effects caused by bug OCPBUGS-29003.
It needs to land in tandem with openshift/image-registry#392.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 9, 2024
@flavianmissi flavianmissi changed the title WIP: OCPBUGS-29003: move azure storage blobs from docker back into /docker OCPBUGS-29003: move azure storage blobs from docker back into /docker Feb 13, 2024
@openshift-ci-robot
Copy link
Contributor

@flavianmissi: An error was encountered querying GitHub for users with public email (wewang@redhat.com) for bug OCPBUGS-29003 on the Jira server at https://issues.redhat.com/. No known errors were detected, please see the full error message for details.

Full error message. Post "http://ghproxy/graphql": dial tcp 172.30.229.2:80: connect: connection refused

Please contact an administrator to resolve this issue, then request a bug refresh with /jira refresh.

In response to this:

This undoes the effects caused by bug OCPBUGS-29003.
It needs to land in tandem with openshift/image-registry#392.


Testing

On a 4.13.z cluster, push a few images to the internal registry.
(Optional) Start some deployments from the pushed images.
Upgrade to 4.14.z (NOTE! use a 4.14.z that does not contain the bug fix).
Pulling will no longer work for images pushed on 4.13.z.
Push new images. To test more parts of the code, ensure that you push both images that were not pushed to 4.13.z as well as SOME images that were already pushed to 4.13.z (NOTE! make sure you still have images pushed to 4.13.z that cannot be pulled at this point).
Update the cluster to a 4.14.z version that contains the fix. I think if you use ClusterBot to create a release build, then you can upgrade to that build using oc adm upgrade --to-image= (but I have not tried this).

While the migration job is running, the operator will change to "Progressing: true". Once the migration job successfully finishes running, the operator should become available again.

Test the above steps, but using with a cluster wide proxy.
If you know other configurations for the registry that could affect the migration pod job, test them too.

Test deploying a cluster containing the fix to a different cloud. The migration job azure-path-fix should not be created and the operator should become available as it normally would.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@flavianmissi
Copy link
Member Author

flavianmissi commented Feb 13, 2024

azure tests are failing due to:

AzurePathFixControllerDegraded: Migration failed: /bin/sh: line 1: /usr/bin/move-blobs: No such file or directory

I got this error during my tests and the reason was that the image used by the controller to launch the job container was not the same as the image I had built. I need to investigate and see if the same is happening on the test.

@flavianmissi
Copy link
Member Author

I don't know why yet, but the image used by the job pod is (and it's fetched from an env var IMAGE):

registry.build05.ci.openshift.org/ci-op-w54twd3x/stable@sha256:ee79443ab0a07572145d49943a1d5c594cf33af3f9e4470397a7611a4784f54b

but the image used by the operator pod is:

registry.build05.ci.openshift.org/ci-op-w54twd3x/stable@sha256:a33923b7600aa3f970c05293a54ad0ff8956b2bef92982cb8204422fa38e4633

we want them to match.

@flavianmissi
Copy link
Member Author

Looks like the IMAGE env var is the image-registry image. I'm looking into the best way to use the operator image for the job instead.

flavianmissi and others added 3 commits February 15, 2024 14:46
* Dockerfile: add move-blobs binary to image
* cmd/move-blobs: prioritise account key for auth when present
this fixes the path bug on the storage level by migrating blobs from the
`docker` virtual directory into the `/docker` virtual directory instead.

* azurepathfix: give job required env vars
* handle different job conditions on azure path fix controller
* azurepathfixjob: set restart policy to never
* azurepathfixjob: export account key from secret
* azurepathfixjob: add azure account key env var to job container
* pass account key to path fix job (this time for reals)
* azurepathfix: only allow new controller to run on Azure
* also mount trusted-ca volume to job so the script works when a
  cluster-wide proxy is in use.
* fix a couple of mispellings
* manifests: export operator image env var to container
* manifests: update generated files
* mount volumes for proxy certs on azure path fix job
* azurepathfixcontroller: return error if account name is not present
@flavianmissi
Copy link
Member Author

/hold cancel

@flavianmissi
Copy link
Member Author

/cherry-pick release-4.15

@openshift-cherrypick-robot

@flavianmissi: once the present PR merges, I will cherry-pick it on top of release-4.15 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@dmage
Copy link
Member

dmage commented Feb 15, 2024

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 15, 2024
Copy link
Contributor

openshift-ci bot commented Feb 15, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dmage, flavianmissi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@flavianmissi
Copy link
Member Author

/hold cancel

@flavianmissi
Copy link
Member Author

/retest
/hold cancel

Copy link
Contributor

openshift-ci bot commented Feb 15, 2024

@flavianmissi: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@flavianmissi
Copy link
Member Author

/hold cancel

@flavianmissi flavianmissi changed the title OCPBUGS-29003: move azure storage blobs from docker back into /docker WIP: OCPBUGS-29003: move azure storage blobs from docker back into /docker Feb 16, 2024
@flavianmissi flavianmissi changed the title WIP: OCPBUGS-29003: move azure storage blobs from docker back into /docker OCPBUGS-29003: move azure storage blobs from docker back into /docker Feb 16, 2024
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 16, 2024
@openshift-merge-bot openshift-merge-bot bot merged commit 9bda27f into openshift:master Feb 16, 2024
12 checks passed
@openshift-ci-robot
Copy link
Contributor

@flavianmissi: Jira Issue OCPBUGS-29003: Some pull requests linked via external trackers have merged:

The following pull requests linked via external trackers have not merged:

These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-29003 has not been moved to the MODIFIED state.

In response to this:

This undoes the effects caused by bug OCPBUGS-29003.
It needs to land in tandem with openshift/image-registry#392.


Testing

On a 4.13.z cluster, push a few images to the internal registry.
(Optional) Start some deployments from the pushed images.
Upgrade to 4.14.z (NOTE! use a 4.14.z that does not contain the bug fix).
Pulling will no longer work for images pushed on 4.13.z.
Push new images. To test more parts of the code, ensure that you push both images that were not pushed to 4.13.z as well as SOME images that were already pushed to 4.13.z (NOTE! make sure you still have images pushed to 4.13.z that cannot be pulled at this point).
Update the cluster to a 4.14.z version that contains the fix. I think if you use ClusterBot to create a release build, then you can upgrade to that build using oc adm upgrade --to-image= (but I have not tried this).

While the migration job is running, the operator will change to "Progressing: true". Once the migration job successfully finishes running, the operator should become available again.

Test the above steps, but using with a cluster wide proxy.
If you know other configurations for the registry that could affect the migration pod job, test them too.

Test deploying a cluster containing the fix to a different cloud. The migration job azure-path-fix should not be created and the operator should become available as it normally would.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-cherrypick-robot

@flavianmissi: new pull request created: #1000

In response to this:

/cherry-pick release-4.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-cluster-image-registry-operator-container-v4.16.0-202402161440.p0.g9bda27f.assembly.stream.el9 for distgit ose-cluster-image-registry-operator.
All builds following this will include this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants