Bug 1972572: Fix image incompatibility between 4.7 and 4.8 while upgrade #48
Conversation
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
/test e2e-metal-ipi-upgrade |
|
@stbenjam: The specified target(s) for
Use
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
ardaguclu
force-pushed
the
fix-upgrade-image-leakage
branch
from
4f700c0
to
511f8b8
Compare
ardaguclu
changed the title
openshift-ci
bot
added
bugzilla/severity-high
|
@ardaguclu: This pull request references Bugzilla bug 1972572, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Bugzilla (augol@redhat.com), skipping review request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@ardaguclu: This pull request references Bugzilla bug 1972572, which is valid. 3 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Bugzilla (augol@redhat.com), skipping review request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
get-resource.sh
Outdated
| # Added to fix image path incompatibility between 4.7 and 4.8 | ||
| # as filed in https://bugzilla.redhat.com/show_bug.cgi?id=1972572 | ||
| if [[ -s "/shared/html/images/$RHCOS_IMAGE_FILENAME_QCOW/$RHCOS_IMAGE_FILENAME_COMPRESSED.md5sum" && ! -s "/shared/html/images/$RHCOS_IMAGE_FILENAME_QCOW/$RHCOS_IMAGE_FILENAME_CACHED.md5sum" ]]; then | ||
| exit 0 |
There was a problem hiding this comment.
I guess this will work, but would it be better to move $RHCOS_IMAGE_FILENAME_COMPRESSED to $RHCOS_IMAGE_FILENAME_CACHED - then the filenames become consistent after upgrade (e.g same as a freshly installed 4.8 cluster)?
Also that would mean that eventually we can remove the workaround and simplify the script (hopefully reducing the chances of more issues like this...
There was a problem hiding this comment.
Thanks @hardys.
Do you mean by workaround this change or this one;
ironic-rhcos-downloader/get-resource.sh
Line 88 in 09c293e
Because if we rename files, above workaround seems needs to be removed.
There was a problem hiding this comment.
I'm wrong, that does not need to be removed.
I'll change according to your suggestions.
ardaguclu
force-pushed
the
fix-upgrade-image-leakage
branch
from
511f8b8
to
18451f5
Compare
ardaguclu
changed the title
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
|
/test e2e-metal-ipi-ovn-ipv6 |
|
/approve |
get-resource.sh
Outdated
|
|
||
| # Added to fix image path incompatibility between 4.7 and 4.8 | ||
| # as filed in https://bugzilla.redhat.com/show_bug.cgi?id=1972572 | ||
| if [[ -s "/shared/html/images/$RHCOS_IMAGE_FILENAME_QCOW/$RHCOS_IMAGE_FILENAME_COMPRESSED.md5sum" && ! -s "/shared/html/images/$RHCOS_IMAGE_FILENAME_QCOW/$RHCOS_IMAGE_FILENAME_CACHED.md5sum" ]]; then |
There was a problem hiding this comment.
We ought to check that -openstack is in the name (as on line 101) otherwise this will do the Wrong Thing if it isn't. AFAIK it always is today, but the whole goal of the patch that messed everything up was to make it not required any more, so we might as well at least get that benefit.
There was a problem hiding this comment.
Note that this prevents the problem, but does not allow us to recover from the problem if a buggy version of this script has already been run.
If this is not a blocker for 4.8 then I think we need to also remove the directory /shared/html/images/${RHCOS_IMAGE_FILENAME_QCOW} if it exists around line 91.
There was a problem hiding this comment.
Thanks @zaneb, I made changes you suggested.
| if [[ -s "/shared/html/images/$RHCOS_IMAGE_FILENAME_QCOW/$RHCOS_IMAGE_FILENAME_COMPRESSED.md5sum" && ! -s "/shared/html/images/$RHCOS_IMAGE_FILENAME_QCOW/$RHCOS_IMAGE_FILENAME_CACHED.md5sum" ]]; then | ||
| cd /shared/html/images | ||
| mv "${RHCOS_IMAGE_FILENAME_QCOW}/$RHCOS_IMAGE_FILENAME_COMPRESSED.md5sum" "${RHCOS_IMAGE_FILENAME_QCOW}/$RHCOS_IMAGE_FILENAME_CACHED.md5sum" | ||
| mv "${RHCOS_IMAGE_FILENAME_QCOW}/$RHCOS_IMAGE_FILENAME_COMPRESSED" "${RHCOS_IMAGE_FILENAME_QCOW}/$RHCOS_IMAGE_FILENAME_CACHED" |
There was a problem hiding this comment.
Fun fact: this means that on an upgraded cluster we won't edit the kernel command line to add ip=dhcp[6], whereas before we accidentally were. Is that OK?
There was a problem hiding this comment.
That is a good point, I think we have to discuss.
There was a problem hiding this comment.
I think these dual-stack problems started as a result of RHCOS moving from the network-legacy to network-manager dracut modules, which introduced a slight change in behavior meaning that the protocol chosen depends on a race between the two DHCP servers.
Looking at the coreos config it seems this change happened several releases ago, but presumably the support status of dual-stack meant it wasn't noticed.
@dustymabe can you confirm which RHCOS versions will be using NM in the ramdisk?
I think we may want to handle this on upgrade, and also backport the kernel commandline fix to previous releases, but that can probably be considered outside the scope of this PR, given that the commandline fix only currently exists on 4.8+ ?
There was a problem hiding this comment.
After sleep, it occurs to me that this PR is our best chance to fix this for those upgrading to 4.8, because at this point in the code we know we are dealing with a non-modified qcow. After the rename that happens here, we will have to go spelunking in the image to determine if it has been modified. (We're already doing it twice in the image cache and that's bad enough.)
OTOH, we will need to do the spelunking thing in 4.7 anyway.
There was a problem hiding this comment.
@dustymabe can you confirm which RHCOS versions will be using NM in the ramdisk?
4.6 and newer.
ardaguclu
force-pushed
the
fix-upgrade-image-leakage
branch
from
18451f5
to
f3e1a2f
Compare
|
/test e2e-metal-ipi |
|
/test e2e-metal-ipi-serial-ipv4 |
1 similar comment
|
/test e2e-metal-ipi-serial-ipv4 |
ardaguclu
changed the title
| if [[ -s "/shared/html/images/$RHCOS_IMAGE_FILENAME_QCOW/$RHCOS_IMAGE_FILENAME_COMPRESSED.md5sum" && ! -s "/shared/html/images/$RHCOS_IMAGE_FILENAME_QCOW/$RHCOS_IMAGE_FILENAME_CACHED.md5sum" ]]; then | ||
| cd /shared/html/images | ||
| mv "${RHCOS_IMAGE_FILENAME_QCOW}/$RHCOS_IMAGE_FILENAME_COMPRESSED.md5sum" "${RHCOS_IMAGE_FILENAME_QCOW}/$RHCOS_IMAGE_FILENAME_CACHED.md5sum" | ||
| mv "${RHCOS_IMAGE_FILENAME_QCOW}/$RHCOS_IMAGE_FILENAME_COMPRESSED" "${RHCOS_IMAGE_FILENAME_QCOW}/$RHCOS_IMAGE_FILENAME_CACHED" |
There was a problem hiding this comment.
After sleep, it occurs to me that this PR is our best chance to fix this for those upgrading to 4.8, because at this point in the code we know we are dealing with a non-modified qcow. After the rename that happens here, we will have to go spelunking in the image to determine if it has been modified. (We're already doing it twice in the image cache and that's bad enough.)
OTOH, we will need to do the spelunking thing in 4.7 anyway.
|
After discussion, we're going to get this in and worry about backporting the dual-stack fix to multiple releases later. |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ardaguclu, hardys, sadasu, zaneb The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
@ardaguclu: All pull requests linked via external trackers have merged: Bugzilla bug 1972572 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/cherry-pick release-4.8 |
|
@ardaguclu: new pull request created: #51 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Upgrade from 4.7 to 4.8 jobs are consuming huge disk space. Because 4.8 uses image caching mechanism and 4.7 not, that's why, 4.8 always downloads same image. To prevent this, this PR adds check and return if the image already exist. This not only saves disk usage but also reduces upgrade duration.