Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

schutzbot: enable F33 testing #1014

Merged
merged 5 commits into from Nov 11, 2020
Merged

Conversation

ondrejbudai
Copy link
Member

@ondrejbudai ondrejbudai commented Oct 1, 2020

Update as of 2020-11-11

This is now done, see today's comment.

Progress as of 2020-10-02

Done so far

  • ✔️ Fedora 33 beta images are available in both PSI and AWS
  • ✔️ Jenkins could not connect to F33 instances. The cause is not entirely clear but it's probably connected to changes in crypto. Major was able to find a workaround. (set crypto policy to LEGACY)
  • ✔️ kolo/xmlrpc was rebased in F33 and has a slightly different API. This PR ships a somewhat ugly workaround, see individual commits for more info.
  • ✔️ s3cmd used in mockbuild.sh does not work on F33 because of changes in Python 3.9. I've sent a PR upstream and a bug downstream. This PR applies the same patch to s3cmd as is in the upstream PR.
  • ✔️ F33 cloud base image doesn't have openssl installed so I have to install it manually so we can generate TLS keys for remote worker/cloud/koji API.

Needs work

  • 🚫 auth tests are broken because they rely on common names in TLS certificates that are deprecated in Go 1.15. This is just an issue in the test, actual implementation should be good.
  • 🚫 image tests need updated cases because the base repo is still moving. We should use a snapshot.
  • 🚫 koji test is broken. Not sure why, looks like an issue with importing schema into the database.
  • 🚫 test/image-tests/ostree.sh: line 167: IMAGE_TYPE: unbound variable - no idea, I haven't had time to investigate yet.

@ondrejbudai ondrejbudai marked this pull request as draft October 2, 2020 09:44
@ondrejbudai ondrejbudai force-pushed the fedora33-testing branch 15 times, most recently from 567aa2a to 965d12e Compare October 7, 2020 12:55
@dvdhrm
Copy link
Contributor

dvdhrm commented Oct 8, 2020

FYI: F33 snapshots are now available. First one is:

https://rpmrepo.osbuild.org/v1/anon/f33/f33-x86_64-devel-20201006/repodata/repomd.xml

I will make another one the coming weekend. You can enumerate the snapshots here:

https://rpmrepo.osbuild.org/control/snapshots

If you need another snapshot earlier than next Monday, lemme know!

@ondrejbudai ondrejbudai force-pushed the fedora33-testing branch 3 times, most recently from d18d623 to fcdf2cf Compare October 13, 2020 12:55
@ondrejbudai ondrejbudai force-pushed the fedora33-testing branch 3 times, most recently from 0f1a6f5 to 4f88274 Compare November 10, 2020 13:23
The downloaded image may not fit inside tmpfs, especially when testing
on a constrained VM. This commit makes the test script use a different
temporary directory while handling the possibly big image.
Let's use the same repository as for Fedora 32, it proved to be very stable.
If the ostree test was run on an unsupported distro, it failed but with a
very weird error message. This commit makes the test fail fast and with a
nice message.
This commit does several things:

1) Changes the Fedora 33 repos in the test case generator from development
   to release ones.

2) Fixes format-request-map.json so we can generate fedora-iot-commit
   "images".

3) Regenerates all the cases.
osbuild-composer now has support, let's fire up more VMs! Currently, F33 beta
is used in both AWS and PSI.
@ondrejbudai
Copy link
Member Author

After 42 days, this is finally finished and green! This is just the tip of the iceberg, many stuff needed to properly support Fedora 33 testing was merged earlier:

This PR does basically 3 things:

  1. Adds support for F33 to Ostree tests
  2. Fixes the test case generator, updates the repos used by it and regenerates the test cases
  3. Enables F33 testing on Schutzbot

But there's one more thing!

Qemu tests downloaded the image to /tmp that doesn't have enough space on F33 (possibly connected with the ZRAM change?[1]. I modified the test to use /var/lib/osbuild-composer-tests/ for this purpose. This is btw the same dir as image-tests use for temporary data.

[1]: https://fedoraproject.org/wiki/Changes/SwapOnZRAM

@ondrejbudai ondrejbudai marked this pull request as ready for review November 11, 2020 07:10
Copy link
Contributor

@msehnout msehnout left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't checked the test cases properly, they are rather long, but otherwise this looks very good!

}
post {
always {
preserve_logs('fedora33-image')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
preserve_logs('fedora33-image')
preserve_logs('fedora33-image')
sh (
label: "Run cloud cleaner just in case something failed",
script: "schutzbot/run_cloud_cleaner.sh"
)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, thanks! I added those lines before the cloud cleaner was introduced.

# Current $PWD is inside /tmp, there may not be enough space for an image.
# Let's use a bigger temporary directory for this operation.
BIG_TEMP_DIR=/var/lib/osbuild-composer-tests
sudo rm -rf "${BIG_TEMP_DIR}" || true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the || true part? Shouldn't it be fine with non-existing or empty directory because of -f?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right, didn't know about this -f's behavior.

@ondrejbudai ondrejbudai mentioned this pull request Nov 11, 2020
Merged
Copy link
Contributor

@msehnout msehnout left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My previous comments are not blockers and CI is all green. Thank you @ondrejbudai !

@msehnout msehnout merged commit a07783c into osbuild:master Nov 11, 2020
@ondrejbudai ondrejbudai deleted the fedora33-testing branch February 2, 2021 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants