Use Qemu emulation on GHA for Vagrant tests #503

mikedep333 · 2020-12-04T17:22:13Z

Adapted from:
https://github.com/pulp/pulplift/pull/66
"RFC: Testing nested Virtualization"

Implementation Includes:
1. Upgrade Qemu from 4.4 to 5.2 from our PPA to address a
severe bug affecting CentOS 7 guests, they could not even validate
SSL certs with curl / yum or create the Pulp postgres database.
2. Upgrade the rest of the virtualization stack on Ubuntu
3. Address the EL8 vagrant-sshfs workaround task failing due to
a GPG signature mismatch.
4. Workaround a bug with VM storage on the newer virtualization
stack.
5. Switch the boxes used on CentOS 7 for more recent updates.

workaround #8095: FIPS failure in geerlingguy.postgresql by
using an old version.
https://pulp.plan.io/issues/8095

workaround #7993:
pulp_installer fails to create the database on EL7 when LANG=C.UTF-8
https://pulp.plan.io/issues/7993

fixes: #7884
Move the pulp_installer Vagrant tests off Travis
https://pulp.plan.io/issues/7884

To ensure that git recognizes that they are existing ones rather than new ones. This would likely be a problem as we are changing them heavily in the next commit. [noissue]

fao89 · 2021-02-08T21:49:28Z

.github/scripts/script.sh

+fi
+
+sed -i -e 's/memory: 10500/memory: 5500/g' vagrant/boxes.d/*
+sed -i -e 's/cpus: 4/cpus: 2/g' vagrant/boxes.d/*


does GHA not handle 4?

GHA only has 2 virtual CPUs, just like Travis.

These 2 lines only have any effect if when we run pulp2-nightly-pulp3-source-centos7 on CI, which we currently do not do.

fao89 · 2021-02-08T21:53:40Z

.github/workflows/pull_request.yaml

@@ -99,3 +99,31 @@ jobs:
          PY_COLORS: '1'
          ANSIBLE_FORCE_COLOR: '1'
        shell: bash
+  vagrant:


I'm wondering if we should run for branches instead of PRs

So we'd test immediately after merging?

I'm open to that. I figured we'd adjust the schedule according to dev feedback. Or look into enabling merging while these tests still have not completed yet.

sounds good to me

fao89 · 2021-02-08T21:56:46Z

roles/pulp_api/templates/pulpcore-api.service.j2

+# timeout 90 is needed Pulp to service its 1st request on extremely slow
+# machines, such as qemu-emulated 2-core machines


I think it needs a new changelog entry

Yeah, I agree. I'll make it a separate commit.

@gerrod3 this is the change I mentioned

fao89 · 2021-02-08T21:57:09Z

roles/pulp_content/templates/pulpcore-content.service.j2

+# timeout 90 is needed Pulp to service its 1st request on extremely slow
+# machines, such as qemu-emulated 2-core machines


I think it needs a new changelog entry

I'll combine it with pulpcore-api's commit/changelog entry, yeah. It may not even be needed for pulpcore-content, but I did it at the same time.

@gerrod3 same here

roles/pulp_health_check/tasks/main.yml

fao89 · 2021-02-08T21:59:08Z

roles/pulp_health_check/tasks/main.yml

 - name: Ensure Pulp is up and healthy
  uri:
    url: "http://{{ __pulp_default_host if pulp_api_bind.startswith('unix:') else pulp_api_bind }}/pulp/api/v3/status/"
    status_code: "200"
    unix_socket: "{{ pulp_api_bind | regex_replace('^unix:', '') if pulp_api_bind.startswith('unix:') else omit }}"
+    # Alternate between 30 second timeouts and 5 second timeouts when handling the situation of


why these magic numbers?

30 is the default. It handles hypothetical extremely long times for the webserver to produce the page.

We intermix it with 5 second waits, because if the server isn't up yet and suddenly is, a 5 second wait usually is sufficient I would think. Just my guestimate.

If your question is about why I combine a short and a long time out together, the purpose is 2:

Is to lower the total amount of time that the CI or the user may wait on the health check when we have a "connection timed out". 38 tries * (30 seconds + 6 second wait) + 37 tries * (5 seconds + 6 second wait) = 29 minutes, rather than 45 minutes.

Still passing a theoretical 29.9 second wait.

Ensuring a large number of checks over a medium period of time (75 tries * 6 second wait = 7.5 min) when the server does "connection refused," which takes under a second.

it was more from where they came from, thank you for all the explanation

Solution: Raise the the Pulp server's gunicorn worker timeout to 90 seconds. fixes: #8228 Pulp Connection Timed Out on slow emulated machines https://pulp.plan.io/issues/8228

Adapted from: pulp/pulplift#66 "RFC: Testing nested Virtualization" Implementation Includes: 1. Upgrade Qemu from 4.4 to 5.2 from our PPA to address a severe bug affecting CentOS 7 guests, they could not even validate SSL certs with curl / yum or create the Pulp postgres database. 2. Upgrade the rest of the virtualization stack on Ubuntu 3. Address the EL8 vagrant-sshfs workaround task failing due to a GPG signature mismatch. 4. Workaround a bug with VM storage on the newer virtualization stack. 5. Switch the boxes used on CentOS 7 for more recent updates. 6. Reducing how long the pulp health check may take, particularly when there is a connection timed out. workaround #8095: FIPS failure in geerlingguy.postgresql by using an old version. https://pulp.plan.io/issues/8095 workaround #7993: pulp_installer fails to create the database on EL7 when LANG=C.UTF-8 https://pulp.plan.io/issues/7993 fixes: #7884 Move the pulp_installer Vagrant tests off Travis https://pulp.plan.io/issues/7884

fao89

Thank you Mike!

pulpbot · 2021-02-09T12:56:01Z

Attached issue: https://pulp.plan.io/issues/8228

Attached issue: https://pulp.plan.io/issues/7884

mikedep333 force-pushed the qemu-emulation branch 11 times, most recently from 4a7e9a8 to c86a2f6 Compare December 4, 2020 20:00

mikedep333 added the WorkInProgress label Dec 4, 2020

mikedep333 force-pushed the qemu-emulation branch 18 times, most recently from f5f4efe to 95794c4 Compare December 9, 2020 01:07

mikedep333 force-pushed the qemu-emulation branch 7 times, most recently from ae082c4 to 64c8add Compare January 25, 2021 18:01

mikedep333 changed the title ~~Test Qemu emulation~~ Use Qemu emulation for Vagrant Tests Jan 25, 2021

mikedep333 changed the title ~~Use Qemu emulation for Vagrant Tests~~ Use Qemu emulation on GHA for Vagrant tests Jan 25, 2021

mikedep333 removed the WorkInProgress label Jan 25, 2021

mikedep333 force-pushed the qemu-emulation branch 3 times, most recently from 3742f78 to 3c0df58 Compare January 26, 2021 15:29

mikedep333 requested a review from dkliban January 26, 2021 16:17

mikedep333 force-pushed the qemu-emulation branch 2 times, most recently from 55ef148 to c7fa77b Compare January 27, 2021 20:46

mikedep333 force-pushed the qemu-emulation branch from c7fa77b to 2d17f86 Compare February 4, 2021 20:14

Move the Travis script files to .github/scripts/

61e8f74

To ensure that git recognizes that they are existing ones rather than new ones. This would likely be a problem as we are changing them heavily in the next commit. [noissue]

mikedep333 force-pushed the qemu-emulation branch 2 times, most recently from 9312900 to c47ac0b Compare February 8, 2021 20:13

fao89 reviewed Feb 8, 2021

View reviewed changes

Problem: Pulp Connection Timed Out on slow emulated machines

c800f0a

Solution: Raise the the Pulp server's gunicorn worker timeout to 90 seconds. fixes: #8228 Pulp Connection Timed Out on slow emulated machines https://pulp.plan.io/issues/8228

mikedep333 force-pushed the qemu-emulation branch from c47ac0b to 00d820c Compare February 8, 2021 22:50

mikedep333 force-pushed the qemu-emulation branch from 00d820c to 13db1da Compare February 9, 2021 01:21

pulp deleted a comment from pulpbot Feb 9, 2021

fao89 approved these changes Feb 9, 2021

View reviewed changes

mikedep333 merged commit 011c2cb into pulp:master Feb 9, 2021

mikedep333 deleted the qemu-emulation branch February 9, 2021 15:09

Use Qemu emulation on GHA for Vagrant tests #503

Use Qemu emulation on GHA for Vagrant tests #503

mikedep333 commented Dec 4, 2020 •

edited

fao89 Feb 8, 2021

mikedep333 Feb 8, 2021

fao89 Feb 8, 2021

mikedep333 Feb 8, 2021

fao89 Feb 9, 2021

fao89 Feb 8, 2021

mikedep333 Feb 8, 2021

fao89 Mar 1, 2021

fao89 Feb 8, 2021

mikedep333 Feb 8, 2021 •

edited

fao89 Mar 1, 2021

fao89 Feb 8, 2021

mikedep333 Feb 8, 2021 •

edited

fao89 Feb 9, 2021

fao89 left a comment

pulpbot commented Feb 9, 2021

		# timeout 90 is needed Pulp to service its 1st request on extremely slow
		# machines, such as qemu-emulated 2-core machines

Use Qemu emulation on GHA for Vagrant tests #503

Use Qemu emulation on GHA for Vagrant tests #503

Conversation

mikedep333 commented Dec 4, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikedep333 Feb 8, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikedep333 Feb 8, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fao89 left a comment

Choose a reason for hiding this comment

pulpbot commented Feb 9, 2021

mikedep333 commented Dec 4, 2020 •

edited

mikedep333 Feb 8, 2021 •

edited

mikedep333 Feb 8, 2021 •

edited