[TEST] Build the fully airgapped on bare metal machines for release upgrade testing #967
Comments
|
These are the initial concept, need further discussion and update by time. Pipeline design concept
Pipeline request
|
|
Just come up the initial idea of the fully airgapped infrastructure diagram for further discussion
|
I think in general this looks pretty good 😄 👍 I would only mention that there would be much more benefit to separating out |
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
* provision resilant docker-registry * provision rancher that utilizes registry Resolves: harvester/tests#967
|
The Rancher instance, docker registry, DNS, name server implementation would be implemented in #942 |
|
Another idea is we can consider moving the
|
|
|
|
There is a slight blocker at: Means we will need to bake in additional logic in the pipeline to compensate for that bug. |
|
We're currently encountering something that we will need to redesign logic for. Seemingly related to something within Jenkins / Groovy : Investigating.... |
|
Even pivoting, now we are hitting a limitation of the script string... |
|
Was able to reduce it, but still it is too big: We'll need to pivot to something else w/ jobdls plugin... |
|
Based on some more investigation, I'm not entirely sure all integrations can be within a single pipeline job... Where the JobDSL plugin, possibly is calling: That it still ultimately in that, where we read a file... it seems to do from the jobdsl repo: Ultimately, reading the file into a String, so we're back in the same place we would be even if we defined it as: ^ because that also just yields a "String". Though... I'm not entirely sure about this.
So that then scales, all our jobs from 1 (that provisions all integrations)... to needing to be "multiple"... one per airgap integration... possibly to just avoid this Groovy limitation of the string being too big 😅 ... |
|
With: But still, the underlying error is now present as it's just seeing the script function as being too large in general... Investigating the new error of: |
|
Trying: Specifically: As suggested: Has led to the same result... |
|
Timeboxing... It's really not working... getting the
With the idea, we'd give a specific script path like: to split apart script / pipeline May pivot back to cpsScm -> scm -> git & scriptPath... The Jenkins JobDSL Plugin Docs + Jenkins Docs don't seem to have "dynamic" examples... |
|
Re-investigating the environment. |
|
It's difficult to overcome the "environment variable" limit... there are probably still some more ways around it... Pivoted instead to: With:
Methods, that get around the "method too large" error, since we pull then the logic of the multiple parallel running stages out into two separate methods -> one to build out the local.tfvars for the respective service, since we can't leverage the default environment If we could, we'd avoid an entire parallel stage that's needed to build out the local.tfvars. That leverages the second bullet point from: |
|
What ended up working for interpolation and also syncing with the needed style of the local.tfvars for each service is using the def string = $/
string-goes-here
${params.interpolation}
other things like newline\n
/$To seemingly help |
|
Currently, testing pipeline on staging... |
|
So, the temporary loop to do a few iterations when we shift the VM NIC/NAD and run a separate playbook for airgap seems to help buffer:
But the second iteration we're still seeing: Implementing an arbitrary timeout of sleep, prior to the next iteration so the VM can make the DHCP request outbound and get the IPv4 assigned regardless of network, will need to happen. |
|
Something is happening and the investigating... |
|
This was an issue: Now fixed from Sunday's update. So cert-manager & nginx are present. Additionally all: rescue:"rescue" blocks in Ansible, will re-trigger an |
|
While we are still waiting to implement Stage 7 & Stage 8 - we now have a new lab, where Stage 6's last part to allow for our Seeder to run AirGap can be a reality once more infrastructure work is done. |
|
For reference, this was leveraged successfully, but outside of our lab env - to provision our needed integrations for v1.4.0 testing.
|
What's the test to develop? Please describe
The epic issue was created to track the progress of building a fully airgapped environment for the upgrade testing in each Harvester release candidate.
Since the current fully airgapped environment was built upon ipxe-example virtual machines inside another powerful virtual machine. This would usually cause some unexpected upgrade failure due to performance or resource bottleneck.
Scope
Prerequisite
Any prerequisite environment and pre-condition required for this test.
Provide test case dependency here if any.
The fully airgapped environment requires the following components:
Test case reference
Roles
Harvester cluster on bare metal machines
On the same bare metal machine host VMs (tenative)
On the same VM (tenative)
Describe the items of the test development (DoD, definition of done) you'd like
Stage 1 Design discussion
Stage 2 Build Out Baseline Provision Harvester Airgap Pipeline
Stage 3 Convert Vagrant Logic To Terraform VMs (per service) For Harvester
Stage 4 Build Out All Airgap Integrations Jenkins Pipeline Utilities
Stage 5 Implement All Airgap Integrations Jenkins Pipeline on Staging
Stage 6 Implement additional pipeline parameters to baseline Harvester Airgap Pipeline
Stage 7 Move Both Pipelines To Prod
Stage 8 Implement New Pipeline To Run Subsection Of Tests Against Airgap
The text was updated successfully, but these errors were encountered: