diff --git a/content/en/docs/release-oversight/disruption-testing/data-architecture.md b/content/en/docs/release-oversight/disruption-testing/data-architecture.md index e45226fa..21cb1006 100644 --- a/content/en/docs/release-oversight/disruption-testing/data-architecture.md +++ b/content/en/docs/release-oversight/disruption-testing/data-architecture.md @@ -1,6 +1,7 @@ --- title: "Architecture Data Flow" description: A high level look at how the disruption historical data is gathered and updated. +weight: 1 --- ### Resources diff --git a/content/en/docs/release-oversight/disruption-testing/job-primer.md b/content/en/docs/release-oversight/disruption-testing/job-primer.md new file mode 100644 index 00000000..b392c077 --- /dev/null +++ b/content/en/docs/release-oversight/disruption-testing/job-primer.md @@ -0,0 +1,99 @@ +--- +title: "Job Primer" +description: Job Primer ci-tool used to generate job names for Big Query. +--- + +## Overview + +{{% alert title="⚠️ NOTE" color="warning" %}} +In `Job Primer` a job name is very important. Please make sure that the job names contain correct information. ([see options below](#naming-convention)) +{{% /alert %}} + +[JobPrimer](https://github.com/openshift/ci-tools/tree/master/pkg/jobrunaggregator/jobtableprimer) is the `ci-tool` that is used to populate the `BigQuery` `Jobs` table. The `Jobs` table is what dictates the periodic jobs are grabbed during `disruption` data gathering. Currently this tool is ran manually. + +## High Level Diagram + +{{< inlineSVG file="/static/job_primer_diagram.svg" >}} + +### How The Data Flows + +1. We first look at the `origin/release` repo to gather a list of the current release jobs that were created. The below command is ran to look through the current configuration and generate the job names. + + ```sh + ./job-run-aggregator generate-job-names > pkg/jobrunaggregator/jobtableprimer/generated_job_names.txt + ``` + +1. That `generated_jobs_names.txt` is then committed to the repo. + + **You must then rebuild the binary so the newly generated list is correctly embedded.** + +1. We then create the jobs in the BigQuery table by running the `prime-job-table` command. This will use the embedded `generated_jobs_names.txt` data and generate the `Jobs` rows based off of the naming convention (see below). After which the `Jobs` table should be updated with the latest jobs. + + ```sh + ./job-run-aggregator prime-job-table + ``` + +### Naming Convention + +Please make sure your job names follow the convention defined below. All job names must include addiqute information to allow proper data aggragetion. + +{{% pageinfo color="primary" %}} + +- Platform: + - aws, gcp, azure, etc... +- Architecture: (default: `amd64`) + - arm64, ppc64le, s390x +- Upgrade: (default: `assumes NOT upgrade`) + - upgrade +- Network: (default: `sdn && ipv4`) + - sdn, ovn + - ipv6, ipv4 +- Topology: (default: `assumes ha`) + - single +- Serial: (default: `assumes parallel`) + - serial + +{{% /pageinfo %}} + +{{% card-code header="[Code Location](https://github.com/openshift/ci-tools/blob/659fc3fed6ebe7ed7fb0bde25330fe2f47e20d0b/pkg/jobrunaggregator/jobtableprimer/job_typer.go#L13-L114)" %}} + +```go +func newJob(name string) *jobRowBuilder { + platform := "" + switch { + case strings.Contains(name, "gcp"): + platform = gcp + case strings.Contains(name, "aws"): + platform = aws + case strings.Contains(name, "azure"): + platform = azure + case strings.Contains(name, "metal"): + platform = metal + case strings.Contains(name, "vsphere"): + platform = vsphere + case strings.Contains(name, "ovirt"): + platform = ovirt + case strings.Contains(name, "openstack"): + platform = openstack + case strings.Contains(name, "libvirt"): + platform = libvirt + } + + architecture := "" + switch { + case strings.Contains(name, "arm64"): + architecture = arm64 + case strings.Contains(name, "ppc64le"): + architecture = ppc64le + case strings.Contains(name, "s390x"): + architecture = s390x + default: + architecture = amd64 + } + +... + + +``` + +{{% /card-code %}} diff --git a/layouts/partials/head-css.html b/layouts/partials/head-css.html index dedf6d07..8ea8cd19 100644 --- a/layouts/partials/head-css.html +++ b/layouts/partials/head-css.html @@ -38,4 +38,11 @@ tr.shown td.details-control { background: url('https://datatables.net/examples/resources/details_close.png') no-repeat center center; } + +.card-body > .highlight, +.card-body > .highlight pre { + overflow-x: auto; + max-width: 100%; + margin: 0; +} diff --git a/layouts/shortcodes/card-code.html b/layouts/shortcodes/card-code.html index c2e310a3..59d8d814 100644 --- a/layouts/shortcodes/card-code.html +++ b/layouts/shortcodes/card-code.html @@ -1,10 +1,10 @@ -
+
{{- with $.Get "header" -}}
{{- $.Get "header" | markdownify -}}
{{end}} -
+
{{ $.Inner }}
diff --git a/static/job_primer_diagram.svg b/static/job_primer_diagram.svg new file mode 100644 index 00000000..ade51f10 --- /dev/null +++ b/static/job_primer_diagram.svg @@ -0,0 +1,58 @@ + + + + + + + GCP + + Big Query + openshift-ci-data-analysis + + + + + Jobs Table + + + + + Job Primer + + GitHub + + + + + openshift/release + + + + + + + + + + + + Fetch List of Jobs + Generate list text file + + Parse job names + (network, topology, type) + + + + + + + Update Jobs Table + + + + + + \ No newline at end of file