Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix cpu partitioning struct tag/field #5227

Merged

Conversation

carbonin
Copy link
Member

@carbonin carbonin commented May 18, 2023

The struct tag for cpu partitioning was using but in order for the
install config to be rendered correctly in all cases it need the yaml
tag like the other fields.

Without the yaml tag cpu partitioning was being included in the install
config even if it was empty with an invlid name.

This meant that all 4.10 installs always failed to prepare for
installation with an event like:

Failed to prepare the installation due to an unexpected error: failed
generating install config for cluster
1583dc9b-e059-48fc-96b9-5e711d1de6b1: error running openshift-install
manifests, level=fatal msg=failed to fetch Master Machines: failed to
load asset "Install Config": failed to unmarshal install-config.yaml:
failed to parse first occurence of unknown field: error unmarshaling
JSON: while decoding JSON: json: unknown field "cpupartitioning" : exit
status 1. Please retry later

To correct this while also maintaining the current overrides behavior
the field name is updated to match the API name (cpuPartitioningMode)
and the json struct tag is changed to yaml.

This allows for overrides to work in a case-insensitive way (as is
supported currently for all other fields) as well as for the install
config yaml to render properly.

NOTE:

This field name is intentionally different than the one in the openshift
installer repo. This allows for us to take advantage of the
case-insensitive matching allowed by json unmarshaling while also using
the correct name.

Originally committed in #5207

List all the issues related to this PR

https://issues.redhat.com/browse/OCPBUGS-13310

  • New Feature
  • Enhancement
  • Bug fix
  • Tests
  • Documentation
  • CI/CD

What environments does this code impact?

  • Automation (CI, tools, etc)
  • Cloud
  • Operator Managed Deployments
  • None

How was this code tested?

  • assisted-test-infra environment
  • dev-scripts environment
  • Reviewer's test appreciated
  • Waiting for CI to do a full test run
  • Manual (Elaborate on how it was tested)
  • No tests needed

Checklist

  • Title and description added to both, commit and PR.
  • Relevant issues have been associated (see [CONTRIBUTING] guide)
  • This change does not require a documentation update (docstring, docs, README, etc)
  • Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

  • Are the title and description (in both PR and commit) meaningful and clear?
  • Is there a bug required (and linked) for this change?
  • Should this PR be backported?

@carbonin carbonin requested review from omertuc and filanov May 18, 2023 16:27
@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 18, 2023
@openshift-ci-robot
Copy link

@carbonin: This pull request references Jira Issue OCPBUGS-13310, which is invalid:

  • expected the bug to target the "4.14.0" version, but it targets "4.13.z" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

The struct tag for cpu partitioning was using json, but it should have been yaml.

This lead to cpu partitioning being included in the default install config, but it is only available in later versions of openshift.

This meant that all 4.10 installs always failed to prepare for installation with an event like:

Failed to prepare the installation due to an unexpected error: failed
generating install config for cluster
1583dc9b-e059-48fc-96b9-5e711d1de6b1: error running openshift-install
manifests, level=fatal msg=failed to fetch Master Machines: failed to
load asset "Install Config": failed to unmarshal install-config.yaml:
failed to parse first occurence of unknown field: error unmarshaling
JSON: while decoding JSON: json: unknown field "cpupartitioning" : exit
status 1. Please retry later

List all the issues related to this PR

https://issues.redhat.com/browse/OCPBUGS-13310

  • New Feature
  • Enhancement
  • Bug fix
  • Tests
  • Documentation
  • CI/CD

What environments does this code impact?

  • Automation (CI, tools, etc)
  • Cloud
  • Operator Managed Deployments
  • None

How was this code tested?

  • assisted-test-infra environment
  • dev-scripts environment
  • Reviewer's test appreciated
  • Waiting for CI to do a full test run
  • Manual (Elaborate on how it was tested)
  • No tests needed

Checklist

  • Title and description added to both, commit and PR.
  • Relevant issues have been associated (see [CONTRIBUTING] guide)
  • This change does not require a documentation update (docstring, docs, README, etc)
  • Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

  • Are the title and description (in both PR and commit) meaningful and clear?
  • Is there a bug required (and linked) for this change?
  • Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@carbonin
Copy link
Member Author

/test ?

@openshift-ci openshift-ci bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label May 18, 2023
@openshift-ci
Copy link

openshift-ci bot commented May 18, 2023

@carbonin: The following commands are available to trigger required jobs:

  • /test e2e-agent-compact-ipv4
  • /test edge-assisted-operator-catalog-publish-verify
  • /test edge-ci-index
  • /test edge-e2e-ai-operator-ztp
  • /test edge-e2e-ai-operator-ztp-sno-day2-workers
  • /test edge-e2e-ai-operator-ztp-sno-day2-workers-late-binding
  • /test edge-e2e-metal-assisted
  • /test edge-e2e-metal-assisted-4-10
  • /test edge-e2e-metal-assisted-4-11
  • /test edge-e2e-metal-assisted-4-12
  • /test edge-e2e-metal-assisted-4-8
  • /test edge-e2e-metal-assisted-4-9
  • /test edge-e2e-metal-assisted-cnv
  • /test edge-e2e-metal-assisted-lvm
  • /test edge-e2e-metal-assisted-ocs
  • /test edge-e2e-metal-assisted-odf
  • /test edge-images
  • /test edge-lint
  • /test edge-subsystem-aws
  • /test edge-subsystem-kubeapi-aws
  • /test edge-unit-test
  • /test edge-verify-generated-code
  • /test images
  • /test mce-images

The following commands are available to trigger optional jobs:

  • /test e2e-agent-ha-dualstack
  • /test e2e-agent-sno-ipv6
  • /test edge-e2e-ai-operator-ztp-3masters
  • /test edge-e2e-ai-operator-ztp-capi
  • /test edge-e2e-ai-operator-ztp-compact-day2-masters
  • /test edge-e2e-ai-operator-ztp-compact-day2-workers
  • /test edge-e2e-ai-operator-ztp-disconnected
  • /test edge-e2e-ai-operator-ztp-hypershift-zero-nodes
  • /test edge-e2e-ai-operator-ztp-ipv4v6-3masters
  • /test edge-e2e-ai-operator-ztp-ipv4v6-3masters-ocp-411
  • /test edge-e2e-ai-operator-ztp-ipv4v6-sno
  • /test edge-e2e-ai-operator-ztp-ipv4v6-sno-ocp-411
  • /test edge-e2e-ai-operator-ztp-multiarch-3masters-ocp-411
  • /test edge-e2e-ai-operator-ztp-multiarch-sno-ocp-411
  • /test edge-e2e-ai-operator-ztp-sno-day2-masters
  • /test edge-e2e-ai-operator-ztp-sno-day2-workers-ignitionoverride
  • /test edge-e2e-metal-assisted-4-13
  • /test edge-e2e-metal-assisted-4-14
  • /test edge-e2e-metal-assisted-day2
  • /test edge-e2e-metal-assisted-day2-arm-workers
  • /test edge-e2e-metal-assisted-day2-single-node
  • /test edge-e2e-metal-assisted-ipv4v6
  • /test edge-e2e-metal-assisted-ipv6
  • /test edge-e2e-metal-assisted-kube-api-late-binding-single-node
  • /test edge-e2e-metal-assisted-kube-api-late-unbinding-ipv4-single-node
  • /test edge-e2e-metal-assisted-kube-api-net-suite
  • /test edge-e2e-metal-assisted-mce
  • /test edge-e2e-metal-assisted-mce-4-10
  • /test edge-e2e-metal-assisted-mce-sno
  • /test edge-e2e-metal-assisted-none
  • /test edge-e2e-metal-assisted-onprem
  • /test edge-e2e-metal-assisted-single-node
  • /test edge-e2e-metal-assisted-static-ip-suite
  • /test edge-e2e-metal-assisted-tang
  • /test edge-e2e-metal-assisted-tpmv2
  • /test edge-e2e-metal-assisted-upgrade-agent
  • /test edge-e2e-nutanix-assisted
  • /test edge-e2e-nutanix-assisted-2workers
  • /test edge-e2e-vsphere-assisted
  • /test edge-e2e-vsphere-assisted-4-12
  • /test edge-e2e-vsphere-assisted-4-13
  • /test edge-e2e-vsphere-assisted-umn
  • /test edge-push-pr-image
  • /test push-pr-image

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-openshift-assisted-service-master-e2e-agent-compact-ipv4
  • pull-ci-openshift-assisted-service-master-edge-ci-index
  • pull-ci-openshift-assisted-service-master-edge-e2e-ai-operator-ztp
  • pull-ci-openshift-assisted-service-master-edge-e2e-metal-assisted
  • pull-ci-openshift-assisted-service-master-edge-images
  • pull-ci-openshift-assisted-service-master-edge-lint
  • pull-ci-openshift-assisted-service-master-edge-subsystem-aws
  • pull-ci-openshift-assisted-service-master-edge-subsystem-kubeapi-aws
  • pull-ci-openshift-assisted-service-master-edge-unit-test
  • pull-ci-openshift-assisted-service-master-edge-verify-generated-code
  • pull-ci-openshift-assisted-service-master-images
  • pull-ci-openshift-assisted-service-master-mce-images

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot requested a review from rccrdpccl May 18, 2023 16:28
@openshift-ci
Copy link

openshift-ci bot commented May 18, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: carbonin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 18, 2023
@openshift-ci-robot
Copy link

@carbonin: This pull request references Jira Issue OCPBUGS-13310, which is invalid:

  • expected the bug to target the "4.14.0" version, but it targets "4.13.z" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

The struct tag for cpu partitioning was using json, but it should have been yaml.

This lead to cpu partitioning being included in the default install config, but it is only available in later versions of openshift.

This meant that all 4.10 installs always failed to prepare for installation with an event like:

Failed to prepare the installation due to an unexpected error: failed
generating install config for cluster
1583dc9b-e059-48fc-96b9-5e711d1de6b1: error running openshift-install
manifests, level=fatal msg=failed to fetch Master Machines: failed to
load asset "Install Config": failed to unmarshal install-config.yaml:
failed to parse first occurence of unknown field: error unmarshaling
JSON: while decoding JSON: json: unknown field "cpupartitioning" : exit
status 1. Please retry later

Originally committed in #5207

List all the issues related to this PR

https://issues.redhat.com/browse/OCPBUGS-13310

  • New Feature
  • Enhancement
  • Bug fix
  • Tests
  • Documentation
  • CI/CD

What environments does this code impact?

  • Automation (CI, tools, etc)
  • Cloud
  • Operator Managed Deployments
  • None

How was this code tested?

  • assisted-test-infra environment
  • dev-scripts environment
  • Reviewer's test appreciated
  • Waiting for CI to do a full test run
  • Manual (Elaborate on how it was tested)
  • No tests needed

Checklist

  • Title and description added to both, commit and PR.
  • Relevant issues have been associated (see [CONTRIBUTING] guide)
  • This change does not require a documentation update (docstring, docs, README, etc)
  • Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

  • Are the title and description (in both PR and commit) meaningful and clear?
  • Is there a bug required (and linked) for this change?
  • Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@carbonin carbonin changed the title OCPBUGS-13310: Fix cpu partitioning struct tag Fix cpu partitioning struct tag May 18, 2023
@openshift-ci-robot openshift-ci-robot removed jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 18, 2023
@openshift-ci-robot
Copy link

@carbonin: No Jira issue is referenced in the title of this pull request.
To reference a jira issue, add 'XYZ-NNN:' to the title of this pull request and request another refresh with /jira refresh.

In response to this:

The struct tag for cpu partitioning was using json, but it should have been yaml.

This lead to cpu partitioning being included in the default install config, but it is only available in later versions of openshift.

This meant that all 4.10 installs always failed to prepare for installation with an event like:

Failed to prepare the installation due to an unexpected error: failed
generating install config for cluster
1583dc9b-e059-48fc-96b9-5e711d1de6b1: error running openshift-install
manifests, level=fatal msg=failed to fetch Master Machines: failed to
load asset "Install Config": failed to unmarshal install-config.yaml:
failed to parse first occurence of unknown field: error unmarshaling
JSON: while decoding JSON: json: unknown field "cpupartitioning" : exit
status 1. Please retry later

Originally committed in #5207

List all the issues related to this PR

https://issues.redhat.com/browse/OCPBUGS-13310

  • New Feature
  • Enhancement
  • Bug fix
  • Tests
  • Documentation
  • CI/CD

What environments does this code impact?

  • Automation (CI, tools, etc)
  • Cloud
  • Operator Managed Deployments
  • None

How was this code tested?

  • assisted-test-infra environment
  • dev-scripts environment
  • Reviewer's test appreciated
  • Waiting for CI to do a full test run
  • Manual (Elaborate on how it was tested)
  • No tests needed

Checklist

  • Title and description added to both, commit and PR.
  • Relevant issues have been associated (see [CONTRIBUTING] guide)
  • This change does not require a documentation update (docstring, docs, README, etc)
  • Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

  • Are the title and description (in both PR and commit) meaningful and clear?
  • Is there a bug required (and linked) for this change?
  • Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@carbonin
Copy link
Member Author

/test edge-e2e-metal-assisted-4-10

@codecov
Copy link

codecov bot commented May 18, 2023

Codecov Report

Merging #5227 (492ff1b) into master (9a7bef9) will increase coverage by 0.10%.
The diff coverage is n/a.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #5227      +/-   ##
==========================================
+ Coverage   67.46%   67.57%   +0.10%     
==========================================
  Files         218      218              
  Lines       32683    32775      +92     
==========================================
+ Hits        22051    22149      +98     
+ Misses       8650     8644       -6     
  Partials     1982     1982              

see 1 file with indirect coverage changes

@carbonin
Copy link
Member Author

/test edge-e2e-metal-assisted-4-10

@omertuc
Copy link
Contributor

omertuc commented May 18, 2023

I don't understand what this have to do with JSON vs YAML and why do we still keep both the JSON and YAML tags after this change?

@carbonin
Copy link
Member Author

The failure was caused because we serialized a struct with an empty cpu partitioning field into yaml, but without the yaml tag.

This left us with a field like cpupartitioning: "" in the yaml which isn't recognized by the openshift installer.

All the other fields don't have json tags but this field is the only one where the serialized name doesn't match the field name so it also needs to be specified to ensure it's consistent. We use json deserialization to do the install config overrides.

That said, after doing a brief test with this I noticed that all the json field names are capitalized because we are not currently providing the json struct tags. I'll change this one to agree with the others for consistency.

@carbonin
Copy link
Member Author

/hold

Going to ensure consistency for overrides

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 19, 2023
@carbonin
Copy link
Member Author

/unhold

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 19, 2023
The struct tag for cpu partitioning was using but in order for the
install config to be rendered correctly in all cases it need the yaml
tag like the other fields.

Without the yaml tag cpu partitioning was being included in the install
config even if it was empty with an invlid name.

This meant that all 4.10 installs always failed to prepare for
installation with an event like:

```
Failed to prepare the installation due to an unexpected error: failed
generating install config for cluster
1583dc9b-e059-48fc-96b9-5e711d1de6b1: error running openshift-install
manifests, level=fatal msg=failed to fetch Master Machines: failed to
load asset "Install Config": failed to unmarshal install-config.yaml:
failed to parse first occurence of unknown field: error unmarshaling
JSON: while decoding JSON: json: unknown field "cpupartitioning" : exit
status 1. Please retry later
```

To correct this while also maintaining the current overrides behavior
the field name is updated to match the API name (cpuPartitioningMode)
and the json struct tag is changed to yaml.

This allows for overrides to work in a case-insensitive way (as is
supported currently for all other fields) as well as for the install
config yaml to render properly.

NOTE:

This field name is intentionally different than the one in the openshift
installer repo. This allows for us to take advantage of the
case-insensitive matching allowed by json unmarshaling while also using
the correct name.

https://issues.redhat.com/browse/OCPBUGS-13310
@carbonin carbonin changed the title Fix cpu partitioning struct tag Fix cpu partitioning struct tag/field May 19, 2023
@carbonin
Copy link
Member Author

/test edge-e2e-metal-assisted-4-10

@carbonin
Copy link
Member Author

This should be ready for final review whenever you get the chance @omertuc

@omertuc
Copy link
Contributor

omertuc commented May 19, 2023

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 19, 2023
@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD fa5e039 and 2 for PR HEAD 492ff1b in total

@carbonin
Copy link
Member Author

/retest

@openshift-ci
Copy link

openshift-ci bot commented May 19, 2023

@carbonin: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit d2dfbcc into openshift:master May 19, 2023
15 checks passed
@osherdp
Copy link
Contributor

osherdp commented May 21, 2023

/cherry-pick cloud_hotfix_releases

@openshift-cherrypick-robot

@osherdp: new pull request created: #5231

In response to this:

/cherry-pick cloud_hotfix_releases

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@carbonin carbonin deleted the yaml_cpu_partitioning branch May 22, 2023 12:28
danielerez pushed a commit to danielerez/assisted-service that referenced this pull request Oct 15, 2023
The struct tag for cpu partitioning was using but in order for the
install config to be rendered correctly in all cases it need the yaml
tag like the other fields.

Without the yaml tag cpu partitioning was being included in the install
config even if it was empty with an invlid name.

This meant that all 4.10 installs always failed to prepare for
installation with an event like:

```
Failed to prepare the installation due to an unexpected error: failed
generating install config for cluster
1583dc9b-e059-48fc-96b9-5e711d1de6b1: error running openshift-install
manifests, level=fatal msg=failed to fetch Master Machines: failed to
load asset "Install Config": failed to unmarshal install-config.yaml:
failed to parse first occurence of unknown field: error unmarshaling
JSON: while decoding JSON: json: unknown field "cpupartitioning" : exit
status 1. Please retry later
```

To correct this while also maintaining the current overrides behavior
the field name is updated to match the API name (cpuPartitioningMode)
and the json struct tag is changed to yaml.

This allows for overrides to work in a case-insensitive way (as is
supported currently for all other fields) as well as for the install
config yaml to render properly.

NOTE:

This field name is intentionally different than the one in the openshift
installer repo. This allows for us to take advantage of the
case-insensitive matching allowed by json unmarshaling while also using
the correct name.

https://issues.redhat.com/browse/OCPBUGS-13310
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants