MGMT-16262: Part1 of implementing usage of custom OCP releases

[danmanor]

In General

This PR is part 1 of the Implementation of this enhancement proposal - https://github.com/openshift/assisted-service/blob/master/docs/enhancements/dynamic-fetching-of-openshift-versions.md, which is enabling usage of dynamic openshift releases.

Main flow

This part is about creating a periodic utility that updates current OCP releases in postgresql DB, for later use of assisted service -

The main changes are:

1. main fetches new configuration called RELEASE_SOURCES which specifies which OCP releases we want to keep in the DB and support (new table called release_images) as well as from which channel, what CPU architectures, etc. Then it starts a new go routine that runs periodically.
2. This go routine fetches releases from the API according to the configuration, and OCP versions support levels from another API.
3. It labels the default release (latest stable release), supporting levels.
4. Adds release images from the configuration (with precedence in case of conflicts).
5. Replaces this data with the old one in the DB.

Plan forward (part 2)

This part is about retrieving the new list of supported openshift versions in endpoint /api/assisted-install/v2/openshift-versions, and enabling their installation.

The main changes are:

1. In openshift-versions endpoint - We retrieve the release images from the DB (from part (1), which contains release images from configuration and API), and in addition we filter by 2 new query parameters (only_latest, version_pattern) that will be used by the UI to present short list of the versions. We also filter by a new configuration parameter IGNORED_RELEASE_IMAGES, which contains a list of versions we don't want to retrieve (as one might have a blocker bug).
2. We change the release handler struct to use the new source of truth (DB) instead of the configuration, to enable their installation.

Remarks

1. As proposed in:

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 145 in e604bc4

   In addition, we should expose a new API to show the current configuration

   I exposed new end point in - /api/assisted-install/v2/release-sources
2. This feature will will be active only in connected REST API deployments, using a new configuration called ENABLE_OPENSHIFT_RELEASE_SYNCER. This is because this new feature does not have much meaning in kubeAPI deployments as then openshift release images are managed through a resource called ClusterIMageSet, also specified in the enhancement -

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 71 in ee0a350

   * Handling OpenShift versions in a kubernetes API interface. For this use-case

   In order to prevent this feature from crashing the service in case of isolated environments (such as in agent installations)
3. I deployed a test instance with extra endpoint available in - http://rdu-infra-edge-06.infra-edge.lab.eng.rdu2.redhat.com:6008/api/assisted-install/v2/release-images
4. I chose to add the configuration releases to the DB for simplicity sake.
5. I chose not to keep the cpu_architectures for each release image in the DB for simplicity sake, as it can be derived from cpu_architecture, and it requires using SQL arrays
6. The current release sources configuration is a suggestion, we can remove some architectures/channels/versions to reduce the releases list size.
7. As specified in the enhancement:

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 237 in f5fc195

   Several cases can require a specific list of images to be configurable out of

   I kept RELEASE_IMAGES in order to enable unsupported release images. It is merged into the API release images during fetching process in order to have one source of truth.

List all the issues related to this PR

• New Feature
• Enhancement
• Bug fix
• Tests
• Documentation
• CI/CD

What environments does this code impact?

• Automation (CI, tools, etc)
• Cloud
• Operator Managed Deployments
• None

How was this code tested?

• assisted-test-infra environment
• dev-scripts environment
• Reviewer's test appreciated
• Waiting for CI to do a full test run
• Manual (Elaborate on how it was tested)
• No tests needed

Checklist

• Title and description added to both, commit and PR.
• Relevant issues have been associated (see CONTRIBUTING guide)
• This change does not require a documentation update (docstring, docs, README, etc)
• Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

• Are the title and description (in both PR and commit) meaningful and clear?
• Is there a bug required (and linked) for this change?
• Should this PR be backported?

[openshift-ci-robot]

@danmanor: This pull request references MGMT-16262 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.16.0" version, but no target version was set.

In response to this:

List all the issues related to this PR

• New Feature
• Enhancement
• Bug fix
• Tests
• Documentation
• CI/CD

What environments does this code impact?

• Automation (CI, tools, etc)
• Cloud
• Operator Managed Deployments
• None

How was this code tested?

• assisted-test-infra environment
• dev-scripts environment
• Reviewer's test appreciated
• Waiting for CI to do a full test run
• Manual (Elaborate on how it was tested)
• No tests needed

Checklist

• Title and description added to both, commit and PR.
• Relevant issues have been associated (see CONTRIBUTING guide)
• This change does not require a documentation update (docstring, docs, README, etc)
• Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

• Are the title and description (in both PR and commit) meaningful and clear?
• Is there a bug required (and linked) for this change?
• Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@danmanor: This pull request references MGMT-16262 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.16.0" version, but no target version was set.

In response to this:

In General

This PR is an Implementation of this enhancement proposal - https://github.com/openshift/assisted-service/pull/4893/files. There are two main parts to this implementation that are in the same PR because they are dependent on one another:

1. A utility that periodically and fetches OCP releases from an API, and stored them in the DB.
2. Using the release from part (1) to allow installation any supported OCP version

part 1

This part is about creating a periodic utility that updates current OCP releases in postgresql DB, for later use of assisted service.

The main flow is:

1. main fetches new configuration called RELEASE_SOURCES which specifies which OCP releases we want to keep in the DB and support (new table called openshift_versions) as well as from which channel, what CPU architectures, etc. Then it starts a new go routine that runs periodically.
2. This go routine fetches releases from the API according to the configuration, and OCP versions support levels from another API.
3. It labels the default release (latest stable release), supporting levels.
4. Replaces this data with the old one in the DB.

part 2

This part is about retrieving the new list of supported openshift versions in endpoint /api/assisted-install/v2/openshift-versions, and allowing their installation.

The main changes are:

1. In openshift-versions endpoint - In addition to the releases that we retrieved before this PR, which are the releases we configured using RELEASE_IMAGES, we retrieve the ones from the DB (from part (1)) as well, such that the configuration has precedence, meaning if a version appears in both DB and config, the config i the one being retrieved. We also filter by 2 new query parameters (only_latest, version_pattern) that will be used by the UI to present short list of the versions. We also filter by a new configuration parameter IGNORED_RELEASE_IMAGES, which contains a list of versions we don't want to retrieve (as one might have a blocker bug).
2. In order to allow the actual installation of these new releases, we Adding lookup in the DB releases for version handler functions if they are not found in the configuration.

Remarks

1. As proposed in:

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 145 in e604bc4

   In addition, we should expose a new API to show the current configuration

   I exposed new end point in - /api/assisted-install/v2/release-sources

List all the issues related to this PR

• New Feature
• Enhancement
• Bug fix
• Tests
• Documentation
• CI/CD

What environments does this code impact?

• Automation (CI, tools, etc)
• Cloud
• Operator Managed Deployments
• None

How was this code tested?

• assisted-test-infra environment
• dev-scripts environment
• Reviewer's test appreciated
• Waiting for CI to do a full test run
• Manual (Elaborate on how it was tested)
• No tests needed

Checklist

• Title and description added to both, commit and PR.
• Relevant issues have been associated (see CONTRIBUTING guide)
• This change does not require a documentation update (docstring, docs, README, etc)
• Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

• Are the title and description (in both PR and commit) meaningful and clear?
• Is there a bug required (and linked) for this change?
• Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[danmanor]

/test all

[openshift-ci-robot]

@danmanor: This pull request references MGMT-16262 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.16.0" version, but no target version was set.

In response to this:

In General

This PR is part 1 of the Implementation of this enhancement proposal - https://github.com/openshift/assisted-service/pull/4893/files, which is enabling usage of dynamic OpenShift releases.

1. A utility that periodically and fetches OCP releases from an API, and stored them in the DB.
2. Using the release from part (1) to allow installation any supported OCP version

part 1

This part is about creating a periodic utility that updates current OCP releases in postgresql DB, for later use of assisted service.

The main flow is:

1. main fetches new configuration called RELEASE_SOURCES which specifies which OCP releases we want to keep in the DB and support (new table called openshift_versions) as well as from which channel, what CPU architectures, etc. Then it starts a new go routine that runs periodically.
2. This go routine fetches releases from the API according to the configuration, and OCP versions support levels from another API.
3. It labels the default release (latest stable release), supporting levels.
4. Replaces this data with the old one in the DB.

part 2

This part is about retrieving the new list of supported openshift versions in endpoint /api/assisted-install/v2/openshift-versions, and allowing their installation.

The main changes are:

1. In openshift-versions endpoint - In addition to the releases that we retrieved before this PR, which are the releases we configured using RELEASE_IMAGES, we retrieve the ones from the DB (from part (1)) as well, such that the configuration has precedence, meaning if a version appears in both DB and config, the config i the one being retrieved. We also filter by 2 new query parameters (only_latest, version_pattern) that will be used by the UI to present short list of the versions. We also filter by a new configuration parameter IGNORED_RELEASE_IMAGES, which contains a list of versions we don't want to retrieve (as one might have a blocker bug).
2. In order to allow the actual installation of these new releases, we Adding lookup in the DB releases for version handler functions if they are not found in the configuration.

Remarks

1. As proposed in:

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 145 in e604bc4

   In addition, we should expose a new API to show the current configuration

   I exposed new end point in - /api/assisted-install/v2/release-sources

List all the issues related to this PR

• New Feature
• Enhancement
• Bug fix
• Tests
• Documentation
• CI/CD

What environments does this code impact?

• Automation (CI, tools, etc)
• Cloud
• Operator Managed Deployments
• None

How was this code tested?

• assisted-test-infra environment
• dev-scripts environment
• Reviewer's test appreciated
• Waiting for CI to do a full test run
• Manual (Elaborate on how it was tested)
• No tests needed

Checklist

• Title and description added to both, commit and PR.
• Relevant issues have been associated (see CONTRIBUTING guide)
• This change does not require a documentation update (docstring, docs, README, etc)
• Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

• Are the title and description (in both PR and commit) meaningful and clear?
• Is there a bug required (and linked) for this change?
• Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@danmanor: This pull request references MGMT-16262 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.16.0" version, but no target version was set.

In response to this:

In General

This PR is part 1 of the Implementation of this enhancement proposal - https://github.com/openshift/assisted-service/pull/4893/files, which is enabling usage of dynamic OpenShift releases.

1. A utility that periodically and fetches OCP releases from an API, and stored them in the DB.
2. Using the release from part (1) to allow installation any supported OCP version

Plan farward

This part is about creating a periodic utility that updates current OCP releases in postgresql DB, for later use of assisted service.

The main flow is:

1. main fetches new configuration called RELEASE_SOURCES which specifies which OCP releases we want to keep in the DB and support (new table called openshift_versions) as well as from which channel, what CPU architectures, etc. Then it starts a new go routine that runs periodically.
2. This go routine fetches releases from the API according to the configuration, and OCP versions support levels from another API.
3. It labels the default release (latest stable release), supporting levels.
4. Replaces this data with the old one in the DB.

part 2

This part is about retrieving the new list of supported openshift versions in endpoint /api/assisted-install/v2/openshift-versions, and allowing their installation.

The main changes are:

1. In openshift-versions endpoint - In addition to the releases that we retrieved before this PR, which are the releases we configured using RELEASE_IMAGES, we retrieve the ones from the DB (from part (1)) as well, such that the configuration has precedence, meaning if a version appears in both DB and config, the config i the one being retrieved. We also filter by 2 new query parameters (only_latest, version_pattern) that will be used by the UI to present short list of the versions. We also filter by a new configuration parameter IGNORED_RELEASE_IMAGES, which contains a list of versions we don't want to retrieve (as one might have a blocker bug).
2. In order to allow the actual installation of these new releases, we Adding lookup in the DB releases for version handler functions if they are not found in the configuration.

Remarks

1. As proposed in:

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 145 in e604bc4

   In addition, we should expose a new API to show the current configuration

   I exposed new end point in - /api/assisted-install/v2/release-sources

List all the issues related to this PR

• New Feature
• Enhancement
• Bug fix
• Tests
• Documentation
• CI/CD

What environments does this code impact?

• Automation (CI, tools, etc)
• Cloud
• Operator Managed Deployments
• None

How was this code tested?

• assisted-test-infra environment
• dev-scripts environment
• Reviewer's test appreciated
• Waiting for CI to do a full test run
• Manual (Elaborate on how it was tested)
• No tests needed

Checklist

• Title and description added to both, commit and PR.
• Relevant issues have been associated (see CONTRIBUTING guide)
• This change does not require a documentation update (docstring, docs, README, etc)
• Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

• Are the title and description (in both PR and commit) meaningful and clear?
• Is there a bug required (and linked) for this change?
• Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@danmanor: This pull request references MGMT-16262 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.16.0" version, but no target version was set.

In response to this:

In General

This PR is part 1 of the Implementation of this enhancement proposal - https://github.com/openshift/assisted-service/pull/4893/files, which is enabling usage of dynamic OpenShift releases.

1. A utility that periodically and fetches OCP releases from an API, and stored them in the DB.
2. Using the release from part (1) to allow installation any supported OCP version

Main flow

This part is about creating a periodic utility that updates current OCP releases in postgresql DB, for later use of assisted service -

1. main fetches new configuration called RELEASE_SOURCES which specifies which OCP releases we want to keep in the DB and support (new table called openshift_versions) as well as from which channel, what CPU architectures, etc. Then it starts a new go routine that runs periodically.
2. This go routine fetches releases from the API according to the configuration, and OCP versions support levels from another API.
3. It labels the default release (latest stable release), supporting levels.
4. Replaces this data with the old one in the DB.

Later on

This part is about retrieving the new list of supported openshift versions in endpoint /api/assisted-install/v2/openshift-versions, and allowing their installation.

The main changes are:

1. In openshift-versions endpoint - In addition to the releases that we retrieved before this PR, which are the releases we configured using RELEASE_IMAGES, we retrieve the ones from the DB (from part (1)) as well, such that the configuration has precedence, meaning if a version appears in both DB and config, the config i the one being retrieved. We also filter by 2 new query parameters (only_latest, version_pattern) that will be used by the UI to present short list of the versions. We also filter by a new configuration parameter IGNORED_RELEASE_IMAGES, which contains a list of versions we don't want to retrieve (as one might have a blocker bug).
2. In order to allow the actual installation of these new releases, we Adding lookup in the DB releases for version handler functions if they are not found in the configuration.

Remarks

1. As proposed in:

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 145 in e604bc4

   In addition, we should expose a new API to show the current configuration

   I exposed new end point in - /api/assisted-install/v2/release-sources

List all the issues related to this PR

• New Feature
• Enhancement
• Bug fix
• Tests
• Documentation
• CI/CD

What environments does this code impact?

• Automation (CI, tools, etc)
• Cloud
• Operator Managed Deployments
• None

How was this code tested?

• assisted-test-infra environment
• dev-scripts environment
• Reviewer's test appreciated
• Waiting for CI to do a full test run
• Manual (Elaborate on how it was tested)
• No tests needed

Checklist

• Title and description added to both, commit and PR.
• Relevant issues have been associated (see CONTRIBUTING guide)
• This change does not require a documentation update (docstring, docs, README, etc)
• Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

• Are the title and description (in both PR and commit) meaningful and clear?
• Is there a bug required (and linked) for this change?
• Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@danmanor: This pull request references MGMT-16262 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.16.0" version, but no target version was set.

In response to this:

In General

This PR is part 1 of the Implementation of this enhancement proposal - https://github.com/openshift/assisted-service/pull/4893/files, which is enabling usage of dynamic OpenShift releases.

1. A utility that periodically and fetches OCP releases from an API, and stored them in the DB.
2. Using the release from part (1) to allow installation any supported OCP version

Main flow

This part is about creating a periodic utility that updates current OCP releases in postgresql DB, for later use of assisted service -

1. main fetches new configuration called RELEASE_SOURCES which specifies which OCP releases we want to keep in the DB and support (new table called openshift_versions) as well as from which channel, what CPU architectures, etc. Then it starts a new go routine that runs periodically.
2. This go routine fetches releases from the API according to the configuration, and OCP versions support levels from another API.
3. It labels the default release (latest stable release), supporting levels.
4. Replaces this data with the old one in the DB.

Plan farward - part 2

This part is about retrieving the new list of supported openshift versions in endpoint /api/assisted-install/v2/openshift-versions, and allowing their installation.

The main changes are:

1. In openshift-versions endpoint - In addition to the releases that we retrieved before this PR, which are the releases we configured using RELEASE_IMAGES, we retrieve the ones from the DB (from part (1)) as well, such that the configuration has precedence, meaning if a version appears in both DB and config, the config i the one being retrieved. We also filter by 2 new query parameters (only_latest, version_pattern) that will be used by the UI to present short list of the versions. We also filter by a new configuration parameter IGNORED_RELEASE_IMAGES, which contains a list of versions we don't want to retrieve (as one might have a blocker bug).
2. In order to allow the actual installation of these new releases, we Adding lookup in the DB releases for version handler functions if they are not found in the configuration.

Remarks

1. As proposed in:

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 145 in e604bc4

   In addition, we should expose a new API to show the current configuration

   I exposed new end point in - /api/assisted-install/v2/release-sources

List all the issues related to this PR

• New Feature
• Enhancement
• Bug fix
• Tests
• Documentation
• CI/CD

What environments does this code impact?

• Automation (CI, tools, etc)
• Cloud
• Operator Managed Deployments
• None

How was this code tested?

• assisted-test-infra environment
• dev-scripts environment
• Reviewer's test appreciated
• Waiting for CI to do a full test run
• Manual (Elaborate on how it was tested)
• No tests needed

Checklist

• Title and description added to both, commit and PR.
• Relevant issues have been associated (see CONTRIBUTING guide)
• This change does not require a documentation update (docstring, docs, README, etc)
• Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

• Are the title and description (in both PR and commit) meaningful and clear?
• Is there a bug required (and linked) for this change?
• Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[codecov]

Codecov Report

Attention: Patch coverage is 79.40299% with 69 lines in your changes are missing coverage. Please review.

Project coverage is 68.39%. Comparing base (fe09431) to head (d516839).
Report is 20 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5916      +/-   ##
==========================================
+ Coverage   68.25%   68.39%   +0.13%     
==========================================
  Files         236      239       +3     
  Lines       35053    35453     +400     
==========================================
+ Hits        23927    24248     +321     
- Misses       9049     9100      +51     
- Partials     2077     2105      +28     

Files	Coverage Δ
internal/common/db.go	7.14% <100.00%> (+0.41%)	⬆️
internal/common/test_configuration.go	0.00% <ø> (ø)
internal/versions/api.go	97.46% <ø> (+0.09%)	⬆️
internal/common/version.go	65.30% <83.33%> (+5.84%)	⬆️
internal/releasesources/clients.go	96.07% <96.07%> (ø)
internal/releasesources/release_sources.go	76.01% <76.01%> (ø)

... and 13 files with indirect coverage changes

[openshift-ci-robot]

@danmanor: This pull request references MGMT-16262 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.16.0" version, but no target version was set.

In response to this:

In General

This PR is part 1 of the Implementation of this enhancement proposal - https://github.com/openshift/assisted-service/pull/4893/files, which is enabling usage of dynamic openshift releases.

Main flow

This part is about creating a periodic utility that updates current OCP releases in postgresql DB, for later use of assisted service -

The main changes are:

1. main fetches new configuration called RELEASE_SOURCES which specifies which OCP releases we want to keep in the DB and support (new table called release_images) as well as from which channel, what CPU architectures, etc. Then it starts a new go routine that runs periodically.
2. This go routine fetches releases from the API according to the configuration, and OCP versions support levels from another API.
3. It labels the default release (latest stable release), supporting levels.
4. Replaces this data with the old one in the DB.

Plan forward (part 2)

This part is about retrieving the new list of supported openshift versions in endpoint /api/assisted-install/v2/openshift-versions, and allowing their installation.

The main changes are:

1. In openshift-versions endpoint - In addition to the releases that we retrieved before this PR, which are the releases we configured using RELEASE_IMAGES, we retrieve the ones from the DB (from part (1)) as well, such that the configuration has precedence, meaning if a version appears in both DB and config, the config i the one being retrieved. We also filter by 2 new query parameters (only_latest, version_pattern) that will be used by the UI to present short list of the versions. We also filter by a new configuration parameter IGNORED_RELEASE_IMAGES, which contains a list of versions we don't want to retrieve (as one might have a blocker bug).
2. In order to allow the actual installation of these new releases, we Adding lookup in the DB releases for version handler functions if they are not found in the configuration.

Remarks

1. As proposed in:

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 145 in e604bc4

   In addition, we should expose a new API to show the current configuration

   I exposed new end point in - /api/assisted-install/v2/release-sources

List all the issues related to this PR

• New Feature
• Enhancement
• Bug fix
• Tests
• Documentation
• CI/CD

What environments does this code impact?

• Automation (CI, tools, etc)
• Cloud
• Operator Managed Deployments
• None

How was this code tested?

• assisted-test-infra environment
• dev-scripts environment
• Reviewer's test appreciated
• Waiting for CI to do a full test run
• Manual (Elaborate on how it was tested)
• No tests needed

Checklist

• Title and description added to both, commit and PR.
• Relevant issues have been associated (see CONTRIBUTING guide)
• This change does not require a documentation update (docstring, docs, README, etc)
• Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

• Are the title and description (in both PR and commit) meaningful and clear?
• Is there a bug required (and linked) for this change?
• Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[danmanor]

/test all

[openshift-ci-robot]

@danmanor: This pull request references MGMT-16262 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.16.0" version, but no target version was set.

In response to this:

In General

This PR is part 1 of the Implementation of this enhancement proposal - https://github.com/openshift/assisted-service/pull/4893/files, which is enabling usage of dynamic openshift releases.

Main flow

This part is about creating a periodic utility that updates current OCP releases in postgresql DB, for later use of assisted service -

The main changes are:

1. main fetches new configuration called RELEASE_SOURCES which specifies which OCP releases we want to keep in the DB and support (new table called release_images) as well as from which channel, what CPU architectures, etc. Then it starts a new go routine that runs periodically.
2. This go routine fetches releases from the API according to the configuration, and OCP versions support levels from another API.
3. It labels the default release (latest stable release), supporting levels.
4. Adds release images from the configuration (with precedence in case of conflicts).
5. Replaces this data with the old one in the DB.

Plan forward (part 2)

This part is about retrieving the new list of supported openshift versions in endpoint /api/assisted-install/v2/openshift-versions, and enabling their installation.

The main changes are:

1. In openshift-versions endpoint - We retrieve the release images from the DB (from part (1), which contains release images from configuration and API), and in addition we filter by 2 new query parameters (only_latest, version_pattern) that will be used by the UI to present short list of the versions. We also filter by a new configuration parameter IGNORED_RELEASE_IMAGES, which contains a list of versions we don't want to retrieve (as one might have a blocker bug).
2. We change the release handler struct to use the new source of truth (DB) instead of the configuration, to enable their installation.

Remarks

1. As proposed in:

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 145 in e604bc4

   In addition, we should expose a new API to show the current configuration

   I exposed new end point in - /api/assisted-install/v2/release-sources
2. This feature will will be active only in connected REST API deployments, using a new configuration called ENABLE_OPENSHIFT_RELEASE_SYNCER. This is because this new feature does not have much meaning in kubeAPI deployments as then openshift release images are managed through a resource called ClusterIMageSet, also specified in the enhancement -

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 71 in ee0a350

   * Handling OpenShift versions in a kubernetes API interface. For this use-case

   In order to prevent this feature from crashing the service in case of isolated environments (such as in agent installations)
3. I deployed a test instance with extra endpoint available in - http://rdu-infra-edge-06.infra-edge.lab.eng.rdu2.redhat.com:6008/api/assisted-install/v2/release-images

List all the issues related to this PR

• New Feature
• Enhancement
• Bug fix
• Tests
• Documentation
• CI/CD

What environments does this code impact?

• Automation (CI, tools, etc)
• Cloud
• Operator Managed Deployments
• None

How was this code tested?

• assisted-test-infra environment
• dev-scripts environment
• Reviewer's test appreciated
• Waiting for CI to do a full test run
• Manual (Elaborate on how it was tested)
• No tests needed

Checklist

• Title and description added to both, commit and PR.
• Relevant issues have been associated (see CONTRIBUTING guide)
• This change does not require a documentation update (docstring, docs, README, etc)
• Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

• Are the title and description (in both PR and commit) meaningful and clear?
• Is there a bug required (and linked) for this change?
• Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@danmanor: This pull request references MGMT-16262 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.16.0" version, but no target version was set.

In response to this:

In General

This PR is part 1 of the Implementation of this enhancement proposal - https://github.com/openshift/assisted-service/pull/4893/files, which is enabling usage of dynamic openshift releases.

Main flow

This part is about creating a periodic utility that updates current OCP releases in postgresql DB, for later use of assisted service -

The main changes are:

1. main fetches new configuration called RELEASE_SOURCES which specifies which OCP releases we want to keep in the DB and support (new table called release_images) as well as from which channel, what CPU architectures, etc. Then it starts a new go routine that runs periodically.
2. This go routine fetches releases from the API according to the configuration, and OCP versions support levels from another API.
3. It labels the default release (latest stable release), supporting levels.
4. Adds release images from the configuration (with precedence in case of conflicts).
5. Replaces this data with the old one in the DB.

Plan forward (part 2)

This part is about retrieving the new list of supported openshift versions in endpoint /api/assisted-install/v2/openshift-versions, and enabling their installation.

The main changes are:

1. In openshift-versions endpoint - We retrieve the release images from the DB (from part (1), which contains release images from configuration and API), and in addition we filter by 2 new query parameters (only_latest, version_pattern) that will be used by the UI to present short list of the versions. We also filter by a new configuration parameter IGNORED_RELEASE_IMAGES, which contains a list of versions we don't want to retrieve (as one might have a blocker bug).
2. We change the release handler struct to use the new source of truth (DB) instead of the configuration, to enable their installation.

Remarks

1. As proposed in:

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 145 in e604bc4

   In addition, we should expose a new API to show the current configuration

   I exposed new end point in - /api/assisted-install/v2/release-sources
2. This feature will will be active only in connected REST API deployments, using a new configuration called ENABLE_OPENSHIFT_RELEASE_SYNCER. This is because this new feature does not have much meaning in kubeAPI deployments as then openshift release images are managed through a resource called ClusterIMageSet, also specified in the enhancement -

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 71 in ee0a350

   * Handling OpenShift versions in a kubernetes API interface. For this use-case

   In order to prevent this feature from crashing the service in case of isolated environments (such as in agent installations)
3. I deployed a test instance with extra endpoint available in - http://rdu-infra-edge-06.infra-edge.lab.eng.rdu2.redhat.com:6008/api/assisted-install/v2/release-images
4. I chose to add the configuration releases to the DB for simplicity sake.
5. I chose not to keep the cpu_architectures for each release image in the DB for simplicity sake, as it can be driven from cpu_architecture, and it requires using SQL arrays

List all the issues related to this PR

• New Feature
• Enhancement
• Bug fix
• Tests
• Documentation
• CI/CD

What environments does this code impact?

• Automation (CI, tools, etc)
• Cloud
• Operator Managed Deployments
• None

How was this code tested?

• assisted-test-infra environment
• dev-scripts environment
• Reviewer's test appreciated
• Waiting for CI to do a full test run
• Manual (Elaborate on how it was tested)
• No tests needed

Checklist

• Title and description added to both, commit and PR.
• Relevant issues have been associated (see CONTRIBUTING guide)
• This change does not require a documentation update (docstring, docs, README, etc)
• Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

• Are the title and description (in both PR and commit) meaningful and clear?
• Is there a bug required (and linked) for this change?
• Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@danmanor: This pull request references MGMT-16262 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.16.0" version, but no target version was set.

In response to this:

In General

This PR is part 1 of the Implementation of this enhancement proposal - https://github.com/openshift/assisted-service/pull/4893/files, which is enabling usage of dynamic openshift releases.

Main flow

This part is about creating a periodic utility that updates current OCP releases in postgresql DB, for later use of assisted service -

The main changes are:

1. main fetches new configuration called RELEASE_SOURCES which specifies which OCP releases we want to keep in the DB and support (new table called release_images) as well as from which channel, what CPU architectures, etc. Then it starts a new go routine that runs periodically.
2. This go routine fetches releases from the API according to the configuration, and OCP versions support levels from another API.
3. It labels the default release (latest stable release), supporting levels.
4. Adds release images from the configuration (with precedence in case of conflicts).
5. Replaces this data with the old one in the DB.

Plan forward (part 2)

This part is about retrieving the new list of supported openshift versions in endpoint /api/assisted-install/v2/openshift-versions, and enabling their installation.

The main changes are:

1. In openshift-versions endpoint - We retrieve the release images from the DB (from part (1), which contains release images from configuration and API), and in addition we filter by 2 new query parameters (only_latest, version_pattern) that will be used by the UI to present short list of the versions. We also filter by a new configuration parameter IGNORED_RELEASE_IMAGES, which contains a list of versions we don't want to retrieve (as one might have a blocker bug).
2. We change the release handler struct to use the new source of truth (DB) instead of the configuration, to enable their installation.

Remarks

1. As proposed in:

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 145 in e604bc4

   In addition, we should expose a new API to show the current configuration

   I exposed new end point in - /api/assisted-install/v2/release-sources
2. This feature will will be active only in connected REST API deployments, using a new configuration called ENABLE_OPENSHIFT_RELEASE_SYNCER. This is because this new feature does not have much meaning in kubeAPI deployments as then openshift release images are managed through a resource called ClusterIMageSet, also specified in the enhancement -

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 71 in ee0a350

   * Handling OpenShift versions in a kubernetes API interface. For this use-case

   In order to prevent this feature from crashing the service in case of isolated environments (such as in agent installations)
3. I deployed a test instance with extra endpoint available in - http://rdu-infra-edge-06.infra-edge.lab.eng.rdu2.redhat.com:6008/api/assisted-install/v2/release-images
4. I chose to add the configuration releases to the DB for simplicity sake.
5. I chose not to keep the cpu_architectures for each release image in the DB for simplicity sake, as it can be driven from cpu_architecture, and it requires using SQL arrays
6. The current release sources configuration is a suggestion, we can remove some architectures/channels/versions to reduce the releases list size

List all the issues related to this PR

• New Feature
• Enhancement
• Bug fix
• Tests
• Documentation
• CI/CD

What environments does this code impact?

• Automation (CI, tools, etc)
• Cloud
• Operator Managed Deployments
• None

How was this code tested?

• assisted-test-infra environment
• dev-scripts environment
• Reviewer's test appreciated
• Waiting for CI to do a full test run
• Manual (Elaborate on how it was tested)
• No tests needed

Checklist

• Title and description added to both, commit and PR.
• Relevant issues have been associated (see CONTRIBUTING guide)
• This change does not require a documentation update (docstring, docs, README, etc)
• Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

• Are the title and description (in both PR and commit) meaningful and clear?
• Is there a bug required (and linked) for this change?
• Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@danmanor: This pull request references MGMT-16262 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the sub-task to target the "4.16.0" version, but no target version was set.

In response to this:

In General

This PR is part 1 of the Implementation of this enhancement proposal - https://github.com/openshift/assisted-service/blob/master/docs/enhancements/dynamic-fetching-of-openshift-versions.md, which is enabling usage of dynamic openshift releases.

Main flow

This part is about creating a periodic utility that updates current OCP releases in postgresql DB, for later use of assisted service -

The main changes are:

1. main fetches new configuration called RELEASE_SOURCES which specifies which OCP releases we want to keep in the DB and support (new table called release_images) as well as from which channel, what CPU architectures, etc. Then it starts a new go routine that runs periodically.
2. This go routine fetches releases from the API according to the configuration, and OCP versions support levels from another API.
3. It labels the default release (latest stable release), supporting levels.
4. Adds release images from the configuration (with precedence in case of conflicts).
5. Replaces this data with the old one in the DB.

Plan forward (part 2)

This part is about retrieving the new list of supported openshift versions in endpoint /api/assisted-install/v2/openshift-versions, and enabling their installation.

The main changes are:

1. In openshift-versions endpoint - We retrieve the release images from the DB (from part (1), which contains release images from configuration and API), and in addition we filter by 2 new query parameters (only_latest, version_pattern) that will be used by the UI to present short list of the versions. We also filter by a new configuration parameter IGNORED_RELEASE_IMAGES, which contains a list of versions we don't want to retrieve (as one might have a blocker bug).
2. We change the release handler struct to use the new source of truth (DB) instead of the configuration, to enable their installation.

Remarks

1. As proposed in:

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 145 in e604bc4

   In addition, we should expose a new API to show the current configuration

   I exposed new end point in - /api/assisted-install/v2/release-sources
2. This feature will will be active only in connected REST API deployments, using a new configuration called ENABLE_OPENSHIFT_RELEASE_SYNCER. This is because this new feature does not have much meaning in kubeAPI deployments as then openshift release images are managed through a resource called ClusterIMageSet, also specified in the enhancement -

   assisted-service/docs/enhancements/dynamic-fetching-of-openshift-versions.md

   Line 71 in ee0a350

   * Handling OpenShift versions in a kubernetes API interface. For this use-case

   In order to prevent this feature from crashing the service in case of isolated environments (such as in agent installations)
3. I deployed a test instance with extra endpoint available in - http://rdu-infra-edge-06.infra-edge.lab.eng.rdu2.redhat.com:6008/api/assisted-install/v2/release-images
4. I chose to add the configuration releases to the DB for simplicity sake.
5. I chose not to keep the cpu_architectures for each release image in the DB for simplicity sake, as it can be driven from cpu_architecture, and it requires using SQL arrays
6. The current release sources configuration is a suggestion, we can remove some architectures/channels/versions to reduce the releases list size

List all the issues related to this PR

• New Feature
• Enhancement
• Bug fix
• Tests
• Documentation
• CI/CD

What environments does this code impact?

• Automation (CI, tools, etc)
• Cloud
• Operator Managed Deployments
• None

How was this code tested?

• assisted-test-infra environment
• dev-scripts environment
• Reviewer's test appreciated
• Waiting for CI to do a full test run
• Manual (Elaborate on how it was tested)
• No tests needed

Checklist

• Title and description added to both, commit and PR.
• Relevant issues have been associated (see CONTRIBUTING guide)
• This change does not require a documentation update (docstring, docs, README, etc)
• Does this change include unit-tests (note that code changes require unit-tests)

Reviewers Checklist

• Are the title and description (in both PR and commit) meaningful and clear?
• Is there a bug required (and linked) for this change?
• Should this PR be backported?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[danmanor]

/cc @gamli75 @osherdp @carbonin @rccrdpccl

[danmanor]

/cc @rccrdpccl

[carbonin]

I don't think this needs to be a pointer and I think we should create an OpenshiftReleaseChannel enum instead of using string and constants.

[carbonin]

I think make the non-thread version of this public and use it here to properly report if it fails.

[carbonin]

Can we do these validations on initialization?
I don't see a reason to do them every run and then we could also save the static enriched images as a part of the handler state and not run getStaticReleaseImages every loop also.

[carbonin]

This will need to be removed if we persist enrichedReleaseImages for static.

Currently this will always we empty for static release images, but if we save them it will always have the value from the previous loop. To deal with this we can just always set the level to the new one for this loop.

[danmanor]

/hold

[danmanor]

/test edge-subsystem-aws /test e2e-agent-compact-ipv4

[danmanor]

/test edge-subsystem-aws

[danmanor]

/unhold

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: carbonin, danmanor

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [carbonin,danmanor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@danmanor: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-agent-compact-ipv4	d516839	link	false	/test e2e-agent-compact-ipv4
ci/prow/edge-e2e-ai-operator-disconnected-capi	d516839	link	false	/test edge-e2e-ai-operator-disconnected-capi

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-bot]

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-agent-installer-api-server-container-v4.16.0-202402261639.p0.gec0d30d.assembly.stream.el8 for distgit ose-agent-installer-api-server.
All builds following this will include this PR.