Skip to content

Reconcile installation upon apply#1764

Merged
carolynvs merged 5 commits intogetporter:release/v1from
carolynvs:reconcile
Sep 28, 2021
Merged

Reconcile installation upon apply#1764
carolynvs merged 5 commits intogetporter:release/v1from
carolynvs:reconcile

Conversation

@carolynvs
Copy link
Member

@carolynvs carolynvs commented Sep 17, 2021

What does this change

Validate the input file during apply

Currently we only validate the resulting merged data structure before saving it to the database. But that doesn't catch when they import a document with an old schema.

This will enforce that the schema of the important document is supported.

Move yq editor into the yaml package

Moving the yq editor into the yaml package so that it's more of a generic editor for any yaml, not just the porter manifest. I'm going to use it in our tests.

Improve output capture in the smoke tests

When we capture output we should capture:

  • stdout only (so that we can parse it for json)
  • combined stdout + stderr, which is what the user would see in the console. This helps us search the output in tests, or just see what the user would see.
  • stderr as a Go error

Call reconcile during porter installation apply

Reconcile the imported changes against the current state of the installation, and call the appropriate command to match the desired state. Right now we only trigger under the following conditions:

  • different bundle digest
  • different resolved parameters
  • different credential set names (I don't compare resolved credentials)

I've added a --force flag to apply so that you can fix stuff when porter detects no changes but things are borked in reality, and also a --dry-run flag so that someone can easily tell if porter detects any changes without running the bundle.

https://deploy-preview-1764--porter.netlify.app/quickstart/desired-state/
https://deploy-preview-1764--porter.netlify.app/end-users/installations/
https://deploy-preview-1764--porter.netlify.app/operator/

What issue does it fix

Closes #1704

Notes for the reviewer

  • Do we want to resolve and compare credentials? Right now I am only comparing credential names because we do not store credential values used in a run.

Checklist

  • Unit Tests
  • Documentation - Don't let me merge until I document the following:
    • Managing installations with GitOps (files instead of imperative commands)
    • Explain what is compared, what changes trigger a run, and how to force it
    • Update the schema files for credential sets/parameter sets and add one for installations
  • Schema (porter.yaml)

Currently we only validate the resulting merged data structure before
saving it to the database. But that doesn't catch when they import a
document with an old schema.

This will enforce that the schema of the important document
is supported.

Signed-off-by: Carolyn Van Slyck <me@carolynvanslyck.com>
Moving the yq editor into the yaml package so that it's more of a
generic editor for any yaml, not just the porter manifest. I'm going to
use it in our tests.

Signed-off-by: Carolyn Van Slyck <me@carolynvanslyck.com>
@carolynvs
Copy link
Member Author

/azp run porter-integration

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

When we capture output we should capture:
* stdout only (so that we can parse it for json)
* combined stdout + stderr, which is what the user would see in the
console. This helps us search the output in tests, or just see what the
user would see.
* stderr as a Go error

Signed-off-by: Carolyn Van Slyck <me@carolynvanslyck.com>
@carolynvs carolynvs marked this pull request as ready for review September 27, 2021 14:12
@carolynvs carolynvs changed the title Reconcile Reconcile installation upon apply Sep 27, 2021
Reconcile the imported changes against the current state of the
installation, and call the appropriate command to match the desired
state. Right now we only trigger under the following conditions:

* different bundle digest
* different resolved parameters
* different credential set _names_ (I don't compare resolved
credentials)

I've added a --force flag to apply so that you can fix stuff when porter
detects no changes but things are borked in reality.

Signed-off-by: Carolyn Van Slyck <me@carolynvanslyck.com>
@carolynvs
Copy link
Member Author

@vdice Okay all the documentation is completed in the PR (links in the OP). Please take a look.

Copy link
Member

@vdice vdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Only really had a few clarifying questions/thoughts...

Do we want to resolve and compare credentials? Right now I am only comparing credential names because we do not store credential values used in a run.

Hmmm... not sure. Should we file an issue in the backlog (if not already there)?

The following will result in Porter executing the bundle:
* The installation has not completed successfully yet.
* The bundle reference has changed. The bundle reference is resolved using the bundleRepository, bundleVersion, bundleDigest, and bundleTag fields.
* The resolved parameter values have changed, either because an associated parameter set has change, the parameters defined on the bundle have changed, or the values resolved by any parameter sets have changed.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* The resolved parameter values have changed, either because an associated parameter set has change, the parameters defined on the bundle have changed, or the values resolved by any parameter sets have changed.
* The resolved parameter values have changed, either because an associated parameter set has changed, the parameters defined on the bundle have changed, or the values resolved by any parameter sets have changed.


```yaml
parameters:
user: vdice
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😺

@carolynvs
Copy link
Member Author

I have added an issue to further improve reconciliation by storing hashes of the resolved parameters and credentials when Porter is run.

#1781

Signed-off-by: Carolyn Van Slyck <me@carolynvanslyck.com>
Copy link
Member

@vdice vdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@carolynvs carolynvs merged commit 2601541 into getporter:release/v1 Sep 28, 2021
@carolynvs carolynvs deleted the reconcile branch September 28, 2021 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants