Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable migration to store inventory in separate ResourceGroup file #2705

Merged
merged 1 commit into from Feb 8, 2022

Conversation

rquitales
Copy link
Contributor

@rquitales rquitales commented Feb 1, 2022

This is a PR that implements the kpt live migrate functionality defined within the kpt live apply from STDIN design doc.

Additions:

The files changed in this PR are:

internal/cmdmigrate/migratecmd.go
internal/cmdmigrate/migratecmd_test.go

@rquitales rquitales changed the title Stdin actuation migrate Enable migration to store inventory in separate ResourceGroup file Feb 1, 2022
@rquitales rquitales force-pushed the stdin-actuation-migrate branch 2 times, most recently from c679065 to b5abbc9 Compare February 1, 2022 18:47
This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.
@mortent
Copy link
Contributor

mortent commented Feb 4, 2022

It is not clear to me how this addition will work with the current migration. Is it a separate migration from the existing one, or do a migration from now on simply put the inventory into the separate file rather than update the Kptfile? If so, why do we need the two different implementations?

It also seems like a lot of duplicated code between the Run and runLiveMigrateWithRGFile function. Is there a way to avoid some of this?

@rquitales
Copy link
Contributor Author

It is not clear to me how this addition will work with the current migration. Is it a separate migration from the existing one, or do a migration from now on simply put the inventory into the separate file rather than update the Kptfile? If so, why do we need the two different implementations?

It also seems like a lot of duplicated code between the Run and runLiveMigrateWithRGFile function. Is there a way to avoid some of this?

The logic in runLiveMigrateWithRGFile will completely replace the logic in Run when we expose this functionality externally. We essentially have "dead code" currently so we don't break existing functionality for kpt releases, but can also introduce the new migrate CUJ logic for PR review and integration testing.

@mortent mortent merged commit 58b5e35 into kptdev:main Feb 8, 2022
martinmaly pushed a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022
…kptdev#2705)

This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.
martinmaly pushed a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022
feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705)

This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.

make kpt binary optional in test harness (kptdev#2758)

feat: enable STDIN apply and destroy using RG inventory (kptdev#2709)

This commit enables actuation from STDIN using inventory information
that is stored in a ResourceGroup file.

This feature is currently behind a feature gate and will be exposed to
users as a CLI flag in a future commit/PR.

updated the version to 1.0.0-beta.13 (kptdev#2806)

Merge main into porch
martinmaly pushed a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022
feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705)

This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.

make kpt binary optional in test harness (kptdev#2758)

feat: enable STDIN apply and destroy using RG inventory (kptdev#2709)

This commit enables actuation from STDIN using inventory information
that is stored in a ResourceGroup file.

This feature is currently behind a feature gate and will be exposed to
users as a CLI flag in a future commit/PR.

updated the version to 1.0.0-beta.13 (kptdev#2806)

Merge main into porch
martinmaly pushed a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022
Save inventory information into resourcegroup.yaml (kptdev#2615)

Fix site sidebar to also show annotation references (kptdev#2734)

effective customizations chapter (kptdev#2659)

merging this as an MVP and will iterate based on additional requests and customer feedback.

docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)

Merge main into porch

Combined Main Commit

feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705)

This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.

make kpt binary optional in test harness (kptdev#2758)

feat: enable STDIN apply and destroy using RG inventory (kptdev#2709)

This commit enables actuation from STDIN using inventory information
that is stored in a ResourceGroup file.

This feature is currently behind a feature gate and will be exposed to
users as a CLI flag in a future commit/PR.

updated the version to 1.0.0-beta.13 (kptdev#2806)

Merge main into porch

Combined Porch Commit

Streamline Docker Image Building (kptdev#2752)

* Use the same naming pattern
* Enable nested Makefiles to inherit values from parent
* Enable explicit image tagging (i.e. using
  `IMAGE_TAG=$(git rev-parse --short HEAD) make build-images`
* Enable `make build-images` and `make push-images`

Run Porch With Consistent Working Dir (kptdev#2753)

`go run` doesn't support setting working directory; use `go build`
instead.

Add subpkgs test for porch (kptdev#2754)

* Add subpkgs test for porch

* Add runtime

Basic e2e Test (kptdev#2755)

* Basic e2e Test
* Simple harness to reduce some boilerplate
* Add a simple git repo test

Refactor Server Startup (kptdev#2770)

* Start background processing in Run function
* Shut down when stopCh is closed

Tidy in 1.17 compatible mode (kptdev#2769)

Occasionally `make tidy` fails complaining about 1.16 vs. 1.17 golang;
this should keep our mods 1.17 compatible with fewer `make tidy`
failures.

Create Engine using Options (kptdev#2771)

Create options for the common engine configuration options.

Support lazy credential resolution (kptdev#2772)

* Add CredentialResolver interface
* Add WithCredentialResolver engine option
* Use lazy credential resolution when interacting with Git

Pass Context to OpenRepository (kptdev#2773)

Use Context to Drive Server Shutdown (kptdev#2774)

Turns out k8s apiserver supports both and we can get the core context to
listen on instead of just getting the close channel.

Clean up Required/Optional API Fields (kptdev#2778)

Git branch and directory can be optional (defaulting to `main` and `/` respectively,
while `SecretRef.Name` is required since nameless secret reference is unhelpful.

Easier deployment onto GKE (kptdev#2776)

Support Creating Porch Deployment Config (kptdev#2777)

* Create `deployment-config` and `push-and-deploy` make targets
* Use kpt function to set images on the deployment config

Update Porch Deployment and Instructions (kptdev#2782)

* Set workload identity service accounts via set-annotations
* Simplify instructions
* Build at Git tag
* Combine Deployment Config in Same Directory
* Rename config files, assign 0 to CRDs.
* `kubectl apply` recursively just in case we add more config later

Use controller-gen v0.8.0 (kptdev#2780)

Use consistent version of controller-gen (v0.8.0)
crd:preserveUnknownFields marker has been removed (`false` was the
required value for v1 CRDs).

Set renderer when building CaDEngine (kptdev#2787)

Otherwise we crash when trying to render a package.

Fix missing error handling (kptdev#2784)

We weren't checking errors when building a CaDEngine.

Fix small typo (kptdev#2793)

Don't call into kpt render if we don't have a package (kptdev#2788)

When creating an empty package, we get an error otherwise (as kpt
render doesn't work if it doesn't have a package).

Combined Porch Commit

Split git_test To Use Git Server (kptdev#2797)

Enable e2e tests to use git server

Add CreatePackageRevision Test (kptdev#2800)

Simple test that creates a package by cloning from upstream repository.

Cache apply-setters:v0.1 function (kptdev#2799)

Some blueprints we encounter in tests use the older version of the function.

Return Git PackageRevisionResources Correctly (kptdev#2801)

Newly created Git package returned package resources at its old commit
SHA where no package contents existed, thus returning empty resources.
As package changes, advance the reference to return up-to-date package
revision contents.

Make Git Server hostable in cluster (kptdev#2798)

* Make Git Server hostable in cluster
* Add makefile targets to build and push its images

Add PackageRevision Approval API (kptdev#2810)

* Add PackageRevision .../approval subresource
* Generate client

Add Clone Package Test (kptdev#2807)

* Add Clone Package Test
* Fix issues with git server's reporting of symbolic references
martinmaly pushed a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022
feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705)

This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.

make kpt binary optional in test harness (kptdev#2758)

feat: enable STDIN apply and destroy using RG inventory (kptdev#2709)

This commit enables actuation from STDIN using inventory information
that is stored in a ResourceGroup file.

This feature is currently behind a feature gate and will be exposed to
users as a CLI flag in a future commit/PR.

updated the version to 1.0.0-beta.13 (kptdev#2806)

Merge main into porch

Correct Apache License Text (kptdev#2675)

LICENSE file is supposed to be an exact copy of
https://www.apache.org/licenses/LICENSE-2.0.txt.
martinmaly pushed a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022
feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705)

This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.

make kpt binary optional in test harness (kptdev#2758)

feat: enable STDIN apply and destroy using RG inventory (kptdev#2709)

This commit enables actuation from STDIN using inventory information
that is stored in a ResourceGroup file.

This feature is currently behind a feature gate and will be exposed to
users as a CLI flag in a future commit/PR.

updated the version to 1.0.0-beta.13 (kptdev#2806)

Merge main into porch

Correct Apache License Text (kptdev#2675)

LICENSE file is supposed to be an exact copy of
https://www.apache.org/licenses/LICENSE-2.0.txt.
martinmaly pushed a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022
feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705)

This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.

make kpt binary optional in test harness (kptdev#2758)

feat: enable STDIN apply and destroy using RG inventory (kptdev#2709)

This commit enables actuation from STDIN using inventory information
that is stored in a ResourceGroup file.

This feature is currently behind a feature gate and will be exposed to
users as a CLI flag in a future commit/PR.

updated the version to 1.0.0-beta.13 (kptdev#2806)

Merge main into porch

Correct Apache License Text (kptdev#2675)

LICENSE file is supposed to be an exact copy of
https://www.apache.org/licenses/LICENSE-2.0.txt.
martinmaly pushed a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022
feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705)

This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.

make kpt binary optional in test harness (kptdev#2758)

feat: enable STDIN apply and destroy using RG inventory (kptdev#2709)

This commit enables actuation from STDIN using inventory information
that is stored in a ResourceGroup file.

This feature is currently behind a feature gate and will be exposed to
users as a CLI flag in a future commit/PR.

updated the version to 1.0.0-beta.13 (kptdev#2806)

Merge main into porch

Correct Apache License Text (kptdev#2675)

LICENSE file is supposed to be an exact copy of
https://www.apache.org/licenses/LICENSE-2.0.txt.
martinmaly pushed a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022
feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705)

This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.

make kpt binary optional in test harness (kptdev#2758)

feat: enable STDIN apply and destroy using RG inventory (kptdev#2709)

This commit enables actuation from STDIN using inventory information
that is stored in a ResourceGroup file.

This feature is currently behind a feature gate and will be exposed to
users as a CLI flag in a future commit/PR.

updated the version to 1.0.0-beta.13 (kptdev#2806)

Merge main into porch

Correct Apache License Text (kptdev#2675)

LICENSE file is supposed to be an exact copy of
https://www.apache.org/licenses/LICENSE-2.0.txt.
martinmaly pushed a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022
feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705)

This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.

make kpt binary optional in test harness (kptdev#2758)

feat: enable STDIN apply and destroy using RG inventory (kptdev#2709)

This commit enables actuation from STDIN using inventory information
that is stored in a ResourceGroup file.

This feature is currently behind a feature gate and will be exposed to
users as a CLI flag in a future commit/PR.

updated the version to 1.0.0-beta.13 (kptdev#2806)

Merge main into porch

Correct Apache License Text (kptdev#2675)

LICENSE file is supposed to be an exact copy of
https://www.apache.org/licenses/LICENSE-2.0.txt.
martinmaly pushed a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022
feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705)

This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.

make kpt binary optional in test harness (kptdev#2758)

feat: enable STDIN apply and destroy using RG inventory (kptdev#2709)

This commit enables actuation from STDIN using inventory information
that is stored in a ResourceGroup file.

This feature is currently behind a feature gate and will be exposed to
users as a CLI flag in a future commit/PR.

updated the version to 1.0.0-beta.13 (kptdev#2806)

Merge main into porch

Correct Apache License Text (kptdev#2675)

LICENSE file is supposed to be an exact copy of
https://www.apache.org/licenses/LICENSE-2.0.txt.
martinmaly pushed a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022
feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705)

This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.

make kpt binary optional in test harness (kptdev#2758)

feat: enable STDIN apply and destroy using RG inventory (kptdev#2709)

This commit enables actuation from STDIN using inventory information
that is stored in a ResourceGroup file.

This feature is currently behind a feature gate and will be exposed to
users as a CLI flag in a future commit/PR.

updated the version to 1.0.0-beta.13 (kptdev#2806)

Merge main into porch

Correct Apache License Text (kptdev#2675)

LICENSE file is supposed to be an exact copy of
https://www.apache.org/licenses/LICENSE-2.0.txt.
martinmaly pushed a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022
Add structs for ResourceGroup objects (kptdev#2619)

* feat: Add structs for ResourceGroup objects

This commit introduces the required types to move inventory information
out from thee Kptfile and into a separate resourcegroup.yaml resource
file.

Since the ResourceGroup CRD was hard coded previously, and ResourceGroup
objects were always handled as unstructured objects, these type
definitions did not exist prior to this commit.

* refactor: Rename default meta struct for ResourceGroup

propose ownership change to facilitate PR reviews and improving kpt.dev docs (kptdev#2679)

docs: update kpt completion docs (kptdev#2673)

This change makes two key updates to the completion docs:
- Provide instructions to remove an artifact of previous installations
which breaks kpt completion functionality. This issue was reported by a
user of kpt.
- Direct the user to the help command for per-shell instructions to
enable kpt completion. Since the completion functionality is provided by
a third party library (cobra), this ensures the user is provided with
accurate and up to date instructions.

Ensure release license file exists (kptdev#2682)

* ci: ensure existence of lib.zip for releases

ensure existence of the lib.zip file for downstream releases. This is
done by creating a base zip file containing a README and updating it
with mozilla_repos if mozilla_repos is not empty.

* ci: use array for mozilla_repos

Update mozilla_repos to use an array rather than a string with space
delimited elements. This is intended to provide a more straightforward
type for storing a list of elements as well as provide more explicit
word splitting in line with SC2086.

fix: report NotFound status for deleted KCC resources (kptdev#2689)

* fix: report NotFound status for deleted KCC resources

The current custom StatusReader for Config Connector resources will
report Unknown status when a Config Connector resource is not found (aka
deleted). This causes the `kpt live destroy` reconcile loop to run
forever since it expects a NotFound status to end. This commit ensures
that deleted resources report a NotFound status instead.

* refactor: Fix linting issue for unkeyed fields in composite literal

Fix missing colon in design doc (kptdev#2693)

This helps people that copy and paste from the examples!

Main

Save inventory information into resourcegroup.yaml (kptdev#2615)

Fix site sidebar to also show annotation references (kptdev#2734)

effective customizations chapter (kptdev#2659)

merging this as an MVP and will iterate based on additional requests and customer feedback.

docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)

Mods: Merge main into porch

Main

feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705)

This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.

make kpt binary optional in test harness (kptdev#2758)

feat: enable STDIN apply and destroy using RG inventory (kptdev#2709)

This commit enables actuation from STDIN using inventory information
that is stored in a ResourceGroup file.

This feature is currently behind a feature gate and will be exposed to
users as a CLI flag in a future commit/PR.

updated the version to 1.0.0-beta.13 (kptdev#2806)

Merge main into porch

Correct Apache License Text (kptdev#2675)

LICENSE file is supposed to be an exact copy of
https://www.apache.org/licenses/LICENSE-2.0.txt.
martinmaly pushed a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022
Add structs for ResourceGroup objects (kptdev#2619)

* feat: Add structs for ResourceGroup objects

This commit introduces the required types to move inventory information
out from thee Kptfile and into a separate resourcegroup.yaml resource
file.

Since the ResourceGroup CRD was hard coded previously, and ResourceGroup
objects were always handled as unstructured objects, these type
definitions did not exist prior to this commit.

* refactor: Rename default meta struct for ResourceGroup

propose ownership change to facilitate PR reviews and improving kpt.dev docs (kptdev#2679)

docs: update kpt completion docs (kptdev#2673)

This change makes two key updates to the completion docs:
- Provide instructions to remove an artifact of previous installations
which breaks kpt completion functionality. This issue was reported by a
user of kpt.
- Direct the user to the help command for per-shell instructions to
enable kpt completion. Since the completion functionality is provided by
a third party library (cobra), this ensures the user is provided with
accurate and up to date instructions.

Ensure release license file exists (kptdev#2682)

* ci: ensure existence of lib.zip for releases

ensure existence of the lib.zip file for downstream releases. This is
done by creating a base zip file containing a README and updating it
with mozilla_repos if mozilla_repos is not empty.

* ci: use array for mozilla_repos

Update mozilla_repos to use an array rather than a string with space
delimited elements. This is intended to provide a more straightforward
type for storing a list of elements as well as provide more explicit
word splitting in line with SC2086.

fix: report NotFound status for deleted KCC resources (kptdev#2689)

* fix: report NotFound status for deleted KCC resources

The current custom StatusReader for Config Connector resources will
report Unknown status when a Config Connector resource is not found (aka
deleted). This causes the `kpt live destroy` reconcile loop to run
forever since it expects a NotFound status to end. This commit ensures
that deleted resources report a NotFound status instead.

* refactor: Fix linting issue for unkeyed fields in composite literal

Fix missing colon in design doc (kptdev#2693)

This helps people that copy and paste from the examples!

Main

Save inventory information into resourcegroup.yaml (kptdev#2615)

Fix site sidebar to also show annotation references (kptdev#2734)

effective customizations chapter (kptdev#2659)

merging this as an MVP and will iterate based on additional requests and customer feedback.

docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)

Mods: Merge main into porch

Main

feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705)

This commit enables `kpt live migrate` to migrate inventory information
from either a ConfigMap or Kptfile to a separate ResourceGroup file.

This functionality is currently behind a feature gate and is not exposed
to the user via any CLI flags. Enabling of this feature to users will be
done later.

make kpt binary optional in test harness (kptdev#2758)

feat: enable STDIN apply and destroy using RG inventory (kptdev#2709)

This commit enables actuation from STDIN using inventory information
that is stored in a ResourceGroup file.

This feature is currently behind a feature gate and will be exposed to
users as a CLI flag in a future commit/PR.

updated the version to 1.0.0-beta.13 (kptdev#2806)

Merge main into porch

Correct Apache License Text (kptdev#2675)

LICENSE file is supposed to be an exact copy of
https://www.apache.org/licenses/LICENSE-2.0.txt.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants