Save inventory information into resourcegroup.yaml #2615

rquitales · 2021-12-09T18:19:16Z

This PR that implements the kpt live init functionality defined within the kpt live apply from STDIN design doc.

Additions:

Go struct type defining the new resourcegroup object
Logic to save inventory information within the new resourcegroup.yaml file

Additional work required:

Updating tests
Refactoring code for better maintainability

internal/cmdliveinit/cmdliveinit.go

internal/pkg/pkg.go

pkg/api/resourcegroup/v1alpha1/types.go

droot

Some comments.

internal/cmdliveinit/cmdliveinit.go

internal/pkg/pkg.go

pkg/api/resourcegroup/v1alpha1/types.go

pkg/resourcegroup/resourcegrouputil/util.go

e2e/live/end-to-end-test.sh

internal/cmdliveinit/cmdliveinit.go

internal/cmdmigrate/migratecmd_test.go

pkg/api/resourcegroup/v1alpha1/types.go

yuwenma · 2022-01-28T21:19:57Z

Per offline discussion about disabling e2e tests in this PR.
I think we do not want to disable it since it will make other PRs lose e2e coverage. @rquitales proposed using feature branch which is fine to me, except that it may cause regression when merging back. Considering the inventory to resourcegroup change will touch many files, this may not be a trivial issue.

After taking a deeper look at the e2e test and the PR, to not disable e2e tests and contribute to main branch, you can 1. keep the inventory in Kptfile, 2. do not change surfaces like adding new flags 3. create/modify resourcegroup file from args which also write/modify the inventory in Kptfile. What do you think?

@droot Will the porch (or part of the porch code) be merged to main in Feb?

mortent

A few comments. My biggest question is how the functionality is split across the different PRs and if it is safe to merge them one-by-one?

e2e/live/end-to-end-test.sh

internal/cmdliveinit/cmdliveinit.go

internal/cmdliveinit/cmdliveinit_test.go

rquitales · 2022-01-31T19:54:38Z

@yuwenma @mortent The PRs have been reworked for this functionality to be behind a feature gate. This will enable kpt releases to keep occurring without the current user flow breaking. The existing e2e tests have been re-enabled to ensure these flows are unchanged by the new additions.

internal/cmdliveinit/cmdliveinit_test.go

internal/pkg/pkg.go

This commit enables `kpt live init` to write the package inventory information to a separate resourcegroup file. This enables kpt to allow inputs from other hydration tools. This feature is currently being feature gated, and not exposed to users. A future commit/PR will enable this once all `migrate`, `apply` and `detroy` functionality exists.

internal/cmdliveinit/cmdliveinit_test.go

internal/pkg/pkg.go

yuwenma · 2022-02-03T20:48:16Z

/lgtm

+cc @droot FYI this PR touches pkg (right now it only adds a field to Pkg).

I still have several questions, but since you mentioned they are side effect of code refactoring, I'm fine to leave them as it is. Just to make sure you mark them with the TODO since this feature touches many files other people are actively working on.

As a backlog reminder, please make sure the following items are resolved once you finished the code refactoring:

"readRGFile" should not belong to ./internal/pkg/pkg.go. Because the main goal is to separate resourcegroup which is a live specific config from Kptfile. So please make sure those RG are moved to internal/*live*.go or live util packages.
there's an ambiguity between RG and inventory. inventory in Kptfile is named under Inventory, but referring inventory in resourcegroup.yaml will further confuse users. For example, InvInRGExistsError and InvInKfExistsError both tells user the inventory already exist. Please make sure RG no longer surfaces inventory.
Since kpt now has two managed files Kptfile and resourcegroup.yaml, as we discussed in the comments, suggest use share logic for file operations like Read Write etc.

Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)

Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739) Merge main into porch Combined Main Commit feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705) This commit enables `kpt live migrate` to migrate inventory information from either a ConfigMap or Kptfile to a separate ResourceGroup file. This functionality is currently behind a feature gate and is not exposed to the user via any CLI flags. Enabling of this feature to users will be done later. make kpt binary optional in test harness (kptdev#2758) feat: enable STDIN apply and destroy using RG inventory (kptdev#2709) This commit enables actuation from STDIN using inventory information that is stored in a ResourceGroup file. This feature is currently behind a feature gate and will be exposed to users as a CLI flag in a future commit/PR. updated the version to 1.0.0-beta.13 (kptdev#2806) Merge main into porch Combined Porch Commit Streamline Docker Image Building (kptdev#2752) * Use the same naming pattern * Enable nested Makefiles to inherit values from parent * Enable explicit image tagging (i.e. using `IMAGE_TAG=$(git rev-parse --short HEAD) make build-images` * Enable `make build-images` and `make push-images` Run Porch With Consistent Working Dir (kptdev#2753) `go run` doesn't support setting working directory; use `go build` instead. Add subpkgs test for porch (kptdev#2754) * Add subpkgs test for porch * Add runtime Basic e2e Test (kptdev#2755) * Basic e2e Test * Simple harness to reduce some boilerplate * Add a simple git repo test Refactor Server Startup (kptdev#2770) * Start background processing in Run function * Shut down when stopCh is closed Tidy in 1.17 compatible mode (kptdev#2769) Occasionally `make tidy` fails complaining about 1.16 vs. 1.17 golang; this should keep our mods 1.17 compatible with fewer `make tidy` failures. Create Engine using Options (kptdev#2771) Create options for the common engine configuration options. Support lazy credential resolution (kptdev#2772) * Add CredentialResolver interface * Add WithCredentialResolver engine option * Use lazy credential resolution when interacting with Git Pass Context to OpenRepository (kptdev#2773) Use Context to Drive Server Shutdown (kptdev#2774) Turns out k8s apiserver supports both and we can get the core context to listen on instead of just getting the close channel. Clean up Required/Optional API Fields (kptdev#2778) Git branch and directory can be optional (defaulting to `main` and `/` respectively, while `SecretRef.Name` is required since nameless secret reference is unhelpful. Easier deployment onto GKE (kptdev#2776) Support Creating Porch Deployment Config (kptdev#2777) * Create `deployment-config` and `push-and-deploy` make targets * Use kpt function to set images on the deployment config Update Porch Deployment and Instructions (kptdev#2782) * Set workload identity service accounts via set-annotations * Simplify instructions * Build at Git tag * Combine Deployment Config in Same Directory * Rename config files, assign 0 to CRDs. * `kubectl apply` recursively just in case we add more config later Use controller-gen v0.8.0 (kptdev#2780) Use consistent version of controller-gen (v0.8.0) crd:preserveUnknownFields marker has been removed (`false` was the required value for v1 CRDs). Set renderer when building CaDEngine (kptdev#2787) Otherwise we crash when trying to render a package. Fix missing error handling (kptdev#2784) We weren't checking errors when building a CaDEngine. Fix small typo (kptdev#2793) Don't call into kpt render if we don't have a package (kptdev#2788) When creating an empty package, we get an error otherwise (as kpt render doesn't work if it doesn't have a package). Combined Porch Commit Split git_test To Use Git Server (kptdev#2797) Enable e2e tests to use git server Add CreatePackageRevision Test (kptdev#2800) Simple test that creates a package by cloning from upstream repository. Cache apply-setters:v0.1 function (kptdev#2799) Some blueprints we encounter in tests use the older version of the function. Return Git PackageRevisionResources Correctly (kptdev#2801) Newly created Git package returned package resources at its old commit SHA where no package contents existed, thus returning empty resources. As package changes, advance the reference to return up-to-date package revision contents. Make Git Server hostable in cluster (kptdev#2798) * Make Git Server hostable in cluster * Add makefile targets to build and push its images Add PackageRevision Approval API (kptdev#2810) * Add PackageRevision .../approval subresource * Generate client Add Clone Package Test (kptdev#2807) * Add Clone Package Test * Fix issues with git server's reporting of symbolic references

Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)

Add structs for ResourceGroup objects (kptdev#2619) * feat: Add structs for ResourceGroup objects This commit introduces the required types to move inventory information out from thee Kptfile and into a separate resourcegroup.yaml resource file. Since the ResourceGroup CRD was hard coded previously, and ResourceGroup objects were always handled as unstructured objects, these type definitions did not exist prior to this commit. * refactor: Rename default meta struct for ResourceGroup propose ownership change to facilitate PR reviews and improving kpt.dev docs (kptdev#2679) docs: update kpt completion docs (kptdev#2673) This change makes two key updates to the completion docs: - Provide instructions to remove an artifact of previous installations which breaks kpt completion functionality. This issue was reported by a user of kpt. - Direct the user to the help command for per-shell instructions to enable kpt completion. Since the completion functionality is provided by a third party library (cobra), this ensures the user is provided with accurate and up to date instructions. Ensure release license file exists (kptdev#2682) * ci: ensure existence of lib.zip for releases ensure existence of the lib.zip file for downstream releases. This is done by creating a base zip file containing a README and updating it with mozilla_repos if mozilla_repos is not empty. * ci: use array for mozilla_repos Update mozilla_repos to use an array rather than a string with space delimited elements. This is intended to provide a more straightforward type for storing a list of elements as well as provide more explicit word splitting in line with SC2086. fix: report NotFound status for deleted KCC resources (kptdev#2689) * fix: report NotFound status for deleted KCC resources The current custom StatusReader for Config Connector resources will report Unknown status when a Config Connector resource is not found (aka deleted). This causes the `kpt live destroy` reconcile loop to run forever since it expects a NotFound status to end. This commit ensures that deleted resources report a NotFound status instead. * refactor: Fix linting issue for unkeyed fields in composite literal Fix missing colon in design doc (kptdev#2693) This helps people that copy and paste from the examples! Main Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739) Mods: Merge main into porch Main feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705) This commit enables `kpt live migrate` to migrate inventory information from either a ConfigMap or Kptfile to a separate ResourceGroup file. This functionality is currently behind a feature gate and is not exposed to the user via any CLI flags. Enabling of this feature to users will be done later. make kpt binary optional in test harness (kptdev#2758) feat: enable STDIN apply and destroy using RG inventory (kptdev#2709) This commit enables actuation from STDIN using inventory information that is stored in a ResourceGroup file. This feature is currently behind a feature gate and will be exposed to users as a CLI flag in a future commit/PR. updated the version to 1.0.0-beta.13 (kptdev#2806) Merge main into porch Correct Apache License Text (kptdev#2675) LICENSE file is supposed to be an exact copy of https://www.apache.org/licenses/LICENSE-2.0.txt.

rquitales requested review from droot, mengqiy, mortent and phanimarupaka as code owners December 9, 2021 18:19

rquitales force-pushed the stdin-actuation-1 branch from 5fa1ee6 to 2f47198 Compare December 9, 2021 18:39

rquitales marked this pull request as draft December 9, 2021 20:20

rquitales mentioned this pull request Dec 9, 2021

Enable kpt live migrate to work with resourcegroup object file #2617

Closed

yuwenma reviewed Dec 9, 2021

View reviewed changes

internal/cmdliveinit/cmdliveinit.go Outdated Show resolved Hide resolved