effective customizations chapter #2659
Conversation
There was a problem hiding this comment.
I think the content appears to be more abstract and i have a few suggestions mostly around: starting with a concrete example to describe the scenario, follow up with concrete solution and then discuss how existing tools etc. solve and discuss the trade offs.
I feel it might take us a few iteration to nail down the exact content, we should iterate over the content as effective customization guide in Guides and once it starts to shape up as more authoritative, we can pull it in as a chapter in book.
| ## Solutions: | ||
|
|
||
| 1. When a complex configuration needs to be generated the package author can create a generator function using turing complete languages and debugging tools. Example of such a function is [folder generation]. The output of the function is plain old KRM. | ||
|
|
There was a problem hiding this comment.
I would suggest a ordering change because reading it as a user feels we are anchoring too much on the negatives of other existing techniques before even showing how to do it well in kpt.
Starting scenario with a concrete use-case. For ex. where we talk a bit about the folder generation use-cases show sparse config and what resources we want to stamp out and then discussing how to do it the kpt way and then discussing other techniques (and how other tools do it) and explain the trade offs will be better ordering.
There was a problem hiding this comment.
The generation pattern isn't well supported, so I don't think the folder generator is a good example
There was a problem hiding this comment.
The case for a general-purpose programming language would be similar to Pulumi or cdk8s. The difference between representing the configuration as code and assuming exclusive actuation and operating on the configuration with code is composability and interoperability.
| @@ -0,0 +1,15 @@ | |||
| ## Scenario: | |||
|
|
|||
| I’d like to limit what my package consumers can do with my package and it feels safer to just provide a string replacement in one place so they know not to alter the configuration outside of the few places that I designated as OK places to change. | |||
There was a problem hiding this comment.
I had to re-read the sentence couple of times to understand the intent :) Can we start with a concrete use-case to describe the problem that can help the reader visualize the scenario better.
And then suggest how to do it the kpt way and then follow up with how existing tools (or techniques do) and discuss the trade-offs.
|
|
||
| ## Solutions: | ||
|
|
||
| 1. General ways to describe policy already exist. kpt has a [gatekeeper] function that allows the author to describe intended limitations for a class of resources or the entire package giving the consumer the freedom to customize and get an error or a warning when the policy is violated. |
There was a problem hiding this comment.
I think this is very generic and as a user it's not clear to me how to really apply it. We should describe the solution a bit more concretely here in context of an example. For ex.. in the tenant use case, enforcing the package to have only one namespace was one invariant we enforced using a custom function.
|
@bgrant0607 @justinsb would be good get another pair of eyes on it. |
Happy to re-root this into the guides section, that's not a problem. I feel like getting the right beginning packages and showcasing the problems at scale is a bit hard though. Let me think about the best way to create it. We did have an internal document around the learnings of not using parameters. I do feel a bit like this is turning into a bigger project than what I signed up for 😄 . Let me see if I can get something with more concrete examples though. |
Sorry about that :) How about we turn this into a guide with minimal changes and get it in. Then I can iterate on it. |
| Kubernetes configuration packages and customizations go hand in hand, all the | ||
| packaging tools enable package customization. In this chapter we cover effective | ||
| customizations techniques the kpt hydration and packaging enables. We show how | ||
| providing customization through parameters has some pitfalls and recommend |
There was a problem hiding this comment.
| providing customization through parameters has some pitfalls and recommend | ||
| alternatives where the contents of the package are no hidden behind a facade. | ||
| Some of these alternatives are only possible because kpt has made an investment | ||
| into bulk editing with KRM functions and upstream merging. |
There was a problem hiding this comment.
site/book/07-effective-customizations/01-single-value-replacement.md
Outdated
Show resolved
Hide resolved
|
|
||
| ## Problems | ||
|
|
||
| 1. With popularity the single values inevitably expand to provide a facade to a large portion of the data defeating the purpose of minimizing the cognitive load. |
There was a problem hiding this comment.
It would be pretty easy to find a helm chart to link to as an example. I've used the jenkins chart in the past. A search for "universal helm chart" found this:
https://github.com/OleksandrUA/universal-helm-charts/blob/master/chart-service/_common/templates/deployment.yaml
There was a problem hiding this comment.
Also, I call parameterizing every field the struct constructor pattern.
There was a problem hiding this comment.
There was a problem hiding this comment.
I agree that it's not hard to find a chart that suffers from the parameter problem. I think what @droot is asking for here is before and after so I'd have to create a feasible package for whatever it is that I showcase. There is also a limit to the size of the example to show case the end result. Maybe I can figure out how to just show code snippets here.
|
|
||
| ## Suggestions | ||
|
|
||
| 1. kpt allows the user to edit a particular value directly in the configuration data and will handle upstream merge. When [editing the yaml] directly the consumers are not confined to the parameters that the package author has |
There was a problem hiding this comment.
We could also explain this as similar to kustomize but where the patch is automatically derived
|
|
||
| ## Problems: | ||
|
|
||
| 1. Over time the templating logic becomes its own language that becomes very complex. Debugging the template generation becomes a task of its own. |
There was a problem hiding this comment.
There was a problem hiding this comment.
thanks for linking to this, this is a good size package that we can possibly adopt since it's small and very readable.
The guides section has no introduction or order to it. I am not sure how additional sections are supposed to get plugged in. It's much more clear with book chapters. |
|
update: added code samples to the second recipe. |
There was a problem hiding this comment.
This is a difficult PR :) and we agree we need to do a lot more to convey the ideas here. I am fine with merging this and iterating along the way given that we will have some more understanding after @yuwenma and @bgrant0607 have completed their work on the blueprints.
site/book/07-effective-customizations/01-single-value-replacement.md
Outdated
Show resolved
Hide resolved
| pipeline: | ||
| validators: | ||
| - image: gcr.io/kpt-fn/gatekeeper:v0.2 | ||
| ``` |
There was a problem hiding this comment.
I am having a bit of difficulty in coming up with the best way to convey this scenario. The solution is coming across more as guardrails becomes the guide (you reach the destination by hitting the guardrails which is not a guided experience).
I am okay getting this in and then figure out what is the best way to convey this.
merging this as an MVP and will iterate based on additional requests and customer feedback.
Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)
Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)
Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739) Merge main into porch Combined Main Commit feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705) This commit enables `kpt live migrate` to migrate inventory information from either a ConfigMap or Kptfile to a separate ResourceGroup file. This functionality is currently behind a feature gate and is not exposed to the user via any CLI flags. Enabling of this feature to users will be done later. make kpt binary optional in test harness (kptdev#2758) feat: enable STDIN apply and destroy using RG inventory (kptdev#2709) This commit enables actuation from STDIN using inventory information that is stored in a ResourceGroup file. This feature is currently behind a feature gate and will be exposed to users as a CLI flag in a future commit/PR. updated the version to 1.0.0-beta.13 (kptdev#2806) Merge main into porch Combined Porch Commit Streamline Docker Image Building (kptdev#2752) * Use the same naming pattern * Enable nested Makefiles to inherit values from parent * Enable explicit image tagging (i.e. using `IMAGE_TAG=$(git rev-parse --short HEAD) make build-images` * Enable `make build-images` and `make push-images` Run Porch With Consistent Working Dir (kptdev#2753) `go run` doesn't support setting working directory; use `go build` instead. Add subpkgs test for porch (kptdev#2754) * Add subpkgs test for porch * Add runtime Basic e2e Test (kptdev#2755) * Basic e2e Test * Simple harness to reduce some boilerplate * Add a simple git repo test Refactor Server Startup (kptdev#2770) * Start background processing in Run function * Shut down when stopCh is closed Tidy in 1.17 compatible mode (kptdev#2769) Occasionally `make tidy` fails complaining about 1.16 vs. 1.17 golang; this should keep our mods 1.17 compatible with fewer `make tidy` failures. Create Engine using Options (kptdev#2771) Create options for the common engine configuration options. Support lazy credential resolution (kptdev#2772) * Add CredentialResolver interface * Add WithCredentialResolver engine option * Use lazy credential resolution when interacting with Git Pass Context to OpenRepository (kptdev#2773) Use Context to Drive Server Shutdown (kptdev#2774) Turns out k8s apiserver supports both and we can get the core context to listen on instead of just getting the close channel. Clean up Required/Optional API Fields (kptdev#2778) Git branch and directory can be optional (defaulting to `main` and `/` respectively, while `SecretRef.Name` is required since nameless secret reference is unhelpful. Easier deployment onto GKE (kptdev#2776) Support Creating Porch Deployment Config (kptdev#2777) * Create `deployment-config` and `push-and-deploy` make targets * Use kpt function to set images on the deployment config Update Porch Deployment and Instructions (kptdev#2782) * Set workload identity service accounts via set-annotations * Simplify instructions * Build at Git tag * Combine Deployment Config in Same Directory * Rename config files, assign 0 to CRDs. * `kubectl apply` recursively just in case we add more config later Use controller-gen v0.8.0 (kptdev#2780) Use consistent version of controller-gen (v0.8.0) crd:preserveUnknownFields marker has been removed (`false` was the required value for v1 CRDs). Set renderer when building CaDEngine (kptdev#2787) Otherwise we crash when trying to render a package. Fix missing error handling (kptdev#2784) We weren't checking errors when building a CaDEngine. Fix small typo (kptdev#2793) Don't call into kpt render if we don't have a package (kptdev#2788) When creating an empty package, we get an error otherwise (as kpt render doesn't work if it doesn't have a package). Combined Porch Commit Split git_test To Use Git Server (kptdev#2797) Enable e2e tests to use git server Add CreatePackageRevision Test (kptdev#2800) Simple test that creates a package by cloning from upstream repository. Cache apply-setters:v0.1 function (kptdev#2799) Some blueprints we encounter in tests use the older version of the function. Return Git PackageRevisionResources Correctly (kptdev#2801) Newly created Git package returned package resources at its old commit SHA where no package contents existed, thus returning empty resources. As package changes, advance the reference to return up-to-date package revision contents. Make Git Server hostable in cluster (kptdev#2798) * Make Git Server hostable in cluster * Add makefile targets to build and push its images Add PackageRevision Approval API (kptdev#2810) * Add PackageRevision .../approval subresource * Generate client Add Clone Package Test (kptdev#2807) * Add Clone Package Test * Fix issues with git server's reporting of symbolic references
Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)
Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)
Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)
Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)
Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)
Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)
Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)
Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739)
Add structs for ResourceGroup objects (kptdev#2619) * feat: Add structs for ResourceGroup objects This commit introduces the required types to move inventory information out from thee Kptfile and into a separate resourcegroup.yaml resource file. Since the ResourceGroup CRD was hard coded previously, and ResourceGroup objects were always handled as unstructured objects, these type definitions did not exist prior to this commit. * refactor: Rename default meta struct for ResourceGroup propose ownership change to facilitate PR reviews and improving kpt.dev docs (kptdev#2679) docs: update kpt completion docs (kptdev#2673) This change makes two key updates to the completion docs: - Provide instructions to remove an artifact of previous installations which breaks kpt completion functionality. This issue was reported by a user of kpt. - Direct the user to the help command for per-shell instructions to enable kpt completion. Since the completion functionality is provided by a third party library (cobra), this ensures the user is provided with accurate and up to date instructions. Ensure release license file exists (kptdev#2682) * ci: ensure existence of lib.zip for releases ensure existence of the lib.zip file for downstream releases. This is done by creating a base zip file containing a README and updating it with mozilla_repos if mozilla_repos is not empty. * ci: use array for mozilla_repos Update mozilla_repos to use an array rather than a string with space delimited elements. This is intended to provide a more straightforward type for storing a list of elements as well as provide more explicit word splitting in line with SC2086. fix: report NotFound status for deleted KCC resources (kptdev#2689) * fix: report NotFound status for deleted KCC resources The current custom StatusReader for Config Connector resources will report Unknown status when a Config Connector resource is not found (aka deleted). This causes the `kpt live destroy` reconcile loop to run forever since it expects a NotFound status to end. This commit ensures that deleted resources report a NotFound status instead. * refactor: Fix linting issue for unkeyed fields in composite literal Fix missing colon in design doc (kptdev#2693) This helps people that copy and paste from the examples! Main Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739) Mods: Merge main into porch Main feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705) This commit enables `kpt live migrate` to migrate inventory information from either a ConfigMap or Kptfile to a separate ResourceGroup file. This functionality is currently behind a feature gate and is not exposed to the user via any CLI flags. Enabling of this feature to users will be done later. make kpt binary optional in test harness (kptdev#2758) feat: enable STDIN apply and destroy using RG inventory (kptdev#2709) This commit enables actuation from STDIN using inventory information that is stored in a ResourceGroup file. This feature is currently behind a feature gate and will be exposed to users as a CLI flag in a future commit/PR. updated the version to 1.0.0-beta.13 (kptdev#2806) Merge main into porch Correct Apache License Text (kptdev#2675) LICENSE file is supposed to be an exact copy of https://www.apache.org/licenses/LICENSE-2.0.txt.
Add structs for ResourceGroup objects (kptdev#2619) * feat: Add structs for ResourceGroup objects This commit introduces the required types to move inventory information out from thee Kptfile and into a separate resourcegroup.yaml resource file. Since the ResourceGroup CRD was hard coded previously, and ResourceGroup objects were always handled as unstructured objects, these type definitions did not exist prior to this commit. * refactor: Rename default meta struct for ResourceGroup propose ownership change to facilitate PR reviews and improving kpt.dev docs (kptdev#2679) docs: update kpt completion docs (kptdev#2673) This change makes two key updates to the completion docs: - Provide instructions to remove an artifact of previous installations which breaks kpt completion functionality. This issue was reported by a user of kpt. - Direct the user to the help command for per-shell instructions to enable kpt completion. Since the completion functionality is provided by a third party library (cobra), this ensures the user is provided with accurate and up to date instructions. Ensure release license file exists (kptdev#2682) * ci: ensure existence of lib.zip for releases ensure existence of the lib.zip file for downstream releases. This is done by creating a base zip file containing a README and updating it with mozilla_repos if mozilla_repos is not empty. * ci: use array for mozilla_repos Update mozilla_repos to use an array rather than a string with space delimited elements. This is intended to provide a more straightforward type for storing a list of elements as well as provide more explicit word splitting in line with SC2086. fix: report NotFound status for deleted KCC resources (kptdev#2689) * fix: report NotFound status for deleted KCC resources The current custom StatusReader for Config Connector resources will report Unknown status when a Config Connector resource is not found (aka deleted). This causes the `kpt live destroy` reconcile loop to run forever since it expects a NotFound status to end. This commit ensures that deleted resources report a NotFound status instead. * refactor: Fix linting issue for unkeyed fields in composite literal Fix missing colon in design doc (kptdev#2693) This helps people that copy and paste from the examples! Main Save inventory information into resourcegroup.yaml (kptdev#2615) Fix site sidebar to also show annotation references (kptdev#2734) effective customizations chapter (kptdev#2659) merging this as an MVP and will iterate based on additional requests and customer feedback. docs: Describe how reconcile status is computed for Config Connector resources (kptdev#2739) Mods: Merge main into porch Main feat: enable migrating from Kptfile & CM to resourcegroup inventories (kptdev#2705) This commit enables `kpt live migrate` to migrate inventory information from either a ConfigMap or Kptfile to a separate ResourceGroup file. This functionality is currently behind a feature gate and is not exposed to the user via any CLI flags. Enabling of this feature to users will be done later. make kpt binary optional in test harness (kptdev#2758) feat: enable STDIN apply and destroy using RG inventory (kptdev#2709) This commit enables actuation from STDIN using inventory information that is stored in a ResourceGroup file. This feature is currently behind a feature gate and will be exposed to users as a CLI flag in a future commit/PR. updated the version to 1.0.0-beta.13 (kptdev#2806) Merge main into porch Correct Apache License Text (kptdev#2675) LICENSE file is supposed to be an exact copy of https://www.apache.org/licenses/LICENSE-2.0.txt.
Adding effective customizations chapter with 3 scenarios for now.