Create function-runner Deployment #2733

martinmaly · 2022-02-04T05:47:39Z

Create k8s configuration to deploy the function-runner gRPC service

Create function-runner Deployment (kptdev#2733) Use kpt Renderer in Porch (kptdev#2735) * Use kpt Renderer in Porch * Accept gRPC function runner endpoint RemoteRootSyncSet: use specified OCI image (kptdev#2722) Stop hard-coding and use the image the user specifies! RemoteRootSyncSet: apply changes with force (kptdev#2723) This lets us switch from client-side-applied to server-side-applied. RemoteRootSyncSet: Remove hack where we set namespace (kptdev#2724) It's now causing problems; we shouldn't need it and we should fix it later in the logic anyway (with a default namespace). Create push-images target to push all our images (kptdev#2721) Also a few tweaks to make the porch image build faster. Create dockerfile / makefile / manifest for pushing controllers to cluster (kptdev#2694) This lets us start to run these on the server, instead of just locally. Introduce mapTaskToMutation function (kptdev#2744) This is a small refactor so that we can reuse this logic in future.

Use apply-setters directly (kptdev#2719) Upgrade kpt dependencies (kptdev#2727) Use Lower-Level Function Runner Interface (kptdev#2730) * Add capability to override function runner during fnrender * Use Lower-Level Function Runner Interface The function framework operates at the level of IO streams so we need to use that in order to benefit from comment healing and other features. * Fix tests to rely on files-on-disk. Co-authored-by: droot <sunilarora@google.com> Use kyaml In-Memory Filesystem (kptdev#2731) Just noticed kyaml comes with in-memory filesystem implementation. Use it instead of custom one. Upgrade porch dependencies (kptdev#2726) Remove direct dependency on deprecated kustomize/api package Build Function Runner Docker Image (kptdev#2729) tests: Fix test flake from timestamp sanitization (kptdev#2713) (kptdev#2742) * Fix test flake from timestamp sanitization When timestamps "overlap", sanitization did not work correctly. For example: in 1s ... in 2.1s The first substitution would replace 1s => 0s in 0s ... in 2.0s The second substitution is predetermined, and replaces 2.1s => 0s, but 2.1s is no longer in the string. * Use new sanitization function in tests, remove old function Co-authored-by: Justin Santa Barbara <justinsb@google.com> Combined Porch Commit Create function-runner Deployment (kptdev#2733) Use kpt Renderer in Porch (kptdev#2735) * Use kpt Renderer in Porch * Accept gRPC function runner endpoint RemoteRootSyncSet: use specified OCI image (kptdev#2722) Stop hard-coding and use the image the user specifies! RemoteRootSyncSet: apply changes with force (kptdev#2723) This lets us switch from client-side-applied to server-side-applied. RemoteRootSyncSet: Remove hack where we set namespace (kptdev#2724) It's now causing problems; we shouldn't need it and we should fix it later in the logic anyway (with a default namespace). Create push-images target to push all our images (kptdev#2721) Also a few tweaks to make the porch image build faster. Create dockerfile / makefile / manifest for pushing controllers to cluster (kptdev#2694) This lets us start to run these on the server, instead of just locally. Introduce mapTaskToMutation function (kptdev#2744) This is a small refactor so that we can reuse this logic in future. Fix kpt file system interactions (kptdev#2746) * missing file.Close() call (filesys' memory filesystem doesn't support files opened multiple times) * pass filesystem into newFnConfig (it reads fn config from file) Combined Porch Commit Pass Function Config to Eval (kptdev#2747) * Move eval mutation into own file * Pass Function Config to Eval * Remove unused field (runner.fn) * Move ConfigMap construction back to engine * Eval preserves comments Connect to gRPC function runner (kptdev#2745) Fix crash (kptdev#2748) If gRPC call fails, res is nill :( Fix clone directory rewriting (kptdev#2749) After changes to git cloning we return PackageResources with configs at the "root directory". Therefore we don't need to rewrite config paths anymore. client generation: fix plural of packagerevisionresources (kptdev#2751) The client was using packagerevisionresourceses (extra "es"), which doesn't match the server name. Update kpt dependencies (kptdev#2743) * Upgrade dependencies * Upgrade go to 1.17

Create function-runner Deployment (kptdev#2733) Use kpt Renderer in Porch (kptdev#2735) * Use kpt Renderer in Porch * Accept gRPC function runner endpoint RemoteRootSyncSet: use specified OCI image (kptdev#2722) Stop hard-coding and use the image the user specifies! RemoteRootSyncSet: apply changes with force (kptdev#2723) This lets us switch from client-side-applied to server-side-applied. RemoteRootSyncSet: Remove hack where we set namespace (kptdev#2724) It's now causing problems; we shouldn't need it and we should fix it later in the logic anyway (with a default namespace). Create push-images target to push all our images (kptdev#2721) Also a few tweaks to make the porch image build faster. Create dockerfile / makefile / manifest for pushing controllers to cluster (kptdev#2694) This lets us start to run these on the server, instead of just locally. Introduce mapTaskToMutation function (kptdev#2744) This is a small refactor so that we can reuse this logic in future.

Remove Placeholder kptlib (kptdev#2665) * Fix Makefile Formatting * Remove dependency on kptlib * Tidy modules * Delete kptlib * Remove kptlib replace Clause Add Porch Workflow Implement Direct Package Fetch (kptdev#2667) * Update VSCode Workspace * Add launch.json * Move Clone Logic Into Own File * Implement Git pacakge fetch Initial fn Interface (kptdev#2676) Initial version of the function interfaces. Upgrade go-containerregistry to v0.8.0 (kptdev#2674) * Tidy Porch Modules * Ignore Modules In .build Directory Porch git: handle resolution of per-package versions (kptdev#2681) We can have a git tag that looks like `packageDir/version`, this allows for independent versioning of packages that share a git repo. We need to try both ref forms when resolving git packages. PackageDraft interface should not inherit from Package (kptdev#2678) In particular, some of the methods such as GetResources are ambiguous until we finish the draft (do we want the in-progress resources or the upstream resources?) Minor fixups to better surface errors (kptdev#2680) In general, we should always return the error, unless we actually want to mask it e.g. for security reasons (and in that case, we would normally expect to log the underlying error instead of wrapping). Clean up Makefile (kptdev#2685) * Remove init and deinit (no longer useful) * Fix module discovery * Clean up directory references Render and Eval via Simple Memory FS (kptdev#2684) * Render and Eval via Simple Memory FS For now the filesystem is inserted at lower level; In future changes the mutation Apply will accept the filesystem abstraction to avoid back-and-forth between filesystem and PackageResources. * Clean up Function Evaluation to use new interface * Remove unnecessary kpt module Working towards applying packages from a repository (kptdev#2686) First steps (with lots of hard-coding and hacks) towards applying from a repository to a cluster. Add priority and fairness to RBAC roles (kptdev#2692) aggregated-apiservers seem to query these resources, and spam the logs (at least) if they don't have permission. Applying: publish MVP status (kptdev#2695) We start to expose the status of the sync operation. Fix Docker Build (kptdev#2696) Remove kpt directory from Docker build. Tidy Go Modules (kptdev#2699) * Fix Go Modules * Run Tidy in Porch Workflow Add Roundtrip Tests (kptdev#2703) Expose Renderer api with FileSystem abstraction (kptdev#2683) * Expose Renderer api with FileSystem abstraction * Merge Fs methods in pkg.go * Update name * Merge ReadKptfile and address changes * Update porch go.sum * Update results write * Suggested changes Porch Create fix-all and verify-fix-all (kptdev#2702) * Create fix-all and verify-fix-all We run all the formatters / header makers etc. In CI, we run them and make sure that the git checkout is still clean. Ideally we would also run generate, but it is very slow, and it also requires GOPATH to be set. We can do that in CI later. * Add missing headers Fix few file headers (kptdev#2707) Define a FunctionEvaluator gRPC Interface (kptdev#2700) * Add install-protoc.sh script * Define a FunctionEvaluator interface * Create an empty server implementation Integrate porch with renderer interface (kptdev#2710) * Integrate porch with renderer interview * Suggested changes * Add implementation Porch Introduce applyset abstraction (kptdev#2697) Replacing our very primitive apply logic with something better using server-side apply. Git repository test (kptdev#2706) We create a simple git server, and we try round-tripping a package. Some hacks/todos uncovered that we need to tackle. Create Kptfile in config/deploy (kptdev#2708) Our package logic requires a Kptfile, so adding one (for now). Update FunctionRunner Interface (kptdev#2714) And make names more consistent. Simple gRPC kpt function runner service (kptdev#2711) Several functions are built into the server Docker image and are exec'd with input. Added simple client which can be used (in ./func directory): ``` go run ./client \ --address=192.168.8.202:9445 \ --package=./config/ \ --image=gcr.io/kpt-fn/set-namespace:v0.2.0 \ -- \ namespace=foo ``` Use simple evaluator in tests (kptdev#2717) This avoids dependency on docker in unit tests. The price is that the simple functions don't support full semantics, but they also run much faster. Fix os calls (kptdev#2718) git: create ApprovePackageRevision method, remove draft hack (kptdev#2712) * git: create ApprovePackageRevision method, remove draft hack Now that we can approve a package (in our tests at least), we can now fetch the package on the non-draft ref. * Update go-git to HEAD We want to pick up go-git/go-git#325 Use apply-setters directly (kptdev#2719) Upgrade kpt dependencies (kptdev#2727) Use Lower-Level Function Runner Interface (kptdev#2730) * Add capability to override function runner during fnrender * Use Lower-Level Function Runner Interface The function framework operates at the level of IO streams so we need to use that in order to benefit from comment healing and other features. * Fix tests to rely on files-on-disk. Co-authored-by: droot <sunilarora@google.com> Use kyaml In-Memory Filesystem (kptdev#2731) Just noticed kyaml comes with in-memory filesystem implementation. Use it instead of custom one. Upgrade porch dependencies (kptdev#2726) Remove direct dependency on deprecated kustomize/api package Build Function Runner Docker Image (kptdev#2729) tests: Fix test flake from timestamp sanitization (kptdev#2713) (kptdev#2742) * Fix test flake from timestamp sanitization When timestamps "overlap", sanitization did not work correctly. For example: in 1s ... in 2.1s The first substitution would replace 1s => 0s in 0s ... in 2.0s The second substitution is predetermined, and replaces 2.1s => 0s, but 2.1s is no longer in the string. * Use new sanitization function in tests, remove old function Co-authored-by: Justin Santa Barbara <justinsb@google.com> Porch Create function-runner Deployment (kptdev#2733) Use kpt Renderer in Porch (kptdev#2735) * Use kpt Renderer in Porch * Accept gRPC function runner endpoint RemoteRootSyncSet: use specified OCI image (kptdev#2722) Stop hard-coding and use the image the user specifies! RemoteRootSyncSet: apply changes with force (kptdev#2723) This lets us switch from client-side-applied to server-side-applied. RemoteRootSyncSet: Remove hack where we set namespace (kptdev#2724) It's now causing problems; we shouldn't need it and we should fix it later in the logic anyway (with a default namespace). Create push-images target to push all our images (kptdev#2721) Also a few tweaks to make the porch image build faster. Create dockerfile / makefile / manifest for pushing controllers to cluster (kptdev#2694) This lets us start to run these on the server, instead of just locally. Introduce mapTaskToMutation function (kptdev#2744) This is a small refactor so that we can reuse this logic in future. Fix kpt file system interactions (kptdev#2746) * missing file.Close() call (filesys' memory filesystem doesn't support files opened multiple times) * pass filesystem into newFnConfig (it reads fn config from file) Porch Pass Function Config to Eval (kptdev#2747) * Move eval mutation into own file * Pass Function Config to Eval * Remove unused field (runner.fn) * Move ConfigMap construction back to engine * Eval preserves comments Connect to gRPC function runner (kptdev#2745) Fix crash (kptdev#2748) If gRPC call fails, res is nill :( Fix clone directory rewriting (kptdev#2749) After changes to git cloning we return PackageResources with configs at the "root directory". Therefore we don't need to rewrite config paths anymore. client generation: fix plural of packagerevisionresources (kptdev#2751) The client was using packagerevisionresourceses (extra "es"), which doesn't match the server name. Update kpt dependencies (kptdev#2743) * Upgrade dependencies * Upgrade go to 1.17 Porch Streamline Docker Image Building (kptdev#2752) * Use the same naming pattern * Enable nested Makefiles to inherit values from parent * Enable explicit image tagging (i.e. using `IMAGE_TAG=$(git rev-parse --short HEAD) make build-images` * Enable `make build-images` and `make push-images` Run Porch With Consistent Working Dir (kptdev#2753) `go run` doesn't support setting working directory; use `go build` instead. Add subpkgs test for porch (kptdev#2754) * Add subpkgs test for porch * Add runtime Basic e2e Test (kptdev#2755) * Basic e2e Test * Simple harness to reduce some boilerplate * Add a simple git repo test Refactor Server Startup (kptdev#2770) * Start background processing in Run function * Shut down when stopCh is closed Tidy in 1.17 compatible mode (kptdev#2769) Occasionally `make tidy` fails complaining about 1.16 vs. 1.17 golang; this should keep our mods 1.17 compatible with fewer `make tidy` failures. Create Engine using Options (kptdev#2771) Create options for the common engine configuration options. Support lazy credential resolution (kptdev#2772) * Add CredentialResolver interface * Add WithCredentialResolver engine option * Use lazy credential resolution when interacting with Git Pass Context to OpenRepository (kptdev#2773) Use Context to Drive Server Shutdown (kptdev#2774) Turns out k8s apiserver supports both and we can get the core context to listen on instead of just getting the close channel. Clean up Required/Optional API Fields (kptdev#2778) Git branch and directory can be optional (defaulting to `main` and `/` respectively, while `SecretRef.Name` is required since nameless secret reference is unhelpful. Easier deployment onto GKE (kptdev#2776) Support Creating Porch Deployment Config (kptdev#2777) * Create `deployment-config` and `push-and-deploy` make targets * Use kpt function to set images on the deployment config Update Porch Deployment and Instructions (kptdev#2782) * Set workload identity service accounts via set-annotations * Simplify instructions * Build at Git tag * Combine Deployment Config in Same Directory * Rename config files, assign 0 to CRDs. * `kubectl apply` recursively just in case we add more config later Use controller-gen v0.8.0 (kptdev#2780) Use consistent version of controller-gen (v0.8.0) crd:preserveUnknownFields marker has been removed (`false` was the required value for v1 CRDs). Set renderer when building CaDEngine (kptdev#2787) Otherwise we crash when trying to render a package. Fix missing error handling (kptdev#2784) We weren't checking errors when building a CaDEngine. Fix small typo (kptdev#2793) Don't call into kpt render if we don't have a package (kptdev#2788) When creating an empty package, we get an error otherwise (as kpt render doesn't work if it doesn't have a package). Porch Split git_test To Use Git Server (kptdev#2797) Enable e2e tests to use git server Add CreatePackageRevision Test (kptdev#2800) Simple test that creates a package by cloning from upstream repository. Cache apply-setters:v0.1 function (kptdev#2799) Some blueprints we encounter in tests use the older version of the function. Return Git PackageRevisionResources Correctly (kptdev#2801) Newly created Git package returned package resources at its old commit SHA where no package contents existed, thus returning empty resources. As package changes, advance the reference to return up-to-date package revision contents. Make Git Server hostable in cluster (kptdev#2798) * Make Git Server hostable in cluster * Add makefile targets to build and push its images Add PackageRevision Approval API (kptdev#2810) * Add PackageRevision .../approval subresource * Generate client Add Clone Package Test (kptdev#2807) * Add Clone Package Test * Fix issues with git server's reporting of symbolic references

* Simple package orchestration (#2663) * Add Porch Workflow * Implement Direct Package Fetch (#2667) * Initial fn Interface (#2676) * handle resolution of per-package versions of git packages (#2681) * PackageDraft interface should not inherit from Package (#2678) * Add priority and fairness to RBAC roles (#2692) * Applying: publish MVP status (#2695) * Add Roundtrip Tests (#2703) * Expose Renderer api with FileSystem abstraction (#2683) * Define a FunctionEvaluator gRPC Interface (#2700) * Integrate porch with renderer interface (#2710) * Introduce applyset abstraction (#2697) * Git repository test (#2706) * Update FunctionRunner Interface (#2714) * Simple gRPC kpt function runner service (#2711) * Use simple evaluator in tests (#2717) * Use Lower-Level Function Runner Interface (#2730) * Use kyaml In-Memory Filesystem (#2731) * Build Function Runner Docker Image (#2729) * Create function-runner Deployment (#2733) * Use kpt Renderer in Porch (#2735) * RemoteRootSyncSet: use specified OCI image (#2722) * RemoteRootSyncSet: apply changes with force (#2723) * Create push-images target to push all our images (#2721) * Pass Function Config to Eval (#2747) * Connect to gRPC function runner (#2745) * Streamline Docker Image Building (#2752) * Basic e2e Test (#2755) * Refactor Server Startup (#2770) * Create Engine using Options (#2771) * Support lazy credential resolution (#2772) * Pass Context to OpenRepository (#2773) * Use Context to Drive Server Shutdown (#2774) * Easier deployment onto GKE (#2776) * Support Creating Porch Deployment Config (#2777) * Update Porch Deployment and Instructions (#2782) * Set renderer when building CaDEngine (#2787) * Split git_test To Use Git Server (#2797) * Add CreatePackageRevision Test (#2800) * Return Git PackageRevisionResources Correctly (#2801) * Make Git Server hostable in cluster (#2798) * Add PackageRevision Approval API (#2810) * Add Clone Package Test (#2807) Co-authored-by: Louis DeJardin <louis.dejardin@gmail.com> Co-authored-by: Louis DeJardin <dejardin@google.com> Co-authored-by: Justin Santa Barbara <justinsb@google.com> Co-authored-by: phani <phanitejamarupaka@gmail.com> Co-authored-by: droot <sunilarora@google.com> Co-authored-by: phani <pmarupaka@google.com>

Create function-runner Deployment

d9a27ca

martinmaly requested a review from justinsb February 4, 2022 05:47

justinsb approved these changes Feb 4, 2022

View reviewed changes

martinmaly merged commit 8f6e79f into kptdev:porch Feb 4, 2022

martinmaly deleted the deploy branch February 4, 2022 20:55

martinmaly added a commit to martinmaly/kpt that referenced this pull request Feb 18, 2022

Create function-runner Deployment (kptdev#2733)

156bc76

martinmaly mentioned this pull request Feb 18, 2022

Merge Porch into main branch #2815

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create function-runner Deployment #2733

Create function-runner Deployment #2733

martinmaly commented Feb 4, 2022

Create function-runner Deployment #2733

Create function-runner Deployment #2733

Conversation

martinmaly commented Feb 4, 2022