Skip to content

profile-pod/profile-pod-operator

Repository files navigation

profile-pod-operator

A Kubernetes operator for profiling applications running inside pods in a simple way with low-overhead.

Description

The profile-pod-operator visualizing CPU time spent in the application functions using FlameGraph. The operator's goal is to help you pin down where your application spend too much time in a simple way with low-overhead and without any modification. By not require any modification or restart for existing applications and by using low-overhead profilers, this is great for recording flame graph data from an already running application in production environment that you don't want to interrupt.

Profile an application

To profile an application, a PodFlame custom resource needs to be created in the same namespace as the target application. Run the following command to create it:

cat << EOF | kubectl apply -f -
apiVersion: profilepod.io/v1alpha1
kind: PodFlame
metadata:
  name: my-app-flame
  namespace: my-app-namespace
spec:
  targetPod: my-app-54674f9647-jvm98 # The name of the pod you want to profile.
EOF

The following optional fields can be added to the PodFlame resource spec:

    duration: 30s # The profiling duration in seconds (s/S) or minutins (m/M). default: 2m.
    containerName: myapp # Require when the pod contains more then one container. 

Note: the PodFlame resource is immutable, if changes are required to a PodFlame resource, destroying the current resource and rebuilding that resource with required changes.

After PodFlame resource is created, an agent pod will be created by the operator in the same node as the target pod who was specified in the PodFlame spec. The agent pod is a high privileged pod, which detect the target application programming language and the target application process id, and runs a profiler suitable for the requested application. Once the Profile is done and flamegraph is generated for the application, it is placed in the .status.flameGraph of the corresponding PodFlame resource. Run the following command to get it:

kubectl get pf my-app-flame -n my-app-namespace -o jsonpath='{.status.flameGraph}' | base64 -d | gunzip > myapp-flamegraph.html

Note: the high privileged agent pod is created in the operator namespace, therefore, allow any unrestrictive policy in all profiled namespaces when using Pod Security admission controller (PSA) or similar enforcement tools should not be a concern.

Getting Started

You’ll need a Kubernetes cluster to run against. You can use KIND to get a local cluster for testing, or run against a remote cluster. Note: Your controller will automatically use the current context in your kubeconfig file (i.e. whatever cluster kubectl cluster-info shows).

Running on the cluster

  1. Install Instances of Custom Resources:
kubectl apply -f config/samples/
  1. Build and push your image to the location specified by IMG:
make docker-build docker-push IMG=<some-registry>/profile-pod-operator:tag
  1. Deploy the controller to the cluster with the image specified by IMG:
make deploy IMG=<some-registry>/profile-pod-operator:tag

Uninstall CRDs

To delete the CRDs from the cluster:

make uninstall

Undeploy controller

UnDeploy the controller from the cluster:

make undeploy

Test It Out

  1. Install the CRDs into the cluster:
make install
  1. Run your controller (this will run in the foreground, so switch to a new terminal if you want to leave it running):
make run

NOTE: You can also run this in one step by running: make install run

Modifying the API definitions

If you are editing the API definitions, generate the manifests such as CRs or CRDs using:

make manifests

NOTE: Run make --help for more information on all potential make targets

More information can be found via the Kubebuilder Documentation

License

Copyright 2023.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published