From 2511ec0f3c7a9139127a9420dd2acc581d1c8760 Mon Sep 17 00:00:00 2001 From: Michelle Nguyen Date: Fri, 8 Apr 2022 15:40:48 -0700 Subject: [PATCH 1/6] add new arch overview page Signed-off-by: Michelle Nguyen --- .../03-architecture/01-overview.md | 64 +++++++++++++++++++ .../02-pixie-ebpf.md} | 4 +- .../01-about-pixie/03-architecture/index.md | 11 ++++ content/en/01-about-pixie/index.md | 2 +- 4 files changed, 78 insertions(+), 3 deletions(-) create mode 100644 content/en/01-about-pixie/03-architecture/01-overview.md rename content/en/01-about-pixie/{03-pixie-ebpf.md => 03-architecture/02-pixie-ebpf.md} (98%) create mode 100644 content/en/01-about-pixie/03-architecture/index.md diff --git a/content/en/01-about-pixie/03-architecture/01-overview.md b/content/en/01-about-pixie/03-architecture/01-overview.md new file mode 100644 index 00000000..b07c2b07 --- /dev/null +++ b/content/en/01-about-pixie/03-architecture/01-overview.md @@ -0,0 +1,64 @@ +--- +title: "Overview" +metaTitle: "About Pixie | Architecture | Overview" +metaDescription: "A detailed overview of Pixie's components" +order: 1 +--- + +## Overview + +Pixie consists of three main components: + +- **Vizier**: Pixie's data plane for data collection and processing. An instance of Vizier is deployed to each cluster which needs to be monitored. +- **Cloud**: Pixie's control plane for serving Pixie's API and UI, and managing and tracking metadata (e.g. orgs, users, Viziers). +- **Vizier Operator**: Pixie's [Kubernetes operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) which is used to help deploy and manage Vizier on a cluster, such as keeping all components running and up-to-date. + +## Vizier + +Vizier is Pixie's data plane. It is responsible for collecting and processing data within the cluster that is being monitored. + +### Components + +1. **Pixie Edge Module (PEM)**: The data collectors, deployed to each node via a daemonset. PEMs serve as a short-term in-memory store (up to 24h) for all data collected in the cluster. The PEMs are also responsible for processing this data when they receive a script execution request. +2. **Collector (Kelvin)**: The Kelvin is the higher-level data collector, where PEMs send data to for further aggregation (e.g. joins, unions) and processing. +3. **Cloud Connector**: The cloud connector communicates with Pixie Cloud and passes messages between the two systems. These messages range from heartbeats, used to indicate the Vizier's overall status, and script execution requests and results. +4. **Metadata**: The metadata service serves as the hub for all of Vizier's metadata. This includes K8s metadata, used to contextualize any collected data, and the status of each PEM running in the cluster. +5. **Query Broker**: The query broker handles all script execution. It compiles any scripts need to be executed, distributes those scripts to the relevant PEMs, and helps proxy those results back to the cloud. + +### Dependencies + +Vizier relies on a couple of dependencies that are critical to its operation. + +1. **[NATS](https://nats.io/)**: Pixie's message bus system, used to send messages between pods and services within Vizier. For example, PEMs send routine heartbeat messages to the metadata pod to inform it of its status. +2. **[etcd](https://etcd.io/)**: Only used when the cluster does not support persistent volumes, or when explicitly chosen. etcd serves as the Metadata pod's in-memory datastore. + +## Cloud + +Pixie's cloud components serves as the control plane. The cloud serves Pixie's API and UI to make it easy to query and access data from Viziers. It also manages metadata in the system, such as orgs, users, and Viziers. + +### Components + +1. **Cloud Proxy**: An NGINX and Envoy proxy which serves as the main entrypoint when requests are made to the Cloud. The proxy also hosts the UI. +2. **API**: The API service serves as a proxy for any incoming API requests (GQL, GRPC, Python, Go), and redirects the request to the relevant downstream services in the Cloud. +3. **Auth**: The Auth service is responsible for handling authentication/authorization for ensuring that any tokens/cookies/API keys are valid before servicing a request. +4. **Vizier Connector (VZConn)**: VZConn serves as the communication point between Cloud and all connected Viziers. It proxies data between the Vizier and the relevant downstream services in the Cloud. +5. **Vizier Manager (VZMgr)**: VZMgr tracks and manages the state of all connected Viziers, such as the version or current health of a Vizier. +6. **Artifact Tracker**: The artifact tracker manages version and release information for all Pixie-related artifacts, such as the Vizier and CLI. For example, it can be queried to obtain the necessary download links for a Vizier's base deployment YAMLs. +7. **Config Manager**: The config manager is responsible for customizing a Vizier's deployment YAMLs based on an user's/org's specified deployment options. +8. **Cron Script**: The cron script service handles registering and distributing any PxL scripts that should run on a scheduled cadence on a Vizier. For example, scripts can be scheduled to run data exports to an external datastore at regular intervals. +9. **Indexer**: The indexer receives any metadata that should be stored and indexed for future querying (for example, to perform autocompletes when searching for K8s namespaces within a cluster). +10. **Plugin**: The plugin service manages all plugin information in the Pixie Plugin System, such as which plugins are available and enabled. +11. **Profile**: The profile service manages all users and orgs registered in Pixie Cloud. + +### Dependencies + +1. **[Elastic](https://www.elastic.co/)**: Elastic serves as Pixie Cloud's indexing engine, for any data that needs to be easily searchable. +2. **[NATS/STAN](https://nats.io/)**: The Cloud's message bus system for distributing messages between services in Pixie Clouds. STAN is addtionally used for any messages that may need persistence. + +## Vizier Operator + +The Vizier Operator is used to help manage Vizier on a cluster. The operator executes the initial deploy of Vizier, deciding which Vizier configurations are best fit for an environment (if not already specified), and monitors the overall state of the Vizier. This includes restarting any failed dependencies and keeping Vizier up-to-date, based on the latest release information from Pixie Cloud. + +### Dependencies + +1. **[Operator Lifecycle Manager](https://olm.operatorframework.io/)**: OLM is responsible for deploying the Vizier Operator to a cluster and keeping the Vizier Operator up-to-date. diff --git a/content/en/01-about-pixie/03-pixie-ebpf.md b/content/en/01-about-pixie/03-architecture/02-pixie-ebpf.md similarity index 98% rename from content/en/01-about-pixie/03-pixie-ebpf.md rename to content/en/01-about-pixie/03-architecture/02-pixie-ebpf.md index 68c3e1c0..bcd04a7f 100644 --- a/content/en/01-about-pixie/03-pixie-ebpf.md +++ b/content/en/01-about-pixie/03-architecture/02-pixie-ebpf.md @@ -1,8 +1,8 @@ --- title: "How Pixie uses eBPF" -metaTitle: "About Pixie | How Pixie uses eBPF" +metaTitle: "About Pixie | Architecture | How Pixie uses eBPF" metaDescription: "How Pixie uses eBPF to automatically collect data" -order: 3 +order: 2 --- Pixie uses [eBPF](https://www.brendangregg.com/ebpf.html) to drive much of its data collection. diff --git a/content/en/01-about-pixie/03-architecture/index.md b/content/en/01-about-pixie/03-architecture/index.md new file mode 100644 index 00000000..ae0a0413 --- /dev/null +++ b/content/en/01-about-pixie/03-architecture/index.md @@ -0,0 +1,11 @@ +--- +title: "Architecture" +metaTitle: "Architecture" +metaDescription: "Guides which dive in Pixie's architecture and how it works" +order: 2 +--- + +This section contains guides detailing Pixie's architecture and how it works. + +- [Overview](/about-pixie/architecture/overview) +- [How Pixie uses eBPF](/about-pixie/architecture/pixie-ebpf) diff --git a/content/en/01-about-pixie/index.md b/content/en/01-about-pixie/index.md index ece17e34..2376171c 100644 --- a/content/en/01-about-pixie/index.md +++ b/content/en/01-about-pixie/index.md @@ -11,6 +11,6 @@ This section gives an overview of Pixie: - [What is Pixie?](/about-pixie/what-is-pixie) - [What data sources does Pixie capture?](/about-pixie/data-sources) -- [How Pixie uses eBPF](/about-pixie/pixie-ebpf) +- [Architecture](/about-pixie/architecture) - [Pixie's roadmap](/about-pixie/roadmap) - [FAQs](/about-pixie/faq) From c6c5cf787a2becf123c555cd22041a4e6d5f1ff4 Mon Sep 17 00:00:00 2001 From: Michelle Nguyen Date: Fri, 8 Apr 2022 15:50:18 -0700 Subject: [PATCH 2/6] add troubleshooting section Signed-off-by: Michelle Nguyen --- content/en/01-about-pixie/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/en/01-about-pixie/index.md b/content/en/01-about-pixie/index.md index 2376171c..9b7e529b 100644 --- a/content/en/01-about-pixie/index.md +++ b/content/en/01-about-pixie/index.md @@ -14,3 +14,4 @@ This section gives an overview of Pixie: - [Architecture](/about-pixie/architecture) - [Pixie's roadmap](/about-pixie/roadmap) - [FAQs](/about-pixie/faq) +- [Troubleshooting](/about-pixie/troubleshooting) From c707c3475e8999f1e8f7d2e62a4b3f926622a036 Mon Sep 17 00:00:00 2001 From: Michelle Nguyen Date: Fri, 8 Apr 2022 16:37:00 -0700 Subject: [PATCH 3/6] move architecture to references Signed-off-by: Michelle Nguyen --- content/en/01-about-pixie/03-architecture/index.md | 11 ----------- .../02-pixie-ebpf.md => 03-pixie-ebpf.md} | 0 content/en/01-about-pixie/06-troubleshooting.md | 2 +- content/en/01-about-pixie/index.md | 4 ++-- .../02-architecture.md} | 6 +++--- .../en/05-reference/{02-api => 03-api}/01-overview.md | 0 content/en/05-reference/{02-api => 03-api}/02-go.md | 0 content/en/05-reference/{02-api => 03-api}/index.md | 0 .../02-grafana.md => 04-plugins/01-grafana.md} | 0 .../05-reference/{03-plugins => 04-plugins}/index.md | 0 content/en/05-reference/index.md | 3 +++ 11 files changed, 9 insertions(+), 17 deletions(-) delete mode 100644 content/en/01-about-pixie/03-architecture/index.md rename content/en/01-about-pixie/{03-architecture/02-pixie-ebpf.md => 03-pixie-ebpf.md} (100%) rename content/en/{01-about-pixie/03-architecture/01-overview.md => 05-reference/02-architecture.md} (98%) rename content/en/05-reference/{02-api => 03-api}/01-overview.md (100%) rename content/en/05-reference/{02-api => 03-api}/02-go.md (100%) rename content/en/05-reference/{02-api => 03-api}/index.md (100%) rename content/en/05-reference/{03-plugins/02-grafana.md => 04-plugins/01-grafana.md} (100%) rename content/en/05-reference/{03-plugins => 04-plugins}/index.md (100%) diff --git a/content/en/01-about-pixie/03-architecture/index.md b/content/en/01-about-pixie/03-architecture/index.md deleted file mode 100644 index ae0a0413..00000000 --- a/content/en/01-about-pixie/03-architecture/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -title: "Architecture" -metaTitle: "Architecture" -metaDescription: "Guides which dive in Pixie's architecture and how it works" -order: 2 ---- - -This section contains guides detailing Pixie's architecture and how it works. - -- [Overview](/about-pixie/architecture/overview) -- [How Pixie uses eBPF](/about-pixie/architecture/pixie-ebpf) diff --git a/content/en/01-about-pixie/03-architecture/02-pixie-ebpf.md b/content/en/01-about-pixie/03-pixie-ebpf.md similarity index 100% rename from content/en/01-about-pixie/03-architecture/02-pixie-ebpf.md rename to content/en/01-about-pixie/03-pixie-ebpf.md diff --git a/content/en/01-about-pixie/06-troubleshooting.md b/content/en/01-about-pixie/06-troubleshooting.md index 288d9b97..6f936d4b 100644 --- a/content/en/01-about-pixie/06-troubleshooting.md +++ b/content/en/01-about-pixie/06-troubleshooting.md @@ -5,7 +5,7 @@ metaDescription: "Troubleshooting problems with Pixie." order: 6 --- -This page describes how to troubleshoot Pixie. We frequently answer questions on our [community Slack](https://slackin.px.dev/) channel and in response to [GitHub issues](https://github.com/pixie-io/pixie/issues). You can also check those two places to see if your question has already been addressed. To better understand how Pixie's various components interact, please see the [Architecture](/about-pixie/architecture/overview) overview. +This page describes how to troubleshoot Pixie. We frequently answer questions on our [community Slack](https://slackin.px.dev/) channel and in response to [GitHub issues](https://github.com/pixie-io/pixie/issues). You can also check those two places to see if your question has already been addressed. To better understand how Pixie's various components interact, please see the [Architecture](/reference/architecture) overview. #### Troubleshooting Deployment diff --git a/content/en/01-about-pixie/index.md b/content/en/01-about-pixie/index.md index 9b7e529b..eac0581d 100644 --- a/content/en/01-about-pixie/index.md +++ b/content/en/01-about-pixie/index.md @@ -11,7 +11,7 @@ This section gives an overview of Pixie: - [What is Pixie?](/about-pixie/what-is-pixie) - [What data sources does Pixie capture?](/about-pixie/data-sources) -- [Architecture](/about-pixie/architecture) -- [Pixie's roadmap](/about-pixie/roadmap) +- [How does Pixie use eBPF?](/about-pixie/pixie-ebpf) +- [What is on Pixie's roadmap?](/about-pixie/roadmap) - [FAQs](/about-pixie/faq) - [Troubleshooting](/about-pixie/troubleshooting) diff --git a/content/en/01-about-pixie/03-architecture/01-overview.md b/content/en/05-reference/02-architecture.md similarity index 98% rename from content/en/01-about-pixie/03-architecture/01-overview.md rename to content/en/05-reference/02-architecture.md index b07c2b07..d8dbfcb6 100644 --- a/content/en/01-about-pixie/03-architecture/01-overview.md +++ b/content/en/05-reference/02-architecture.md @@ -1,8 +1,8 @@ --- -title: "Overview" -metaTitle: "About Pixie | Architecture | Overview" +title: "Architecture" +metaTitle: "Reference | Architecture" metaDescription: "A detailed overview of Pixie's components" -order: 1 +order: 2 --- ## Overview diff --git a/content/en/05-reference/02-api/01-overview.md b/content/en/05-reference/03-api/01-overview.md similarity index 100% rename from content/en/05-reference/02-api/01-overview.md rename to content/en/05-reference/03-api/01-overview.md diff --git a/content/en/05-reference/02-api/02-go.md b/content/en/05-reference/03-api/02-go.md similarity index 100% rename from content/en/05-reference/02-api/02-go.md rename to content/en/05-reference/03-api/02-go.md diff --git a/content/en/05-reference/02-api/index.md b/content/en/05-reference/03-api/index.md similarity index 100% rename from content/en/05-reference/02-api/index.md rename to content/en/05-reference/03-api/index.md diff --git a/content/en/05-reference/03-plugins/02-grafana.md b/content/en/05-reference/04-plugins/01-grafana.md similarity index 100% rename from content/en/05-reference/03-plugins/02-grafana.md rename to content/en/05-reference/04-plugins/01-grafana.md diff --git a/content/en/05-reference/03-plugins/index.md b/content/en/05-reference/04-plugins/index.md similarity index 100% rename from content/en/05-reference/03-plugins/index.md rename to content/en/05-reference/04-plugins/index.md diff --git a/content/en/05-reference/index.md b/content/en/05-reference/index.md index 346ab79b..a2e5bd51 100644 --- a/content/en/05-reference/index.md +++ b/content/en/05-reference/index.md @@ -20,6 +20,9 @@ order: 50 - [User Management & Sharing](/reference/admin/user-mgmt/) - [Version History](/reference/admin/product-updates) +#### Architecture +- [Architecture](/reference/architecture) + #### API - [API Overview](/reference/api/overview) From 79512dfc305d3c8d06b135ec0e73c6749deb3c64 Mon Sep 17 00:00:00 2001 From: Michelle Nguyen Date: Mon, 11 Apr 2022 10:04:04 -0700 Subject: [PATCH 4/6] comment Signed-off-by: Michelle Nguyen --- content/en/01-about-pixie/03-pixie-ebpf.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/01-about-pixie/03-pixie-ebpf.md b/content/en/01-about-pixie/03-pixie-ebpf.md index bcd04a7f..6101c96a 100644 --- a/content/en/01-about-pixie/03-pixie-ebpf.md +++ b/content/en/01-about-pixie/03-pixie-ebpf.md @@ -1,8 +1,8 @@ --- title: "How Pixie uses eBPF" -metaTitle: "About Pixie | Architecture | How Pixie uses eBPF" +metaTitle: "About Pixie | How Pixie uses eBPF" metaDescription: "How Pixie uses eBPF to automatically collect data" -order: 2 +order: 3 --- Pixie uses [eBPF](https://www.brendangregg.com/ebpf.html) to drive much of its data collection. From 6377c96649868a926f31684c77529310e7360af9 Mon Sep 17 00:00:00 2001 From: Michelle Nguyen Date: Mon, 11 Apr 2022 10:52:31 -0700 Subject: [PATCH 5/6] sign all commits Signed-off-by: Michelle Nguyen --- content/en/01-about-pixie/03-pixie-ebpf.md | 2 +- content/en/05-reference/02-architecture.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/en/01-about-pixie/03-pixie-ebpf.md b/content/en/01-about-pixie/03-pixie-ebpf.md index 6101c96a..68c3e1c0 100644 --- a/content/en/01-about-pixie/03-pixie-ebpf.md +++ b/content/en/01-about-pixie/03-pixie-ebpf.md @@ -2,7 +2,7 @@ title: "How Pixie uses eBPF" metaTitle: "About Pixie | How Pixie uses eBPF" metaDescription: "How Pixie uses eBPF to automatically collect data" -order: 3 +order: 3 --- Pixie uses [eBPF](https://www.brendangregg.com/ebpf.html) to drive much of its data collection. diff --git a/content/en/05-reference/02-architecture.md b/content/en/05-reference/02-architecture.md index d8dbfcb6..ae818c03 100644 --- a/content/en/05-reference/02-architecture.md +++ b/content/en/05-reference/02-architecture.md @@ -20,9 +20,9 @@ Vizier is Pixie's data plane. It is responsible for collecting and processing da ### Components 1. **Pixie Edge Module (PEM)**: The data collectors, deployed to each node via a daemonset. PEMs serve as a short-term in-memory store (up to 24h) for all data collected in the cluster. The PEMs are also responsible for processing this data when they receive a script execution request. -2. **Collector (Kelvin)**: The Kelvin is the higher-level data collector, where PEMs send data to for further aggregation (e.g. joins, unions) and processing. -3. **Cloud Connector**: The cloud connector communicates with Pixie Cloud and passes messages between the two systems. These messages range from heartbeats, used to indicate the Vizier's overall status, and script execution requests and results. -4. **Metadata**: The metadata service serves as the hub for all of Vizier's metadata. This includes K8s metadata, used to contextualize any collected data, and the status of each PEM running in the cluster. +2. **Collector (Kelvin)**: Kelvin is the high-level data collector. During script execution, PEMs send partially processed data to Kelvins. Kelvins aggregate the node level view from each PEM into a cluster-level view and complete the script execution steps that requires all of the data, such as aggregates and joins. +3. **Cloud Connector**: The cloud connector handles message passing between Vizier and Pixie Cloud. These messages range from heartbeats, used to indicate the Vizier's overall status, and script execution requests and results. +4. **Metadata**: The metadata service serves as the hub for all of Vizier's metadata. The metadata stored includes k8s objects (ie pods, services, namespaces) for adding context to execute scripts as well as metadata about the Vizier state (ie PEM status, [dynamic logs deployed](https://docs.px.dev/tutorials/custom-data/dynamic-go-logging/)) 5. **Query Broker**: The query broker handles all script execution. It compiles any scripts need to be executed, distributes those scripts to the relevant PEMs, and helps proxy those results back to the cloud. ### Dependencies From 01cb6826dd3ccae10e16138acdec0dd3629e316f Mon Sep 17 00:00:00 2001 From: Michelle Nguyen Date: Tue, 19 Apr 2022 13:12:52 -0700 Subject: [PATCH 6/6] fix merge conflict Signed-off-by: Michelle Nguyen --- .../05-reference/{03-plugins => 04-plugins}/01-plugin-system.md | 0 .../en/05-reference/04-plugins/{01-grafana.md => 02-grafana.md} | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename content/en/05-reference/{03-plugins => 04-plugins}/01-plugin-system.md (100%) rename content/en/05-reference/04-plugins/{01-grafana.md => 02-grafana.md} (100%) diff --git a/content/en/05-reference/03-plugins/01-plugin-system.md b/content/en/05-reference/04-plugins/01-plugin-system.md similarity index 100% rename from content/en/05-reference/03-plugins/01-plugin-system.md rename to content/en/05-reference/04-plugins/01-plugin-system.md diff --git a/content/en/05-reference/04-plugins/01-grafana.md b/content/en/05-reference/04-plugins/02-grafana.md similarity index 100% rename from content/en/05-reference/04-plugins/01-grafana.md rename to content/en/05-reference/04-plugins/02-grafana.md