Sandbox followup for Pixie and YARP

[caniszczyk]

The @cncf/toc had some questions regarding Pixie which is being contributed to sandbox: https://github.com/pixie-labs/pixie along with YARP: https://github.com/microsoft/reverse-proxy

For Pixie, there a supportive discussion but a question around whether the whole github organization was being contributed (not just the client) so there needs to be some clarity from the project maintainers for clarity, also the presentation included in the proposal wasn't available for viewing: https://docs.google.com/presentation/d/1s75vY7NKPr-r3QMggh2WUJ5EnCyIioJe7ToPEuHp9uo/edit?usp=sharing - Also there was some discussion whether CNCF SIG Observability had opinions on the project.

For YARP, there was questions of where the project was used in the .NET ecosystem and how it may integrate with other projects and how it may help spread cloud native practices in the .NET ecosystem. Also, the project is currently under the MIT license and need to confirm they are open to re-licensing to Apache v2.0 if accepted. There was a desire to have CNCF SIG Network weigh in on this project too.

[zasgar]

Pixie maintainer here. Thanks, @caniszczyk! Happy to help with the process. Here are some clarifications:

We have open-sourced the entire Pixie platform and would like to contribute all of it to CNCF. This includes the following repos:

Pixie https://github.com/pixie-labs/pixie
This is the primary Pixie repository containing an entire self-hosted version of Pixie. It is fully open-source (Apache 2.0) and has no dependence on external services or vendors. This repo contains Pixie’s CLI (src/pixie_cli), UI (src/ui), cloud (src/cloud), and the monitoring backend (src/vizier).

Pixie API https://github.com/pixie-labs/pxapi.go
This repository contains the API for querying Pixie. It includes our Golang and GRPC client APIs.

Pixie Docs https://github.com/pixie-labs/pixie-docs
This repository contains all of our documentation, including tutorials and reference docs.

In addition, we will be contributing the website, domain names (px.dev, pixie.io), and the open-source logos. We are also planning to publish Prometheus and Grafana connectors, which will also be donated to CNCF.

With this, the community will get access to a fully self-hosted version of Pixie, with no dependence on external/vendor services. In addition, they will have access to all of our deployment/CI scripts, documentation, and website.

This project is managed by the governance board of Pixie. The overall project presentation is located here (link fixed).

Happy to answer any questions and excited to be part of the CNCF community.

[caniszczyk]

Thanks @zasgar, that's more clear cc: @cncf/toc

[lizrice]

@zasgar for Pixie, is this product page signup step necessary?

[caniszczyk]

As an FYI @zasgar, CNCF projects follow website guidelines that are usually taken care of before the project enters the foundation, fairly straight forward changes: https://github.com/cncf/foundation/blob/master/website-guidelines.md

[zasgar]

@lizrice, it's not necessary. Pixie can be used completely self-hosted. We are still updating all our assets after open-sourcing pixie. We'll take care of making these updates today.

[samsp-msft]

Hi, I am the PM for the YARP project at Microsoft. I will try and answer some of your questions inline:

For YARP, there was questions of where the project was used in the .NET ecosystem

YARP is a project being run by two of the sub teams within .NET - ASP.NET for the front end, and the Networking team for http client, TLS, sockets etc. We have a number of teams who had built their own Http/1.1 proxies, because the off the shelf solutions didn't handle their needs, but with APIs transitioning to gRPC they needed solutions that support HTTP/2. So they wanted our help in how to use the existing components from .NET to make a proxy. After we got asked more than a couple of times, we realized that there was a hole in our ecosystem for an adaptable proxy.

The customization ability we have built into YARP is because of the needs of these internal customers, both in terms of where and how to pull route configuration from, and the ability to wholesale replace parts of the proxy pipeline to fit in with their infrastructure such health checking.

Externally we have a couple of smaller deployments of YARP. We are finding that the customizability aspect is resonating well with the people who are using it. It turns out that most app deployments are slightly off the beaten path and have some oddities that need to be accounted for, and that a reverse proxy is great way to paper over some of those.

We recently had a consultant working with their (non .NET!) Azure customer on how to add a waiting room concept to their website that is currently under a lot of load. They had tried extending for a week, before they found YARP and then had a proof of concept working in a couple of hours. We only found out about it when they wanted a code review for their extension code.

YARP is not a product Microsoft intends to sell. Its using the building blocks from .NET to solve problems faced by internal partners, that we realized that others can also make use of. For .NET, the teams working on YARP own the underlying libraries YARP is built on, and building YARP is a great use case to drive improvements into those sub components.

and how it may integrate with other projects and how it may help spread cloud native practices in the .NET ecosystem.

Since we submitted the original request to join the sandbox, we have made progress in bridging the technologies:

• We have a prototype k8s ingress config provider for YARP so that you can use YARP as an ingress controller and it will pull all the config from k8s, but you still get the benefits of being able to customize it and the other proxy handling of YARP.
• A side effect of that is that the YARP team became custodians for an operator framework library for .NET/c#. It had been developed by another team, who had shelved it when they had switched directions. We want to find the most appropriate home for it so that it can be used by other .NET apps/libraries wanting to interoperate with k8s.
• We have a prometheus monitoring sample for YARP showing how to use the prometheus-net library to export an endpoint from YARP for prometheus to collect metrics.

The reason that we are looking at the CNCF rather than the .NET foundation is because we think YARP would be a good starting point for bringing more cross pollination between .NET and the CNCF ecosystems. We ship .NET for Windows, Linux and Mac, and see a lot of interest and growth both from internal services and public customers using .NET in (linux) containers.

Customers deploying cloud scale .NET apps have the same infrastructure needs as those working in Java, Go, Rust etc. They are not just using Azure, but being deployed on-prem, at AWS and other cloud providers. Typically the larger customers with .NET apps also have Java, Node etc and want to use the same management, monitoring, scaling systems etc between them.

AFAIK this would be the first .NET component in the CNCF, and hopefully not the last.

Also, the project is currently under the MIT license and need to confirm they are open to re-licensing to Apache v2.0 if accepted.

Yes - we acknowledged that as part of the submission form. We are currently under the Microsoft github org, and plan to move to an independent org (yarpproxy) if accepted, and would change the license at the same time. Its easier to do that when outside of the Microsoft github org.

There was a desire to have CNCF SIG Network weigh in on this project too.

We are open and happy to talk with whoever wants to know more about the project. You can reach me either via github, or the part of my alias before the - @microsoft.com.

[halcyondude]

In the past 2 weeks @RichiH and I had an in-depth overview and Q&A with @zasgar on Pixie which was informative and positive. Today the TAG welcomed Pixie to give a demo which was well received.

Thank you @samsp-msft for articulating some of the context and current work in flight. I have a few questions and observations.

Points to consider:

• The .Net developer community is massive, and .Net Core has been consistently gaining in popularity and dev mindshare for the past few years. It's built for cloud environments and leverages an ecosystem of developer-focused tooling that is rich in its feature set and history.
• .Net 5, released in 2020 has solidified the open sourced .Net Core as the clear path forward for the ecosystem.

We have a prototype k8s ingress config provider for YARP so that you can use YARP as an ingress controller and it will pull all the config from k8s, but you still get the benefits of being able to customize it and the other proxy handling of YARP.

TAG Network should absolutely provide feedback. Ingress and its future is an interesting topic (cncf blog) and I look forward to following the progress of an yarp ingress.

A side effect of that is that the YARP team became custodians for an operator framework library for .NET/c#. It had been developed by another team, who had shelved it when they had switched directions. We want to find the most appropriate home for it so that it can be used by other .NET apps/libraries wanting to interoperate with k8s.

Please do provide a link, I would be interested in learning more. TAG App Delivery could be consulted here, there's work in flight on an Operator Whitepaper that appears to be wrapping up.

We have a prometheus monitoring sample for YARP showing how to use the prometheus-net library to export an endpoint from YARP for prometheus to collect metrics.

The Prometheus sample for YARP is a fantastic thing to see. We've recommended following up with the Prometheus team, and exploring having it listed as an exporter (https://prometheus.io/docs/instrumenting/exporters). @RichiH is a domain expert and has provided additional resources and materials directly.

I too would heartily support expanding the CNCF's portfolio of projects to include .Net. Microsoft has for many many years embraced cloud native approaches to software well before the inception of the CNCF. Here's their take:

https://docs.microsoft.com/en-us/dotnet/architecture/cloud-native/definition

And a look at how they are actively pushing their ecosystem of existing applications --> Cloud Native approaches:

https://docs.microsoft.com/en-us/dotnet/architecture/cloud-native/candidate-apps

I think both communities would be well served by cross-pollenating, and YARP is somehow in my 🧠 perfect for a first project. It's thin, light, and purpose-built. That it's so very customizable makes it ironically fitting, as it will have good company in the broad assortment of Ingress resources which provide rich customization and configuration.

It turns out that most app deployments are slightly off the beaten path and have some oddities that need to be accounted for, and that a reverse proxy is great way to paper over some of those.

On a more serious note, we should not underestimate the importance of pragmatism when wrangling real applications and services. YARP could facilitate the migration of existing applications that would otherwise require the CNCF's newest members (C# developers learning k8s/cncf ecosystem in this case) to sort out issues needing to be papered over with unfamiliar mechanisms and tools.

TAG Observability looks forward to engaging with YARP and Pixie should they be admitted to Sandbox, as I hope they are.

[samsp-msft]

Thanks @halcyondude. The operator framework I was talking about can be found at https://github.com/microsoft/reverse-proxy/tree/main/src/OperatorFramework

[halcyondude]

I see, so is reverse-proxy is the base/root home for OperatorFramework, or is this a snapshot to freeze a dependency?

[Tratcher]

@halcyondude it's a temporary home for OperatorFramework while we find it a better one.

[leecalcote]

Point of clarification: Is the OperatorFramework included in the YARP SandBox proposal?

[leecalcote]

@halcyondude, YARP leverages the existing https://github.com/prometheus-net/prometheus-net library for instruct mention of its metrics. @samsp-msft, will you confirm whether this is the extent of the integration with Prometheus? I assume there are no plans to create a YARP-specific exporter, is that correct?

[Tratcher]

Point of clarification: Is the OperatorFramework included in the YARP SandBox proposal?

OperatorFramework should be considered an internal/private dependency until we find a proper home for it in another project. We don't plan to publish that as a consumable package right now.

[samsp-msft]

YARP has a series of counters that you can listen to. The prometheus sample extends the proxy using prometheus-net to create an endpoint with the response format that prometheus needs. It then exports 25 counters for different aspects of the proxy, for example

        private static readonly Histogram _outboundHttp20RequestQueueDuration = Metrics.CreateHistogram(
            "yarp_outbound_http20_request_queue_duration",
            "Average time spent on queue for HTTP 2.0 requests that hit the MAX_CONCURRENT_STREAMS limit on the connection in the last metrics interval",
            new HistogramConfiguration
            {
                Buckets = Histogram.ExponentialBuckets(10, CUBE_ROOT_10, 10)
            });

This means you can change the way the counters are exposed, such as changing the names, units, histogram buckets etc.

One of the design points for YARP that is different from other proxies is that you get to customize the proxy with the functionality you need for your deployment. Where you need to integrate with other systems, it may be built into the YARP library directly, but we typically have the extension points, the .NET ecosystem has libraries to work with external systems, and as the app author you can tie those together, usually with very little code. This gives a lot of flexibility without YARP having to have solutions built-in for every conceivable scenario.

[samsp-msft]

@leecalcote a thing I realized after our conversation last week is that I think we have a unique perspective from most people building proxies. The Microsoft people working on YARP are all from developer platform teams/backgrounds, and the internal customers we have are each building their own solutions as a combination of YARP plus service specific functionality. Each has service specific requirements that are of the beaten path of capabilities of most proxies. Offering YARP as a toolkit consisting of a library and samples makes it easy for customers to create a proxy that suits their explicit needs, without having to fork and modify the innards which would be needed with other proxies.

[caniszczyk]

closing after the TOC meeting today: https://lists.cncf.io/g/cncf-toc/message/5915