Health of Linkerd project

[dims]

The TOC has determined the recently announced changes to Linkerd releases warrants discussion with the project concerning their health and this recent change. This issue is created to track the discussion and engagement with project and evaluate their current alignment with the expectations of a graduated project.

[monadic]

People and companies work on projects for economic and intrinsic motivations.

Eg

Economics of CNCF needs to justified to vendors.

Intrinsic motivations include belief that the project will flourish and be sustainable, popular, cool.

End users often are a mix of both. They will need support over time (in 3-10 year horizon) and in short term want to mitigate new tech risk

CNCF needs to take this a lot more seriously or it will be the home of poorly staffed second tier projects.

TOC+GB+EUC needs to engage at the level of product, marketing and long term business support, or find people who can.

[dims]

@shinebayar-g @monadic thanks for the comments, but let's please keep this issue focused for now. feel free to use other channels of communication (email thread, another issue, slack ..) for the larger debate.

[monadic]

admirable to address separately if possible.

unlikely to succeed because issues are tangled. linkerd governance seems to have fallen into disuse and obviously needs explaining & renewal. but real issue is why linkerd main developers felt obliged to adjust to an inferior OSS offering, which as far as I can see they are free to do, if they have to support it.

[kzap]

At our company, we have been using Linkerd in production for 4+ years. We have been thoroughly impressed by the level of maintenance on this project and work put into the project despite us paying $0 to the vendor Buoyant. It's quite understandable how this decision could come to fruition given the economic challenges that everyone in the industry including us are still going through. Things could have been far worse.

---

I have some suggestions for the CNCF:

From https://www.cncf.io/projects/

Graduated and incubating projects are considered stable and are used successfully in production environments.

Perhaps there can be some modifications to the Graduated project qualifications to encourage seeking more diversity in maintainers for a project to be listed as graduated.

The requirement of Committers from 2 organizations seems like a low bar given the reach and usage of Graduated projects. Making this atleast maintainers from 2 organizations (or having atleast 33% of maintainers from a different company) could be more helpful in using the Graduated status to incentivize having competing concerns at the maintainer level and lead to a healthier project. The CNCF and its community has grown so much since Linkerd graduated that it would only be natural to raise the bar.

If we look at Cilium maintainers we see a similar pattern of majority 1 vendor being the maintainers and thus could have commercial pressures to have those maintainers perform similar decisions to a graduated CNCF project like Cilium.

Also hope there could be some expectation of standards for releases and the naming of these versions. The edge vs stable is certainly confusing for end users of a graduated project.

This project has been well maintained by 1 vendor, perhaps too well. It is obvious that they have been wearing both open source and vendor hats at the same time so it is hard to discern if the maintainers are speaking with their open source hats or their vendor hats.

---

I am glad that this project is stewarded by the CNCF and all the trademarks and IP is owned by them. I am sure a positive outcome for the entire eco-system will come out of this, because thats how open source works and its why we put the time and effort into it whether we are end users or maintainers.

[lizrice]

There’s some healthy conversation to be had here about requirements and expectations, but I’m not sure it’s helpful to imply that Cilium is on the same path as Linkerd. I want to set the record straight before these misapprehensions take root! Yes, the majority of Cilium committers (the term Cilium uses for “maintainers” i.e. people with the commit bit) are from Isovalent, but we also have

• committers from 6 different companies plus another two non-affiliated committers
• a voting cap so that no company has a majority
• most importantly, an incredibly healthy flow of contributions from hundreds of contributors across dozens of organizations
• absolutely no plan to stop doing OSS stable builds!

It’s only a few months since the TOC gave Cilium’s governance a clean bill of health for graduation. Which isn’t to say that it’s perfect, and we’re actively working on improvements. At the moment we have 21% of GitHub contributions coming from non-Isovalent people, and we’d love to increase that number. I’ve had conversations in the last few weeks with folks from two additional companies, telling me they are assigning folks to working on Cilium full-time, in the hope of building up the expertise and trust that they can in time become committers too - this is great!

The reality is that projects need huge investments of time and resources, and that investment comes almost entirely from companies. Many foundation projects are majority supported by one company that has a commercial distribution, and is existentially bound to the success of the project. As a foundation community, we largely close our eyes to the financial health of those businesses, although it’s an incredibly important indicator of the investment into and therefore health of the underlying projects.

[steve-gray]

I'd like to raise five points, and I'm not really expecting answers to any of them:

• The change to what branches are made or binaries produced breaks no specific rule that anyone is able to articulate.

  • Does this mean there ought to be some? Maybe, but right here and now the rules aren't broken.
  • People don't like it, and you can argue semantics to try and find a violation-ish thing.

• The metrics being chosen here are poor, and people are selecting them to suit the discussion they want to have.

  • Cillium might have more numerical committers from other companies, but at the same time it's all about the measurement axis you want to use.
  • Being specifc, 42 out of 49 listed committers in MAINTAINERS.md are Isovalent, but on a pure lines of code and commits basis, its very much clear that the ratio is far more skewed than this selected metric indicates, with the ratio actually bordering on an effective totality.
  • Looking at commits by volume and originator, it is evident the withdrawl of Isovalent support would essentially have an impact as great as Buoyant's contribution to Linkerd, on a proportional basis.
  • The recent changes in ownership do put a valid question over the continuity of support for the open development of a product. A change in policy by Cisco would have a dramatic, likely cataclysmic impact on forward velocity.

• Members of the CNCF do not appear to be approaching this with clean hands, at least from an optics perspective.

  • Are there policies that indicate if you have a employment or other commercial interest that is geared towards a competing project, you recuse from deliberations involving those projects?
  • I would like to single out for example, Craig Box, as someone who is unbashedly partisan, with what appears to be a sustained grudge against the Linkerd project and its key people. But @lizrice too, obviously as an outsider to this process - on what basis can someone like myself feel your take is fair, genuine and not otherwise influenced by these relations?
  • It would stand to reason that recusal of people with such interests would still leave sufficient people for any deliberations on issues, given the deep bench the CNCF has to play with in terms of people?

• The reality is that the more deeply technical a project is, it lends itself to contributor concentration, as there generally is only ever one truly succesful commercialisation of a CNCF project.

  • This means that generally outside of projects favoured by one of the backbone technology companies, this pattern of concentration repeats itself.
  • The choice of licence and structure itself is explicitly intended to help vendors both contribute, yet commercialise - but beyond this ideological fork in the road, there is limited to no support. Many folks, for example Adam Jacob formerly of Chef have got a very nuanced take on this on X/Twitter, where he calls out the bind that CNCF often leaves companies in - but its echo'd by folks from WeaveWorks too (Edit: He's commented on this further up here too, I didnt put the github names together).

Food for thought.

[JorTurFer]

Honestly, I don't think that this issue is the place to talk about cilium or any other project, as this is for checking the health status of Linkerd as CNCF project. If there is any other project whose health can be discussed, let's do it in their own health checks

I've checked some of the commitments done during the graduation process and I'd just want to add my 2c providing some information for who is interested on it.

Have committers from at least two organizations

This is a requirement that all the graduated project have to address, in this case Linkerd was an exception, but they did some commitments:

Throughout its history, Linkerd has always been open to contributors and
maintainers from anywhere in the world. Linkerd has publicly committed to open
governance
and features a steering committee comprising end-users.

Checking the governance section, I can see that there is a maintainers section, where current maintainers are listed:

But Buoyant's CEO assumed the ownership of the decision in their own blog, despite he isn't present in the current maintainers panel. He is just who pays their salaries (he told that, not me):

There is another interesting point related with the steering committee. They (in theory) meet periodically not less than once a quarter. Recordings and minutes will be posted publicly., but checking the channel where those records are posted, there isn't any record since 2021. The last one happened only 2 days after graduation and never more. I think that this is interesting because one of the responsibilities is to "Provide neutral mediation for non-technical disputes." and this looks as a good point to be discussed there and records posted publicly.

Disclaimer: I'm not saying that they haven't met, I'm just saying that the public place where they shared the meetings haven't shared any other meeting after that and I can't find any other place where they are posted (but definitively it can exist and I just haven't found it)

[steve-gray]

Honestly, I don't think that this issue is the place to talk about cilium or any other project, as this is for checking the health status of Linkerd as CNCF project. If there is any other project whose health can be discussed, let's do it in their own health checks

If folks want to signal that somehow this is an issue here, and that "Project X" and "Project Y" they happen to be involved in is somehow good examplar, then I'm afraid that that does imply that a discussion of the comparative behaviours and realities of projects is fair game. It also lends itself to the question "Is this a Linkerd issue, or a wider problem with the framework under which projects operate in the CNCF?" - I feel ultimately the deck is rarely stacked for success.

I think any discussion of this issue would be remiss to not point out that a dramatic subset of the CNCF graduated projects have similar issues, and that list of 25 would whittle down pretty hard if people want to start drawing lines in the sand. I cited specific projects when they were brought out by those involved them directly, but it's obvious that you don't have to spin the spotlight around too far on its mount to find many, many other similar situations. Now maybe that is as important to discuss?

Have committers from at least two organizations

This is a requirement that all the graduated project have to address, in this case Linkerd was an exception

Is there a policy around reviewing the grants of these exceptions, a process for periodically renewing or revoking them? Going back and changing the deal years later, once the trademarks were signed over and after many years of hard service from the team behind the project is poor form.

But Buoyant's CEO assumed the ownership of the decision in their own blog, despite he isn't present in the current maintainers panel.

To break this down:

• He's able to put into effect whatever policy he wants around Buoyant though, and given their role in the maintenance of the project, he's as good a voice as any to be the figurehead on the discussion - I don't think it'd be fair or proper to just to grab one of the MAINTAINERS.md list and have them take the heat on this?
• I think this assumes that the maintainers didn't collectively agree to it, and that Will's not just putting a voice to it. If that's the case, is there a policy that these announcements have to come from someone in MAINTAINERS.md?
• The reality is that it's pretty much the norm people from the "vendor" side to be involved in public-facing statements, blog posts or media from the OSS projects without being in MAINTAINERS.md. I'm happy to provide citations of specific examples of this from other projects (it took minutes to find a few).
• Just like the general vendor concentration issue, this is an issue prevalent in CNCF graduates. I can see one particular example of a project where someone effectively speaks as the voice of the project, is in MAINTAINERS.md, but actually hasn't maintained the project from a technical level personally in years?

I'm actually not saying this is a problem either, but its far from unusual.

There is another interesting point related with the [steering committee]...........
.... I think that this is interesting because one of the responsibilities is to "Provide neutral mediation for non-technical disputes." and this looks as a good point to be discussed there and records posted publicly.

If the principal duty of that committee apart from "assisting" maintainers is in various ways is to discuss disputes, has anyone raised one on say, GitHub issues for the project, or somewhere else? I mention it because apart from one thread in Slack, nobodies actually bothered to formally raise a dispute with it. Going to be a quiet chat otherwise. The reality is that the decision has been met with quiet acceptance on aggregate, people don't like it fully, sure, but the reality is that the majority of discussion on the issue has been heavily brigaded in nature, taking place in non Linkerd forums.

(To qualify here, its not a Github issue on the project yet, nor has it had any discussion on Slack beyond one thread where a few folks who are patently not users of the project turned up to presumably controversy-farm for quotes in the public channels).

[JorTurFer]

I fully agree that this is something deeper than just Linkerd, but once again, this issue is to track Linkerd health.

I don't think it'd be fair or proper to just to grab one of the MAINTAINERS.md list and have them take the heat on this?

The election of the words is quite important for official communications, something like "the project", "maintainers", etc... could be more appropriated than the CEO of the company which will offer the enterprise license saying that it's his decision

To qualify here, its not a Github issue on the project yet, nor has it had any discussion on Slack beyond one thread where a few folks who are patently not users of the project turned up to presumably controversy-farm for quotes in the public channels

I personally asked in CNCF's Linkerd channel, as can be checked here 2 days ago, and I'm also a Linkerd user.

Honestly, I'm not trying to say that Buoyant is the evil, I fully understand their movement because they are a company trying to be sustainable. But it doesn't mean that I can't say that IMHO the project as CNCF project isn't doing the things well. There isn't any hidden roadmap behind my opinion at all, but as maintainer from another graduated project, I had to pass through a process that at least I thought that was to prevent these situations.

This coin has two sides, what will happen if all the projects from the CNCF landscape only offer stable artifacts behind a paywall? Will it be acceptable too? That's why I strongly suggest maintaining the issue's goal as other projects can be interested too on the result.

[steve-gray]

I fully agree that this is something deeper than just Linkerd, but once again, this issue is to track Linkerd health.

Understood. However I feel that if this discussion is taking place on what is.... very soft sand in terms of anything being done wrong, then it's pretty clearly going to need to morph into a "Do the rules need to change for projects being graduted" chat.

I personally asked in CNCF's Linkerd channel, as can be checked here 2 days ago, and I'm also a Linkerd user.

I stand corrected - in my defence had no idea that was an official project channel - and I suspect given the moderate activity in there generally nor did almost anyone, it's probably a forum that people pass an eye over.... irregularly. It's not even mentioned on Linkerd.io anywhere.

Honestly, I'm not trying to say that Buoyant is the evil, I fully understand their movement because they are a company trying to be sustainable. But it doesn't mean that I can't say that IMHO the project as CNCF project isn't doing the things well.

Thats a perfectly valid question. I think the issue is that 'the things' are generally poorly defined, and that general setup of things is such that there are many slow-moving fireballs moving roughly in the same direction amongst CNCF projects.

There isn't any hidden roadmap behind my opinion at all, but as maintainer from another graduated project, I had to pass through a process that at least I thought that was to prevent these situations.

I wasn't referring to you on anything in particular on this front - but I simply meant that people with a vested commercial or personal interest in Linkerd getting a bruise being part of these discussions feels a little warped, especially when wheeling out their own projects to justify their criticisms. You'd be a good example of someone without a fighter in the ring on this issue.

This coin has two sides, what will happen if all the projects from the CNCF landscape only offer stable artifacts behind a paywall? Will it be acceptable too? That's why I strongly suggest maintaining the issue's goal as other projects can be interested too on the result.

Its an interesting one, and one that probably requires some decisions about how to come up with a more prescriptive policy around this that outlines the expectations and requirements. I mean the requirement totality for a graduated project is A clear versioning scheme.. Now I do feel two things are true concurrently:

• The scheme they're moving to is clear
• Its controversial.

A clearer, more prescriptive standard on what's expected here wouldn't hurt the CNCF to have - but applying it to all projects fairly, backporting it into their processes etc will be interesting - especially once those projects with a single key vendor start seeing the scope of mandatory process increasing.

[christianhuening]

I read @kzap 's comment to be less about Cilium, but rather about the pattern and that this situation could relatively easily happen to more projects not by ill intention but mere chance and the dynamics of the cloud-native (business) world.

If Isovalent would for some reason have to make the decision to stop investing time into stable OSS builds, the responsibility to do so would rely on the remaining non-Isovalent committers/maintainers.

What I am trying to say here, is that the decision to stop providing builds tagged as "stable", has been done by the company currently investing the majority of maintainers to the project LinkerD. So if there were committers from other companies, that would make for a more diverse (healthier?) ecosystem but would also not guarantee the publishing of stable, quality checked release builds in and as of itself.

So the discussion should not be about builds or no builds, but rather about what we do with projects that fall below the required minimum (for maintainer diversity) and whether the minimum requirements contain the right metrics to look at.