RFC: Stop accepting new plugins to the backstage/backstage repository

[Rugvip]

🔖 Need

Starting out the Backstage project we thought it was important to have a large number of plugins available in the main repository. Both in order to help us make more informed decisions when building out the core framework and features, and also to help keep the project healthy and active with a lot of communication centered in one place.

The project has now grown (a lot! 🎉🙏), and we have plenty of plugins to keep the project active and help guide the evolution of the framework. A little bit too many in fact. We've had previous calls for contributions like #8294, but with the core maintainers being the owners of any plugin without a Project Area or Plugin Maintainer, it remains to be a significant workload for the core maintainers team. We want to reduce this workload, both to free up time for improving the project in other ways, but also to make sure that broadly used plugins receive the care that they deserve.

🎉 Proposal

We will stop accepting new plugins to the backstage/backstage repository. Exceptions are made for existing project areas that want to expand their set of plugins, as well as new library packages for existing plugins.

We recognize that there is some work involved in setting up a repository to publish plugins from. We will provide a GitHub template repo with a base setup to help out. Plugins will not be able to use the @backstage namespace on NPM. They should instead use the @<org>/backstage-plugin-* or backstage-plugin-* naming pattern. Plugin authors can keep adding their plugins to https://backstage.io/plugins to make them easier to discover.

This is a step towards ensuring that the Backstage has a growing and healthy ecosystem of plugins, and we will keep working towards this in other ways. If you are a plugin maintainer that would like to help out by moving your plugin out of the backstage/backstage repo, please let us know and we can help get that sorted. There will be more to come on this in the future, so this is just an early opportunity for those alright with a slightly rockier road as we figure things out.

〽️ Alternatives

We have many times discussed whether we should have a "community plugins" or "plugins contrib" repository in the Backstage GitHub organization. In the end we feel that it would encounter the same issues as what we're seeing in the main repository right now, so it is something we'd like to avoid. The Backstage Open Source project is from the very beginning built towards enabling an open and distributed ecosystem of plugins, with limited need for centralized control.

❌ Risks

There are some benefits to be had to have all plugins in one repository when it comes to keeping plugins up to date, as you can make broader changes more easily. At the same time it also adds friction since you end up in situations where you are forced to make upgrades early. The Backstage framework is in many ways designed to allow for plugins to not always be on the latest and greatest version of the core APIs. It is fine for plugins to lag behind and only do upgrades when necessary or to access new features. This means that while broad upgrades can be slightly more effective, more narrow ones can often be skipped altogether. There is of course also the fact that we are looking to distribute this workload, it is the reason for this RFC.

Another risk is that we simply do not have enough plugins in the main repository to properly test new features that we are building in the core framework. This is not a problem we have right now, but in the future we may need further investment into tooling that allows for smoother cross-repo development.

👀 Have you spent some time to check if this RFC has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

[awanlin]

Thanks for starting the discussion on this topic @Rugvip

As a plugin author I'm torn by this as being able to contribute plugins has helped me contribute more and become more involved in the overall Community (this might be a risk as well) but I do think it is the right path forward. The number of PRs used to sit about 60-70 and now is regularly around 100-110, part of this is due to growth which is great but part of it is due to the number of plugins in the backstage/backstage repo.

I've also been thinking about the idea of providing the Community a way to "spin up" a repo with everything you need to build and support open source plugins - release pipelines, changesets, etc. - all in one command so that you can hit the ground rolling. This would be a pretty awesome feature and would lower some of the barriers for Orgs hosting plugins themselves.

For this to work well I do think we need clear guidance as to what would make sense to be part of the backstage/backstage repo and what should be external. Using the plugins I'm an author/co-author of as an example:

• Azure DevOps: This to me would stay, it feels very much a Day 1 or 2 plugin you would want to add
• DevTools: This one to me is more tricky, it's terribly helpful but I would be open to it being in either
• Linguist: This easily can be external, it's a great plugin but it's one that you may not be adding early in your adoption path

This feels a bit unclear, and is really opinion based, having some guidelines published would help with this aspect.

Finally I think we need to make Plugin Discovery easier - the Plugin Directory is nice but it's also getting large, hard to find things and doesn't really help showcase the capabilities of plugins very well.

[jcarres-mdsol]

Have you considered as alternative allowing a monorepo for all those other plugins?

For instance, the current backstage/backstage can be divided:

• backstage/framework Same as current repo but without any plugin.
• backstage/plugins-core Plugins the core team would review and merge PRs for (catalog, search..)
• backstage/plugins-third-party An easy place for other plugins. Ownership here is delegated to other authors, not the core team.

This last repository then may eventually become many repos in other orgs, at that point does not matter much, the ownership of the core team is low on this one.

[jcarres-mdsol]

* Azure DevOps: This to me would stay, it feels very much a Day 1 or 2 plugin you would want to add

* DevTools: This one to me is more tricky, it's terribly helpful but I would be open to it being in either

* Linguist: This easily can be external, it's a great plugin but it's one that you may not be adding early in your adoption path

As we do not use Azure, to me Azure Devops is 100% not core. I would never use it.
Linguist probably also external but debatable
Devtools looks core to me. I have not installed myself but looks like a Backstage native functionality more than the others.

[awanlin]

As we do not use Azure, to me Azure Devops is 100% not core. I would never use it. Linguist probably also external but debatable Devtools looks core to me. I have not installed myself but looks like a Backstage native functionality more than the others.

@jcarres-mdsol based on your previous suggestion to have 3 repos - framework, core, and plugins - that makes sense to me but it also highlights the point I was trying to make around the need for clear guidance.

[freben]

Note also the comment regarding a community plugins repo in the Alternatives section above.

[awanlin]

Just wanted to share some recent issues that are worth considering:

• Grafana Plugin: Community is seeing that it's not being maintained and are wanting to fork it, they did want to bring that into the backstage/backstage repo, Discord thread here: https://discord.com/channels/687207715902193673/1156136524484718665
• Azure DevOps Scaffolder Actions: these don't support the latest version which changed the YAML syntax and don't look like that will be done in the near term, Discord thread here: https://discord.com/channels/687207715902193673/1154845617756065932

These seem to strengthen the Alternative proposal of having a "community plugins" repo.

[antoniobergas]

Hi, this looks like an important topic, I can imagine the huge workload... In fact these last months every time I turned around a few weeks a lot of new plugins appeared in the marketplace!

Based on the little experience I've had maintaining a Backstage instance, the best part is being able to update automatically with the cli and tools you provide and enjoy an easily and constantly updated developer portal.

The only problems we have had are due to external plugins (we only use plugins that are on the list or that you mention in the documentation).

All the proposals look like good candidates but is there any way to ensure compatibility of the plugins regardless of the Backstage version? Also I guess that in case most of plugins became external we will lose the possibility to update plugins with backstage-cli right?

[jcarres-mdsol]

In my experience backstage core has done a good job having deprecation periods for changes that would break plugins.
Even with the change to the new backend system you could basically run both side by side.
I would expect if a plugin breaks is because it has been unmaintained for a quite long period of time.

[antoniobergas]

I totally agree with you on that, I was just having some thoughts about what @Rugvip mentioned here
but in the future we may need further investment into tooling that allows for smoother cross-repo development.

[P1llus]

Totally understandable situation to be in, and I wholeheartedly agree that plugins outside of core components should really live outside, either in a separate repo from backstage or from the plugin maintainer.

TLDR: New CLI/Documentation steps for plugin devs specific for contributions would be great. I also think both core and community plugins should be moved out of core into separate repo's, and that community plugins be allowed to live in a repo still under backstage, so it can reuse many of the points below.

A few notes as someone who is and wants to continue to develop plugins for the purpose of contribution, but is not part of the core maintainer group, many points which @awanlin already mentioned, mostly about what I feel will be missed now that plugins would be developed outside the main repo:

1. A already good CI structure for testing a plugin against backstage.
2. I think certain tooling/linting is handy that is setup only on the main repo, the usage of api-extractor is just one example.
3. Currently most "get started" documentation and CLI commands are expecting you to be developing plugins for your own internal usage, I think both documentation focusing solely on setting up a single or mono repo for plugins meant to be shared would be very useful. The same could be said about CLI commands, the templates used could use some tweaking to take this into consideration?
4. As a plugin would not be part of a PR to the main repo, would we still be able to request feedback/reviews on plugins from the core team? Without any promises or timelines of course.

[Pike]

I've started my train-of-thought with a clear "yes, that helps the machine move".

Now I'm thinking: Yes, we have a people problem. I'm concerned that creating dozens of people problems isn't making the ecosystem better.

From a contributor perspective, we "contribute to backstage" right now. An external plugin is an obligation to check for deprecations at least every other release. That moves plugins from a contribution to an obligation. I'm also concerned that many plugins will struggle to find a community in their own right. So we'd often get a people problem for each plugin, to do triage, maintenance, development, reviews, and succession of ownership.

From an adopter perspective, I stopped using 3rd party plugins a year ago. I moved over to "steal and own". Back then, I was frequently blocked from updating backstage by the lack of a release on the 3rd party side. I don't know if that's better today.

Also, I'm one of those weird folks that cares about their supply chain. Having various versions of the same code bothers me, and they're mostly aligned inside the backstage repo. And easy to fix in a monorepo, compared to a multi-repo setup.

PS: I've been in a couple of different places on my train of thought, and some might be interesting technically, but not relevant to this comment.

[andrewthauer]

I can definitely try and see both sides of the fence, but obviously can't sit in the maintainers shoes. I share much of @Pike's experience and sentiment. We also don't use many 3rd party plugins at all for similar reasons and/or challenges upgrading Backstage core, etc.

I think this also comes down to an indirectly related aspect to this. Sure having a lot of plugins is nice. We use x and backstage has a plugin for x. That said, the quality and usefulness of plugins varies drastically. Just having a check mark that says oh yeah Backstage has a plugin for x doesn't cut it. As early adopters we tried this and the kitchen sink approach didn't really benefit anyone. Instead we are much more selective of what plugins we adopt and if they actually solve a business problem. So, from that perspective more is not necessarily better and quality and maintenance of specific plugins is more important. This brings the question, how does this RFC impact this?

The other thing I've often been pondering, is even within the same monorepo there are cases where concepts are duplicated when they could be generalized and re-used. This is at the best of times a hard problem to solve as probjects get bigger. However, splitting things up will likely make consolidating and generalizing concepts a bit more challenging. At the very least potentially discouraging collaboration to contribute some cross cutting things back to core.

I haven't been as active in the community in the last few months, but I have been following things. The ideas around project areas, etc. seems sound, etc. but I can't comment on how well that has worked. I've also been following OpenTelemetry a bit and albeit the project being much larger, they also have these concepts and have quite a few contrib repos in the main org that are maintained by different project area owners, etc. I can't comment to say how well this works for them, but I'm kind of wondering if this model might also work for Backstage. Similar to the points @awanlin mentioned.

[jcarres-mdsol]

I think the cross-pollination of ideas/components and the creation of generalizations is a huge pro of a monorepo. I think one more reason to have a backstage-plugins repo that have them all instead of dispersing them around.
That say, responding to the comment about plugins having different quality. A problem with the monorepo is that eventually some of those plugins will have no maintainer and eventually backstage will do some backwards incompatible change. Do we keep broken plugins there? What it the CI does not pass, who fixes it. That could be a big problem with a monorepo.

[Rugvip]

Awesome feedback on this RFC, thank you all 😁 🎉

Starting off with a couple of individual replies:

@awanlin

Finally I think we need to make Plugin Discovery easier - the Plugin Directory is nice but it's also getting large, hard to find things and doesn't really help showcase the capabilities of plugins very well.

Very much this. There's a lot of discussions around this topic here. There's both the discoverability angle, but also curation and other tools to help make sound decisions for which plugins to install and use.

---

@jcarres-mdsol

Have you considered as alternative allowing a monorepo for all those other plugins?

Yes, and to be clear we still are. The proposal in this RFC would be more of a short-term adjustment to help alleviate capacity issues, but we are still considering many paths forward. That said we are still leaning towards the proposal in this RFC of moving plugins out of the main repo in some form, but the discussion here is valuable feedback that will help us find a good path.

---

@antoniobergas

All the proposals look like good candidates but is there any way to ensure compatibility of the plugins regardless of the Backstage version? Also I guess that in case most of plugins became external we will lose the possibility to update plugins with backstage-cli right?

Backwards compatibility in the core is a very high priority for us, the compatibility with older versions of @backstage/core-plugin-api goes very far back. There are more moving parts than that though, and especially when you look at plugins that build upon other plugins there might be more frequent breakage. It's something that we are constantly working towards though and we believe the evolved frontend and backend systems will help a lot here too.

For the second point: absolutely not! not in the long term anyway 😁 We want to evolve the versions:bump command to consider all Backstage plugins, regardless of package name, and likely make it interactive by default too. Btw, if anyone wants to pick up and drive this work I'd be happy to help out guiding that contribution 🙏

To provide a bit more context on the "smoother cross-repo development", that's be in a world where we have plugin backstage/backstage into multiple repos. Let's for example say that there is a plugins repo and core, we'd still want it to make it very simple to iterate on changes to the core and trying them out locally in the plugins repo with the same or better devex than we have in the main repo today.

---

@P1llus

As a plugin would not be part of a PR to the main repo, would we still be able to request feedback/reviews on plugins from the core team? Without any promises or timelines of course.

That's an interesting idea and of course hard to scale, but I do think it would be nice to have in some form. Perhaps a more general plugin review request issue type in this repo or similar.

---

@Pike

From a contributor perspective, we "contribute to backstage" right now. An external plugin is an obligation to check for deprecations at least every other release. That moves plugins from a contribution to an obligation. I'm also concerned that many plugins will struggle to find a community in their own right. So we'd often get a people problem for each plugin, to do triage, maintenance, development, reviews, and succession of ownership.

Yep the problem right now is that those obligations instead fall on the core maintainers, which isn't scaleable. As discussed there are more solutions to that other than moving all plugins to separate repos. No decisions are made yet and we're still figuring out the way forward.

From an adopter perspective, I stopped using 3rd party plugins a year ago. I moved over to "steal and own". Back then, I was frequently blocked from updating backstage by the lack of a release on the 3rd party side. I don't know if that's better today.

I would say this is quite problematic but also expected for some cases. I think it's alright for some cases, but shouldn't be the default approach to adopting external plugins. It's certainly a good method for building internal plugins though.

---

@andrewthauer

The other thing I've often been pondering, is even within the same monorepo there are cases where concepts are duplicated when they could be generalized and re-used. This is at the best of times a hard problem to solve as probjects get bigger. However, splitting things up will likely make consolidating and generalizing concepts a bit more challenging. At the very least potentially discouraging collaboration to contribute some cross cutting things back to core.

I agree and it's one of the reasons we went with the backstage/backstage approach to begin with. It's an important thing to consider as we find the way forward here. Exactly how things are sliced up can have a pretty big effect, Conway's law etc.

I've also been following OpenTelemetry a bit and albeit the project being much larger, they also have these concepts and have quite a few contrib repos in the main org that are maintained by different project area owners, etc. I can't comment to say how well this works for them, but I'm kind of wondering if this model might also work for Backstage. Similar to the points @awanlin mentioned.

Might've looked in that direction once or twice 😁 Been speaking to some of them at KubeCon too about their governance structure. OpenTelemetry has the contrib repos with quite a lot of activity, and also a lot of separate repos for SDKs and other components.

---

@jcarres-mdsol

I think the cross-pollination of ideas/components and the creation of generalizations is a huge pro of a monorepo.

It is, but I don't think it's the only way to do it. I've actually found that the varying quality of plugins is a bit of a trap in backstage/backstage. When searching for inspiration it's a bit too easy to find some old and less optimal code. A repository split where we're able to maintain high code quality throughout the entire repo would help out here I'd think.

---

To finish I have a more general question on the topic of using external plugins that I'd love some feedback on in order to help us move forward and improve the ecosystem:

What pain-points do you experience when using plugins that do not belong to @backstage/*?

I'll mention again that we plan to improve the versions:bump to make it more usable for the entire ecosystem, and please let us know if you want to help contribute towards that 🙏

[antoniobergas]

I appreciate the detailed responses and insights! 😁

I wanted to highlight a current pain point that @ahhhndre briefly mentioned earlier. Starting from Backstage version 1.18.0, there have been changes in the Azure DevOps integration syntax, particularly for those using Personal Access Tokens (PAT). This transition has affected our usage, and I'd like to outline the differences:

Before 1.18.0

integrations:
  azure:
    - host: dev.azure.com
      token: ${AZURE_DEVOPS_TOKEN}

With 1.18.0

integrations:
  azure:
    - host: dev.azure.com
      credentials:
        - personalAccessToken: ${AZURE_DEVOPS_TOKEN}

We rely on two scaffolder plugins that are not under the @backstage/* namespace:

• Scaffolder Azure DevOps Pipelines
• Scaffolder Azure DevOps Repositories

These plugins are affected by this change as they still use the old syntax in their code. Here's an excerpt of the code snippet from one of these plugins:

  if (!integrationConfig.config.token && !ctx.input.token) {
    throw new InputError(`No token provided for Azure Integration ${host}`);
  }

  const token = ctx.input.token ?? integrationConfig.config.token!;

I've attempted to use both syntaxes concurrently, but it hasn't yielded the desired results.

Previously, I could easily update Backstage every week when a new release was shipped. However, with these template changes, updating Backstage now leads to broken configurations.

I've been in communication with the maintainers of these plugins, and my initial suggestion was to migrate them to use the @backstage/* namespace. However, after becoming aware of this RFC (Thanks @ahhhndre ❤️), I'm now waiting to see what options we have moving forward.

That's the current situation, and I hope this information is helpful in making any decisions. 😁

P.S. After wrapping up with the Azure DevOps plugins and a few other tasks, I'd be happy to contribute to the improvement of the version:bump. Currently, I'm working on a Node/Bun CLI project, so any learnings would be greatly appreciated!

[Rugvip]

@antoniobergas I think this is a pretty good example of how most of the time, this is a technical issue, in particular a lack of abstraction. The proper solution to this is to use an injected AzureDevOpsCredentialsProvider as a layer of indirection between config and the plugin code. That can in turn then consume different versions of the azure config, independent of the plugin's access to credentials. These are the kind of abstractions that we want to keep building out both for plugins not to have to implement it themselves, but more importantly to unify configuration and provide backwards and forwards compatibility.

It's only a technical problem in the sense that it needs to be designed and built though. There's also the angle of identifying, encouraging and making it easy to build these abstractions. That in the end is related to @andrewthauer's point above where a monorepo(s) can really help out in the identification and creation of solutions to these kind of common problems. For this and other reasons we are considering a path forward with a repo structure that includes repo(s) oriented towards these kind of integrations.

[jcarres-mdsol]

Another datapoint backstage/mkdocs-techdocs-core#145

I think it is significant that someone has these questions on a repo in this very github org.
The backstage main repo is awesomely well maintained. But confidence falls quick outside this main repo.

[tudi2d]

The idea of a Community Repository is great! To align with the RFC's goal of reducing core maintainer workload, the question remains what a sustainable model for maintaining the Community Repository could look like. Can the SIGs and project areas play a central role in maintaining the repository, are there individuals from the community who would be keen to help out?

An alternative to a single community repository could be a "Contribution" Organization for Backstage in GitHub. Examples for this approach are coming from Nuxt and Crossplane, which are using it for their plugin-like ecosystem.

[jmezach]

Just adding my two cents here, as also mentioned to @benjdlambert offline at KubeCon today, I think having a community plugins repo within the Backstage org makes the most sense here. That way the open source project would at least have the ability to assign new ownership to plugins that have been abandoned by their original maintainers. That would be really hard to do if all community created plugins would live under various different orgs or users.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[awanlin]

@Rugvip, is this genuinely stale and you have the feedback you need?

[Rugvip]

@awanlin yep let's close this. Our goal is to move forward with a community plugins repo, thank you all for your feedback! 🙏 🎉