🚀 Feature: Workflow-Orchestration Plugin

[alex-gunning]

🔖 Feature description

This feature aims to add a workflow-orchestration system into Backstage. This plugin would allow creating workflows which are generic enough to handle automating the work of a user-access team. This orchestration system should be able to save and load state from the database and transition according to the workflow rules.

More use-cases for generic workflows would most-likely be discovered in the future.

🎤 Context

Tooling access-control for developers is usually pain-point in companies. Sometimes it is controlled at an IDP level but more often than not requires some sort of manual process for onboarding - emails approvals, slack messages etc.

Creating automated systems for onboarding can sometimes be complicated depending upon the use case. For example, to onboard a user onto security tooling one might need the following flow in Backstage.

• See if user's group is eligible in the catalog.
• Create a approval request (in-app and/or email) to user's line-manager.
• Approval Logic:
  • Approved ->
    • Create calls to SASS service to onboard user.
    • Succeeded?
      • Send notification about approval.
      • Log domain event occurrence.
    • Failed?
      • Log failure.
      • Send error notification.
  • Rejected ->
    • Send notification about rejection.

There multiple interactions here with different actors. (User, manager, API response) and these would all need their appropriate entrypoints into Backstage. The various entrypoints would not necessarily form part of this work. For example, exposing a router to listen to email clicks.

✌️ Possible Implementation

One fairly easy option would be to use something like x-state, which is a minimal-dependency typescript library.
It can create complex state-machines, persist state and run side-effects. It has a nice visualisation library as well.

👀 Have you spent some time to check if this feature request has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

Are you willing to submit PR?

Yes I am willing to submit a PR!

[awanlin]

Hi @alex-gunning, can you contrast this with the current Scaffolder and Software Templates? I feel like there's something I'm missing in why you are asking for this as I feel like the Scaffolder could do this.

Also, the idea of approvals or gates for the Scaffolder and Software Templates was requested here: #16622

[jbadeau]

Are you aware of https://github.com/janus-idp/backstage-plugins/tree/main/plugins/orchestrator

[alex-gunning]

@awanlin Thanks for the reply! You'll have to excuse my ignorance here as I'm not intimately familiar with scaffolder (My company has been mainly concerned with read-mode)

I had a similar intuition to you but from what I understand scaffolder is more of a resource/infrastructure creation tool. While it seems fairly flexible in what it can do, it seems very heavyweight and not suited to small complex interactions between steps (for instance parallelisation, races or logical retries). Yes, gating could go some ways in accomplishing what I want here.

Ideally, I'd like something that's lightweight, easy to build, and can persist and rehydrate on demand. Think an action occurring on a listener which pulls up a workflow, rehydrates it and advances it by the rules already present inside it before persisting it again.

If that's a possible and realistic expectation of scaffolder's future then I guess I have my answer. I'm all in favour of not reinventing the wheel.

@jbadeau Thanks for this. I heard whispers. I'll take a look.

[awanlin]

Yeah, I just don't think that having multiple plugins with similar usage makes sense but I'm not sure what the full roadmap for the scaffolder is. Given your comments I'm not sure the Orchestrator plugin that @jbadeau mentioned would be any less "heavywieght" but I could be wrong it's pretty new to me.

It might be worth joining one of the Scaffolder SIGs to chat about this. The next one is actually tomorrow: https://calendar.google.com/calendar/u/0/embed?src=c_qup9gbhn9sqpuao6trttd8mk5s@group.calendar.google.com

[jbadeau]

In general, my feeling is that orchestrating things directly in the scaffolder should be kept to a minimum as it’s not a workflow engine and has very basic or no support for common wotkflow features. Running complex workflows should prob start with collecting user information in the scaffolder and then triggering workflows via a controlplane or api. The Janus orchestration plugin is a frontend for sonataflow which supports complex orchestrated and choreographed workflows.

[alex-gunning]

@jbadeau the capabilities seem very advanced but I think it's probably too much for what I was explaining above - given it's using an external process to orchestrate the workflows and all.

@awanlin Thanks for the link. I'll see if I can join tomorrow.

[alex-gunning]

Hey @awanlin I did bring this topic up at the SIG. You can see the writeup here

https://docs.google.com/document/d/19VMhugnvmmX_F6tu-53_BsNknCxLme5VUzUCLa4nzSM/edit#heading=h.q3mrd0yojm8o

Just a TLDR I think we can conclude that scaffolder in its current form isn't really designed to handle workflows. In the future it may be able to handle certain aspects (retries, gating, resumes) but it's still a way off and once there it would definitely be more flexible but still wouldn't be able to handle many common workflow actions that a workflow engine could.

[awanlin]

Thanks for the follow up @alex-gunning, given that I'm not sure where this lands then as it won't be a core feature. Should this be changed to a plugin request and maybe someone from the Community will build it?

Personally you seem to have some very specific requirements that are partially covered by some things - Scaffolder, Orchestrator plugin - but not entirely. I'm not sure I've seen lots of demand for these specific things in the Community. My opinion is that your best bet is to build this yourself. If you do build something and want to contribute it keep in mind that would be submitted to the new community-plugins repo and not the main Backstage repo: https://github.com/backstage/community-plugins

[alex-gunning]

@awanlin Yeah, I'd rather not speculate about the adoption rate. What I know is what I mentioned above, this is a pain point for us and I imagine many other companies. Self-service is our aim here.

As you mentioned It's probably best I just build this with community help. After we get some adoption we'll be able to tell how useful it is where/if there is a place for it in the community.

[awanlin]

Awesome, sounds good to me @alex-gunning 🚀

[Alok650]

Hi @alex-gunning
I have a use-case, which involves having an approval based workflow (maker-checker mechanism) within scaffolder templates. Please let me know if you were able to implement this gating feature.

[freben]

possibly related: #16622