Application Pipelines

[joepavitt]

Description

Introduces a new "Pipelines" view at the Application level.

Functionality:

• Can add a pipeline
• Can delete a pipeline
• Can add a stage to a pipeline and define which instance that stage links to
• Can manually run a single stage of a pipeline (iteration 1 - future would be fully automated pipeline)

To-do:

• Test Coverage:
  • Server-side
  • E2E Frontend
• Polling of live status & "last deployed" for instances in a stage
• Audit Logging (Added in separate PR for easier reviewing: Add Audit Events for DevOps Pipelines #2098)
• PostHog Events
• Improved accuracy of "last updated" - seems to change when a project is added to a pipeline? Would ideally be last restart/deployment of the nodes
• Check needsPermission for new endpoints

Related Issue(s)

Closes #2082
Closes #2076

Checklist

• I have read the contribution guidelines
• Suitable unit/system level tests have been added and they pass
• Documentation has been updated
  • Upgrade instructions
  • Configuration details
  • Concepts

[joepavitt]

Polling of live status & "last deployed" for instances in a stage

@Pezmc would appreciate some advice/input on this outstanding task. We still just repeating a HTTP request every X seconds, or did something more sophisticated get added that I haven't seen?

[joepavitt]

Development work should now be complete, apart from:

• Improved accuracy of "last updated" - seems to change when a project is added to a pipeline? Would ideally be last restart/deployment of the nodes
• Check needsPermission for new endpoints (@knolleary need your input here - can connect early next week)

For early next week, need to prioritise test coverage for the new work. I'm happy to take the E2E front-end testing, but will need assistance server-side. @Pezmc if you have the bandwidth, would be great to get the assist.

TLDR for server-side changes and where test coverage is likely required:

• New API endpoints (/applications/:applicationid/pipelines and /pipelines/:pipelineid/:stages)
• New models, views & controllers: Pipeline and PipelineStage

[joepavitt]

Have added limited E2E testing as currently, we have no way of testing licensed features in the E2E tests - also the same issue with the Team Library.

[knolleary]

The unit test failure on postgres comes down to how CE/EE features co-exist in the database models and when/if migrations get applied.

For a clean run of the forge app (no existing database), the code relies 100% on the sequelize models to generate the database structure. The migrations do not get used.

This PR introduces a migration that adds a relation between the (CE) Projects table and (EE) PipelineStages table. However, in a clean run, that migration will not get applied. So the Projects table does not have the expected PipelineStageId column.

Furthermore, the existing migration will fail when upgrading an existing instance of FF - because the migrations run before sequelize initiates the models, so the PipelineStages table won't exist at that point in time.

I need to spend some more time working out how to fix all of this.

In summary it involves:

1. Adding migrations to create the new tables
2. Consider adding a through table relation between Project and PipelineStage (like we have Team -> TeamMember -> User). This will remove the need to modify the Projects table at all.

This is a few hours work, including tests.

[knolleary]

Having been poking at this to understand the database requirements a bit better. Hit a couple UX things:

1. Bug: When creating a pipeline stage it is possible to pick an instance that is already assigned to a stage. If you do that, the pipeline view no longer renders the pipeline - so you cannot edit/delete it. Had to delete the pipeline/stages in the database to get things back.
   Recreate:
   1. Create pipeline
   2. Create first stage with Instance A
   3. Create second stage, pick Instance A again
   4. The pipeline view is now blank - no way to add/edit/delete the pipeline
2. Missing feature: No ability to edit anything - if you make a typo in a name or pick the wrong instance you have to delete the whole pipeline and start again

[Pezmc]

Approving for myself, see: https://flowforgeworkspace.slack.com/archives/C032Q63FGG1/p1685094359788189

[Pezmc]

Approving for myself, see: https://flowforgeworkspace.slack.com/archives/C032Q63FGG1/p1685094359788189