Pipelines: Backend re-work

[Pezmc]

Merge target is feature branch 2075-application-pipelines (#2094)

Description

All behind the scenes changes, the front-end presents exactly the same as in #2094, bugs and all.

• Re-works the backend to use a through table to connect instances to pipeline stages
• Explicitly creates all the required tables as part of a migration
• Updates the backend to allow multiple instances per pipeline stage
• Updates the frontend to only allow one instance per pipeline stage (for now)
• Adds some validations for instances and guards against instance being null at pipeline stage creation time
• Small misc cleanup

Related Issue(s)

Part of #2124

Checklist

• I have read the contribution guidelines
• [-] Suitable unit/system level tests have been added and they pass
  • Full unit coverage coming as a follow up
• [-] Documentation has been updated
  • Upgrade instructions
  • Configuration details
  • Concepts
• [-] Changes flowforge.yml?
  • Issue/PR raised on flowforge/helm to update ConfigMap Template
  • Issue/PR raised on flowforge/CloudProject to update values for Staging/Production

Labels

• Backport needed? -> add the backport label
• Includes a DB migration? -> add the area:migration label

[joepavitt]

"the front-end presents exactly the same as in #2094, bugs and all."

Just checking, what bugs are these? Any others than:

1. No edit functionality
2. Can still select an already assigned Instance to a Stage?

[joepavitt]

Adding a stage to a new pipeline results in:

[2023-05-11T09:03:37.766Z] INFO: AUDIT: event: application.pipeline.deleted, type: team, id: 1
[2023-05-11T09:04:03.339Z] INFO: AUDIT: event: application.pipeline.created, type: team, id: 1
ValidationError [SequelizeValidationError]: Validation error: this.getInstance is not a function
    at InstanceValidator._validate (/Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/node_modules/sequelize/lib/instance-validator.js:50:13)
    at async InstanceValidator._validateAndRunHooks (/Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/node_modules/sequelize/lib/instance-validator.js:60:7)
    at async InstanceValidator.validate (/Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/node_modules/sequelize/lib/instance-validator.js:54:12)
    at async model.save (/Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/node_modules/sequelize/lib/model.js:2426:7)
    at async PipelineStage.create (/Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/node_modules/sequelize/lib/model.js:1362:12)
    at async Object.addPipelineStage (/Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/packages/flowforge/forge/ee/db/controllers/Pipeline.js:16:23)
    at async Object.<anonymous> (/Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/packages/flowforge/forge/ee/routes/pipeline/index.js:63:21) {
  errors: [
    ValidationErrorItem {
      message: 'this.getInstance is not a function',
      type: 'Validation error',
      path: 'instancesHaveSameApplication',
      value: null,
      origin: 'FUNCTION',
      instance: [PipelineStage],
      validatorKey: 'instancesHaveSameApplication',
      validatorName: null,
      validatorArgs: [],
      original: TypeError: this.getInstance is not a function
          at model.instancesHaveSameApplication (/Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/packages/flowforge/forge/ee/db/models/PipelineStage.js:21:47)
          at InstanceValidator._invokeCustomValidator (/Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/node_modules/sequelize/lib/instance-validator.js:156:31)
          at /Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/node_modules/sequelize/lib/instance-validator.js:93:28
          at /Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/node_modules/lodash/lodash.js:4967:15
          at baseForOwn (/Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/node_modules/lodash/lodash.js:3032:24)
          at /Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/node_modules/lodash/lodash.js:4936:18
          at Function.forEach (/Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/node_modules/lodash/lodash.js:9410:14)
          at InstanceValidator._customValidators (/Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/node_modules/sequelize/lib/instance-validator.js:89:7)
          at InstanceValidator._validate (/Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/node_modules/sequelize/lib/instance-validator.js:47:12)
          at InstanceValidator._validateAndRunHooks (/Users/joepavitt/Documents/flowforge/development/flowforge-dev-env/node_modules/sequelize/lib/instance-validator.js:60:18)
    }
  ]
}

[joepavitt]

problem is in ee/db/models/PipelineStage.js - line 21:

const instancesPromise = this.getInstance()

[joepavitt]

Trying to fix this myself, but can't see where this.getInstance() or this.getPipeline() are/were defined?

[Pezmc]

Trying to fix this myself, but can't see where this.getInstance() or this.getPipeline() are/were defined?

built in sequelize method

[Pezmc]

Just checking, what bugs are these? Any others than:

Those, plus:

• Stuck loading state if a request fails on create/delete
• Creating a pipeline stage without an instance causing the pipeline not to render.

Am tracking in #2124

[knolleary]

Approving to allow progress - but noting the TODOs.