Feature Infrastructure Service

[gerardcl]

Proposal to fix opendevstack/ods-quickstarters#631

For now the proposal is to have stage odsComponentStageInfrastructureAWS that uses InfrastructureService which wraps the Makefile make executions (similar approach as other tooling stages like Aqua). The config param expected are the envPath, which the default is as per current AWS quickstarter envs path ./environments, and the resourceName used for tracking timings.

In future we could consider naming it odsComponentStageInfrastructure and pass the provider as a config param, so any quickstarter specialized on a provider (e.g.: AWS, Azure, GCP,...) would just need the right environments and Makefile in place.

The current approach will also let users have in the same component used for business logic (e.g.: nodejs backend) to leveraging the related resources in their infrastructure (e.g.: RDS or S3), keeping a nice micro-service config approach.

This is how the current QS Jenkinsfile would look like now:

/* generated jenkins file used for building and deploying AWS-infrastructure in projects */

@Library('ods-jenkins-shared-library@4.x') _

odsComponentPipeline(
  imageStreamTag: 'ods/jenkins-agent-terraform:4.x',
  branchToEnvironmentMapping: [
    'master': 'dev',
  ]
) { context ->
    odsComponentStageInfrastructureAWS(context)
}

TODO:

• Add docs
• Add tests
• Changelog

@metmajer shall I move the target issue to this repo?

FYI @metmajer @michaelsauter @tbugfinder @nichtraunzer @clemensutschig let me know your thoughts!
Question: would you agree and see no issues on adding (sub)stages within curren proposed stage (so to expose to jenkins specific stages for testing, build,... all called inside the main IaC stage)? Basically, with the new approach we only get one stage one:

And the previous one (which we could add back as suggested, with sub-stages) looks like:

[gerardcl]

hi all, we have also tested this within RM context and worked without issues (and LeVa docs looks good too!) :)

this is the metadata.yml content used:

id: grd
name: Project grd
description: Description of grd.

services:
  bitbucket:
    credentials:
      id: grd-cd-cd-user-with-password

# Uncomment (below) to connect to jira
#  jira:
#    credentials:
#      id: grd-cd-cd-user-with-password
  nexus:
    repository:
      name: leva-documentation

repositories:
  - id: terraform
    name: grd-terraform
    type: ods-infra

#  - id: A
#    url: https://github.com/my-org/my-repo-A.git
#    branch: master
#    type: ods
#  - id: B
#    name: my-repo-B
#    branch: master
#    type: ods
#  - id: C
#    type: ods
#  - id: D
#    type: ods
#    forceRebuild: true

# Uncomment (below) to generate documents	  
capabilities:
 - LeVADocs:
     GAMPCategory: 5
     templatesVersion: "1.1"

# Uncomment (below) to allow partial rebuilding during development
# allowPartialRebuild: true

Screenshots:
1- within current proposal:

2- within the current existing QS implementation:

[nichtraunzer]

Hi @gerardcl
great job and thanks for the exeptional support :)

Question: would you agree and see no issues on adding (sub)stages within curren proposed stage (so to expose to jenkins specific stages for testing, build,... all called inside the main IaC stage)? Basically, with the new approach we only get one stage one:

And the previous one (which we could add back as suggested, with sub-stages) looks like:

Regarding the stages I actually changed my mind ...
Bug tracing might be easier when sub stages are used because I immediately could jump to the stage which has an error (when using BlueOcean) instead of browsing trough tons of log output.

[nichtraunzer]

BTW: @gerardcl and I will wait for your highly appreciated feedback before we proceed with our proposal.

[nichtraunzer]

Looks good to me

[tbugfinder]

Hi @gerardcl ,
would we have the option to skip specifc stages e.g. test or drop the test-env completely after prod deployment to reduce costs?

@nichtraunzer Above pipeline shows Smoke - could the full stage name be displayed?

[tbugfinder]

Is the cloud provider available as a parameter?

[metmajer]

@gerardcl @nichtraunzer @tbugfinder Nice work, guys, but you currently leave out the end-user perspective. It would be good to see what the corresponding Jenkinsfile.template of the AWS Quickstarter would look like for the user. Here I expect a corresponding change. The result of this change should be that this is now very slim. The benefit besides improved modularity and maintainability is that a support team can more easily assess if a component has been manipulated, or not.

[metmajer]

Hi @gerardcl , would we have the option to skip specifc stages e.g. test or drop the test-env completely after prod deployment to reduce costs?

Installation tests are required after an installation into any environment. Pre-installation test optional, post-installation are not.

[metmajer]

@nichtraunzer Note that the Release Manager currently schedules aws-infra components in the Deploy phase, not in the build phase. This seems useful, since the Build stage should provide quick feedback on the question: do my application components build and unit-test correctly?

[nichtraunzer]

Hi @gerardcl , would we have the option to skip specifc stages e.g. test or drop the test-env completely after prod deployment to reduce costs?

Good point - we already considered providing a list of stages as optional parameter to odsComponentStageInfrastructure to specify only stages needed. Might require a wider discussion ...

@nichtraunzer Above pipeline shows Smoke - could the full stage name be displayed?

Do you want to see "Smoke Test" Screenshots listed here are from 3.x/4.x branches - we did not touch naming of stages as far as I know ...

[nichtraunzer]

Is the cloud provider available as a parameter?

Do we need that? Isn't that self contained as part of the Quickstarter?

[nichtraunzer]

@nichtraunzer Note that the Release Manager currently schedules aws-infra components in the Deploy phase, not in the build phase. This seems useful, since the Build stage should provide quick feedback on the question: do my application components build and unit-test correctly?

I can't follow - the question was if we want see substages or not - and I said I want to have substages :)

[nichtraunzer]

We think that the current version is ready for a review @metmajer @tbugfinder @michaelsauter @kuebler
The current proposal has shift to contain the initial steps to support multiple cloud providers - see

https://github.com/opendevstack/ods-quickstarters/pull/749/files#diff-a58bdbe85174e45a7fbe5363b108c78b2936fca8e38562836482589084f56c9bR11

[metmajer]

Great work overall, @nichtraunzer @gerardcl @tbugfinder. Some minor improvements asked for in the questions.

[gerardcl]

hi! so this is now ready to get a final review from all involved in this. Please, note that once this is merged (to master) we will need (correct me if I am wrong) to port this changes to ODS 4.x too.

[nichtraunzer]

Thanks @gerardcl for the great support

[michaelsauter]

I only had a brief glance, but overall this looks fine to me.

[nichtraunzer]

Thanks @gerardcl and get well soon!

[metmajer]

@nichtraunzer can we please have the same changes in the 4.x branch via a dedicated PR?

[nichtraunzer]

@metmajer I would like to run a complete end 2 end test together with the EDP first before moving to 4.x