:four_leaf_clover: `Proposal`: A new way to configure: App

[aFlyBird0]

What Would You Like to Add? Why Is This Needed?

Background

After further discussion of the value of the dtm, we found that the value of dtm revolves heavily around a few specific apps themselves, especially the management of multiple projects in the microservices environment.

Aim & Benefits

• Reduce the length of configuration files in multi-microservice scenarios to make it easier for users to get started
• Maximize the value of DevStream around the app

Design

Core Concepts

We have introduced two new concepts, apps and pipelineTemplates

• apps: a list of apps, representing a set of devops pipeline of microservices
• pipelineTemplates: The devops pipeline for most projects may be almost the same ( e.g. only the project name is different) and the user can customize the template and then reference it within the app and populate it with different variables.

PipelineTemplates

pipelineTemplates is a list starting from the pipelineTemplates key.

each pipelineTemplate has three filed: name, type, options.

• name: the name of pipeline, it is used to identify the pipeline so that you can use them in app
• type: corresponding to a plugin. (We will convert them into tools finally)
• options: options. Maybe there will be a little different from the origin plugins(simpler and with more defaults, auto-completion of fields)

You can define vars in the pipelineTemplate just like [[ dockerUser ]] in the following example. Dtm will try to render variables from :

1. global vars defined in the config files(That is, the way dtm defined variables previously)
2. local vars. You could define local vars in apps when you want to use a template. And you could import the same template many times and pass them to different local vars.
3. local vars will override global vars if the vars' names are the same.

pipelineTemplates:
  - name: ci-pipeline-1
    type: githubactions # corresponding to a plugin
    options:
      branch: main # optional, default is main
      docker:
        registry:
          type: dockerhub
          username: [[ dockerUser ]]
          repository: [[ app ]]
  - name: cd-pipeline-1
    type: argocdapp
    options:
      app:
        namespace: [[ argocdNamespace ]] # you can use global vars in templates
      destination:
        server: https://kubernetes.default.svc
        namespace: default
      source:
        valuefile: values.yaml
        path: helm/[[ app ]]
        repoURL: ${{repo-scaffolding.myapp.outputs.repoURL}}

Apps

Apps is a list starting from the apps key.

Each app has these fields:

• name: app name
• spec: some useful common specs for this app
• repo: source code repo
• repoTemplate: optional. If this block is not empty, dtm will use repo-scaffolding to create the repo from repo template.
• ci: a list of ci pipelines. Each ci has these fields:
  • type: template or a plugin name.
  • templateName(optional): You should specify this field when type is set to template. It is used to find the specific template from pipelineTemplates
  • vars(optional): the local vars passed to the template you import from pipelineTemplates (only valid when type is set to template)
  • options(optional)
    • when type is a plugin name: just the options of this plugin
    • when type is set to template: To override the specify keys defined in options of pipelineTemplate. You should explicitly write out each level of the key of the options to be covered.
    • 
• cd: similar to ci. You could also import pipelineTemplates here. dtm will execute all ci first of an app, then cd.

...
var1: value1 # global vars
var2: value2
var3: value3

apps:
  - name: service-A
    spec:
      language: python
      framework: django
    repo:
      scmType: github
      owner: devstream-io
      org: devstream-io # choose between owner and org
      name: # optional, default is the same as app name
      url: github.com/devstream-io/service-A # optional，if url is specified，we can infer scm/owner/org/name from url
      apiURL: gitlab.com/some/path/to/your/api # optional, if you want to create a repo from repo template
    # if repoTemplate is not empty，we could help user to create repo from scaffoldingRepo
    repoTemplate: # optional
      scmType: github
      owner: devstream-io
      org: devstream-io # choose between owner and org
      name: dtm-scaffolding-golang
      url: github.com/devstream-io/dtm-scaffolding-golang # optional，if url is specified，we can infer scm/owner/org/name from url
    ci:
      - type: template
        templateName: ci-pipeline-1
        options: # overwrite options in pipelineTemplates
          docker:
            registry:
              type: [[ var3 ]] # while overridden, you can even use global variables
        vars: # optional, use to render vars in template（valid only if the ci.type is template）
          dockerUser: dockerUser1
          app: service-A
    cd:
      - type: cd-pipeline-custom # if the type is not "template", it means plugins
        options: # options to the plugins
          key: [[ var1 ]] # use global vars
      - type: template
        templateName: cd-pipeline-1
        options: # overwrite options in pipelineTemplates
          destination:
            namespace: devstream-io
        vars: # optional, use to render vars in template（valid only if the cd.type is template）
          app: service-A
...

Multiple files

You could put apps and pipelineTemplates definitions in the separated files(like var files):

config.yaml

# main config 
varFile: var.yaml # relative to main config file
toolFile: tool.yaml
appFile: app.yaml
templateFile: template.yaml
pluginDir: ./plugins
state: # state config, backend can be local or s3
  backend: local
  options:
    stateFile: devstream.state

dtm init -f config.yaml

What will happen when dtm apply

dtm will execute the following in order:

1. parse all templates and vars
2. handle default values of apps
3. convert apps into tools calls
4. execute the origin tools definition
5. execute the tools converted from apps

Details(default values, options, spec ...)

@steinliber Please help me to complete this section, thank you! 🚀

Anything else

[daniel-hutao]

New format: #1251