From c119e23e71ebf0ed9530d5ff66a2df862f0d7496 Mon Sep 17 00:00:00 2001 From: Tiexin Guo Date: Tue, 29 Nov 2022 11:35:31 +0800 Subject: [PATCH 1/3] docs: update app-centric docs Signed-off-by: Tiexin Guo --- docs/core-concepts/config.md | 302 +++++++++++++++++++++++------- docs/core-concepts/overview.md | 15 +- docs/core-concepts/stateconfig.md | 0 docs/core-concepts/tools-apps.md | 10 +- 4 files changed, 247 insertions(+), 80 deletions(-) delete mode 100644 docs/core-concepts/stateconfig.md diff --git a/docs/core-concepts/config.md b/docs/core-concepts/config.md index 476d736df..b99a1e8df 100644 --- a/docs/core-concepts/config.md +++ b/docs/core-concepts/config.md @@ -1,120 +1,282 @@ -# DevStream Configuration +# Config -DevStream uses YAML files to describe your DevOps toolchain configuration. +Now let's have a look at some config examples. -## Main Config File +TL;DR: [see the complete config file examle at the end of this doc](#7-putting-it-all-together). -By default, `dtm` tries to use `./config.yaml` (under your current directory) as the main config. +--- -The main config contains three sections: +## 1 The Config File -- `varFile`: the path/to/your/variables file -- `toolFile`: the path/to/your/tools configuration file -- `pluginDir`: the path/to/your/plugins directory, default: `~/.devstream/plugins`, or use `-d` flag to specify a directory -- `state`: configuration of DevStream state. For example, +As mentioned in the overview section, the main config contains many sections, like: -### Example Main Config File +- `config` +- `vars` +- `tools` +- `apps` +- `pipelineTemplates` -See the `config.yaml` example below: +By default, `dtm` tries to use `./config.yaml` (under your working directory) as the main config. + +You can override the default value with `-f` or `--config-file`. Examples: + +```shell +dtm apply -f path/to/your/config.yaml +dtm apply --config-file path/to/your/config.yaml +``` + +--- + +## 2 State Config + +The `config` section specifies where to store the DevStream state. + +### 2.1 Local ```yaml -varFile: "./variables.yaml" -toolFile: "./tools.yaml" -appFile: "./apps.yaml" -templateFile: "./templates.yaml" -state: # state config, backend can be "local", "s3", or "k8s" - backend: local - options: - stateFile: devstream.state +config: + state: + backend: local + options: + stateFile: devstream.state +``` + +_Note: The `stateFile` under the `options` section is mandatory for the local backend._ + +### 2.2 Kubernetes + +```yaml +config: + state: + backend: k8s + options: + namespace: devstream # optional, default is "devstream", will be created if not exists + configmap: state # optional, default is "state", will be created if not exists +``` + +### 2.3 S3 + +```yaml +config: + state: + backend: s3 + options: + bucket: devstream-remote-state + region: ap-southeast-1 + key: devstream.state ``` -### Variables File +_Note: the `bucket`, `region`, and `key` under the `options` section are all mandatory fields for the s3 backend._ -The var file is a YAML file containing key-value pairs. +--- -_At the moment, nested/composite values (for example, the value is a list/dictionary) are not supported yet._ +## 3 Variables -See the `variables.yaml` example below: +To not repeat yourself (Don't repeat yourself, DRY, see [here](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself),) we can define some variables in a var file and use the vars in the config file. + +The vars section is a YAML dict containing key-value pairs. Example: + +Example: ```yaml -githubUsername: daniel-hutao -repoName: dtm-test-go -defaultBranch: main -dockerhubUsername: exploitht +vars: + repoOwner: RepoOwner + repoTemplateBaseURL: github.com/devstream-io + imageRepoOwner: ImageRepoOwner ``` -### Tool File +To use these variables in a config file, use the following syntax: -Tool file contains a list of tools. +```yaml +[[ varNameHere ]] +``` -The tool file contains: +--- -- Only one section (at the moment), which is `tools`. -- `tools` is a list of dictionaries. -- Each dictionary defines a DevOps "tool" which is managed by a DevStream plugin -- Each dictionary (tool) has the following mandatory fields: - - `name`: the name of the tool/plugin, string, without underscore - - `instanceID`: the id of this tool instance - - you can have duplicated `name` in one config file, and you can also have duplicated `instanceID` in one config file, but the `name + instanceID` combination must be unique in one config file -- Each dictionary (tool) has an optional field which is `options`, which in turn is a dictionary containing parameters for that specific plugin. For plugins' parameters, see the "plugins" section of this document. -- Each directory (tool) has an optional field which is `dependsOn`. Continue reading for detail about dependencies. +## 4 Tools -See the `tools.yaml` example down below: +The tools section contains a list of tools. ```yaml tools: - name: repo-scaffolding - instanceID: golang-github + instanceID: myapp options: destinationRepo: - owner: [[ githubUsername ]] - org: "" - repo: [[ repoName ]] - branch: [[ defaultBranch ]] + owner: [[ githubUser ]] + repo: [[ app ]] + branch: main repoType: github - vars: - ImageRepo: "[[ dockerhubUsername ]]/[[ repoName ]]" sourceRepo: org: devstream-io - repo: dtm-scaffolding-golang + repo: dtm-scaffolding-python repoType: github -- name: jira-github-integ + vars: + imageRepo: [[ dockerUser ]]/[[ app ]] +- name: githubactions-python instanceID: default - dependsOn: [ "repo-scaffolding.golang-github" ] + dependsOn: [ repo-scaffolding.myapp ] options: - owner: [[ githubUsername ]] - repo: [[ repoName ]] - jiraBaseUrl: https://xxx.atlassian.net - jiraUserEmail: foo@bar.com - jiraProjectKey: zzz + owner: [[ githubUser ]] + repo: [[ app ]] + language: + name: python branch: main + docker: + registry: + type: dockerhub + username: [[ dockerUser ]] + repository: [[ app ]] +- name: helm-installer + instanceID: argocd +- name: argocdapp + instanceID: default + dependsOn: [ "helm-installer.argocd", "githubactions-python.default" ] + options: + app: + name: [[ app ]] + namespace: argocd + destination: + server: https://kubernetes.default.svc + namespace: default + source: + valuefile: values.yaml + path: helm/[[ app ]] + repoURL: ${{repo-scaffolding.myapp.outputs.repoURL}} ``` -### State +If you want tool A to be installed before tool B, you can let tool B depend on tool A. -The `state` section specifies where to store DevStream state. As of now (v0.5.0), we only support local backend. +The syntax for dependency is: -From v0.6.0 on, we will support both "local" and "s3" backend store the DevStream state. +```yaml +dependsOn: [ "ToolName.ToolInstanceID" ] +``` -Read the section [The State Section in the Main Config](./stateconfig.md) for more details. +Since `dependsOn` is a list, a tool can have multiple dependencies: -## Default Values +``` +dependsOn: [ "ToolName1.ToolInstanceID1", "ToolName2.ToolInstanceID2", "..." ] +``` -By default, `dtm` uses `config.yaml` as the main config file. +--- -### Specifying a Main Config File Explicitly +## 5 Apps -You can override the default value with `-f` or `--config-file`. Examples: +The Apps section looks like the following: -```shell -dtm apply -f path/to/your/config.yaml -dtm apply --config-file path/to/your/config.yaml +```yaml +apps: +- name: service-A + spec: + language: python + framework: django + repo: + scmType: github + owner: devstream-io + org: devstream-io # either owner or org must exist + name: service-A # defaults to "app.name" + url: github.com/devstream-io/repo-name # optional. if exists, no need for the scm/owner/org/name sections + apiURL: gitlab.com/some/path/to/your/api # optional, in case of self-managed git + repoTemplate: # optional. if repoTemplate isn't empty, create repo according to the template + scmType: github + owner: devstream-io + org: devstream-io # either owner or org must exist + name: dtm-scaffolding-golang + url: github.com/devstream-io/repo-name # optional. if exists, no need for the scm/owner/org/name sections + ci: + - type: template # type template means it's a reference to the pipeline template. read the next section. + templateName: ci-pipeline-1 + options: # optional, used to override defaults in the template + vars: # optional, key/values to be passed to the template + key: value + cd: + - type: template # type template means it's a reference to the pipeline template. read the next section. + templateName: cd-pipeline-1 + options: # optional, used to override defaults in the template + vars: # optional, key/values to be passed to the template + key: value +``` + +Since many key/values have default values, it's possible to use the following minimum config for the apps section: + +```yaml +apps: +- name: myapp1 + spec: + language: python + framework: django + repo: + url: github.com/ironcore864/myapp1 + repoTemplate: + url: github.com/devstream-io/dtm-scaffolding-python + ci: + - type: githubactions + cd: + - type: argocdapp +``` + +--- + +## 6 Pipeline Templates + +You can define some pipeline templates in the pipelineTemplates section: + +```yaml +pipelineTemplates: +- name: ci-pipeline-1 + type: githubactions # corresponds to a DevStream plugin + options: + branch: main # optional + docker: + registry: + type: dockerhub + username: [[ dockerUser ]] + repository: [[ app ]] +- name: cd-pipeline-1 + type: argocdapp + options: + app: + namespace: argocd + destination: + server: https://kubernetes.default.svc + namespace: default + source: + valuefile: values.yaml + path: helm/[[ app ]] + repoURL: ${{repo-scaffolding.myapp.outputs.repoURL}} ``` -### No Defaults for varFile and toolFile +Then, you can refer to these pipeline templates in the apps file. + +--- -For `varFile` and `toolFile`, no default values are provided. +## 7 Putting It All Together -If `varFile` isn't specified in the main config, `dtm` will not use any var files, even if there is already a file named `variables.yaml` under the current directory. +Here's a complete example: -Similarly, if `toolFile` isn't specified in the main config, `dtm` will throw an error, even if there is a `tools.yaml` file under the current directory. +```yaml +config: + state: + backend: local + options: + stateFile: devstream.state + +tools: +- name: helm-installer + instanceID: argocd + +apps: +- name: myapp1 + spec: + language: python + framework: django + repo: + url: github.com/ironcore864/myapp1 + repoTemplate: + url: github.com/devstream-io/dtm-scaffolding-python + ci: + - type: githubactions + cd: + - type: argocdapp +``` diff --git a/docs/core-concepts/overview.md b/docs/core-concepts/overview.md index f62c82121..9ad092166 100644 --- a/docs/core-concepts/overview.md +++ b/docs/core-concepts/overview.md @@ -29,14 +29,13 @@ The main config file, which defaults to `config.yaml` in the working directory, There are a few different configs, but please don't be overwhelmed because some are not mandatory, and [you can define all things within a single file](https://stackoverflow.com/questions/50788277/why-3-dashes-hyphen-in-yaml-file). -Configurations in the main config: - -- `pluginDir`: the directory where DevStream plugins are stored, defaults to `~/.devstream/plugins`. You can also use the `-d' flag to specify a directory. -- `state`: state-related settings. Read more [here](./state.md). -- `varFile`: the location of the variable file used by DevStream. The variable file is a YAML of key/values, which can be referred to in the toolFile/appFile/templateFile. -- `toolFile`: the location of the tool file used by DevStream. The tool file is a list of DevStream _Tools_, each containing its name, instanceID (unique identifier), and options. Read more [here](./tools-apps.md). -- `appFile`: the location of the app file used by DevStream. The app file is a list of _Apps_, another DevStream concept, each corresponding to a microservice. Read more [here](./tools-apps.md). -- `templateFile`: the location of the template file used by DevStream. The templateFile can be referred to by DevStream _Apps_. Read more [here](./tools-apps.md). +Configurations in the main config contains multiple sections: + +- `config`: basic configuration of DevStream, at the moment mainly state-related settings. Read more [here](./state.md). +- `vars`: variable definitions. Key/value pairs, which can be referred to in the tools/apps/pipelineTemplates sections. +- `tools`: a list of DevStream _Tools_, each containing its name, instanceID (unique identifier), and options. Read more [here](./tools-apps.md). +- `apps`: a list of _Apps_, another DevStream concept, each corresponding to a microservice. Read more [here](./tools-apps.md). +- `pipelineTemplates`: a list of templates which can be referred to by DevStream _Apps_. Read more [here](./tools-apps.md). --- diff --git a/docs/core-concepts/stateconfig.md b/docs/core-concepts/stateconfig.md deleted file mode 100644 index e69de29bb..000000000 diff --git a/docs/core-concepts/tools-apps.md b/docs/core-concepts/tools-apps.md index 12637a05a..7e6eeaa25 100644 --- a/docs/core-concepts/tools-apps.md +++ b/docs/core-concepts/tools-apps.md @@ -1,6 +1,6 @@ # Tools and Apps -## 1 Tools +## 1 tools DevStream treats everything as a concept named _Tool_: @@ -14,7 +14,7 @@ Each dependency is named in the format of "TOOL_NAME.INSTANCE_ID". --- -## 2 Apps +## 2 apps Sometimes, you have to define multiple _Tools_ for a single app/microservice. For example, for a web application, you might need to specify the following tools: @@ -27,3 +27,9 @@ If you have multiple apps to manage, you'd have to create many _Tools_ in the co To manage multiple apps/microservices more easily, DevStream has another level of abstraction called _Apps_. You can define everything within one app (like the aforementioned repository scaffolding, CI, CD, etc.) with only a few config lines, making the config much easier to read and manage. Under the hood, DevStream would still convert your _Apps_ configuration into _Tools_ definition, but you do not have to worry about it. + +--- + +## 3 pipelineTemplates + +A DevStream _App_ can refer to one or multiple elements of pipelineTemplates, which are mainly CI/CD definitions. In this way, the _Apps_ definition can be shorter, sharing common CI/CD pipelines between multiple microservices. From 575a8b9dd4ae8163edf1588b8cb378af95dd9d2d Mon Sep 17 00:00:00 2001 From: Tiexin Guo Date: Tue, 29 Nov 2022 11:38:50 +0800 Subject: [PATCH 2/3] docs: fix broken links Signed-off-by: Tiexin Guo --- docs/development/git-workflow/reviewing.md | 2 +- docs/development/git-workflow/reviewing.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/development/git-workflow/reviewing.md b/docs/development/git-workflow/reviewing.md index 385a6e202..310a77bbf 100644 --- a/docs/development/git-workflow/reviewing.md +++ b/docs/development/git-workflow/reviewing.md @@ -64,7 +64,7 @@ Below are a set of common questions that apply to all pull requests: Reviewers are encouraged to read the following articles for help with common reviewer tasks: - [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) -- [Project Layout](https://docs.devstream.io/en/latest/development/devstream/project-layout/) +- [Project Layout](https://docs.devstream.io/en/latest/development/project-layout/) - [The Art of Closing: How to closing an unfinished or rejected pull request](https://blog.jessfraz.com/post/the-art-of-closing/) - [Kindness and Code Reviews: Improving the Way We Give Feedback](https://product.voxmedia.com/2018/8/21/17549400/kindness-and-code-reviews-improving-the-way-we-give-feedback) - [Code Review Guidelines for Humans: Examples of good and back feedback](https://phauer.com/2018/code-review-guidelines/#code-reviews-guidelines-for-the-reviewer) diff --git a/docs/development/git-workflow/reviewing.zh.md b/docs/development/git-workflow/reviewing.zh.md index 377daa589..adf726a3e 100644 --- a/docs/development/git-workflow/reviewing.zh.md +++ b/docs/development/git-workflow/reviewing.zh.md @@ -66,7 +66,7 @@ 我们鼓励审稿人阅读以下文章，这会对常见的审查工作有所帮助： - [常规提交](https://www.conventionalcommits.org/en/v1.0.0/) -- [项目结构](https://docs.devstream.io/en/latest/development/devstream/project-layout.zh/) +- [项目结构](https://docs.devstream.io/en/latest/development/project-layout.zh/) - [关闭的艺术：如何关闭未完成或被拒绝的拉取请求](https://blog.jessfraz.com/post/the-art-of-closing/) - [善意和代码审查：改进我们反馈的方式](https://product.voxmedia.com/2018/8/21/17549400/kindness-and-code-reviews-improving-the-way-we-give-feedback) - [代码审查指南：良好反馈和反向反馈的示例](https://phauer.com/2018/code-review-guidelines/#code-reviews-guidelines-for-the-reviewer) From 761873f81d37c5b8bd622bb8e536e3fa885971d7 Mon Sep 17 00:00:00 2001 From: Tiexin Guo Date: Tue, 29 Nov 2022 12:02:02 +0800 Subject: [PATCH 3/3] docs: add vars into repoTemplate example config Signed-off-by: Tiexin Guo --- docs/core-concepts/config.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/core-concepts/config.md b/docs/core-concepts/config.md index b99a1e8df..ab33a2258 100644 --- a/docs/core-concepts/config.md +++ b/docs/core-concepts/config.md @@ -184,6 +184,8 @@ apps: org: devstream-io # either owner or org must exist name: dtm-scaffolding-golang url: github.com/devstream-io/repo-name # optional. if exists, no need for the scm/owner/org/name sections + vars: # optional + foo: bar # variables used for repoTemplate specifically ci: - type: template # type template means it's a reference to the pipeline template. read the next section. templateName: ci-pipeline-1