Docs how to organize project

docs/GettingStarted/HowToOrganizeDevlakeProjects.md

@@ -0,0 +1,227 @@
+---
+title: "How to Organize DevLake Projects"
+sidebar_position: 1
+description: >
+  How to Organize DevLake Projects
+---
+
+## 1. Introduction
+This guide provides step-by-step instructions for organizing projects in DevLake. 
+It is intended for team leads and developers who want to optimize their workflow. 
+Herein, we will explore various use cases to demonstrate the practical application of these instructions.
+
+Development teams typically manage `pull requests`, `deployments`, and `incidents` using various boards.
+
+Based on such, we want to measure their productivity and stability. This is how [DORA](../DORA.md) does that:
+- Productivity:
+  - How many times does the team `deploy`? (a.k.a. [Deployment Frequency](../Metrics/DeploymentFrequency.md))
+  - How fast are the `pull requests` resolved? (a.k.a. [Lead Time](../Metrics/LeadTimeForChanges.md))
+- Stability:
+  - How many `incidents` per `deploys` does the team have? (a.k.a. [Change Failure Rate](../Metrics/CFR.md))
+  - How fast are these `incidents` solved? (a.k.a. [Median Time to Restore](../Metrics/MTTR.md))
+
+All these questions/metrics are based on either `pull requests`, `deployments`, or `incidents`.
+
+Note: _All three are completely separate entities and are associated only with their project._
+
+#### General advice
+
+There are 3 red lines when it comes to structuring your DevLake `projects`:
+
+But when we scale this up, a few problems arise:
+- A team usually works with multiple `repos`
+- A team also might work on different projects, and we want to measure these projects separately (e.g. it is not the same to work on a big old legacy as on a greenfield)
+- There may be multiple teams
+- A `board` contains incidents of multiple teams or projects
+- A `repository` is managed by multiple teams or projects, e.g. a monorepo
+- A `pipeline` can trigger deployments in multiple repos
+- Some organizations want to measure DORA based on projects, and some want to measure it by teams
+
+This is where the `project` concept comes into play.
+
+## 2. What is a DevLake project?
+In practical terms, a project usually involves building or researching solutions to address specific problems 
+or explore new opportunities. A DevLake project is a grouping of `pull requests`, `deployments`, or `incidents`. 
+It can be seen as a real-world project or product line. DevLake measures DORA metrics for each project.
+
+![](../Configuration/images/HowToOrganizeDevlakeProjects/project_pipeline.png)
+
+Note: _It does not matter if a team works on a particular repository more than another.
+The metrics are calculated over the entire set of repos, and the values are accumulated.
+More on that: [Debugging DORA Issue Metrics](../Troubleshooting/Dashboard.md#debugging-dora-issue-metrics)._
+
+## 3. As a team lead, how many DevLake projects do I need?
+
+DevLake's flexible structure enables you to organize `pull requests`, `deployments`, and `incidents` 
+in a manner tailored to your specific projects, teams, or technologies.
+
+The examples below show the patterns of how to organize your projects.
+To keep things simple we assume that we work with **GitHub** repos, **Jira** boards, 
+and **Jenkins CI/CD** deploys in each GitHub repo.
+
+The same would apply to other repos (e.g. GitLab or BitBucket), boards (e.g. TAPD), 
+or CI/CD (e.g. GitLab CI, Azure DevOps).
+
+## 4. Use Cases
+This section demonstrates real-life situations and how they get reflected in DevLake.
+
+Disclaimer: _To keep this guide shorter, some technical details are only mentioned in
+[Use Case 1](HowToOrganizeDevlakeProjects.md#41-use-case-1-apache-projects),
+so if you read this page for the first time, make sure to go through them in order._
+
+Note: _If you use webhooks, check the [quick note](HowToOrganizeDevlakeProjects.md#5-about-webhooks) about them below._
+
+### 4.1. Use Case 1: Apache Projects
+Apache Software Foundation (ASF) has and is developing many
+[projects](https://en.wikipedia.org/wiki/List_of_Apache_Software_Foundation_projects).
+
+To take an example we will analyze 2 `projects`: DevLake and [Spark](https://spark.apache.org/).
+Both are independent of each other. Assume that ASF wants to check the health of the development
+and maintenance of these projects with DORA.
+
+DevLake manages 3 `repos`: [incubator-devlake](https://github.com/apache/incubator-devlake), 
+[incubator-devlake-website](https://github.com/apache/incubator-devlake-website), 
+and [incubator-devlake-helm-chart](https://github.com/apache/incubator-devlake-helm-chart).
+There are many repos related to _Spark_ in one way or another. To keep it simple, 
+we will also pick 3 `repos`: [spark](https://github.com/apache/spark),
+[spark-website](https://github.com/apache/spark-website), and [incubator-livy](https://github.com/apache/incubator-livy).
+
+![](../Configuration/images/HowToOrganizeDevlakeProjects/project_use_case_1.png)
+
+Both projects use GitHub for storing code (including `pull requests`), `deployments` on GitHub Actions, and `incidents`.
+
+Note: _To avoid confusion between DevLake as a `project` in this use case and DevLake as a platform,
+we will use complete names i.e. `project DevLake` and `platform DevLake` respectively._
+
+#### 4.1.1. Organizing Projects
+First, create two projects on the DevLake platform, one for DevLake and one for Spark. 
+These will represent real-world projects.
+
+![](../Configuration/images/HowToOrganizeDevlakeProjects/create_project_1.png)
+![](../Configuration/images/HowToOrganizeDevlakeProjects/create_project_2.png)
+
+Once these are created, the connections created in the following steps will be bound to them.
+
+#### 4.1.2. Creating Connections
+
+Since all is on GitHub in this case, we can use just 1 connection with the following properties:
+- it includes all the project's `repos`
+- its scope includes everything we work with (i.e. `pull requests`, `deployments`, and `incidents`)
+
+If you store `incidents` on Jira, for example, you will need to create a separate connection just for them.
+The same applies to `deployments`, a separate connection is needed in case they are stored in Jenkins (or any other host for `deployments`).
+
+#### 4.1.3. Configuring Connections
+This part is described in [GitHub](/docs/Configuration/GitHub.md) connection configuration.
+For other platforms check the [Configuration Guide](/docs/Configuration)
+
+#### 4.1.4. Using connections
+
+At this point, we have projects and connections created on the platform DevLake. 
+It is time to bind those connections to the projects. To do so, follow the steps
+described in the [Tutorial](/docs/Configuration/Tutorial.md).
+
+#### 4.1.5. Resulting Metrics
+
+To know if the data of a project is successfully collected go to your DORA Dashboard:
+
+![](../Configuration/images/HowToOrganizeDevlakeProjects/navigate_to_dora_1.png)
+![](../Configuration/images/HowToOrganizeDevlakeProjects/navigate_to_dora_2.png)
+
+If everything goes well, you should see all the 4 charts.
+If something is wrong, and you are puzzled as to why, check out the
+[Debugging Dora Issue Metrics](../Troubleshooting/Dashboard.md#debugging-dora-issue-metrics) page.
+
+#### 4.1.6. How can I observe metrics by project?
+In the same DORA dashboard check out this menu point:
+![](../Configuration/images/HowToOrganizeDevlakeProjects/observe_metrics_by_project_panel.png)
+
+The metrics should change when you select or deselect projects, representing the projects you selected.
+
+### 4.2. Use Case 2: Multiple Teams with Distinct Projects
+
+Consider a scenario where a company operates with several teams, each managing one or more projects. 
+For illustration, we will explore two such teams: the Payments team and the Internal Tools team. 
+Here's a simplified representation of this scenario:
+
+#### Quick Overview:
+- The Payments team works on a single project: “payments”.
+- The Internal Tools team manages two projects: “it-legacy” and “it-new”.
+- Both teams use different sets of tools and boards.
+
+![](../Configuration/images/HowToOrganizeDevlakeProjects/project_use_case_2.png)
+
+#### Step-by-Step Simplification:
+1. **Define the Teams and Projects:**
+   - **Payments Team**:
+     - One project: "payments". 
+   - **Internal Tools Team**:
+     - Two projects: "it-legacy" and "it-new". 
+2. **Understand the Tools**:
+   - Assume both teams utilize GitHub for `repos` and Jenkins for CI/CD.
+   - The _Payments_ team uses Jira boards.
+   - The _Internal Tools_ team uses webhooks for reporting incidents.
+
+#### 4.2.1. Organizing Projects
+
+DORA is effective for observing the impacts of methodology changes within a team.
+From DORA’s standpoint, the concept of distinct `teams` is not recognized; only `projects` exist. 
+Adding a `team` concept introduces unnecessary complexity without providing any substantial benefit. 
+
+In DevLake, we create three `projects`: _payments_, _it-legacy_, and _it-new_.
+
+It is crucial to maintain **atomic** `projects`, representing the smallest, independent units, 
+to prevent complexity and ensure precise data representation. **Atomic** `projects` allow for a more flexible 
+and accurate data comparison and combination between `projects`.
+
+#### 4.2.2. Adding Connections
+
+Create just one connection and reuse it across projects by adding data scopes.
+This method optimizes data collection, minimizing redundancy and ensuring more efficient use of resources.
+
+It is NOT recommended to create multiple connections, for instance, GitHub repos, as it  
+will increase the time to collect the data due to the storage of multiple copies of shared repos in the database.
+
+The only exception is the webhooks: **we must have 1 connection per project**, 
+as this is the only way DevLake can accurately assign `incidents` to the corresponding `project`.
+
+So, in total we will have only these connections:
+- 1 connection for all GitHub `repos` to collect `pull requests`
+- 1 connection to Jenkins to collect all `deployments`
+- 1 connection to Jira to collect `incidents`
+- 2 webhook connections to collect `incidents`: 1 per each `project` that uses webhooks (_it-legacy_ and _it-new_)
+
+The step-by-step [Configuration Guide](/docs/Configuration) shows how to both add connections 
+and set scopes as described in the next chapter.
+
+#### 4.2.3. Setting Scopes
+Now, add the connections to our projects and set the scope to them:
+
+For payments `project`: 
+- add 1 scope to GitHub connection for _p1...p10_ `repos` to collect their `pull requests`
+- add 1 scope to Jenkins for `deployments` of _p1...p10_ `repos`
+- add 1 scope to Jira to collect `incidents`
+
+For it-legacy `project`:
+- add 1 scope to GitHub for `repos` _it-legacy-1_, _it-legacy-2_, _it-core-1_ and _it-core-2_ to collect their `pull requests`
+- add 1 scope to Jenkins for `deployments` of _it-legacy-1_, _it-legacy-2_, _it-core-1_ and _it-core-2_ `repos`
+- include the _it-legacy_ webhook for collecting `incidents`
+
+For it-new `project`:
+- add 1 scope to GitHub for `repos` _it-new-1_, _it-new-2_, _it-core-1_ and _it-core-2_ to collect their `pull requests`
+- add 1 scope to Jenkins for `deployments` of _it-new-1_, _it-new-2_, _it-core-1_ and _it-core-2_ `repos`
+- include the _it-new_ webhook for collecting `incidents`
+
+#### 4.2.4. Resulting Metrics
+See [4.1.5 Resulting Metrics](HowToOrganizeDevlakeProjects.md#415-resulting-metrics)
+
+## 5. About Webhooks
+**Assigning a UNIQUE webhook to each project is critical.** This ensures that the DevLake platform 
+correctly associates the incoming data with the corresponding project through the webhook.
+
+If you use the same webhook across multiple projects, the data sent by it **will be replicated per each
+project that uses that webhook**. More information available on the [Webhook](/docs/Plugins/webhook.md) page
+
+## 6. Troubleshooting
+
+If you run into any problem, please check the [Troubleshooting](/docs/Troubleshooting/Configuration.md) or [create an issue](https://github.com/apache/incubator-devlake/issues)