-
- Steps 2 to 7 occur in the background and are transparent to the user.
-
-
-1. A user logs in to CDSP
-2. The user is redirected to Codefresh Service Provider (SP) to initiate SSO
-3. The user’s browser is then redirected to the customer IdP
-4. Once authenticated by the corporate side, a SAML token is sent to the user’s browser
-5. The SAML assertion is then forwarded to Codefresh SP
-6. If you are a valid Codefresh user for this SSO connection, an SSO token is returned to the user’s browser
-7. The user’s browser then returns a token to Codefresh and access is granted for your account
-
-### Configure SAML SSO settings in Codefresh
-
-1. In Codefresh, select **Account settings**.
-1. From the sidebar expand **Collaboration**, and select **Single Sign-on**.
- OR
- Go directly to [https://g.codefresh.io/account-admin/sso](https://g.codefresh.io/account-admin/sso))
-
-
- {% include image.html
- lightbox="true"
-file="/images/administration/sso/add-sso-dropdown.png"
-url="/images/administration/sso/add-sso-dropdown.png"
-alt="SSO provider settings"
-caption="SSO provider settings"
-max-width="70%"
-%}
-
-{:start="3"}
-1. Select **Add single-sign-on**, and then select **SAML**.
-1. Enter the following:
-
- * **Client Name**: For auto-generation, leave empty. Codefresh generates the client name once you save the settings.
- * **Display Name**: The name you want to give to this integration.
- * **IDP Entry**: The SSO endpoint of your Identity Provider. For Azure SAML, for example, this is the Login URL.
- * **Application Certificate**: The security certificate of your Identity Provider. Paste the value directly in the field. Do not convert to base64 or any other encoding by hand. (For Azure SAML, this will be Certificate (Base64) and the value needed is between the -----BEGIN ... and -----END... from the downloaded cert)
- * **Assertion URL**: `https://g.codefresh.io/api/auth/- Application state
A snapshot that displays a breakdown of the deployed applications by their health status.
Click a status to filter by applications that match it.
Codefresh tracks Argo CD's set of health statuses. See the official documentation on Health sets. . - Application attributes
Attribute filters support multi-selection, and results are based on an OR relationship within the same filter with multiple options, and an AND relationship between filters.
Clicking More Filters gives you options to filter by Health status, Cluster names, Namespace, and Type.- Application Type: Can be any of the following
- Applications: Standalone applications. See the official documentation on Applications.
- ApplicationSet: Applications created using the ApplicationSet Custom Resource (CR) template. An ApplicationSet can generate single or multiple applications. See the official documentation on Generating Applications with ApplicationSet.
- Git Source: Applications created by Codefresh that includes other applications and CI resources. See Git Sources.
- Application Type: Can be any of the following
- Labels:The K8s labels defined for the applications. The list displays labels of all the applications, even if you have applied filters.
To see the available labels, select Add, and then select the required label and one or more values.
To filter by the labels, select Add and then Apply.
See the official documentation on Labels and selectors.
{:/}| Star applications as favorites and view only the starred applications.{::nomarkdown}Select the
to star the application as a favorite.To filter by favorite applications, on the filters bar, select
.{:/} TIP: If you star applications as favorites in the Applications dashboard, you can filter by the same applications in the [DORA metrics dashboard]({{site.baseurl}}/docs/reporting/dora-metrics/#metrics-for-favorite-applications). | -|Application actions| Options to monitor/manage applications through the application's context menu. {::nomarkdown}
- Quick view
A comprehensive read-only view of the deployment and definition information for the application. {:/}See [Application Quick View](#view-deployment-and-configuration-info-for-selected-application) in this article.{::nomarkdown} - Synchronize/Sync
Manually synchronize the application. {:/}See [Manually sync applications]({{site.baseurl}}/docs/deployment/manage-application/#manually-synchronize-an-application).{::nomarkdown} - Edit
Modify application definitions. {:/}See [Edit application definitions]({{site.baseurl}}/docs/deployment/manage-application/#edit-application-definitions).{::nomarkdown} - Refresh and Hard Refresh: Available in Card view only. In List view, you must first select the application.
- Refresh: Retrieve desired (Git) state, compare with the live (cluster) state, and refresh the application to sync with the desired state.
- Hard Refresh: Refresh the application to sync with the Git state, while removing the cache.
{:/}| Star applications as favorites and view only the starred applications.{::nomarkdown}
Select the to star the application as a favorite.
To filter by favorite applications, on the filters bar, select
.
{:/} TIP: If you star applications as favorites in the Applications dashboard, you can filter by the same applications in the [DORA metrics dashboard]({{site.baseurl}}/docs/reporting/dora-metrics/#metrics-for-favorite-applications). | +|Application actions| Options to monitor/manage applications through the application's context menu. {::nomarkdown}- Quick view
A comprehensive read-only view of the deployment and definition information for the application. {:/}See [Application Quick View](#view-deployment-and-configuration-info-for-selected-application) in this article.{::nomarkdown} - Synchronize/Sync
Manually synchronize the application. {:/}See [Manually sync applications]({{site.baseurl}}/docs/deployments/gitops/manage-application/#manually-synchronize-an-application).{::nomarkdown} - Edit
Modify application definitions. {:/}See [Edit application definitions]({{site.baseurl}}/docs/deployments/gitops/manage-application/#edit-application-definitions).{::nomarkdown} - Refresh and Hard Refresh: Available in Card view only. In List view, you must first select the application.
- Refresh: Retrieve desired (Git) state, compare with the live (cluster) state, and refresh the application to sync with the desired state.
- Hard Refresh: Refresh the application to sync with the Git state, while removing the cache.
{:/} -#### Warning: Missing Rollouts reporter in cluster +### Warning: Missing Rollouts reporter in cluster **Reason**: Codefresh has detected that Argo Rollouts is not installed on the target cluster. Rollout instructions are therefore not executed and the application is not deployed. Applications with `rollout` resources need Argo Rollouts on the target cluster, both to visualize rollouts in the Applications dashboard and control rollout steps with the Rollout Player. @@ -108,7 +109,7 @@ Applications with `rollout` resources need Argo Rollouts on the target cluster,
{:/} -#### Warning: Long sync +### Warning: Long sync **Reason**: Ongoing sync for application exceeds 30 minutes (Argo CD's default duration for a sync operation). **Corrective Action**: @@ -119,7 +120,7 @@ Applications with `rollout` resources need Argo Rollouts on the target cluster, * Drill down into the application to investigate the issue and make changes. -### View deployment and configuration info for selected application +## View deployment and configuration info for selected application View deployment, definition, and event information for the selected application in a centralized location through the Quick View. A read-only view, the Quick View displays information on the application state and location, labels and annotations, parameters, sync options, manifest, status and sync events. @@ -165,7 +166,8 @@ max-width="50%" -##### Quick View: Summary +### Quick View: Summary + Displays health, sync status, and source and destination definitions. {% include @@ -178,7 +180,9 @@ caption="Application Quick View: Summary" max-width="30%" %} -##### Quick View: Metadata + +### Quick View: Metadata + Displays labels and annotations for the application. {% include @@ -191,7 +195,9 @@ caption="Application Quick View: Metadata" max-width="30%" %} -##### Quick View: Parameters + +### Quick View: Parameters + Displays parameters configured for the application, based on the tool used to create the application's manifests. The parameters displayed differ according to the tool: `directory` (as in the screenshot below), `Helm` charts, or `Kustomize` manifests, or the specific plugin. @@ -205,7 +211,8 @@ caption="Application Quick View: Parameters" max-width="30%" %} -##### Quick View: Sync Options +### Quick View: Sync Options + Displays sync options enabled for the application. {% include @@ -218,7 +225,8 @@ caption="Application Quick View: Parameters" max-width="30%" %} -##### Quick View: Manifest +### Quick View: Manifest + Displays the YAML version of the application manifest. {% include @@ -231,7 +239,8 @@ caption="Application Quick View: Manifest" max-width="30%" %} -##### Quick View: Events +### Quick View: Events + Displays status and sync events for the application. {% include @@ -244,7 +253,7 @@ caption="Application Quick View: Events" max-width="30%" %} -### Monitor health and sync statuses for selected application +## Monitor health and sync statuses for selected application Monitor the health status of the selected application, the current sync status, and the result of the previous sync operation. Once you select an application, the quickest option to monitor statuses is through the application header which is always displayed, no matter what tab you navigate to. @@ -274,7 +283,7 @@ max-width="40%" You can also view the current health and sync status for the application as a resource in the Current State tab. -### Monitor resources for selected application +## Monitor resources for selected application Monitor the resources deployed in the current version of the selected application in the Current State tab. Selecting an application from the Applications dashboard takes you to the Current State tab, which as its title indicates, displays the @@ -302,7 +311,7 @@ You can view application resources in [List or Tree views](#view-modes-for-appli > To quickly see which resources have been added, modified, or removed for the current or for a specific deployment, switch to the Timeline tab and expand the deployment record to show Updated Resources. See [Monitor resource updates for deployments](#monitor-resource-updates-for-deployments). -#### View modes for application resources +### View modes for application resources The Current State tab supports Tree and List view formats. * Tree view (default): A hierarchical, interactive visualization of the application and its resources. Useful for complex deployments with multiple clusters and large numbers of resources. See also [Working with resources in Tree view](#working-with-resources-in-tree-view). @@ -335,7 +344,7 @@ max-width="50%" -##### Working with resources in Tree view +#### Working with resources in Tree view The Tree view is designed to impart key information at a glance. Review the sections that follow for pointers to quickly get to what you need in the Tree view. **Context menu** @@ -415,7 +424,7 @@ max-width="50%" %} -#### Filters for application resources +### Filters for application resources Filters are common to both Tree and List views, and when applied are retained when switching between views. `IgnoreExtraneous` is a filter in [Argo CD](https://argo-cd.readthedocs.io/en/stable/user-guide/compare-options){:target="\_blank"} that allows you to hide specific resources from the Current State views. These resources are usually generated by a tool and their sync statuses have no impact on the sync status of the application. For example, `ConfigMap` and `pods`. The application remains in-sync even when such resources are syncing or out-of-sync. @@ -450,7 +459,7 @@ max-width="50%" %} -#### Health status for application resources +### Health status for application resources View and monitor health status of the selected application's resources in the Current State tab, in Tree or List views. Identify the health of an application resource through the color-coded border and the resource-type icon (Tree view), or the textual labels at the right of the resource (List view). @@ -458,18 +467,20 @@ Identify the health of an application resource through the color-coded border an {: .table .table-bordered .table-hover} | Health status | Description | Display in Tree view | | -------------- | ------------| ------------------| -| **Healthy** | Resource is functioning as required. | {::nomarkdown}
{:/} |
-| **Progressing** | Resource is not healthy but can become healthy before the timeout occurs.| {::nomarkdown}
{:/} |
-| **Suspended** | Resource is not functioning, and is either suspended or paused. For example, Cron job or a canary rollout.| {::nomarkdown}
{:/} |
+
+| **Healthy** | Resource is functioning as required. | {::nomarkdown}
{:/} |
+| **Progressing** | Resource is not healthy but can become healthy before the timeout occurs.| {::nomarkdown}
{:/} |
+| **Suspended** | Resource is not functioning, and is either suspended or paused. For example, Cron job or a canary rollout.| {::nomarkdown}
{:/} |
| **Missing** | Resource is not present on the cluster. |{::nomarkdown}
{:/} |
-| **Degraded** | Resource is not healthy, or a timeout occurred before it could reach a healthy status.| {::nomarkdown}
{:/} |
-| **Unknown** | Resource does not have a health status, or the health status is not tracked in Argo CD. For example,`ConfigMaps` resource types. | {::nomarkdown}
{:/} |
+| **Degraded** | Resource is not healthy, or a timeout occurred before it could reach a healthy status.| {::nomarkdown}
{:/} |
+| **Unknown** | Resource does not have a health status, or the health status is not tracked in Argo CD. For example,`ConfigMaps` resource types. | {::nomarkdown}
{:/} |
+
See also [Argo CD's set of health checks](https://argo-cd.readthedocs.io/en/stable/operator-manual/health/){:target="\_blank"}.
-#### Sync status for application resources
+### Sync status for application resources
Similar to the health status, the Current State also tracks the sync status of all application resources. The sync status identifies if the live state of the application resource on the cluster is synced with its desired state in Git.
Identify the sync status through the icon on the left of the resource name and the color of the resource name (Tree view), or the textual labels at the right of the resource (List view).
@@ -479,15 +490,17 @@ The table describes the possible sync statuses for an application resource, and
{: .table .table-bordered .table-hover}
| Sync state | Description |Display in Tree view |
| -------------- | ---------- | ---------- |
-| **Synced** | The live state of the resource on the cluster is identical to the desired state in Git.| {::nomarkdown}
{:/} |
-| **Syncing** | The live state of the resource was not identical to the desired state, and is currently being synced.| {::nomarkdown}
{:/} |
-| **Out-of-Sync** | {::nomarkdown}The live state is not identical to the desired state.
To sync a resource, select the Sync option from the resource's context menu in Tree view. {:/}| {::nomarkdown}
{:/} |
-| **Unknown** | The sync status could not be determined. | {::nomarkdown}
{:/} |
+
+| **Synced** | The live state of the resource on the cluster is identical to the desired state in Git.| {::nomarkdown}
{:/} |
+| **Syncing** | The live state of the resource was not identical to the desired state, and is currently being synced.| {::nomarkdown}
{:/} |
+| **Out-of-Sync** | {::nomarkdown}The live state is not identical to the desired state.
To sync a resource, select the Sync option from the resource's context menu in Tree view. {:/}| {::nomarkdown}
{:/} |
+| **Unknown** | The sync status could not be determined. | {::nomarkdown}
{:/} |
+
> The application header displays the statuses of the current and previous sync operations. Clicking **More** opens the Sync panels with Sync Info, Sync Result and Commit Info.
The Application Warnings/Errors panel surfaces sync errors on exceeding the maximum number of retries and when a sync operation extends beyond 30 minutes.
-#### Manifests for application resources
+### Manifests for application resources
In either Tree or List views, double-click an application resource to see its manifests. The manifests are displayed in the Summary tab.
> Based on the selected resource type, you can also view logs, and events. Endpoints for example show only manifests, while pods show manifests, logs, and events.
@@ -519,7 +532,7 @@ Here's what you can see and do in the Summary tab:
{:/} -#### Logs for application resources +### Logs for application resources In either Tree or List views, double-click an application resource to see its logs. Logs are available only for resource types such as pods. {% include @@ -541,7 +554,7 @@ max-width="50%"
{:/} -#### Events for application resources +### Events for application resources In either Tree or List views, double-click an application resource to see events in the Events tab. > If your runtime is lower than the version required to view events, you are notified to upgrade to the required version. @@ -563,7 +576,7 @@ max-width="50%" -### Monitor deployments for selected application +## Monitor deployments for selected application Monitor an ongoing deployment for the selected application, and review its historical deployments. The Timeline tab displays the history of deployments for the selected application, sorted by the most recent deployment (default), labeled **Current Version** at the top. @@ -600,7 +613,7 @@ caption="Applications Dashboard: Deployment chart" max-width="30%" %} -#### Monitor CI details by deployment +### Monitor CI details by deployment Each deployment record displays the complete CI history for that deployment. @@ -611,7 +624,7 @@ Each deployment record displays the complete CI history for that deployment. * The **Committer** who made the changes. -#### Monitor updated resources by deployment +### Monitor updated resources by deployment Each deployment record also identifies the resources that were changed (created, updated, or removed) as part of that deployment in **Updated Resources**. You can trace the history of a resource, from the original to their final versions. For each version, you can see the actual change or changes through the Diff view. The Full View shows the complete resource manifest, with the diff view of the changes, while the Compact View shows only those lines with the changes. > For detailed information on the current state of a resource, switch to the Current State tab and click the resource node. See [Monitoring application resources](#monitoring-application-resources). @@ -657,15 +670,15 @@ max-width="70%" -#### Monitor rollouts by deployment +### Monitor rollouts by deployment A rollout is initiated when there is an Argo CD sync due to a change in the desired state. Visualize ongoing and completed rollouts by deployments in **Services**. -> To view and manage a rollout, you must have an Argo `rollout` resource defined for your application, and [install Argo Rollouts in the cluster]({site.baseurl}}/docs/_docs/deployment/install-argo-rollouts). +> To view and manage a rollout, you must have an Argo `rollout` resource defined for your application, and [install Argo Rollouts in the cluster]({site.baseurl}}/docs/deployments/gitops/install-argo-rollouts). For detailed information on Argo Rollouts, see [Argo Rollouts documentation](https://argoproj.github.io/argo-rollouts/){:target="\_blank"}. -##### Rollout progress +#### Rollout progress For an ongoing rollout, the rollout bar displays the progress of the rollout. You can also visualize the steps in the rollout, and control the rollout using the options in the Rollout Player. Here is an example of an ongoing rollout for a canary deployment in Updated Services. The rollout comprising four steps has not started, and no traffic has not been routed as yet to the new version of the application. @@ -693,7 +706,7 @@ caption="Rollout completed for deployment" max-width="50%" %} -##### Manage ongoing rollout +#### Manage ongoing rollout Click the rollout name to visualize its steps. Manually manage the rollout through the controls in the Rollout Player. Here you can see that two out of four steps have been completed, 25% of the traffic has been routed, and the rollout has been paused for the defined length of time. @@ -720,7 +733,7 @@ The table lists the controls in the Rollout Player to manage an ongoing rollout. -##### View analysis run +#### View analysis run If you have defined an analysis template for the rollout, you can check the run results and the manifest. The result of an analysis run determines if the rollout is completed, paused, or aborted. For detailed information, see the [Analysis section in Argo Rollouts](https://argoproj.github.io/argo-rollouts/features/analysis/){:target="\_blank"}. @@ -749,7 +762,7 @@ max-width="50%" %} -### Monitor services for selected application +## Monitor services for selected application The Services tab shows the K8s services for each deployment of the application. Each service shows the number of replicas, the endpoint IP, the labels that reference the application, and the health status. @@ -765,6 +778,11 @@ caption="Applications Dashboard: Services tab" max-width="50%" %} +## Related articles +[Creating GitOps applications]({{site.baseurl}}/docs/deployments/gitops/create-application) +[Managing GitOps applications]({{site.baseurl}}/docs/deployments/gitops/manage-applications) +[Home dashboard]({{site.baseurl}}/docs/reporting/home-dashboard) +[DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics/) diff --git a/_docs/deployment/create-application.md b/_docs/deployments/gitops/create-application.md similarity index 96% rename from _docs/deployment/create-application.md rename to _docs/deployments/gitops/create-application.md index 47aea2d7..27425172 100644 --- a/_docs/deployment/create-application.md +++ b/_docs/deployments/gitops/create-application.md @@ -1,7 +1,8 @@ --- -title: "Creating applications" +title: "Creating GitOps applications" description: "" -group: deployment +group: deployments +sub_group: gitops toc: true --- @@ -19,7 +20,7 @@ Codefresh provides all the options and functionality to create and manage Argo C * Edit and delete applications Once the application is created and synced to the cluster, it is displayed in the Applications dashboard. Here, you can select an application to update the application's configuration settings, or delete it. - To monitor the health and sync status, deployments, and resources for the application, see [Monitoring applications]({{site.baseurl}}/docs/deployment/applications-dashboard/). + To monitor the health and sync status, deployments, and resources for the application, see [Monitoring GitOps applications]({{site.baseurl}}/docs/deployments/gitops/applications-dashboard/). ### Application: Definitions Application definitions include the name, runtime, and the name of the YAML manifest. By default, the YAML manifest has the same name as that of the application. @@ -225,7 +226,7 @@ Track the application in the [Applications dashboard](https://g.codefresh.io/2.0 ### Related articles -[Monitoring applications]({{site.baseurl}})/docs/deployment/applications-dashboard) -[Managing applications]({{site.baseurl}})/docs/deployment/manage-applications) -[Home dashboard]({{site.baseurl}})docs/reporting/home-dashboard) -[DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics/) \ No newline at end of file +[Monitoring GitOps applications]({{site.baseurl}})/docs/deployments/gitops/applications-dashboard) +[Managing GitOps applications]({{site.baseurl}})/docs/deployments/gitops/manage-applications) +[Home dashboard]({{site.baseurl}}/docs/reporting/home-dashboard) +[DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics/) \ No newline at end of file diff --git a/_docs/deployment/images.md b/_docs/deployments/gitops/images.md similarity index 92% rename from _docs/deployment/images.md rename to _docs/deployments/gitops/images.md index d4538a52..80984226 100644 --- a/_docs/deployment/images.md +++ b/_docs/deployments/gitops/images.md @@ -1,7 +1,8 @@ --- title: "Images in Codefresh" description: "" -group: deployment +group: deployments +sub_group: gitops toc: true --- @@ -18,7 +19,7 @@ Complete the mandatory steps to see your Images in the Codefresh UI. Each step h 1. (Mandatory) Report image information to Codefresh. See the [report-image-info](https://github.com/codefresh-io/argo-hub/blob/main/workflows/codefresh-csdp/versions/0.0.6/docs/report-image-info.md){:target="\_blank"} example. -> If you are using an external GitHub Actions-based pipeline, we have a new template that combines image reporting and enrichment. See [Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/). +> If you are using an external GitHub Actions-based pipeline, we have a new template that combines image reporting and enrichment. See [Image enrichment with integrations]({{site.baseurl}}/docs/integrations/gitops/image-enrichment-overview/). ### Image views in Codefresh * In the Codefresh UI, go to [Images](https://g.codefresh.io/2.0/images){:target="\_blank"}. @@ -111,3 +112,10 @@ Selecting **more details** for an image tag. | **3** | The Git details for this image tag, such as the Git hash, the Jira issue number, Git Pull Request, commit information, the name of the user who performed the commit. | | **4** | The workflow for the image step. Select to go to the Workflow.| | **5** | The log information for the build image step in the relevant workflow. Select to view Logs panel. | + +## Related articles + +[Creating GitOps applications]({{site.baseurl}}/docs/deployments/gitops/create-application) +[Managing GitOps applications]({{site.baseurl}}/docs/deployments/gitops/manage-applications) +[Image enrichment with integrations]({{site.baseurl}}/integrations/image-enrichment-overview) +[Home dashboard]({{site.baseurl}}/docs/reporting/home-dashboard) diff --git a/_docs/deployment/install-argo-rollouts.md b/_docs/deployments/gitops/install-argo-rollouts.md similarity index 74% rename from _docs/deployment/install-argo-rollouts.md rename to _docs/deployments/gitops/install-argo-rollouts.md index 22f6afb7..0847b58c 100644 --- a/_docs/deployment/install-argo-rollouts.md +++ b/_docs/deployments/gitops/install-argo-rollouts.md @@ -1,12 +1,13 @@ --- -title: "Install Argo Rollouts" +title: "Progressive delivery with GitOps" description: "" -group: deployment +group: deployments +sub_group: gitops toc: true --- -Install Argo Rollouts on managed clusters with a single click. With Argo Rollouts installed on your cluster, you can visualize rollout progress for deployed applications in the [Applications dashboard]({{site.baseurl}}/docs/deployment/applications-dashboard/#rollout-progress-visualization). +Install Argo Rollouts on managed clusters with a single click. With Argo Rollouts installed on your cluster, you can visualize rollout progress for deployed applications in the [Applications dashboard]({{site.baseurl}}/docs/deployments/gitops/applications-dashboard/#rollout-progress-visualization). If Argo Rollouts has not been installed, an **Install Argo Rollouts** button is displayed on selecting the managed cluster. 1. In the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. @@ -24,4 +25,4 @@ If Argo Rollouts has not been installed, an **Install Argo Rollouts** button is %} ### Related articles -[Add external clusters to runtimes]({{site.baseurl}}/docs/runtime/managed-cluster/) \ No newline at end of file +[Add external clusters to runtimes]({{site.baseurl}}/docs/installation/managed-cluster/) \ No newline at end of file diff --git a/_docs/deployment/manage-application.md b/_docs/deployments/gitops/manage-application.md similarity index 96% rename from _docs/deployment/manage-application.md rename to _docs/deployments/gitops/manage-application.md index 2ccd3e16..15a0049d 100644 --- a/_docs/deployment/manage-application.md +++ b/_docs/deployments/gitops/manage-application.md @@ -1,7 +1,8 @@ --- -title: "Managing applications" +title: "Managing GitOps applications" description: "" -group: deployment +group: deployments +sub_group: gitops toc: true --- @@ -49,8 +50,8 @@ Update General or Advanced configuration settings for a deployed application thr {:start="3"} 1. Update the **General** or **Advanced** configuration settings as needed: - [General configuration]({{site.baseurl}}/docs/deployment/create-application/#application-general-configuration-settings) - [Advanced configuration]({{site.baseurl}}/docs/deployment/create-application/#application-advanced-configuration-settings) + [General configuration]({{site.baseurl}}/docs/deployments/gitops/create-application/#application-general-configuration-settings) + [Advanced configuration]({{site.baseurl}}/docs/deployments/gitops/create-application/#application-advanced-configuration-settings) When you change a setting, the Commit and Discard Changes buttons are displayed. {% include @@ -218,7 +219,7 @@ For example, if you made changes to `api` resources or `audit` resources, type ` Delete an application from Codefresh. Deleting an application deletes the manifest from the Git repository, and then from the cluster where it is deployed. When deleted from the cluster, the application is removed from the Applications dashboard in Codefresh. >**Prune resources** in the application's General settings determines the scope of the delete action. -When selected, both the application and its resources are deleted. When cleared, only the application is deleted. For more information, review [Sync settings]({{site.baseurl}}/docs/deployment/create-application/#sync-settings). +When selected, both the application and its resources are deleted. When cleared, only the application is deleted. For more information, review [Sync settings]({{site.baseurl}}/docs/deployments/gitops/create-application/#sync-settings). Codefresh warns you of the implication of deleting the selected application in the Delete form. 1. In the Codefresh UI, go to the [Applications dashboard](https://g.codefresh.io/2.0/applications-dashboard/list){:target="\_blank"}. @@ -351,10 +352,9 @@ The table describes the options for the `Rollout` resource. ### Related articles -[Creating applications]({{site.baseurl}}/docs/deployment/create-application) +[Creating GitOps applications]({{site.baseurl}}/docs/deployments/gitops/create-application) [Home dashboard]({{site.baseurl}}/docs/reporting/home-dashboard) -[DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics) - +[DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics) diff --git a/_docs/deployments/helm/custom-helm-uploads.md b/_docs/deployments/helm/custom-helm-uploads.md new file mode 100644 index 00000000..46da0a4d --- /dev/null +++ b/_docs/deployments/helm/custom-helm-uploads.md @@ -0,0 +1,125 @@ +--- +title: "Creating and uploading Helm packages" +description: "Manually create and upload Helm packages" +group: deployments +sub_group: helm +redirect_from: + - /docs/create-helm-artifacts-using-codefresh-pipeline/ +toc: true +--- + +Helm packages are just TAR files. Helm repositories are simple file hierarchies with an extra [index.yaml](https://helm.sh/docs/developing_charts/#the-chart-repository-structure){:target="\_blank"}. +You can run custom commands and manually upload indexes and packages to a Helm repo. + +>This articles shows some non-standard Helm examples. + For the basic use cases, or if you are just getting started with Helm, see our [Helm quick start guide]({{site.baseurl}}/docs/getting-started/helm-quick-start-guide/) and [Using Helm in CI pipelines]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/). + +## Package a Helm chart +Below is an example of a freestyle step in a Codefresh pipeline that packages the Helm chart and then extracts the chart name from the command output. It also saves that package name in an environment variable for later use. + + `YAML` +{% highlight yaml %} +{% raw %} +helm_package: + image: devth/helm + commands: + - cf_export PACKAGE=$(helm package| cut -d " " -f 8) +{% endraw %} +{% endhighlight %} + +The `helm package` command expects a path to an unpacked chart. Replace ` ` in the example with the directory that holds your chart files. Note that this directory must have the same name as the chart name, as per Helm requirements.
+See [Helm package docs](https://github.com/kubernetes/helm/blob/master/docs/helm/helm_package.md){:target="_blank"} and [Helm charts overview](https://github.com/kubernetes/helm/blob/master/docs/charts.md){:target="_blank"} for more information. + +{{site.data.callout.callout_info}} +To use `cf_export`and make the variable available to other steps in the pipeline, see [Variables in pipelines]({{site.baseurl}}/docs/pipelines/variables). +{{site.data.callout.end}} + +## Example 1: Push the chart to GCS based Helm Repository +The first example pushes the packaged chart into a public cloud storage service, like AWS S3, Azure Storage, or Google Cloud Storage. We chose Google Cloud Storage (GCS) for this example. +Our pipeline has three steps: + +{:start="1"} +1. download_index: download the Helm `index.yaml` file from GCS, or create one of it's not there. + +{:start="2"} +2. helm_package_merge: package the chart as described earlier, and also merge the new package into the downloaded `index.yaml` file, using the `helm repo index --merge` command. + +{:start="3"} +3. push_gcs: upload the updated `index.yaml` file and the newly created package to GCS. + + `YAML` +{% highlight yaml %} +{% raw %} +steps: + download_index: + image: appropriate/curl:latest + commands: + - 'curl https://storage.googleapis.com/$GOOGLE_BUCKET_NAME/index.yaml --output ./index.yaml --fail || :' + - '[ ! -f ./index.yaml ] && echo "apiVersion: v1">./index.yaml' + helm_package_merge: + image: devth/helm + commands: + - cf_export PACKAGE=$(helm package| cut -d " " -f 8) + - helm repo index --merge ./index.yaml . + push_gcs: + image: camil/gsutil + commands: + - echo -E $GOOGLE_CREDENTIALS > /gcs-creds.json + - echo -e "[Credentials]\ngs_service_key_file = /gcs-creds.json\n[GSUtil]\ndefault_project_id = $GOOGLE_PROJECT_ID" > /root/.boto + - gsutil cp ./index.yaml gs://$GOOGLE_BUCKET_NAME + - gsutil cp $PACKAGE gs://$GOOGLE_BUCKET_NAME +{% endraw %} +{% endhighlight %} + + +### Environment setup + +This pipeline references some predefined environment variables such as `GOOGLE_BUCKET_NAME`, `GOOGLE_PROJECT_ID` and `GOOGLE_CREDENTIALS`. +For this example, we created a service account with appropriate permissions in Google Cloud, and saved the credentials into `GOOGLE_CREDENTIALS` as a Codefresh Secret.
+For more information, see: +[Authenticating with Google services](https://cloud.google.com/storage/docs/authentication#service_accounts){:target="_blank"}.
+[Codefresh pipeline configuration and secrets](https://codefresh.io/docs/docs/codefresh-yaml/variables/#user-provided-variables){:target="_blank"}. + +## Example 2: Push the chart to Chart Museum +Chart Museum is a Helm repository *server* that has an HTTP API, pluggable backends, authentication, and more. +Read more about [Chart Museum](https://github.com/kubernetes-helm/chartmuseum){:target="_blank"}. + +In this example, we already have a Chart Museum server running, so we'll push the packaged chart to it. + +The steps will be: + +{:start="1"} +1. helm_package: package the chart as described earlier. + +{:start="2"} +2. get_repo_url: In order to avoid hard-coding the repository URL into the pipeline, we will retrieve it from the Codefresh Helm integration. +In this case, we have added our repository with Codefresh as described in [Using external Helml repos in Codefresh pipelines]({{site.baseurl}}/docs/deployments/helm/helm-charts-and-repositories). +Replace `` in the example with the name you gave to your repository when you added it to Codefresh. + +{:start="3"} +3. helm_push: call the Chart Museum HTTP api to just upload the package. Chart Museum will take care of the rest. + + `YAML` +{% highlight yaml %} +{% raw %} +steps: + helm_package: + image: devth/helm + commands: + - cf_export PACKAGE=$(helm package | cut -d " " -f 8) + get_repo_url: + image: codefresh/cli:latest + commands: + - cf_export HELM_URL=$(codefresh get ctx -o=yaml | grep repositoryUrl | cut -d "'" -f 2) + helm_push: + image: appropriate/curl + commands: + - curl --data-binary "@$PACKAGE" $HELM_URL/api/charts +{% endraw %} +{% endhighlight %} + + +## Related articles +[Using Helm in a Codefresh pipeline]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/) +[Using a managed Helm repository]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/) +[Helm environment promotion]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion) diff --git a/_docs/deployments/helm/helm-charts-and-repositories.md b/_docs/deployments/helm/helm-charts-and-repositories.md new file mode 100644 index 00000000..8edfe049 --- /dev/null +++ b/_docs/deployments/helm/helm-charts-and-repositories.md @@ -0,0 +1,111 @@ +--- +title: "Using external Helml repos in Codefresh pipelines" +description: "Use external Helm Charts and repositories in Codefresh pipelines" +group: deployments +sub_group: helm +toc: true +--- +Codefresh allows you to integrate with external Helm repositories and Helm charts in the Helm Charts page. +It is optional to use external Helm repositories as all Codefresh accounts already include a [built-in Helm repository]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/). + +## Add an external Helm repository + +Easily add your own Helm charts. +By default, we show charts from the [official Helm repository](https://github.com/kubernetes/charts){:target="_blank"}. + +1. In the Codefresh UI, from the Artifacts section in the sidebar, select [**Helm Charts**](https://g.codefresh.io/helm/releases/releasesNew/){:target="\_blank"}. +1. On the top right, click **Add Existing Helm Repository**. + You are taken to Pipeline Integrations. +1. In the Integrations page, click **Add Helm Repository**, and then select the type of Helm repo to add from the list. +1. Enter the **Helm repository name** and **URL**. + Do not include the specific path to `index.yaml` in the URL. + +{% include image.html +lightbox="true" +file="/images/deployments/helm/quick-helm-integration.png" +url="/images/deployments/helm/quick-helm-integration.png" +alt="Adding a Helm repository" +caption="Adding a Helm repository" +max-width="70%" +%} + +1. If your repository doesn't require authentication, to complete the process, click **Save**. + +For more details on adding Helm repositories, see [Helm integrations]({{site.baseurl}}/docs/integrations/helm/). + +## Use a Helm repository in a Codefresh pipeline + +Once connected, inject any Helm repository context into Codefresh pipelines. + +1. From the Pipelines page, select the pipeline into which to import the Helm configuation. +1. In the Workflows tab, do one of the following: + * Click **Variables** on the right, and then click the **Settings** (gear) icon. + * Click the context menu next to the settings icon. +1. Click on **Import from/Add shared configuration**, and select the name of the repository. + The repository settings are injected as environment variables into the pipeline. + +{% include image.html +lightbox="true" +file="/images/deployments/helm/connect-helm-repo.png" +url="/images/deployments/helm/connect-helm-repo.png" +alt="Connecting a Helm repository in the pipeline" +caption="Connecting a Helm repository in the pipeline" +max-width="70%" +%} + +1. If you are using the Helm step, the step uses these settings to connect to your authenticated repository automatically. For details, see [Using Helm in Codefresh pipelines]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/). + +## Install a chart from your Helm repository +Install a chart from a Helm repository to your cluster. + +* Values in the Chart Install wizard are provided in the following order: + 1. Chart Default Values (implicitly part of the chart). + 2. Overridden default values (provided as values file, provided only if edited by the user). + 3. Supplied values files from Yaml Shared Configuration. + 4. Override variables are provided as `--set` arguments. +* Variables available for custom pipelines: + If you select a custom pipeline, the following variables are available: + * `CF_HELM_RELEASE` - name of release + * `CF_HELM_KUBE_CONTEXT` - kubectl context name of target cluster (cluster name from [dashboard]({{site.baseurl}}/docs/deploy-to-kubernetes/manage-kubernetes/#work-with-your-services)) + * `CF_HELM_INSTALLATION_NAMESPACE` - desired namespace for the release + * `CF_HELM_CHART_VERSION` - Chart Version, + * `CF_HELM_CHART_NAME` - Chart Name + * `CF_HELM_CONTEXTS` - values from [shared configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/#using-shared-helm-values) + * `CF_HELM_VALUES` - extra values + * `CF_HELM_SET` - extra values, + * `CF_HELM_CHART_REPO_URL` - URL of Chart repository + * `CF_HELM_COMMIT_MESSAGE` - Message to show in Helm GUI, + +
+ +**Before you begin** +* Make sure tht you have a Kubernetes integration with the cluster and namespace, as described [here]({{site.baseurl}}/docs/deploy-to-kubernetes/add-kubernetes-cluster/) + +**How to** +1. In the Codefresh UI, from the Artifacts section in the sidebar, select [**Helm Charts**](https://g.codefresh.io/helm/releases/releasesNew/){:target="\_blank"}. +1. In the row with the chart to install, click **Install**. +1. Enter the **Release name** for the chart, and select the **Chart version** to install. +1. From Cluster Information, select a Kubernetes **Cluster** and the **Namespace** to install to. +1. Select the **Pipeline** to install to. +1. If required, edit the **Default Chart Values** to view and override them. + When the default values yaml is changed, it is provided to Helm install as a values file. You can revert to the original values cby clicking Revert. +1. To provide additional values files, do the following: + * From the **Import from configuration** list, select **Add new context of type: YAML**. + * Enter the **Context name**. + * Insert your values YAML, and click **Save**. + The YAML is saved and added to the list of configuration files that you can import from. +1. To override variable values, click **+Add variable**, and then enter the Key and Value. + > The order of value configurations matter for Helm: most recently provided values override earlier ones. +1. Click **Install**. You can observe the newly installed release in Helm Releases. + +You can also install Helm releases from [any Helm environment board]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion). + + +## Related articles +[Using Helm in a Codefresh pipeline]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/) +[Helm integrations]({{site.baseurl}}/docs/integrations/helm/) +[Helm Dashboard]({{site.baseurl}}/docs/deployments/helm/helm-releases-management) +[Helm Promotion boards]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion) +[Helm best practices]({{site.baseurl}}/docs/ci-cd-guides/helm-best-practices/) + + diff --git a/_docs/deployments/helm/helm-environment-promotion.md b/_docs/deployments/helm/helm-environment-promotion.md new file mode 100644 index 00000000..21466e5d --- /dev/null +++ b/_docs/deployments/helm/helm-environment-promotion.md @@ -0,0 +1,290 @@ +--- + +title: "Promoting Helm Environments" +description: "Manage your Helm Environments with the Codefresh UI" +group: deployments +sub_group: helm +toc: true +--- +Apart from the [Helm Releases]({{site.baseurl}}/docs/deployments/helm/helm-releases-management) that show your Kubernetes clusters at the application level, Codefresh also comes with a special environment board that allows you to track one or more applications as they move within your infrastructure (example, Dev, QA, Prod). + +The environment board can function both as an overview of the whole lifecycle of the application, as well as a tool to shift-left/right Helm releases between environments. + +Here is an example board: + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/board.png" +url="/images/deployments/helm/promotion/board.png" +alt="Helm Environment Dashboard" +caption="Helm Environment Dashboard" +max-width="80%" +%} + +This board has three environments that correspond to Kubernetes clusters: + * A Load-testing environment where applications are stress-tested + * A Staging environment where smoke tests are performed + * The Production environment where applications go live + +You can see that a Python example app at version 0.2.0 is already in production. Version 0.3.0 is waiting in the staging environment for smoke tests. Once it is tested it can be dragged to the production column therefore *promoting* it to production status. + + +## Using the Helm Environment Board + +You can create and manage as many Helm promotion boards as you want. +For each board, you define how many columns it will contain, where each column is a Helm-enabled Kubernetes cluster. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/helm-environments.png" +url="/images/deployments/helm/promotion/helm-environments.png" +alt="Helm environments column structure" +caption="Helm environments column structure" +max-width="80%" +%} + +You can use different clusters for each column or different namespaces from the same cluster. You can even mix and match both approaches. +As an example, you could create a Helm board with the following environments: + +* Column 1, dev cluster showing all namespaces (DEV) +* Column 2, namespace qa from cluster staging (QA) +* Column 3, namespace staging from cluster staging (STAGING) +* Column 4, namespace production from cluster prod (PRODUCTION) + +Once you have your columns in place, you can move Helm releases between clusters/namespaces by drag-n-drop. Each Helm release can be dragged to any other column either promoting it, for example, from QA to Production, or shifting it left, for example, from Production to QA. + +## Creating a custom Helm Board + +Create your own Helm board with a single or multiple Helm applications. You can create as many boards as you want. + +1. In the Codefresh UI, from the DevOps Insights section in the sidebar, select [**Helm Boards**](https://g.codefresh.io/helm/helm-kanban/){:target="\_blank"}. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/board-selection.png" +url="/images/deployments/helm/promotion/board-selection.png" +alt="Helm board selection" +caption="Helm board selection" +max-width="80%" +%} + +{:start="2"} +1. On the top-right, click **Add board**. +1. Enter the title of your board as the **Board Name**. +1. Optional. In the **Release name regex expression** field, enter the Regex expression for this board to filter all its environments to show only Helm releases that match this regular expression. + Regex expressions are very helpful if you want your environment board to focus only on a single or set of Helm applications. + To see all Helm releases of your clusters, leave empty. + +You can edit both options for an existing board if you change your mind later. + +### Define Clusters/Namespaces for each Environment + +Once you create your Helm environment board, you are ready to define its columns. + +* To add a column, on the top-right, click **Add environment***. + You will see the environment details dialog: + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/edit-helm-environment.png" +url="/images/deployments/helm/promotion/edit-helm-environment.png" +alt="Edit Helm environment" +caption="Edit Helm environment" +max-width="50%" +%} + + For each environment you can select: + * A name for that column + * The Kubernetes cluster it corresponds to + * One or more namespaces that define this environment (You can even toggle the switch for a regex match) + * A custom pipeline that will be used when a Helm release is installed for the first time in this column + * A custom pipeline that will be used when a Helm release is dragged in this column (promoted from another column) + * Optional. One or more charts to use for the environment. Defining charts for the environment saves you from having to search through all the charts in your Helm repository. When you install an application from the install graphical dialog, only the selected chart(s) are displayed. + * A presentation color to easily identify the environment on the board (For example, a "production" environment should have a red color) + +You can also select no namespace at all. In that case, the column will show Helm releases for all namespaces in that cluster. +You can change all these options after creation, so feel free to change your mind. + +Repeat the same process for additional environments. Remember that you can name your environment as you want and define any combination of cluster/namespace for any of the columns. This gives you a lot of power to define a Helm environment board that matches exactly your own process. + +You don't have to define the environments in order. You can drag-n-drop columns to change their order after the initial creation. + + +### Installing Helm Releases on each Environment + +If you already have [pipelines that deploy Helm releases]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/), your columns are populated automatically with information. + +For each Helm release, you will get some basic details such as the chart version and the name of the release. You can expand a release by clicking on the arrow button to get additional information such as the docker images and the replicas of each pod that are contained in the release. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/expand.png" +url="/images/deployments/helm/promotion/expand.png" +alt="Helm release details" +caption="Helm release details" +max-width="50%" +%} + +You can even install manually a Helm release from any external repository by clicking on the *PLUS* button at the header of each column. In that case you will see a list of possible Helm applications to choose from. + +You will be able to select the target cluster and namespace as well as the chart values [as any other Helm release]({{site.baseurl}}/docs/deployments/helm/helm-charts-and-repositories/#install-chart-from-your-helm-repository). + + +## Moving Releases between Environments + +A Helm environment board can be used by different stakeholders in order to get the detailed status of all defined environments. In that aspect it can act as a read-only tool that simply shows the results of Codefresh pipelines that deploy Helm applications. + +### Promoting Helm Releases with the UI + +You can also use the board as an action tool in order to promote/demote a Helm release between individual environments. To move a Helm release between environments just drag-n-drop it to a different column. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/shift-right.png" +url="/images/deployments/helm/promotion/shift-right.png" +alt="Promoting a Helm release" +caption="Promoting a Helm release" +max-width="80%" +%} + +Once you drop the release you will also see the promotion dialog. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/promote-settings.png" +url="/images/deployments/helm/promotion/promote-settings.png" +alt="Promotion Settings" +caption="Promotion Settings" +max-width="40%" +%} + +All fields here will be auto-filled according to the Helm release that you dragged. You can also choose a custom pipeline (see below) for the promotion if you don't want to use the default one. + +By clicking the *Variables* button you can override the chart values, import a specific shared configuration or add new values. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/value-options.png" +url="/images/deployments/helm/promotion/value-options.png" +alt="Changing deployment values" +caption="Changing deployment values" +max-width="40%" +%} + +By default Codefresh will use a built-in install/upgrade pipeline for performing the promotion. You can choose your own pipeline from the promotion dialog. That pipeline will be automatically provided with the following [environment variables]({{site.baseurl}}/docs/deployments/helm/helm-releases-management/#overriding-the-default-helm-actions): + +* `CF_HELM_RELEASE` - name of release +* `CF_HELM_KUBE_CONTEXT` - `kubectl` context name of target cluster (cluster name from [dashboard]({{site.baseurl}}/docs/deploy-to-kubernetes/manage-kubernetes/#work-with-your-services)) +* `CF_HELM_NAMESPACE` - Tiller Namespace if you use Helm 2 +* `CF_HELM_INSTALLATION_NAMESPACE` - namespace where release is promoted to +* `CF_HELM_CONTEXTS` - [shared configuration]({{site.baseurl}}/docs/configure-ci-cd-pipeline/shared-configuration) Helm contexts +* `CF_HELM_VALUES` - Helm chart values +* `CF_HELM_SET` - Additional values there were overriden +* `CF_HELM_CHART_JSON_GZIP` - Gzipped JSON of Helm chart (only for Helm 3) +* `CF_HELM_CHART_JSON` - JSON of Helm chart (only for Helm 2) +* `CF_HELM_BOARD` - Name of the board that is used for the drag-n-drop-action +* `CF_HELM_TARGET_SECTION` - Name of the Source Environment that you are promoting from +* `CF_HELM_SOURCE_SECTION` - Name of the Target Environment that you are promoting to + + +Note that the variable `CF_HELM_CHART_JSON_GZIP` is both compressed and base64 encoded. To get the raw value you need a command like `echo $CF_HELM_CHART_JSON_GZIP | base64 -d | gunzip` + +>Overriding the default pipeline can only be done by [Codefresh admin users]({{site.baseurl}}/docs/administration/access-control/#users-and-administrators). + +Once you click the *update* button, a new build will run that will perform the deployment. + +Note that you can move releases to any column both on the right and on the left of the current column. This is helpful if for example you find a bug in your production environment and you want to bring it back to a staging environment for debugging. + +### Promoting Helm releases programmatically + +You can also promote Helm releases with the [Codefresh CLI](https://codefresh-io.github.io/cli/predefined-pipelines/promote-helm-release/){:target="\_blank"}. + +Once you have [installed the CLI](https://codefresh-io.github.io/cli/getting-started/){:target="\_blank"}, you can use it from an external script or terminal with the `helm-promotion` parameter: + +{% highlight shell %} +{% raw %} +codefresh helm-promotion --board MySampleBoard --source Staging --target Production --source-release my-app --set myenv=prod +{% endraw %} +{% endhighlight %} + +Here we promote the Helm release `my-app` to the *Production* column overriding also the `myenv` value. + +Remember that the Codefresh CLI can also run in a Codefresh pipeline with a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). +Here is an example of a Helm promotion from within a Codefresh pipeline. + + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + triggerstep: + title: trigger + image: codefresh/cli + commands: + - 'codefresh helm-promotion --board MySampleBoard --source Staging --target Production --source-release my-app --namespace my-namespace --set myenv=prod' +{% endraw %} +{% endhighlight %} + +## Viewing the promotion pipeline + +When you promote a Helm Release for a Board, you can view the pipeline for that release. + +1. Click on Boards under the Helm section on the left-hand side +2. Select the board you want to view +3. Select the Builds tab on the top +4. Here, you can see the Promotion Pipelines / builds for promoting a Release + +## Editing your Helm Boards + +For any existing Helm board, you have the following options: + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/board-management.png" +url="/images/deployments/helm/promotion/board-management.png" +alt="Editing a Helm environment" +caption="Editing a Helm environment" +max-width="80%" +%} + + +1. The refresh button will update the board with the current state of the clusters +1. The filtering menu can be used to further constrain the Helm releases shown in each column. +1. The *edit properties* button allows you to change again the title of the board as well as a global filter for Helm releases +1. The *remove board* completely deletes the present board from the Codefresh UI +1. The environment details on the environment header are: +* The edit button to change again the options for this column (shown on mouse hover) +* The delete button to remove this column from the board (shown on mouse hover) +* The plus button to install a new chart. If you selected one or more charts when you defined your environment, only the selected charts are displayed. +* A numeric value that shows how many releases are contained on this environment +1. The delete button allows you to uninstall a Helm release for an environment + +The filtering options allow you to further constrain the Helm release shown for the whole board. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/filter.png" +url="/images/deployments/helm/promotion/filter.png" +alt="Filtering options" +caption="Filtering options" +max-width="50%" +%} + +The filters are especially helpful in Helm boards with large numbers of environments and/or releases. + +## Related articles +[Using Helm in a Codefresh pipeline]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/) +[Using external Helml repos in Codefresh pipelines]({{site.baseurl}}/docs/deployments/helm/helm-charts-and-repositories/#add-helm-repository) +[Managing Helm releases]({{site.baseurl}}/docs/deployments/helm/helm-releases-management) +[Environment Dashboard]({{site.baseurl}}/docs/deploy-to-kubernetes/environment-dashboard/) diff --git a/_docs/deployments/helm/helm-releases-management.md b/_docs/deployments/helm/helm-releases-management.md new file mode 100644 index 00000000..991701f7 --- /dev/null +++ b/_docs/deployments/helm/helm-releases-management.md @@ -0,0 +1,263 @@ +--- +title: "Managing Helm releases" +description: "Manage Helm deployments from the Codefresh UI" +group: deployments +sub_group: helm +redirect_from: + - /docs/helm-releases-management/ + - /docs/deployments/helm/helm3/ +toc: true +--- +Codefresh has built-in integration for Helm that provides a unique view into your production Kubernetes cluster. +In Helm Releases, you can see the current status of your cluster, including the currently deployed releases, their previous revisions including change tracking, and even roll back to a previous release. + +Codefresh also offers [an environment view for Helm releases]({{site.baseurl}}/docs/deploy-to-kubernetes/environment-dashboard/) as well as [a promotion dashboard]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion). + + +## View Helm releases and release information + +View all the Helm releases in your cluster, and drill down into a specific release to see its services, deployed versions, manifests and more. + +> Make sure you have [connected your Kubernetes cluster]({{site.baseurl}}/docs/integrations/kubernetes/adding-non-gke-kubernetes-cluster/) to Codefresh. + +1. In the Codefresh UI, from the DevOps Insights section in the sidebar, select [**Helm Releases**](https://g.codefresh.io/helm/releases/releasesNew/){:target="\_blank"}. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/dashboard/helm-release-dashboard.png" +url="/images/deployments/helm/dashboard/helm-release-dashboard.png" +alt="Helm Releases" +caption="Helm Releases" +max-width="90%" +%} + + + + +{:start="2"} +1. To see details for a specific release, click the release name. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/dashboard/services.png" +url="/images/deployments/helm/dashboard/services.png" +alt="Kubernetes Services" +caption="Kubernetes Services" +max-width="70%" +%} + +The History tab shows all previous releases. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/dashboard/history.png" +url="/images/deployments/helm/dashboard/history.png" +alt="Helm History" +caption="Helm History" +max-width="60%" +%} + +You can further expand a release revision to see exactly what files were changed in this release. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/dashboard/diff.png" +url="/images/deployments/helm/dashboard/diff.png" +alt="Helm diff" +caption="Helm diff" +max-width="60%" +%} + +There are other tabs that show you the chart used, the values as well as the final manifests that were actually deployed. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/dashboard/manifests.png" +url="/images/deployments/helm/dashboard/manifests.png" +alt="Final rendered manifests" +caption="Final rendered manifests" +max-width="50%" +%} + +## Add labels to Kubernetes services + +For better visibility into services, add the [recommended labels](https://helm.sh/docs/topics/chart_best_practices/labels/){:target="\_blank"} to your Kubernetes service. + +{% highlight yaml %} +{% raw %} + apiVersion: v1 +kind: Service +metadata: + name: {{ template "fullname" . }} + labels: + app.kubernetes.io/name: "{{ template "name" . }}" + helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}" + app.kubernetes.io/managed-by: "{{ .Release.Service }}" + app.kubernetes.io/instance: "{{ .Release.Name }}" +{% endraw %} +{% endhighlight %} + +To use the instance label for something different, you can also use a release label instead: + +{% highlight yaml %} +{% raw %} +release: {{ .Release.Name }} +{% endraw %} +{% endhighlight %} + + + +## Add an upgrade message + +Codefresh allows you to display a meaningful description for each release in the release history. This message +can help show the main reason behind each release, or any other message that is convenient for you. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/dashboard/helm-commit-message.png" +url="/images/deployments/helm/dashboard/helm-commit-message.png" +alt="Helm release message" +caption="Helm release message" +max-width="70%" +%} + +You can set this message for your Helm release in three ways: + +1. When you manually install a Helm release from the [Helm charts screen]({{site.baseurl}}/docs/deployments/helm/helm-charts-and-repositories/#install-chart-from-your-helm-repository), there is a field for this message. +1. Set the property `commit_message` inside the [notes.txt](https://helm.sh/docs/chart_template_guide/notes_files/){:target="\_blank"} file of your chart. +1. By providing an environment variable called `COMMIT_MESSAGE` within your [pipeline Helm step]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/). + + +## Roll back a Helm release + +You can rollback to a previous revision of a release in the History tab. + +1. Click the Helm release for which to perform a rollback, and then click the **History** tab. +1. To rollback to a specific release, click **Rollback** in the row. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/dashboard/rollback.png" +url="/images/deployments/helm/dashboard/rollback.png" +alt="Rolling back to a previous release" +caption="Rolling back to a previous release" +max-width="50%" +%} + +>It takes time to complete rollback for a release, and the change in the cluster is not instantly updated in the Codefresh UI. If you also use a [custom rollback pipeline](#overriding-the-default-helm-actions), the delay between the cluster update and the UI refresh is even longer. + +## Helm UI actions + +From the main release screen, you have some additional actions. + +You can issue a [Helm test](https://github.com/kubernetes/helm/blob/master/docs/chart_tests.md) by clicking on the 'Run Test' button on the desired chart row. + +You can delete a release by clicking on the 'Delete' button on the desired chart row. +For deletion options, see the [helm delete documentation](https://github.com/kubernetes/helm/blob/master/docs/helm/helm_delete.md){:target="\_blank"}, for example, *purge* will remove the revision from the release history. + +## Helm deployment badge + +Similar to a [build badge]({{site.baseurl}}/docs/pipelines/build-status/#using-the-build-badge), you can also get a deployment badge for a Helm release. + +1. In the Codefresh UI, from the DevOps Insights section in the sidebar, select [**Helm Releases**](https://g.codefresh.io/helm/releases/releasesNew/){:target="\_blank"}. +1. In the row with the Helm release for which to add a deployment badge, click the **Settings** (gear) icon. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/dashboard/helm-badge.png" +url="/images/deployments/helm/dashboard/helm-badge.png" +alt="Helm Deployment badge" +caption="Helm Deployment badge" +max-width="60%" +%} + +{:start="3"} +1. To get deployment information, click **Badge**. + Codefresh provides the Markdown/HTML/Link segment that you can embed in README or other documents to show deployment information. + +## Overriding default Helm actions for releases + +By default, when you take an action in the UI, Codefresh executes the native Helm command corresponding to that action: + +* `helm test` for testing a chart +* `helm rollback` for rollbacks +* `helm delete` or `helm uninstall --keep-history` for delete +* `helm delete --purge ` or `helm uninstall ` for purging a release + +You can override these actions for a specific Helm release by defining custom pipelines for each action. This way you can add your extra logic on top of these actions. For example your own Helm uninstall pipeline might also have a notification step that posts a message to a Slack channel after a release is removed. + +>Only [Codefresh admin users]({{site.baseurl}}/docs/administration/access-control/#users-and-administrators) can override the default pipelines defined for a Helm release. + +1. In the Codefresh UI, from the DevOps Insights section in the sidebar, select [**Helm Releases**](https://g.codefresh.io/helm/releases/releasesNew/){:target="\_blank"}. +1. In the row with the Helm release for which to override default actions, click the **Settings** (gear) icon. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/dashboard/override-helm-actions.png" +url="/images/deployments/helm/dashboard/override-helm-actions.png" +alt="Changing default Helm actions" +caption="Changing default Helm actions" +max-width="50%" +%} + +{:start="3"} +1. Select the pipeline to use for the respective actions. + +### Environment variables for custom Helm commands +If you do override any of these actions, the following [environment variables]({{site.baseurl}}/docs/codefresh-yaml/variables/) are available in the respective pipeline, so that you can use your own custom Helm command. + +**Helm Test pipeline** +* `CF_HELM_RELEASE`: Name of release +* `CF_HELM_KUBE_CONTEXT`: `kubectl` context name of target cluster (cluster name from [dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/#work-with-your-services)) +* `CF_HELM_NAMESPACE`: Namespace where release is stored +* `CF_HELM_TIMEOUT`: Time in seconds to wait for any individual Kubernetes operation +* `CF_HELM_CLEANUP`: Delete test pods upon completion + + + +**Helm Rollback pipeline** +* `CF_HELM_VERSION`: Helm version, ex.: 3.0.1, 2.7.0 +* `CF_HELM_RELEASE`: Name of release on cluster +* `CF_HELM_REVISION`: Revision to use for rollback +* `CF_HELM_KUBE_CONTEXT`: `kubectl` context name of target cluster (cluster name from [dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/#work-with-your-services)) +* `CF_HELM_NAMESPACE`: Namespace where release is stored + + +**Helm Delete pipeline** +* `CF_HELM_PURGE`: Boolean, delete release from store +* `CF_HELM_RELEASE`: Name of release +* `CF_HELM_TIMEOUT`: Time in seconds to wait for any individual Kubernetes operation +* `CF_HELM_HOOKS`: Prevent hooks from running during install +* `CF_HELM_KUBE_CONTEXT`: `kubectl` context name of target cluster (cluster name from [dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/#work-with-your-services)) +* `CF_HELM_VERSION`: Helm version, ex.: 3.0.1, 2.7.0 +* `CF_HELM_NAMESPACE`: Namespace where release is stored + + +## Related articles +[Using Helm in a Codefresh pipeline]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/) +[Helm charts and repositories]({{site.baseurl}}/docs/deployments/helm/helm-charts-and-repositories/) +[Codefresh-managed Helm Repositories]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/) +[Helm promotion boards]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion) \ No newline at end of file diff --git a/_docs/deployments/helm/managed-helm-repository.md b/_docs/deployments/helm/managed-helm-repository.md new file mode 100644 index 00000000..5dcd43b5 --- /dev/null +++ b/_docs/deployments/helm/managed-helm-repository.md @@ -0,0 +1,137 @@ +--- +title: "Using a managed Helm repository" +description: "Use the Codefresh integrated Helm repository" +group: deployments +sub_group: helm +toc: true +--- + +Codefresh provides fully managed, hosted Helm repositories for users. +While we automatically create a default managed repo for every Codefresh account, you can also add [external Helm repositories]({{site.baseurl}}/docs/deployments/helm/helm-charts-and-repositories/). + +The built-in Helm repo that Codefresh creates, is private by default, allowing access only via Codefresh or via a Codefresh API token. + +> Tip: + You may be familiar with the popular open source Helm repository implementation called 'ChartMuseum', that Codefresh sponsors. Codefresh-managed repositories are based on, and therefore compatible with, ChartMuseum and its unique features. For details, see [ChartMuseum](https://github.com/kubernetes-helm/chartmuseum){:target="\_blank"}. + +## View Helm repository integrations + +The Codefresh-managed Helm repo is displayed with other Helm repositories you have added to Helm integrations. + +>You cannot delete the built-in Helm repo that Codefresh creates for you. + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select **Pipeline Integrations**. +1. Scroll to **Helm Repositories**, and then click **Configure**. + All the Helm integrations you set up are displayed. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/private-helm-repo/managed-helm-repo.png" +url="/images/deployments/helm/private-helm-repo/managed-helm-repo.png" +alt="Codefresh built-in Helm repository" +caption="Codefresh built-in Helm repository" +max-width="50%" +%} + + +## Get the chart repository URL +Get the chart repository URL for any Helm integration. +The URL is in the format: `cm://h.cfcr.io// `, where the default repo is `default`. + +* From the list of Helm integrations, select the integration and then click the **Edit** icon on the left. + The Helm Repository URL field displays the chart URL. + +## Codefresh Helm dashboards + +The Helm Charts and Helm Releases dashboards are automatically configured to work with your default managed repo to easily install charts and manage releases. +For more information, see [install chart from a Helm repository]({{site.baseurl}}/docs/deployments/helm/helm-charts-and-repositories/#install-chart-from-your-helm-repository) and [Managing Helm releases]({{site.baseurl}}/docs/deployments/helm/helm-releases-management/). + +## Use Codefresh CLI for advanced Helm repo management + +The Codefresh CLI supports advanced management options for your managed repository, without having to log in to the Codefresh UI. +For more information on CLI support for Helm repos, see the [CLI documentation on Helm Repos](https://codefresh-io.github.io/cli/helm-repos/){:target="\_blank"}. + + +## Set access level for managed repo + +The managed Helm repository supports two modes of access: +* Private +* Public + +By default, the managed Helm repo is created with `Private` access, meaning that read/write access is protected by Codefresh authentication. + +You can switch the access level to `Public`, which will make the repository accessible to anonymous users only *for read operations*. Write operations, even in public access mode, always require authentication. +Be very careful when you make your repo public, as the whole world will be able to access your charts. We recommend this setting only for quick demos and POCs. + +**How to** + +* Use the Codefresh CLI to toggle access level on a managed repo: + +{% highlight bash %} +codefresh patch helm-repo mycfrepo -public +{% endhighlight %} + +For more info, see the relevant section in the [Codefresh CLI documentation](https://codefresh-io.github.io/cli/helm-repos/update-helm-repo/){:target="\_blank"}. + +## Working with Helm CLI + +The private Helm repository offered by Codefresh is a standard Helm repo and will work with the vanilla Helm executable even outside of the Codefresh UI. +We suggest using the private [Helm repo from Codefresh pipelines]({{site.baseurl}}/docs/example-catalog/cd-examples/helm/), but you can also use it from your workstation. + +### Add a Public repo to Helm + +If your repo is set to `public` access mode, you can use it just like any other HTTP Helm repository. +You can: + +{% highlight bash %} +helm repo add mycfrepo https://h.cfcr.io/ / +{% endhighlight %} + +### Add a Private repo to Helm + +If your repo is set to `private` access mode, the default, then the Helm client needs to authenticate with Codefresh. +To authenticate, you can use ChartMuseum's 'Helm Push' CLI plugin which adds support for authentication and chart manipulation on top of the basic Helm CLI functionality. + +We highly recommend that you familiarize yourself with the [Helm Push plugin](https://github.com/chartmuseum/helm-push){:target="\_blank"}. + +#### Install the Helm Push plugin + +{% highlight bash %} +helm plugin install https://github.com/chartmuseum/helm-push +{% endhighlight %} + +#### Configure the Helm Push plugin + +If you have the Codefresh CLI installed and configured, there's nothing you need to do. The Helm Push plugin picks up your settings automatically. +To learn about getting started with Codefresh CLI, see [CLI getting started](https://codefresh-io.github.io/cli/getting-started/). +To learn about manual authentication without depending on the Codefresh CLI, see [here](https://github.com/chartmuseum/helm-push#token). + +#### Add the private repo + +{% highlight bash %} +helm repo add mycfrepo cm://h.cfcr.io/kostis-codefresh/default +{% endhighlight %} + +Notice the protocol is `cm://` instead of `https://`. This indicates the custom authentication scheme supported by ChartMuseum Helm Push plugin. + +## Using in a Codefresh pipeline + +The Codefresh Helm plugin automatically handles authentication for managed repositories. You can use the plugin as you usually would. For more information, see the [Codefresh Helm plugin]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/). + +## Removing a Helm chart from a private Codefresh repository + +You can delete a Helm chart from your own Helm repository with the following HTTP call. + +{% highlight bash %} +curl -X DELETE -v -H "Authorization: Bearer " https://h.cfcr.io/api/ / /charts/ / +{% endhighlight %} + +Replace values in `<>` with your own (also removing `<>` in the process). + +Generate an api key from [https://g.codefresh.io/user/settings](https://g.codefresh.io/user/settings) as explained in the [API page]({{site.baseurl}}/docs/integrations/codefresh-api/). + +## Related articles +[Using Helm in a Codefresh pipeline]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/) +[Helm integration]({{site.baseurl}}/docs/integrations/helm/) +[Managing Helm releases]({{site.baseurl}}/docs/deployments/helm/helm-releases-management) diff --git a/_docs/deployments/helm/using-helm-in-codefresh-pipeline.md b/_docs/deployments/helm/using-helm-in-codefresh-pipeline.md new file mode 100644 index 00000000..f96ddd4f --- /dev/null +++ b/_docs/deployments/helm/using-helm-in-codefresh-pipeline.md @@ -0,0 +1,346 @@ +--- +title: "Using Helm in a Codefresh pipeline" +description: "Deploy and push Helm charts with Codefresh" +group: deployments +sub_group: helm +redirect_from: + - /docs/deployments/helm/create-helm-artifacts-using-codefresh-pipeline/ + - /docs/install-helm-chart-using-codefresh-pipeline/ +toc: true +--- + +We created a [special Helm step](https://codefresh.io/steps/step/helm){:target="\_blank"} for easy integration of Helm in Codefresh pipelines. The Helm step facilitates authentication, configuration, and execution of Helm commands. + +> If you have a special use case that is not covered by the Codefresh Helm step, you can always use the regular `helm` cli in a freestyle step. + In this case, you can use the simpler container `codefresh/kube-helm` which includes only Kubectl and helm tools. `kube-helm` is available on DockerHub: [https://hub.docker.com/r/codefresh/kube-helm/](https://hub.docker.com/r/codefresh/kube-helm/){:target="\_blank"}. + +If you are just starting with Helm, refer to our [Helm quick start guide]({{site.baseurl}}/docs/getting-started/helm-quick-start-guide/). And, if you prefer to work directly with code, see our [full Helm example]({{site.baseurl}}/docs/example-catalog/cd-examples/helm/). + +## Helm setup + + + +To use Helm in your Codefresh pipeline you must do the following: + +1. Make sure that your application has a [Helm chart](https://helm.sh/docs/chart_template_guide/getting_started/) +1. Create a Helm package for your application from the chart +1. [Add a Kubernetes cluster]({{site.baseurl}}/docs/deploy-to-kubernetes/add-kubernetes-cluster/) in Codefresh +1. Define a Helm repository or use the [one offered by Codefresh to all accounts]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/) +1. Import the Helm [configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/) into your pipeline variables +1. Use the Helm step in your [yml build definition]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) + +Let's see these steps in order. + +### Step 1: Create a Helm chart for your application + +Helm applications are bundled in special archives called *Charts*. You can create a Helm +chart for your application by following [the official documentation on charts](https://helm.sh/docs/chart_template_guide/getting_started/){:target="\_blank"}. + +The example Codefresh application includes a [sample chart](https://github.com/codefresh-contrib/python-flask-sample-app/tree/with-helm/charts/python){:target="\_blank"}, used in our Helm quick start guide, mentioned earlier in this article. + +You can create the chart manually or by using the [helm create](https://helm.sh/docs/helm/#helm-create){:target="\_blank"} command on your workstation. There are also several third-party tools that can create Helm packages for you such as [Draft](https://draft.sh/){:target="\_blank"}. + +Once your Helm chart is ready, commit it to a folder called `charts`, in the same Git repository that contains the source code of your application. Codefresh can also work with Helm charts that are in different Git repositories. We suggest however that you keep both the source code and the Helm chart of an application in the same Git repository to make chart management much easier. + + +### Step 2: Select Kubernetes cluster for deployment + +The Helm pipeline step requires the configuration of a `kube_context` variable that determines the Kubernetes cluster used for the deployment. + +1. Connect your Kubernetes cluster with Codefresh, as described [here]({{site.baseurl}}/docs/deploy-to-kubernetes/add-kubernetes-cluster/). + +1. Provide the cluster to the Helm step by adding the `KUBE_CONTEXT` variable, where the value is the connection *name* entered when you created the connection. +> The connection name also appears as the title of the cluster in Kubernetes integration settings (Account Settings >Integrations > Kubernetes). + +{% include image.html +lightbox="true" +file="/images/deployments/helm/k8s-name.png" +url="/images/deployments/helm/k8s-name.png" +alt="Name of Kubernetes cluster" +caption="Name of Kubernetes cluster" +max-width="70%" +%} + +1. Verify that your cluster is set up for Helm, from the sidebar, below DevOps Insights, select **Helm Releases**. + The [Helm releases]({{site.baseurl}}/docs/deployments/helm/helm-releases-management/) in your cluster are displayed. If you have just started using Helm, the release page may be empty. + +### Step 3: Define a Helm repository + +To push your chart to a Helm repository, configure the target repository to work with. +Always a good practice to save Helm charts in Helm repositories, Codefresh supports a variety of private, authenticated Helm repositories +in addition to public HTTP repositories. Codefresh also provides a free, managed Helm repository for every account. + +* Either [connect your repository with Codefresh]({{site.baseurl}}/docs/deployments/helm/add-helm-repository/) +OR +* Obtain your [managed Helm repository URL]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/#chart-repository-url) + + +### Step 4: (Optional) Import the Helm configuration into your pipeline definition + +Once you have a connected to a Helm repository, attach it to the pipeline. + +1. Frpm the Pipelines page, select the pipeline into which to import the Helm configuation. +1. In the Workflows tab, do one of the following: + * Click **Variables** on the right, and then click the Settings (gear) icon in the variables section on the right. + * Click the context menu next to the settings icon. +1. Click on **Import from/Add shared configuration**, and from the list, select `CF_HELM_DEFAULT`. See [shared configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/). + +{% include image.html +lightbox="true" +file="/images/deployments/helm/import-helm-configuration.png" +url="/images/deployments/helm/import-helm-configuration.png" +alt="Connecting a Helm repository in the pipeline" +caption="Connecting a Helm repository in the pipeline" +max-width="50%" +%} + + +### Step 5: Use the Helm freestyle step in the pipeline + +You can now use the Helm freestyle step in the `codefresh.yml` file. This step is only needed in pipelines that actually upload/fetch Helm charts to/from Helm repositories. If your pipeline directly installs a Helm chart from the Git filesystem, there is no need to import a Helm configuration. + +>Currently, you can use only one Helm configuration in the same pipeline. We are aware +of this limitation and will soon improve the way Codefresh works with multiple Helm configurations. + + + +* Use the Helm typed step from the [Step Marketplace](https://codefresh.io/steps/step/helm){:target="\_blank"}. +* Configure the Helm step using environment variables, as described [here]({{site.baseurl}}/docs/codefresh-yaml/variables/#user-provided-variables). + +The example below illustrates how to provide variables as part of the Helm step definition: + +```yaml +deploy: + type: helm + arguments: + action: install + chart_name: test_chart + release_name: first + helm_version: 3.0.3 + kube_context: my-kubernetes-context + custom_values: + - 'pat.arr="{one,two,three}"' + - 'STR_WITH_COMAS="one\,two\,three"' +``` + + + +#### Helm step action modes + +The Helm step can operate in one of three modes, as defined by the `action` field: + +1. `install`: Installs the chart into a Kubernetes cluster. This is the default mode if not explicitly set. +2. `push`: Packages the chart and pushes it to the repository. +3. `auth`: Authenticate only. Only sets up authentication and adds the repo to the Helm. This mode is useful to write your own Helm commands using the freestyle step's `commands` property, but still allow the step to handle authentication. + + +#### Helm values + +* To supply a value file, add to the Helm step, `custom_values_file`, with the value pointing to an existing values file. +* To override specific values, add to the Helm step, `custom_values` followed by the path to the value to set. For example, `myservice_imageTag`. Note that `.` (dot) should be replaced with `_` (underscore). The value of the variable is used to override or set the templated property. + +Examples: +```yaml +... + custom_values: + - 'myimage_pullPolicy=Always' +... +``` +results in: +`--set myimage.pullPolicy=Always` + +```yaml +... + custom_value_files: + - 'values-prod.yaml' +... +``` +results in: +`--values values-prod.yaml` + +If a variable already contains a `_` (underscore) in its name, replace it with `__` (double underscore). + +## Helm usage examples + +The following sections illustrate all three modes of Helm usage. + +You can also look at the [GitHub repository](https://github.com/codefresh-contrib/helm-sample-app){:target="\_blank"} of [our Helm example]({{site.baseurl}}/docs/example-catalog/cd-examples/helm/) for full pipelines: + +* Pipeline YAML for [deploying a chart](https://github.com/codefresh-contrib/helm-sample-app/blob/master/codefresh-do-not-store.yml){:target="\_blank"} +* Pipeline YAML for [both storing and deploying a chart](https://github.com/codefresh-contrib/helm-sample-app/blob/master/codefresh.yml){:target="\_blank"} + +### Helm usage example: Installing a Helm Chart + +The following example includes the minimum configuration to install a Helm chart from a repository. For more configuration options, see the [Arguments reference](https://codefresh.io/steps/step/helm){:target="\_blank"}. + +```yaml +deploy: + type: helm + arguments: + action: install + chart_name: path/to/charts + release_name: first + helm_version: 3.0.3 + kube_context: my-kubernetes-context +``` + +### Helm usage example: Pushing a Helm Chart + +The following example illustrates how to package and push a Helm chart into a repository. + +```yaml +deploy: + type: helm + arguments: + action: push + chart_name: /codefresh/volume/repo/chart + chart_repo_url: 'cm://h.cfcr.io/useraccount/default' +``` + +> **Notes**: + - Assumes that a Git repository with the Helm chart files was cloned as a part of the pipeline. + - The Git repository contains the chart files in the `chart` directory. + - `chart_repo_url` is optional. If a [Helm repository configuration](#step-4-optional-import-the-helm-configuration-in-your-pipeline-definition) is attached to the pipeline, this setting is ignored. + +### Helm usage example: Authenticating only + +The following example illustrates the Helm mode for authentication only. + +```yaml +deploy: + type: helm + arguments: + action: auth + kube_context: my-kubernetes-context + commands: + - helm list +``` + +### Helm usage example: Custom Helm commands + +The following example illustrates executing custom Helm commands. + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +my_custom_helm_command: + type: helm + arguments: + action: auth + kube_context: my-kubernetes-context + commands: + - source /opt/bin/release_chart + - helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/ + - helm repo add stable https://kubernetes-charts.storage.googleapis.com + - helm repo list + - helm repo update + - helm list +{% endraw %} +{% endhighlight %} + +> Notes: +- The directory that contains a chart MUST have the same name as the chart. Thus, a chart named `my-chart` MUST be created in a directory called `my-chart/`. This is a requirement of the [Helm Chart format](https://helm.sh/docs/chart_template_guide/). + +## Helm configuration fields + +Name|Required|Description +---|---|--- +action|Defaults to 'install'|Operation mode: `install`/`push`/`auth` +chart_name|required for install/push|Chart reference to use, adhering to Helm's lookup rules (path to chart folder, or name of packaged chart). There's no need to prefix with `/reponame` if referencing a chart in a repository, this is handled automatically. a.k.a `CHART_NAME` but `CHART_NAME` shouldn't be used anymore. +chart_repo_url|optional|Helm chart repository URL. If a [Helm repository configuration](#step-4-optional---import-the-helm-configuration-in-your-pipeline-definition) is attached to the pipeline, this setting is ignored. +chart_version|optional|Override or set the chart version. +cmd_ps|optional|When defined, Command Postscript is appended as is to the generated Helm command string. Can be used to set additional parameters supported by the command but not exposed as configuration options.| +commands|optional|Commands to execute in plugin after `auth` action. +custom_value_files|optional|Values file to provide to Helm as `--values` or `-f`.| +custom_values|optional|Values to provide to Helm as `--set` +helm_version|optional|Version of [cfstep-helm image](https://hub.docker.com/r/codefresh/cfstep-helm/tags){:target="\_blank"} +kube_context|required for install|Kubernetes context to use. The name of the cluster as [configured in Codefresh]({{site.baseurl}}/docs/deploy-to-kubernetes/add-kubernetes-cluster/). +namespace|optional|Target Kubernetes namespace to deploy to. +release_name|required for install|Helm release name. If the release exists, it is upgraded. +repos|optional|Array of custom repositories. + + +## Full Helm pipeline example + +The pipeline in this example builds a docker image, runs unit tests, stores the Helm chart in the Codefresh private Helm repository and finally deploys the Helm chart to a cluster. + +{% include image.html +lightbox="true" +file="/images/deployments/helm/full-helm-pipeline.png" +url="/images/deployments/helm/full-helm-pipeline.png" +alt="Helm pipeline" +caption="Helm pipeline" +max-width="90%" +%} + +This is the pipeline definition: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - checkout + - build + - test +steps: + clone: + title: Cloning main repository... + stage: checkout + type: git-clone + arguments: + repo: 'codefresh-contrib/python-flask-sample-app' + revision: with-helm + git: github + MyAppDockerImage: + title: Building Docker Image + stage: build + type: build + working_directory: '${{clone}}' + arguments: + image_name: kostis-codefresh/python-flask-sample-app + tag: 'master' + dockerfile: Dockerfile + MyUnitTests: + title: Running Unit tests + stage: test + type: freestyle + working_directory: '${{clone}}' + arguments: + image: ${{MyAppDockerImage}} + commands: + - python setup.py test + StoreChart: + title: Storing Helm Chart + type: helm + stage: store + working_directory: ./python-flask-sample-app + arguments: + action: push + chart_name: charts/python + kube_context: kostis-demo@FirstKubernetes + DeployMyChart: + type: helm + stage: deploy + working_directory: ./python-flask-sample-app + arguments: + action: install + chart_name: charts/python + release_name: my-python-chart + helm_version: 3.0.2 + kube_context: kostis-demo@FirstKubernetes + custom_values: + - 'buildID=${{CF_BUILD_ID}}' + - 'image_pullPolicy=Always' + - 'image_tag=master' + - 'image_pullSecret=codefresh-generated-r.cfcr.io-cfcr-default' +{% endraw %} +{% endhighlight %} + +You can see the source code in our [example section]({{site.baseurl}}/docs/example-catalog/cd-examples/helm/). + + +## Related articles +[Helm Charts and repositories]({{site.baseurl}}/docs/deployments/helm/add-helm-repository/) +[Using managed Helm repositories]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/) +[Helm Promotion boards]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion) diff --git a/_docs/deployments/kubernetes/custom-kubectl-commands.md b/_docs/deployments/kubernetes/custom-kubectl-commands.md new file mode 100644 index 00000000..77bc7411 --- /dev/null +++ b/_docs/deployments/kubernetes/custom-kubectl-commands.md @@ -0,0 +1,184 @@ +--- +title: "Custom kubectl commands" +description: "Use kubectl in your Codefresh pipelines" +group: deployments +sub_group: kubernetes +toc: true +--- + +As explained in [Kubernetes deployment options]({{site.baseurl}}/docs/deployments/kubernetes/deployment-options-to-kubernetes/), Codefresh has built-in functionality for deploying to Kubernetes clusters. + +For maximum flexibility with cluster deployments, you can run your own custom `kubectl` commands in a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). +[Kubectl](https://kubernetes.io/docs/reference/kubectl/overview/){:target="\_blank"} is the command line interface for managing kubernetes clusters. + +Codefresh automatically sets up your [config context](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/){:target="\_blank"} with your connected clusters. + +The config context is automatically placed for you at the path of the [variable]({{site.baseurl}}/docs/pipelines/variables/) `$CF_KUBECONFIG_PATH`. +In the current Codefresh implementation, this expands to `/codefresh/volume/sensitive/.kube/config`, within the [shared step volume]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/#sharing-the-workspace-between-build-steps). + +When you use custom `kubectl` commands, it is your responsibility to template your manifests using any of the available options. To employ Codefresh for templating, it is better to use the dedicated [cf-deploy-kubernetes step]({{site.baseurl}}/docs/deployments/ci-cd-guides/kubernetes-templating/), which provides simple templating capabilities. + +## Using the Codefresh kubectl image + +Codefresh already offers a public Docker image with `kubectl` at [https://hub.docker.com/r/codefresh/kubectl/tags](https://hub.docker.com/r/codefresh/kubectl/tags){:target="\_blank"}. You can choose a specific version of `kubectl` with the appropriate tag or just select `latest` for the most up-to-date version. + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + MyCustomKubectlCommands: + title: Running Kubectl + image: codefresh/kubectl:1.13.3 + commands: + - echo $CF_KUBECONFIG_PATH + - kubectl help +{% endraw %} +{% endhighlight %} + +If you run the pipeline, you can see the help options for `kubectl`. + +## Getting a config context + +The important thing to know when running custom `kubectl` commands is that Codefresh automatically sets up +your [kubeconfig files](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/){:target="\_blank"} for you with the cluster information present in [integrations]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster). + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/kube-context.png" +url="/images/deployments/kubernetes/kube-context.png" +alt="Codefresh cluster names" +caption="Codefresh cluster names" +max-width="50%" +%} + +If you run this pipeline, you will see the names of all your connected clusters: + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + MyCustomKubectlCommands: + title: Running Kubectl + image: codefresh/kubectl + commands: + - kubectl config get-contexts +{% endraw %} +{% endhighlight %} + +With two sample clusters, the output of this pipeline is the following: + +``` +Running freestyle step: Running Kubectl +Pulling image codefresh/kubectl:latest +Status: Image is up to date for codefresh/kubectl:latest +NAME CLUSTER AUTHINFO NAMESPACE +gke-kostisdemo-codefresh-kostis gke-kostisdemo-codefresh-kostis gke-kostisdemo-codefresh-kostis default +kostis-demo@FirstKubernetes kostis-demo@FirstKubernetes kostis-demo@FirstKubernetes default + +``` + +You can modify the current config context and run any `kubectl` command you want applied to that context. The next pipeline will print all the nodes of the first cluster: + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + MyCustomKubectlCommands: + title: Running Kubectl + image: codefresh/kubectl + commands: + - kubectl config get-contexts + - kubectl config use-context "gke-kostisdemo-codefresh-kostis" + - kubectl get nodes +{% endraw %} +{% endhighlight %} + +## Example of parallel deployment with kubectl + +Let's see a full example. In this pipeline, we will create two Docker images and deploy them on two separate clusters, using custom `kubectl` commands. We will also use the [parallel capability]({{site.baseurl}}/docs/pipelines/advanced-workflows/) of Codefresh pipelines. + +Here is the pipeline: + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/parallel-kubectl.png" +url="/images/deployments/kubernetes/parallel-kubectl.png" +alt="Parallel kubectl deployment" +caption="Parallel kubectl deployment" +max-width="100%" +%} + +And here is the complete `codefresh.yml`: + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' + +stages: +- build +- deploy + +steps: + BuildingApps: + type: parallel + stage: 'build' + steps: + BuildingApp1: + title: Building App 1 + type: build + stage: build + image_name: nestjs-app + working_directory: ./my-nestjs-project/ + dockerfile: Dockerfile + BuildingApp2: + title: Building App 2 + type: build + stage: build + image_name: rails + working_directory: ./my-rails-project/ + dockerfile: Dockerfile + DeployingApps: + type: parallel + stage: 'deploy' + steps: + DeployApp1: + title: Deploying App 1 + stage: deploy + image: codefresh/kubectl + working_directory: ./my-nestjs-project/ + commands: + - kubectl config get-contexts + - kubectl config use-context "gke-kostisdemo-codefresh-kostis" + - kubectl apply -f service.yml deployment.yml + DeployApp2: + title: Deploying App 2 + stage: deploy + image: codefresh/kubectl + working_directory: ./my-rails-project/ + commands: + - kubectl config get-contexts + - kubectl config use-context "kostis-demo@FirstKubernetes" + - kubectl apply -f service.yml deployment.yml configmap.yml +{% endraw %} +{% endhighlight %} + +In the example above, we select one of the clusters in each deployment step, and then apply several Kubernetes manifests that constitute an application. + +## Related articles +[Managing your cluster]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) +[Accessing a Docker registry]({{site.baseurl}}/docs/ci-cd-guides/access-docker-registry-from-kubernetes/) + + + + + + + + + + + \ No newline at end of file diff --git a/_docs/deployments/kubernetes/deployment-options-to-kubernetes.md b/_docs/deployments/kubernetes/deployment-options-to-kubernetes.md new file mode 100644 index 00000000..ed56a0b1 --- /dev/null +++ b/_docs/deployments/kubernetes/deployment-options-to-kubernetes.md @@ -0,0 +1,141 @@ +--- +title: "Deployment options for Kubernetes" +description: "Deploy to Kubernetes with the declarative deploy step" +group: deployments +sub_group: kubernetes +redirect_from: + - /docs/deploy-to-kubernetes/ + - /docs/deployment-to-kubernetes-quick-start-guide/ + - /docs/deploy-to-kubernetes/deployment-to-kubernetes-quick-start-guide/ + - /docs/deploy-to-kubernetes/get-ready-to-deploy/ +toc: true +--- + +Codefresh offers several options when it comes to Kubernetes deployments: + +1. Codefresh UI for on demand deployments + This is the easiest deployment option for Kubernetes. See our [Kubernetes quick start guide]({{site.baseurl}}/docs/getting-started/deployment-to-kubernetes-quick-start-guide/). +1. Through a dedicated [deploy step]({{site.baseurl}}/docs/pipelines/steps/deploy/) in a pipeline + Described in this article. +1. Through the [cf-deploy-kubernetes step]({{site.baseurl}}/docs/ci-cd-guides/kubernetes-templating/) in a pipeline + Use this to also perform simple templating on Kubernetes manifests. +1. Through a [freestyle]({{site.baseurl}}/docs/pipelines/steps/freestyle/) step with [Kustomize](https://kustomize.io){:target="\_blank"}. + See [Deployment with Kustomize]({{site.baseurl}}/docs/example-catalog/cd-examples/deploy-with-kustomize). +1. Using a [freestyle]({{site.baseurl}}/docs/codefresh-yaml/steps/freestyle/) step with your own `kubectl` commands + This deployment option gives you great flexibility, but assumes that you know how to work with `kubectl`. See [Custom kubectl commands]({{site.baseurl}}/docs/deployments/kubernetes/custom-kubectl-commands/). +1. Using Helm as a package manager + See our [Helm quick start guide]({{site.baseurl}}/docs/getting-started/helm-quick-start-guide/). + +## Prerequisites + +* A K8s cluster in Codefresh (see [Connecting a Kubernetes cluster]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster/) +* Familiarity with the [Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/), basic [pipeline steps ]({{site.baseurl}}/docs/pipelines/steps/), and how to describe them +* [Integrate your Docker registry]({{site.baseurl}}/docs/integrations/docker-registries/) with Codefresh + +## Build and push your image +Here is a basic Codefresh pipeline scenario to build and push your image to Dockerhub registry. + + `YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + BuildImage: + type: build + image_name: ' / ' #specify your future image reference here + dockerfile: Dockerfile + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + + PushToDockerRegistry: + type: push + candidate: '${{BuildImage}}' + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + registry: 'dockerhub' #the name of the registry you added to Codefresh +{% endraw %} +{% endhighlight %} + +Using this YAML example, we'll add an additional step to deploy the image in Dockerhub to Kubernetes. + +## Describe your deployment +The following instructions describe how to create a new service in your Kubernetes cluster in order to deploy to it. +>If you're deploying to an existing service in your Kubernetes cluster, please skip to the [next step]({{site.baseurl}}/docs/getting-started/deployment-to-kubernetes-quick-start-guide/#add-a-deployment-step). + + + 1. Go to the **`Kubernetes` → `Services page`**. + 1. Click the button **“Add Service”**. + 1. Select the **cluster**. + 1. Select the **namespace**. + 1. Type an arbitrary **service name**. + 1. Specify the **number of replicas**. + 1. Type the name of your **pushed image**. + 1. In the **“Internal Ports”** field specify the port which your application listens to. + 1. In the **“Expose port”** field specify the port to be exposed to the Internet and check the checkbox. + 1. Click the button **“Deploy”** to deploy the application. + +Wait until the deployment is completed, and you can open the deployed application in your browser by clicking on the "endpoint" link. + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/describe-k8s-deployment.png" +url="/images/deployments/kubernetes/describe-k8s-deployment.png" +alt="Describe Kubernetes deployment" +caption="Describe Kubernetes deployment" +max-width="60%" +%} + +## Add a Deployment step +So now you have deployed your image manually, which is great. But how to trigger the deployment within your pipeline? For that you will need to add a step of a “Deploy” type to the Codefresh YAML manifest file: + + `YAML` +{% highlight yaml %} +{% raw %} +RunningDeployScript: + title: Running Deploy Script + type: deploy + kind: kubernetes + cluster: ' ' #the name specified when you added the cluster + namespace: #the namespace you wish to deploy into + service: #the service you would like to update the deployment in + candidate: + image: '${{BuildImage}}' + registry: 'dockerhub' +{% endraw %} +{% endhighlight %} + +The full Codefresh YAML looks like this: + + `YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + BuildImage: + type: build + image_name: ' / ' + dockerfile: Dockerfile + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + + PushToDockerRegistry: + type: push + candidate: '${{BuildImage}}' + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + registry: 'dockerhub' #the name of the registry you added to Codefresh + + RunningDeployScript: + title: Running Deploy Script + type: deploy + kind: kubernetes + cluster: ' ' #the name specified when you added the cluster + namespace: #the namespace you wish to deploy into + service: #the service you would like to update the deployment in + candidate: + image: '${{BuildImage}}' + registry: 'dockerhub' +{% endraw %} +{% endhighlight %} + +You can now run the whole pipeline that builds your application from source to a docker image, pushes it to a docker registry and deploys it to your Kubernetes cluster. + +## Related articles +[Manage your Kubernetes cluster]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) +[Environment dashboard]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/) diff --git a/_docs/deployments/kubernetes/manage-kubernetes.md b/_docs/deployments/kubernetes/manage-kubernetes.md new file mode 100644 index 00000000..986fb08d --- /dev/null +++ b/_docs/deployments/kubernetes/manage-kubernetes.md @@ -0,0 +1,169 @@ +--- +title: "Managing Kubernetes clusters" +description: "Use the graphical Kubernetes dashboard in Codefresh" +group: deployments +sub_group: kubernetes +redirect_from: + - /docs/deploy-to-kubernetes/codefresh-kubernetes-integration-beta/ + - /docs/codefresh-kubernetes-integration-beta/ +toc: true +--- + +Codefresh includes a built-in Kubernetes Dashboard that allows you to see the state of your clusters, and even make changes if you have the appropriate access privileges. + +## Accessing the Kubernetes Dashboard + +After [adding a cluster]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster), you will be able to manage your Kubernetes assets via the *Kubernetes tab* on the left pane. Clicking on the Kubernetes icon will take you to your services dashboard. + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/kubernetes-dashboard.png" +url="/images/deployments/kubernetes/kubernetes-dashboard.png" +alt="Codefresh Kubernetes Dashboard" +caption="Codefresh Kubernetes Dashboard" +max-width="80%" + %} + +With the graphical dashboard it is very easy to locate problematic services or deploy new ones quickly. If there are clusters that are not accessible to your user you can hide them by enabling the *Hide inaccessible clusters* option at the top right of the window in order to simplify the view. + +## Viewing your Kubernetes services + +If you have too many clusters you can choose the *add filter* button at the top of the window to hide specific clusters or namespaces. + +You will be able to see the following parameters for each service: +* Name +* Cluster +* Namespace +* Replica count +* Docker image +* Selector +* A status check + +You can also switch to a Grid view if you prefer that over the default List view: + + +{% include image.html +lightbox="true" +file="/images/kubernetes/dashboard/grid-view.png" +url="/images/kubernetes/dashboard/grid-view.png" +alt="Kubernetes Dashboard grid view" +caption="Kubernetes Dashboard grid view" +max-width="80%" + %} + + If there are clusters that are not accessible to your user you can hide them by enabling the *Hide inaccessible clusters* option at the top right of the window in order to simplify the view. + + +## Work with your services + +In this view, you will be able to perform the following actions: + +* Add new service +* Edit/Update existing services +* Remove service + + +## Deploying a new service + +The Kubernetes dashboard provides a GUI dialog to quickly deploy new services in your cluster. + +### Choose a Docker image + +To add a service, click the "Add Service" button on the top or the "plus" button on a specific namespace. Then fill in the details for your new service. + +You can add images built in Codefresh which were pushed to Codefresh registry or provide a name for Docker image that will be pulled from an [external Docker registry]({{site.baseurl}}/docs/integrations/docker-registries/). Notice that images which are not from Dockerhub must be mentioned with their full domain name. + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/quick-ui-deploy.png" +url="/images/deployments/kubernetes/quick-ui-deploy.png" +alt="Deploying with the quick UI dialog" +caption="Deploying with the quick UI dialog" +max-width="60%" +%} + + +Use the following steps in order to add Image and pull secrets from the [connected Docker Registry]({{site.baseurl}}/docs/docker-registries/external-docker-registries/): +* Specify the image name in the format ` / / : ` +* Provide and image pull secret - this will be done for each namespace + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/deploying-private-cf-registry.png" +url="/images/deployments/kubernetes/deploying-private-cf-registry.png" +alt="Deploying from the private Codefresh registry" +caption="Deploying from the private Codefresh registry" +max-width="60%" +%} + + +From this screen you can also [create Kubernetes image secrets]({{site.baseurl}}/docs/ci-cd-guides/access-docker-registry-from-kubernetes/) without actually deploying anything. + + +### Set environment variables and resources + +You can add extra environment variables that will passed to the deployment image. + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/environment-variables-deployment.png" +url="/images/deployments/kubernetes/environment-variables-deployment.png" +alt="Environment variables for the deployment" +caption="Environment variables for the deployment" +max-width="60%" +%} + + + +You can also define resource limits for your pods. +It is a good practice to place maximum limits so that your services do not experience resource starvation. + + +### Adding a service with a manifest file + +If you are an advanced Kubernetes user, toggle the Deployment option button to the `YAML` position on the top right corner of the screen. +In this mode you can define exactly the contents for the service and deployment Kubernetes resources. + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/define-k8s-service-resource.png" +url="/images/deployments/kubernetes/define-k8s-service-resource.png" +alt="Define a Kubernetes Service Resource" +caption="Define a Kubernetes Service Resource" +max-width="60%" +%} + +You can type directly in the browser window or paste content from a text editor. + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/define-k8s-deployment-resource.png" +url="/images/deployments/kubernetes/define-k8s-deployment-resource.png" +alt="Define a Kubernetes Deployment Resource" +caption="Define a Kubernetes Deployment Resource" +max-width="60%" +%} + + +Congratulations! Your service is now deployed to your Kubernetes cluster. + +You can update an existing service in a similar manner from your Kubernetes services window - Just hit the "edit" icon and update your service using the same steps as in "Add new service" section. + +## Automate your deployment + +After your service is deployed to your Kubernetes cluster, you can automate image deployment using Codefresh pipelines. + +Some of the possible options are: + +1. The dedicated [deploy step]({{site.baseurl}}/docs/pipelines/steps/deploy/) in a pipeline. +1. The [cf-deploy-kubernetes step]({{site.baseurl}}/docs/ci-cd-guides/kubernetes-templating/) in a pipeline. This can also perform simple templating on Kubernetes manifests. + +See more choices in the [Deployment options page]({{site.baseurl}}/docs/deployments/kubernetes/deployment-options-to-kubernetes/). + +## Related articles +[Environment dashboard]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/) +[Add Config Maps]({{site.baseurl}}/docs/ci-cd-guides/add-config-maps-to-your-namespaces/) +[Kubernetes deployment quick start]({{site.baseurl}}/docs/getting-started/deployment-to-kubernetes-quick-start-guide/) + + + diff --git a/_docs/example-catalog/cd-examples/amazon-ecs.md b/_docs/example-catalog/cd-examples/amazon-ecs.md new file mode 100644 index 00000000..89905c22 --- /dev/null +++ b/_docs/example-catalog/cd-examples/amazon-ecs.md @@ -0,0 +1,155 @@ +--- +title: "Amazon ECS/Fargate" +description: "Use Codefresh to deploy Docker containers to ECS/Fargate" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/amazon-ecs/ + - /docs/deploy-your-containers/ + - /docs/deploy-your-containers/amazon-ecs/ +toc: true +--- +Codefresh can deploy to any ECS or Fargate cluster created in Amazon. + +{% include image.html +lightbox="true" +file="/images/examples/amazon-ecs/ecs-pipeline-deployment.png" +url="/images/examples/amazon-ecs/ecs-pipeline-deployment.png" +alt="Deploying to Amazon ECS" +caption="Deploying to Amazon ECS" +max-width="100%" +%} + +## Prerequisites + + +1. Configure an ECS (or Fargate) Cluster with at least one running instance. +1. Configure an ECS Service and Task Definition with a reference to **the image that you are going to build and push.** See [the official amazon docs](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) for more details. +1. Connect your [ECR to Codefresh]({{site.baseurl}}/docs/docker-registries/external-docker-registries/amazon-ec2-container-registry/) so that it can be used by name in Codefresh pipelines. +1. Verify you have AWS Credentials (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`), with the following privileges: + + `JSON` +{% highlight json %} +{% raw %} +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "Stmt1479146904000", + "Effect": "Allow", + "Action": [ + "ecs:DescribeServices", + "ecs:DescribeTaskDefinition", + "ecs:DescribeTasks", + "ecs:ListClusters", + "ecs:ListServices", + "ecs:ListTasks", + "ecs:RegisterTaskDefinition", + "ecs:UpdateService" + ], + "Resource": [ + "*" + ] + } + ] +} +{% endraw %} +{% endhighlight %} + + + +## Create a CI/CD pipeline for ECS/Fargate + +Here is the complete pipeline: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - "clone" + - "build" + - "deploy" +steps: + main_clone: + type: "git-clone" + description: "Cloning main repository..." + repo: "${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}" + revision: "${{CF_BRANCH}}" + stage: "clone" + git: github + BuildingDockerImage: + stage: "build" + title: Building Docker Image + type: build + image_name: ${{IMAGE}} + tag: '${{CF_SHORT_REVISION}}' + dockerfile: Dockerfile.multistage + Push: + title: "Pushing image to ECR" + stage: "deploy" + type: "push" + tag: '${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}' + registry: "ecr" + candidate: "${{BuildingDockerImage}}" + DeployToFargate: + stage: "deploy" + image: codefreshplugins/cf-deploy-ecs + commands: + - cfecs-update ${{REGION}} ${{ECS_CLUSTER_NAME}} ${{ECS_SERVICE_NAME}} --image-name ${{IMAGE_PREFIX}}/${{IMAGE}} --image-tag '${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}' + environment: + - AWS_ACCESS_KEY_ID=${{AWS_ACCESS_KEY_ID}} + - AWS_SECRET_ACCESS_KEY=${{AWS_SECRET_ACCESS_KEY}} + +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code with a [Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/) +1. Uses a [build step]({{site.baseurl}}/docs/pipelines/steps/build/) to create a Docker image +1. Uses a [push step]({{site.baseurl}}/docs/cpipelines/steps/push/) to push the docker image to ECR. The registry was previously [connected in Codefresh]({{site.baseurl}}/docs/docker-registries/external-docker-registries/) with the `ecr` identifier. +1. Runs `codefreshplugins/cf-deploy-ecs` to perform the actual deployment + + +The pipeline needs [environment variables]({{site.baseurl}}/docs/pipelines/pipelines/#pipeline-settings) that hold all the required parameters. + +{% include image.html +lightbox="true" +file="/images/examples/amazon-ecs/ecs-variables.png" +url="/images/examples/amazon-ecs/ecs-variables.png" +alt="ECS environment variables" +caption="ECS environment variables" +max-width="80%" +%} + + + + +Note that the **`--image-name`** and **`--image-tag`** pair should comprise the **full name** of the image that was pushed to the registry (including the registry name) in order to be correctly referred by the corresponding Task Definition. + + + +## Deployment Flow + +The `codefreshplugins/cf-deploy-ecs` step performs the following: + + +1. Gets the ECS service by specified `aws-region`, `ecs-cluster`, and `service-names`. +1. Creates a new revision from the current task definition of the service. If `--image-name` and `--image-tag` are provided, it replaces the image tag. +1. Runs the `update-service` command with the new task definition revision. +1. Waits for the deployment to complete. + * Deployment is successfully completed if `runningCount == desiredCount` for PRIMARY deployment - see `aws ecs describe-services` + * The `cfecs-update` command exits with a timeout error if after --timeout (default = 900s) `runningCount` does not equal `desiredCount` + * The `cfecs-update` exits with an error if --max-failed (default = 2) or more ECS tasks were stopped with error for the task definition that you are deploying. ECS continuously retries failed tasks. + +You can also find the same step in the form of a [Codefresh plugin](https://codefresh.io/steps/step/ecs-deploy). + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[External Registries]({{site.baseurl}}/docs/integration/docker-registries/) + + diff --git a/_docs/example-catalog/cd-examples/deploy-to-heroku.md b/_docs/example-catalog/cd-examples/deploy-to-heroku.md new file mode 100644 index 00000000..100a56ec --- /dev/null +++ b/_docs/example-catalog/cd-examples/deploy-to-heroku.md @@ -0,0 +1,212 @@ +--- +title: "Deploy to Heroku" +description: "Deploy your application or image to Heroku" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +Heroku is a container-based cloud PaaS (Platform as a Service) software that allows you to deploy, run, and manage your applications. Built on top of AWS, it supports Ruby, Node.js, Java, Python, Clojure, Scala, Go and PHP. + +This tutorial will cover two examples, depending on your use case. If you are not using containers, your use case is covered using the Codefresh heroku-deployer plugin ([Example #1](#pipeline-example-1-deploying-source-code-to-heroku-using-the-codefresh-heroku-plugin)). If you are using containers, you can achieve deployment by using a combination of build, push, and freestyle steps ([Example #2](#pipeline-example-2-deploy-a-docker-image-to-heroku)). + +## Example Django Application + +You can find the example project on [GitHub](https://github.com/codefresh-contrib/heroku-python-django-sample-app). + +The repository contains a Django starter project with the following commands: + +- `pip install -r requirements.txt` to install dependencies. +- `python -m unittest composeexample.utils` runs unit tests. +- `python manage.py runserver 0.0.0.0:8000` to start the application locally. + +Once launched the application presents the Django starter page at localhost:8000. + +## Pipeline Example #1: Deploying Source Code to Heroku Using the Codefresh Heroku Plugin + +### Prerequisites + +- A [free Codefresh account]({{site.baseurl}}/docs/administration/account-user-management/#create-a-codefresh-account/) +- A [free Heroku account](https://signup.heroku.com){:target="\_blank"} +- A Heroku API token (you can find this under **Account Settings** and then scrolling down, you will find the API Key) + +### Create the pipeline + +This pipeline has three stages: clone, test, and deploy. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/heroku-deployer-pipeline.png" +url="/images/examples/deployments/heroku-deployer-pipeline.png" +alt="Codefresh UI Pipeline View" +caption="Codefresh UI Pipeline View" +max-width="100%" +%} + +You should be able to copy and paste this YAML in the in-line editor of the Codefresh UI. It will automatically clone the project for you. + +Note that you need to change the environment variables in the deploy stage to your respective values. You can do this directly [in the YAML itself]({{site.baseurl}}/docs/how-to-guides/migrating-from-travis-ci/#environment-variables), or through the Codefresh UI. Navigate to the in-line editor, and to the right you will find a tab lebeled **Variables**. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/heroku-deployer-variables2.png" +url="/images/examples/deployments/heroku-deployer-variables2.png" +alt="Codefresh UI Pipeline Variables View" +caption="Codefresh UI Pipeline Variables View" +max-width="100%" +%} + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - clone + - test + - deploy +steps: + clone: + title: "Cloning main repository..." + stage: "clone" + type: "git-clone" + arguments: + repo: "codefresh-contrib/heroku-python-django-sample-app" + revision: "master" + git: "github" + run_unit_tests: + title: "Running unit tests..." + stage: "test" + type: "freestyle" + working_directory: "${{clone}}" + arguments: + image: "python:3.6-slim" + commands: + - "pip install -r requirements.txt --cache-dir=/codefresh/volume/pip-cache" + - "python -m unittest composeexample.utils" + deploy_to_heroku: + title: "Deploying to Heroku..." + stage: "deploy" + type: "heroku-deployer" + arguments: + APP_NAME: $APP_NAME + EMAIL: $EMAIL + API_TOKEN: $API_TOKEN +{% endraw %} +{% endhighlight %} + +The above pipeline has the following steps: + +1. A [git-clone]({{site.baseurl}}/docs/pipelines/steps/git-clone/) step that clones the main repository +2. A [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/) that installs dependencies and runs the unit tests +3. A freestyle step that deploys the application to Heroku using the heroku-deployer plugin from the [Step Marketplace](https://codefresh.io/steps/step/heroku-deployer) + +## Pipeline Example #2: Deploy a Docker Image to Heroku + +This example differs from the plugin usage, as it deploys a built Docker image to Heroku. + +Note that you need to change the environment variables to your respective values. You can do this directly [in the YAML itself]({{site.baseurl}}/docs/how-to-guides/migrating-from-travis-ci/#environment-variables), or through the Codefresh UI. Navigate to the in-line editor, and to the right you will find a tab lebeled **Variables**. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/heroku-deployer-variables.png" +url="/images/examples/deployments/heroku-deployer-variables.png" +alt="Codefresh UI Pipeline Variables View" +caption="Codefresh UI Pipeline Variables View" +max-width="100%" +%} + +## Prerequisites + +- A [free Codefresh account]({{site.baseurl}}/docs/administration/account-user-management/#create-a-codefresh-account/) +- A [free Heroku account](https://signup.heroku.com){:target="\_blank"} +- An empty repository already created in Heroku using the `heroku create ` command +- A Heroku registry [connected to Codefresh]({{site.baseurl}}/docs/docker-registries/external-docker-registries/other-registries/#heroku-registries) +- A Heroku API token (you can find this under **Account Settings** and then scrolling down, you will find the API Key) + +### Create the pipeline + +This pipeline has five stages: clone, build, test, push, and release. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/heroku-vanilla-push-pipeline.png" +url="/images/examples/deployments/heroku-vanilla-push-pipeline.png" +alt="Codefresh UI Pipeline View" +caption="Codefresh UI Pipeline View" +max-width="100%" +%} + +You should be able to copy and paste this YAML in the in-line editor of the Codefresh UI. It will automatically clone the project for you. + +`codefresh-heroku-push-image.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +version: '1.0' +stages: + - clone + - build + - test + - push + - release +steps: + clone: + title: "Cloning main repository..." + stage: "clone" + type: "git-clone" + arguments: + repo: "codefresh-contrib/heroku-python-django-sample-app" + revision: "master" + git: "github" + build: + title: "Building Docker Image..." + stage: "build" + type: "build" + working_directory: "./heroku-python-django-sample-app" + arguments: + image_name: "${{IMAGE_NAME}}" + tag: "master" + dockerfile: "Dockerfile" + run_unit_tests: + title: "Running unit tests..." + stage: "test" + type: "freestyle" + working_directory: "./heroku-python-django-sample-app" + arguments: + image: '${{build}}' + commands: + - "python -m unittest composeexample.utils" + push_image: + title: "Pushing image to Heroku..." + stage: "push" + type: "push" + arguments: + candidate: '${{build}}' + image_name: "${{IMAGE_NAME}}/web" + registry: "heroku" + release_image: + title: "Releasing image..." + stage: "release" + type: "freestyle" + arguments: + image: "nazarcodefresh/heroku-cli:alpine" + commands: + - >- + printf "machine api.heroku.com\n login $EMAIL\n password + $API_TOKEN\nmachine git.heroku.com\n login $EMAIL\n password + $API_TOKEN\n" > ~/.netrc + - "heroku container:release --app $IMAGE_NAME web" +{% endraw %} +{% endhighlight %} + +The pipeline does the following: +1. Clones the main repository through the [git-clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +1. Builds our Docker image through a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). +1. Runs unit tests on our Docker image through another freestyle step. +1. Pushes to the Heroku registry through a [push step]({{site.baseurl}}/docs/pipelines/steps/push/). +1. Releases the Docker image through another freestyle step. + + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) diff --git a/_docs/example-catalog/cd-examples/deploy-to-tomcat-via-scp.md b/_docs/example-catalog/cd-examples/deploy-to-tomcat-via-scp.md new file mode 100644 index 00000000..fa66c1c2 --- /dev/null +++ b/_docs/example-catalog/cd-examples/deploy-to-tomcat-via-scp.md @@ -0,0 +1,122 @@ +--- +title: "Deploy to a VM via SCP" +description: "Deploy your application to Tomcat using SCP" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +## Prerequisites + +- A [free Codefresh account]({{site.baseurl}}/docs/administration/account-user-management/create-codefresh-account) +- A distribution of [Tomcat](https://tomcat.apache.org/download-90.cgi){:target="\_blank"} setup on a remote server (running with port 8080 exposed) + +## The example Java Application + +You can find the example project on [GitHub](https://github.com/codefresh-contrib/scp-war-app){:target="\_blank"}. + +The example application is a simple Hello World Java application using the [Spark Java framework](http://sparkjava.com/){:target="\_blank"}: + +{% include image.html +lightbox="true" +file="/images/examples/deployments/scp-hello-world.png" +url="/images/examples/deployments/scp-hello-world.png" +alt="Hello World!" +caption="Hello World!" +max-width="100%" +%} + + +```java + @Override + public void init() { + get("/hello", (req, res) -> "Hello World"); + } +``` + +## Create the pipeline + +Our pipeline has three stages: clone, package, and transfer. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/scp-pipeline.png" +url="/images/examples/deployments/scp-pipeline.png" +alt="SCP pipeline" +caption="Codefresh UI Pipeline View" +max-width="100%" +%} + +You should be able to copy and paste this YAML in the in-line editor of the Codefresh UI. It will automatically clone the project for you. + +Note that you need to change the environment variables under the `transfer` step to your respective values. + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +# More examples of Codefresh YAML can be found at +# https://codefresh.io/docs/docs/example-catalog/example + +version: "1.0" +# Stages can help you organize your steps in stages +stages: + - "clone" + - "package" + - "transfer" + +steps: + clone: + title: "Cloning repository..." + type: "git-clone" + stage: "clone" + arguments: + repo: "codefresh-contrib/scp-war-app" + + package: + title: "Packaging war..." + type: "freestyle" + stage: "package" + arguments: + image: "maven:3.5.2-jdk-8-alpine" + working_directory: "${{clone}}" + commands: + - "mvn -Dmaven.repo.local=/codefresh/volume/m2_repository clean package" + + transfer: + title: "Transferring war to Tomcat..." + type: "freestyle" + stage: "transfer" + arguments: + image: "ictu/sshpass:latest" + working_directory: "${{package}}/target" + environment: + - USER= + - HOST= + - PASSWORD= + - TOMCAT_DIR= + commands: + - "echo | ssh-keygen -P '' -t rsa" + - "sshpass -p $PASSWORD ssh-copy-id -i /root/.ssh/id_rsa.pub -o StrictHostKeyChecking=no $USER@$HOST" + - "scp sparkjava-hello-world-1.0.war $USER@$HOST:$TOMCAT_DIR" +{% endraw %} +{% endhighlight %} + +The above pipeline does the following: + +1. Clones the main repository through the [git-clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +2. Installs the dependencies via Maven and packages our `war` file through a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). +3. Transfers our application via scp to a Tomcat server through another freestyle step. + +Note that you will need to change the listed environment variables accordingly, either through the YAML itself, or through your pipeline settings: + +{% include image.html +lightbox="true" +file="/images/examples/deployments/scp-variables.png" +url="/images/examples/deployments/scp-variables.png" +alt="Pipeline variables" +caption="Pipeline variables" +max-width="100%" +%} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) diff --git a/_docs/example-catalog/cd-examples/deploy-with-kustomize.md b/_docs/example-catalog/cd-examples/deploy-with-kustomize.md new file mode 100644 index 00000000..21b3089f --- /dev/null +++ b/_docs/example-catalog/cd-examples/deploy-with-kustomize.md @@ -0,0 +1,244 @@ +--- +title: "Deploy with Kustomize" +description: "Deploy your services to Kubernetes using Kustomize" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +[Kustomize](https://kustomize.io) is a tool included with kubectl 1.14 that "lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is." + +Kustomize is more of an overlay engine, as opposed to a templating engine. You create a base configuration and overlays. Your overlays contain a *kustomization.yaml* file, and any variants/changes are applied over top of the base configuration. Kustomize does not use templates at all. + +While it is good for simple scenarios, we suggest that you use Helm for managing your Kubernetes applications. Helm is a full package manager for Kubernetes manifests that also provides templating capabilities. See [this example]({{site.baseurl}}/docs/example-catalog/cd-examples/helm/){:target="\_blank"} for more information. + +## The example application + +You can find the example project on [GitHub](https://github.com/codefresh-contrib/kustomize-sample-app){:target="\_blank"}. + +The sample application is a simple Spring Boot web app, that displays an environment variable, `MY_MYSQL_DB` on the page: + +```java +public class HelloController { + + String my_sql_db = System.getenv("MY_MYSQL_DB"); + + @RequestMapping("/") + public String index() { + return my_sql_db; + } +``` + +The project contains a [base](https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#base){:target="\_blank"} and two [overlays](https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#overlay){:target="\_blank"}, one for a staging environment and one for production. + +The base manifest holds a dummy variable for `MY_MYSQL_DB` which will be overlayed once we call the kustomize command in our pipeline. + +`base/deployment.yaml` +```yaml +... + env: + - name: MY_MYSQL_DB + valueFrom: + configMapKeyRef: + name: the-map + key: mysqlDB +``` + +We will overlay on top of the manifests a different value for `MY_MYSQL_DB` for the staging environment and production environment. + +`overlays/staging/config-map.yaml` +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: the-map +data: + mysqlDB: "staging-mysql.example.com:3306" +``` + +`overlays/production/config-map.yaml` +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: the-map +data: + mysqlDB: "prod-mysql.example.com:3306" +``` + +In addition, for the production environment, the number of replicas will be overlayed to 3 instead of 1 (as [defined in the base deployment](https://github.com/codefresh-contrib/kustomize-sample-app/blob/32e683f82940de0bf2de2da40fa6b150e2b24b23/base/deployment.yaml#L8)){:target="\_blank"}. + +`overlays/production/deployment.yaml` +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: the-deployment +spec: + replicas: 3 +``` + +## Prerequisites + +- A [free Codefresh account]({{site.baseurl}}/docs/administration/account-user-management/create-codefresh-account) + +- A Kubernetes cluster [connected to your Codefresh account](https://codefresh.io/docs/docs/integrations/kubernetes/#connect-a-kubernetes-cluster) + +## Create the staging environment pipeline + +This pipeline will have two stages: clone and deploy. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/k8s-kustomize-staging-pipeline.png" +url="/images/examples/deployments/k8s-kustomize-staging-pipeline.png" +alt="Codefresh UI Pipeline View" +caption="Codefresh UI Pipeline View" +max-width="100%" +%} + +You should be able to copy and paste this YAML in the in-line pipeline editor of the Codefresh UI. However, make sure to replace cluster context for the kubectl command under the arguments section with your own that you integrated with Codefresh. It will automatically clone the project for you and deploy. + +`staging-codefresh.yml` +{% highlight yaml %} +{% raw %} +# More examples of Codefresh YAML can be found at +# https://codefresh.io/docs/docs/example-catalog/ + +version: "1.0" +# Stages can help you organize your steps in stages + +stages: + - clone + - deploy + +steps: + clone: + title: Cloning main repository... + type: git-clone + stage: clone + arguments: + repo: https://github.com/codefresh-contrib/kustomize-sample-app.git + git: github + revision: master + + deploy: + title: Deploying to Staging using Kustomize... + type: freestyle + stage: deploy + working_directory: ${{clone}} + arguments: + image: codefresh/kubectl:1.14.9 + commands: + - kubectl config use-context anna-sandbox@codefresh-support + - kubectl apply -k overlays/staging +{% endraw %} +{% endhighlight %} + +The above pipeline does the following: +1. Clones the main repository through a [git-clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +2. Connects to our Kubernetes cluster we have integrated with Codefresh using `kubectl`, and deploys the application as a staging environment with the appropriate value for `MY_MYSQL_DB` as defined in our configMap using Kustomize (the `-k` flag), through a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). + +>If you are using `kubectl` prior to 1.14, you can use the following command to deploy with Kustomize: + `kustomize build overlays/production | kubectl apply -f` + +## Create the production environment pipeline + +Likewise, this pipeline will have two stages: clone and deploy. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/k8s-kustomize-prod-pipeline.png" +url="/images/examples/deployments/k8s-kustomize-prod-pipeline.png" +alt="Codefresh UI Pipeline View" +caption="Codefresh UI Pipeline View" +max-width="100%" +%} + +You should be able to copy and paste this YAML in the in-line editor of the Codefresh UI and remember to replace cluster context for the kubectl command again with your own. Click Save and Run and it will automatically clone the project for you. + +`prod-codefresh.yml` +{% highlight yaml %} +{% raw %} +# More examples of Codefresh YAML can be found at +# https://codefresh.io/docs/docs/example-catalog/ + +version: "1.0" +# Stages can help you organize your steps in stages + +stages: + - clone + - deploy + +steps: + clone: + title: Cloning main repository... + type: git-clone + stage: clone + arguments: + repo: https://github.com/codefresh-contrib/kustomize-sample-app.git + git: github + revision: master + + deploy: + title: Deploying to Production using Kustomize... + type: freestyle + stage: deploy + working_directory: ${{clone}} + arguments: + image: codefresh/kubectl:1.14.9 + commands: + - kubectl config use-context anna-sandbox@codefresh-support + - kubectl apply -k overlays/production +{% endraw %} +{% endhighlight %} + +The above pipeline does the following: + +1. Clones the main repository through a [git-clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +1. Connects to our Kubernetes cluster we have integrated with Codefresh using `kubectl`, and deploys the application as a staging environment with the appropriate value for `MY_MYSQL_DB` as defined in our configMap using Kustomize (the `-k` flag), through a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). + + +>Note that if you are using kubectl prior to 1.14, you can use the following command to deploy with Kustomize: +>`kustomize build overlays/production | kubectl apply -f` + +## Verification + +After you run these pipelines, your deployments are displayed in the [Kubernetes dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/#accessing-the-kubernetes-dashboard). + +{% include image.html +lightbox="true" +file="/images/examples/deployments/k8s-kustomize-dashboard.png" +url="/images/examples/deployments/k8s-kustomize-dashboard.png" +alt="Codefresh Kubernetes Deployments" +caption="Codefresh Kubernetes Deployments" +max-width="100%" +%} + +You can test that the application deployed correctly to both environments by accessing the endpoints: + +{% include image.html +lightbox="true" +file="/images/examples/deployments/k8s-kustomize-staging-endpoint.png" +url="/images/examples/deployments/k8s-kustomize-staging-endpoint.png" +alt="Staging endpoint" +caption="Staging endpoint" +max-width="100%" +%} + +{% include image.html +lightbox="true" +file="/images/examples/deployments/k8s-kustomize-prod-endpoint.png" +url="/images/examples/deployments/k8s-kustomize-prod-endpoint.png" +alt="Production endpoint" +caption="Production endpoint" +max-width="100%" +%} + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Deployment options to Kubernetes]({{site.baseurl}}/docs/deployments/kubernetes/deployment-options-to-kubernetes) +[Running custom kubectl commands]({{site.baseurl}}/docs/deployments/kubernetes/custom-kubectl-commands/) +[Deploy with Helm]({{site.baseurl}}/docs/example-catalog/cd-examples/helm/) + diff --git a/_docs/example-catalog/cd-examples/docker-swarm.md b/_docs/example-catalog/cd-examples/docker-swarm.md new file mode 100644 index 00000000..ad6dfbe1 --- /dev/null +++ b/_docs/example-catalog/cd-examples/docker-swarm.md @@ -0,0 +1,221 @@ +--- +title: "Deploy to Docker SWARM" +description: "Deploy to Docker Swarm with Codefresh" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/docker-swarm/ + - /docs/deploy-to-docker-swarm/ + - /docs/deploy-your-containers/docker-swarm/ +toc: true +--- + +Codefresh can easily deploy your application to [Docker Swarm](https://docs.docker.com/engine/swarm/){:target="\_blank"} using [Codefresh pipelines]({{site.baseurl}}/docs/pipelines/pipelines/). + +You will need to provide: + +1. The `docker-stack.yml` that contains the definition of the application +1. The host where your Docker Swarm is running +1. An SSH key that Codefresh can use to access remotely the Docker Swarm host +1. The stack name that will be used once the application is deployed + +All this information will be passed to the pipeline in the form of build parameters. + + +## Example application + +For an example Docker Swarm application, see [https://github.com/codefreshdemo/example-voting-app](https://github.com/codefreshdemo/example-voting-app){:target="\_blank"} + +To launch it locally you need to download [Docker](https://www.docker.com/products/overview){:target="\_blank"}. +If you are on Mac or Windows, [Docker Compose](https://docs.docker.com/compose){:target="\_blank"} is automatically installed. +On Linux, make sure you have the latest version of [Compose](https://docs.docker.com/compose/install/){:target="\_blank"}. + + +Run in this root directory: + +{% highlight bash %} +{% raw %} + +docker-compose up + +{% endraw %} +{% endhighlight %} + +The app runs at [http://localhost:5000](http://localhost:5000), and the results are at [http://localhost:5001](http://localhost:5001). + +Alternately, if you want to run it on a Docker Swarm, first make sure you have a Swarm. +If you don't, run: + +{% highlight bash %} +{% raw %} + +docker swarm init + +{% endraw %} +{% endhighlight %} + +Once you have your swarm, in this directory run: + +{% highlight bash %} +{% raw %} + +docker stack deploy --compose-file docker-stack.yml vote + +{% endraw %} +{% endhighlight %} + +{{site.data.callout.callout_warning}} +The swarm master must have Python installed. +{{site.data.callout.end}} + +## Deploy to Remote Swarm with Codefresh + +First you need to set up the following environment variables in your Codefresh pipeline: + +{: .table .table-bordered .table-hover} +| `RDOCKER_HOST` | remote Docker Swarm master machine, accessible over SSH (for example, ubuntu@ec2-public-ip) | +| `STACK_NAME` | is new Docker stack name (use \"vote\", for example) | +| `SSH_KEY` | private SSH key, used to access Docker Swarm master machine | +| `SPLIT_CHAR` | split character, you've used to replace `newline` in SSH key. Recommendation: use `,` (`comma` character). | + +The `SSH_KEY` variable has the contents of the [SSH key](https://www.ssh.com/ssh/public-key-authentication){:target="\_blank"} that can access the Docker Swarm host. Currently, in order to pass SSH key through Codefresh UI, you need to convert it to single line string (replacing `newline` with `comma`), like this: + +{% highlight bash %} +{% raw %} +SSH_KEY=$(cat ~/.ssh/my_ssh_key_file | tr '\n' ',') +{% endraw %} +{% endhighlight %} + +The `SPLIT_CHAR` variable should hold the replacement character that was used for the SSH key (in the example above it is the comma character) + +{% include image.html +lightbox="true" +file="/images/2f1884a-codefresh_env_vars.png" +url="/images/2f1884a-codefresh_env_vars.png" +alt="Docker Swarm build parameters" +caption="Docker Swarm build parameters" +max-width="70%" +%} + + +## Deploy to Docker Swarm with a YAML step + +Once you have defined all the variables, deploy to your cluster using the following [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). + + `codefresh.yml` +{% highlight yaml %} +{% raw %} + +deploy_to_swarm: + image: codefresh/remote-docker + working_directory: ${{main_clone}} + commands: + - rdocker ${{RDOCKER_HOST}} docker stack deploy --compose-file docker-stack.yml ${{STACK_NAME}} + environment: + - SSH_KEY=${{SSH_KEY}} + when: + branch: + only: + - master + +{% endraw %} +{% endhighlight %} + +You can also pass custom credentials like this: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} + +deploy_to_swarm: + image: codefresh/remote-docker + working_directory: ${{main_clone}} + commands: + - rdocker ${{RDOCKER_HOST}} docker login ${{MY_REGISTRY}} -u ${{MY_REGISTRY_USER}} -p ${{MY_REGISTRY_PASSWORD}} \&\& docker stack deploy --compose-file docker-compose.yml --with-registry-auth ${{STACK_NAME}} + environment: + - SSH_KEY=${{SSH_KEY}} + when: + branch: + only: + - master +{% endraw %} +{% endhighlight %} + + + +## Create a CI/CD pipeine for Docker Swarm + +Here is the complete pipeline: + +{% include +image.html +lightbox="true" +file="/images/examples/docker-swarm/docker-swarm-pipeline.png" +url="/images/examples/docker-swarm/docker-swarm-pipeline.png" +alt="Docker Swarm pipeline" +caption="Docker Swarm pipeline" +max-width="100%" +%} + +And here is the pipeline definition: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - build + - deploy +steps: + main_clone: + title: Cloning main repository... + stage: prepare + type: git-clone + repo: 'codefreshdemo/example-voting-app' + revision: master + git: github-1 + MyResultDockerImage: + title: Building Result Docker Image + stage: build + type: build + image_name: resultApp + working_directory: ./result/ + tag: master + dockerfile: Dockerfile + MyVoteDockerImage: + title: Building Vote Docker Image + stage: build + type: build + image_name: voteApp + working_directory: ./vote/ + tag: master + dockerfile: Dockerfile + MyWorkerDockerImage: + title: Building Worker Docker Image + stage: build + type: build + image_name: workedApp + working_directory: ./worker/ + tag: master + dockerfile: Dockerfile + DeployToSwarmNow: + image: codefresh/remote-docker + working_directory: ${{main_clone}} + stage: deploy + commands: + - rdocker ${{RDOCKER_HOST}} docker login ${{MY_REGISTRY}} -u ${{MY_REGISTRY_USER}} -p ${{MY_REGISTRY_PASSWORD}} \&\& docker stack deploy --compose-file docker-compose.yml --with-registry-auth ${{STACK_NAME}} + environment: + - SSH_KEY=${{SSH_KEY}} +{% endraw %} +{% endhighlight %} + +The values of `MY_REGISTRY`, `MY_REGISTRY_USER` and `MY_REGISTRY_PASSWORD` depend upon the type of [your connected registry]({{site.baseurl}}/docs/integration/docker-registries/). + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) + diff --git a/_docs/example-catalog/cd-examples/elastic-beanstalk.md b/_docs/example-catalog/cd-examples/elastic-beanstalk.md new file mode 100644 index 00000000..cd0b6949 --- /dev/null +++ b/_docs/example-catalog/cd-examples/elastic-beanstalk.md @@ -0,0 +1,136 @@ +--- +title: "Deploy to Elastic Beanstalk" +description: "" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/elastic-beanstalk/ + - /docs/deploy-your-containers/elastic-beanstalk/ +toc: true +--- + + +## Prerequisites + +- Configured Application in Elastic Beanstalk service
+ See: [http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/GettingStarted.html](http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/GettingStarted.html){:target="_blank"} + + +## Deployment with Codefresh +- Add encrypted environment variables for AWS credentials: + * `AWS_ACCESS_KEY_ID` + * `AWS_SECRET_ACCESS_KEY` + +- Provide the following environment variables: + * `AWS_REGION` + * `AWS_ENV_NAME` + * `AWS_VERSION` + * `AWS_BRANCH` + +{% include +image.html +lightbox="true" +file="/images/examples/elastic-beanstalk/codefresh_eb_env_vars.png" +url="/images/examples/elastic-beanstalk/codefresh_eb_env_vars.png" +alt="codefresh_eb_env_vars.png" +max-width="40%" +%} + +{{site.data.callout.callout_info}} +{% raw %} +The ``${{AWS_VERSION}}`` of application you can find in the Elastic Beanstalk service. +{% endraw %} +{{site.data.callout.end}} + +{% include +image.html +lightbox="true" +file="/images/examples/elastic-beanstalk/codefresh_eb_version_label.png" +url="/images/examples/elastic-beanstalk/codefresh_eb_version_label.png" +alt="codefresh_eb_version_label.png" +max-width="40%" +%} + +{{site.data.callout.callout_info}} +{% raw %} +The ``${{AWS_ENV_NAME}}`` of application you can find in the Elastic Beanstalk service. +{% endraw %} +{{site.data.callout.end}} + +{% include +image.html +lightbox="true" +file="/images/examples/elastic-beanstalk/codefresh_eb_environment.png" +url="/images/examples/elastic-beanstalk/codefresh_eb_environment.png" +alt="codefresh_eb_environment.png" +max-width="40%" +%} + +Add the following step to codefresh.yml: + + `deploy_step` +{% highlight yaml %} +{% raw %} +deploy-elastic-beanstalk: + fail-fast: false + image: garland/aws-cli-docker:latest + commands: + - sh -c "aws configure set region '${{AWS_REGION}}' && aws elasticbeanstalk update-environment --environment-name '${{AWS_ENV_NAME}}' --version-label '${{AWS_VERSION}}' " + when: + condition: + all: + masterBranch: "'${{CF_BRANCH}}' == '${{AWS_BRANCH}}'" +{% endraw %} +{% endhighlight %} + +{:.text-secondary} +## Deployment Flow +- Go to the Elastic Beanstalk service and create an application and environment. + + +{% include +image.html +lightbox="true" +file="/images/examples/elastic-beanstalk/codefresh_eb_environment-deploy.png" +url="/images/examples/elastic-beanstalk/codefresh_eb_environment-deploy.png" +alt="codefresh_eb_environment.png" +max-width="40%" +%} + +- Perform the following commands from root of your project: + * eb init + * eb create {% raw %}`${{AWS_ENV_NAME}}`{% endraw %} + + + +>Note: + If you don't have awsebcli - install EB CLI [http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-install.html](http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-install.html){:target="_blank"}. + + +{% include +image.html +lightbox="true" +file="/images/examples/elastic-beanstalk/codefresh_eb_health.png" +url="/images/examples/elastic-beanstalk/codefresh_eb_health.png" +alt="codefresh_eb_health.png" +max-width="40%" +%} + +- Add this repository to Codefresh, provide the necessary environments variables and build this service + +{% include +image.html +lightbox="true" +file="/images/examples/elastic-beanstalk/codefresh_eb_cf_step_deploy.png" +url="/images/examples/elastic-beanstalk/codefresh_eb_cf_step_deploy.png" +alt="codefresh_eb_cf_step_deploy.png" +max-width="40%" +%} + +## Example + +* [cf-example-deploy-elasticbeanstalk](https://github.com/codefreshdemo/cf-example-deploy-elasticbeanstalk){:target="_blank"} + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) diff --git a/_docs/example-catalog/cd-examples/helm.md b/_docs/example-catalog/cd-examples/helm.md new file mode 100644 index 00000000..1b104663 --- /dev/null +++ b/_docs/example-catalog/cd-examples/helm.md @@ -0,0 +1,225 @@ +--- +title: "Deploy with Helm" +description: "Use Helm in a Codefresh pipeline" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +[Helm](https://helm.sh/){:target=\_blank"} is the package manager for Kubernetes. +Codefresh has comprehensive support for Helm: + +* Free [built-in Helm repository]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/) with each Codefresh account +* [Helm chart dashboard]({{site.baseurl}}/docs/docs/deployments/add-helm-repository/) to track your charts +* [Helm Release dashboard]({{site.baseurl}}/docs/docs/deployments/helm-releases-management/) to view your deployments +* [Environment dashsboard]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/) to view Helm releases +* [Helm promotion dashboard]({{site.baseurl}}/docs/deployments/helm-environment-promotion/) to promote Helm releases +* Add any external Helm repository on any other cloud provider + +Codefresh also provides a [pipeline step]({{site.baseurl}}/docs/new-helm/using-helm-in-codefresh-pipeline/) for deploying with Helm. + +For more insights on Helm charts see also our [Helm best practices]({{site.baseurl}}/docs/new-helm/helm-best-practices/) guide. + + +## The example Helm project + +You can see the example project at [https://github.com/codefresh-contrib/helm-sample-app](https://github.com/codefresh-contrib/helm-sample-app){:target=\_blank"}. The repository contains a simple Go application, a Dockerfile and an example chart. + + +## Prerequisites + +[At least one Kubernetes cluster]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster/) in your Codefresh account. + +>Notice that if you still use Helm 2 you should also have installed the server side of Helm 2 (Tiller) using `helm init`. This command is best run from the cloud console of your cluster. The respective pipelines of this guide are in the [helm-2 branch](https://github.com/codefresh-contrib/helm-sample-app/tree/helm-2){:target=\_blank"}. + + + +## CI/CD pipeline with Helm deployment + +It is possible to deploy directly a Helm chart as it exists on the filesystem. This is not the recommended way to use Helm, because you are bypassing the Helm chart repository, but it is certainly the simplest Helm pipeline possible. + +{% include image.html +lightbox="true" +file="/images/examples/helm/helm-deploy-pipeline.png" +url="/images/examples/helm/helm-deploy-pipeline.png" +alt="Pipeline for Helm deployment" +caption="Pipeline for Helm deployment" +max-width="100%" +%} + +Here is the whole pipeline: + + `codefresh-do-not-store.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - build + - deploy +steps: + clone: + title: Cloning main repository... + stage: prepare + type: git-clone + arguments: + repo: codefresh-contrib/helm-sample-app + revision: master + git: github + build: + title: Building Docker Image + stage: build + type: build + working_directory: ./helm-sample-app + arguments: + image_name: helm-sample-app-go + tag: multi-stage + dockerfile: Dockerfile + deploy: + title: Deploying Helm Chart + type: helm + stage: deploy + working_directory: ./helm-sample-app + arguments: + action: install + chart_name: charts/helm-example + release_name: my-go-chart-prod + helm_version: 3.0.2 + kube_context: my-demo-k8s-cluster + custom_values: + - 'buildID=${{CF_BUILD_ID}}' + - 'image_pullPolicy=Always' + - 'image_tag=multi-stage' + - 'replicaCount=3' + - 'image_pullSecret=codefresh-generated-r.cfcr.io-cfcr-default' +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code through a [Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/) +1. Builds a docker image through a [build step]({{site.baseurl}}/docs/pipelines/steps/build/) +1. Deploys the Helm chart to a cluster named `my-demo-k8s-cluster` using the Helm step [from the Step Marketplace](https://codefresh.io/steps/step/helm){:target=\_blank"}. + +In this example, `charts/helm-example` refers to the [filesystem location in the code](https://github.com/codefresh-contrib/helm-sample-app/tree/master/charts/helm-example){:target=\_blank"} that was just checked out. + +The deployment will be visible in the [Helm releases dashboard]({{site.baseurl}}/docs/new-helm/helm-releases-management/). + +{% include image.html +lightbox="true" +file="/images/examples/helm/helm-release.png" +url="/images/examples/helm/helm-release.png" +alt="Helm release view" +caption="Helm release view" +max-width="100%" +%} + +If you want to run this example yourself, make sure to edit the chart and put your own values there for the Docker image. + +## CI/CD pipeline with Helm deployment that also stores the chart + +It is recommended to use a Helm repository to store your chart before deploying it. This way you know what is deployed in your clusters +and you can also reuse charts in other installations. + +First of all you need to import in your pipeline from the [shared configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/) the settings for the internal Helm repository (or any other external repository that you have setup in Codefresh). + This will make available the internal Helm repository to your pipeline so that it can push/pull Helm charts from it. + + {% include image.html + lightbox="true" + file="/images/examples/helm/import-helm-configuration.png" + url="/images/examples/helm/import-helm-configuration.png" + alt="Using the default Helm repository in a Pipeline" + caption="Using the default Helm repository in a Pipeline" + max-width="40%" + %} + +Once that is done you can change your pipeline to also store the chart first and *then* deploy it. + + +{% include image.html +lightbox="true" +file="/images/examples/helm/helm-push-and-deploy-pipeline.png" +url="/images/examples/helm/helm-push-and-deploy-pipeline.png" +alt="Pipeline for Helm deployment that stores chart" +caption="Pipeline for Helm deployment that stores chart" +max-width="100%" +%} + +Here is the whole pipeline: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - build + - store + - deploy +steps: + clone: + title: Cloning main repository... + stage: prepare + type: git-clone + arguments: + repo: codefresh-contrib/helm-sample-app + revision: master + git: github + build: + title: Building Docker Image + stage: build + type: build + working_directory: ./helm-sample-app + arguments: + image_name: helm-sample-app-go + tag: multi-stage + dockerfile: Dockerfile + store: + title: Storing Helm Chart + type: helm + stage: store + working_directory: ./helm-sample-app + arguments: + action: push + chart_name: charts/helm-example + kube_context: my-demo-k8s-cluster + deploy: + type: helm + stage: deploy + working_directory: ./helm-sample-app + arguments: + action: install + chart_name: charts/helm-example + release_name: my-go-chart-prod + helm_version: 3.0.2 + kube_context: my-demo-k8s-cluster + custom_values: + - 'buildID=${{CF_BUILD_ID}}' + - 'image_pullPolicy=Always' + - 'image_tag=multi-stage' + - 'replicaCount=3' + - 'image_pullSecret=codefresh-generated-r.cfcr.io-cfcr-default' +{% endraw %} +{% endhighlight %} + + +After you finish running your pipeline, not only the deployment will take place, but you will also see your chart in your [Helm Chart dashboard]({{site.baseurl}}/docs/new-helm/add-helm-repository/): + +{% include image.html +lightbox="true" +file="/images/examples/helm/helm-chart.png" +url="/images/examples/helm/helm-chart.png" +alt="Stored Helm chart" +caption="Stored Helm chart" +max-width="80%" +%} + +It is also possible to [run your own Helm commands]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/#example-custom-helm-commands) in a Codefresh pipeline. + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) diff --git a/_docs/example-catalog/cd-examples/import-data-to-mongodb.md b/_docs/example-catalog/cd-examples/import-data-to-mongodb.md new file mode 100644 index 00000000..68a6c79a --- /dev/null +++ b/_docs/example-catalog/cd-examples/import-data-to-mongodb.md @@ -0,0 +1,60 @@ +--- + +title: "Import data to MongoDB" +description: "" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/import-data-to-mongodb-in-composition/ + - /docs/on-demand-test-environment/example-compositions/import-data-to-mongodb/ +toc: true +--- + +If you want to import/restore or to do something else before using MongoDB in your application, you can look at the following example. + +You just need to create Dockerfile for mongo seed service and provide the command to prepare MongoDB. In this case it's command `mongoimport` + + `Dockerfile mongo_seed` +{% highlight docker %} +FROM mongo +COPY init.json /init.json +CMD mongoimport --host mongodb --db exampleDb --collection contacts --type json --file /init.json --jsonArray +{% endhighlight %} + +## Looking around +In the root of this repository you'll find a file named `docker-compose.yml`. +Let's quickly review the contents of this file: + + `docker-compose.yml` +{% highlight yaml %} +{% raw %} +version: '3' +services: + mongodb: + image: mongo + command: mongod --smallfiles + ports: + - 27017 + + mongo_seed: + image: ${{mongo_seed}} + links: + - mongodb + + client: + image: ${{build_prj}} + links: + - mongodb + ports: + - 9000 + environment: + - MONGO_URI=mongodb:27017/exampleDb +{% endraw %} +{% endhighlight %} + +{{site.data.callout.callout_info}} +You can add the following example to your GitHub or Bitbucket account, and build the [example](https://github.com/codefreshdemo/cf-example-manage-mongodb){:target="_blank"}. +{{site.data.callout.end}} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) diff --git a/_docs/example-catalog/cd-examples/nodejs-angular2-mongodb.md b/_docs/example-catalog/cd-examples/nodejs-angular2-mongodb.md new file mode 100644 index 00000000..f4e69839 --- /dev/null +++ b/_docs/example-catalog/cd-examples/nodejs-angular2-mongodb.md @@ -0,0 +1,52 @@ +--- +title: "NodeJS + Angular2 + MongoDB" +description: "" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/nodejs-angular2-mongodb/ + - /docs/on-demand-test-environment/example-compositions/nodejs-angular2-mongodb/ +toc: true +--- +This tutorial will walk you through the process of adding the following: + +- Build client +- Build server +- Launch composition + +## Looking around +In the root of this repository you'll find a file named `docker-compose.yml`. +Let's quickly review the contents of this file: + + `docker-compose.yml` +{% highlight yaml %} +{% raw %} +version: '3' +services: + mongodb: + image: mongo + ports: + - 28017 + server: + image: ${{build_server}} + environment: + - MONGO_URI=mongodb://mongodb/exampleDb + links: + - mongodb + ports: + - 9000 + client: + image: ${{build_client}} + ports: + - 3000 +{% endraw %} +{% endhighlight %} + +{{site.data.callout.callout_info}} +##### Example + +Just head over to the example [__repository__](https://github.com/codefreshdemo/nodejs-angular2-mongo){:target="_blank"} in GitHub and follow the instructions there. +{{site.data.callout.end}} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) diff --git a/_docs/example-catalog/cd-examples/nomad.md b/_docs/example-catalog/cd-examples/nomad.md new file mode 100644 index 00000000..a7e78d79 --- /dev/null +++ b/_docs/example-catalog/cd-examples/nomad.md @@ -0,0 +1,225 @@ +--- +title: "Deploy to Nomad" +description: "Deploy Docker images to a Nomad cluster with Codefresh" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +Even though Codefresh has great support for Kubernetes and Helm deployments, there is no lock-in on using just Kubernetes. Codefresh can deploy on any infrastructure. + + +[Nomad](https://www.nomadproject.io/){:target=\_blank"} is an alternative scheduling platform from Hashicorp. It supports docker containers (like Kubernetes), but you can also use Nomad to schedule VMs, Java apps, Go apps or any other standalone executable. + +There are several public Docker Images with Nomad, so it is very easy to use Codefresh pipelines to deploy to a Nomad cluster. + + +{% include image.html +lightbox="true" +file="/images/examples/nomad/nomad-ci-pipeline.png" +url="/images/examples/nomad/nomad-ci-pipeline.png" +alt="Deploying to Nomad with Codefresh" +caption="Deploying to Nomad with Codefresh" +max-width="80%" +%} + +In this example, we will use the image at [https://hub.docker.com/r/djenriquez/nomad](https://hub.docker.com/r/djenriquez/nomad){:target=\_blank"}. + +## The example Nomad project + +You can see the example project at [https://github.com/codefresh-contrib/nomad-sample-app](https://github.com/codefresh-contrib/nomad-sample-app){:target=\_blank"}. The repository contains a simple job specification that deploys a docker container on nomad cluster. + + +Here is the whole job file: + + `docker-job.hcl` +{% highlight hcl %} +{% raw %} +job "example-job" { + # Specify this job should run in the region named "us". Regions + # are defined by the Nomad servers' configuration. + #region = "us" + + # Spread the tasks in this job between us-west-1 and us-east-1. + datacenters = ["dc1"] + + # Run this job as a "service" type. Each job type has different + # properties. See the documentation below for more examples. + type = "service" + + # Specify this job to have rolling updates, two-at-a-time, with + # 30 second intervals. + update { + stagger = "30s" + max_parallel = 1 + } + + # A group defines a series of tasks that should be co-located + # on the same client (host). All tasks within a group will be + # placed on the same host. + group "example-group" { + # Specify the number of these tasks we want. + count = 3 + + # Create an individual task (unit of work). This particular + # task utilizes a Docker container to front a web application. + task "example-task" { + # Specify the driver to be "docker". Nomad supports + # multiple drivers. + driver = "docker" + + # Configuration is specific to each driver. + config { + image = "r.cfcr.io/$CF_ACCOUNT/$CF_REPO_NAME:$CF_BRANCH_TAG_NORMALIZED" + + auth { + username = "$CF_ACCOUNT" + password = "$CFCR_LOGIN_TOKEN" + server_address = "r.cfcr.io" + } + + port_map { + http = 8080 + } + } + + # The service block tells Nomad how to register this service + # with Consul for service discovery and monitoring. + service { + # This tells Consul to monitor the service on the port + # labelled "http". Since Nomad allocates high dynamic port + # numbers, we use labels to refer to them. + port = "http" + + check { + type = "http" + path = "/" + interval = "10s" + timeout = "2s" + } + } + + # Specify the maximum resources required to run the task, + # include CPU, memory, and bandwidth. + resources { + cpu = 500 # MHz + memory = 128 # MB + + network { + mbits = 100 + + + port "http" {} + + + + } + } + } + } +} + +{% endraw %} +{% endhighlight %} + +Notice that the job specification has several [Codefresh variables]({{site.baseurl}}/docs/pipelines/variables/) embedded. We will use [envsubst](https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html){:target=\_blank"} in our pipeline to replace +them with the correct values. + +## Prerequisites + +You need to create a Codefresh account and have a Nomad cluster running. You need to decide on how Codefresh will communicate +with the nomad cluster. In this simple example we just use the `NOMAD_ADDR` variable to point the nomad client to our cluster. In a production environment you should use proper [ACL](https://www.nomadproject.io/guides/security/acl.html){:target=\_blank"} and [certificate](https://www.nomadproject.io/guides/security/securing-nomad.html){:target=\_blank"} variables as well. + + +In this example the Nomad cluster is already setup on a VM at Google cloud. + +You also need to create a [token for the Docker registry]({{site.baseurl}}/docs/integrations/docker-registries/) so that Nomad can pull your private images on the cluster. + +## Create a CI/CD pipeline for Nomad deployments + +Here is the whole pipeline: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +stages: + - "clone" + - "build" + - "deploy" +steps: + main_clone: + type: "git-clone" + title: "Clone main repository..." + repo: "codefresh-contrib/nomad-sample-app" + revision: "${{CF_BRANCH}}" + stage: "clone" + build: + title: "Building Docker Image" + type: "build" + image_name: "nomad-sample-app" + tag: "${{CF_BRANCH_TAG_NORMALIZED}}" + dockerfile: "Dockerfile" + stage: "build" + prepareJob: + title: "Preparing Nomad job" + image: bhgedigital/envsubst + stage: deploy + commands: + - envsubst < docker-job.hcl > docker-job-export.hcl + - cat docker-job-export.hcl + runJob: + title: "Deploying Nomad job" + image: djenriquez/nomad + stage: deploy + commands: + - nomad run docker-job-export.hcl +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code through a [Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +1. Creates a Docker image for a simple Go application through a [build step]({{site.baseurl}}/docs/pipelines/steps/build/). The image is automatically pushed to the default Docker registry. +1. Replaces all variables in the job spec by running `envsubst`. These include: + * The Registry token so that Nomad can access the default Docker registry + * The docker image name and tag to be deployed +1. Runs the job to deploy the image to Nomad through a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). + + +Run the pipeline and see your deployment succeed. + +Here are the environment variables defined for this pipeline. + +{% include image.html +lightbox="true" +file="/images/examples/nomad/nomad-variables.png" +url="/images/examples/nomad/nomad-variables.png" +alt="Pipeline variables for Nomad deployments" +caption="Pipeline variables for Nomad deployments" +max-width="50%" +%} + + +The `NOMAD_ADDR` variable is holding the URL of the cluster. The `CFCR_LOGIN_TOKEN` variable holds authentication for the Codefresh Docker registry. + +## Verify the deployment + +Nomad also comes with its own UI that can show you the result of a deployment. + +{% include image.html +lightbox="true" +file="/images/examples/nomad/nomad-ui-deployment.png" +url="/images/examples/nomad/nomad-ui-deployment.png" +alt="Nomad UI deployment" +caption="Nomad UI deployment" +max-width="80%" +%} + +You can also use [Terraform]({{site.baseurl}}/docs/example-catalog/cd-examples/terraform/) in Codefresh pipelines. + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/cd-examples/packer-gcloud.md b/_docs/example-catalog/cd-examples/packer-gcloud.md new file mode 100644 index 00000000..58d12ade --- /dev/null +++ b/_docs/example-catalog/cd-examples/packer-gcloud.md @@ -0,0 +1,132 @@ +--- +title: "Deploy to a Virtual Machine" +description: "Deploy to Google Cloud in a Codefresh pipeline with Packer" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +Even though Codefresh is Kubernetes-native and designed for containers, it can still deploy traditional applications in the form of Virtual Machines to any Cloud provider. + +In this example, we will use [Packer](http://www.packer.io/){:target="\_blank"} to package an application into a VM disk image that will then be launched in Google Cloud. +Because Packer itself is already offered [in a Docker container](https://hub.docker.com/r/hashicorp/packer/){:target="\_blank"}, it is very easy to run Packer in a Codefresh pipeline. + +Google also offers a [Docker image for GCloud](https://hub.docker.com/r/google/cloud-sdk/){:target="\_blank"} making the launching of the VM straightforward in a Codefresh pipeline. + + +{% include image.html +lightbox="true" +file="/images/examples/packer-gcloud/packer-codefresh-pipeline.png" +url="/images/examples/packer-gcloud/packer-codefresh-pipeline.png" +alt="Running Packer inside Codefresh" +caption="Running Packer inside Codefresh" +max-width="80%" +%} + +This Codefresh pipeline creates a VM image and then uses it to launch a Google Compute instance. + + +## The example Packer/Gcloud project + +You can see the example project at [https://github.com/codefresh-contrib/vm-packer-sample-app](https://github.com/codefresh-contrib/vm-packer-sample-app){:target="\_blank"}. The repository contains a simple Go application as well as a packer template. + +You can play with it locally after installing the `packer` and `gcloud` executables. + +## Prerequisites + +You need to create a Codefresh account and a Google account first. Then you need to create a [Service account Key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys){:target="\_blank"} which will allow `packer` and `gcloud` to communicate with Google cloud. + + +Add your service account json as a pipeline variable called `SERVICE_ACCOUNT`. The content of this variable will be used +in order to authenticate to Google cloud. + +{% include image.html +lightbox="true" +file="/images/examples/packer-gcloud/service-account-variable.png" +url="/images/examples/packer-gcloud/service-account-variable.png" +alt="Using a Service Account JSON in Codefresh" +caption="Using a Service Account JSON in Codefresh" +max-width="50%" +%} + +## Create a CI/CD pipeline for Packer/GCloud + +Here is the whole pipeline: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - build + - deploy +steps: + main_clone: + title: 'Cloning main repository...' + type: git-clone + repo: 'codefresh-contrib/vm-packer-sample-app' + git: github + revision: 'master' + stage: prepare + SetupAuth: + title: 'Setup GCloud Auth' + image: 'alpine' + stage: prepare + commands: + - echo $SERVICE_ACCOUNT > account.json + BuildMyApp: + title: Compiling App code + stage: build + image: 'golang:1.12' + commands: + - go build -o sample src/sample/trivial-web-server.go + CreatePackerImage: + title: Baking VM image + stage: build + image: 'hashicorp/packer' + commands: + - packer validate my-google-cloud-example.json + - packer build -force my-google-cloud-example.json + DeployToVM: + title: Deploying to VM + stage: deploy + image: 'google/cloud-sdk' + commands: + - gcloud auth activate-service-account --key-file=account.json + - gcloud config set project firstkubernetes-176201 + - gcloud compute instances create packer-demo-codefresh --image codefresh-simple-ubuntu-vm --zone europe-west1-b --metadata-from-file startup-script=startup.sh --tags http-server --preemptible --quiet + +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code through a [Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +1. Saves the content of the variable that holds the Google account as a file called `account.json`. +1. Compiles the Go application through a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). +1. Runs `packer` to create a VM image based on Ubuntu that also contains the simple Go application. +1. Runs `gcloud` to launch a VM with the image that was just created. + + +Run the pipeline and see your deployment succeed. You can customize the image by editing the [Packer template](https://github.com/codefresh-contrib/vm-packer-sample-app/blob/master/my-google-cloud-example.json){:target="\_blank"}. + +Once the VM has finished launching you can access it with your web browser. + +{% include image.html +lightbox="true" +file="/images/examples/packer-gcloud/web-app-url.png" +url="/images/examples/packer-gcloud/web-app-url.png" +alt="Accessing the VM application" +caption="Accessing the VM application" +max-width="70%" +%} + + +You can follow the same procedure for any other cloud that has an API/CLI (such as AWS, Azure, Digital Ocean etc). + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/cd-examples/pulumi.md b/_docs/example-catalog/cd-examples/pulumi.md new file mode 100644 index 00000000..74c1f3f7 --- /dev/null +++ b/_docs/example-catalog/cd-examples/pulumi.md @@ -0,0 +1,116 @@ +--- +title: "Deploy with Pulumi" +description: "Use Pulumi in a Codefresh pipeline with Docker" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +[Pulumi](https://pulumi.io/){:target="\_blank"} is a platform for *Infrastructure as Code*. It works like Terraform but allows you to use a proper programming language (TypeScript, Python, Go) to describe your infrastructure (instead of a configuration language). + +You can use Pulumi to deploy to Kubernetes or any other supported cloud platform. Because Pulumi itself is already offered [in a Docker container](https://hub.docker.com/r/pulumi/pulumi), it is very easy to run Pulumi in a Codefresh pipeline. + + +{% include image.html +lightbox="true" +file="/images/examples/pulumi/pulumi-pipeline.png" +url="/images/examples/pulumi/pulumi-pipeline.png" +alt="Running Pulumi inside Codefresh" +caption="Running Pulumi inside Codefresh" +max-width="80%" +%} + +## The example Pulumi project + +You can see the example project at [https://github.com/codefresh-contrib/pulumi-sample-app](https://github.com/codefresh-contrib/pulumi-sample-app){:target="\_blank"}. The repository contains a simple Pulumi stack based on Kubernetes and TypeScript. + +You can play with it locally after installing the `pulumi` executable. + +## Prerequisites + +You need to create a Codefresh account and a Pulumi account first. Then you need to create a [Pulumi token](https://app.pulumi.com/account/tokens){:target="\_blank"} which will allows Codefresh to communicate with Pulumi. + +[Add a Kubernetes cluster]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster/) in your Codefresh account from any cloud provider. + +Codefresh automatically creates a kubeconfig in any [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/) with all your clusters. This is the same way that Pulumi communicated with Kubernetes, so the integration between Codefresh and Pulumi is ready out of the box. + +Create a [stack](https://pulumi.io/reference/stack.html){:target="\_blank"} in Pulumi or use the one provided in the example. + +Finally add you Pulumi token as a pipeline variable called `PULUMI_ACCESS_TOKEN`. All freestyle steps have automatic access to all pipeline variables, and Pulumi will search for a token by default with this name when logging in. + + +## Create a CI/CD pipeline for Pulumi + +Here is the whole pipeline: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - build + - deploy +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}' + revision: '${{CF_REVISION}}' + stage: prepare + git: github-1 + BuildProject: + title: Build project + stage: build + image: pulumi/pulumi + commands: + - yarn install + SelectMyCluster: + title: Select K8s cluster + stage: deploy + image: codefresh/kubectl:1.13.3 + commands: + - kubectl config get-contexts + - kubectl config use-context "kostis-demo@FirstKubernetes" + RunPulumi: + title: Deploying + stage: deploy + image: pulumi/pulumi + commands: + - pulumi stack select dev --non-interactive + - pulumi stack --non-interactive + - pulumi up --non-interactive +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code through a [Git clone step]({{site.baseurl}}/docs/pipelines/pipelines/git-clone/). +1. Runs `yarn install` to download dependencies. In this example we use TypeScript, but Go and Python would work as well (or any other language supported by Pulumi). +1. Chooses the cluster that will be used for deployments, if you have more than one. Use your own cluster name as seen in the [Kubernetes dashboard]({{site.baseurl}}/docs/deploy-to-kubernetes/manage-kubernetes/) of Codefresh. +1. Runs `pulumi up` with the same target cluster. + +The pipeline needs a [single environment variable]({{site.baseurl}}/docs/pipelines/pipelines/#pipeline-settings) that holds the content of your Pulumi Token. + + +{% include image.html +lightbox="true" +file="/images/examples/pulumi/pulumi-access-token.png" +url="/images/examples/pulumi/pulumi-access-token.png" +alt="Passing the Pulumi Token in the pipeline parameters" +caption="Passing the Pulumi Token in the pipeline parameters" +max-width="60%" +%} + +Run the pipeline and see your deployment succeed. + +## Handling Pull requests + +You can easily use the same pipeline or a different one for pull requests. In this case replace the `pulumi up` command with `pulumi preview`. Even better you can add an [approval step]({{site.baseurl}}/docs/pipelines/steps/approval/) to allows humans to inspect the pipeline first. + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) diff --git a/_docs/example-catalog/cd-examples/secure-a-docker-container-using-http-basic-auth.md b/_docs/example-catalog/cd-examples/secure-a-docker-container-using-http-basic-auth.md new file mode 100644 index 00000000..b7e2884c --- /dev/null +++ b/_docs/example-catalog/cd-examples/secure-a-docker-container-using-http-basic-auth.md @@ -0,0 +1,92 @@ +--- +title: "Secure a Docker Container using HTTP Basic Auth" +description: "" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/securing-docker-container-with-http-basic-auth/ + - /docs/on-demand-test-environment/examples-compositions/securing-docker-container-with-http-basic-auth/ + - /docs/on-demand-test-environment/example-compositions/secure-a-docker-container-using-http-basic-auth/ +toc: true +--- +Before making a product publicly available, you might want to restrict access to certain users. These are some options to accomplish this goal: + + - Implement custom authentication within the system + - Configure the server to act as a proxy between the user and the application + - Limit access to specific IP addresses + +This article explains how to secure a container by exposing public ports, using an extra NGINX container to act as a proxy. + +## Expose Web App Public Port + + `webapp` +{% highlight yaml %} +{% raw %} +version: '3' +services: + web: + image: codefreshio/webapp + ports: + - "3000" +{% endraw %} +{% endhighlight %} + +The architecture for this step is displayed in the diagram below. In this step example, Docker is forwarding an internal 3000 port to the host 80 port. + +{% include +image.html +lightbox="true" +file="/images/examples/docker-https/codefresh_webapp_container.png" +url="/images/examples/docker-https/codefresh_webapp_container.png" +alt="codefresh_webapp_container.png" +max-width="40%" +%} + +## Add NGINX Proxy +To secure the web-app we are going to specify these commands in the ```docker-compose.yml``` file. + +1. Remove the port that maps from the web-app (it won't be directly accessible) +2. Add an extra NGINX container with custom configuration (proxy all traffic) +3. Configure NGINX to communicate with the web-app + + `docker-compose.yml` +{% highlight yaml %} +{% raw %} +version: '3' +services: + web: + image: ${{build-prj}} + auth: + image: ${{build-nginx}} + ports: + - 80 + links: + - web + environment: + USER: ${{USERNAME}} + PASS: ${{PASSWORD}} +{% endraw %} +{% endhighlight %} + +The architecture for the ```docker-compose.yml``` file is displayed in the diagram below. + +{% include +image.html +lightbox="true" +file="/images/examples/docker-https/codefresh_nginx_container.png" +url="/images/examples/docker-https/codefresh_nginx_container.png" +alt="codefresh_nginx_container.png" +max-width="40%" +%} + +{{site.data.callout.callout_info}} +##### Example + +Just head over to the example [__repository__](https://github.com/codefreshdemo/cf-example-basic-auth-container){:target="_blank"} in GitHub and follow the instructions there. +{{site.data.callout.end}} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/cd-examples/spring-boot-kafka-zookeeper.md b/_docs/example-catalog/cd-examples/spring-boot-kafka-zookeeper.md new file mode 100644 index 00000000..2134ff17 --- /dev/null +++ b/_docs/example-catalog/cd-examples/spring-boot-kafka-zookeeper.md @@ -0,0 +1,203 @@ +--- +title: "Spring Boot + Kafka + Zookeeper" +description: "" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/spring-boot-kafka-zookeeper/ + - /docs/on-demand-test-environment/example-compositions/spring-boot-kafka-zookeeper/ +toc: true +--- +This project uses `Java, Spring Boot, Kafka, Zookeeper` to show you how to integrate these services in the composition. + +{{site.data.callout.callout_info}} +##### Example + +Just head over to the example [__repository__](https://github.com/codefreshdemo/example-springboot-kafka){:target="_blank"} in GitHub and follow the instructions there. +{{site.data.callout.end}} + +## Zookeeper Docker image + +Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one + + `docker-compose.yml` +{% highlight yaml %} +{% raw %} + zookeeper: + image: wurstmeister/zookeeper + ports: + - "2181:2181" +{% endraw %} +{% endhighlight %} + +## Kafka Docker image +Now start the Kafka server. In the `docker-compose.yml` it can be something like this + + `docker-compose.yml` +{% highlight yaml %} +{% raw %} + kafka: + build: + context: kafka + dockerfile: Dockerfile + links: + - zookeeper:zk + ports: + - "9092:9092" + environment: + KAFKA_ADVERTISED_HOST_NAME: $CF_HOST_IP + KAFKA_ZOOKEEPER_CONNECT: zk:2181 + KAFKA_MESSAGE_MAX_BYTES: 2000000 + KAFKA_CREATE_TOPICS: "Topic1:1:1" + volumes: + - /var/run/docker.sock:/var/run/docker.sock + depends_on: + - zookeeper +{% endraw %} +{% endhighlight %} + +To start the Kafka server with the certain per-configuration, you need to use Environment variables. Below, you can see which Environment variables are available for this service. + +__Broker IDs__ + +You can configure the broker id in different ways: + +1. Explicitly, using ```KAFKA_BROKER_ID``` +2. Via a command, using ```BROKER_ID_COMMAND```, e.g. ```BROKER_ID_COMMAND: "hostname | awk -F'-' '{print $2}'"``` + +If you don't specify a broker id in your docker-compose file, it will automatically be generated (see [https://issues.apache.org/jira/browse/KAFKA-1070](https://issues.apache.org/jira/browse/KAFKA-1070){:target="_blank"}. This allows scaling up and down. In this case it is recommended to use the ```--no-recreate``` option of docker-compose to ensure that containers are not re-created and thus keep their names and ids. + + +__Automatically create topics__ + +If you want to have kafka-docker automatically create topics in Kafka during +creation, a ```KAFKA_CREATE_TOPICS``` environment variable can be +added in ```docker-compose.yml```. + +Here is an example snippet from ```docker-compose.yml```: + + environment: + KAFKA_CREATE_TOPICS: "Topic1:1:3,Topic2:1:1:compact" + +```Topic 1``` will have 1 partition and 3 replicas, ```Topic 2``` will have 1 partition, 1 replica and a `cleanup.policy` set to `compact`. + +__Advertised hostname__ + +You can configure the advertised hostname in different ways: + +1. Explicitly, using ```KAFKA_ADVERTISED_HOST_NAME``` +2. Via a command, using ```HOSTNAME_COMMAND```, e.g. ```HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"``` + +When using commands, make sure you review the "Variable Substitution" section in [https://docs.docker.com/compose/compose-file/](https://docs.docker.com/compose/compose-file/){:target="_blank"} + +If ```KAFKA_ADVERTISED_HOST_NAME``` is specified, it takes precedence over ```HOSTNAME_COMMAND``` + +For AWS deployment, you can use the Metadata service to get the container host's IP: +``` +HOSTNAME_COMMAND=wget -t3 -T2 -qO- http://169.254.169.254/latest/meta-data/local-ipv4 +``` +Reference: [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html){:target="_blank"} + +__JMX__ + +For monitoring purposes, you may wish to configure JMX. Additional to the standard JMX parameters, problems could arise from the underlying RMI protocol used to connect + +* java.rmi.server.hostname - interface to bind listening port. +* com.sun.management.jmxremote.rmi.port - the port to service RMI requests. + +For example, to connect to a kafka running locally (assumes exposing port 1099) + + KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=1099" + JMX_PORT: 1099 + +## Spring Boot + Kafka +Then grab the spring-kafka JAR and all of its dependencies - the easiest way to do that is to declare a dependency in your build tool, e.g. for Maven: + + `Text` +{% highlight xml %} +{% raw %} ++ +org.springframework.kafka +spring-kafka +${spring-kafka.version} ++ +{% endraw %} +{% endhighlight %} + +Using plain Java to send and receive a message: + + `Java` +{% highlight java %} +{% raw %} +private static String BOOT_TOPIC = "boot.t"; + +@Autowired +private Sender sender; + +@Autowired +private Receiver receiver; + +@ClassRule +public static KafkaEmbedded embeddedKafka = new KafkaEmbedded(1, true, BOOT_TOPIC); + +@BeforeClass +public static void setUpBeforeClass() throws Exception { + System.setProperty("spring.kafka.bootstrap-servers", embeddedKafka.getBrokersAsString()); +} + +@Test +public void testReceive() throws Exception { + sender.send(BOOT_TOPIC, "Hello Boot!"); + + receiver.getLatch().await(10000, TimeUnit.MILLISECONDS); + assertThat(receiver.getLatch().getCount()).isEqualTo(0); +} +{% endraw %} +{% endhighlight %} + +Maven will download the needed dependencies, compile the code and run the unit test case. The result should be a successful build during which following logs are generated: + + `Java` +{% highlight java %} +{% raw %} +. ____ _ __ _ _ + /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ +( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ + \\/ ___)| |_)| | | | | || (_| | ) ) ) ) + ' |____| .__|_| |_|_| |_\__, | / / / / + =========|_|==============|___/=/_/_/_/ + :: Spring Boot :: (v1.5.2.RELEASE) + +08:36:56.175 [main] INFO c.c.kafka.SpringKafkaApplicationTest - Starting SpringKafkaApplicationTest on cnf-pc with PID 700 (started by CodeNotFound in c:\code\st\spring-kafka\spring-kafka-avro) +08:36:56.175 [main] INFO c.c.kafka.SpringKafkaApplicationTest - No active profile set, falling back to default profiles: default +08:36:56.889 [main] INFO c.c.kafka.SpringKafkaApplicationTest - Started SpringKafkaApplicationTest in 1.068 seconds (JVM running for 5.293) +08:36:58.223 [main] INFO c.codenotfound.kafka.producer.Sender - sending user='{"name": "John Doe", "favorite_number": null, "favorite_color": "green"}' +08:36:58.271 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-L-1] INFO c.c.kafka.consumer.Receiver - received user='{"name": "John Doe", "favorite_number": null, "favorite_color": "green"}' +08:37:00.240 [main] ERROR o.a.zookeeper.server.ZooKeeperServer - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes +Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.871 sec - in com.codenotfound.kafka.SpringKafkaApplicationTest + +Results: + +Tests run: 3, Failures: 0, Errors: 0, Skipped: 0 + +[INFO] ------------------------------------------------------------------------ +[INFO] BUILD SUCCESS +[INFO] ------------------------------------------------------------------------ +[INFO] Total time: 41.632 s +[INFO] Finished at: 2017-04-17T08:37:31+02:00 +[INFO] Final Memory: 18M/212M +[INFO] ------------------------------------------------------------------------ +{% endraw %} +{% endhighlight %} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/cd-examples/terraform.md b/_docs/example-catalog/cd-examples/terraform.md new file mode 100644 index 00000000..0dd05f46 --- /dev/null +++ b/_docs/example-catalog/cd-examples/terraform.md @@ -0,0 +1,113 @@ +--- +title: "Deploy with Terraform" +description: "Use Terraform in a Codefresh pipeline with Docker" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +[Terraform](https://www.terraform.io/){:target="\_blank"} is a platform for *Infrastructure as Code*. It allows you to describe your cloud infrastructure in a declarative manner. + +You can use Terraform to deploy to Kubernetes or any other supported cloud platform. Because Terraform itself is already offered [in a Docker container](https://hub.docker.com/r/hashicorp/terraform/){:target="\_blank"}, it is very easy to run Terraform in a Codefresh pipeline. + + +{% include image.html +lightbox="true" +file="/images/examples/terraform/terraform-pipeline.png" +url="/images/examples/terraform/terraform-pipeline.png" +alt="Running Terraform inside Codefresh" +caption="Running Terraform inside Codefresh" +max-width="80%" +%} + +## The example Terraform project + +You can see the example project at [https://github.com/codefresh-contrib/terraform-sample-app](https://github.com/codefresh-contrib/terraform-sample-app){:target="\_blank"}. The repository contains a simple Terraform definition that creates a VM on Google cloud. + +You can play with it locally after installing the `terraform` executable. + +## Prerequisites + +You need to [create a Codefresh account]({{site.baseurl}}/docs/administration/create-a-codefresh-account/) and a Google account first. Then you need to create a [Service account Key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys){:target="\_blank"} which will allow terraform to communicate with Google cloud. + + +Add your service account json as a pipeline variable called `ACCOUNT_JSON_CONTENT`. The content of this variable will be used +in order to authenticate to Google cloud. + +## Create a CI/CD pipeline for Terraform + +Here is the whole pipeline: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - checkout + - prepare + - deploy +steps: + main_clone: + title: Cloning main repository... + stage: checkout + type: git-clone + repo: 'codefresh-contrib/terraform-sample-app' + revision: master + git: github + SetupAuth: + image: alpine:3.9 + title: Setting up Google cloud auth + stage: prepare + commands: + - echo $ACCOUNT_JSON_CONTENT > /codefresh/volume/account.json + - cf_export GOOGLE_CLOUD_KEYFILE_JSON=/codefresh/volume/account.json + DeployWithTerraform: + image: hashicorp/terraform:0.12.0 + title: Deploying Terraform plan + stage: deploy + commands: + - terraform init + - terraform apply -auto-approve + +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code through a [Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +1. Creates a pipeline variable with the path of the Google service account by running [cf_export]({{site.baseurl}}/docs/pipelines/variables/#exporting-environment-variables-from-a-freestyle-step). +1. Creates the VM on Google cloud by running `terraform init/apply`. + +>For simplicity, we auto-approve the Terraform plan in the example pipeline. In a production pipeline, you would instead use an [approval step]({{site.baseurl}}/docs/pipelines/steps/approval/) to inspect the plan before actually applying it. + +The pipeline needs a [single environment variable]({{site.baseurl}}/docs/pipelines/pipelines/#pipeline-settings) that holds the content of the service account. + + +{% include image.html +lightbox="true" +file="/images/examples/terraform/google_cloud_json.png" +url="/images/examples/terraform/google_cloud_json.png" +alt="Passing the Google account in the pipeline parameters" +caption="Passing the Google account in the pipeline parameters" +max-width="60%" +%} + + +Run the pipeline and see your deployment succeed. + + +Note that in a production pipeline you should also handle the [Terraform state](https://www.terraform.io/docs/state/){:target="\_blank"} in a proper manner. The example provided is using a file for [state storage](https://www.terraform.io/docs/backends/index.html){:target="\_blank"} which is not appropriate when using Terraform in a team environment. Instead you should use one of the [storage backends](https://www.terraform.io/docs/backends/types/index.html){:target="\_blank"} that support High Availability and Locking. + + + + +## Handling Pull requests + +You can easily use the same pipeline or a different one for pull requests. In this case replace the `terraform apply` command with `terraform plan`. Even better, you can add an [approval step]({{site.baseurl}}/docs/pipelines/steps/approval/) to allow humans to inspect the pipeline first. + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) diff --git a/_docs/example-catalog/cd-examples/transferring-php-ftp.md b/_docs/example-catalog/cd-examples/transferring-php-ftp.md new file mode 100644 index 00000000..56aa1d27 --- /dev/null +++ b/_docs/example-catalog/cd-examples/transferring-php-ftp.md @@ -0,0 +1,118 @@ +--- +title: "Deploy to VM via FTP" +description: "Deploying a PHP application to a VM using FTP" +group: example-catalog +sub_group: cd-examples +toc: true +redirect_from: + - /docs//learn-by-example/java/spring-mvc-jdbc-template/ +--- + +## Prerequisites + +- A [free Codefresh account]({{site.baseurl}}/docs/administration/account-management/create-a-codefresh-account/){:target="\_blank"} +- A remote machine with an FTP server and SSH setup (ensure that your FTP directory, I.e., `/srv/ftp/pub` has the proper write permissions for the FTP user). + +>Note that as you may already know, FTP is extremely insecure as it relies on plain-text passwords and usernames, making data very vulnerable to sniffing. A more secure solution would be to use SFTP or SCP. + +## Example PHP project + +The example project can be found on [GitHub](https://github.com/codefresh-contrib/ftp-php-app){:target="\_blank"}. The application is a simple PHP application that displays an example timer. + +{% include image.html +lightbox="true" +file="/images/examples/php-file-transfer/test-environment.png" +url="/images/examples/php-file-transfer/test-environment.png" +alt="Example PHP Application" +caption="Example PHP Application" +max-width="90%" +%} + +## Create the pipeline + +Our pipeline includes four stages: + +- A stage for cloning +- A stage for packaging +- A stage for transferring files + +{% include image.html +lightbox="true" +file="/images/examples/php-file-transfer/pipeline.png" +url="/images/examples/php-file-transfer/pipeline.png" +alt="Codefresh UI Pipeline View" +caption="Codefresh UI Pipeline View" +max-width="90%" +%} + +Here is the entire pipeline: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +# More examples of Codefresh YAML can be found at +# https://codefresh.io/docs/docs/example-catalog/ + +version: "1.0" +# Stages can help you organize your steps in stages +stages: + - "clone" + - "install" + - "transfer" +steps: + clone: + title: "Cloning main repository..." + type: "git-clone" + arguments: + repo: "codefresh-contrib/ftp-php-app" + git: "github" + stage: "clone" + install_dependencies: + title: "Collecting Php dependencies..." + type: "freestyle" + working_directory: "./ftp-php-app" + arguments: + image: "composer:1.9.3" + commands: + - "composer install --ignore-platform-reqs --no-interaction --no-plugins --no-scripts --prefer-dist" + stage: "install" + steps: + ftp_transfer: + title: "Transferring application to VM via ftp..." + type: "freestyle" + working_directory: "./ftp-php-app" + arguments: + image: "dockito/lftp-client:latest" + environment: + - USER=org.springframework.kafka +spring-kafka-test +${spring-kafka.version} +test ++ - PASSWORD= + - HOST= + - PUB_FTP_DIR= + commands: + - lftp -e "set ftp:use-site-utime2 false; mirror -x ^\.git/$ -X flat-logo.png -p -R ftp-php-ap $PUB_FTP_DIR/ftp-php-app; exit" -u $USER,$PASSWORD $HOST + stage: "transfer" +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the main repository through a [Git-clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +2. Installs the necessary PHP dependencies for our application through a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). +3. Transfers our application via FTP through another freestyle step. Note that you will need to change the environment variables to your respective values, either in the YAML itself (above), or through the pipeline settings: + +{% include image.html +lightbox="true" +file="/images/examples/php-file-transfer/variables.png" +url="/images/examples/php-file-transfer/variables.png" +alt="Codefresh Environment Variables" +caption="Codefresh Environment Variables" +max-width="90%" +%} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) + + diff --git a/_docs/example-catalog/cd-examples/trigger-a-k8s-deployment-from-docker-registry.md b/_docs/example-catalog/cd-examples/trigger-a-k8s-deployment-from-docker-registry.md new file mode 100644 index 00000000..c15084dd --- /dev/null +++ b/_docs/example-catalog/cd-examples/trigger-a-k8s-deployment-from-docker-registry.md @@ -0,0 +1,135 @@ +--- +title: "Trigger a Kubernetes Deployment from a Docker Hub Push Event" +description: "Learn how to trigger a Kubernetes deployment when an image is updated" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +In this example, we will cover how to trigger a Kubernetes deployment from a Dockerhub Push event using a Dockerhub [registry trigger]({{site.baseurl}}/docs/pipelines/triggers/dockerhub-triggers/#create-a-new-dockerhub-trigger). + +Our example has two pipelines: one for packaging code (CI), and the second for deploying code (CD). + +## Prerequisites + +- A [free Codefresh account](https://codefresh.io/docs/docs/getting-started/create-a-codefresh-account/) +- A DockerHub registry [connected to your Codefresh account]({{site.baseurl}}/docs/integrations/docker-registries/#docker-hub) +- A Kubernetes cluster [connected to your Codefresh account]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster) +- A service for your application [deployed to your cluster]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/#viewing-your-kubernetes-services) + +## Example Project + +You can see the example project on [GitHub](https://github.com/codefresh-contrib/registry-trigger-sample-app/tree/master){:target=\_blank"}. The repository contains a simple Hello World NodeJs app as well as 2 pipelines. + +## Create the CI Pipeline + +As mentioned before, our first pipeline will handle the CI process. +The pipeline has three stages: + +- A stage for cloning +- A stage for building the image +- A stage for pushing the image to DockerHub + +{% include image.html +lightbox="true" +file="/images/examples/deployments/k8s-deployment-ci-pipeline.png" +url="/images/examples/deployments/k8s-deployment-ci-pipeline.png" +alt="Codefresh UI CI Pipeline View" +caption="Codefresh UI CI Pipeline View" +max-width="90%" +%} + + `codefresh-CI-pipeline.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' + +stages: +- checkout +- build +- push + +steps: + clone: + title: Cloning main repository... + type: git-clone + stage: checkout + arguments: + repo: 'codefresh-contrib/registry-trigger-sample-app' + revision: 'master' + git: github + build_my_app: + title: Building image... + type: build + stage: build + arguments: + image_name: registry-trigger-sample-app + working_directory: ${{clone}} + tag: 'master' + dockerfile: Dockerfile + push_to_my_registry: + stage: 'push' + type: push + title: Pushing to Dockerhub... + arguments: + candidate: ${{build_my_app}} + tag: 'latest' + registry: dockerhub + image_name: annabaker/registry-trigger-sample-app +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code through a [Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +2. Builds a docker image tagged with the Application version through a [build step]({{site.baseurl}}/docs/pipelines/steps/build/). +3. Pushes the Docker image through a [push step](https://codefresh.io/docs/docs/pipelines/steps/push/) to the Docker Hub registry you have integrated with Codefresh. + +## Create the CD Pipeline + +This pipeline contains one stage/step, for deploying. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/k8s-deployment-CD-pipeline.png" +url="/images/examples/deployments/k8s-deployment-CD-pipeline.png" +alt="Codefresh UI CD Pipeline View" +caption="Codefresh UI CD Pipeline View" +max-width="90%" +%} + +Note that for the trigger mechanism to take place, you will need to [add a Docker Hub registry trigger]({{site.baseurl}}/docs/pipelines/triggers/dockerhub-triggers/#create-a-new-dockerhub-trigger) to the pipeline. + + `codefresh-CD-pipeline.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" + +stages: + - "deploy" + +steps: + deploy_to_k8s: + title: Running Deploy Script... + type: deploy + kind: kubernetes + arguments: + cluster: anna-demo@FirstKubernetes + namespace: default + service: registry-trigger-sample-app + candidate: + image: annabaker/registry-trigger-sample-app:latest + registry: 'dockerhub' +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Deploys the image to Kubernetes through a [deploy step]({{site.baseurl}}/docs/pipelines/steps/deploy/). The deploy step uses a [Registry trigger]({{site.baseurl}}/docs/pipelines/triggers/dockerhub-triggers/#create-a-new-dockerhub-trigger) to kick off the pipeline when the updated image is pushed to the registry. + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) +[Triggers in pipelines]({{site.baseurl}}/docs/pipelines/triggers/) diff --git a/_docs/example-catalog/cd-examples/use-kubectl-as-part-of-freestyle-step.md b/_docs/example-catalog/cd-examples/use-kubectl-as-part-of-freestyle-step.md new file mode 100644 index 00000000..b228a895 --- /dev/null +++ b/_docs/example-catalog/cd-examples/use-kubectl-as-part-of-freestyle-step.md @@ -0,0 +1,42 @@ +--- +title: "Use kubectl as part of freestyle step" +description: "How to run manually kubectl in a Codefresh pipeline" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/use-kubectl-as-part-of-freestyle-step/ +toc: true +--- + + +Running Kubernetes commands in Codefresh as part of the workflow is very easy. + + +Codefresh is adding all your clusters into the workflow ready to be used as part of your CI/CD pipeline. +The context remains the same as it appears in the [Codefresh Kubernetes dashboard]({{site.baseurl}}/docs/deploy-to-kubernetes/manage-kubernetes/). + +>If your cluster name includes spaces then make sure that you use quotes in the `kubectl` command. + +* Use image: `codefresh/kubectl` +* Add your commands: + * `kubectl config get-contexts`. Will print the cluster that we added to the workflow + * `kubectl config use-context "my-cluster-name"`. The name is the same as in `Account settings` → `Integrations` → `Kubernetes` + * `kubectl get po -owide` + * `kubectl get nodes` + + +## Follow the example + +* Add this [Git repo](https://github.com/Codefresh-Examples/kubectl-in-freestyle-step){:target="_blank"} to your account +* Change the pipeline configuration to use `codefresh.yml`. +* Build. + +## Running parallel steps with kubectl + +More complex examples can be found in the [custom kubectl commands]({{site.baseurl}}/docs/deploy-to-kubernetes/custom-kubectl-commands/) documentation page. + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/cd-examples/web-terminal.md b/_docs/example-catalog/cd-examples/web-terminal.md new file mode 100644 index 00000000..37151528 --- /dev/null +++ b/_docs/example-catalog/cd-examples/web-terminal.md @@ -0,0 +1,48 @@ +--- +title: "Web terminal" +description: "" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/web-terminal/ + - /docs/on-demand-test-environment/example-compositions/web-terminal/ +toc: true +--- +This example shows you how to access containers running in a Codefresh standup environment. + +## Looking around +In the root of this repository you'll find a file named `docker-compose.yml`. +Here are the contents of this file: + + `Composition.yml` +{% highlight yaml %} +version: '3' +services: + my-service: + image: 'containers101/whomi:master' + volumes: + - my-service:/app + ports: + - '1337' + terminal: + image: 'containers101/cfterminal:master' + ports: + - '8000' + volumes_from: + - my-service +volumes: + my-service: + driver: local +{% endhighlight %} + +{{site.data.callout.callout_info}} +##### Example + +Just head over to the example [__repository__](https://github.com/codefreshdemo/cf-example-web-termial){:target="_blank"} in GitHub and follow the instructions there. +{{site.data.callout.end}} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/ci-examples/android.md b/_docs/example-catalog/ci-examples/android.md new file mode 100644 index 00000000..a02d66b1 --- /dev/null +++ b/_docs/example-catalog/ci-examples/android.md @@ -0,0 +1,80 @@ +--- +title: "Compile and package an Android application" +description: "Using Codefresh pipelines" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +Android applications use Java/Gradle for their build system. Because Codefresh already supports [Gradle]({{site.baseurl}}/docs/example-catalog/ci-examples/gradle/), it is also very easy to build Android projects. + +Any Gradle command can run inside a Docker image that contains the Android SDK. As an example, we will use a [Nextcloud](https://hub.docker.com/r/nextcloudci/android){:target="\_blank"} image from Dockerhub. + + +## The example project + +You can see the example project at [https://github.com/codefresh-contrib/android-sample-app](https://github.com/codefresh-contrib/android-sample-app){:target="\_blank"}. The repository contains a Hello World Android project with the following tasks: + +* `./gradlew test` runs unit tests +* `./gradlew build` builds the application + + +## Create a CI pipeline that compiles/releases Android + +In most cases you would create a similar pipeline to a Gradle project. + +{% include image.html +lightbox="true" +file="/images/learn-by-example/mobile/android-ci-pipeline.png" +url="/images/learn-by-example/mobile/android-ci-pipeline.png" +alt="Building and Testing an Android app" +caption="Building and Testing an Android app" +max-width="80%" +%} + +Here is the [full pipeline](https://github.com/codefresh-contrib/android-sample-app/blob/master/codefresh.yml){:target="\_blank"} that uses a Docker image with the Android SDK in order to run Gradle. + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - test + - build +steps: + main_clone: + title: Cloning main repository... + stage: prepare + type: git-clone + repo: 'codefresh-contrib/android-sample-app' + revision: master + git: github + TestIt: + title: Running Tests + stage: test + image: nextcloudci/android:android-48 + commands: + - chmod +x ./gradlew + - ./gradlew test --no-daemon --gradle-user-home=/codefresh/volume/.gradle + BuildIt: + title: Packaging Android App + stage: build + image: nextcloudci/android:android-48 + commands: + - ./gradlew build --no-daemon --gradle-user-home=/codefresh/volume/.gradle +{% endraw %} +{% endhighlight %} + +This pipeline clones the source code, runs unit tests and finally builds the Android application. + +Codefresh is smart enough that [caches automatically]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/#how-caching-works-in-codefresh) for us the workspace of a build (`/codefresh/volume`). This works great for build tools that keep their cache in the project folder, but not for Maven/Gradle which keep their cache externally. By changing the location of the Gradle cache we make sure that Codefresh will cache automatically the Gradle libraries resulting in much faster builds. + + + +## Related articles +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) + diff --git a/_docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository.md b/_docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository.md new file mode 100644 index 00000000..d81e5363 --- /dev/null +++ b/_docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository.md @@ -0,0 +1,94 @@ +--- +title: "Build an Image from a different Git repository" +description: "Build microservices from other repositories" +group: example-catalog +sub_group: ci-examples +redirect_from: + - /docs/build-an-image-from-a-different-git-repository/ +toc: true +--- + +In most cases, your Codefresh pipeline checks out a single Git repository. Codefresh has great support also for [monorepos]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/#using-the-modified-files-field-to-constrain-triggers-to-specific-folderfiles) if you have placed all your applications in a single repository. + +A Codefresh pipeline is not really tied to a specific Git repository, which means that by [checking out multiple Git repositories]({{site.baseurl}}/docs/example-catalog/ci-examples/git-checkout/#cloning-multiple-repositories) you can build Docker images from other unrelated repositories in a single pipeline if you wish to do so. + +## Building Docker images from other Git repositories + + +Here is a Codefresh pipeline that checks out two microservices from two different Git repositories. + +{% include image.html +lightbox="true" +file="/images/examples/docker-build/build-from-other-git-repo.png" +url="/images/examples/docker-build/build-from-other-git-repo.png" +alt="Checkout and build docker images" +caption="Checkout and build docker images" +max-width="100%" +%} + +And here is the [pipeline definition]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/). + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - 'clone phase' + - 'build phase' +steps: + checkoutApp1: + title: 'Cloning first repository...' + type: git-clone + repo: kostis-codefresh/example_nodejs_postgres + revision: experiment1 + git: github + stage: 'clone phase' + checkoutApp2: + title: 'Cloning second repository...' + type: git-clone + repo: kostis-codefresh/trivial-go-web + revision: master + git: github + stage: 'clone phase' + myFirstDockerImage: + title: 'Building Microservice A' + type: build + dockerfile: Dockerfile + image_name: my-nodejs-image + tag: from-develop-branch + working_directory: './example_nodejs_postgres' + stage: 'build phase' + mySecondDockerImage: + title: 'Building Microservice B' + type: build + dockerfile: Dockerfile + working_directory: './trivial-go-web' + image_name: my-app-image + tag: from-master-branch + stage: 'build phase' +{% endraw %} +{% endhighlight %} + +The pipeline first checks out two different Git repositories, which themselves contain Dockerfiles. Then it creates a Docker image for each one using the respective Dockerfile. + +You can see both images in the [Docker image dashboard]({{site.baseurl}}/docs/docker-registries/#viewing-docker-images) . + +{% include image.html +lightbox="true" +file="/images/examples/docker-build/two-docker-images.png" +url="/images/examples/docker-build/two-docker-images.png" +alt="Docker images from other Git repos" +caption="Docker images from other Git repos" +max-width="100%" +%} + + +Notice that there are no explicit push steps in the pipeline, as all successful Codefresh pipelines automatically push to the private Docker registry. + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/) +[Build step in pipelines in pipelines]({{site.baseurl}}/docs/pipelines/steps/build/) +[Build and Push an image]({{site.baseurl}}/docs/pipelines/examples/build-and-push-an-image/) +[Parallel pipelines]({{site.baseurl}}/docs/pipelines/advanced-workflows/) diff --git a/_docs/example-catalog/ci-examples/build-an-image-specify-dockerfile-location.md b/_docs/example-catalog/ci-examples/build-an-image-specify-dockerfile-location.md new file mode 100644 index 00000000..75d5b67f --- /dev/null +++ b/_docs/example-catalog/ci-examples/build-an-image-specify-dockerfile-location.md @@ -0,0 +1,74 @@ +--- +title: "Build an Image by specifying a Dockerfile location" +description: "How to choose a Dockerfile to build with Codefresh pipelines" +group: example-catalog +sub_group: ci-examples +redirect_from: + - /docs/build-an-image-specify-dockerfile-location/ +toc: true +--- + +You may have a project where the Dockerfile is **not** in the root folder of the project. Maybe the repository has multiple projects inside, each with its own Dockerfile, or you simply want to use a different folder for the Docker context. + +>The source code of the repository is at [https://github.com/codefreshdemo/cf-example-dockerfile-other-location](https://github.com/codefreshdemo/cf-example-dockerfile-other-location){:target="\_blank"}. Feel free to fork it if you want to follow along. + +If you don't have a Codefresh account already, you can easily create a free one from the [sign-up page]({{site.baseurl}}/docs/administration/create-a-codefresh-account/). + + +## Building a Dockerfile from a different folder + +By default, if you run a single command like the one below, Docker uses the Dockerfile of the current folder: + +``` +docker build . -t my-web-app +``` + +If your Dockerfile is in a different folder, specify it explicitly with: + +``` +docker build . -t my-web-app -f subfolder/Dockerfile +``` + +Codefresh supports a similar syntax as well. The `dockerfile` property of the [build step]({{site.baseurl}}/docs/pipelines/steps/build/) can accept a full path. + +Here is the full pipeline: + + `codefresh.yml` +{% highlight yaml %} +version: '1.0' +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'codefreshdemo/cf-example-dockerfile-other-location' + revision: 'master' + git: github + build_my_app: + title: Building Node.Js Docker Image + type: build + image_name: my-app + working_directory: '.' + tag: 'master' + dockerfile: docker/Dockerfile +{% endhighlight %} + +This pipeline checks out the source code of the repository and then builds a Dockerfile found at the subfolder `docker` while still keeping as Docker context the root directory. + +{% include image.html +lightbox="true" +file="/images/examples/docker-build/build-spefify-dockerfile.png" +url="/images/examples/docker-build/build-spefify-dockerfile.png" +alt="Building a Docker image with specific Dockerfile" +caption="Building a Docker image with specific Dockerfile" +max-width="100%" +%} + +You could also change the Docker build context by editing the `working_directory` property. By default, it looks at the root folder of the project, but any subfolder path is also valid. + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Build step in pipelines]({{site.baseurl}}/docs/pipelines/steps/build/) +[Build an Image with the Dockerfile in root directory]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-dockerfile-in-root-directory/) +[Build an Image from a different Git repository]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository) +[Build and push an Image]({{site.baseurl}}/docs/yaml-examples/example-catalog/ci-examples/build-and-push-an-image) +[Build an Image With build arguments]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-with-build-arguments) \ No newline at end of file diff --git a/_docs/example-catalog/ci-examples/build-an-image-with-build-arguments.md b/_docs/example-catalog/ci-examples/build-an-image-with-build-arguments.md new file mode 100644 index 00000000..a7a62356 --- /dev/null +++ b/_docs/example-catalog/ci-examples/build-an-image-with-build-arguments.md @@ -0,0 +1,133 @@ +--- +title: "Build an Image with build arguments" +description: "Use Docker arguments in Codefresh pipelines" +group: example-catalog +sub_group: ci-examples +redirect_from: + - /docs/build-an-image-with-build-arguments/ +toc: true +--- + +Building a Docker image that requires build arguments is very easy with Codefresh pipelines. + +The source code of the repository is at [https://github.com/codefreshdemo/cf-example-build-arguments](https://github.com/codefreshdemo/cf-example-build-arguments){:target="\_blank"}. Feel free to fork it if you want to follow along. + +If you don't have a Codefresh account already, you can easily create a free one from the [sign-up page]({{site.baseurl}}/docs/administration/create-a-codefresh-account/). + +## Using Docker build arguments + +The example application is a very simple NodeJS application with the following DYouockerfile: + +`Dockerfile` +{% highlight docker %} +{% raw %} +ARG NODE_VERSION +FROM node:$NODE_VERSION + +ARG APP_DIR + +RUN mkdir -p $APP_DIR + +WORKDIR $APP_DIR + +COPY package.json . +RUN npm install --silent +COPY . . +EXPOSE 3000 + +ENV PORT 3000 + +CMD [ "npm", "start" ] +{% endraw %} +{% endhighlight %} + +This Dockerfile expects two [build arguments](https://docs.docker.com/engine/reference/builder/#/arg){:target="\_blank"}: + +* `NODE_VERSION` is the version of Node image to use as base +* `APP_DIR` is the source directory to be used inside the container + +## Building a Dockerfile passing values for build arguments + +When you build an image locally on your workstation, you can define build arguments with the `--build-arg` syntax: + +``` +docker build . -t my-node-app --build-arg NODE_VERSION=8 --build-arg APP_DIR=/usr/src/app +``` + +You can get the same result within a Codefresh pipeline: + + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'codefreshdemo/cf-example-build-arguments' + revision: 'master' + git: github + build_my_app: + title: Building Node.Js Docker Image + type: build + image_name: my-app + working_directory: '.' + tag: 'master' + dockerfile: Dockerfile + build_arguments: + - NODE_VERSION=8 + - APP_DIR=/usr/src/app +{% endraw %} +{% endhighlight %} + +This pipeline checks out the source code of the repository and then builds the Dockerfile by passing the values `8` and `/usr/src/app` to the two arguments. + +{% include image.html +lightbox="true" +file="/images/examples/docker-build/docker-build-arguments.png" +url="/images/examples/docker-build/docker-build-arguments.png" +alt="Using Docker build arguments in a pipeline" +caption="Using Docker build arguments in a pipeline" +max-width="100%" +%} + +## Using Codefresh variables as build arguments + +In the previous pipeline, the Docker build arguments are defined in the pipeline itself, but you can also use [pipeline variables]({{site.baseurl}}/docs/pipelines/pipelines/#creating-new-pipelines), [shared configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/), or any other standard mechanism you already have in place. + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'codefreshdemo/cf-example-build-arguments' + revision: 'master' + git: github + build_my_app: + title: Building Node.Js Docker Image + type: build + image_name: my-app + working_directory: '.' + tag: 'master' + dockerfile: Dockerfile + build_arguments: + - NODE_VERSION=${{NODE_VERSION_FROM_SHARED_CONFIG}} + - APP_DIR=${{APP_DIR_PIPELINE_VARIABLE}} +{% endraw %} +{% endhighlight %} + +In this case, you can also use any of the built-in [Codefresh variables]({{site.baseurl}}/docs/pipelines/variables/). + + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Build step in pipelines]({{site.baseurl}}/docs/pipelines/steps/build/) +[Build an Image with the Dockerfile in root directory]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-dockerfile-in-root-directory/) +[Build an Image by specifying the Dockerfile location]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-specify-dockerfile-location) +[Build an Image from a different Git repository]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository) +[Build and push an Image]({{site.baseurl}}/docs/yaml-examples/example-catalog/ci-examples/build-and-push-an-image) diff --git a/_docs/example-catalog/ci-examples/build-an-image-with-the-dockerfile-in-root-directory.md b/_docs/example-catalog/ci-examples/build-an-image-with-the-dockerfile-in-root-directory.md new file mode 100644 index 00000000..a9c5cb2e --- /dev/null +++ b/_docs/example-catalog/ci-examples/build-an-image-with-the-dockerfile-in-root-directory.md @@ -0,0 +1,67 @@ +--- +title: "Build an Image with the Dockerfile in root directory" +description: "Get started quickly with building Docker images" +group: example-catalog +sub_group: ci-examples +toc: true +--- +Building a Docker image is one of the basic operations in Codefresh pipelines. + +>The source code of the repository is at [https://github.com/codefreshdemo/cf-yml-example-build-dockerfile-inroot](https://github.com/codefreshdemo/cf-yml-example-build-dockerfile-inroot){:target="\_blank"}. Feel free to fork it if you want to follow along. + +If you don't have a Codefresh account already, you can easily create a free one from the [sign-up page]({{site.baseurl}}/docs/administration/create-a-codefresh-account/). + + +## Building a Dockerfile from the root folder + +By default, if you run a single command like the one below, Docker uses the Dockerfile of the current folder: + +``` +docker build . -t my-web-app +``` + +You can get the same result within a Codefresh pipeline: + + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'codefreshdemo/cf-yml-example-build-dockerfile-inroot' + revision: 'master' + git: github + build_my_app: + title: Building Node.Js Docker Image + type: build + image_name: my-app + working_directory: '${{main_clone}}' + tag: 'master' + dockerfile: Dockerfile +{% endraw %} +{% endhighlight %} + +This pipeline checks out the source code of the repository and then builds a dockerfile found at the root folder of the project. + +{% include image.html +lightbox="true" +file="/images/examples/docker-build/build-dockerfile-root.png" +url="/images/examples/docker-build/build-dockerfile-root.png" +alt="Building a Docker image with a default Dockerfile" +caption="Building a Docker image with a default Dockerfile" +max-width="100%" +%} + +You can also change the Docker build context by editing the `working_directory` property. By default, it looks at the root folder of the project, but any subfolder path is also valid. + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Build step in pipelines]({{site.baseurl}}/docs/pipelines/steps/build/) +[Build an Image by specifying the Dockerfile location]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-specify-dockerfile-location) +[Build an Image from a different Git repository]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository) +[Build and push an Image]({{site.baseurl}}/docs/yaml-examples/example-catalog/ci-examples/build-and-push-an-image) +[Build an Image With build arguments]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-with-build-arguments) diff --git a/_docs/example-catalog/ci-examples/build-and-push-an-image.md b/_docs/example-catalog/ci-examples/build-and-push-an-image.md new file mode 100644 index 00000000..33ebac63 --- /dev/null +++ b/_docs/example-catalog/ci-examples/build-and-push-an-image.md @@ -0,0 +1,137 @@ +--- +title: "Build and push an Image" +description: "Build Docker images and push them to registries with Codefresh" +group: example-catalog +sub_group: ci-examples +redirect_from: + - /docs/build-and-push-an-image/ + - /docs/docker-registries/push-image-to-a-docker-registry/ +toc: true +--- + +Building a Docker image and then pushing it to a registry is one of the most basic scenarios for creating a pipeline. +In this example we will use a demo Node.js application that will be packaged in a Docker image. + +The source code of the repository is at [https://github.com/codefreshdemo/cf-example-build-and-push](https://github.com/codefreshdemo/cf-example-build-and-push){:target="\_blank"}. Feel free to fork it if you want to follow along. + +If you don't have a Codefresh account already, you can easily create a free one from the [sign-up page]({{site.baseurl}}/docs/administration/create-a-codefresh-account/). + + +## Building and push Docker image to default registry + +Building a Docker image with Codefresh is easy, and only requires a simple step. In addition, all successful pipelines in Codefresh automatically push to [your default Docker registry]({{site.baseurl}}/docs/docker-registries/#the-default-registry), without additional configuration, if you have one. + +Here is the most basic pipeline that clones a repo and builds an image: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: +- checkout +- build +steps: + clone: + title: Cloning main repository... + type: git-clone + stage: checkout + repo: 'codefreshdemo/cf-example-build-and-push' + revision: 'master' + git: github + build_my_app: + title: Building Node.Js Docker Image + type: build + stage: build + image_name: my-node-js-app + working_directory: {{clone}} + tag: 'master' + dockerfile: Dockerfile +{% endraw %} +{% endhighlight %} + +## Building and pushing Docker image to _any registry_. + +You can push your image to any [registry]({{site.baseurl}}/docs/docker-registries/). + +* First you need to connect your external registry in the integrations page. Here are the instructions for: + + * [Docker Hub]({{site.baseurl}}/docs/integrations/docker-registries/docker-hub/) + * [Google Container Registry]({{site.baseurl}}/docs/integrations/docker-registries/google-container-registry/) + * [Amazon EC2 Container Registry]({{site.baseurl}}/docs/integrations/docker-registries/amazon-ec2-container-registry/) + * [Bintray.io]({{site.baseurl}}/docs/integrations/docker-registries/bintray-io/) + * [Quay.io]({{site.baseurl}}/docs/integrations/docker-registries/quay-io/) + * [Other Registries]({{site.baseurl}}/docs/integrations/docker-registries/other-registries/) + +* Then add a [push step]({{site.baseurl}}/docs/pipelines/steps/push/) in your pipeline and use the registry name of your integration. + +Here is the full example: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: +- checkout +- build +- push +steps: + clone: + title: Cloning main repository... + type: git-clone + stage: checkout + repo: 'codefreshdemo/cf-example-build-and-push' + revision: 'master' + git: github + build_my_app: + title: Building Node.Js Docker Image + type: build + stage: build + image_name: my-node-js-app + working_directory: {{clone}} + tag: 'master' + dockerfile: Dockerfile + push_to_my_registry: + stage: 'push' + type: push + title: Pushing to a registry + candidate: ${{build_my_app}} + tag: 'v1.0.0' + registry: dockerhub + image_name: kkapelon/my-node-js-app +{% endraw %} +{% endhighlight %} + +Here we use a specific tag - `v1.0.0` but +Codefresh has several variables that you can use to tag images. Common examples are `CF_BRANCH_TAG_NORMALIZED`, `CF_SHORT_REVISION` or `CF_BUILD_ID`. Read more on [variables]({{site.baseurl}}/docs/pipelines/variables/). + +{% include image.html + lightbox="true" + file="/images/examples/docker-build/build-and-push-pipeline.png" + url="/images/examples/docker-build/build-and-push-pipeline.png" + alt="Pushing image to external registry" + caption="Pushing image to external registry" + max-width="100%" + %} + + +If you run the pipeline, the Docker image is pushed *both* to the private Docker regisry (by the build step) *and* the external docker registry (by the push step). + + +## More options for pushing images + +Codefresh has several options when it comes to pushing images: + +* You can specify multiple tags to be pushed +* You can use directly ECR registries +* You can embed credentials in the push steps + +Read more in [push steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/push/). + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Build step in pipelines]({{site.baseurl}}/docs/pipelines/steps/build/) +[Build an Image with the Dockerfile in root directory]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-dockerfile-in-root-directory/) +[Build an Image by specifying the Dockerfile location]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-specify-dockerfile-location) +[Build an Image from a different Git repository]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository) +[Build an Image With Build arguments]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-with-build-arguments) diff --git a/_docs/example-catalog/ci-examples/c-make.md b/_docs/example-catalog/ci-examples/c-make.md new file mode 100644 index 00000000..06b95d76 --- /dev/null +++ b/_docs/example-catalog/ci-examples/c-make.md @@ -0,0 +1,74 @@ +--- +title: "Compile and test a C application" +description: "Using Codefresh pipelines" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +Codefresh can work with any C/C++ application very easily as both `gcc` and `g++` are already offered in Dockerhub. There is also another example available with [C++ and cmake]({{site.baseurl}}/docs/example-catalog/ci-examples/cpp-cmake). + +## The example C project + +You can see the example project at [https://github.com/codefresh-contrib/c-sample-app](https://github.com/codefresh-contrib/c-sample-app){:target="\_blank"}. The repository contains a C starter project with a `Makefile` and several targets: + +* `make` compiles the code. +* `make test` runs unit tests +* `make clean` removes artifacts and binaries. + +There are also extra targets for `tags` and `etags`. + +## Create a CI pipeline for C applications + +Creating a CI/CD pipeline for C is very easy, because Codefresh can run any [gcc image](https://hub.docker.com/_/gcc/){:target="\_blank"} that you wish. Gcc docker images already contain the `make` utility. + +{% include image.html +lightbox="true" +file="/images/learn-by-example/cc/c-make-pipeline.png" +url="/images/learn-by-example/cc/c-make-pipeline.png" +alt="Compiling a C application in a pipeline" +caption="Compiling a C application in a pipeline" +max-width="80%" +%} + +Here is the [full pipeline](https://github.com/codefresh-contrib/c-sample-app/blob/master/codefresh.yml){:target="\_blank"} that compiles the application after checking out the code. + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - checkout + - build +steps: + main_clone: + title: Cloning main repository... + stage: checkout + type: git-clone + repo: 'codefresh-contrib/c-sample-app' + revision: master + git: github + compile_my_sources: + title: Compile + stage: build + image: gcc + commands: + - make + run_my_tests: + title: Test + stage: build + image: gcc + commands: + - make test +{% endraw %} +{% endhighlight %} + +This pipeline clones the source code, compiles the code and runs unit tests. In all cases we use the public Docker image of Gcc that also contains `make`. + + +## Related articles +[C++ example]({{site.baseurl}}/docs/example-catalog/ci-examples/cpp-cmake/) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/ci-examples/call-child-pipelines.md b/_docs/example-catalog/ci-examples/call-child-pipelines.md new file mode 100644 index 00000000..fd83b5b7 --- /dev/null +++ b/_docs/example-catalog/ci-examples/call-child-pipelines.md @@ -0,0 +1,108 @@ +--- +title: "Call a CD pipeline from a CI pipeline" +description: "How to call child pipelines from a parent pipeline" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +In Codefresh you can easily create nested pipelines by calling other pipelines from within an existing pipeline. The [codefresh-run plugin](https://codefresh.io/steps/step/codefresh-run){:target="\_blank"} allows you to launch another pipeline, and optionally wait for its completion. + +{% include image.html +lightbox="true" +file="/images/examples/nested-pipelines/call-other-pipeline.png" +url="/images/examples/nested-pipelines/call-other-pipeline.png" +alt="Parent and child pipelines" +caption="Parent and child pipelines" +max-width="80%" +%} + +A very common pattern in Codefresh is to have a parent pipeline responsible for Continuous Integration (packaging code), that calls a child pipeline for Continuous Delivery (taking care of deployment). + +## Example project + +You can see the example project at [https://github.com/codefresh-contrib/call-child-pipeline-sample-app](https://github.com/codefresh-contrib/call-child-pipeline-sample-app){:target="\_blank"}. The repository contains a NodeJs app as well as three - one parent and two child pipelines. + +## Create a pipeline that calls other pipelines + +Here is the definition of the parent pipeline: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - package + - deploy +steps: + main_clone: + title: 'Cloning main repository...' + type: git-clone + repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}' + revision: '${{CF_REVISION}}' + git: github + stage: prepare + read_my_app_version: + title: Reading Application version + stage: prepare + image: node:latest + commands: + - export PACKAGE_VERSION=$(node -p "require('./package.json').version") + - cf_export PACKAGE_VERSION + build_my_docker_image: + title: 'Building My Docker Image' + stage: package + type: build + dockerfile: Dockerfile + image_name: my-app-image + tag: ${{PACKAGE_VERSION}} + call_qa_pipeline: + title: Deploy to QA + stage: deploy + type: codefresh-run + arguments: + PIPELINE_ID: child-pipelines/qa-pipeline + VARIABLE: + - CF_BRANCH=${{CF_BRANCH}} + - CF_REVISION=${{CF_REVISION}} + - APP_VERSION=${{PACKAGE_VERSION}} + when: + branch: + only: + - develop + call_prod_pipeline: + title: Deploy to Prod + stage: deploy + type: codefresh-run + arguments: + PIPELINE_ID: child-pipelines/prod-pipeline + VARIABLE: + - CF_BRANCH=${{CF_BRANCH}} + - CF_REVISION=${{CF_REVISION}} + - APP_VERSION=${{PACKAGE_VERSION}} + when: + branch: + only: + - /^release.*/i + + +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code through a [Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +1. Creates a variable that contains the Application version as specified in `package.json` through [cf_export]({{site.baseurl}}/docs/pipelines/variables/#exporting-environment-variables-from-a-freestyle-step). +1. Builds a docker image tagged with the Application version through a [build step]({{site.baseurl}}/docs/pipelines/steps/build/). +1. Optionally runs the downstream QA pipeline if the branch is named `develop`. It also passes several environment variables to the child pipeline (including the Application version). +1. Optionally runs the downstream Prod pipeline if the branch name starts with `release`. It also passes several environment variables to the child pipeline (including the Application version). + +The last two steps use [conditions]({{site.baseurl}}/docs/pipelines/conditional-execution-of-steps/) to decide if they will run or not. + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[Pipeline plugins](https://codefresh.io/steps/){:target="\_blank"} \ No newline at end of file diff --git a/_docs/example-catalog/ci-examples/cc.md b/_docs/example-catalog/ci-examples/cc.md new file mode 100644 index 00000000..c23c08fc --- /dev/null +++ b/_docs/example-catalog/ci-examples/cc.md @@ -0,0 +1,10 @@ +--- +title: "C/C++" +description: "How to build C/C++ applications with Codefresh CI/CD pipelines" +group: example-catalog +toc: true +--- +This section contains Codefresh examples based on C and C++. + +- [C Example with make]({{site.baseurl}}/docs/learn-by-example/cc/c-make) +- [C++ Example with cmake]({{site.baseurl}}/docs/learn-by-example/cc/cpp-cmake) \ No newline at end of file diff --git a/_docs/example-catalog/ci-examples/codacy-testing.md b/_docs/example-catalog/ci-examples/codacy-testing.md new file mode 100644 index 00000000..bd2e437b --- /dev/null +++ b/_docs/example-catalog/ci-examples/codacy-testing.md @@ -0,0 +1,174 @@ +--- +title: "Codacy coverage reports" +description: "How to forward coverage reports to Codacy" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +[Codacy](https://www.codacy.com/){:target="\_blank"} is a code review tool that allows automatic analysis, code coverage tracking, and extensive reports, for you and your team to improve your code quality over time. + +Analysis reports displayed within Codacy dashboard: +{% include image.html +lightbox="true" +file="/images/testing/codacy/codacy-report.png" +url="/images/testing/codacy/codacy-report.png" +alt="Codacy UI with coverage reports" +max-width="100%" +%} + +## Prerequisites for using Codacy + +* A simple [Codefresh pipeline, up and running]({{site.baseurl}}/docs/getting-started/create-a-basic-pipeline/) +* A [Codacy account](https://www.codacy.com/){:target="\_blank"} (free, pro or enterprise) +* A testing tool added to your project that produces coverage reports + +Codacy supports over [30 different language integrations](https://docs.codacy.com/getting-started/supported-languages-and-tools/){:target="\_blank"}. Depending on the programming language used, it requires little to no set-up. + +You could try it out by cloning our [node example application](https://github.com/codefresh-contrib/codacy-sample-app){:target="\_blank"} that utilises [jest](https://jestjs.io/){:target="\_blank"}. + +## Create an account with Codacy +Codacy has a free version, a pro version, and an on-premises version. The latter two have a free trial, which allows you to test all features over the course of two weeks. You can sign-up via GitHub, Bitbucket, or GitLab. + +When you log into Codacy for the first time, it will ask you to provide access to a repository. At this stage, Codacy will not download any code from your repository but merely access its names. You can then either provide access to selective repositories or your entire git account. + +{% include image.html +lightbox="true" +file="/images/testing/codacy/codacy-add-repo.png" +url="/images/testing/codacy/codacy-add-repo.png" +alt="Add repository to codacy" +max-width="80%" +%} + +## Generate Project API token +To use Codacy, we need a project API token. To generate the token, select your project => go to settings => integrations => add integration => select “Project API”. Make sure that you select the API token from here and not your general project settings. + +{% include image.html +lightbox="true" +file="/images/testing/codacy/create-api-token.png" +url="/images/testing/codacy/create-api-token.png" +alt="Create Project API token" +max-width="80%" +%} + +## Codefresh pipeline + +In case the project that you want to use Codacy in does not have a pipeline, [create a new pipeline]({{site.baseurl}}/docs/getting-started/create-a-basic-pipeline/). + +{% include image.html +lightbox="true" +file="/images/testing/codacy/create-codacy-pipeline.png" +url="/images/testing/codacy/create-codacy-pipeline.png" +alt="Create Codacy Pipeline" +max-width="80%" +%} + +**Setting-up step** + +This step is based on our [TypeScript application](https://github.com/codefresh-contrib/codacy-sample-app){:target="\_blank"}. Before we set up our pipeline, we will add our Project API token as our environment variable. Note that we have specified our token in the variables section on the right, as displayed in the following screenshot. + +{% include image.html +lightbox="true" +file="/images/testing/codacy/codacy-variable.png" +url="/images/testing/codacy/codacy-variable.png" +alt="Provide Codacy ENV variable" +max-width="80%" +%} + +Once the variable is called through the [Codefresh yml syntax]({{site.baseurl}}/docs/pipelines/variables/), it automatically uses the value provided within the variables section. If you are using this example as your pipeline, please delete anything in your pipeline. We can then add the following pipeline to our Inline YAML within the Workflow section in our UI: + +{% highlight yaml %} +{% raw %} +version: "1.0" +# Stages can help you organize your steps in stages +stages: + - "clone" + - "build" + - "test" + +steps: + clone: + title: "Cloning repository" + type: "git-clone" + repo: "anais-codefresh/codacy-sample-app" + # CF_BRANCH value is auto set when pipeline is triggered + # Learn more at codefresh.io/docs/docs/pipelines/variables/ + revision: "${{CF_BRANCH}}" + git: "github" + stage: "clone" + + build: + title: "Building Docker image" + type: "build" + image_name: "anaisurlichs/codacy-sample-app" + working_directory: "${{clone}}" + tag: "${{CF_BRANCH_TAG_NORMALIZED}}" + dockerfile: "Dockerfile" + stage: "build" + registry: "dockerhub" + + tests: + title: "Running test" + type: "freestyle" + working_directory: '${{clone}}' + arguments: + image: 'node:15.2' + commands: + - "npm install --save-dev jest" + - "npm run test" + stage: "test" + + codacy: + title: "Pushing reports to codacy" + type: "freestyle" + working_directory: '${{clone}}' + arguments: + image: 'alpine:3.8' + commands: + - "export CODACY_PROJECT_TOKEN=${{CODACY_PROJECT_TOKEN}}" + - "wget -qO - https://coverage.codacy.com/get.sh | sh" + stage: "test" +{% endraw %} +{% endhighlight %} + +The last two steps, ’tests’ and ’codacy’, are used to run our tests, create our coverage reports and forward those to Codacy. If you are using your own project and existing pipeline, add those two steps to your pipeline. In case you are using your own application, make sure to adapt the commands within the test step to run the tests of your application. Additionally, ensure that both the ’repo’ and the ’image_name’ point to your integrations. + +Once you run the pipeline, the steps will create the coverage report and forwards it to Codacy. + +{% include image.html +lightbox="true" +file="/images/testing/codacy/codacy-pipeline.png" +url="/images/testing/codacy/codacy-pipeline.png" +alt="Pipeline with Codacy step" +max-width="80%" +%} + +## View reports + +You can view the updated coverage reports within Codacy's UI every time you make a commit and/or run the Codefresh pipeline directly. + +{% include image.html +lightbox="true" +file="/images/testing/codacy/codacy-report.png" +url="/images/testing/codacy/codacy-report.png" +alt="Codacy UI Analysis Dashboard" +max-width="80%" +%} + +You can access further information on the coverage report by opening the file tab and accessing a specific file from your repository. + +{% include image.html +lightbox="true" +file="/images/testing/codacy/file-analysis.png" +url="/images/testing/codacy/file-analysis.png" +alt="Codacy report details" +max-width="90%" +%} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Unit tests]({{site.baseurl}}/docs/testing/unit-tests/) +[Integration tests]({{site.baseurl}}/docs/testing/integration-tests/) +[Sonarqube Integration]({{site.baseurl}}/docs/testing/sonarqube-integration/) diff --git a/_docs/example-catalog/ci-examples/codecov-testing.md b/_docs/example-catalog/ci-examples/codecov-testing.md new file mode 100644 index 00000000..82e06f88 --- /dev/null +++ b/_docs/example-catalog/ci-examples/codecov-testing.md @@ -0,0 +1,128 @@ +--- +title: "Codecov coverage reports" +description: "How to forward coverage reports to Codecov" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +[Codecov account](https://codecov.io/){:target="\_blank"} is a code analysis tool with which users can group, merge, archive, and compare coverage reports. Code coverage describes which lines of code were executed by the test suite and which ones were not. However, this is not to be confused with a testing tool. + +Analysis reports displayed within the Codecov dashboard: +{% include image.html +lightbox="true" +file="/images/testing/codecov/analysis-report.png" +url="/images/testing/codecov/analysis-report.png" +alt="Codecov UI Analysis reports" +max-width="50%" +%} + +## Prerequisites for using Codecov + +* A simple [Codefresh pipeline up and running](https://codefresh.io/docs/docs/getting-started/create-a-codefresh-account/) +* A [Codecov account](https://codecov.io/){:target="\_blank"} (free or enterprise) +* A testing tool added to your project that produces coverage reports + +Note that reports should ideally be written in .json, .xml, or txt. To be sure, please double check that your coverage [report format](https://docs.codecov.io/docs/supported-report-formats){:target="\_blank"} is supported. You can find a variety of examples for different programming languages and suggestions for respective testing tools in the [Codecov docs](https://docs.codecov.io/docs/supported-languages){:target="\_blank"}. + +To test Codecov and follow along with the next section, you can clone our [Codecov sample app](https://github.com/codefresh-contrib/codecov-sample-app){:target="\_blank"}. + +## Create a Codecov account + +Once you sign up to Codecov, you can add a new repository. The UI will then provide you with an access token to the repository. While it is recommended that you take note of the token, you will still be able to access it within the **Settings** tap. + +{% include image.html +lightbox="true" +file="/images/testing/codecov/codecov-interface.png" +url="/images/testing/codecov/codecov-interface.png" +alt="Codecov Project Repository UI" +max-width="50%" +%} + +## Codefresh pipeline + +In this case, we divided testing and connecting Codefresh to Codecov into two different steps. If they can be run within the same image, you could also connect them. + +**Testing step** +Runs the command(s) for our testing tool. This will generate the code coverage report upon running the pipeline. Please refer to the Codecov documentation for [supported testing frameworks](https://docs.codecov.io/docs/supported-report-formats){:target="\_blank"}. The [README of each example](https://docs.codecov.io/docs/supported-languages){:target="\_blank"} refers to possible frameworks that can be used. + +In general, ensure that the framework you use for testing and generating code coverage reports: +* Produce code coverage reports in the supported file format +* Is compatible with the programming language that your program is written in + +{% highlight yaml %} +{% raw %} + test: + title: "Running test" + type: "freestyle" # Run any command + image: "node:14.19.0" # The image in which command will be executed + working_directory: "${{clone}}" # Running command where code cloned + commands: + - "npm install --save-dev jest" + - "npx jest --coverage" + stage: "test" +{% endraw %} +{% endhighlight %} + +**Codecov step** + +{% highlight yaml %} +{% raw %} +upload: + title: "Running test" + type: "freestyle" # Run any command + image: "node:14.19.0" # The image in which command will be executed + working_directory: "${{clone}}" # Running command where code cloned + commands: + - "ci_env=`curl -s https://codecov.io/env`" + - "npm install codecov -g" + - "codecov -t ${{CODECOV_TOKEN}} -f ./coverage/clover.xml" + stage: "upload" +{% endraw %} +{% endhighlight %} + +The commands run inside of the node Docker image: +* `ci_env= curl -s https://codecov.io/env`: Sets the CI environment variable to take note that we are using Codefresh +* `npm install codecov -g`: Installs the odecov CLI +* `codecov -t ${{CODECOV_TOKEN}} -f ./coverage/clover.xml`: Sets the Codevoc access token provided in the UI when we connect to a new Git repository and point to the file that contains our coverage report. + +Once you run the pipeline, the steps will create the coverage report and forward it to Codecov. + +{% include image.html +lightbox="true" +file="/images/testing/codecov/codecov-pipeline.png" +url="/images/testing/codecov/codecov-pipeline.png" +alt="Pipeline with codecov step" +max-width="50%" +%} + +## View reports + +You can view the updated coverage reports within the Codecov UI every time you make a commit and/or run the Codefresh pipeline directly. + +{% include image.html +lightbox="true" +file="/images/testing/codecov/codecov-report.png" +url="/images/testing/codecov/codecov-report.png" +alt="Pipeline with codecov step" +max-width="50%" +%} + +You can access further information on the coverage report by opening the link to the file displayed in the table. + +{% include image.html +lightbox="true" +file="/images/testing/codecov/codecov-report-details.png" +url="/images/testing/codecov/codecov-report-details.png" +alt="Codecov report details" +max-width="50%" +%} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Unit tests]({{site.baseurl}}/docs/testing/unit-tests/) +[Integration tests]({{site.baseurl}}/docs/testing/integration-tests/) +[Sonarqube Integration]({{site.baseurl}}/docs/testing/sonarqube-integration/) + diff --git a/_docs/example-catalog/ci-examples/coveralls-testing.md b/_docs/example-catalog/ci-examples/coveralls-testing.md new file mode 100644 index 00000000..dd060c20 --- /dev/null +++ b/_docs/example-catalog/ci-examples/coveralls-testing.md @@ -0,0 +1,221 @@ +--- +title: "Coveralls coverage reports" +description: "How to forward coverage reports to Coveralls" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +[Coveralls](https://coveralls.io/){:target="\_blank"} is a web service that allows users to track the code coverage of their application over time in order to optimize the effectiveness of their unit tests. This section details how coverage reports can be generated and forwarded to Coveralls with every Codefresh build. + +Analysis reports displayed within Coveralls dashboard: +{% include image.html +lightbox="true" +file="/images/testing/coveralls/coveralls-sample-app.png" +url="/images/testing/coveralls/coveralls-sample-app.png" +alt="Coveralls UI Analysis reports" +max-width="80%" +%} + +## Prerequisites for using Coveralls + +* A simple [Codefresh pipeline up and running](https://codefresh.io/docs/docs/getting-started/create-a-codefresh-account/) +* A [Coveralls account](https://coveralls.io/) (free or enterprise) -- Note that all open-source projects are free on Coveralls +* A testing tool added to your project that produces coverage reports + +Coveralls supports [22 different language integrations](https://docs.coveralls.io/about-coveralls){:target="\_blank"}. Each example provided in the official documentation suggests several coverage report tools that can be used in combination with Coveralls. + +You could try it out by cloning our [node example application](https://github.com/codefresh-contrib/coveralls-sample-app){:target="\_blank"} that utilises [jest](https://jestjs.io/){:target="\_blank"}. + +## Prepare your repository + +If you are using your own application as an example, you have to make a few modifications to the repository. Please have a look at the Coveralls example section for other languages. + +First, install Coveralls in your project: +{% highlight yaml %} +{% raw %} +npm install coveralls --save-dev +{% endraw %} +{% endhighlight %} + +Coveralls requires a [script](https://github.com/nickmerwin/node-coveralls){:target="\_blank"} that takes standard input and sends it to coveralls.io to report your code coverage. Depending on the framework that you are using, you will have to add a different script to your application. + +Any coverage reports can be forwarded that are within a [lcov data format](http://ltp.sourceforge.net/coverage/lcov/geninfo.1.php){:target="\_blank"} (including [mocha's LCOV reporter](https://www.npmjs.com/package/mocha-lcov-reporter){:target="\_blank"}). For this, we are going to set-up a “bin” folder, and within the folder a coveralls.js file that contains the following content: + +{% highlight yaml %} +{% raw %} +#!/usr/bin/env node + +'use strict'; + +const { handleInput } = require('..'); + +process.stdin.resume(); +process.stdin.setEncoding('utf8'); + +let input = ''; + +process.stdin.on('data', chunk => { + input += chunk; +}); + +process.stdin.on('end', () => { + handleInput(input, err => { + if (err) { + throw err; + } + }); +}); +{% endraw %} +{% endhighlight %} + +## Create a Coveralls account + +Once you sign-up to Coveralls, you can add a new repository. The UI will then provide you with an access token to the repository. Take note of the token since it will be required in the next sections. + +{% include image.html +lightbox="true" +file="/images/testing/coveralls/add-repository.png" +url="/images/testing/coveralls/add-repository.png" +alt="Coveralls repository" +max-width="80%" +%} + +## Codefresh pipeline + + +In case the project that you want to use Coveralls in does not have a pipeline, [create a new pipeline]({{site.baseurl}}/docs/getting-started/create-a-basic-pipeline/). + +{% include image.html +lightbox="true" +file="/images/testing/coveralls/create-coveralls-pipeline.png" +url="/images/testing/coveralls/create-coveralls-pipeline.png" +alt="Create Coveralls Pipeline" +max-width="80%" +%} + +Once you ’create’ the pipeline, a standard codefresh.yml file is generated with three steps: +* The first step will clone your repository; +* The second step will both, build and push your repository to the container registry that you have connected with Codefresh; +* And the third step currently does not do much. +In the next section, we will modify the testing step. + +**Testing step** + +The testing step requires three different environment variables to connect to Coveralls: +* `export COVERALLS_SERVICE_NAME="codefresh"` +* `export COVERALLS_GIT_BRANCH="insert the branch that you will be using with your application"` +* `export COVERALLS_REPO_TOKEN="insert the secret repo token from coveralls.io"` + +{% highlight yaml %} +{% raw %} + test: + title: "Running test" + type: "freestyle" # Run any command + image: "node:15.2" # The image in which command will be executed + working_directory: "${{clone}}" # Running command where code cloned + commands: + - "export COVERALLS_SERVICE_NAME=${{COVERALLS_SERVICE_NAME}}" + - "export COVERALLS_GIT_BRANCH=${{CF_BRANCH}}" + - "export COVERALLS_REPO_TOKEN=${{COVERALLS_REPO_TOKEN}}" + - "npm install --save-dev jest" + - "npm run test" + stage: "test" +{% endraw %} +{% endhighlight %} + +We specify several variables within this step. Those, which start with ’CF’ are [Codefresh-specific steps]({{site.baseurl}}/docs/pipelines/variables/) and the value is automatically provided by Codefresh once you run the pipeline. Our entire codefresh.yml will look as such: + +{% highlight yaml %} +{% raw %} +version: "1.0" +stages: + - "clone" + - "build" + - "test" + +steps: + clone: + title: "Cloning repository" + type: "git-clone" + repo: "anais-codefresh/coveralls-sample-app" + # CF_BRANCH value is auto set when pipeline is triggered + # Learn more at codefresh.io/docs/docs/pipelines/variables/ + revision: "${{CF_BRANCH}}" + git: "github" + stage: "clone" + + build: + title: "Building Docker image" + type: "build" + image_name: "anaisurlichs/coveralls-sample-app" + working_directory: "${{clone}}" + tag: "${{CF_BRANCH_TAG_NORMALIZED}}" + dockerfile: "Dockerfile" + stage: "build" + registry: "dockerhub" + + test: + title: "Running test" + type: "freestyle" # Run any command + image: "node:15.2" # The image in which command will be executed + working_directory: "${{clone}}" # Running command where code cloned + commands: + - "export COVERALLS_SERVICE_NAME=${{COVERALLS_SERVICE_NAME}}" + - "export COVERALLS_GIT_BRANCH=${{CF_BRANCH}}" + - "export COVERALLS_REPO_TOKEN=${{COVERALLS_REPO_TOKEN}}" + - "npm install --save-dev jest" + - "npm run test" + stage: "test" +{% endraw %} +{% endhighlight %} + +Once you run the pipeline the steps will create the coverage report and forward it to Coveralls. + +{% include image.html +lightbox="true" +file="/images/testing/coveralls/coveralls-pipeline.png" +url="/images/testing/coveralls/coveralls-pipeline.png" +alt="Pipeline with Coveralls step" +max-width="80%" +%} + +## View reports + +You can view the updated coverage reports within Coveralls UI every time you make a commit and/or run the Codefresh pipeline directly. + +{% include image.html +lightbox="true" +file="/images/testing/coveralls/coveralls-sample-app.png" +url="/images/testing/coveralls/coveralls-sample-app.png" +alt="Coveralls UI Analysis reports" +max-width="80%" +%} + +You can access further information on the coverage report by opening the link to the file displayed in the table. + +{% include image.html +lightbox="true" +file="/images/testing/coveralls/coveralls-specific-report.png" +url="/images/testing/coveralls/coveralls-specific-report.png" +alt="Coveralls report details" +max-width="80%" +%} + +And view a the code coverage of a specific file: +{% include image.html +lightbox="true" +file="/images/testing/coveralls/coveralls-coverage.png" +url="/images/testing/coveralls/coveralls-coverage.png" +alt="Coveralls report details" +max-width="80%" +%} + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Unit tests]({{site.baseurl}}/docs/testing/unit-tests/) +[Integration tests]({{site.baseurl}}/docs/testing/integration-tests/) +[Sonarqube Integration]({{site.baseurl}}/docs/testing/sonarqube-integration/) diff --git a/_docs/example-catalog/ci-examples/cpp-cmake.md b/_docs/example-catalog/ci-examples/cpp-cmake.md new file mode 100644 index 00000000..11f8e963 --- /dev/null +++ b/_docs/example-catalog/ci-examples/cpp-cmake.md @@ -0,0 +1,125 @@ +--- +title: "Compile and test a C++ application" +description: "Using Codefresh pipelines" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +Codefresh can work with any C/C++ application very easily as both `gcc` and `g++` are already offered in Dockerhub. There is also another example available with [C and make]({{site.baseurl}}/docs/example-catalog/ci-examples/c-make). + +## The example C++ project + +You can see the example project at [https://github.com/codefresh-contrib/cpp-sample-app](https://github.com/codefresh-contrib/cpp-sample-app){:target="\_blank"}. The repository contains a C++ starter project with a `CMakeLists.txt` file: + +* `cmake .` creates the makefiles. +* `make test` runs unit tests +* `make` compiles the code + +The project is also using the [boost testing libraries](https://www.boost.org/){:target="\_blank"}. + +## Cmake, g++ and Docker + +Creating a CI/CD pipeline for C is very easy, because Codefresh can run any [gcc image](https://hub.docker.com/_/gcc/){:target="\_blank"} that you wish. Gcc docker images already contain the `make` utility but not the the `cmake` one. Therefore we will first create a Dockerfile that has `g++`, cmake and the boost libraries. You can follow the same pattern for other development tools that you use. + + +Here is the Dockerfile: + + `Dockerfile` +{% highlight docker %} +{% raw %} +FROM gcc:9.2 + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && apt-get install -y cmake libgtest-dev libboost-test-dev && rm -rf /var/lib/apt/lists/* + +CMD ["cmake"] + +{% endraw %} +{% endhighlight %} + +This docker build does the following: + +1. Starts from the GCC image +1. Installs cmake and boost +1. Sets cmake as the default command + +## Create a CI pipeline for C++ applications + +We can now use the custom Docker image in order to compile/test the C++ application: + +{% include image.html +lightbox="true" +file="/images/learn-by-example/cc/cpp-cmake-pipeline.png" +url="/images/learn-by-example/cc/cpp-cmake-pipeline.png" +alt="Compiling a C++ application in a pipeline" +caption="Compiling a C++ application in a pipeline" +max-width="80%" +%} + +Here is the [full pipeline](https://github.com/codefresh-contrib/cpp-sample-app/blob/master/codefresh.yml){:target="\_blank"} that compiles the application after checking out the code. + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - checkout + - prepare + - build +steps: + main_clone: + title: Cloning main repository... + stage: checkout + type: git-clone + repo: 'codefresh-contrib/cpp-sample-app' + revision: master + git: github + build_dev_image: + title: Building Dev Image + stage: prepare + type: build + image_name: cmake + working_directory: ./dev/ + tag: 'latest' + dockerfile: Dockerfile + create_makefiles: + title: Create Makefiles + stage: prepare + image: ${{build_dev_image}} + commands: + - cmake . + compile_my_sources: + title: Compile + stage: build + image: ${{build_dev_image}} + commands: + - make + run_my_tests: + title: Test + stage: build + image: ${{build_dev_image}} + commands: + - make test +{% endraw %} +{% endhighlight %} + +This pipeline: + +1. clones the source code +1. Creates a development docker image that has g++, cmake and boost +1. Runs cmake on the source code to create the make files +1. Compiles the source code +1. Runs unit tests + +You can add additional tools in the pipeline by extending the Dockerfile mentioned in the previous section. You can also +change the version of Gcc/g++ by starting from a different public or private Docker image. + + +## Related articles +[C example]({{site.baseurl}}/docs/example-catalog/ci-examples/c-make/) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/codefresh-yaml/steps/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/ci-examples/decryption-with-mozilla-sops.md b/_docs/example-catalog/ci-examples/decryption-with-mozilla-sops.md new file mode 100644 index 00000000..d091ca24 --- /dev/null +++ b/_docs/example-catalog/ci-examples/decryption-with-mozilla-sops.md @@ -0,0 +1,177 @@ +--- +title: "Decrypt with Mozilla SOPS" +description: "Store secrets in your repository and decrypt them using Mozilla SOPS" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +## Prerequisites + +- A [free Codefresh account]({{site.baseurl}}/docs/administration/create-a-codefresh-account/) +- A public and private GnuGP key pair +- A credentials yaml, that is encrypted using Mozilla SOPS, and stored in your repository + +## Example Java application + +You can find the example project on [GitHub](https://github.com/codefresh-contrib/mozilla-sops-app){:target="\_blank"}. + +The example application retrieves the system variable "password," from the pipeline and uses it to authenticate to a Redis database, but you are free to use any type of database of your choosing. + +```java + String password = System.getenv("password"); + String host = System.getProperty("server.host"); + + RedisClient redisClient = new RedisClient( + RedisURI.create("redis://" + password + "@" + host + ":6379")); + RedisConnection : ' + VAULT_PATH: 'path/to/secret' + VAULT_AUTH_TOKEN: ' ' + clone: + title: Cloning main repository... + type: git-clone + arguments: + repo: 'codefresh-contrib/vault-sample-app' + git: github + stage: clone + package_jar: + title: Packaging jar and running unit tests... + stage: package + working_directory: ${{clone}} + arguments: + image: maven:3.5.2-jdk-8-alpine + commands: + - mvn -Dmaven.repo.local=/codefresh/volume/m2_repository -Dserver.host=my-redis-db-host clean package + services: + composition: + my-redis-db-host: + image: 'redis:4-alpine' + command: 'redis-server --requirepass $password' + ports: + - 6379 +``` + +The pipeline does the following: + +1. Imports the key-value pairs from the Vault server and exports them into the pipeline under `/meta/env_vars_to_export`. +2. Clones the main repository (note the special use of naming the step `main_clone`). This ensures that all subsequent commands are run [inside the project that was checked out]({{site.baseurl}}/docs/pipelines/steps/git-clone/#basic-clone-step-project-based-pipeline). +3. The `package_jar`, does a few special things to take note of: + - Spins up a [Service Container]({{site.baseurl}}/docs/pipelines/service-containers/) running Redis on port 6379 , and sets the password to the database using our exported environment variable + - Sets `maven.repo.local` to cache Maven dependencies into the local codefresh volume to [speed up builds]({{site.baseurl}}/docs/example-catalog/ci-examples/spring-boot-2/#caching-the-maven-dependencies) + - Runs unit tests and packages the jar. Note how you can directly refer to the service container's name (`my-redis-db-host`) when we set `server.host` + +You will see that the variable was correctly exported to the pipeline by running a simple `echo` command: + {% include image.html + lightbox="true" + file="/images/examples/secrets/vault-pipeline2.png" + url="/images/examples/secrets/vault-pipeline2.png" + alt="Vault pipeline variable" + caption="Vault pipeline variable" + max-width="100%" + %} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/ci-examples/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) + diff --git a/_docs/example-catalog/ci-examples/voting-app.md b/_docs/example-catalog/ci-examples/voting-app.md new file mode 100644 index 00000000..08cb4a5c --- /dev/null +++ b/_docs/example-catalog/ci-examples/voting-app.md @@ -0,0 +1,93 @@ +--- +title: "Voting app" +description: "" +excerpt: "" +group: example-catalog +sub_group: ci-examples +redirect_from: + - /docs/voting-app-1/ + - /docs/python/voting-app/ +toc: true +--- +This voting application is a demo with which you can build an advanced composition that uses `Python, Redis, Postgres, Node.js, and .Net`. + +## Looking around +In the root of this repository you'll find a file named codefresh.yml, this is our build descriptor and it describes the different steps that comprise our process. Let's quickly review the contents of this file: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + unit-tests: + image: codefresh/buildpacks:nodejs-5 + working-directory : ${{initial-clone}} + commands: + - echo Installing npm modules silent + - npm install + - gulp test + - echo $(date) + + build-step: + #title: Build My Image #Display name for the step + type: build + image-name: containers101/cf-example-result + tag: ${{CF_BRANCH}} + build_arguments: + - OPTION_A=${{OPTION_A}} + - OPTION_B=${{OPTION_B}} + + push-to-registry: + type: push + #candidate: the image from the build step + candidate: ${{build-step}} + tag: ${{CF_BRANCH}} + + integration-tests-step: + type: composition + #location of the compostion on the filesystem of the cloned image + composition: './cf-compositions/voting-app-full.yml' + #run integration only when pushing to master + when: + branch: + only: + - master #can also be regex + composition-candidates: + #this will be the image that we will test + integ-test: + image: containers101/cf-example-tests:master + command: ./tests.sh + composition-variables: + - VOTING_OPTION_A=${{OPTION_A}} + - VOTING_OPTION_B=${{OPTION_B}} + + launch-composition: + type: launch-composition + environment-name: 'Test composition after build' + composition: './cf-compositions/voting-app-full.yml' + composition-variables: + - VOTING_OPTION_A=${{OPTION_A}} + - VOTING_OPTION_B=${{OPTION_B}} + + deploy to ecs: + image: codefresh/cf-deploy-ecs + commands: + - cfecs-update --image-name containers101/cf-example-result --image-tag ${{CF_BRANCH}} eu-west-1 vote-app result + environment: + - AWS_ACCESS_KEY_ID=${{AWS_ACCESS_KEY_ID}} + - AWS_SECRET_ACCESS_KEY=${{AWS_SECRET_ACCESS_KEY}} + when: + condition: + all: + pushCommit: 'includes(lower("${{CF_COMMIT_MESSAGE}}"), "[deploy]") == true' +{% endraw %} +{% endhighlight %} + +{{site.data.callout.callout_info}} +##### Example + +Just head over to the example [__repository__](https://github.com/containers101/cf-example-result){:target="_blank"} in GitHub and follow the instructions there. +{{site.data.callout.end}} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/ci-examples/) diff --git a/_docs/example-catalog/examples.md b/_docs/example-catalog/examples.md new file mode 100644 index 00000000..9dee4525 --- /dev/null +++ b/_docs/example-catalog/examples.md @@ -0,0 +1,127 @@ +--- +title: "CI/CD pipeline examples" +description: "A collection of examples for Codefresh pipelines" +group: example-catalog +redirect_from: + - /docs/examples-v01/ + - examples.html + - /docs/catalog-examples/ + - /docs/examples/ + - /docs/pipelines-examples/ + - /docs/pipelines/pipelines-examples/ +toc: true +--- +Codefresh enables you to define the steps of your pipeline in a [YAML file]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/). By default, the file is named `codefresh.yml`, and is located in the root directory of the repository. + +## CI examples + +### Programming-language specific examples + +Codefresh is agnostic as far as programming languages are concerned. All major programming languages are supported: + +- [Go Web App]({{site.baseurl}}/docs/example-catalog/ci-examples/golang-hello-world/) or [Go CLI]({{site.baseurl}}/docs/example-catalog/golang/goreleaser) +- [Spring Java app with Maven]({{site.baseurl}}/docs/example-catalog/ci-examples/spring-boot-2/) or [Gradle]({{site.baseurl}}/docs/example-catalog/ci-examples/gradle/). Also how to [upload JAR to Nexus/Artifactory]({{site.baseurl}}/docs/example-catalog/ci-examples/publish-jar/) +- Node [Express.js App]({{site.baseurl}}/docs/example-catalog/ci-examples/lets-chat/) or [React.js App]({{site.baseurl}}/docs/example-catalog/ci-examples/react/) +- [Php App]({{site.baseurl}}/docs/example-catalog/ci-examples/php) +- [Python Django App]({{site.baseurl}}/docs/example-catalog/ci-examples/django/) +- [Ruby On Rails App]({{site.baseurl}}/docs/example-catalog/ci-examples/ruby) +- [C]({{site.baseurl}}/docs/example-catalog/ci-examples/c-make/) or [C++]({{site.baseurl}}/docs/example-catalog/ci-examples/cpp-cmake) +- [Rust]({{site.baseurl}}/docs/example-catalog/ci-examples/rust/) +- [C# .NET core]({{site.baseurl}}/docs/example-catalog/ci-examples/dotnet/) +- [Scala App]({{site.baseurl}}/docs/example-catalog/ci-examples/scala-hello-world/) +- [Android (Mobile)]({{site.baseurl}}/docs/example-catalog/ci-examples/android/) + +### Source code checkout examples + +You can check out code from one or more repositories in any pipeline phase. Codefresh includes [built-in GIT integration]({{site.baseurl}}/docs/integrations/git-providers/) with all the popular GIT providers and can be used with [git-clone]({{site.baseurl}}/docs/pipelines/steps/git-clone/) steps. + +- [Cloning Git repositories using the built-in integration]({{site.baseurl}}/docs/example-catalog/ci-examples/git-checkout/) +- [Cloning Git repositories using manual Git commands]({{site.baseurl}}/docs/example-catalog/ci-examples/git-checkout-custom/) +- [Checking out from Subversion, Perforce, Mercurial, etc ]({{site.baseurl}}/docs/example-catalog/ci-examples/non-git-checkout/) + +### Build/push examples + +Codefresh has native support for [building]({{site.baseurl}}/docs/pipelines/steps/build/) and [pushing]({{site.baseurl}}/docs/pipelines/steps/push/) Docker containers. +You can also compile traditional applications that are not Dockerized yet. + +- [Build an Image with the Dockerfile in root directory]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-dockerfile-in-root-directory/) +- [Build an Image by specifying the Dockerfile location]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-specify-dockerfile-location) +- [Build an Image from a different Git repository]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository) +- [Build and Push an Image]({{site.baseurl}}/docs/example-catalog/ci-examples/build-and-push-an-image) +- [Build an Image with build arguments]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-with-build-arguments) +- [Share data between steps]({{site.baseurl}}/docs/example-catalog/ci-examples/shared-volumes-between-builds) +- [Upload or download from a Google Storage Bucket]({{site.baseurl}}/docs/example-catalog/ci-examples/uploading-or-downloading-from-gs/) +- [Get Short SHA ID and use it in a CI process]({{site.baseurl}}/docs/example-catalog/ci-examples/get-short-sha-id-and-use-it-in-a-ci-process) +- [Call a CD pipeline from a CI pipeline]({{site.baseurl}}/docs/example-catalog/ci-examples/call-child-pipelines) +- [Trigger a Kubernetes Deployment from a Dockerhub Push Event]({{site.baseurl}}/docs/example-catalog/ci-examples/trigger-a-k8s-deployment-from-docker-registry/) + + +### Unit and integration test examples + +Codefresh has support for both [unit]({{site.baseurl}}/docs/testing/unit-tests/) and [integration tests]({{site.baseurl}}/docs/testing/integration-tests/) as well as [test reporting]({{site.baseurl}}/docs/testing/test-reports/). + +- [Run unit tests]({{site.baseurl}}/docs/example-catalog/ci-examples/run-unit-tests) +- [Run integration tests]({{site.baseurl}}/docs/example-catalog/ci-examples/run-integration-tests/) +- [Run integration tests with MongoDB]({{site.baseurl}}/docs/example-catalog/ci-examples/integration-tests-with-mongo/) +- [Run integration tests with MySQL]({{site.baseurl}}/docs/example-catalog/ci-examples/integration-tests-with-mysql/) +- [Run integration tests with PostgreSQL]({{site.baseurl}}/docs/example-catalog/ci-examples/integration-tests-with-postgres/) +- [Run integration tests with Redis]({{site.baseurl}}/docs/example-catalog/ci-examples/integration-tests-with-redis/) +- [Populate a database with existing data]({{site.baseurl}}/docs/example-catalog/populate-a-database-with-existing-data) + +- [Shared volumes of service from composition step for other yml steps]({{site.baseurl}}/docs/example-catalog/shared-volumes-of-service-from-composition-step-for-other-yml-steps) +- [Launch Composition]({{site.baseurl}}/docs/example-catalog/ci-examples/launch-composition) +- [Launch Composition and define Service Environment variables using a file]({{site.baseurl}}/docs/example-catalog/ci-examples/launching-a-composition-and-defining-a-service-environment-variables-using-a-file) +- [Run multiple kinds of unit tests using fan-in-fan-out parallel pipeline]({{site.baseurl}}/docs/example-catalog/fan-in-fan-out) + +### Code coverage examples + +- [Run coverage reports with Codecov]({{site.baseurl}}/docs/example-catalog/ci-examples/codecov-testing) +- [Run coverage reports with Coveralls]({{site.baseurl}}/docs/example-catalog/ci-examples/coveralls-testing) +- [Run coverage reports with Codacy]({{site.baseurl}}/docs/example-catalog/ci-examples/codacy-testing) + +### Secrets examples + +Codefresh can automatically export secret key-value pairs using the Vault plugin from the [Step Marketplace](https://codefresh.io/steps/step/vault). + +- [Vault secrets in the Pipeline]({{site.baseurl}}/docs/example-catalog/ci-examples/vault-secrets-in-the-pipeline) +- [Decryption with Mozilla SOPS]({{site.baseurl}}/docs/example-catalog/ci-examples/ci-examples/decryption-with-mozilla-sops) +- [GitOps with Bitnami sealed secrets]({{site.baseurl}}/docs/example-catalog/ci-examples/gitops-secrets) + +### Notification examples + +- [Send notification to Slack]({{site.baseurl}}/docs/example-catalog/ci-examples/sending-the-notification-to-slack) +- [Send notification to Jira]({{site.baseurl}}/docs/example-catalog/ci-examples/sending-the-notification-to-jira) + + +## CD examples + +### Preview environment examples + +Codefresh can automatically launch environments (powered by Docker swarm) to [preview a Pull Reqest or feature]({{site.baseurl}}/docs/getting-started/on-demand-environments/). The definition of the environment can come from an [existing composition]({{site.baseurl}}/docs/testing/create-composition/), a docker-compose file or an inline YAML. Preview environments can be launched manually or [automatically from pipelines]({{site.baseurl}}/docs/pipelines/steps/launch-composition/). + +- [MongoDB preload data]({{site.baseurl}}/docs/example-catalog/cd-examples/import-data-to-mongodb/) +- [NodeJS + Angular2 + MongoDB]({{site.baseurl}}/docs/example-catalog/cd-examples/nodejs-angular2-mongodb/) +- [NGINX Basic Auth]({{site.baseurl}}/docs/example-catalog/cd-examples/secure-a-docker-container-using-http-basic-auth/) +- [Spring Boot + Kafka + Zookeeper]({{site.baseurl}}/docs/example-catalog/cd-examples/spring-boot-kafka-zookeeper/) +- [Web terminal]({{site.baseurl}}/docs/example-catalog/cd-examples/web-terminal/) + +### Deployment examples + +Codefresh can deploy to any platform such as VMs, FTP/SSH/S3 sites, app servers, but of course it has great support for [Kubernetes clusters]({{site.baseurl}}/docs/deploy-to-kubernetes/deployment-options-to-kubernetes/) and [Helm releases]({{site.baseurl}}/docs/new-helm/helm-releases-management/): + +- [Deploy to a VM with packer]({{site.baseurl}}/docs/example-catalog/cd-examples/packer-gcloud/) +- [Deploy to a VM with FTP]({{site.baseurl}}/docs/example-catalog/cd-examples/transferring-php-ftp) +- [Deploy to Tomcat using SCP]({{site.baseurl}}/docs/example-catalog/cd-examples/deploy-to-tomcat-via-scp) +- [Deploy Demochat to a Kubernetes cluster]({{site.baseurl}}/docs/cd-examples/deploy-to-kubernetes/codefresh-kubernetes-integration-demochat-example/) +- [Use kubectl as part of freestyle step]({{site.baseurl}}/docs/example-catalog/cd-examples/use-kubectl-as-part-of-freestyle-step) +- [Deploy with Kustomize]({{site.baseurl}}/docs/example-catalog/cd-examples/deploy-with-kustomize) +- [Deploy with Helm]({{site.baseurl}}/docs/example-catalog/cd-examples/helm) +- [Deploy with Terraform]({{site.baseurl}}/docs/example-catalog/cd-examples/terraform) +- [Deploy with Pulumi]({{site.baseurl}}/docs/example-catalog/cd-examples/pulumi) +- [Deploy to Nomad]({{site.baseurl}}/docs/example-catalog/cd-examples/nomad) +- [Deploy to Heroku]({{site.baseurl}}/docs/example-catalog/cd-examples/deploy-to-heroku/) +- [Deploy to Docker swarm]({{site.baseurl}}/docs/example-catalog/cd-examples/docker-swarm/) +- [Deploy to Elastic Beanstalk]({{site.baseurl}}/docs/example-catalog/cd-examples/elastic-beanstalk/) +- [Deploy to Amazon ECS/Fargate]({{site.baseurl}}/docs/example-catalog/cd-examples/amazon-ecs/) + + diff --git a/_docs/example-catalog/gitops-example.md b/_docs/example-catalog/gitops-example.md new file mode 100644 index 00000000..ba1c727d --- /dev/null +++ b/_docs/example-catalog/gitops-example.md @@ -0,0 +1,9 @@ +--- +title: "GitOps examples" +description: "A collection of examples for GitOps deployments" +group: example-catalog +sub_group: gitops-examples +toc: true +--- + +TBD \ No newline at end of file diff --git a/_docs/getting-started/architecture.md b/_docs/getting-started/architecture.md deleted file mode 100644 index 584fd507..00000000 --- a/_docs/getting-started/architecture.md +++ /dev/null @@ -1,199 +0,0 @@ ---- -title: "Architecture" -description: "" -group: getting-started -toc: true ---- - -Codefresh GitOps is built around an enterprise version of the Argo Ecosystem, fully compliant with the GitOps paradigm, with industry-standard security. -To cater to differing requirements and degrees of enterprise security, Codefresh supports hosted and hybrid installation environments for Codefresh runtimes. - -The sections that follow illustrate the architecture of the different installation environments, starting with a high-level overview of the Codefresh Platform. - -### Codefresh architecture - -The diagram shows a high-level view of the Codefresh Platform and its core components, the Codefresh Control Plane, the Codefresh Runtime, and the Codefresh Clients. - -{% include - image.html - lightbox="true" - file="/images/getting-started/architecture/arch-codefresh-simple.png" - url="/images/getting-started/architecture/arch-codefresh-simple.png" - alt="Codefresh Platform architecture" - caption="Codefresh Platform architecture" - max-width="100%" -%} - -{::nomarkdown} -
-{:/} - -#### Codefresh Control Plane -The Codefresh Control Plane is the SaaS component in the platform. External to the enterprise firewall, it does not have direct communication with the Codefresh Runtime, Codefresh Clients, or the customer's organizational systems. The Codefresh Runtime and the Codefresh Clients communicate with the Codefresh Control Plane to retrieve the required information. - - -{::nomarkdown} -
-{:/} - -#### Codefresh Runtime -The Codefresh Runtime is installed on a Kubernetes cluster, and houses the enterprise distribution of the Codefresh Application Proxy and the Argo Project. -Depending on the type of installation environment, the Codefresh Runtime is installed either in the Codefresh platform (hosted), or in the customer environment (hybrid). Read more in [Codefresh runtime architecture](#codefresh-runtime-architecture). - - -{::nomarkdown} -
-{:/} - -#### Codefresh Clients - -Codefresh Clients include the Codefresh UI and the Codefresh CLI. -The Codefresh UI provides a unified, enterprise-wide view of deployments (runtimes and clusters), and CI/CD operations (Delivery Pipelines, workflows, and deployments) in the same location. -The Codefresh CLI includes commands to install hybrid runtimes, add external clusters, and manage runtimes and clusters. - -### Codefresh runtime architecture -The sections that follow show detailed views of runtime architecture in the different installation environments, and descriptions of the Codefresh Runtime components. - -* [Hosted GitOps runtime architecture](#hosted-gitops-runtime-architecture) - In this installation environment, the Codefresh Runtime is installed on a _Codefresh-managed cluster_ in the Codefresh platform. -* Hybrid runtime architecture: - In this installation environment, the Codefresh Runtime is installed on a _customer-managed cluster_ in the customer environment. The Codefresh Runtime with or without ingress controllers: - * [Ingress controller](#ingress-controller-hybrid-runtime-architecture) - * [Ingress-less](#ingress-less-hybrid-runtime-architecture) -* Runtime components - * [Codefresh Application Proxy](#codefresh-application-proxy) - * [Argo Project](#argo-project) - * [Request Routing Service](#request-routing-service) - * [Tunnel Server](#codefresh-tunnel-server) - * [Tunnel Client](#codefresh-tunnel-client) - - -#### Hosted GitOps runtime architecture -In the hosted environment, the Codefresh Runtime is installed on a K8s cluster managed by Codefresh. - -{% include - image.html - lightbox="true" - file="/images/getting-started/architecture/arch-hosted.png" - url="/images/getting-started/architecture/arch-hosted.png" - alt="Hosted runtime architecture" - caption="Hosted runtime architecture" - max-width="100%" -%} - -#### Ingress controller hybrid runtime architecture -Runtimes with ingress use an ingress controller to control communication between the Codefresh Runtime in the customer cluster and the Codefresh Platform. Ingress controllers are optimal when the cluster with the Codefresh Runtime is exposed to the internet. - - - -{% include - image.html - lightbox="true" - file="/images/getting-started/architecture/arch-hybrid-ingress.png" - url="/images/getting-started/architecture/arch-hybrid-ingress.png" - alt="Ingress-based hybrid runtime architecture" - caption="Ingress-based hybrid runtime architecture" - max-width="100%" -%} - -#### Ingress-less hybrid runtime architecture -Ingress-less runtimes uses tunneling to control communication between the Codefresh Runtime in the customer cluster and the Codefresh Platform. Ingress-less runtimes are optimal when the cluster with the Codefresh Runtime is not exposed to the internet. - -{% include - image.html - lightbox="true" - file="/images/getting-started/architecture/arch-hybrid-ingressless.png" - url="/images/getting-started/architecture/arch-hybrid-ingressless.png" - alt="Ingress-less hybrid runtime architecture" - caption="Ingress-less hybrid runtime architecture" - max-width="100%" -%} - - - -#### Codefresh Application Proxy -The Codefresh Application Proxy (App-Proxy) functions as the Codefresh agent, and is deployed as a service in the Codefresh Runtime. -For hybrid runtimes with ingress, the App-Proxy is the single point-of-contact between the Codefresh Runtime, and the Codefresh Clients, the Codefresh Platform, and any organizational systems in the customer environment. -For ingress-less hybrid runtimes, the Tunnel Client forwards the incoming traffic from the Tunnel Server using internal reverse proxy to the App-Proxy. - -The App-Proxy: -* Accepts and serves requests from Codefresh Clients either via the Codefresh UI or CLI -* Retrieves a list of Git repositories for visualization in Codefresh Clients -* Retrieves permissions from the Codefresh Control Plane to authenticate and authorize users for the required operations. -* Implements commits for GitOps-controlled entities, such as Delivery Pipelines and other CI resources -* Implements state-change operations for non-GitOps controlled entities, such as terminating Argo Workflows - -{::nomarkdown} -
-{:/} - -#### Argo Project - -The Argo Project includes: -* Argo CD for declarative continuous deployment -* Argo Rollouts for progressive delivery -* Argo Workflows as the workflow engine -* Argo Events for event-driven workflow automation framework - - -{::nomarkdown} -
-{:/} - -#### Request Routing Service -The Request Routing Service is installed on the same cluster as the Codefresh Runtime in the customer environment. -It receives requests from the ingress controller (ingress) or the Tunnel Client (ingress-less), and forwards the request URLs to the Application Proxy, and webhooks directly to the Event Sources. - ->Important: - The Request Routing Service is available from runtime version 0.0.543 and higher. - Older runtime versions are not affected as there is complete backward compatibility, and the ingress controller continues to route incoming requests. - -#### Tunnel Server -Applies only to _ingress-less_ runtimes in hybrid installation environments. -The Codefresh Tunnel Server is installed in the Codefresh platform. It communicates with the enterprise cluster located behind a NAT or firewall. - -The Tunnel Server: -* Forwards traffic from Codefresh Clients to the client (customer) cluster. -* Manages the lifecycle of the Codefresh Tunnel Client. -* Authenticates requests from the Codefresh Tunnel Client to open tunneling connections. - -{::nomarkdown} -
-{:/} - -#### Tunnel Client -Applies only to _ingress-less_ runtimes in hybrid installation environments. - -Installed on the same cluster as the Codefresh Runtime, the Codefresh Tunnel Client establishes the tunneling connection to the Codefresh Tunnel Server via the WebSocket Secure (WSS) protocol. -A single Codefresh Runtime can have a single Tunnel Client. - -The Codefresh Tunnel Client: -* Initiates the connection with the Codefresh Tunnel Server. -* Forwards the incoming traffic from the Tunnel Server through the Request Routing Service to App-Proxy, and other services. - -{::nomarkdown} -
-{:/} - - -### Customer environment -The customer environment that communicates with the Codefresh Runtime and the Codefresh Platform, generally includes: -* Ingress controller for ingress hybrid runtimes - The ingress controller is configured on the same Kubernetes cluster as the Codefresh Runtime, and implements the ingress traffic rules for the Codefresh Runtime. - See [Ingress controller requirements]({{site.baseurl}}/docs/runtime/requirements/#ingress-controller). -* Managed clusters - Managed clusters are external clusters registered to provisioned hosted or hybrid runtimes for application deployment. - Hosted runtimes requires you to connect at least one external K8s cluster as part of setting up the Hosted GitOps environment. - Hybrid runtimes allow you to add external clusters after provisioning the runtimes. - See [Add external clusters to runtimes]({{site.baseurl}}/docs/runtime/managed-cluster/). -* Organizational systems - Organizational Systems include the customer's tracking, monitoring, notification, container registries, Git providers, and other systems. They can be entirely on-premises or in the public cloud. - Either the ingress controller (ingress hybrid environments), or the Tunnel Client (ingress-less hybrid environments), forwards incoming events to the Codefresh Application Proxy. - -### Related articles -[Set up a hosted runtime environment]({{site.baseurl}}/docs/runtime/hosted-runtime/) -[Install a hybrid runtime]({{site.baseurl}}/docs/runtime/installation/) - - - - diff --git a/_docs/getting-started/csdp-introduction.md b/_docs/getting-started/csdp-introduction.md deleted file mode 100644 index 6d48105f..00000000 --- a/_docs/getting-started/csdp-introduction.md +++ /dev/null @@ -1,208 +0,0 @@ ---- -title: "Introducing Codefresh" -description: "" -group: getting-started -toc: true ---- - -Codefresh is a full-featured, turn-key solution for application deployments and releases. Powered by Argo, Codefresh uses Argo CD, Argo Workflows, Argo Events, and Argo Rollouts, extended with unique functionality and features essential for enterprise deployments. - -Codefresh offers security, maintainability, traceability, and most importantly, a single control plane for all stakeholders, be they developers, operators, product owners or project managers. - -With Codefresh, you can: - -* Deliver software at scale by managing hundreds or thousands of deployment targets and applications -* Get a secure, enterprise-ready distribution of Argo with built-in identity, RBAC (role-based access control), and secrets -* Gain clear visibility across all deployments and trace changes and regressions from code to cloud in seconds -* Get enterprise-level dedicated support for Argo deployments -* Get insights into every aspect of your CI/CD with smart dashboards -* Manage multiple runtimes and multiple clusters in a single pane of glass - - -### Codefresh deployment models - -Codefresh supports hosted and hybrid deployments: - -* **Hosted** deployment or Hosted GitOps, a hosted and managed version of Argo CD. The SaaS version of Codefresh, the runtime is hosted on a Codefresh cluster (easy setup) and managed by Codefresh (zero maintenance overhead). -Click once to provision the hosted runtime, and start deploying applications to clusters without having to install and maintain Argo CD. - - -* **Hybrid** deployment, with the runtime hosted on the customer cluster and managed by the customer. -The hybrid offering retains runtimes within the customer infrastructure while giving you the power of Argo CD with Codefresh's CI and CD tools, to help achieve continuous integration and continuous delivery goals. - -For details, see [Codefresh architecture]({{site.baseurl}}/docs/getting-started/architecture). - - -### Codefresh and open source Argo -Codefresh brings the power of the Argo project to your Kubernetes deployments: - -* Argo CD for declarative continuous deployment -* Argo Rollouts for progressive delivery -* Argo Workflows as the workflow engine -* Argo Events for event-driven workflow automation framework - -Codefresh creates a conformed fork of the Argo project, providing an enterprise-supported version of the same, enhanced with unique functionality. - - - -### Codefresh and GitOps -Codefresh is GitOps-centric, and supports GitOps from the ground up. Codefresh leverages Argo components to have the entire desired state applied from Git to your Kubernetes cluster, and then reported back to Codefresh. -In addition: - -* Every state change operation in Codefresh is made via Git -* Codefresh audit log is derived from the Git changelog -* Codefresh access control is derived from Git permissions - -For details, see [entity model]({{site.baseurl}}/docs/getting-started/entity-model) and [access control]({{site.baseurl}}/docs/administration/access-control). - - -### Insights in Codefresh -Codefresh makes it easy to both access and visualize critical information for any CI/CD resource at any stage, at any level, and for anyone, from managers to DevOps engineers. - -{::nomarkdown} -
- {:/} - -#### Global deployment analytics - -The Home dashboard presents system-wide highlights in real-time, making it an ideal tool for management. -Get insights into important KPIs for entities across runtimes and clusters, in the same location. View status of runtimes and managed clusters, deployments, failed deployments with rollbacks, most active applications, and Delivery Pipelines. - -{% include - image.html - lightbox="true" - file="/images/incubation/home-dashboard.png" - url="/images/incubation/home-dashboard.png" - alt="Global deployment analytics" - caption="Global deployment analytics" - max-width="70%" -%} - -{::nomarkdown} -
- {:/} - -#### DORA metrics - -DORA metrics has become integral to enterprises wanting to quantify DevOps performance, and Codefresh has out-of-the-box support for it. - - -Apart from the metrics themselves, the DORA dashboard in Codefresh has several features such as the Totals bar with key metrics, filters that allow you to pinpoint just which applications or runtimes are contributing to problematic metrics, and the ability to set a different view granularity for each DORA metric. - -See [DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics/). - -{% include - image.html - lightbox="true" - file="/images/incubation/intro-dora-metrics.png" - url="/images/incubation/intro-dora-metrics.png" - alt="DORA metrics" - caption="DORA metrics" - max-width="60%" -%} - -{::nomarkdown} -
- {:/} - -#### Application analytics and analysis - -The Applications dashboard displays a unified view of applications across runtimes and clusters. No matter what the volume and frequency of your deployments, the Applications dashboard makes it easy to track them. Search for Jira issues, commit messages, committers, and see exactly when and if the change was applied to a specific application. - -See [Monitoring applications]({{site.baseurl}}/docs/deployment/applications-dashboard/). - -{::nomarkdown} -
- {:/} - -#### Delivery Pipelines -The Delivery Pipelines dashboard displays aggregated performance analytics based on the pipeline’s workflows, including step analytics across all the workflows in the pipeline. - -{::nomarkdown} -
- {:/} - -#### Workflows -View and monitor submitted workflows across all pipelines in the Workflows dashboard. Select a time range, or view up to fifty of the most recent workflows for all the pipelines in the runtime. Drill down to any workflow for further analysis. -{::nomarkdown} -
- {:/} - -### CI/CD resources in Codefresh -Wizards make it easy to create delivery pipelines and applications. Smart views and options make it easier to monitor and manage them. -{::nomarkdown} -
- {:/} - -#### Delivery Pipelines - -Delivery Pipelines are where the CI magic happens in Codefresh. Our pipeline creation wizard removes the complexity from creating, validating, and maintaining pipelines. Every stage has multi-layered views of all the related Git change information for the pipeline. -See [Create delivery pipelines]({{site.baseurl}}/docs/pipelines/create-pipeline/). - -{::nomarkdown} -
- {:/} - -#### Workflows -Drill down into a workflow to visualize the connections between the steps in the workflow. -A unique feature is the incorporation of Argo Events into the workflow visualization. You get a unified view of Argo Events and Argo Workflows in the same location, the events that triggered the workflow combined with the workflow itself. - -{::nomarkdown} -
- {:/} - -#### Workflow Templates -Select from ready-to-use Workflow Templates in the Codefresh Hub for Argo or create your own custom template. The **Run** option allows you to test a new Workflow Template, or changes to an existing template, without needing to first commit the changes. - - {% include - image.html - lightbox="true" - file="/images/whats-new/wrkflow-template-main.png" - url="/images/whats-new/wrkflow-template-main.png" - alt="Workflow Templates" - caption="Workflow Templates" - max-width="70%" - %} - -{::nomarkdown} -
- {:/} - -#### Applications -Create GitOps-compliant applications, and manage the application lifecycle in the Codefresh UI. - -Define all application settings in a single location through the intuitive Form mode or directly in YAML, and commit all changes to Git. -For easy access, after commit, the configuration settings are available in the Applications dashboard along with the deployment and resource information. - -See [Applications]({{site.baseurl}}/docs/deployment/create-application/). - -{% include - image.html - lightbox="true" - file="/images/applications/add-app-general-settings.png" - url="/images/applications/add-app-general-settings.png" - alt="Application creation in Codefresh" - caption="Application creation in Codefresh" - max-width="60%" -%} - -### GitOps CI integrations - -If you have Hosted GitOps, and your own CI tools for pipelines and workflows, enrich your deployments with CI information without disrupting existing processes. -Simply connect your CI tools to Codefresh, and our new report image template retrieves the information. For example, add the report image step in your GitHub Actions pipeline and reference the different integrations for Codefresh to retrieve and enrich the image with Jira ticket information. - -See [Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/). - -{% include - image.html - lightbox="true" - file="/images/incubation/github-action-int-settings.png" - url="/images/incubation/github-action-int-settings.png" - alt="Image enrichment with GitHub Actions integration" - caption="Image enrichment with GitHub Actions integration" - max-width="60%" -%} - - -### What to read next -[Quick start tutorials]({{site.baseurl}}/docs/getting-started/quick-start) \ No newline at end of file diff --git a/_docs/getting-started/entity-model.md b/_docs/getting-started/entity-model.md deleted file mode 100644 index 302940c1..00000000 --- a/_docs/getting-started/entity-model.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -title: "Entity model" -description: "" -group: getting-started -toc: true ---- - -The Codefresh entity model is derived from these entity types: -* Codefresh account/user management entities -* Argo ecosystem entities -* Workflow, runtime, and Git Source entities -* Codefresh-specific entities such as pipelines, images, and applications - - - -### Codefresh account/user management entities -The account/user management entity types includes entities that do not share a direct relationship to the Codefresh domain. These are enterprise-specific entities in standard SAAS solutions. - -#### Account -Every user who signs in to Codefresh gets a private administrator user account. - -If you received an invitation to Codefresh, instead of a private user account, you are added as a collaborator to the main account. Your permissions are based on those explicitly assigned to you. - -The number of collaborators in an account is defined by the current plan associated with it. - -#### User -A user in Codefresh is one who has completed the sign-up process, and can log in using authorized third-party systems such as: -* GitHub -* Bitbucket -* GitLab -* Azure -* Google - -> If you configure SSO (Single Sign-On) for the account, the user can log in using only the configured SSO. - -#### Billing -For details, please contact [Sales](mailto:sales@codefresh.io?subject=[Codefresh] Codefresh billing inquiry). - -#### Single Sign-On (SSO) -Enterprise accounts can configure SSO. For details, see [Federated Single Sign-On (SSO) overview]({{site.baseurl}}/docs/administration/single-sign-on/). - -#### Security configuration -Security settings include: -* Inactivity timeout per collaborator account -* Domain restriction for invitations - -### Argo ecosystem entities -Codefresh is built on top of the successful open source Argo project, and as such, supports all the native Argo project-entities. -You can apply every supported entity that exists in the open source projects to your Codefresh account. - -### Workflow -Codefresh shows all the workflows executed with Argo Workflows. -Workflows with pipelines display links to the pipelines. Users can terminate or retry a workflow, and view its logs. - -### Runtime -A runtime represents an installation of Codefresh on the customer's K8s cluster, and contains all the components required to perform all tasks on the cluster. - -Review [Codefresh architecture]({{site.baseurl}}/docs/getting-started/architecture/), and [runtime installation ]({{site.baseurl}}/docs/runtime/installation/). - -### Git Source -A Git Source is a link to a Git repository that stores GitOps-controlled entities. You can create as many as Git Sources as you require. - -To understand how to control Git Sources using GitOps, see [access control]({{site.baseurl}}/docs/administration/access-control/). - -### Codefresh high-level entities -Codefresh creates high-level views that better represents, abstracts, and connects all the different entities in the Argo ecosystem. - -#### CI/CD pipeline -A pipeline is a Codefresh-representation of Argo Events, comprising an Argo Events Sensor and Argo Events Triggers. Every trigger in a sensor becomes a different pipeline in Codefresh. The same sensor can be associated with multiple pipelines, if it has different trigger conditions. - -A pipeline links to the following Argo Events entities: -* Sensor -* Event Source -* Workflow Template (or a cluster-level Workflow Template) - -A pipeline also shows all the workflows created from the triggered event associated with that pipeline. - -#### Image -An image represents a built artifact of a Docker image, reported to Codefresh using a dedicated interface. - -Users can use a predefined [Argo Workflow Template](https://codefresh.io/argohub/workflow-template/codefresh-csdp) to help with transferring the image information to Codefresh. - -#### Application -A holistic view of all your Argo CD and Argo Rollouts deployments that link to the underlying artifacts and workflows. diff --git a/_docs/getting-started/faq.md b/_docs/getting-started/faq.md deleted file mode 100644 index bd189aab..00000000 --- a/_docs/getting-started/faq.md +++ /dev/null @@ -1,52 +0,0 @@ ---- -title: "Frequently asked questions" -description: "" -group: getting-started ---- -We have collected a few of the common questions on the Codefresh solution. - -For questions on Codefresh Classic, navigate to our [FAQs for Codefresh Classic](https://codefresh.io/docs/docs/getting-started/faq/){:target="\_blank"}. - - -**Q. What is the Codefresh platform?** - -A. The Codefresh platform is a full-featured, turn-key solution for application deployments and releases. Powered by the Argo Project, Codefresh uses Argo CD, Argo Workflows, Argo Events, and Argo Rollouts, extended with unique functionality and features essential for enterprise deployments. - -**Q. Which deployment environments does Codefresh support?** - -A. The current release of Codefresh supports hosted and hybrid deployment environments. Stay tuned for our announcement on support for on-premises deployments. - -**Q. How does Codefresh relate to Open Source Argo?** - -A. Codefresh creates a conformed fork of the Argo Project. You get an enterprise-supported version of the Argo Project comprising Argo Workflows, Argo Events, Argo CD, and Argo Rollouts. You can take advantage of the Argo Project offering, with the extended functionality that Codefresh brings to it. - -**Q. I already have a Kubernetes cluster with Argo CD. Can I install Codefresh on the same cluster?** - -A. Hybrid runtimes must be installed on a clean Kubernetes cluster without any Argo Project components. Because we create a conformed fork of the Argo Project in Codefresh, installing it on a cluster with Argo components creates a conflict that will cause the installation to fail. - -**Q. I have resources on my Kubernetes cluster that I want to use in Codefresh. What should I do?** - -A. We will be giving detailed instructions on migrating resources from Kubernetes clusters to Codefresh-based Kubernetes clusters. - -**Q. Does Codefresh support all Git providers?** -A. At the time of writing, Codefresh supports GitHub. We are working to quickly extend support to GitLab and Bitbucket. Stay tuned. - -**Q. What are the browser requirements for the Codefresh UI?** - -A. Officially, we support the latest version of the Chrome browser. Any browser released in the last couple of years should work without major issues. -The following browser versions are **NOT** supported: - -{: .table .table-bordered .table-hover} -| Browser | Version | Date released | -| -------------- | ---------------------------- |-------------------------| -| Chrome | < 51 | May 2016 | -| Firefox | < 54 | Jun 2017 | -| Edge | < 14 | Aug 2016 | -| Safari | < 10 | Sep 2016 | - - -## Migration from Codefresh Classic - -**Q. I have Codefresh Classic. Can I migrate to Codefresh?** -A. At the time of writing, we are working on making the migration from Codefresh Classic to Codefresh as seamless as possible. Stay tuned for the migration announcement. - diff --git a/_docs/getting-started/gitops.md b/_docs/getting-started/gitops.md deleted file mode 100644 index d7542fbb..00000000 --- a/_docs/getting-started/gitops.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: "GitOps approach" -description: "" -group: getting-started -toc: true ---- - -> In the documentation, Kubernetes and K8s are used interchangeably. - -### GitOps - -The Codefresh platform is built entirely around the concept of GitOps, a set of best practices where the entire code delivery process is controlled via Git, including infrastructure and application definition, and automation to complete updates and rollbacks. - -To fully understand the benefits of Codefresh, let's briefly recap GitOps, and how it can help: - -#### Infrastructure as code, the entire system described declaratively - Infrastructure as code is a modern approach that "declaratively" describes the state of a system as code, while having that single source of truth applied to an end-system. The end-systems in most cases are modern cloud native tools. - - Declarative means that configuration is guaranteed by a set of facts, instead of by a set of instructions. With your end system's declarations versioned in Git, you have a single source of truth. You can then both easily deploy and roll back your end system according to the state changes in Git. And more important, if and when disaster strikes, you can also reproduce your cluster’s infrastructure reliably and quickly. - - GitOps is just a specific case of infrastructure as code where the end system is a Kubernetes cluster. - -#### Desired system state versioned in Git - With the declaration of your system stored in a version control system, and serving as your canonical source of truth, you have a single place from which everything is derived and driven. Now not only your application code is in Git, but also all the information required to install and manage your application, including service definition, deployment information, and more. - - Developers can continue with the familiar and convenient approaches they are already using for their applicative code. In addition, Git makes complicated tasks like collaboration (via pull requests), security (via signed commits), permissions (repository permissions), and rollback, as trivial as they can get. - - -#### Use dedicated tools to implement transfer of desired state into the end system - Once the state of your end-system is declared and kept under version control, you need a tool and process to apply the updated desired state into the end system. - - One of the tools for implementing infrastructure as code in the realm of DevOps is [Terraform](https://www.terraform.io/), for example. - - While you can implement GitOps (infrastructure as code for Kubernetes), using a battle-ready tool like Terraform which has a plugin system that also supports Kubernetes, K8s has many nuances that differ from a traditional sync process to a cloud system or some other standard REST API end system. - - To address the specific use cases of Kubernetes, there are new tools dedicated to implementing GitOps (infrastructure as code for k8s), such as [ArgoCD](https://github.com/argoproj/argo-cd). - - diff --git a/_docs/getting-started/main-concepts.md b/_docs/getting-started/main-concepts.md deleted file mode 100644 index c2d4f418..00000000 --- a/_docs/getting-started/main-concepts.md +++ /dev/null @@ -1,37 +0,0 @@ ---- -title: "Main concepts" -description: "" -group: getting-started -toc: true ---- - -### Built on top of the open source Argo -Codefresh maintains a [conformed](https://github.com/argoproj/argo-conformance-program) fork of the following Argo components, providing an enterprise-supported version of them: -* [Argo CD](https://github.com/argoproj/argo-cd): Declarative continuous deployment for Kubernetes. -* [Argo Rollouts](https://argoproj.github.io/argo-rollouts/): Progressive Delivery for Kubernetes. -* [Argo Workflows](https://github.com/argoproj/argo-workflows): Workflow engine for Kubernetes. -* [Argo Events](https://github.com/argoproj/argo-events): Event-driven workflow automation framework. - -For details, see [Codefresh architecture]({{site.baseurl}}/docs/getting-started/architecture/). - -### Hybrid behind firewall model -Codefresh performs an installation, called a Runtime, on the user's K8s cluster. The Runtime contains all required components for the Codefresh experience. - -For details, see [Codefresh architecture]({{site.baseurl}}/docs/getting-started/architecture/). - -### GitOps native approach -Codefresh is built entirely on the heavily-adopted concept of GitOps. Read the detailed explanation on our [GitOps approach]({{site.baseurl}}/docs/getting-started/gitops/).
-Codefresh leverages Argo components (Argo CD and Argo Events), to have the entire desired state applied from Git to the user's K8s cluster, and also reported back to Codefresh platform. - -### Every state change operation in Codefresh is made via Git -Codefresh has taken the GitOps approach a step forward by making our entire entity model fully controlled by GitOps via Codefresh, meaning that the entire state of your account is maintained in Git. For details, see [entity model]({{site.baseurl}}/docs/getting-started/entity-model/). - -Codefresh provides a full front-end experience powered by a strong API layer (GraphQL), and every state change (via GraphQL mutation) actually performs a commit on behalf of the user to Git. - -### Audit log derived from Git changelog -Codefresh has built its sophisticated but simple audit log on all operations to the system, for both the Git change and the log of API calls that have been made to the system. -For details, see [audit]({{site.baseurl}}/docs/administration/audit/). - -### Access control derived from Git permissions -Codefresh has built its sophisticated but simple access control model on top of the existing Git operations that are defined externally to the system.
-For details, see [access control]({{site.baseurl}}/docs/administration/access-control/). diff --git a/_docs/getting-started/quick-start.md b/_docs/getting-started/quick-start.md deleted file mode 100644 index 203c03d0..00000000 --- a/_docs/getting-started/quick-start.md +++ /dev/null @@ -1,55 +0,0 @@ ---- -title: "Quick start" -description: "" -group: getting-started -toc: true ---- - -Check out our quick start tutorial to get up and running in the Codefresh platform with hosted or hybrid runtimes. - -The tutorial is divided into these sections: -* Provisioning runtimes -* Creating and deploying an application -* Triggering and creating a Delivery Pipeline - -Each section indicates the runtime environment it is relevant to. - -### Provision runtimes -Based on your deployment model, start by provisioning the hosted or hybrid runtime. Hosted and hybrid runtimes can co-exist with each other. - - -#### Hosted -Hosted runtimes are hosted on a Codefresh cluster and managed by Codefresh. You need to provision your hosted runtime once for your account. - -1. [Provision a hosted runtime]({{site.baseurl}}/docs/getting-started/quick-start/install-hosted) - Provision the hosted runtime with a single click, and complete the setup for your hosted environment. - -{::nomarkdown} -
-{:/} - -#### Hybrid -Hybrid runtimes: Hosted on a customer cluster and managed by the customer. You can provision multiple hybrid runtimes in the same account. - -1. [Prepare for hosted runtime installation]({{site.baseurl}}/docs/getting-started/quick-start/verify-requirements) - Verify your environment matches the requirements for installing Codefresh runtime. -1. [Install hybrid runtime]({{site.baseurl}}/docs/getting-started/quick-start/runtime) - Install the Codefresh runtime by downloading the CLI, installing the runtime, and validate successful installation in the UI - -### Deploy an application - -1. [Create an application]({{site.baseurl}}/docs/getting-started/quick-start/create-app-ui) - Create the `codefresh-guestbook` application in the Codefresh UI. -1. [Create and commit resources for application]({{site.baseurl}}/docs/getting-started/quick-start/create-app-specs) - Create rollout and service resources, and commit these resources to deploy the `codefresh-guestbook` application. -1. [Update the image tag for application]({{site.baseurl}}/docs/getting-started/quick-start/create-rollout) - Update the image for the `codefresh-guestbook` application to trigger a rollout. - -### Trigger/create a Delivery Pipeline -> Available for hybrid deployments. - -1. [Trigger the Hello World example pipeline]({{site.baseurl}}/docs/getting-started/quick-start/hello-world) - Configure the Git event to trigger the demo pipeline. -1. [Create a basic CI delivery pipeline]({{site.baseurl}}/docs/getting-started/quick-start/create-ci-pipeline) - Create a new CI delivery pipeline in Codefresh. - diff --git a/_docs/getting-started/quick-start/create-app-specs.md b/_docs/getting-started/quick-start/create-app-specs.md deleted file mode 100644 index 1c44be1f..00000000 --- a/_docs/getting-started/quick-start/create-app-specs.md +++ /dev/null @@ -1,131 +0,0 @@ ---- -title: "Create and commit resources for application" -description: "" -group: getting-started -sub-group: quick-start -toc: true ---- - -Now that you have created an application, you need to deploy the application. Let's deploy the `codefresh-guestbook` application by creating and commiting resources. -You will create and commit the following resources: -1. A folder in Git to save resources for the application -1. `Rollout` resource defining the deployment strategy -1. `Service` resource to expose the application to external traffic - -### Before you begin -* [Create an application]({{site.baseurl}}/docs/getting-started/quick-start/create-app-ui) -* Make sure [Argo Rollouts is installed]({{site.baseurl}}/docs/deployment/install-argo-rollouts) on the target cluster - -### Create folder in Git for application resources -Create a folder in the Git repo in which to save all the resources for the `codefresh-guestbook` application. - -* In your Git repo, create a folder to store the resources needed to deploy the application. - For example, `/quick-start/` - -### Create rollout.yaml - -Create a rollout resource for the application you want to deploy. - - -To leverage Argo Rollouts' deployment capabilities, we are using the Argo's `rollout` resource instead of the native Kubernetes Deployment object. -For detailed information on the fields you can define, see [Argo Rollout specification](https://argoproj.github.io/argo-rollouts/features/specification/){:target="\_blank"}. - - -* In the Git repository create the `rollout.yaml` file, as in the example below. - - -```yaml -apiVersion: argoproj.io/v1alpha1 -kind: Rollout -metadata: - name: codefresh-guestbook-rollout -spec: - replicas: 4 - revisionHistoryLimit: 2 - selector: - matchLabels: - app: codefresh-guestbook - template: - metadata: - labels: - app: codefresh-guestbook - spec: - containers: - - image: gcr.io/heptio-images/ks-guestbook-demo:0.1 - name: codefresh-guestbook - ports: - - name: http - containerPort: 80 - protocol: TCP - minReadySeconds: 30 - strategy: - canary: - steps: - - setWeight: 25 - - pause: {duration: 20s} - - setWeight: 75 - - pause: {duration: 15s} -``` - -#### Fields in `rollout.yaml` - -{: .table .table-bordered .table-hover} -| Rollout Field | Notes | -| -------------- | -------------| -| `replicas` | When deployed, the rollout creates four replicas of the `codefresh-guestbook` application.| -| `revisionHistoryLimit` | The number of replica sets to retain. | -| `matchLabels` | The pods to select for this rollout. In our example, all pods with the label `codefresh-guestbook` are selected.| -| `image` | The container image for the application with the version tag, `gcr.io/heptio-images/ks-guestbook-demo:0.1` in our example.| -| `name` | The name of the application, `codefresh-guestbook` in our example. | -| `canary` | The deployment strategy, `canary` meaning that the traffic is gradually routed to the new application. Starting with `setWeight` of `25%` followed by a `pause` of 20 seconds, and the remaining `75%` after verification.| -| `templateName` | The analysis template used to validate the application metrics. Our example has the `background-analysis` template, and interfaces with Prometheus to monitor and validate metric thresholds.| - - -### Create a service resource -Create a service resource to expose your application to external traffic. - -* Create a `service.yaml` resource for the application you want to deploy, as in the example below. - > Create it in the same folder in which you saved `rollout.yaml`. - -```yaml -apiVersion: v1 -kind: Service -metadata: - name: codefresh-guestbook-svc -spec: - ports: - - port: 8080 - targetPort: 80 - selector: - app: codefresh-guestbook # must be the same as the selector defined in rollouts.yaml - type: LoadBalancer -``` - -#### Fields in `service.yaml` - -{: .table .table-bordered .table-hover} -| Service field | Notes | -| -------------- | -------------- | -| `spec.ports` | The internal `port`, 8080 in our example, and external `targetPort`, 80 in our example.| -| `selector.app` | The pods to select, and MUST be identical to that defined in `rollouts.yaml`, `codefresh-guestbook` in our example.| - -### View application resources in Codefresh -Once you create and commit the `rollout` and `service` resources, return to the Applications dashboard. The Current State to see these resources. - -1. In the Codefresh UI, go to the [Applications dashboard](https://g.codefresh.io/2.0/applications-dashboard?sort=desc-lastUpdated){:target="\_blank"}. -1. Select the application. - The Current State tab is now populated with the `rollout` and `service` resources you added. - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/cdops-app-current-state.png" - url="/images/getting-started/quick-start/cdops-app-current-state.png" - alt="Current State with resources for application" - caption="Current State with resources for application" - max-width="70%" - %} - -### What to do next - -[(Optional) Update image tag for application]({{site.baseurl}}/docs/getting-started/quick-start/create-rollout) \ No newline at end of file diff --git a/_docs/getting-started/quick-start/create-app-ui.md b/_docs/getting-started/quick-start/create-app-ui.md deleted file mode 100644 index b2fdff6b..00000000 --- a/_docs/getting-started/quick-start/create-app-ui.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -title: "Create an application" -description: "" -group: getting-started -sub-group: quick-start -toc: true ---- - -Let's start by creating a simple application, the `codefresh-guestbook` application in the Codefresh UI. -We'll create the application without resources and then define/add resources in the next step. - - -For detailed information, see [Create an application]({{site.baseurl}}/docs/deployment/create-application). - - -**How to** - - -1. In the Codefresh UI, go to the [Applications](https://g.codefresh.io/2.0/applications-dashboard?sort=desc-lastUpdated){:target="\_blank"} dashboard. -1. Select **Add Application** on the top-right. -1. In the Add Application panel, add definitions for the application: - * **Application name**: `codefresh-guestbook` for the quick start. - * **Runtime**: The runtime to associate with the application, `hosted-runtime` for the quick start. - * **Name for YAML file**: The name of the application's configuration manifest, assigned on commit to Git. By default, the manifest is assigned the application name. - You can click the Edit icon and change the name, if needed. - - >You cannot change the application definitions once you continue to the Configuration settings. - -{% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/cdops-add-app-settings.png" - url="/images/getting-started/quick-start/cdops-add-app-settings.png" - alt="Add Application panel" - caption="Add Application panel" - max-width="50%" - %} - -{:start="4"} -1. Select **Next** to go to the Configuration tab. - By default you are in Form mode. You can toggle between the Form and YAML modes as you define the application's configuration settings. -1. Define the **General** settings for the application: - * **Repository URL**: The URL to the repo in Git where you created the YAML resource files for the application. - * **Revision**: The branch in Git with the resource files. - * **Path**: The folder in Git with the resource files. - * **Namespace**: Optional. For the quick start, we'll create a namespace for the application, entitled `quick-start`. - * **Sync Policy**: Change to **Automatic**, and select **Prune resources** to automatically remove unused resources. - * **Sync Options**: If you defined a namespace, select **Auto-create namespace** to ensure that the namespace is created if it doesn't exist. - -{% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/cdops-add-app-configuration.png" - url="/images/getting-started/quick-start/cdops-add-app-configuration.png" - alt="Add Application Quick Start: General settings" - caption="Add Application Quick Start: General settings" - max-width="70%" - %} - - -{:start="6"} -1. Retain the default **Advanced Settings**. -1. To commit all your changes, select **Commit**. - The Commit form is displayed with the application's definitions on the left, and the read-only version of the manifest with the configuration settings you defined on the right. -1. Select the **Git Source** to which to commit. - -{% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/cdops-add-app-commit.png" - url="/images/getting-started/quick-start/cdops-add-app-commit.png" - alt="Add Application Quick Start: Commit to Git" - caption="Add Application Quick Start: Commit to Git" - max-width="70%" - %} - -{:start="9"} -1. Add a commit message and then select **Commit** at the bottom-right of the panel. - You are directed to the [Applications dashboard](https://g.codefresh.io/2.0/applications-dashboard?sort=desc-lastUpdated){:target="\_blank"}. - You may have to wait for a few seconds until the application is synced to the cluster. - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/cdops-add-app-dashboard.png" - url="/images/getting-started/quick-start/cdops-add-app-dashboard.png" - alt="Application dashboard with new application" - caption="Application dashboard with new application" - max-width="70%" - %} - -{:start="10"} -1. Select the application. The Current State tab does not display any resources as we have not created any resources for the application. - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/cdops-app-empty-current-state.png" - url="/images/getting-started/quick-start/cdops-app-empty-current-state.png" - alt="Empty Current State for new application" - caption="Empty Current State for new application" - max-width="70%" - %} - - -In the next task, you will create and commit resources for the `codefresh-guestbook` application and deploy the application. - - -### What to do next -[Create and commit resources for application]({{site.baseurl}}/docs/getting-started/quick-start/create-app-specs/) diff --git a/_docs/getting-started/quick-start/create-ci-pipeline.md b/_docs/getting-started/quick-start/create-ci-pipeline.md deleted file mode 100644 index 5a297b78..00000000 --- a/_docs/getting-started/quick-start/create-ci-pipeline.md +++ /dev/null @@ -1,186 +0,0 @@ ---- -title: "Create a basic CI delivery pipeline" -description: "" -group: getting-started -sub-group: quick-start -toc: true ---- - -Now that you have configured and run the Hello World demo pipeline, let's create a more advanced pipeline. - -For the quick start, you'll create a basic CI Delivery Pipeline in Codefresh. - -The Delivery Pipeline: -* Clones a Git repository -* Builds a docker image using `kaniko` -* Pushes the built image to a Docker Registry -* Runs an example testing step -* Sends the image information to Codefresh - -Our CI pipeline interacts with third-party services such as GitHub and a Docker Registry. You need to first add secrets to the cluster to store the credentials required. - - -### Create a Personal Access Token (PAT) -You must have a PAT to clone the repository. - - -1. Create your PAT (Personal Access Token) with a valid `expiration` date and `scope`. - Scopes: `repo` and `admin-repo.hook` - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/quick-start-git-event-permissions.png" - url="/images/getting-started/quick-start/quick-start-git-event-permissions.png" - alt="GitHub PAT permissions for CI pipeline" - caption="GitHub PAT permissions for CI pipeline" - max-width="30%" - %} - -{:start="2"} -1. Define your PAT and namespace by replacing the values in these commands: - -``` - export GIT_TOKEN=[PAT token] - export NAMESPACE=[Codefresh runtime namespace] -``` - -1. Create a generic Kubernetes secret with your PAT token: - -``` -kubectl create secret generic github-token \ - --from-literal=token=$GIT_TOKEN --dry-run=client \ - --save-config -o yaml | kubectl apply -f - -n $NAMESPACE -``` - -### Create Docker-registry secret -To push the image to a Docker registry, we'll need the credentials on our cluster. - -> The Docker registry secret is different from the general registry secret. - -1. Export the values for the Docker registry's `server`, `username`, `password`, `email`, and `namespace`: - -``` -export DOCKER_REGISTRY_SERVER=[Server] -export DOCKER_USER=[Username] -export DOCKER_PASSWORD=[Password] -export DOCKER_EMAIL=[Email] -export NAMESPACE=[Codefresh runtime namespace] -``` - -{:start="2"} -1. Create the secret: - -``` -kubectl create secret docker-registry\ - --docker-server=$DOCKER_REGISTRY_SERVER \ - --docker-username=$DOCKER_USER \ - --docker-password=$DOCKER_PASSWORD \ - --docker-email=$DOCKER_EMAIL -n $NAMESPACE -``` - - > In the Workflow Template, the Docker registry name defaults to `docker-config`. - - -### Create general registry secret -Create a general registry secret to send the image information to Codefresh. - -1. Export the values for your registry's `username`, `password`, `domain`, and `namespace`: - -``` -export USER=[Username] -export PASSWORD=[Password] -export DOMAIN=[Domain] -export NAMESPACE=[Codefresh runtime namespace] -``` - -{:start="2"} -1. Create the secret: - -``` -kubectl create secret generic registry-creds \ - --from-literal=username=$USER \ - --from-literal=password=$PASSWORD \ - --from-literal=domain=$DOMAIN \ - --dry-run=client --save-config -o yaml | kubectl apply -f - -n $NAMESPACE -``` - -### Create the CI delivery pipeline -Now that you have defined the secrets, create the CI delivery pipeline in Codefresh. - -1. In the UI, go to [Delivery Pipelines](https://g.codefresh.io/2.0/pipelines){:target="\_blank"}. -1. Select **+ Add Delivery Pipeline**. - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/quick-start-new-pipeline.png" - url="/images/getting-started/quick-start/quick-start-new-pipeline.png" - alt="Add Delivery Pipeline panel in Codefresh" - caption="Add Delivery Pipeline panel in Codefresh" - max-width="30%" - %} - -{:start="3"} -1. Enter a name for the delivery pipeline. - The name is created from the names of the sensor and the trigger event for the delivery pipeline. - * **Sensor Name**: The name of the sensor resource, for example, `sensor-cf-ci`. - * **Trigger Name**: The event configured in the sensor to trigger the Workflow Template, for example, `push-cf-ci`. -1. From the list of **Git Sources**, select the Git Source to which to commit the resources for this delivery pipeline. - > Do not select the marketplace Git Source as you cannot commit to it. - If you have multiple runtimes installed, the Git Source you select also determines the runtime that executes the pipeline. -1. Select **Next**. - In the **Configuration** tab, **Workflow Templates** is selected. This is our CI Starter Workflow Template, that builds a Docker image using Kaniko, reports image metadata to Codefresh, and tests the image. -1. Select **Trigger Conditions**. -1. From the **Add** dropdown, select **Git Events**. -1. In the **Git Repository URLs** field, select one or more GitHub repositories to listen to for the selected event. -1. From the **Event** dropdown, select the event, in our case, **Commit pushed**. - Codefresh displays all the **Arguments** available for the selected event. - You can map each argument to a single or combination of predefined variables, which Codefresh automatically maps to the correct path when you commit the changes. Argo Workflow then instantiates the values from the event payload. - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/quick-start-ci-pipeline-arguments.png" - url="/images/getting-started/quick-start/quick-start-ci-pipeline-arguments.png" - alt="Predefined variables for arguments" - caption="Predefined variables for arguments" - max-width="30%" - %} - - In each field, type `$` and from the list of predefined variables, select each of these in turn: - - * **REPO**: Required. The repository to clone during the build step. Select `Repository name`. - * **IMAGE_NAME**: Required. The name for the built image. Enter the name in the format `([docker_url]/[account]/[image_name]`. - * **TAG**: Optional. The tag for the built image. If not defined, uses the default tag `latest`. Enter `1.0`. - * **GIT_REVISION**: Optional. The Git revision to report to Codefresh. Select `Git revision`. - * **GIT_BRANCH**: Optional. The Git branch to report to Codefresh. Select `Git branch`. - * **GIT_COMMIT_URL**: Optional. The Git commit URL to report to Codefresh. Select `Commit url`. - * **GIT_COMMIT_MESSAGE**: Optional. The Git commit message to report to Codefresh. Select `Commit message`. - - You are now ready to commit the delivery pipeline to the Git Source. - -{:start="10"} -1. Select **Apply**, and then **Commit** on the top-right. - The Commit Changes panel shows the files to be committed. -1. Enter the commit message and then select **Commit**. -1. In the **Delivery Pipelines** page to which you are redirected, verify that your pipeline is displayed. - - Behind the scenes, we committed the pipeline to your Git repository and synced the resources to your cluster. - It may take a few seconds for the Git-to-cluster sync to complete, and then your pipeline should be displayed. - -### Trigger the pipeline with a Git commit event -Make a change to a file in the Git repository to trigger the pipeline. - -1. Go to the Git repository selected for the trigger condition. -1. Make a change to any file to get a commit event. -1. In the UI, go back to [Delivery Pipelines](https://g.codefresh.io/2.0/pipelines){:target="\_blank"} to see the new workflow for the pipeline. - -Continue to tweak the pipeline and enhance its capabilities. - - -### What to do next -If you have not created an application in Codefresh, continue with: - -[Create resources for codefresh-guestbook application]({{site.baseurl}}/docs/getting-started/quick-start/create-app-specs) - diff --git a/_docs/getting-started/quick-start/create-github-action-ci.md b/_docs/getting-started/quick-start/create-github-action-ci.md deleted file mode 100644 index b95544b6..00000000 --- a/_docs/getting-started/quick-start/create-github-action-ci.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: "Connect a GitHub Action CI to enrich image" -description: "" -group: getting-started -sub-group: quick-start -toc: true ---- \ No newline at end of file diff --git a/_docs/getting-started/quick-start/create-rollout.md b/_docs/getting-started/quick-start/create-rollout.md deleted file mode 100644 index 763df47a..00000000 --- a/_docs/getting-started/quick-start/create-rollout.md +++ /dev/null @@ -1,76 +0,0 @@ ---- -title: "Update image tag for application" -description: "" -group: getting-started -sub-group: quick-start -toc: true ---- - -You will now make a change in the application manifest, and update the image tag. Because we selected auto-sync in the application settings, Argo CD detects that the live state in the cluster is out of sync with the desired state in Git, and triggers the new rollout. - -### Before you begin - -* [Create resources for application]({{site.baseurl}}/docs/getting-started/quick-start/create-app-specs/) - - -### Update image tag in rollout.yaml -Update the image tag in the `codefresh-guestbook` application. - -1. Go to the Git repo with `rollout.yaml`. -1. Update the image tag from `0.1` to `0.2` as in the example below. - -```yaml -... -template: - metadata: - labels: - app: codefresh-guestbook - spec: - containers: - - image: gcr.io/heptio-images/ks-guestbook-demo:0.2 - name: codefresh-guestbook - ports: - - name: http - containerPort: 80 - protocol: TCP -... -``` -{:start="3"} -1. Commit the change. - -### View the rollout in the Applications dashboard -When the image tag is updated, the auto-sync initiates the rollout. - -1. Go back to the [Applications dashboard](https://g.codefresh.io/2.0/applications-dashboard?sort=desc-lastUpdated){:target="\_blank"}. -1. Select the application you created. - The deployment entry for the application is displayed as progressing. - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/cdops-app-rollout-in-dashboard.png" - url="/images/getting-started/quick-start/cdops-app-rollout-in-dashboard.png" - alt="Application dashboard with rollout in progress" - caption="Application dashboard with rollout in progress" - max-width="60%" - %} - -{:start="3"} -1. To visualize the rollout analysis, click the rollout name. - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/cdops-app-rollout-panel.png" - url="/images/getting-started/quick-start/cdops-app-rollout-panel.png" - alt="Rollout analysis in progress" - caption="Rollout analysis in progress" - max-width="60%" - %} - -{:start="4"} -1. To view metric validation details, expand **Background Analysis** in the panel. - -You have created and deployed an application in Codefresh. - - \ No newline at end of file diff --git a/_docs/getting-started/quick-start/hello-world.md b/_docs/getting-started/quick-start/hello-world.md deleted file mode 100644 index 02011074..00000000 --- a/_docs/getting-started/quick-start/hello-world.md +++ /dev/null @@ -1,86 +0,0 @@ ---- -title: "Trigger the Hello World example pipeline" -description: "" -group: getting-started -sub-group: quick-start -toc: true ---- - -Now that you have successfully installed the hybrid runtime, you can trigger one of the Hello World demo pipelines included in the runtime package. -The two Hello World example pipelines are triggered by different event conditions: -* Git (GitHub) event -* Calendar (cron) event - -For the quick start, let's focus on the `github/hello-world` pipeline. - -### Create a PAT token -To commit resources for the `github/hello-world` pipeline, you need to add a PAT to Codefresh. - -1. Create your personal token with a valid `expiration` date and `scope` with `base64` encoding. - For the pipeline, you need `repo` and `admin-repo.hook` scopes: - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/quick-start-git-event-permissions.png" - url="/images/getting-started/quick-start/quick-start-git-event-permissions.png" - alt="GitHub PAT permissions for Hello World pipeline" - caption="GitHub PAT permissions for Hello World pipeline" - max-width="30%" - %} - -{:start="2"} -1. In the Codefresh UI, go to [User Settings](https://g.codefresh.io/2.0/user-settings){:target="\_blank"}, add your token. - -### View pipelines -View the pipelines in Codefresh. - -1. In the Codefresh UI, go to [Delivery Pipelines](https://g.codefresh.io/2.0/pipelines){:target="\_blank"}. - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/quick-start-pipelines.png" - url="/images/getting-started/quick-start/quick-start-pipelines.png" - alt="Demo pipelines in the Pipelines page" - caption="Demo pipelines in the Pipelines page" - max-width="30%" - %} - - * The `github/hello-world` pipeline has not been triggered as it requires a Git event to trigger it. - * The `cron/hello-world` pipeline shows statistics as it has already been triggered based on the `cron` interval. - -### View and update manifest -As we don't have a workflow for this pipeline, you will configure the Git Source resource in the pipeline's **Manifest** tab. -1. In the **Pipelines** page, to drill down, select the pipeline name. -1. Select the **Manifest** tab, and click the arrowhead to expand the resource view. - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/quick-start-manifest-expand.png" - url="/images/getting-started/quick-start/quick-start-manifest-expand.png" - alt="Expand resource view in Mainfests tab" - caption="Expand resource view in Mainfests tab" - max-width="30%" - %} - - You can see these resources: - - * Event Source (`event-source.git-source.yaml`). - * Sensor (`sensor.git-source.yaml`) - * Workflow Template (`workflow-template.hellow-world.yaml`) - - - > The pipeline is configured to run on a `PUSH` event in the Git repository. - - -Codefresh does the following: -* Commits the changes to your Git repository. -* Synchronizes the changes in Git back to your cluster, and updates the `event-source.git-source` resource. -* Triggers this pipeline after the `PUSH` event to your repository. -* Creates a workflow. View it in the UI, in the [Workflows](https://g.codefresh.io/2.0/workflows){:target="\_blank"} dashboard. - Select view workflow details to see the workflow log. - -### What to do next -[Create a basic CI pipeline]({{site.baseurl}}/docs/getting-started/quick-start/create-ci-pipeline) diff --git a/_docs/getting-started/quick-start/install-hosted.md b/_docs/getting-started/quick-start/install-hosted.md deleted file mode 100644 index 8d3c6fbf..00000000 --- a/_docs/getting-started/quick-start/install-hosted.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -title: "Provision a hosted runtime" -description: "" -group: getting-started -sub-group: quick-start -toc: true ---- - -If you have Hosted GitOps, set up your hosted runtime environment: - -1. Provision the hosted runtime with a single click -1. Authorize access through your OAuth token to the organization where Codefresh creates the Git runtime repo and the shared configuration repo -1. Connect to an external K8s cluster with access to the internet, to which you can deploy applications -1. Install Argo Rollouts on the cluster - -Read our [blog on Hosted GitOps](https://codefresh.io/blog/codefresh-upends-continuous-delivery-with-hosted-gitops-platform-featuring-dora-dashboards-and-first-class-integrations-for-ci/). -For detailed information on each of the steps below, see [Set up a hosted runtime environment]({{site.baseurl}}/docs/runtime/hosted-runtime/). - -**Before you begin** - -Verify the following: -* If you have hybrid runtimes installed, make sure you have latest version of the CLI - * Check version: - `cf version` - To compare with the latest version from Codefresh, [click here](https://github.com/codefresh-io/cli-v2/releases){:target="\_blank"}. - * [Download the CLI]({{site.baseurl}}/docs/clients/csdp-cli/). -* Kubernetes cluster with access to the internet -* OAuth token - -**How to** -1. In the Codefresh UI, go to Codefresh [Home](https://g.codefresh.io/2.0/?time=LAST_7_DAYS){:target="\_blank"}. - -{% include -image.html -lightbox="true" -file="/images/runtime/hosted-initial-view.png" -url="/images/runtime/hosted-initial-view.png" -alt="Hosted GitOps setup" -caption="Hosted GitOps setup" -max-width="80%" -%} - -{:start="2"} -1. Provision the hosted runtime: - * Click **Install**, and wait for Codefresh to complete provisioning your hosted runtime (may take up to ten minutes). - -{% include -image.html -lightbox="true" -file="/images/runtime/hosted-installing.png" -url="/images/runtime/hosted-installing.png" -alt="Installing hosted runtime" -caption="Installing hosted runtime" -max-width="80%" -%} - -{:start="3"} -1. Select the Git organization for the runtime installation and shared configuration repos: - * Click **Connect**. - * Click **Authorize Access** and enter your OAuth token. - * Select the **Git Organization for which to create the repos**. - * Click **Create**. - Codefresh creates the two Git repositories in the paths shown. - - {% include -image.html -lightbox="true" -file="/images/runtime/hosted-connect-git.png" -url="/images/runtime/hosted-connect-git.png" -alt="Connect to Git provider" -caption="Connect to Git provider" -max-width="80%" -%} - -{:start="4"} -1. Connect a K8s cluster: - * Click **Connect**. - * In the Add Managed Cluster panel, copy the command `cf cluster add`, and run it in the terminal. - * When prompted to select the `kube-context`, select from the list of available clusters as defined in `kubeconfig`. - * Verify that you have configured access to the required IP addresses required. See [Codefresh IP addresses]({{site.baseurl}}/docs/administration/platform-ip-addresses/). - -{% include -image.html -lightbox="true" -file="/images/runtime/hosted-connect-cluster-step.png" -url="/images/runtime/hosted-connect-cluster-step.png" -alt="Connect a K8s cluster for hosted runtime" -caption="Connect a K8s cluster for hosted runtime" -max-width="70%" -%} - -1. Install Argo Rollouts on the cluster you added. You'll need this to apply the `rollout` resource we will create for the application in the next task. - * Go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. - * Select **Topology View**. - * Select the target cluster, and then select **+ Install Argo Rollouts**. - -{% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/cdops-app-install-rollout.png" - url="/images/getting-started/quick-start/cdops-app-install-rollout.png" - alt="Install Argo Rollouts on managed cluster" - caption="Install Argo Rollouts on managed cluster" - max-width="50%" - %} - -### What to do next -[Create resources for codefresh-guestbook application]({{site.baseurl}}/docs/getting-started/quick-start/create-app-specs) - diff --git a/_docs/getting-started/quick-start/runtime.md b/_docs/getting-started/quick-start/runtime.md deleted file mode 100644 index 9ac99d86..00000000 --- a/_docs/getting-started/quick-start/runtime.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -title: "Install a hybrid runtime" -description: "" -group: getting-started -sub-group: quick-start -toc: true ---- - - -Install the hybrid runtime on your K8s cluster. Installing the hybrid runtime installs Argo-project and Codefresh-specific components. The Argo Project is an enterprise-supported version of the Argo CD components, derived from a conformed fork of the Argo ecosystem. - -### About hybrid runtime installation -Installing a hybrid runtime includes installing the: -1. Codefresh CLI. -2. Codefresh hybrid runtime from the CLI in a specific namespace on your cluster. - Every hybrid runtime installation makes commits to two Git repos: - * Runtime installation repo: The installation repo that manages the runtime itself with Argo CD. If the repo URL you provide does not exist, the runtime creates it automatically. - * Git Source repo: Created automatically during runtime installation. The repo with the demo resources required for the sample `Hello World` pipelines we provide. - * Shared configuration repo: A repository that stores configuration manifests shared across runtimes. - -### Before you begin -A hybrid runtime requires a Git token for authentication to the Git installation repository. -Have your GitHub Personal Authentication Token (PAT) ready with a valid expiration date and access permissions: -* Expiration: Either the default of 30 days or any duration you consider logical. -* Access scopes: Set to `repo` and `admin-repo.hook` - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/quick-start-git-event-permissions.png" - url="/images/getting-started/quick-start/quick-start-git-event-permissions.png" - alt="GitHub PAT permissions" - caption="GitHub PAT permissions" - max-width="30%" - %} - - If you need detailed information on GitHub tokens, see the [GitHub article](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token). - -### Download the Codefresh CLI -Downloading the Codefresh CLI requires you to select the download mode and OS, generate an API key, and authentication context, as instructed in the UI. -1. In the Welcome page, select **+ Install Runtime**. -1. Download the Codefresh CLI: - * Select one of the methods. - * Generate the API key and create the authentication context. - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/quick-start-download-cli.png" - url="/images/getting-started/quick-start/quick-start-download-cli.png" - alt="Download Codefresh CLI to install runtime" - caption="Download Codefresh CLI to install runtime" - max-width="30%" - %} -### Install hybrid runtime -For the quick start, install the hybrid runtime through the Codefresh CLI that you downloaded previously. - -1. To start runtime installation, run `cf runtime install`. - >If you don't have a valid SSL certificate for the Ingress controller, and want to continue with the installation, add the `--insecure` flag to the runtime command. -1. Follow the prompts in the CLI wizard to complete the installation: - * **Runtime name**: The name of your runtime, starting with a lower-case character, and including up to 63 characters and numbers. Example: `codefreshproduction` - * **Select Kube context**: Your current context is highlighted. Press Enter to select it, or use the arrow keys to select a different context. - * **Ingress class**: Select the ingress class for runtime installation from the list displayed. - * **Ingress host**: Displays the NGINX host, either from the cluster or the NGINX ingress controller associated with the **Ingress class**. - * **Repository URL**: The GitHub repo for the installation definitions, in the format `https://github.com/[user-or-org-name]/[repo_name]`. Example: `https//:github.com/codefresh/cf_production_install` - * **Git runtime token**: The GitHub PAT for access to the installation repo. - * **Install Codefresh demo resources?** Press Enter to confirm. Demo resources are saved in a new Git Source repo, created by Codefresh. They include resources for two Hello World pipelines, one with a Cron trigger condition, and the other with a Git event trigger condition. - * **Do you wish to continue with runtime install?** Press Enter to confirm and start runtime installation. -1. Wait for the runtime installed successfully message. - -### Validate successful installation -The **Runtimes** dashboard shows the hybrid runtime you just installed. You can drill down into the runtime to see its components and Git Sources. - -1. In the Codefresh UI, go to the [**Runtimes**](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"} dashboard. - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/quick-start-runtime-dashboard.png" - url="/images/getting-started/quick-start/quick-start-runtime-dashboard.png" - alt="Runtime dashboard after successful installation" - caption="Runtime dashboard after successful installation" - max-width="30%" - %} - -{:start="2"} -1. Select the runtime name to drill down, and then select the tabs to view the runtime components and Git Sources. - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/quick-start-runtime-components.png" - url="/images/getting-started/quick-start/quick-start-runtime-components.png" - alt="Runtime components tab" - caption="Runtime Components tab" - max-width="30%" - %} - - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/quick-start-git-source.png" - url="/images/getting-started/quick-start/quick-start-git-source.png" - alt="Git Source tab" - caption="Git Source tab" - max-width="30%" - %} - -### What to do next -[Create resources for codefresh-guestbook application]({{site.baseurl}}/docs/getting-started/quick-start/create-app-specs) -OR -[Trigger the Hello World example pipeline]({{site.baseurl}}/docs/getting-started/quick-start/hello-world) diff --git a/_docs/getting-started/quick-start/verify-requirements.md b/_docs/getting-started/quick-start/verify-requirements.md deleted file mode 100644 index 3fc48c3e..00000000 --- a/_docs/getting-started/quick-start/verify-requirements.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -title: "Prepare for hybrid runtime installation" -description: "" -group: getting-started -sub-group: quick-start -toc: true ---- - - -**New installation** -If this is your first time installing Codefresh, review and confirm that your deployment environment conforms to the minimum requirements for hybrid runtime installation. Check the [system requirements]({{site.baseurl}}/docs/runtime/requirements). - -**Existing installation** -If you already have a hybrid runtime installation on your cluster, you have two options: -1. To install on the same cluster, first uninstall the existing hybrid runtime. Currently, you can install a single hybrid runtime per cluster. -1. Install on a different cluster, verifying that you meet the minimum requirements. - -**Uninstallation tips for existing runtimes** -* Before you run uninstall an existing hybrid runtime from the Codefresh UI, or run `cf runtime uninstall` from the CLI, _delete_ all Codefresh-related namespaces. -* If a namespace is frozen in the `Terminating` status, it could be because the namespace has resources with finalizers that are preventing deletion. - Here's how you can remove finalizers using `k9s`: - * In the `applications` view, do the following for each application: - * Hit `e` to edit the YAML. - * Scroll down to the section entitled `finalizers`. - * Move cursor to the line with the finalizer definition, and then hit `dd` to delete the line. - * Delete also the `finalizers` key. - * To save and exit, hit `escape` `wq:` `enter`. - * Try deleting the namespace again. - -### What to do next -[Install a hybrid runtime]({{site.baseurl}}/docs/getting-started/quick-start/runtime) diff --git a/_docs/incubation/intro-hosted-runtime.md b/_docs/incubation/intro-hosted-runtime.md deleted file mode 100644 index c2a66b69..00000000 --- a/_docs/incubation/intro-hosted-runtime.md +++ /dev/null @@ -1,146 +0,0 @@ ---- -title: "Hosted GitOps" -description: "" -group: incubation -toc: true ---- - - -Codefresh has enhanced our solution offering with Hosted GitOps, the SaaS version of Codefresh. - -What do you get with Hosted GitOps? -In a nutshell, a hosted and managed version of Argo CD. From application analytics, to application creation, rollout, and deployment, you get the best of both worlds: Argo CD with unique features and functionality from Codefresh to help achieve your CD goals. -What it also means is easy set up and zero maintenance overhead. - -Read on for more details. And check out our [blog](https://codefresh.io/blog/codefresh-upends-continuous-delivery-with-hosted-gitops-platform-featuring-dora-dashboards-and-first-class-integrations-for-ci/). - -### Hosted runtimes - -Setting up your hosted environment takes just a few clicks. All you need is a Codefresh account, a Git account, and a Kubernetes cluster to which to deploy your applications. -Codefresh guides you through the simple three-step process of provisioning your hosted runtime. From that point, Codefresh handles administration and maintenance of the hosted runtime, including version and security updates. - -See [Set up a hosted (Hosted GitOps) environment]({{site.baseurl}}/docs/runtime/hosted-runtime/). - -{% include - image.html - lightbox="true" - file="/images/runtime/intro-hosted-hosted-initial-view.png" - url="/images/runtime/intro-hosted-hosted-initial-view.png" - alt="Hosted runtime setup" - caption="Hosted runtime setup" - max-width="80%" -%} - -### Global deployment analytics - -The Home dashboard presents enterprise-wide deployment highlights, making it a useful management tool. - -Get insights into important KPIs and deployments, across runtimes and clusters, all in the same location. View status of runtimes and managed clusters, deployments, failed deployments with rollbacks, most active applications. Use filters to narrow the scope to focus on anything specific. - -{% include - image.html - lightbox="true" - file="/images/incubation/home-dashboard.png" - url="/images/incubation/home-dashboard.png" - alt="Global deployment analytics" - caption="Global deployment analytics" - max-width="80%" -%} - -### Application analytics and analysis - -The Applications dashboard displays applications across runtimes and clusters, from which you can select and analyze individual applications. Individual application information is grouped by current and historical deployments, enriched with Argo, Jira, and Git details, including rollout visualizations for ongoing deployments (Timeline tab), and an interactive tree view of application resources (Current State tab). - -See [Monitoring applications]({{site.baseurl}}/docs/deployment/applications-dashboard/). - -{% include - image.html - lightbox="true" - file="/images/applications/app-dashboard-main-view.png" - url="/images/applications/app-dashboard-main-view.png" - alt="Applications dashboard" - caption="Applications dashboard" - max-width="80%" -%} - -### DORA metrics - -DORA metrics has become integral to enterprises wanting to quantify DevOps performance, and Codefresh has out-of-the-box support for it. - -Apart from the metrics themselves, the DORA dashboard in Codefresh has several features such as the Totals bar with key metrics, filters that allow you to pinpoint just which applications or runtimes are contributing to problematic metrics, and the ability to set a different view granularity for each DORA metric. - -See [DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics/). - -{% include - image.html - lightbox="true" - file="/images/incubation/intro-dora-metrics.png" - url="/images/incubation/intro-dora-metrics.png" - alt="DORA metrics" - caption="DORA metrics" - max-width="60%" -%} - -### Application management - -Manage the application lifecycle in the Codefresh UI, from creating, editing, and deleting them. - -Define all application settings in a single location through the intuitive Form mode or directly in YAML, and commit all changes to Git. -For easy access, the configuration settings are available for editing in the Applications dashboard. - -See [Applications]({{site.baseurl}}/docs/deployment/create-application/). - -{% include - image.html - lightbox="true" - file="/images/applications/add-app-general-settings.png" - url="/images/applications/add-app-general-settings.png" - alt="Application creation in Codefresh" - caption="Application creation in Codefresh" - max-width="60%" -%} - -### Third-party CI integrations - -If you have your own tools for CI pipelines and workflows, Hosted GitOps gives you the option to connect them to Codefresh and enrich deployment information with our new report image template. For example, you can add the report image step in your GitHub Actions pipeline and reference the different integrations for Codefresh to retrieve and enrich the image information. - -* Git PRs (Pull Requests), Commits, and Committer information directly from the code repositories -* Jira ticket information for correlation with deployed features -* Docker Hub or Quay for image information - -See [Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/). - -{% include - image.html - lightbox="true" - file="/images/incubation/github-action-int-settings.png" - url="/images/incubation/github-action-int-settings.png" - alt="Image enrichment with GitHub Actions integration" - caption="Image enrichment with GitHub Actions integration" - max-width="60%" -%} - -### Hosted vs. hybrid environments - -The table below highlights the main differences between hosted and hybrid environments. - -{: .table .table-bordered .table-hover} -| Functionality |Feature | Hosted | Hybrid | -| -------------- | --------------|--------------- | --------------- | -| Runtime | Installation | Provisioned by Codefresh | Provisioned by customer | -| | Runtime cluster |Managed by Codefresh | Managed by customer | -| | Number per account | Only one runtime | Multiple runtimes | -| | Upgrade | Managed by Codefresh | Managed by customer | -| | External cluster | Managed by customer | Managed by customer | -| | Uninstall | Managed by customer | Managed by customer | -| Argo CD | | Codefresh cluster | Customer cluster | -| CI Ops | Delivery Pipelines |Not supported | Supported | -| |Workflows | Not supported | Supported | -| |Workflow Templates | Not supported | Supported | -| CD Ops |Applications | Supported | Supported | -| |Image enrichment | Supported | Supported | -| | Rollouts | Supported | Supported | -|Integrations | | Supported | Supported | -|Dashboards |Home Analytics | Hosted runtime and deployments|Runtimes, deployments, Delivery Pipelines | -| |DORA metrics | Supported |Supported | -| |Applications | Supported |Supported | diff --git a/_docs/installation/codefresh-on-prem-upgrade.md b/_docs/installation/codefresh-on-prem-upgrade.md new file mode 100644 index 00000000..335b3075 --- /dev/null +++ b/_docs/installation/codefresh-on-prem-upgrade.md @@ -0,0 +1,575 @@ +--- +title: "Codefresh On-Premises Upgrade" +description: "Use the Kubernetes Codefresh Installer to upgrade your Codefresh On-Premises platform " +group: installation +redirect_from: + - /docs/enterprise/codefresh-on-prem-upgrade/ +toc: true +--- +Upgrade the Codefresh On-premises platform to the latest version: +* Prepare for the upgrade: _Before_ the upgrade, based on the version you are upgrading to, complete the required tasks +* Upgrade On-premises +* Complete post-upgrade configuration: If needed, also based on the version you are upgrading to, complete the required tasks + + +## Upgrade to 1.1.1 +Prepare for the upgrade to v1.1.1 by performing the tasks listed below. + +### Maintain backward compatibility for infrastructure services +If you have Codefresh version 1.0.202 or lower installed, and are upgrading to v1.1.1, to retain the existing images for the services listed below, update the `config.yaml` for `kcfi`. + +* `cf-mongodb` +* `cf-redis` +* `cf-rabbitmq` +* `cf-postgresql` +* `cf-nats` +* `cf-consul` + +> In the `config.yaml`, as in the example below, if needed, replace the `bitnami` prefix with that of your private repo. + +```yaml +... + +global: + ### Codefresh App domain name. appUrl is manadatory parameter + appUrl: onprem.mydomain.com + appProtocol: https + + mongodbImage: bitnami/mongodb:3.6.13-r0 # (default `mongodbImage: bitnami/mongodb:4.2`) + +mongodb: + image: bitnami/mongodb:3.6.13-r0 # (default `image: bitnami/mongodb:4.2`) + podSecurityContext: + enabled: true + runAsUser: 0 + fsGroup: 0 + containerSecurityContext: + enabled: false + +redis: + image: bitnami/redis:3.2.9-r2 # (default `image: bitnami/redis:6.0.16`) + podSecurityContext: + enabled: false + containerSecurityContext: + enabled: false + +postgresql: + imageTag: 9.6.2 # (default `imageTag:13`) + +nats: + imageTag: 0.9.4 # (default `imageTag:2.7`) + +consul: + ImageTag: 1.0.0 # (default `imageTag:1.11`) +... +``` +## Upgrade to 1.2.0 and higher +This major release **deprecates** the following Codefresh managed charts: +* Ingress +* Rabbitmq +* Redis + +See the instructions below for each of the affected charts. + +> Before the upgrade remove any seed jobs left from previous release with: + `kubectl delete job --namespace ${CF_NAMESPACE} -l release=cf ` + +> Before the upgrade remove PDBs for Redis and RabbitMQ left from previous release with: + `kubectl delete pdb cf-rabbitmq --namespace ${CF_NAMESPACE}`
+ `kubectl delete pdb cf-redis --namespace ${CF_NAMESPACE}` + +### Update configuration for Ingress chart +From version **1.2.0 and higher**, we have deprecated support for `Codefresh-managed-ingress`. +Kubernetes community public `ingress-nginx` chart replaces `Codefresh-managed-ingress` chart. For more information on the `ingress-nginx`, see [kubernetes/ingress-nginx](https://github.com/kubernetes/ingress-nginx). + +> Parameter locations have changed as the ingress chart name was changed from `ingress` to `ingress-nginx`: + **NGINX controller** parameters are now defined under `ingress-nginx` + **Ingress object** parameters are now defined under `ingress` + +You must update `config.yaml`, if you are using: +* External ingress controllers, including ALB (Application Load Balancer) +* Codefresh-managed ingress controller with _custom_ values + +#### Update configuration for external ingress controllers + +For external ingress controllers, including ALB (Application Load Balancer), update the relevant sections in `config.yaml` to align with the new name for the ingress chart: + +* Replace `ingress` with `ingress-nginx` + +*v1.1.1 or lower* +```yaml +ingress: #disables creation of both Nginx controller deployment and Ingress objects + enabled: false +``` + +*v1.2.2 or higher* +```yaml +ingress-nginx: #disables creation of Nginx controller deployment + enabled: false + +ingress: #disables creation of Ingress objects (assuming you've manually created ingress resource before) + enabled: false +``` + +* Replace `annotations` that have been deprecated with `ingressClassName` + +*v1.1.1 or lower* +```yaml +ingress: + annotations: + kubernetes.io/ingress.class: my-non-codefresh-nginx +``` + +*v1.2.2 or higher* +```yaml +ingress-nginx: + enabled: false + +ingress: + ingressClassName: my-non-codefresh-nginx +### `kubernetes.io/ingress.class` annotation is deprecated from kubernetes v1.22+. +# annotations: +# kubernetes.io/ingress.class: my-non-codefresh-nginx +``` + +#### Update configuration for Codefresh-managed ingress with custom values + +If you were running `Codefresh-managed ingress` controller with _custom_ values refer to [values.yaml](https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml) from the official repo. If needed, update the `ingress-nginx` section in `config.yaml`. The example below shows the default values (already provided in Codefresh chart) for `ingress-nginx`: + +```yaml +ingress-nginx: + enabled: true + controller: + ## This section refers to the creation of the IngressClass resource + ## IngressClass resources are supported since k8s >= 1.18 and required since k8s >= 1.19 + ingressClassResource: + # -- Is this ingressClass enabled or not + enabled: true + # -- Is this the default ingressClass for the cluster + default: false + # -- Controller-value of the controller that is processing this ingressClass + controllerValue: "k8s.io/ingress-nginx-codefresh" + # -- Name of the ingressClass + name: nginx-codefresh + # -- For backwards compatibility with ingress.class annotation. + # Algorithm is as follows, first ingressClassName is considered, if not present, controller looks for ingress.class annotation + ingressClass: nginx-codefresh + # -- Process IngressClass per name (additionally as per spec.controller). + ingressClassByName: true + # Limit the scope of the controller to a specific namespace + scope: + # -- Enable 'scope' or not + enabled: true + admissionWebhooks: + enabled: false +``` +> New `ingress-nginx` subchart creates a new `cf-ingress-nginx-controller` service (`type: LoadBalancer`) instead of old `cf-ingress-controller` service. So make sure to update DNS record for `global.appUrl` to point to a new external load balancer IP. + You can get external load balancer IP with: + `kubectl get svc cf-ingress-nginx-controller -o jsonpath={.status.loadBalancer.ingress[0].ip` + + +### Update configuration for RabbitMQ chart +From version **1.2.2 and higher**, we have deprecated support for the `Codefresh-managed Rabbitmq` chart. Bitnami public `bitnami/rabbitmq` chart has replaced the `Codefresh-managed rabbitmq`. For more information, see [bitnami/rabbitmq](https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq). + +> Configuration updates are not required if you are running an **external** RabbitMQ service. + +> RabbitMQ chart was replaced so as a consequence values structure might be different for some parameters. + For the complete list of values, see [values.yaml](https://github.com/bitnami/charts/blob/master/bitnami/rabbitmq/values.yaml) + +**`existingPvc` changed to `existingClaim` and defined under `persistence`** + +*v1.1.1 or lower* +```yaml +rabbitmq: + existingPvc: my-rabbitmq-pvc + nodeSelector: + foo: bar + resources: + limits: + cpu: 2000m + memory: 2Gi + requests: + cpu: 500m + memory: 1Gi + tolerations: + - effect: NoSchedule + key:+ operator: Equal + value: +``` + +*v1.2.2 or higher* +```yaml +rabbitmq: + volumePermissions: ## Enable init container that changes the owner and group of the persistent volume from existing claim + enabled: true + persistence: + existingClaim: my-rabbitmq-pvc + nodeSelector: + foo: bar + resources: + limits: + cpu: 2000m + memory: 2Gi + requests: + cpu: 500m + memory: 1Gi + tolerations: + - effect: NoSchedule + key: + operator: Equal + value: +``` + +**`storageClass` and `size` defined under `persistence`** + +*v1.1.1 or lower* +```yaml +rabbitmq: + storageClass: my-storage-class + storageSize: 32Gi +``` + +*v1.2.2 or higher* +```yaml +rabbitmq: + persistence: + storageClass: my-storage-class + size: 32Gi +``` + +### Update configuration for Redis chart +From version **1.2.2 and higher**, we have deprecated support for the `Codefresh-managed Redis` chart. Bitnami public `bitnami/redis` chart has replaced the `Codefresh-managed Redis` chart. For more information, see [bitnami/redis](https://github.com/bitnami/charts/tree/master/bitnami/redis). + +Redis storage contains **CRON and Registry** typed triggers so you must migrate existing data from the old deployment to the new stateful set. +This is done by backing up the existing data before upgrade, and then restoring the backed up data after upgrade. + +> Configuration updates are not required: + * When running an **external** Redis service. + * If CRON and Registy triggers have not been configured. + +#### Verify existing Redis data for CRON and Registry triggers +Check if you have CRON and Registry triggers configured in Redis. + +* Run `codefresh get triggers` + OR + Directly from the K8s cluster where Codefresh is installed. + +```shell +NAMESPACE=codefresh +REDIS_PASSWORD=$(kubectl get secret --namespace $NAMESPACE cf-redis -o jsonpath="{.data.redis-password}" | base64 --decode) + +kubectl exec -it deploy/cf-redis -- env REDIS_PASSWORD=$REDIS_PASSWORD bash +#once inside cf-redis pod +REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli +info keyspace # list db +select 15 # select db 15 +keys * #show keys +``` + +* If there are results, continue with _Back up existing Redis data_. + +#### Back up existing Redis data +Back up the existing data before the upgrade: + +* Connect to the pod, run `redis-cli`, export AOF data from old `cf-redis-*` pod: + +```shell +NAMESPACE=codefresh +REDIS_PASSWORD=$(kubectl get secret --namespace $NAMESPACE cf-redis -o jsonpath="{.data.redis-password}" | base64 --decode) +REDIS_POD=$(kubectl get pods -l app=cf-redis -o custom-columns=:metadata.name --no-headers=true) +kubectl cp $REDIS_POD:/bitnami/redis/data/appendonly.aof appendonly.aof -c cf-redis +``` + +#### Restore backed-up Redis data +Restore the data after the upgrade: + +* Copy `appendonly.aof` to the new `cf-redis-master-0` pod: + + ```shell + kubectl cp appendonly.aof cf-redis-master-0:/data/appendonly.aof + ```` +* Restart `cf-redis-master-0` and `cf-api` pods: + + ```shell + kubectl delete pod cf-redis-master-0 + + kubectl scale deployment cf-cfapi-base --replicas=0 -n codefresh + kubectl scale deployment cf-cfapi-base --replicas=2 -n codefresh + ``` + +> Redis chart was replaced so as a consequence values structure might be different for some parameters. + For the complete list of values, see [values.yaml](https://github.com/bitnami/charts/blob/master/bitnami/redis/values.yaml). + +**`existingPvc` changed to `existingClaim` and defined under `master.persistence`** + +*v1.1.1 or lower* +```yaml +redis: + existingPvc: my-redis-pvc + nodeSelector: + foo: bar + resources: + limits: + cpu: 1000m + memory: 1Gi + requests: + cpu: 500m + memory: 500Mi + tolerations: + - effect: NoSchedule + key: + operator: Equal + value: +``` + +*v1.2.2 or higher* +```yaml +redis: + volumePermissions: ## Enable init container that changes the owner and group of the persistent volume from existing claim + enabled: true + master: + persistence: + existingClaim: my-redis-pvc + nodeSelector: + foo: bar + resources: + limits: + cpu: 1000m + memory: 1Gi + requests: + cpu: 500m + memory: 500Mi + tolerations: + - effect: NoSchedule + key: + operator: Equal + value: +``` + +**`storageClass` and `size` defined under `master.persistence`** + + +*v1.1.1 or lower* +```yaml +redis: + storageClass: my-storage-class + storageSize: 32Gi +``` + +*v1.2.2 or higher* +```yaml +redis: + master: + persistence: + storageClass: my-storage-class + size: 32Gi +``` + +> If you run the upgrade without redis backup and restore procedure, **Helm Releases Dashboard** page might be empty for a few minutes after the upgrade. + +## Upgrade to 1.3.0 and higher +This major release **deprecates** the following Codefresh managed charts: +* Consul +* Nats + +### Update configuration for Consul +From version **1.3.0 and higher**, we have deprecated the Codefresh-managed `consul` chart, in favor of Bitnami public `bitnami/consul` chart. For more information, see [bitnami/consul](https://github.com/bitnami/charts/tree/master/bitnami/consul). + +Consul storage contains data about **Windows** worker nodes, so if you had any Windows nodes connected to your OnPrem installation, see the following instruction: + +> Use `https:// /admin/nodes` to check for any existing Windows nodes. + +#### Back up existing consul data +_Before starting the upgrade_, back up existing data. + +> Because `cf-consul` is a StatefulSet and has some immutable fields in its spec with both old and new charts having the same names, you cannot perform a direct upgrade. + Direct upgrade will most likely fail with: + `helm.go:84: [debug] cannot patch "cf-consul" with kind StatefulSet: StatefulSet.apps "cf-consul" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy' and 'minReadySeconds' are forbidden` + After backing up existing data, you must delete the old StatefulSet. + + +1. Exec into the consul pod and create a snapshot: +```shell +kubectl exec -it cf-consul-0 -n codefresh -- consul snapshot save backup.snap +``` +1. Copy snapshot locally: +```shell +kubectl cp -n codefresh cf-consul-0:backup.snap backup.snap +``` +1. **Delete the old** `cf-consul` stateful set: + +```shell +kubectl delete statefulset cf-consul -n codefresh +``` + +#### Restore backed up data + +After completing the upgrade to the current version, restore the `consul` data that you backed up. + +1. Copy the snapshot back to the new pod: + +```shell +kubectl cp -n codefresh backup.snap cf-consul-0:/tmp/backup.snap +``` +1. Restore the data: +``` +kubectl exec -it cf-consul-0 -n codefresh -- consul snapshot restore /tmp/backup.snap +``` +> Consul chart was replaced, and values structure might be different for some parameters. + For the complete list of values, see [values.yaml](https://github.com/bitnami/charts/blob/master/bitnami/consul/values.yaml) + + +### Update Nats configuration +From version **1.3.0 and higher**, we have deprecated Codefresh-managed `nats` chart in favor of Bitnami public `bitnami/nats` chart. For more information, see [bitnami/nats](https://github.com/bitnami/charts/tree/master/bitnami/consul). + +> Because `cf-nats` is a StatefulSet and has some immutable fields in its spec, both the old and new charts have the same names, preventing a direct upgrade. + Direct upgrade will most likely fail with: + `helm.go:84: [debug] cannot patch "cf-nats" with kind StatefulSet: StatefulSet.apps "cf-nats" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy' and 'minReadySeconds' are forbidden` + After backing up existing data, you must delete the old StatefulSet. + + +* **Delete the old** `cf-nats` stateful set. + +```shell +kubectl delete statefulset cf-nats -n codefresh +``` + +> Nats chart was replaced, and values structure might be different for some parameters. + For the complete list of values, see [values.yaml](https://github.com/bitnami/charts/blob/master/bitnami/nats/values.yaml). + +### Upgrade to 1.3.1 and higher + +Chart **v1.3.1** fixes duplicated env vars `CLUSTER_PROVIDERS_URI` and `CLUSTER_PROVIDERS_PORT` in `cf-api` deployment. +```yaml +W1010 03:03:55.553842 280 warnings.go:70] spec.template.spec.containers[0].env[94].name: duplicate name "CLUSTER_PROVIDERS_URI" +W1010 03:03:55.553858 280 warnings.go:70] spec.template.spec.containers[0].env[95].name: duplicate name "CLUSTER_PROVIDERS_PORT" +``` + + +> Due to Helm issue [Removal of duplicate array entry removes completely from Kubernetes](https://github.com/helm/helm/issues/10741), you shoud run `kcfi deploy` or `helm upgrade` two times consecutively. + + +With chart **v1.3.1** [insecure registy](https://docs.docker.com/registry/insecure/) property has been moved under `builder` section: + +```yaml +builder: + insecureRegistries: + - "myregistrydomain.com:5000" +``` + +## Upgrade the Codefresh Platform with [kcfi](https://github.com/codefresh-io/kcfi) + +1. Locate the `config.yaml` file you used in the initial installation. +1. Change the release number inside it. + ```yaml + metadata: + kind: codefresh + installer: + type: helm + helm: + chart: codefresh + repoUrl: https://chartmuseum.codefresh.io/codefresh + version: 1.2.14 + ``` +1. Perform a dry run and verify that there are no errors: + `kcfi deploy --dry-run --debug -c codefresh/config.yaml` +1. Run the actual upgrade: + `kcfi deploy --debug -c codefresh/config.yaml` +1. Verify that all the pods are are in running state: + `kubectl -n codefresh get pods --watch` +1. Log in to the Codefresh UI, and check the new version. +1. If needed, enable/disable new feature flags. + +## Codefresh with Private Registry + +If you install/upgrade Codefresh on the air-gapped environment (without access to public registries or Codefresh Enterprise registry) you will have to copy the images to your organization container registry. + +**Obtain [image list](https://github.com/codefresh-io/onprem-images/tree/master/releases) for specific release** + +**Push images to private docker registry** + +There are 3 types of images: + +> localhost:5000 is your
Note: Maintenance support for Kubernetes v1.19 ended on Oct 28, 2021.{:/}| +|Operating systems|{::nomarkdown}- Windows 10/7
- Linux
- OSX {:/}| +|Node requirements| {::nomarkdown}
- Memory: 5000 MB
- CPU: 2
- GitHub: SaaS and on-premises versions
- Bitbucket: SaaS and Bitbucket server (on-premises) 5.4.0 version and above
- GitLab: SaaS and on-premise versions (API v4 only)
- Single node: 8 CPU core and 16GB RAM
- Multi node: master(s) + 3 nodes with 4 CPU core and 8GB RAM (24 GB in total)
- GitHub
- Bitbucket Cloud
- Ambassador {:/}(see [Ambassador ingress configuration](#ambassador-ingress-configuration)){::nomarkdown}
- AWS ALB (Application Load Balancer) {:/} (see [AWS ALB ingress configuration](#aws-alb-ingress-configuration)){::nomarkdown}
- Istio {:/} (see [Istio ingress configuration](#istio-ingress-configuration)){::nomarkdown}
- NGINX Enterprise (nginx.org/ingress-controller) {:/} (see [NGINX Enterprise ingress configuration](#nginx-enterprise-ingress-configuration)){::nomarkdown}
- NGINX Community (k8s.io/ingress-nginx) {:/} (see [NGINX Community ingress configuration](#nginx-community-version-ingress-configuration)){::nomarkdown}
- Trafik {:/}(see [Traefik ingress configuration](#traefik-ingress-configuration))| +|Node requirements| {::nomarkdown}
- Memory: 5000 MB
- CPU: 2
- GitHub
- GitHub Enterprise
- GitLab Cloud
- GitLab Server
- Bitbucket Cloud
- Bitbucket Server
- Valid expiration date
- Scopes:
- GitHub and GitHub Enterprise: repo, admin-repo.hook
- GitLab Cloud and GitLab Server: api, read_repository
- Bitbucket Cloud and Server: Permissions: Read, Workspace membership: Read, Webhooks: Read and write, Repositories: Write, Admin
--api-url ` + `cf integration git register default --runtime --token ` + +{::nomarkdown} + +{:/} + +### Istio ingress configuration +For detailed configuration information, see [Istio ingress controller documentation](https://istio.io/latest/docs/tasks/traffic-management/ingress/kubernetes-ingress){:target="\_blank}. + +The table below lists the specific configuration requirements for Codefresh. + +{: .table .table-bordered .table-hover} +| What to configure | When to configure | +| -------------- | -------------- | +|Valid external IP address |_Before_ installing hybrid runtime | +|Valid TLS certificate| | +|TCP support | | +|Cluster routing service | _After_ installing hybrid runtime | + +{::nomarkdown} + +{:/} + +#### Valid external IP address +Run `kubectl get svc -A` to get a list of services and verify that the `EXTERNAL-IP` column for your ingress controller shows a valid hostname. + +{::nomarkdown} + +{:/} + +#### Valid TLS certificate +For secure runtime installation, the ingress controller must have a valid TLS certificate. +> Use the FQDN (Fully Qualified Domain Name) of the ingress controller for the TLS certificate. + +{::nomarkdown} + +{:/} + +#### TCP support +Configure the ingress controller to handle TCP requests. + +{::nomarkdown} + +{:/} + + + +#### Cluster routing service +> The cluster routing service must be configured _after_ installing the hybrid runtime. + +Based on the runtime version, you need to configure a single or multiple `VirtualService` resources for the `app-proxy`, `webhook`, and `workflow` services. + +##### Runtime version 0.0.543 or higher +Configure a single `VirtualService` resource to route traffic to the `app-proxy`, `webhook`, and `workflow` services, as in the example below. + +```yaml +apiVersion: networking.istio.io/v1alpha3 +kind: VirtualService +metadata: + namespace: pov-codefresh-istio-runtime # replace with your runtime name + name: internal-router +spec: + hosts: + - pov-codefresh-istio-runtime.sales-dev.codefresh.io # replace with your host name + gateways: + - istio-system/internal-router # replace with your gateway name + http: + - match: + - uri: + prefix: /webhooks + route: + - destination: + host: internal-router + port: + number: 80 + - match: + - uri: + prefix: /app-proxy + route: + - destination: + host: internal-router + port: + number: 80 + - match: + - uri: + prefix: /workflows + route: + - destination: + host: internal-router + port: + number: 80 +``` + +##### Runtime version 0.0.542 or lower + +Configure two different `VirtualService` resources, one to route traffic to the `app-proxy`, and the second to route traffic to the `webhook` services, as in the examples below. + +{::nomarkdown} + +{:/} + +**`VirtualService` example for `app-proxy`:** + +```yaml +apiVersion: networking.istio.io/v1alpha3 +kind: VirtualService +metadata: + namespace: test-runtime3 # replace with your runtime name + name: cap-app-proxy +spec: + hosts: + - my.support.cf-cd.com # replace with your host name + gateways: + - my-gateway # replace with your host name + http: + - match: + - uri: + prefix: /app-proxy + route: + - destination: + host: cap-app-proxy + port: + number: 3017 +``` + +**`VirtualService` example for `webhook`:** + +> Configure a `uri.prefix` and `destination.host` for each event-source if you have more than one. + +```yaml +apiVersion: networking.istio.io/v1alpha3 +kind: VirtualService +metadata: + namespace: test-runtime3 # replace with your runtime name + name: csdp-default-git-source +spec: + hosts: + - my.support.cf-cd.com # replace with your host name + gateways: + - my-gateway # replace with your gateway name + http: + - match: + - uri: + prefix: /webhooks/test-runtime3/push-github # replace `test-runtime3` with your runtime name, and `push-github` with the name of your event source + route: + - destination: + host: push-github-eventsource-svc # replace `push-github' with the name of your event source + port: + number: 80 + - match: + - uri: + prefix: /webhooks/test-runtime3/cypress-docker-images-push # replace `test-runtime3` with your runtime name, and `cypress-docker-images-push` with the name of your event source + route: + - destination: + host: cypress-docker-images-push-eventsource-svc # replace `cypress-docker-images-push` with the name of your event source + port: + number: 80 +``` + +{::nomarkdown} + +{:/} + +### NGINX Enterprise ingress configuration + +For detailed configuration information, see [NGINX ingress controller documentation](https://docs.nginx.com/nginx-ingress-controller){:target="\_blank}. + +The table below lists the specific configuration requirements for Codefresh. + +{: .table .table-bordered .table-hover} +| What to configure | When to configure | +| -------------- | -------------- | +|Verify valid external IP address |_Before_ installing hybrid runtime | +|Valid TLS certificate | | +|TCP support| | +|NGINX Ingress: Enable report status to cluster | | +|NGINX Ingress Operator: Enable report status to cluster| | +|Patch certificate secret |_After_ installing hybrid runtime + +{::nomarkdown} + +{:/} + +#### Valid external IP address +Run `kubectl get svc -A` to get a list of services and verify that the `EXTERNAL-IP` column for your ingress controller shows a valid hostname. + +{::nomarkdown} + +{:/} + +#### Valid TLS certificate +For secure runtime installation, the ingress controller must have a valid TLS certificate. +> Use the FQDN (Fully Qualified Domain Name) of the ingress controller for the TLS certificate. + +{::nomarkdown} + +{:/} + +#### TCP support +Configure the ingress controller to handle TCP requests. + +{::nomarkdown} + +{:/} + +#### NGINX Ingress: Enable report status to cluster + +If the ingress controller is not configured to report its status to the cluster, Argo’s health check reports the health status as “progressing” resulting in a timeout error during installation. + +* Pass `--report-ingress-status` to `deployment`. + +```yaml +spec: + containers: + - args: + - --report-ingress-status +``` + +{::nomarkdown} + +{:/} + +#### NGINX Ingress Operator: Enable report status to cluster + +If the ingress controller is not configured to report its status to the cluster, Argo’s health check reports the health status as “progressing” resulting in a timeout error during installation. + +1. Add this to the `Nginxingresscontrollers` resource file: + + ```yaml + ... + spec: + reportIngressStatus: + enable: true + ... + ``` + +1. Make sure you have a certificate secret in the same namespace as the runtime. Copy an existing secret if you don't have one. +You will need to add this to the `ingress-master` when you have completed runtime installation. + +{::nomarkdown} + +{:/} + +#### Patch certificate secret +> The certificate secret must be configured _after_ installing the hybrid runtime. + +Patch the certificate secret in `spec.tls` of the `ingress-master` resource. +The secret must be in the same namespace as the runtime. + +1. Go to the runtime namespace with the NGINX ingress controller. +1. In `ingress-master`, add to `spec.tls`: + + ```yaml + tls: + - hosts: + - + secretName: + ``` + +{::nomarkdown} + +{:/} + +### NGINX Community version ingress configuration + +Codefresh has been tested with and supports implementations of the major providers. For your convenience, we have provided configuration instructions, both for supported and untested providers in [Provider-specific configuration](#provider-specific-configuration). + + +This section lists the specific configuration requirements for Codefresh to be completed _before_ installing the hybrid runtime. +* Verify valid external IP address +* Valid TLS certificate +* TCP support + +{::nomarkdown} + +{:/} + +#### Valid external IP address +Run `kubectl get svc -A` to get a list of services, and verify that the `EXTERNAL-IP` column for your ingress controller shows a valid hostname. + +{::nomarkdown} + +{:/} + +#### Valid TLS certificate +For secure runtime installation, the ingress controller must have a valid TLS certificate. +> Use the FQDN (Fully Qualified Domain Name) of the ingress controller for the TLS certificate. + +{::nomarkdown} + +{:/} + +#### TCP support +Configure the ingress controller to handle TCP requests. + +Here's an example of TCP configuration for NGINX Community on AWS. +Verify that the `ingress-nginx-controller` service manifest has either of the following annotations: + +`service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"` +OR +`service.beta.kubernetes.io/aws-load-balancer-type: nlb` + +{::nomarkdown} + +{:/} + +#### Provider-specific configuration + +> The instructions are valid for `k8s.io/ingress-nginx`, the community version of NGINX. + + ++AWS
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/aws/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ +Azure (AKS)
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ +Bare Metal Clusters
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ +Digital Ocean
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/do/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ +Docker Desktop
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
+Note: By default, Docker Desktop services will provision with localhost as their external address. Triggers in delivery pipelines cannot reach this instance unless they originate from the same machine where Docker Desktop is being used. + +++ + +Exoscale
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/exoscale/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ + +Google (GKE)
+
+Add firewall rules +
+GKE by default limits outbound requests from nodes. For the runtime to communicate with the control-plane in Codefresh, add a firewall-specific rule. + +-
+
- Find your cluster's network:
+ gcloud container clusters describe [CLUSTER_NAME] --format=get"(network)" +
+ - Get the Cluster IPV4 CIDR:
+ gcloud container clusters describe [CLUSTER_NAME] --format=get"(clusterIpv4Cidr)" +
+ - Replace the `[CLUSTER_NAME]`, `[NETWORK]`, and `[CLUSTER_IPV4_CIDR]`, with the relevant values:
+ gcloud compute firewall-rules create "[CLUSTER_NAME]-to-all-vms-on-network"
+ + --network="[NETWORK]" \ +
+ + --source-ranges="[CLUSTER_IPV4_CIDR]" \ +
+ + --allow=tcp,udp,icmp,esp,ah,sctp +
+
+
+Use ingress-nginx
+-
+
- Create a `cluster-admin` role binding:
+ + kubectl create clusterrolebinding cluster-admin-binding \ +
+ + --clusterrole cluster-admin \ +
+ + --user $(gcloud config get-value account) +
+
+ - Apply:
+ + kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml + +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
+
++ + +MicroK8s
+-
+
- Install using Microk8s addon system:
+ microk8s enable ingress +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ + + +MiniKube
+-
+
- Install using MiniKube addon system:
+ minikube addons enable ingress +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ +Oracle Cloud Infrastructure
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ +{::nomarkdown} + +{:/} + +### Traefik ingress configuration +For detailed configuration information, see [Traefik ingress controller documentation](https://doc.traefik.io/traefik/providers/kubernetes-ingress){:target="\_blank}. + +The table below lists the specific configuration requirements for Codefresh. + +{: .table .table-bordered .table-hover} + +| What to configure | When to configure | +| -------------- | -------------- | +|Valid external IP address | _Before_ installing hybrid runtime | +|Valid SSL certificate | | +|TCP support | | +|Enable report status to cluster| | + +{::nomarkdown} + +{:/} + +#### Valid external IP address +Run `kubectl get svc -A` to get a list of services and verify that the `EXTERNAL-IP` column for your ingress controller shows a valid hostname. + +{::nomarkdown} + +{:/} + +#### Valid TLS certificate +For secure runtime installation, the ingress controller must have a valid TLS certificate. +> Use the FQDN (Fully Qualified Domain Name) of the ingress controller for the TLS certificate. + +{::nomarkdown} + +{:/} + +#### TCP support +Configure the ingress controller to handle TCP requests. + +{::nomarkdown} + +{:/} + +#### Enable report status to cluster +By default, the Traefik ingress controller is not configured to report its status to the cluster. If not configured, Argo’s health check reports the health status as “progressing”, resulting in a timeout error during installation. + +To enable reporting its status, add `publishedService` to `providers.kubernetesIngress.ingressEndpoint`. + +The value must be in the format `"Scaleway
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/scw/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
/ "`, where: + ` ` is the Traefik service from which to copy the status + +```yaml +... +providers: + kubernetesIngress: + ingressEndpoint: + publishedService: " / " # Example, "codefresh/traefik-default" +... +``` + +{::nomarkdown} + +{:/} + +## GitOps CLI installation + +### GitOps CLI installation modes +The table lists the modes available to install the Codefresh CLI. + +{: .table .table-bordered .table-hover} +| Install mode | OS | Commands | +| -------------- | ----------| ----------| +| `curl` | MacOS-x64 | `curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-darwin-amd64.tar.gz | tar zx && mv ./cf-darwin-amd64 /usr/local/bin/cf && cf version`| +| | MacOS-m1 |`curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-darwin-arm64.tar.gz | tar zx && mv ./cf-darwin-arm64 /usr/local/bin/cf && cf version` | +| | Linux - X64 |`curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-linux-amd64.tar.gz | tar zx && mv ./cf-linux-amd64 /usr/local/bin/cf && cf version` | +| | Linux - ARM | `curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-linux-arm64.tar.gz | tar zx && mv ./cf-linux-arm64 /usr/local/bin/cf && cf version`| +| `brew` | N/A| `brew tap codefresh-io/cli && brew install cf2`| + +### Install the GitOps CLI +Install the Codefresh CLI using the option that best suits you: `curl`, `brew`, or standard download. +If you are not sure which OS to select for `curl`, simply select one, and Codefresh automatically identifies and selects the right OS for CLI installation. + +1. Do one of the following: + * For first-time installation, go to the Welcome page, select **+ Install Runtime**. + * If you have provisioned a GitOps Runtime, in the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}, and select **+ Add Runtime**. +1. Install the Codefresh CLI: + * Select one of the installation modes. + * Generate the API key. + * Create the authentication context: + `cf config create-context codefresh --api-key ` + + + {% include + image.html + lightbox="true" + file="/images/getting-started/quick-start/quick-start-download-cli.png" + url="/images/getting-started/quick-start/quick-start-download-cli.png" + alt="Download CLI to install runtime" + caption="Download CLI to install runtime" + max-width="30%" + %} + + +{::nomarkdown} + +{:/} + +## Install Hybrid GitOps Runtime + +**Before you begin** +* Make sure you meet the [minimum requirements]({{site.baseurl}}/docs/runtime/requirements/#minimum-requirements) for installation +* Make sure you have [Runtime token with the required scopes from your Git provider]({{site.baseurl}}/docs/reference/git-tokens) +* [Download or upgrade to the latest version of the CLI]({{site.baseurl}}/docs/installation/hybrid-gitops/#hybrid-gitops-upgrade-gitops-cli) +* Review [Hybrid Runtime installation flags](#hybrid-runtime-installation-flags) +* For ingress-based runtimes, make sure your ingress controller is configured correctly: + * [Ambasador ingress configuration]({{site.baseurl}}/docs/runtime/requirements/#ambassador-ingress-configuration) + * [AWS ALB ingress configuration]({{site.baseurl}}/docs/runtime/requirements/#alb-aws-ingress-configuration) + * [Istio ingress configuration]({{site.baseurl}}/docs/runtime/requirements/#istio-ingress-configuration) + * [NGINX Enterprise ingress configuration]({{site.baseurl}}/docs/runtime/requirements/#nginx-enterprise-ingress-configuration) + * [NGINX Community ingress configuration]({{site.baseurl}}/docs/runtime/requirements/#nginx-community-version-ingress-configuration) + * [Traefik ingress configuration]({{site.baseurl}}/docs/runtime/requirements/#traefik-ingress-configuration) + + +{::nomarkdown} + +{:/} + +**How to** + +1. Do one of the following: + * If this is your first Hybrid Runtime installation, in the Welcome page, select **+ Install Runtime**. + * If you have provisioned a Hybrid Runtime, to provision additional runtimes, in the Codefresh UI, go to [**Runtimes**](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. Click **+ Add Runtimes**, and then select **Hybrid Runtimes**. +1. Do one of the following: + * CLI wizard: Run `cf runtime install`, and follow the prompts to enter the required values. + * Silent install: Pass the required flags in the install command: + `cf runtime install --repo --git-token --silent` + For the list of flags, see [Hybrid runtime installation flags](#hybrid-runtime-installation-flags). +1. If relevant, complete the configuration for these ingress controllers: + * [ALB AWS: Alias DNS record in route53 to load balancer]({{site.baseurl}}/docs/runtime/requirements/#alias-dns-record-in-route53-to-load-balancer) + * [Istio: Configure cluster routing service]({{site.baseurl}}/docs/runtime/requirements/#cluster-routing-service) + * [NGINX Enterprise ingress controller: Patch certificate secret]({{site.baseurl}}/docs/runtime/requirements/#patch-certificate-secret) +1. If you bypassed installing ingress resources with the `--skip-ingress` flag for ingress controllers not in the supported list, create and register Git integrations using these commands: + `cf integration git add default --runtime --api-url ` + `cf integration git register default --runtime --token ` + + +{::nomarkdown} + +{:/} + + + +## Hybrid GitOps Runtime installation flags +This section describes the required and optional flags to install a Hybrid GitOps Runtime. +For documentation purposes, the flags are grouped into: +* Runtime flags, relating to Runtime, cluster, and namespace requirements +* Ingress-less flags, for tunnel-based installation +* Ingress-controller flags, for ingress-based installation +* Git provider flags +* Codefresh resource flags + +{::nomarkdown} + +{:/} + +### Runtime flags + +**Runtime name** +Required. +The Runtime name must start with a lower-case character, and can include up to 62 lower-case characters and numbers. +* CLI wizard: Add when prompted. +* Silent install: Add the `--runtime` flag and define the name. + +**Namespace resource labels** +Optional. +The label of the namespace resource to which you are installing the Hybrid Runtime. Labels are required to identify the networks that need access during installation, as is the case when using services meshes, such as Istio for example. + +* CLI wizard and Silent install: Add the `--namespace-labels` flag, and define the labels in `key=value` format. Separate multiple labels with `commas`. + +**Kube context** +Required. +The cluster defined as the default for `kubectl`. If you have more than one Kube context, the current context is selected by default. + +* CLI wizard: Select the Kube context from the list displayed. +* Silent install: Explicitly specify the Kube context with the `--context` flag. + +**Access mode** +The access mode for the runtime, which can be one of the following: +* [Tunnel-based]({{site.baseurl}}/docs/installation/runtime-architecture/#tunnel-based-hybrid-gitops-runtime-architecture), for runtimes without ingress controllers. This is the default. +* [Ingress-based]({{site.baseurl}}/docs/getting-started/architecture/#ingress-based-hybrid-gitops-runtime-architecture) for runtimes with ingress contollers. + + +* CLI wizard: Select the access mode from the list displayed. +* Silent install: + * For tunnel-based, see [Tunnel-based runtime flags](#tunnel-based-runtime-flags) + * For ingress-based, add the [Ingress controller flags](#ingress-controller-flags) + + >If you don't specify any flags, tunnel-based access is automatically selected. + +**Shared configuration repository** +The Git repository per Runtime account with shared configuration manifests. +* CLI wizard and Silent install: Add the `--shared-config-repo` flag and define the path to the shared repo. + +{::nomarkdown} + +{:/} + +### Tunnel-based runtime flags +These flags are required to install tunnel-based Hybrid Runtimes, without an ingress controller. + +**IP allowlist** + +Optional. + +The allowed list of IPs from which to forward requests to the internal customer cluster for ingress-less runtime installations. The allowlist can include IPv4 and IPv6 addresses, with/without subnet and subnet masks. Multiple IPs must be separated by commas. + +When omitted, all incoming requests are authenticated regardless of the IPs from which they originated. + +* CLI wizard and Silent install: Add the `--ips-allow-list` flag, followed by the IP address, or list of comma-separated IPs to define more than one. For example, `--ips-allow-list 77.126.94.70/16,192.168.0.0` + +{::nomarkdown} + +{:/} + +### Ingress controller flags + + +**Skip ingress** +Required, if you are using an unsupported ingress controller. +For unsupported ingress controllers, bypass installing ingress resources with the `--skip-ingress` flag. +In this case, after completing the installation, manually configure the cluster's routing service, and create and register Git integrations. See the last step in [Install the Hybrid GitOps Runtime](#install-hybrid-gitops-runtime). + +**Ingress class** +Required. + +* CLI wizard: Select the ingress class for Runtime installation from the list displayed. +* Silent install: Explicitly specify the ingress class through the `--ingress-class` flag. Otherwise, Runtime installation fails. + +**Ingress host** +Required. +The IP address or host name of the ingress controller component. + +* CLI wizard: Automatically selects and displays the host, either from the cluster or the ingress controller associated with the **Ingress class**. +* Silent install: Add the `--ingress-host` flag. If a value is not provided, takes the host from the ingress controller associated with the **Ingress class**. + > Important: For AWS ALB, the ingress host is created post-installation. However, when prompted, add the domain name you will create in `Route 53` as the ingress host. + +**Insecure ingress hosts** +TLS certificates for the ingress host: +If the ingress host does not have a valid TLS certificate, you can continue with the installation in insecure mode, which disables certificate validation. + +* CLI wizard: Automatically detects and prompts you to confirm continuing the installation in insecure mode. +* Silent install: To continue with the installation in insecure mode, add the `--insecure-ingress-host` flag. + +**Internal ingress host** +Optional. +Enforce separation between internal (app-proxy) and external (webhook) communication by adding an internal ingress host for the app-proxy service in the internal network. +For both CLI wizard and Silent install: + +* For new Runtime installations, add the `--internal-ingress-host` flag pointing to the ingress host for `app-proxy`. +* For existing installations, commit changes to the installation repository by modifying the `app-proxy ingress` and ` .yaml` + See [(Optional) Internal ingress host configuration for existing Hybrid Runtimes](#optional-internal-ingress-host-configuration-for-existing-hybrid-runtimes). + +{::nomarkdown} + +{:/} + + + +### Git provider and repo flags +The Git provider defined for the Runtime. + +>Because Codefresh creates a [shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration) for the Runtimes in your account, the Git provider defined for the first Runtime you install in your account is used for all the other Runtimes in the same account. + +You can define any of the following Git providers: +* GitHub: + * [GitHub](#github) (the default Git provider) + * [GitHub Enterprise](#github-enterprise) +* GitLab: + * [GitLab Cloud](#gitlab-cloud) + * [GitLab Server](#gitlab-server) +* Bitbucket: + * [Bitbucket Cloud](#bitbucket-cloud) + * [Bitbucket Server](#bitbucket-server) + +{::nomarkdown} + +{:/} + + + +#### GitHub +GitHub is the default Git provider for Hybrid Runtimes. Being the default provider, for both the CLI wizard and Silent install, you need to provide only the repository URL and the Git runtime token. + +> For the required scopes, see [GitHub and GitHub Enterprise Runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#github-and-github-enterprise-runtime-token-scopes). + +`--repo --git-token ` + +where: +* `--repo ` (required), is the `HTTPS` clone URL of the Git repository for the Runtime installation, including the `.git` suffix. Copy the clone URL from your GitHub website (see [Cloning with HTTPS URLs](https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls){:target="\_blank"}). + If the repo doesn't exist, copy an existing clone URL and change the name of the repo. Codefresh creates the repository during the installation. + + Repo URL format: + `https://github.com/ /reponame>.git[/subdirectory][?ref=branch]` + where: + * ` / ` is your username or organization name, followed by the name of the repo, identical to the HTTPS clone URL. For example, `https://github.com/nr-codefresh/codefresh.io.git`. + * `[/subdirectory]` (optional) is the path to a subdirectory within the repo. When omitted, the Runtime is installed in the root of the repository. For example, `/runtimes/defs`. + * `[?ref=branch]` (optional) is the `ref` queryParam to select a specific branch. When omitted, the Runtime is installed in the default branch. For example, `codefresh-prod`. + + Example: + `https://github.com/nr-codefresh/codefresh.io.git/runtimes/defs?ref=codefresh-prod` +* `--git-token ` (required), is the Git token authenticating access to the Runtime installation repository (see [GitHub runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#github-and-github-enterprise-runtime-token-scopes)). + +{::nomarkdown} + +{:/} + +#### GitHub Enterprise + +> For the required scopes, see [GitHub and GitHub Enterprise runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#github-and-github-enterprise-runtime-token-scopes). + + +`--provider github --repo --git-token ` + +where: +* `--provider github` (required), defines GitHub Enterprise as the Git provider for the Runtime and the account. +* `--repo ` (required), is the `HTTPS` clone URL of the Git repository for the Runtime installation, including the `.git` suffix. Copy the clone URL for HTTPS from your GitHub Enterprise website (see [Cloning with HTTPS URLs](https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls){:target="\_blank"}). + If the repo doesn't exist, copy an existing clone URL and change the name of the repo. Codefresh creates the repository during the installation. + Repo URL format: + + `https://ghe-trial.devops.cf-cd.com/ / .git[/subdirectory][?ref=branch]` + where: + * ` / ` is your username or organization name, followed by the name of the repo. For example, `codefresh-io/codefresh.io.git`. + * `[/subdirectory]` (optional) is the path to a subdirectory within the repo. When omitted, the Runtime is installed in the root of the repository. For example, `/runtimes/defs`. + * `[?ref=branch]` (optional) is the `ref` queryParam to select a specific branch. When omitted, the Runtime is installed in the default branch. For example, `codefresh-prod`. + + Example: + `https://ghe-trial.devops.cf-cd.com/codefresh-io/codefresh.io.git/runtimes/defs?ref=codefresh-prod` +* `--git-token ` (required), is the Git token authenticating access to the Runtime installation repository (see [GitHub runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#github-and-github-enterprise-runtime-token-scopes)). + + +{::nomarkdown} + +{:/} + +#### GitLab Cloud +> For the required scopes, see [GitLab Cloud and GitLab Server runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#gitlab-cloud-and-gitlab-server-runtime-token-scopes). + + +`--provider gitlab --repo --git-token ` + +where: +* `--provider gitlab` (required), defines GitLab Cloud as the Git provider for the Runtime and the account. +* `--repo ` (required), is the `HTTPS` clone URL of the Git project for the Runtime installation, including the `.git` suffix. Copy the clone URL for HTTPS from your GitLab website. + If the repo doesn't exist, copy an existing clone URL and change the name of the repo. Codefresh creates the repository during the installation. + + > Important: You must create the group with access to the project prior to the installation. + + Repo URL format: + + `https://gitlab.com/ / .git[/subdirectory][?ref=branch]` + where: + * ` ` is either your username, or if your project is within a group, the front-slash separated path to the project. For example, `nr-codefresh` (owner), or `parent-group/child-group` (group hierarchy) + * ` ` is the name of the project. For example, `codefresh`. + * `[/subdirectory]` (optional) is the path to a subdirectory within the repo. When omitted, the Runtime is installed in the root of the repository. For example, `/runtimes/defs`. + * `[?ref=branch]` (optional) is the `ref` queryParam to select a specific branch. When omitted, the Runtime is installed in the default branch. For example, `codefresh-prod`. + + Examples: + `https://gitlab.com/nr-codefresh/codefresh.git/runtimes/defs?ref=codefresh-prod` (owner) + + `https://gitlab.com/parent-group/child-group/codefresh.git/runtimes/defs?ref=codefresh-prod` (group hierarchy) + +* `--git-token ` (required), is the Git token authenticating access to the Runtime installation repository (see [GitLab runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#gitlab-cloud-and-gitlab-server-runtime-token-scopes)). + + +{::nomarkdown} + +{:/} + + +#### GitLab Server + +> For the required scopes, see [GitLab Cloud and GitLab Server runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#gitlab-cloud-and-gitlab-server-runtime-token-scopes). + +`--provider gitlab --repo --git-token ` + +where: +* `--provider gitlab` (required), defines GitLab Server as the Git provider for the Runtime and the account. +* `--repo ` (required), is the `HTTPS` clone URL of the Git repository for the Runtime installation, including the `.git` suffix. + If the project doesn't exist, copy an existing clone URL and change the name of the project. Codefresh creates the project during the installation. + + > Important: You must create the group with access to the project prior to the installation. + + Repo URL format: + `https://gitlab-onprem.devops.cf-cd.com/ / .git[/subdirectory][?ref=branch]` + where: + * ` ` is your username, or if the project is within a group or groups, the name of the group. For example, `nr-codefresh` (owner), or `parent-group/child-group` (group hierarchy) + * ` ` is the name of the project. For example, `codefresh`. + * `[/subdirectory]` (optional) is the path to a subdirectory within the repo. When omitted, the Runtime is installed in the root of the repository. For example, `/runtimes/defs`. + * `[?ref=branch]` (optional) is the `ref` queryParam to select a specific branch. When omitted, the Runtime is installed in the default branch. For example, `codefresh-prod`. + + Examples: + `https://gitlab-onprem.devops.cf-cd.com/nr-codefresh/codefresh.git/runtimes/defs?ref=codefresh-prod` (owner) + + `https://gitlab-onprem.devops.cf-cd.com/parent-group/child-group/codefresh.git/runtimes/defs?ref=codefresh-prod` (group hierarchy) + +* `--git-token ` (required), is the Git token authenticating access to the Runtime installation repository (see [GitLab runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#gitlab-cloud-and-gitlab-server-runtime-token-scopes)). + + +{::nomarkdown} + +{:/} + +#### Bitbucket Cloud +> For the required scopes, see [Bitbucket runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#bitbucket-cloud-and-bitbucket-server-runtime-token-scopes). + + +`--provider bitbucket --repo --git-user --git-token ` + +where: +* `--provider gitlab` (required), defines Bitbucket Cloud as the Git provider for the Runtime and the account. +* `--repo ` (required), is the `HTTPS` clone URL of the Git repository for the Runtime installation, including the `.git` suffix. + If the project doesn't exist, copy an existing clone URL and change the name of the project. Codefresh creates the project during Runtime installation. + >Important: Remove the username, including @ from the copied URL. + + Repo URL format: + + `https://bitbucket.org .git[/subdirectory][?ref=branch]` + where: + * ` ` is your workspace ID. For example, `nr-codefresh`. + * ` ` is the name of the repository. For example, `codefresh`. + * `[/subdirectory]` (optional) is the path to a subdirectory within the repo. When omitted, the Runtime is installed in the root of the repository. For example, `/runtimes/defs`. + * `[?ref=branch]` (optional) is the `ref` queryParam to select a specific branch. When omitted, the Runtime is installed in the default branch. For example, `codefresh-prod`. + + Example: + `https://bitbucket.org/nr-codefresh/codefresh.git/runtimes/defs?ref=codefresh-prod` +* `--git-user ` (required), is your username for the Bitbucket Cloud account. +* `--git-token ` (required), is the Git token authenticating access to the runtime installation repository (see [Bitbucket runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#bitbucket-cloud-and-bitbucket-server-runtime-token-scopes)). + + +{::nomarkdown} + +{:/} + +#### Bitbucket Server + +> For the required scopes, see [Bitbucket runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#bitbucket-cloud-and-bitbucket-server-runtime-token-scopes). + + +`--provider bitbucket-server --repo --git-user --git-token ` + +where: +* `--provider gitlab` (required), defines Bitbucket Cloud as the Git provider for the Runtime and the account. +* `--repo ` (required), is the `HTTPS` clone URL of the Git repository for the Runtime installation, including the `.git` suffix. + If the project doesn't exist, copy an existing clone URL and change the name of the project. Codefresh then creates the project during the installation. + >Important: Remove the username, including @ from the copied URL. + + Repo URL format: + + `https://bitbucket-server-8.2.devops.cf-cd.com:7990/scm/ / .git[/subdirectory][?ref=branch]` + where: + * ` ` is your username or organization name. For example, `codefresh-io.`. + * ` ` is the name of the repo. For example, `codefresh`. + * `[/subdirectory]` (optional) is the path to a subdirectory within the repo. When omitted, the Runtime is installed in the root of the repository. For example, `/runtimes/defs`. + * `[?ref=branch]` (optional) is the `ref` queryParam to select a specific branch. When omitted, the Runtime is installed in the default branch. For example, `codefresh-prod`. + + Example: + `https://bitbucket-server-8.2.devops.cf-cd.com:7990/scm/codefresh-io/codefresh.git/runtimes/defs?ref=codefresh-prod` +* `--git-user ` (required), is your username for the Bitbucket Server account. +* `--git-token ` (required), is the Git token authenticating access to the Runtime installation repository (see [Bitbucket runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#bitbucket-cloud-and-bitbucket-server-runtime-token-scopes)). + +{::nomarkdown} + +{:/} + +### Codefresh resource flags +**Codefresh demo resources** +Optional. +Install demo pipelines to use as a starting point to create your own GitOps pipelines. We recommend installing the demo resources as these are used in our quick start tutorials. + +* Silent install: Add the `--demo-resources` flag, and define its value as `true` (default), or `false`. For example, `--demo-resources=true` + +**Insecure flag** +For _on-premises installations_, if the Ingress controller does not have a valid SSL certificate, to continue with the installation, add the `--insecure` flag to the installation command. + +{::nomarkdown} + +{:/} + + + + + + + +## (Optional) Internal ingress host configuration for existing Hybrid Runtimes +If you already have provisioned Hybrid Runtimes, to use an internal ingress host for app-proxy communication and an external ingress host to handle webhooks, change the specs for the `Ingress` and `Runtime` resources in the Runtime installation repository. Use the examples as guidelines. + +` /apps/app-proxy/overlays/ /ingress.yaml`: change `host` + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: codefresh-cap-app-proxy + namespace: codefresh #replace with your runtime name +spec: + ingressClassName: nginx + rules: + - host: my-internal-ingress-host # replace with the internal ingress host for app-proxy + http: + paths: + - backend: + service: + name: cap-app-proxy + port: + number: 3017 + path: /app-proxy/ + pathType: Prefix +``` + +`../ /bootstrap/ .yaml`: add `internalIngressHost` + +```yaml +apiVersion: v1 +data: + base-url: https://g.codefresh.io + runtime: | + apiVersion: codefresh.io/v1alpha1 + kind: Runtime + metadata: + creationTimestamp: null + name: codefresh #replace with your runtime name + namespace: codefresh #replace with your runtime name + spec: + bootstrapSpecifier: github.com/codefresh-io/cli-v2/manifests/argo-cd + cluster: https://7DD8390300DCEFDAF87DC5C587EC388C.gr7.us-east-1.eks.amazonaws.com + components: + - isInternal: false + name: events + type: kustomize + url: github.com/codefresh-io/cli-v2/manifests/argo-events + wait: true + - isInternal: false + name: rollouts + type: kustomize + url: github.com/codefresh-io/cli-v2/manifests/argo-rollouts + wait: false + - isInternal: false + name: workflows + type: kustomize + url: github.com/codefresh-io/cli-v2/manifests/argo-workflows + wait: false + - isInternal: false + name: app-proxy + type: kustomize + url: github.com/codefresh-io/cli-v2/manifests/app-proxy + wait: false + defVersion: 1.0.1 + ingressClassName: nginx + ingressController: k8s.io/ingress-nginx + ingressHost: https://support.cf.com/ + internalIngressHost: https://my-internal-ingress-host # add this line and replace my-internal-ingress-host with your internal ingress host + repo: https://github.com/NimRegev/my-codefresh.git + version: 99.99.99 +``` + + +## Related articles +[Add external clusters to Hybrid and Hosted Runtimes]({{site.baseurl}}/docs/installation/managed-cluster/) +[Monitoring & managing GitOps Runtimes]({{site.baseurl}}/docs/installation/monitor-manage-runtimes/) +[Add Git Sources to runtimes]({{site.baseurl}}/docs/installation/git-sources/) +[Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration) +[Troubleshoot Hybrid Runtime installation]({{site.baseurl}}/installation/troubleshooting/runtime-issues/) diff --git a/_docs/installation/installation-options.md b/_docs/installation/installation-options.md new file mode 100644 index 00000000..89009723 --- /dev/null +++ b/_docs/installation/installation-options.md @@ -0,0 +1,231 @@ +--- +title: "Installation environments" +description: "" +group: installation +toc: true +--- +To be changed and updated for ProjectOne + +The Codefresh platform supports two different installation environments, each with different installation options. + +* CI/CD installation environment + The CI/CD installation environment is optimized for Continuous Integration/Delivery with Codefresh pipelines. CI pipelines created in Codefresh fetch code from your Git repository, packages/compiles the code, and deploys the final artifact to a target environment. + + The CI/CD installation environment supports these installation options: + * Hybrid, where the Codefresh CI/CD UI runs in the Codefresh cloud, and the builds run on customer premises + * SaaS, a full cloud version that is fully managed by Codefresh + * On-premises, where Codefresh CI/CD runs within the customer datacenter/cloud + + On-premises and Hybrid CI/CD options are available to Enterprise customers looking for a "behind-the-firewall" solution. + +* GitOps installation environment + The GitOps installation environment is a full-featured solution for application deployments and releases. Powered by the Argo Project, Codefresh uses Argo CD, Argo Workflows, Argo Events, and Argo Rollouts, extended with unique functionality and features essential for enterprise deployments. + + GitOps installations support Hosted and Hybrid options. + +## Comparison +Both environments can co-exist giving you the best of both worlds. For + +TBD + + +## Codefresh CI/CD installation options + + + + + + + + + +### Codefresh Cloud CI/CD - likely to be removed + +The Codefresh CI/CD Cloud version is the easiest way to start using Codefresh as it is fully managed and runs 100% on the cloud. Codefresh DevOps handles the maintenance and updates. + +You can also create a [free account]({{site.baseurl}}/docs/getting-started/create-a-codefresh-account/) on the SAAS version right away. The account is forever free with some limitations +on number of builds. + +The cloud version runs on multiple clouds: + +{% include image.html + lightbox="true" + file="/images/installation/codefresh-saas.png" + url="/images/installation/codefresh-saas.png" + alt="sso-diagram.png" + max-width="60%" + %} + +Codefresh Cloud is also compliant with [SOC2 - Type2](https://www.aicpa.org/SOC) showing our commitment to security and availability. + +{% include image.html + lightbox="true" + file="/images/installation/soc2-type2-certified.png" + url="/images/installation/soc2-type2-certified.png" + alt="sso-diagram.png" + max-width="40%" + %} + +The Cloud version has multi-account support with most git providers (GitLab, GitHub, Bitbucket) as well as Azure and Google. + + +### Codefresh Hybrid CI/CD + +The Hybrid CI/CD installation option is for organizations who want their source code to live within their premises, or have other security constraints. For more about the theory and implementation, see [CI/CD behind the firewall installation]({{site.baseurl}}/docs/administration/behind-the-firewall/). + +The UI runs on Codefresh infrastructure, while the builds happen in a Kubernetes cluster in the customer's premises. + +{% include image.html + lightbox="true" + file="/images/installation/hybrid-installation.png" + url="/images/installation/hybrid-installation.png" + alt="sso-diagram.png" + max-width="70%" + %} + + +CI/CD Hybrid installation strikes the perfect balance between security, flexibility, and ease of use. Codefresh still does the heavy lifting for maintaining most of the platform parts. The sensitive data (such as source code and internal services) never leave the premises of the customers. + +With Hybrid CI/CD installation, Codefresh can easily connect to internal [secure services]({{site.baseurl}}/docs/reference/behind-the-firewall/#using-secure-services-in-your-pipelines) that have no public presence. +The UI part is still compliant with Soc2. + + +Here are the security implications of CI/CD Hybrid installation: + +{: .table .table-bordered .table-hover} +| Company Asset | Flow/Storage of data | Comments | +| -------------- | ---------------------------- |-------------------------| +| Source code | Stays behind the firewall | | +| Binary artifacts | Stay behind the firewall | | +| Build logs | Also sent to Codefresh Web application | | +| Pipeline volumes | Stay behind the firewall | | +| Pipeline variables | Defined in Codefresh Web application | | +| Deployment docker images | Stay behind the firewall| Stored on your Docker registry | +| Development docker images | Stay behind the firewall | Stored on your Docker registry| +| Testing docker images | Stay behind the firewall| Stored on your Docker registry | +| Inline pipeline definition | Defined in Codefresh Web application | | +| Pipelines as YAML file | Stay behind the firewall | | +| Test results | Stay behind the firewall | | +| HTML Test reports | Shown on Web application | Stored in your S3 or Google bucket or Azure storage | +| Production database data | Stays behind the firewall | | +| Test database data | Stays behind the firewall | | +| Other services (e.g. Queue, ESB) | Stay behind the firewall | | +| Kubernetes deployment specs | Stay behind the firewall | | +| Helm charts | Stay behind the firewall | | +| Other deployment resources/script (e.g. terraform) | Stay behind the firewall | | +| Shared configuration variables | Defined in Codefresh Web application | | +| Deployment secrets (from git/Puppet/Vault etc) | Stay behind the firewall| | +| Audit logs | Managed via Codefresh Web application | | +| SSO/Idp Configuration | Managed via Codefresh Web application | | +| User emails | Managed via Codefresh Web application | | +| Access control rules | Managed via Codefresh Web application | | + + + +### Codefresh On-premises CI/CD + +For customers who want full control, Codefresh also offers an on-premises option for CI/CD installation. Both the UI and builds run on a Kubernetes cluster fully managed by the customer. + +While Codefresh can still help with maintenance of the CI/CD On-premises, we would recommend the Hybrid CI/CD option first as it offers the most flexibility while maintaining high security. + +### CI/CD installation comparison + +{: .table .table-bordered .table-hover} +| Characteristic | Cloud | Hybrid | On Premise | +| -------------- | ---------------------------- |-------------------------| +| Managed by | Codefresh | Codefresh and Customer | Customer | +| UI runs on | public cloud | public cloud | private cluster | +| Builds run on | public cloud | private cluster | private cluster | +| Access to secure/private services | no | yes | yes | +| Customer maintenance effort | none | some | full | +| Best for | most companies | companies with security constraints | Large scale installations | +| Available to | all customers | [enterprise plans](https://codefresh.io/contact-us/) | [enterprise plans](https://codefresh.io/contact-us/) | + + +## Codefresh GitOps installation options + +Similar to CI/CD installation options, Codefresh GitOps also supports SaaS and hybrid installation options: + + +### Hosted GitOps +The SaaS version of GitOps, has Argo CD installed in the Codefresh cluster. +Hosted GitOps Runtime is installed and provisioned in a Codefresh cluster, and managed by Codefresh. +Hosted enviroments are full-cloud environments, where all updates and improvements are managed by Codefresh, with zero-maintenance overhead for you as the customer. Currently, you can add one Hosted GitOps Runtime per account. +For the architecture, see [Hosted GitOps Runtime architecture]({{site.baseurl}}/docs/installation/architecture/#hosted-gitops-runtime-architecture). + + +{% include + image.html + lightbox="true" + file="/images/runtime/intro-hosted-hosted-initial-view.png" + url="/images/runtime/intro-hosted-hosted-initial-view.png" + alt="Hosted runtime setup" + caption="Hosted runtime setup" + max-width="80%" +%} + + For more information on how to set up the hosted environment, including provisioning hosted runtimes, see [Set up Hosted GitOps]({{site.baseurl}}/docs/installation/hosted-runtime/). + +### Hybrid GitOps +The hybrid version of GitOps, has Argo CD installed in the customer's cluster. +Hybrid GitOps is installed in the customer's cluster, and managed by the customer. +The Hybrid GitOps Runtime is optimal for organizations with security constraints, wanting to manage CI/CD operations within their premises. Hybrid GitOps strikes the perfect balance between security, flexibility, and ease of use. Codefresh maintains and manages most aspects of the platform, apart from installing and upgrading Hybrid GitOps Runtimes which are managed by the customer. + + +{% include + image.html + lightbox="true" + file="/images/runtime/runtime-list-view.png" + url="/images/runtime/runtime-list-view.png" + alt="Runtime List View" + caption="Runtime List View" + max-width="70%" +%} + + For more information on hybrid environments, see [Hybrid GitOps runtime requirements]({{site.baseurl}}/docs/installation/hybrid-gitops/#minimum-system-requirements) and [Installling Hybrid GitOps Runtimes]({{site.baseurl}}/docs/installation/hybrid-gitops/). + + + + + +### Hosted vs.Hybrid GitOps + +The table below highlights the main differences between Hosted and Hybrid GitOps. + +{: .table .table-bordered .table-hover} +| GitOps Functionality |Feature | Hosted | Hybrid | +| -------------- | -------------- |--------------- | --------------- | +| Runtime | Installation | Provisioned by Codefresh | Provisioned by customer | +| | Runtime cluster | Managed by Codefresh | Managed by customer | +| | Number per account | One runtime | Multiple runtimes | +| | External cluster | Managed by customer | Managed by customer | +| | Upgrade | Managed by Codefresh | Managed by customer | +| | Uninstall | Managed by customer | Managed by customer | +| Argo CD | | Codefresh cluster | Customer cluster | +| CI Ops | Delivery Pipelines |Not supported | Supported | +| |Workflows | Not supported | Supported | +| |Workflow Templates | Not supported | Supported | +| CD Ops |Applications | Supported | Supported | +| |Image enrichment | Supported | Supported | +| | Rollouts | Supported | Supported | +|Integrations | | Supported | Supported | +|Dashboards |Home Analytics | Hosted runtime and deployments|Runtimes, deployments, Delivery Pipelines | +| |DORA metrics | Supported |Supported | +| |Applications | Supported |Supported | + +### Related articles +[Architecture]({{site.baseurl}}/docs/installation/runtime-architecture/) +[Add Git Sources to GitOps Runtimes]({{site.baseurl}}/docs/installation/git-sources/) +[Shared configuration repository]({{site.baseurl}}/docs/reference/shared-configuration) + diff --git a/_docs/runtime/managed-cluster.md b/_docs/installation/managed-cluster.md similarity index 69% rename from _docs/runtime/managed-cluster.md rename to _docs/installation/managed-cluster.md index 25ae4546..fb010209 100644 --- a/_docs/runtime/managed-cluster.md +++ b/_docs/installation/managed-cluster.md @@ -1,42 +1,42 @@ --- -title: "Add external clusters to runtimes" +title: "Add external clusters to GitOps Runtimes" description: "" -group: runtime +group: installation toc: true --- -Register external clusters to provisioned hybrid or hosted runtimes in Codefresh. Once you add an external cluster, you can deploy applications to that cluster without having to install Argo CD in order to do so. External clusters allow you to manage multiple clusters through a single runtime. +Register external clusters to provisioned Hybrid or Hosted GitOps Runtimes in Codefresh. Once you add an external cluster, you can deploy applications to that cluster without having to install Argo CD in order to do so. Manage manage multiple external clusters through a single Runtime. -When you add an external cluster to a provisioned runtime, the cluster is registered as a managed cluster. A managed cluster is treated as any other managed K8s resource, meaning that you can monitor its health and sync status, deploy applications on the cluster and view information in the Applications dashboard, and remove the cluster from the runtime's managed list. +When you add an external cluster to a provisioned Runtime, the cluster is registered as a managed cluster. A managed cluster is treated as any other managed K8s resource, meaning that you can monitor its health and sync status, deploy applications to it, view information in the Applications dashboard, and remove the cluster from the Runtime's managed list. Add managed clusters through: * Codefresh CLI * Kustomize -Adding a managed cluster via Codefresh ensures that Codefresh applies the required RBAC resources (`ServiceAccount`, `ClusterRole` and `ClusterRoleBinding`) to the target cluster, creates a `Job` that updates the selected runtime with the information, registers the cluster in Argo CD as a managed cluster, and updates the platform with the new cluster information. +Adding a managed cluster via Codefresh ensures that Codefresh applies the required RBAC resources (`ServiceAccount`, `ClusterRole` and `ClusterRoleBinding`) to the target cluster, creates a `Job` that updates the selected Runtime with the information, registers the cluster in Argo CD as a managed cluster, and updates the platform with the new cluster information. -### Add a managed cluster with Codefresh CLI -Add an external cluster to a provisioned runtime through the Codefresh CLI. When adding the cluster, you can also add labels and annotations to the cluster, which are added to the cluster secret created by Argo CD. +## Add a managed cluster with Codefresh CLI +Add an external cluster to a provisioned GitOps Runtime through the Codefresh CLI. When adding the cluster, you can also add labels and annotations to the cluster, which are added to the cluster secret created by Argo CD. Optionally, to first generate the YAML manifests, and then manually apply them, use the `dry-run` flag in the CLI. **Before you begin** - -* For _hosted_ runtimes: [Configure access to these IP addresses]({{site.baseurl}}/docs/administration/platform-ip-addresses/) +* For _Hosted_ Runtimes: [Configure access to these IP addresses]({{site.baseurl}}/docs/administration/platform-ip-addresses/) * Verify that: - * Your Git personal access token is valid and has the correct permissions - * You have installed the latest version of the Codefresh CLI + * Your Git personal access token is valid and has the [required scopes]({{site.baseurl}}/docs/reference/git-tokens) + * You have installed the [latest version of the Codefresh CLI]({{site.baseurl}}/docs/installation/monitor-manage-runtimes/#hybrid-gitops-upgrade-gitops-cli) **How to** -1. In the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. -1. From either the **Topology** or **List** views, select the runtime to which to add the cluster. +1. In the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. From either the **Topology** or **List** views, select the Runtime to which to add the cluster. 1. Topology View: Select {::nomarkdown} 
{:/}.
List View: Select the **Managed Clusters** tab, and then select **+ Add Cluster**.
1. In the Add Managed Cluster panel, copy and run the command:
- `cf cluster add [--labels label-key=label-value] [--annotations annotation-key=annotation-value][--dry-run]` + `cf cluster add [runtime-name] [--labels label-key=label-value] [--annotations annotation-key=annotation-value][--dry-run]` where: + * `runtime-name` is the name of the Runtime to which to add the cluster. * `--labels` is optional, and required to add labels to the cluster. When defined, add a label in the format `label-key=label-value`. Separate multiple labels with `commas`. * `--annotations` is optional, and required to add annotations to the cluster. When defined, add an annotation in the format `annotation-key=annotation-value`. Separate multiple annotations with `commas`. * `--dry-run` is optional, and required if you want to generate a list of YAML manifests that you can redirect and apply manually with `kubectl`. @@ -54,7 +54,7 @@ Optionally, to first generate the YAML manifests, and then manually apply them, {:start="5"} 1. If you used `dry-run`, apply the generated manifests to the same target cluster on which you ran the command. - Here is an example of the YAML manifest generated with the `--dry-run` flag. Note that there are placeholders in the example, which are replaced with the actual values with `--dry-run`. + Here is an example of the YAML manifest generated with the `--dry-run` flag. Note that the example has placeholders, which are replaced with the actual values during the `--dry-run`. ```yaml @@ -177,9 +177,9 @@ spec: ``` -The new cluster is registered to the runtime as a managed cluster. +The new cluster is registered to the Runtime as a managed cluster. -### Add a managed cluster with Kustomize +## Add a managed cluster with Kustomize Create a `kustomization.yaml` file with the information shown in the example below, and run `kustomize build` on it. ```yaml @@ -222,16 +222,20 @@ resources: ``` -### Work with managed clusters -Work with managed clusters in hybrid or hosted runtimes in either the Topology or List runtime views. For information on runtime views, see [Runtime views]({{site.baseurl}}/docs/runtime/runtime-views). -As the cluster is managed through the runtime, updates to the runtime automatically updates the components on all the managed clusters that include it. +## Work with managed clusters +Work with managed clusters in either the Topology or List Runtime views. For information on Runtime views, see [Runtime views]({{site.baseurl}}/docs/runtime/runtime-views). +As the cluster is managed through the Runtime, updates to the Runtime automatically updates the components on all the managed clusters that include it. View connection status for the managed cluster, and health and sync errors. Health and sync errors are flagged by the error notification in the toolbar, and visually flagged in the List and Topology views. -#### Install Argo Rollouts -Install Argo Rollouts directly from Codefresh with a single click to visualize rollout progress in the [Applications dashboard]({{site.baseurl}}/docs/deployment/applications-dashboard/). If Argo Rollouts has not been installed, an **Install Argo Rollouts** button is displayed on selecting the managed cluster. +### Install Argo Rollouts +Applications with `rollout` resources need Argo Rollouts on the target cluster, both to visualize rollouts in the Applications dashboard and control rollout steps with the Rollout Player. +If Argo Rollouts has not been installed on the target cluster, it displays **Install Argo Rollouts** button. + +Install Argo Rollouts with a single click to execute rollout instructions, deploy the application, and visualize rollout progress in the [Applications dashboard]({{site.baseurl}}/docs/deployment/applications-dashboard/). + -1. In the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. In the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. 1. Select **Topology View**. 1. Select the target cluster, and then select **+ Install Argo Rollouts**. @@ -246,16 +250,16 @@ Install Argo Rollouts directly from Codefresh with a single click to visualize r %} -#### Remove a managed cluster from the Codefresh UI -Remove a cluster from the runtime's list of managed clusters from the Codefresh UI. +### Remove a managed cluster from the Codefresh UI +Remove a cluster from the Runtime's list of managed clusters from the Codefresh UI. > You can also remove it through the CLI. -1. In the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. In the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. 1. Select either the **Topology View** or the **List View** tabs. 1. Do one of the following: - * In the Topology View, select the cluster node from the runtime it is registered to. - * In the List View, select the runtime, and then select the **Managed Clusters** tab. + * In the Topology View, select the cluster node from the Runtime it is registered to. + * In the List View, select the Runtime, and then select the **Managed Clusters** tab. 1. Select the three dots next to the cluster name, and then select **Uninstall** (Topology View) or **Remove** (List View). {% include @@ -269,8 +273,8 @@ Remove a cluster from the runtime's list of managed clusters from the Codefresh %} -#### Remove a managed cluster through the Codefresh CLI -Remove a cluster from the list managed by the runtime, through the CLI. +### Remove a managed cluster through the Codefresh CLI +Remove a cluster from the list managed by the Runtime, through the CLI. * Run: `cf cluster remove --server-url ` @@ -279,7 +283,6 @@ Remove a cluster from the list managed by the runtime, through the CLI. ` ` is the URL of the server on which the managed cluster is installed. -### Related articles -[Add Git Sources to runtimes]({{site.baseurl}}/docs/runtime/git-sources/) -[Manage provisioned hybrid runtimes]({{site.baseurl}}/docs/runtime/monitor-manage-runtimes/) -[(Hybrid) Monitor provisioned runtimes]({{site.baseurl}}/docs/runtime/monitoring-troubleshooting/) \ No newline at end of file +## Related articles +[Add Git Sources to GitOps Runtimes]({{site.baseurl}}/docs/installation/git-sources/) +[Monitoring & managing GitOps Runtimes]({{site.baseurl}}/docs/installation/monitor-manage-runtimes/) diff --git a/_docs/installation/monitor-manage-runtimes.md b/_docs/installation/monitor-manage-runtimes.md new file mode 100644 index 00000000..08267a95 --- /dev/null +++ b/_docs/installation/monitor-manage-runtimes.md @@ -0,0 +1,643 @@ +--- +title: "Monitoring & managing GitOps Runtimes" +description: "" +group: runtime +redirect_from: + - /monitor-manage-runtimes/ + - /monitor-manage-runtimes +toc: true +--- + + +The **Runtimes** page displays the provisioned GitOps Runtimes in your account, both Hybrid, and the Hosted Runtime if you have one. + +View Runtime components and information in List or Topology view formats to monitor and manage them. + +{% include + image.html + lightbox="true" + file="/images/runtime/runtime-list-view.png" + url="/images/runtime/runtime-list-view.png" + alt="Runtime List View" + caption="Runtime List View" + max-width="70%" +%} + +Monitor provisioned GitOps Runtimes for security, health, and sync errors: + +* (Hybrid and Hosted) View/download logs for Runtimes and for Runtime components +* (Hybrid) Restore provisioned Runtimes +* (Hybrid) Configure browsers to allow access to insecure Runtimes +* (Hybrid) Monitor notifications in the Activity Log + + +Manage provisioned GitOps Runtimes: +* [Add managed clusters to GitOps Runtimes]({{site.baseurl}}/docs/installation/managed-cluster/) +* [Add and manage Git Sources for GitOps Runtimes]({{site.baseurl}}/docs/installation/git-sources/) +* +* Upgrade GitOps CLI +* Upgrade Hybrid GitOps Runtimes +* Uninstall GitOps Runtimes + + + +> Unless specified otherwise, all options are common to both types of GitOps Runtimes. If an option is valid only for Hybrid GitOps, it is indicated as such. + + +## GitOps Runtime views + +View provisioned GitOps Runtimes in List or Topology view formats. + +* List view: The default view, displays the list of provisioned Runtimes, the clusters managed by them, and Git Sources associated with them. +* Topology view: Displays a hierarchical view of Runtimes and the clusters managed by them, with health and sync status of each cluster. + +### List view + +The List view is a grid-view of the provisioned Runtimes. + +Here is an example of the List view for runtimes. +{% include + image.html + lightbox="true" + file="/images/runtime/runtime-list-view.png" + url="/images/runtime/runtime-list-view.png" + alt="Runtime List View" + caption="Runtime List View" + max-width="70%" +%} + +Here is a description of the information in the List View. + +{: .table .table-bordered .table-hover} +| List View Item| Description | +| -------------- | ---------------- | +|**Name**| The name of the provisioned GitOps Runtime. | +|**Type**| The type of GitOps Runtime provisioned, and can be **Hybrid** or **Hosted**. | +|**Cluster/Namespace**| The K8s API server endpoint, as well as the namespace with the cluster. | +|**Modules**| The modules installed based on the type of provisioned Runtime. Hybrid Runtimes include CI amnd CD Ops modules. Hosted runtimes include CD Ops. | +|**Managed Cluster**| The number of managed clusters if any, for the runtime. To view list of managed clusters, select the runtime, and then the **Managed Clusters** tab. To work with managed clusters, see [Adding external clusters to runtimes]({{site.baseurl}}/docs/runtime/managed-cluster).| +|**Version**| The version of the runtime currently installed. **Update Available!** indicates there are later versions of the runtime. To see all the commits to the runtime, mouse over **Update Available!**, and select **View Complete Change Log**. +|**Last Updated**| The most recent update information from the runtime to the Codefresh platform. Updates are sent to the platform typically every few minutes. Longer update intervals may indicate networking issues.| +|**Sync Status**| The health and sync status of the runtime or cluster. {::nomarkdown} -
indicates health or sync errors in the runtime, or a managed cluster if one was added to the runtime. The runtime name is colored red. 
indicates that the runtime is being synced to the cluster on which it is provisioned.
indicates the local cluster, always displayed as `in-cluster`. The in-cluster server URL is always set to `https://kubernetes.default.svc/`.
indicates a managed cluster.- select to add a new managed cluster.
- indicates health or sync errors in the Runtime, or a managed cluster if one was added to the runtime. The runtime or cluster node is bordered in red and the name is colored red.
- indicates that the Runtime is being synced to the cluster on which it is provisioned.
- Find a Runtime or its clusters by typing part of the Runtime/cluster name, and then navigate to the entries found.
- Topology view options: Resize to window, zoom in, zoom out, full screen view.
--git-token --silent` + where: + ` ` is a valid Git token with the correct scopes. + +**CLI wizard-based upgrade** + +1. In the Codefresh UI, make sure you are in [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. Switch to either the **List View** or to the **Topology View**. +1. **List view**: + * Select the Runtime name. + * To see all the commits to the Runtime, in the Version column, mouse over **Update Available!**, and select **View Complete Change Log**. + * On the top-right, select **Upgrade**. + + {% include + image.html + lightbox="true" + file="/images/runtime/runtime-list-view-upgrade.png" + url="/images/runtime/runtime-list-view-upgrade.png" + alt="List View: Upgrade runtime option" + caption="List View: Upgrade runtime option" + max-width="30%" + %} + + **Topology view**: + Select the Runtime cluster, and from the panel, select the three dots and then select **Upgrade Runtime**. + {% include + image.html + lightbox="true" + file="/images/runtime/runtiime-topology-upgrade.png" + url="/images/runtime/runtiime-topology-upgrade.png" + alt="Topology View: Upgrade runtime option" + caption="Topology View: Upgrade runtime option" + max-width="30%" +%} + +{:start="4"} + +1. If you have already installed the Codefresh CLI, in the Install Upgrades panel, copy the upgrade command. + + {% include + image.html + lightbox="true" + file="/images/runtime/install-upgrades.png" + url="/images/runtime/install-upgrades.png" + alt="Upgrade runtime" + caption="Upgrade runtime panel" + max-width="30%" +%} + +{:start="5"} +1. In your terminal, paste the command, and do the following: + * Update the Git token value. + * To manually define the shared configuration repo, add the `--shared-config-repo` flag with the path to the repo. +1. Confirm to start the upgrade. + + + + + + +### Uninstall provisioned GitOps Runtimes + +Uninstall provisioned GitOps Runtimes that are not in use. Uninstall a Runtime through a silent uninstall or through the CLI wizard. +> Uninstalling a Runtime removes the Git Sources and managed clusters associated with it. + +**Before you begin** +For both types of uninstalls, make sure you have: + +* The latest version of the GitOps CLI +* A valid runtime Git token +* The Kube context from which to uninstall the provisioned Runtime + +**Silent uninstall** +Pass the mandatory flags in the uninstall command: + `cf runtime uninstall --git-token --silent` + where: + `--git-token` is a valid runtime token with the `repo` and `admin-repo.hook` scopes. + +**CLI wizard uninstall** + +1. In the Codefresh UI, make sure you are in [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. Switch to either the **List View** or to the **Topology View**. +1. **List view**: On the top-right, select the three dots and then select **Uninstall**. + + {% include + image.html + lightbox="true" + file="/images/runtime/uninstall-location.png" + url="/images/runtime/uninstall-location.png" + alt="List View: Uninstall runtime option" + caption="List View: Uninstall runtime option" + max-width="30%" +%} + +**Topology view**: Select the Runtime node, and from the panel, select the three dots and then select **Uninstall Runtime**. + {% include + image.html + lightbox="true" + file="/images/runtime/runtime-topology-uninstall.png" + url="/images/runtime/runtime-topology-uninstall.png" + alt="Topology View: Uninstall runtime option" + caption="Topology View: Uninstall runtime option" + max-width="30%" +%} + +{:start="4"} + +1. If you already have the latest version of the Codefresh CLI, in the Uninstall Codefresh Runtime panel, copy the uninstall command. + + {% include + image.html + lightbox="true" + file="/images/runtime/uninstall.png" + url="/images/runtime/uninstall.png" + alt="Uninstall Codefresh runtime" + caption="Uninstall Codefresh runtime" + max-width="40%" +%} + +{:start="5"} + +1. In your terminal, paste the command, and update the Git token value. +1. Select the Kube context from which to uninstall the Runtime, and then confirm the uninstall. +1. If you get errors, run the uninstall command again, with the `--force` flag. + + + +### Update Git tokens for Runtimes + +Provisioned Runtimes require valid Git tokens at all times to authenticate Git actions by you as a user. +>These tokens are specific to the user, and the same can be used for multiple runtimes. + +There are two different situations when you need to update Git tokens: +* Update invalid, revoked, or expired tokens: Codefresh automatically flags Runtimes with such tokens. It is mandatory to update the Git tokens to continue working with the platform. +* Update valid tokens: Optional. You may want to update Git tokens, even valid ones, by deleting the existing token and replacing it with a new token. + +The methods for updating any Git token are the same regardless of the reason for the update: +* OAuth2 authorization, if your admin has registered an OAuth Application for Codefresh +* Git access token authentication, by generating a personal access token in your Git provider account with the correct scopes + +**Before you begin** +* To authenticate through a Git access token, make sure your token is valid and has [the required scopes]({{site.baseurl}}/docs/reference/git-tokens) + +**How to** +1. Do one of the following: + * If you see a notification in the Codefresh UI about invalid Runtime tokens, click **[Update Token]**. + The Runtimes page shows Runtimes with invalid tokens prefixed by the key icon. Mouse over shows invalid token. + * To update an existing token, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. Select the Runtime for which to update the Git token. +1. From the context menu with the additional actions at the top-right, select **Update Git Runtime token**. + + {% include + image.html + lightbox="true" + file="/images/runtime/update-git-runtime-token.png" + url="/images/runtime/update-git-runtime-token.png" + alt="Update Git runtime token option" + caption="Update Git runtime token option" + max-width="40%" +%} + +{:start="4"} +1. Do one of the following: + * If your admin has set up OAuth access, click **Authorize Access to Git Provider**. Go to _step 5_. + * Alternatively, authenticate with an access token from your Git provider. Go to _step 6_. + +{:start="5"} +1. For OAuth2 authorization: + > If the application is not registered, you get an error. Contact your admin for help. + * Enter your credentials, and select **Sign In**. + * If required, as for example if two-factor authentication is configured, complete the verification. + + {% include + image.html + lightbox="true" + file="/images/administration/user-settings/oauth-user-authentication.png" + url="/images/administration/user-settings/oauth-user-authentication.png" + alt="Authorizing access with OAuth2" + caption="Authorizing access with OAuth2" + max-width="30%" + %} + +{:start="6"} +1. For Git token authentication, expand **Advanced authorization options**, and then paste the generated token in the **Git runtime token** field. + +1. Click **Update Token**. + +## Monitoring GitOps Runtimes +* [View/download logs to troubleshoot Runtimes](#viewdownload-logs-to-troubleshoot-runtimes) +* [(Hybrid GitOps) Restoring provisioned Runtimes](#hybrid-gitops-restoring-provisioned-runtimes) +* [(Hybrid GitOps) Configure browser to allow insecure Runtimes](#hybrid-gitops-configure-browser-to-allow-insecure-runtimes) +* [(Hybrid GitOps) View notifications in Activity Log](#hybrid-gitops-view-notifications-in-activity-log) +* [(Hybrid GitOps) Troubleshoot health and sync errors for Runtimes](#hybrid-gitops-troubleshoot-health-and-sync-errors-for-runtimes) + +### View/download logs to troubleshoot Runtimes +Logs are available for completed Runtimes, both for the Runtime and for individual Runtime components. Download log files for offline viewing and analysis, or view online logs for a Runtime component, and download if needed for offline analysis. Online logs support free-text search, search-result navigation, and line-wrap for enhanced readability. + +Log files include events from the date of the application launch, with the newest events listed first. + +{::nomarkdown} +
+{:/} + +#### Download logs for Runtimes +Download the log file for a Runtime. The Runtime log is downloaded as a `.tar.gz` file, which contains the individual log files for each runtime component. + +1. In the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. If needed, switch to **List View**, and then select the runtime for which to download logs. +1. From the context menu, select **Download All Logs**. + The log file is downloaded to the Downloads folder or the folder designated for downloads, with the filename, `.tar.gz`. For example, `codefreshv2-production2.tar.gz`. + + + {% include + image.html + lightbox="true" + file="/images/runtime/runtime-logs-download-all.png" + url="/images/runtime/runtime-logs-download-all.png" + alt="Download logs for selected runtime" + caption="Download logs for selected runtime" + max-width="40%" +%} + + +{:start="4"} +1. To view the log files of the individual components, unzip the file. + Here is an example of the folder with the individual logs. + + {% include + image.html + lightbox="true" + file="/images/runtime/runtime-logs-folder-view.png" + url="/images/runtime/runtime-logs-folder-view.png" + alt="Individual log files in folder" + caption="Individual log files in folder" + max-width="50%" +%} + +{:start="5"} +1. Open a log file with the text editor of your choice. + +{::nomarkdown} +
+{:/} + +#### View/download logs for Runtime components +View online logs for any Runtime component, and if needed, download the log file for offline viewing and analysis. + +Online logs show up to 1000 of the most recent events (lines), updated in real time. Downloaded logs include all the events, from the application launch to the date and time of download. + +1. In the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. If needed, switch to **List View**, and then select the Runtime. +1. Select the Runtime component and then select **View Logs**. + + {% include + image.html + lightbox="true" + file="/images/runtime/runtime-logs-view-component.png" + url="/images/runtime/runtime-logs-view-component.png" + alt="View log option for individual runtime component" + caption="View log option for individual runtime component" + max-width="40%" +%} + + +{:start="4"} +1. Do the following: + * Search by free-text for any string, and click the next and previous buttons to navigate between the search results. + * To switch on line-wrap for readability, click **Wrap**. + + {% include + image.html + lightbox="true" + file="/images/runtime/runtime-logs-screen-view.png" + url="/images/runtime/runtime-logs-screen-view.png" + alt="Runtime component log example" + caption="Runtime component log example" + max-width="50%" +%} + +{:start="5"} +1. To download the log, click **Download**. + The file is downloaded as `.log`. + +### (Hybrid GitOps) Restoring provisioned Runtimes + +In case of cluster failure, restore the provisioned Hybrid Runtime from the existing runtime installation repository. +For partial or complete cluster failures, you can restore the Runtime to either the failed cluster or to a different cluster. +Restoring the provisioned Runtime reinstalls it, leveraging the resources in the existing Runtime repo. + +Restoring the runtime: +* Applies `argo-cd` from the installation manifests in your repo to your cluster +* Associates `argo-cd` with the existing installation repo +* Applies the Runtime and `argo-cd` secrets to the cluster +* Updates the Runtime config map (` .yaml` in the `bootstrap` directory) with the new cluster configuration for these fields: + `cluster` + `ingressClassName` + `ingressController` + `ingressHost` + +{::nomarkdown} +
+{:/} + +#### Restore a Hybrid Runtime +Reinstall the Hybrid Runtime from the existing installation repository to restore it to the same or a different cluster. + +**Before you begin** + +* Have the following information handy: + > All values must be the identical to the Runtime to be restored. + * Runtime name + * Repository URL + * Codefresh context + * Kube context: Required if you are restoring to the same cluster + +**How to** + +1. Run: + `cf runtime install --from-repo` +1. Provide the relevant values when prompted. +1. If you are performing the runtime recovery in a different cluster, verify the ingress resource configuration for `app-proxy`, `workflows`, and `default-git-source`. + If the health status remains as `Progressing`, do the following: + + * In the Runtime installation repo, check if the `ingress.yaml` files for the `app-proxy` and `workflows` are configured with the correct `host` and `ingressClassName`: + + `apps/app-proxy/overlays//ingress.yaml` + `apps/workflows/overlays/ /ingress.yaml` + + * In the Git Source repository, check the `host` and `ingressClassName` in `cdp-default-git-source.ingress.yaml`: + + `resources_ /cdp-default-git-source.ingress.yaml` + + See the [example](#ingress-example) below. + +{:start="4"} +1. If you have managed clusters registered to the hybrid runtime you are restoring, reconnect them. + Run the command and follow the instructions in the wizard: + `cf cluster add` + +1. Verify that you have a registered Git integration: + `cf integration git list --runtime ` + +1. If needed, create a new Git integration: + `cf integration git add default --runtime --provider github --api-url https://api.github.com` + +{::nomarkdown} +
+{:/} + +#### Ingress example +This is an example of the `ingress.yaml` for `workflows`. + + ```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + annotations: + ingress.kubernetes.io/protocol: https + ingress.kubernetes.io/rewrite-target: /$2 + nginx.ingress.kubernetes.io/backend-protocol: https + nginx.ingress.kubernetes.io/rewrite-target: /$2 + creationTimestamp: null + name: runtime-name-workflows-ingress + namespace: runtime-name +spec: + ingressClassName: nginx + rules: + - host: your-ingress-host.com + http: + paths: + - backend: + service: + name: argo-server + port: + number: 2746 + path: /workflows(/|$)(.*) + pathType: ImplementationSpecific +status: + loadBalancer: {} +``` + + +### (Hybrid GitOps) Configure browser to allow insecure Runtimes + +If at least one of your Hybrid Runtimes was installed in insecure mode (without an SSL certificate for the ingress controller from a CA), the UI alerts you that _At least one runtime was installed in insecure mode_. +{% include + image.html + lightbox="true" + file="/images/runtime/runtime-insecure-alert.png" + url="/images/runtime/runtime-insecure-alert.png" + alt="Insecure runtime installation alert" + caption="Insecure runtime installation alert" + max-width="100%" +%} + +All you need to do is to configure the browser to trust the URL and receive content. + +1. Select **View Runtimes** to the right of the alert. + You are taken to the Runtimes page, where you can see insecure Runtimes tagged as **Allow Insecure**. + {% include + image.html + lightbox="true" + file="/images/runtime/runtime-insecure-steps.png" + url="/images/runtime/runtime-insecure-steps.png" + alt="Insecure runtimes in Runtime page" + caption="Insecure runtimes in Runtime page" + max-width="40%" +%} +{:start="2"} +1. For _every_ insecure Runtime, select **Allow Insecure**, and when the browser prompts you to allow access, do as relevant: + +* Chrome: Click **Advanced** and then **Proceed to site**. +* Firefox: Click **Advanced** and then **Accept the risk and continue**. +* Safari: Click **Show Certificate**, and then select **Always allow content from site**. +* Edge: Click **Advanced**, and then select **Continue to site(unsafe)**. + +### (Hybrid GitOps) View notifications in Activity Log + +The Activity Log is a quick way to monitor notifications for Runtime events such as upgrades. A pull-down panel in the Codefresh toolbar, the Activity Log shows ongoing, success, and error notifications, sorted by date, starting with today's date. + +1. In the Codefresh UI, on the top-right of the toolbar, select  **Activity Log**. +1. To see notifications for provisioned Runtimes, filter by **Runtime**. + + {% include image.html + lightbox="true" + file="/images/runtime/runtime-activity-log.png" + url="/images/runtime/runtime-activity-log.png" + alt="Activity Log filtered by Runtime events" + caption="Activity Log filtered by Runtime events" + max-width="30%" + %} + +{:start="3"} + +1. To see more information on an error, select the **+** sign. + +### (Hybrid GitOps) Troubleshoot health and sync errors for Runtimes +The  icon with the Runtime in red indicates either health or sync errors. + +**Health errors** +Health errors are generated by Argo CD and by Codefresh for Runtime components. + +**Sync errors** +Runtimes with sync errors display an **Out of sync** status in Sync Status column. They are related to discrepancies between the desired and actual state of a Runtime component or one of the Git sources associated with the Runtime. + +**View errors** +For both views, select the Runtime, and then select **Errors Detected**. +Here is an example of health errors for a Runtime. + + {% include image.html + lightbox="true" + file="/images/runtime/runtime-health-sync-errors.png" + url="/images/runtime/runtime-health-sync-errors.png" + alt="Health errors for runtime example" + caption="Health errors for runtime example" + max-width="30%" + %} + + +### Related articles +[Add Git Sources to GitOps Runtimes]({{site.baseurl}}/docs/installation/git-sources/) +[Add external clusters to GitOps Runtimes]({{site.baseurl}}/docs/installation/managed-cluster/) +[Shared configuration repo for GitOps Runtimes]({{site.baseurl}}/docs/reference/shared-configuration) + + diff --git a/_docs/installation/runtime-architecture.md b/_docs/installation/runtime-architecture.md new file mode 100644 index 00000000..1e0841d2 --- /dev/null +++ b/_docs/installation/runtime-architecture.md @@ -0,0 +1,240 @@ +--- +title: "Runtime architectures" +description: "" +group: installation +toc: true +--- + +Overview TBD + +## Codefresh CI/CD architecture + +The most important components are the following: + +**Codefresh VPC:** All internal Codefresh services run in the VPC (analyzed in the next section). Codefresh uses Mongo and PostgreSQL to store user and authentication information. + +**Pipeline execution environment**: The Codefresh engine component is responsible for taking pipeline definitions and running them in managed Kubernetes clusters by automatically launching the Docker containers that each pipeline needs for its steps. + +**External actors**. Codefresh offers a [public API]({{site.baseurl}}/docs/integrations/ci-integrations/codefresh-api/) that is consumed both by the Web user interface and the [Codefresh CLI](https://codefresh-io.github.io/cli/){:target="\_blank"}. The API is also available for any custom integration with external tools or services. + +### CI/CD topology + +If we zoom into Codefresh Services for CI/CD, we will see the following: + +{% include image.html + lightbox="true" + file="/images/installation/topology-new.png" + url="/images/installation/topology-new.png" + alt="Topology diagram" + caption="Topology diagram (click to enlarge)" + max-width="100%" + %} + +### CI/CD core components + +{: .table .table-bordered .table-hover} +|Category | Component | Function | +| -------------- | ----------| ----------| +| Core | **pipeline-manager**| Manages all CRUD operations for CI pipelines.| +| | **cfsign** | Signs server TLS certificates for docker daemons, and generates client TLS certificates for hybrid pipelines. | +| | **cf-api** | Central back-end component that functions as an API gateway for other services, and handles authentication/authorization. | +| | **context-manager**| Manages the authentications/configurations used by Codefresh CI/CD and by the Codefresh engine. | +| | **runtime-environment-manager**| Manages the different runtime environments for CI pipelines. The runtime environment for CI/CD SaaS is fully managed by Codefresh. For CI/CD Hybrid, customers can add their own runtime environments using private Kubernetes clusters. | +| Trigger | **hermes**| Controls CI pipeline trigger management. See [triggers]({{site.baseurl}}/docs/pipelines/triggers/). | +| | **nomios**| Enables triggers from Docker Hub when a new image/tag is pushed.See [Triggers from Docker Hub]({{site.baseurl}}/docs/pipelines/triggers/dockerhub-triggers/). | +| | **cronus**| Enables defining Cron triggers for CI pipelines. See [Cron triggers]({{site.baseurl}}/docs/pipelines/triggers/cron-triggers/).| +| Log | **cf-broadcaster**| Stores build logs from CI pipelines. The UI and CLI stream logs by accessing the **cf-broadcaster** through a web socket. | +| Kubernetes | **cluster-providers** | Provides an interface to define cluster contexts to connect Kubernetes clusters in CI/CD installation environments. | +| | **helm-repo-manager** | Manages the Helm charts for CI/CD installation environments through the Helm repository admin API and ChartMuseum proxy. See [Helm charts in Codefresh]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/). | +| | **k8s-monitor** | The agent installed on every Kubernetes cluster, providing information for the Kubernetes dashboards. See [Kubernetes dashboards]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/). | +| |**charts-manager** | Models the Helm chart view in Codefresh. See [Helm chart view]({{site.baseurl}}/docs/deployments/helm/helm-releases-management/). | +| | **kube-integration** | Provides an interface to retrieve required information from a Kubernetes cluster, can be run either as an http server or an NPM module. | +| | **tasker-kubernetes** | Provides cache storage for Kubernetes dashboards. See [Kubernetes dashboards]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/). | + + +## Codefresh GitOps Platform architecture + +The diagram shows a high-level view of the Codefresh GitOps installation environment, and its core components, the Codefresh Control Plane, the Codefresh Runtime, and the Codefresh Clients. + +{% include +image.html +lightbox="true" +file="/images/getting-started/architecture/arch-codefresh-simple.png" +url="/images/getting-started/architecture/arch-codefresh-simple.png" +alt="Codefresh GitOps Platform architecture" +caption="Codefresh GitOps Platform architecture" +max-width="100%" +%} + +{::nomarkdown} +
+{:/} + +### Codefresh GitOps Control Plane +The Codefresh Control Plane is the SaaS component in the platform. External to the enterprise firewall, it does not have direct communication with the Codefresh Runtime, Codefresh Clients, or the customer's organizational systems. The Codefresh Runtime and the Codefresh Clients communicate with the Codefresh Control Plane to retrieve the required information. + + +{::nomarkdown} +
+{:/} + +### Codefresh GitOps Runtime +The Codefresh Runtime is installed on a Kubernetes cluster, and houses the enterprise distribution of the Codefresh Application Proxy and the Argo Project. +Depending on the type of GitOps installation, the Codefresh Runtime is installed either in the Codefresh platform (Hosted GitOps), or in the customer environment (Hybrid GitOps). Read more in [Codefresh GitOps Runtime architecture](#codefresh-gitops-runtime-architecture). + + +{::nomarkdown} +
+{:/} + +### Codefresh GitOps Clients + +Codefresh Clients include the Codefresh UI and the Codefresh CLI. +The Codefresh UI provides a unified, enterprise-wide view of deployments (runtimes and clusters), and CI/CD operations (Delivery Pipelines, workflows, and deployments) in the same location. +The Codefresh CLI includes commands to install hybrid runtimes, add external clusters, and manage runtimes and clusters. + +### Codefresh GitOps Runtime architecture +The sections that follow show detailed views of the GitOps Runtime architecture for the different installation options, and descriptions of the GitOps Runtime components. + +* [Hosted GitOps runtime architecture](#hosted-gitops-runtime-architecture) + For Hosted GitOps, the GitOps Runtime is installed on a _Codefresh-managed cluster_ in the Codefresh platform. +* Hybrid GitOps runtime architecture: + For Hybrid GitOps, the GitOps Runtime is installed on a _customer-managed cluster_ in the customer environment. The Hybrid GitOps Runtime can be tunnel- or ingress-based: + * [Tunnel-based](#tunnel-based-hybrid-gitops-runtime-architecture) + * [Ingress-based](#ingress-based-hybrid-gitops-runtime-architecture) +* GitOps Runtime components + * [Application Proxy](#application-proxy) + * [Argo Project](#argo-project) + * [Request Routing Service](#request-routing-service) + * [Tunnel Server](#tunnel-server) + * [Tunnel Client](#tunnel-client) + + +#### Hosted GitOps runtime architecture +In the hosted environment, the Codefresh Runtime is installed on a K8s cluster managed by Codefresh. + +{% include + image.html + lightbox="true" + file="/images/getting-started/architecture/arch-hosted.png" + url="/images/getting-started/architecture/arch-hosted.png" + alt="Hosted runtime architecture" + caption="Hosted runtime architecture" + max-width="100%" +%} + +#### Tunnel-based Hybrid GitOps runtime architecture +Tunnel-based Hybrid GitOps runtimes use tunneling instead of ingress controllers to control communication between the GitOps Runtime in the customer cluster and the Codefresh GitOps Platform. Tunnel-based runtimes are optimal when the cluster with the GitOps Runtime is not exposed to the internet. + +{% include + image.html + lightbox="true" + file="/images/getting-started/architecture/arch-hybrid-ingressless.png" + url="/images/getting-started/architecture/arch-hybrid-ingressless.png" + alt="Tunnel-based hybrid runtime architecture" + caption="Tunnel-based hybrid runtime architecture" + max-width="100%" +%} + + +#### Ingress-based Hybrid GitOps runtime architecture +Ingress-based runtimes use ingress controllers to control communication between the GitOps Runtime in the customer cluster and the Codefresh GitOps Platform. Ingress-based runtimes are optimal when the cluster with the GitOps Runtime is exposed to the internet. + + + +{% include + image.html + lightbox="true" + file="/images/getting-started/architecture/arch-hybrid-ingress.png" + url="/images/getting-started/architecture/arch-hybrid-ingress.png" + alt="Ingress-based hybrid runtime architecture" + caption="Ingress-based hybrid runtime architecture" + max-width="100%" +%} + + +#### Application Proxy +The GitOps Application Proxy (App-Proxy) functions as the Codefresh agent, and is deployed as a service in the GitOps Runtime. + +For tunnel-based Hybrid GitOps Runtimes, the Tunnel Client forwards the incoming traffic from the Tunnel Server using the Request Routing Service to the GitOps App-Proxy. +For Hybrid GitOps Runtimes with ingress, the App-Proxy is the single point-of-contact between the GitOps Runtime, and the GitOps Clients, the GitOps Platform, and any organizational systems in the customer environment. + + +The GitOps App-Proxy: +* Accepts and serves requests from GitOps Clients either via the UI or CLI +* Retrieves a list of Git repositories for visualization in the Client interfaces +* Retrieves permissions from the GitOps Control Plane to authenticate and authorize users for the required operations. +* Implements commits for GitOps-controlled entities, such as Delivery Pipelines and other CI resources +* Implements state-change operations for non-GitOps controlled entities, such as terminating Argo Workflows + +{::nomarkdown} +
+{:/} + +#### Argo Project + +The Argo Project includes: +* Argo CD for declarative continuous deployment +* Argo Rollouts for progressive delivery +* Argo Workflows as the workflow engine +* Argo Events for event-driven workflow automation framework + + +{::nomarkdown} +
+{:/} + +#### Request Routing Service +The Request Routing Service is installed on the same cluster as the GitOps Runtime in the customer environment. +It receives requests from the the Tunnel Client (tunnel-based) or the ingress controller (ingress-based), and forwards the request URLs to the Application Proxy, and webhooks directly to the Event Sources. + +>Important: + The Request Routing Service is available from runtime version 0.0.543 and higher. + Older runtime versions are not affected as there is complete backward compatibility, and the ingress controller continues to route incoming requests. + +#### Tunnel Server +Applies only to _tunnel-based_ Hybrid GitOps Runtimes. +The Codefresh Tunnel Server is installed in the Codefresh platform. It communicates with the enterprise cluster located behind a NAT or firewall. + +The Tunnel Server: +* Forwards traffic from Codefresh Clients to the client (customer) cluster. +* Manages the lifecycle of the Tunnel Client. +* Authenticates requests from the Tunnel Client to open tunneling connections. + +{::nomarkdown} +
+{:/} + +#### Tunnel Client +Applies only to _tunnel-based_ Hybrid GitOps Runtimes. + +Installed on the same cluster as the Hybrid GitOps Runtime, the Tunnel Client establishes the tunneling connection to the Tunnel Server via the WebSocket Secure (WSS) protocol. +A single Hybrid GitOps Runtime can have a single Tunnel Client. + +The Tunnel Client: +* Initiates the connection with the Tunnel Server. +* Forwards the incoming traffic from the Tunnel Server through the Request Routing Service to App-Proxy, and other services. + +{::nomarkdown} +
+{:/} + + +#### Customer environment +The customer environment that communicates with the GitOps Runtime and the GitOps Platform, generally includes: +* Ingress controller for ingress hybrid runtimes + The ingress controller is configured on the same Kubernetes cluster as the GitOps Runtime, and implements the ingress traffic rules for the GitOps Runtime. + See [Ingress controller requirements]({{site.baseurl}}/docs/installation/requirements/#ingress-controller). +* Managed clusters + Managed clusters are external clusters registered to provisioned Hosted or Hybrid GitOps runtimes for application deployment. + Hosted GitOps requires you to connect at least one external K8s cluster as part of setting up the Hosted GitOps environment. + Hybrid GitOps allow you to add external clusters after provisioning the runtimes. + See [Add external clusters to runtimes]({{site.baseurl}}/docs/installation/managed-cluster/). +* Organizational systems + Organizational Systems include the customer's tracking, monitoring, notification, container registries, Git providers, and other systems. They can be entirely on-premises or in the public cloud. + Either the ingress controller (ingress hybrid environments), or the Tunnel Client (tunnel-based hybrid environments), forwards incoming events to the Codefresh Application Proxy. + + ## Related articles +[Codefresh pricing](https://codefresh.io/pricing/) +[Codefresh features](https://codefresh.io/features/) + \ No newline at end of file diff --git a/_docs/installation/upgrade-gitops-cli.md b/_docs/installation/upgrade-gitops-cli.md new file mode 100644 index 00000000..30e06096 --- /dev/null +++ b/_docs/installation/upgrade-gitops-cli.md @@ -0,0 +1,87 @@ +--- +title: "Download/upgrade Codefresh CLI" +description: "Have the latest version of the Codefresh CLI for GitOps runtimes" +group: installation +toc: true +--- + +You need the Codefresh CLI to install Hybrid GitOps Runtimes, and have access to all the newest features. +For the initial download, you need to generate an API key and create the API authentication context, which you do from the UI. +When newer versions are available, the CLI automatically notifies you through a banner. You can use the existing API credentials for the upgrade. + + +## GitOps CLI installation modes +The table lists the modes available to install the Codefresh CLI. + +{: .table .table-bordered .table-hover} +| Install mode | OS | Commands | +| -------------- | ----------| ----------| +| `curl` | MacOS-x64 | `curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-darwin-amd64.tar.gz | tar zx && mv ./cf-darwin-amd64 /usr/local/bin/cf && cf version`| +| | MacOS-m1 |`curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-darwin-arm64.tar.gz | tar zx && mv ./cf-darwin-arm64 /usr/local/bin/cf && cf version` | +| | Linux - X64 |`curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-linux-amd64.tar.gz | tar zx && mv ./cf-linux-amd64 /usr/local/bin/cf && cf version` | +| | Linux - ARM | `curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-linux-arm64.tar.gz | tar zx && mv ./cf-linux-arm64 /usr/local/bin/cf && cf version`| +| `brew` | N/A| `brew tap codefresh-io/cli && brew install cf2`|```` + +## Install the GitOps CLI +Install the Codefresh CLI using the option that best suits you: `curl`, `brew`, or standard download. +If you are not sure which OS to select for `curl`, simply select one, and Codefresh automatically identifies and selects the right OS for CLI installation. + +1. Do one of the following: + * For first-time installation, go to the Welcome page, select **+ Install Runtime**. + * If you have provisioned a GitOps Runtime, in the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}, and select **+ Add Runtime**. +1. Install the Codefresh CLI: + * Select one of the installation modes. + * Generate the API key. + * Create the authentication context: + `cf config create-context codefresh --api-key` + + + {% include + image.html + lightbox="true" + file="/images/getting-started/quick-start/quick-start-download-cli.png" + url="/images/getting-started/quick-start/quick-start-download-cli.png" + alt="Download CLI to install runtime" + caption="Download CLI to install runtime" + max-width="30%" + %} + + +{::nomarkdown} + +{:/} + + +## Upgrade the GitOps CLI + +The Codefresh CLI automatically self-checks its version, and if a newer version is available, prints a banner with the notification. + + {% include + image.html + lightbox="true" + file="/images/runtime/cli-upgrade-banner.png" + url="/images/runtime/cli-upgrade-banner.png" + alt="Upgrade banner for Codefresh CLI" + caption="Upgrade banner for Codefresh CLI" + max-width="40%" + %} + + +You can upgrade to a specific version if you so require, or download the latest version to an output folder to upgrade at your convenience. + + +* Do any of the following: + * To upgrade to the latest version, run: + `cf upgrade` + * To upgrade to a specific version, even an older version, run: + `cf upgrade --version v ` + where: + ` ` is the version you want to upgrade to. + * To download the latest version to an output file, run: + `cf upgrade --version v -o ` + where: + * ` ` is the path to the destination file, for example, `/cli-download`. + +## Related articles +[Hosted GitOps Runtime setup]({{site.baseurl}}/docs/installation/hosted-runtime) +[Hybrid GitOps Runtime installation]({{site.baseurl}}/docs/installation/hybrid-gitops) diff --git a/_docs/integrations/ci-integrations.md b/_docs/integrations/ci-integrations.md deleted file mode 100644 index f43ad8e4..00000000 --- a/_docs/integrations/ci-integrations.md +++ /dev/null @@ -1,107 +0,0 @@ ---- -title: "CI integrations" -description: "" -group: integrations -toc: true ---- - -Use Codefresh's Hosted GitOps with any popular Continuous Integration (CI) solution, not just with Codefresh CI. - -You can connect a third-party CI solution to Codefresh, such as GitHub Actions for example, to take care of common CI tasks such as building/testing/scanning source code, and have Codefresh Hosted GitOps still responsible for the deployment, including image enrichment and reporting. -The integration brings in all the CI information to your images which you can see in the Images dashboard. - -See [Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/). - -### Codefresh image reporting and enrichment action -To support the integration between Codefresh and third-party CI platforms and tools, we have created dedicated actions for supported CI tools in the Codefresh Marketplace. These actions combine image enrichment and reporting through integrations with issue tracking and container registry tools. - ->You can also configure the integration directly in the Codefresh UI, as described in [Connect a third-party CI platform/tool to Codefresh](#connect-a-third-party-ci-platformtool-to-codefresh). - - -Use the action as follows: - -1. Create your pipeline with your CI platform/tool as you usually do. -1. Use existing CI actions for compiling code, running unit tests, security scanning etc. -1. Place the final action in the pipeline as the "report image" action provided by Codefresh. - See: - [GitHub Action Codefresh report image](https://github.com/marketplace/actions/codefresh-report-image){:target="\_blank"} - [Codefresh Classic Codefresh report image](https://codefresh.io/steps/step/codefresh-report-image){:target="\_blank"} -1. When the pipeline completes execution, Codefresh retrieves the information on the image that was built and its metadata through the integration names specified (essentially the same data that Codefresh CI would send automatically). -1. View the image in Codefresh's [Images dashboard]({{site.baseurl}}/docs/deployment/images/), and in any [application]({{site.baseurl}}/docs/deployment/applications-dashboard/) in which it is used. - -### Connect a third-party CI platform/tool to Codefresh -Connecting the CI platform/tool to Codefresh from the UI includes configuring the required arguments, and then generating and copying the YAML manifest for the report image to your pipeline. - -1. In the Codefresh UI, go to [Integrations](https://g.codefresh.io/2.0/account-settings/integrations){:target="\_blank"}. -1. Filter by **CI tools**, then select the CI tool and click **Add**. -1. Define the arguments for the CI tool: - [Codefresh Classic]({{site.baseurl}}/docs/integrations/ci-integrations/codefresh-classic/) - [GitHub Action]({{site.baseurl}}/docs/integrations/ci-integrations/github-actions/) - [Jenkins]({{site.baseurl}}/docs/integrations/ci-integrations/jenkins/) - - For the complete list of arguments you can use, see [CI integration argument reference](#ci-integration-argument-reference) in this article. - -1. To generate a YAML snippet with the arguments, on the top-right, click **Generate Manifest**. - Codefresh validates the generated manifest, and alerts you to undefined arguments that are required, and other errors. - - {% include image.html -lightbox="true" -file="/images/integrations/generated-manifest-with-error.png" -url="/images/integrations/generated-manifest-with-error.png" -alt="Example of manifest generated for Codefresh Classic with validation errors" -caption="Example of manifest generated for Codefresh Classic with validation errors" -max-width="50%" -%} - -{:start="5"} -1. If required, click **Close**, update as needed and generate the manifest again. -1. If there are no validation errors, click **Copy**. - -{% include image.html -lightbox="true" -file="/images/integrations/classic/classic-manifest.png" -url="/images/integrationsclassic/classic-manifest.png" -alt="Example of manifest generated for Codefresh Classic" -caption="Example of manifest generated for Codefresh Classic" -max-width="50%" -%} - -{:start="6"} -1. Paste the copied manifest as the last step in your CI pipeline. - -### CI integration argument reference -The table describes _all_ the arguments required for CI integrations in general. The actual arguments required, differs according to the CI integration tool. - -{: .table .table-bordered .table-hover} -| Argument | Description | Required/Optional/Default | -| ---------- | -------- | ------------------------- | -| `CF_HOST` | _Deprecated from v 0.0.460 and higher._ Recommend using `CF_RUNTIME_NAME` instead. {::nomarkdown}
CF_HOST has been deprecated because the URL is not static, and any change can fail the enrichment.
The URL to the cluster with the Codefresh runtime to integrate with. If you have more than one runtime, select the runtime from the list. Codefresh displays the URL of the selected runtime cluster.{:/} | _Deprecated_ | -| `CF_RUNTIME_NAME` | The runtime to use for the integration. If you have more than one runtime, select the runtime from the list. | Required | -| `CF_PLATFORM_URL` | The root URL of the Codefresh application. The default value is `https://g.codefresh.io`. | Optional | -| `CF_API_KEY` | The API key for authentication. Generate the key for the integration. | Required | -| `CF_CONTAINER_REGISTRY_INTEGRATION` | The name of the container registry integration created in Codefresh where the image is stored. See [Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/). | Optional | -| `CF_JIRA_INTEGRATION` | _Deprecated from version 0.0.565 and higher._ Replaced by `CF_ISSUE_TRACKING_INTEGRATION`. | _Deprecated_ -| `CF_ISSUE_TRACKING_INTEGRATION` | The name of the issue tracking integration created in Codefresh to use for image enrichment. Relevant only if Jira enrichment is required for the image. If you don't have a Jira integration, click **Create Atlassian Jira Integration** and configure settings. See [Jira integration]({{site.baseurl}}/docs/integrations/issue-tracking/jira/). | Optional | -| `CF_IMAGE` | The image to be enriched and reported in Codefresh. Pass the `[account-name]/[image-name]:[tag]` built in your CI. | Required | -| `CF_WORKFLOW_NAME` | The name assigned to the workflow that builds the image. When defined, the name is displayed in the Codefresh platform. Example, `Staging step` | Optional | -| `CF_GIT_BRANCH` | The Git branch with the commit and PR (pull request) data to add to the image. Pass the Branch from the event payload used to trigger your action. | Required | -| `CF_GIT_REPO` | The Git repository with the configuration and code used to build the image. {::nomarkdown}- Optional for GitHub Actions.
- Required for Classic and Jenkins.
- Optional when you don't define other related Git provider arguments. When not defined, Codefresh retrieves the required information from the runtime selected for the integration.
- Required when you define at least one of the Git provider arguments. For example, when you define CF_GITLAB_TOKEN, then you must define all Git provider arguments, in this case, CF_GIT_PROVIDER as gitlab, and CF_GITLAB_HOST_URL.
- Optional when you don't define any GitLab-specific arguments. When not defined, Codefresh retrieves the required information from the runtime selected for the integration.
- Required when you define at least one of the GitLab-specific arguments, such as CF_GIT_PROVIDER as gitlab, or CF_GITLAB_HOST_URL.
- Optional when you don't define other related GitLab-specific arguments. When not defined, Codefresh retrieves the required information from the runtime selected for the integration.
- Required when you define at least one of the GitLab-specific arguments, such as CF_GIT_PROVIDER as gitlab, or CF_GITLAB_TOKEN.
- Optional when you don't define other related Bitbucket-specific arguments. When not defined, Codefresh retrieves the required information from the runtime selected for the integration.
- Required when you define at least one of the Bitbucket-specific arguments, such as CF_GIT_PROVIDER as bitbucket, CF_BITBUCKET_PASSWORD or CF_BITBUCKET_HOST_URL.
- Optional when you don't define other related Bitbucket-specific arguments. When not defined, Codefresh retrieves the required information from the runtime selected for the integration.
- Required when you define at least one of the Bitbucket-specific arguments, such as CF_GIT_PROVIDER as bitbucket, CF_BITBUCKET_USERNAME, or CF_BITBUCKET_HOST_URL.
- Optional when you don't define other related Bitbucket Server-specific arguments. When not defined, Codefresh retrieves the required information from the runtime selected for the integration.
- Required when you define at least one of the Bitbucket Server-specific arguments, such as CF_GIT_PROVIDER as bitbucket, CF_BITBUCKET_USERNAME or CF_BITBUCKET_PASSWORD.
- For a GitHub Container registry, select GHCR_GITHUB_TOKEN_AUTHENTICATION even if you have not created an integration in Codefresh.
Codefresh retrieves and provides the explicit credentials for the container registry on generating the integration manifest. - To create a container registry integration if you don't have one, click Create Container Registry Integration, and then configure the settings.
See Container registry integrations. - Optional when you don't define other related Git provider arguments. When not defined, Codefresh retrieves the required information from the runtime selected for the integration.
- Required when you define at least one of the Git provider arguments. For example, when you define CF_GITLAB_TOKEN, then you _must_ define all Git provider arguments, in this case, CF_GIT_PROVIDER as gitlab, and CF_GITLAB_HOST_URL.
- Optional when you don't define any GitLab-specific arguments. When not defined, Codefresh retrieves the required information from the runtime selected for the integration.
- Required when you define at least one of the GitLab-specific arguments, such as CF_GIT_PROVIDER as gitlab, or CF_GITLAB_HOST_URL.
- Optional when you don't define other related GitLab-specific arguments. When not defined, Codefresh retrieves the required information from the runtime selected for the integration.
- Required if you define at least one of the GitLab-specific arguments, such as CF_GIT_PROVIDER as gitlab, or CF_GITLAB_TOKEN.
- Optional when you don't define other related Bitbucket-specific arguments. When not defined, Codefresh retrieves the required information from the runtime selected for the integration.
- Required when you define at least one of the Bitbucket-specific arguments, such as CF_GIT_PROVIDER as bitbucket, CF_BITBUCKET_PASSWORD or CF_BITBUCKET_HOST_URL.
- Optional when you don't define other related Bitbucket-specific arguments. When not defined, Codefresh retrieves the required information from the runtime selected for the integration.
- Required when you define at least one of the Bitbucket-specific arguments, such as CF_GIT_PROVIDER as bitbucket, CF_BITBUCKET_USERNAME, or CF_BITBUCKET_HOST_URL.
- Optional when you don't define other related Bitbucket Server-specific arguments. When not defined, Codefresh retrieves the required information from the runtime selected for the integration.
- Required when you define at least one of the Bitbucket Server-specific arguments, such as
- {:/} | Required |
-| `CF_GIT_PROVIDER` | The Git provider for the integration, and can be either `github`, `gitlab`, or `bitbucket`. {::nomarkdown}
- {:/}| Optional |
-| `CF_GITLAB_TOKEN` | The token to authenticate the GitLab account. {::nomarkdown}
- {:/} | Optional |
-| `CF_GITLAB_HOST_URL` | The URL address of your GitLab Cloud/Server instance. {::nomarkdown}
- {:/} | Optional |
-| `CF_BITBUCKET_USERNAME` | The username for the Bitbucket or the Bitbucket Server (on-prem) account. {::nomarkdown}
- {:/}| Optional |
-| `CF_BITBUCKET_PASSWORD` | The password for the Bitbucket or the BitBucket Server (on-prem) account. {::nomarkdown}
- {:/}| Optional |
-| `CF_BITBUCKET_HOST_URL` | Relevant for Bitbucket Server accounts only. The URL address of your Bitbucket Server instance. Example, `https://bitbucket-server:7990`. {::nomarkdown}
diff --git a/_docs/integrations/ci-integrations/github-actions.md b/_docs/integrations/ci-integrations/github-actions.md deleted file mode 100644 index fa6dd1c8..00000000 --- a/_docs/integrations/ci-integrations/github-actions.md +++ /dev/null @@ -1,258 +0,0 @@ ---- -title: "GitHub Actions" -description: "" -group: integrations -sub_group: ci-integrations -toc: true ---- - -Use Codefresh Hosted GitOps with any popular Continuous Integration (CI) solution, not just with Codefresh CI. -GitHub Actions is one of the third-party CI solutions that you can connect to Codefresh for deployment with image reporting and enrichment. - - Connecting a GitHub Action, adds the CI information to images which are displayed in the Images dashboard, as in the example below. - - {% include - image.html - lightbox="true" - file="/images/integrations/images-dashboard.png" - url="/images/integrations/images-dashboard.png" - alt="Images dashboard with enriched image information" - caption="Images dashboard with enriched image information" - max-width="70%" - %} - -For information on how to use the image reporting action in your GitHub Action pipeline and how to configure the integration, see [CI Integrations]({{site.baseurl}}/docs/integrations/ci-integrations/). - - -### Example of GitHub Actions pipeline with Codefresh report image action - - -Here is an example pipeline that uses GitHub Actions to build a container image, and the Codefresh action to enrich and report the resulting image to Codefresh. - -Because a Jira integration account is configured in Codefresh, the step needs only the name for `CF_JIRA_INTEGRATION`, instead of explicit credentials `CF_JIRA_API_TOKEN`, `CF_JIRA_HOST_URL`, and `CF_JIRA_EMAIL`. - - -{% highlight yaml %} -{% raw %} - -name: Docker Image CI - -on: - push: - branches: [ main ] - pull_request: - branches: [ main ] -jobs: - build: - environment: - name: test - runs-on: ubuntu-latest - steps: - - name: Checkout - uses: actions/checkout@v3 - - name: Login to DockerHub - uses: docker/login-action@v2 - with: - username: ${{ secrets.DOCKERHUB_USERNAME }} - password: ${{ secrets.DOCKERHUB_TOKEN }} - - name: Build & push the Docker image - env: - CF_IMAGE: ${{ secrets.DOCKERHUB_USERNAME }}/build-by-github-action:0.0.1 - run: | - docker build . --file Dockerfile --tag $CF_IMAGE && docker push $CF_IMAGE - echo "Image should be accessible to your local machine (after docker login) by:" - echo "docker pull $CF_IMAGE" - docker pull $CF_IMAGE - echo "On the next step, the report image would use the integration to pull information on the reported image, using the specified enrichers." - - name: report image by action - with: - # Name of runtime to implement the enrichment - CF_RUNTIME_NAME: 'codefresh-hosted' - - # Codefresh API key !! Committing a plain text token is a security risk. We highly recommend using encrypted secrets. !! - # Documentation - https://docs.github.com/en/actions/security-guides/encrypted-secrets - CF_API_KEY: ${{ secrets.USER_TOKEN }} - - # Name of Container registry integration - CF_CONTAINER_REGISTRY_INTEGRATION: 'docker' - - # The git branch which is related for the commit - CF_GIT_BRANCH: 'main' - - # Image path to enrich - CF_IMAGE: ${{ secrets.DOCKERHUB_USERNAME }}/build-by-github-action:0.0.1 - - # GitHub Access token !! Committing a plain text token is a security risk. We highly recommend using encrypted secrets. !! - # Documentation - https://docs.github.com/en/actions/security-guides/encrypted-secrets - CF_GITHUB_TOKEN: ${{ secrets.CF_GITHUB_TOKEN }} - - # Name of Jira integration - CF_ISSUE_TRACKING_INTEGRATION: 'jira' - - # String starting with the issue ID to associate with image - CF_JIRA_MESSAGE: 'CR-11027' - - # Jira project filter - CF_JIRA_PROJECT_PREFIX: "CR" - uses: codefresh-io/codefresh-report-image@latest - - -{% endraw %} -{% endhighlight yaml %} - -### GitHub Action-Codefresh integration arguments -The table describes the arguments required to connect a GitHub Action to Codefresh. - - - {: .table .table-bordered .table-hover} -| Argument | Description | Required/Optional/Default | -| ---------- | -------- | ------------------------- | -| `CF_HOST` | _Deprecated from v 0.0.460 and higher._ Recommend using `CF_RUNTIME_NAME` instead. {::nomarkdown}
CF_HOST has been deprecated because the URL is not static, and any change can fail the enrichment.
The URL to the cluster with the Codefresh runtime to integrate with. If you have more than one runtime, select the runtime from the list. Codefresh displays the URL of the selected runtime cluster.{:/} | Required | -| `CF_RUNTIME_NAME` | The runtime to use for the integration. If you have more than one runtime, select the runtime from the list. | Required | -| `CF_PLATFORM_URL` | The root URL of the Codefresh application. The default value is `https://g.codefresh.io`. | Optional | -| `CF_API_KEY` | The API key to authenticate the GitHub Actions user to Codefresh. Generate the key for the GitHub Action. {::nomarkdown}
Enter this token in GitHub Actions as a secret with the name CF_API_KEY. You can then reference it in all GitHub pipelines as you would any other secret.{:/}| Required | -| `CF_CONTAINER_REGISTRY_INTEGRATION` | The name of the container registry integration created in Codefresh where the image is stored. {::nomarkdown}
-{:/} - -#### CF_IMAGE - -**Example: Report full repo and branch information** -This example illustrates how to define the value for `CF_IMAGE` to report the repo owner, name, and short branch, with the Git hash. - - Value: - {% raw %}`${{ github.repository }}/${{ github.ref_name }}/${{ github.sha }}`{% endraw %} - - where: - * {% raw %}`${{ github.repository }}`{% endraw %} reports the owner of the repository and the name of the repository. For example, `nr-codefresh/codefresh-production`. - * {% raw %}`${{ github.ref_name }}`{% endraw %} reports the short reference to the branch that triggered the workflow. For example, `auth-feature-branch`. - * {% raw %}`${{ github.sha }}`{% endraw %} reports the complete commit SHA that triggered the workflow. For example, `fa53bfa91df14c4c9f46e628a65ee21dd574490a`. - - - -**Example: Report a specific image tag** -This example illustrates how to define the value for `CF_IMAGE` when you know the specific image version you want to report. - -Value: -{% raw %}`${{ github.repository }}:`{% endraw %} - -where: -* {% raw %}`${{ github.repository }}`{% endraw %} reports the owner of the repository and the name of the repository. For example, `nr-codefresh/codefresh-production`. -* ` ` reports the hard-coded tag `v1.0`. - - -**Example: Report the latest Git tag available on repository** -This example illustrates how to define the value for `CF_IMAGE` to report the latest Git tag on the repository. - -Value: -{% raw %}`codefresh/${{ github.repository }}/latest`{% endraw %} - -where: -* {% raw %}`codefresh`{% endraw %} is the hard-coded owner of the image. -* {% raw %}`${{ github.repository }}`{% endraw %} reports the owner of the repository and the name of the repository. For example, `nr-codefresh/codefresh-production`. -* {% raw %}`latest`{% endraw %} reports the latest Git tag available for the repository defined by {% raw %}`${{ github.repository }}`{% endraw %}. For example, `v1.0.4-14-g2414721`. - -{::nomarkdown} -
-{:/} - -#### CF_GIT_BRANCH - -**Example: Report fully-formed reference of the branch or tag** -This example illustrates how to define the value for `CF_GIT_BRANCH` to report the fully-formed reference of the branch or tag that triggered the workflow run. -For workflows triggered by push events, this is the branch or tag ref that was pushed. -For workflows triggered by pull_requests, this is the pull request merge branch. - -Value: -{% raw %}`${{ github.ref }}`{% endraw %} - -where: -* {% raw %}`${{ github.ref }}`{% endraw %} is the reference to the branch or tag. For example, `refs/heads/auth-feature-branch` (branch), and `refs/pull/#843/merge` (pull request). - -**Example: Report short reference name of the branch or tag** -This example illustrates how to define the value for `CF_GIT_BRANCH` to report only the name of the branch or tag that triggered the workflow run. - - -Value: -{% raw %}`${{ github.ref-name }}`{% endraw %} - -where: -* {% raw %}`${{ github.ref-name }}`{% endraw %} is the name of the target branch or tag. For example, `auth-feature-branch`. - -{::nomarkdown} -
-{:/} - -#### CF_JIRA_MESSAGE -The Jira message represents an existing Jira issue, and must be a literal string. - - Value: - `CR-1246` - -### GitHub Action logs -View and analyze logs for GitHub Action workflows through the Logs tab. When a GitHub Action is run, it is added to the Logs tab. -You can: -* Filter by status or by date range to view a subset of actions -* Navigate to the build file in GitHub Actions, and view the Codefresh report image step - -{% include image.html -lightbox="true" -file="/images/integrations/github-actions/github-actions-logs.png" -url="/images/integrations/github-actions/github-actions-logs.png" -alt="GitHub Action: Logs tab" -caption="GitHub Action: Logs tab" -max-width="50%" -%} - -**Build YAML in GitHub Action** - -The Run column includes the link to the build files for the actions. - -Here are examples of the build file for the GitHub Action (top) and of the Codefresh report image step in the action (down). - -{% include image.html -lightbox="true" -file="/images/integrations/github-actions/action-build-yaml.png" -url="/images/integrations/github-actions/action-build-yaml.png" -alt="Build file in GitHub Action" -caption="Build file in GitHub Action" -max-width="50%" -%} - -{% include image.html -lightbox="true" -file="/images/integrations/github-actions/actiosn-cf-report-image-step.png" -url="/images/integrations/github-actions/actiosn-cf-report-image-step.png" -alt="Codefresh report image step in GitHub Action build file" -caption="Codefresh report image step in GitHub Action build file" -max-width="50%" -%} - - -### Related articles -[Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration/) -[Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/) -[Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/) -[Issue-tracking integrations]({{site.baseurl}}/docs/integrations/issue-tracking/) - - diff --git a/_docs/integrations/ci-integrations/jenkins.md b/_docs/integrations/ci-integrations/jenkins.md deleted file mode 100644 index ddfa1eba..00000000 --- a/_docs/integrations/ci-integrations/jenkins.md +++ /dev/null @@ -1,250 +0,0 @@ ---- -title: "Jenkins" -description: "" -group: integrations -sub_group: ci-integrations -toc: true ---- - - Use Hosted GitOps with any popular Continuous Integration (CI) solution, not just with Codefresh CI. Jenkins is one of the third-party CI platform/tools that you can connect to Codefresh for deployment with image enrichment and reporting. - - Connecting a Jenkins pipeline, adds the CI information to images which are displayed in the Images dashboard, as in the example below. - - {% include - image.html - lightbox="true" - file="/images/integrations/images-dashboard.png" - url="/images/integrations/images-dashboard.png" - alt="Images dashboard with enriched image information" - caption="Images dashboard with enriched image information" - max-width="70%" - %} - - -For information on how to use the image reporting action in your Jenkins pipeline and how to configure the integration, see [CI Integrations]({{site.baseurl}}/docs/integrations/ci-integrations/). - -### Example of Jenkins pipeline with report image step - -{% highlight yaml %} -{% raw %} - -pipeline { - - agent any - stages { - stage('Clone repository') { - steps { - checkout scm - } - } - stage ('Build & Push ') { - environment { - CF_IMAGE= credentials('CF_IMAGE') - } - steps { - sh 'echo "Building $CF_IMAGE"' - script { - def app - app = docker.build("${env.CF_IMAGE}") - // require credentials to be stored under DOCKERHUB - docker.withRegistry('https://registry.hub.docker.com', 'DOCKERHUB') { - app.push("latest") - } - } - sh ''' - # test we have image in repository. - docker pull $CF_IMAGE - ''' - } - } - - stage('report image') { - environment { - // Name of runtime to implement the enrichment - CF_RUNTIME_NAME= 'codefresh-hosted' - - // Image path to enrich - CF_IMAGE= credentials('CF_IMAGE') - - // Codefresh API key !! Committing a plain text token is a security risk. We highly recommend using encrypted secrets. !! - // Documentation - https://www.jenkins.io/doc/book/using/using-credentials - CF_API_KEY= credentials('CF_API_KEY') - - // Name of Container registry integration - CF_CONTAINER_REGISTRY_INTEGRATION= 'docker' - - // Name of Jira integration - CF_ISSUE_TRACKING_INTEGRATION= 'jira' - - // String starting with the issue ID to associate with image - CF_JIRA_MESSAGE= 'CR-11027' - - // Jira project filter - CF_JIRA_PROJECT_PREFIX= 'CR' - - // GitHub Access token !! Committing a plain text token is a security risk. We highly recommend using encrypted secrets. !! - // Documentation - https://www.jenkins.io/doc/book/using/using-credentials - CF_GITHUB_TOKEN= credentials('CF_GITHUB_TOKEN') - } - steps { - sh ''' - export CF_CI_TYPE="jenkins" - # add workflow details - export CF_WORKFLOW_NAME="${CF_WORKFLOW_NAME:-$JOB_NAME}" - export CF_WORKFLOW_URL="${CF_WORKFLOW_URL:-$BUILD_URL}" - # add git branch - export CF_GIT_PROVIDER="${CF_GIT_PROVIDER:-github}" - WITHOUT_POSTFIX="${GIT_URL%.*}" - export CF_GIT_REPO="${CF_GIT_REPO:-${WITHOUT_POSTFIX#*//*/}}" - # slice branch name from repo/branch - export CF_GIT_BRANCH="${CF_GIT_BRANCH:-${GIT_BRANCH#*/}}" - env | cut -f 1 -d "=" | grep -E "^CF_" > cf_env - docker run --env-file=cf_env "quay.io/codefresh/codefresh-report-image:latest" - ''' - } - } - - } -} - -{% endraw %} -{% endhighlight yaml %} - -### Jenkins-Codefresh integration arguments -The table describes the arguments to connect Codefresh Classic to Codefresh. - -{: .table .table-bordered .table-hover} -| Argument | Description | Required/Optional/Default | -| ---------- | -------- | ------------------------- | -| `CF_RUNTIME_NAME` | The runtime to use for the integration. If you have more than one runtime, select the runtime from the list. | Required | -| `CF_PLATFORM_URL` | The root URL of the Codefresh application. The default value is `https://g.codefresh.io`. | Optional | -| `CF_API_KEY` | The API key to authenticate the Codefresh Classic user to Codefresh. Generate the key for the integration. | Required | -| `CF_CONTAINER_REGISTRY_INTEGRATION` | The name of the container registry integration created in Codefresh where the image is stored. To create a container registry integration if you don't have one, click **Create Container Registry Integration**, and then configure the settings. See [Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/). | Optional | -| `CF_JIRA_INTEGRATION` | Deprecated from version 0.0.565. Replaced by `CF_ISSUE_TRACKING_INTEGRATION`. | _Deprecated_ -| `CF_ISSUE_TRACKING_INTEGRATION` | The name of the issue tracking integration created in Codefresh to use to enrich the image. Relevant only if Jira enrichment is required for the image. If you don't have a Jira integration, click **Create Atlassian Jira Integration** and configure settings. See [Jira integration]({{site.baseurl}}/docs/integrations/issue-tracking/jira/). | Optional | -| `CF_IMAGE` | The image to be enriched and reported in Codefresh. Pass the `[account-name]/[image-name]:[tag]` built in your CI. | Required | -| `CF_GIT_BRANCH` | The Git branch with the commit and PR (pull request) data to add to the image. Pass the Branch from the event payload used to trigger your action. | Required | -| `CF_GIT_REPO` | The Git repository with the configuration and code used to build the image. | Required | -| `CF_GIT_PROVIDER` | The Git provider for the integration, and can be either `github`, `gitlab`, or `bitbucket`. {::nomarkdown}- {:/}| Optional |
-| `CF_GITLAB_TOKEN` | The token to authenticate the GitLab account. {::nomarkdown}
- {:/} | Optional |
-| `CF_GITLAB_HOST_URL` | The URL address of your GitLab Cloud/Server instance. {::nomarkdown}
- {:/} | Optional |
-| `CF_BITBUCKET_USERNAME` | The username for the Bitbucket or the Bitbucket Server (on-prem) account. {::nomarkdown}
- {:/}| Optional |
-| `CF_BITBUCKET_PASSWORD` | The password for the Bitbucket or the Bitbucket Server (on-prem) account. {::nomarkdown}
- {:/}| Optional |
-| `CF_BITBUCKET_HOST_URL` | Relevant for Bitbucket Server accounts only. The URL address of your Bitbucket Server instance. Example, `https://bitbucket-server:7990`. {::nomarkdown}
] [ --atomic ] [ --debug ] [ helm upgrade parameters ] +``` +### Step 6: Install the Codefresh Kubernetes Agent + +Install the `cf-k8s-agent` on a cluster separate from the installer, or in a different namespace on the same cluster. +The `cf-k8s-agent` accesses Kubernetes resources (pods, deployments, services, etc.) behind the firewall to display them in the Codefresh UI. The agent streams updates from cluster resources and then sends information updates to the `k8s-monitor` service. + +1. Create a staging directory for the agent: + +``` +kcfi init k8s-agent +``` + A staging directory is created, named k8s-agent with a `config.yaml`. +1. Edit k8s-agent/config.yaml ?? for what?? + +1. Run: + +``` +kcfi deploy [ -c config.yaml ] [-n namespace] +``` + where: + [namespace] is the namespace if you are installing the agent in the same cluster. + + + + +## High-Availability (HA) with active-passive clusters +Enable high-availability in the Codefresh platform for disaster recovery with an active-passive cluster configuration. +Review the prerequisites, and then do the following to configure high-availability: +* For new installations, install Codefresh on the active cluster +* Install Codefresh on the passive cluster +* When needed, switch between clusters for disaster recovery + +### Prerequisites + +* **K8s clusters** + Two K8s clusters, one designated as the active cluster, and the other designated as the passive cluster for disaster recovery. + +* **External databases and services** + Databases and services external to the clusters. + + * Postgres database (see [Configuring an external Postgres database](#configuring-an-external-postgres-database)) + * MongoDB (see [Configuring an external MongoDB](#configuring-an-external-mongodb)) + * Redis service (see [Configuring an external Redis service](#configure-an-external-redis-service)) + * RabbitMQ service (see [Configuring an external RabbitMQ service](#configure-an-external-redis-service)) + * Consul service (see [Configuring an external Consul service](#configuring-an-external-consul-service)) + +* **DNS record** + To switch between clusters for disaster recovery + +### Install Codefresh on active cluster + +If you are installing Codefresh for the first time, install Codefresh on the cluster designated as the _active_ cluster. +See [Installing the Codefresh platform]({{site.baseurl}}/docs/administration/codefresh-on-prem/#install-the-codefresh-platform). + +### Install Codefresh on passive cluster + +First get the `values.yaml` file from the current Codefresh installation on the active cluster. Then install Codefresh on the passive cluster using Helm. + +**1. Get values.yaml** +1. Switch your kube context to the active cluster. +1. Get `values.yaml` from the active cluster: + `helm get values ${release_name} -n ${namespace} > cf-passive-values.yaml` + where: + `{release-version}` is the name of the Codefresh release, and is by default `cf`. + `${namespace}` is the namespace with the Codefresh release, and is by default `codefresh`. + +{:start="3"} +1. Update the required variables in `cf-passive-values.yaml`. + > If the variables do not exist, add them to the file. + + * In the `global` section, disable `seedJobs` by setting it to `false`: + + ```yaml + global: + seedJobs: false + ``` + + * Add variable `FREEZE_WORKFLOWS_EXECUTION` to `cfapi`, and set it to `true`. + + ```yaml + cfapi: + env: + FREEZE_WORKFLOWS_EXECUTION: true + ``` + +**2. Install Codefresh on passive cluster** + +1. Download the Helm chart: + `helm repo add codefresh-onprem https://chartmuseum.codefresh.io/codefresh` + `helm fetch codefresh-onprem/codefresh --version ${release-version}` + where: + `{release-version}` is the version of Codefresh you are downloading. + +1. Unzip the Helm chart: + `tar -xzf codefresh-${release-version}.tgz` +1. Go to the folder where you unzipped the Helm chart. +1. Install Codefresh with the Helm command using `cf-passive-values.yaml`: + `helm install cf . -f ${path}/cf-passive-values.yaml -n codefresh` + + +### Switch between clusters for disaster recovery + +For disaster recovery, switch between the active and passive clusters. + +1. In the `cfapi` deployment on the _active_ cluster, change the value of `FREEZE_WORKFLOWS_EXECUTION` from `false` to `true`. + If the variable does not exist, add it, and make sure the value is set to `true`. +1. In the `cfapi` deployment on the _passive_ cluster, change the value of `FREEZE_WORKFLOWS_EXECUTION` from `true` to `false`. +1. Switch DNS from the currently active cluster to the passive cluster. + +### Services without HA + +The following services cannot run in HA, but are not critical in case of downtime or during the process of switchover from active to passive. +These services are not considered critical as they are part of build-handling. In case of failure, a build retry occurs, ensuring that the build is always handled. +* `cronus` +* `cf-sign` + + +## Additional configuration + +After you install Codefresh, these are post-installation operations that you should follow. + +### Selectively enable SSO provider for account +As a Codefresh administrator, you can select the providers you want to enable for SSO in your organization, for both new and existing accounts. +You can always renable a provider when needed. + + +1. Sign in as Codefresh admin. +1. From the left pane, select **Providers**. +1. Disable the providers not relevant for the accounts. +These providers are not displayed as options during sign-up/sign-in. + + +### (Optional) Set up Git integration + +Codefresh supports out-of-the-box Git logins using your local username and password, or logins using your Git provider, as described below.You can also configure login to supported SSO providers after installation, as described in [Setting up OpenID Connect (OIDC) Federated Single Sign-On (SSO)]({{site.baseurl}}/docs/administration/single-sign-on/oidc). + +If you’d like to set up a login to Codefresh using your Git provider, first login using the default credentials (username: `AdminCF`, password: `AdminCF` and add your Git provider OAuth integration details in our admin console: + +**Admin Management** > **IDPs** tab + +To get the Client ID and Client Secret for each of the supported Git providers, follow the instructions according to your VCS provider. + +#### GitHub Enterprise + +Navigate to your GitHub organization settings: https://github.com/organizations/your_org_name/settings. + +On the left-hand side, under **Developer settings**, select **OAuth Apps**, and click **Register an Application**. + +Complete the OAuth application registration as follows: + +- **Application name:** codefresh-on-prem (or a significant name) +- **Homepage URL:** https://your-codefresh-onprem-domain +- **Authorization callback URL:** https://your-codefresh-onprem-domain/api/auth/github/callback + +After registration, note down the created Client ID and Client Secret. They will be required for the settings in **Codefresh Admin**->**IDPs** + +#### GitLab + +Navigate to your Applications menu in GitLab User Settings: https://gitlab.com/profile/applications + +Complete the application creation form as follows: + +- **Name:** codefresh-onprem (or a significant name) +- **Redirect URI:** https://your-codefresh-onprem-domain/api/auth/gitlab/callback +- **Scopes (permissions):** + - API + - read_user + - read_registry + +Click **Save application**. + +After app creation, note down the created Application ID and Client Secret. They will be required for the settings in **Codefresh Admin**->**IDPs**. + +{% include image.html + lightbox="true" + file="/images/installation/git-idp.png" + url="/images/installation/git-idp.png" + %} + +>Note: When configuring the default IDP (for GitHub, GitLab, etc), do not modify the Client Name field. Please keep them as GitHub, GitLab, BitBucket, etc. Otherwise, the signup and login views won’t work. + +### Proxy Configuration + +If your environment resides behind HTTP proxies, you need to uncomment the following section in `config.yaml`: + +```yaml +global: + env: + HTTP_PROXY: "http://myproxy.domain.com:8080" + http_proxy: "http://myproxy.domain.com:8080" + HTTPS_PROXY: "http://myproxy.domain.com:8080" + https_proxy: "http://myproxy.domain.com:8080" + NO_PROXY: "127.0.0.1,localhost,kubernetes.default.svc,.codefresh.svc,100.64.0.1,169.254.169.254,cf-builder,cf-cfapi,cf-cfui,cf-chartmuseum,cf-charts-manager,cf-cluster-providers,cf-consul,cf-consul-ui,cf-context-manager,cf-cronus,cf-helm-repo-manager,cf-hermes,cf-ingress-nginx-controller,cf-kube-integration,cf-mongodb,cf-nats,cf-nomios,cf-pipeline-manager,cf-postgresql,cf-rabbitmq,cf-redis-master,cf-registry,cf-runner,cf-runtime-environment-manager,cf-store" + no_proxy: "127.0.0.1,localhost,kubernetes.default.svc,.codefresh.svc,100.64.0.1,169.254.169.254,cf-builder,cf-cfapi,cf-cfui,cf-chartmuseum,cf-charts-manager,cf-cluster-providers,cf-consul,cf-consul-ui,cf-context-manager,cf-cronus,cf-helm-repo-manager,cf-hermes,cf-ingress-nginx-controller,cf-kube-integration,cf-mongodb,cf-nats,cf-nomios,cf-pipeline-manager,cf-postgresql,cf-rabbitmq,cf-redis-master,cf-registry,cf-runner,cf-runtime-environment-manager,cf-store" +``` +In addition to this, you should also add your Kubernetes API IP address (`kubectl get svc kubernetes`) to both: `NO_PROXY` and `no_proxy`. + +### Storage + +Codefresh is using both cluster storage (volumes) as well as external storage. + +#### Databases + +The following table displays the list of databases created as part of the installation: + +| Database | Purpose | Latest supported version | +|----------|---------| ---------------| +| mongoDB | storing all account data (account settings, users, projects, pipelines, builds etc.) | 4.2.x | +| postgresql | storing data about events that happened on the account (pipeline updates, deletes, etc.). The audit log uses the data from this database. | 13.x | +| redis | mainly used for caching, but also used as a key-value store for our trigger manager. | 6.0.x | + +#### Volumes + +These are the volumes required for Codefresh on-premises: + + +{: .table .table-bordered .table-hover} +| Name | Purpose | Minimum Capacity | Can run on netfs (nfs, cifs) | +|----------------|------------------------|------------------|------------------------------| +| cf-mongodb* | Main database - Mongo | 8GB | Yes** | +| cf-postgresql* | Events databases - Postgres | 8GB | Yes** | +| cf-rabbitmq* | Message broker | 8GB | No** | +| cf-redis* | Cache | 8GB | No** | +| cf-store | Trigger Redis data | 8GB | No** | +| cf-cronus | Trigger crontab data | 1GB | Yes | +| datadir-cf-consul-0 | Consul datadir | 1GB | Yes | +| cf-chartmuseum | chartmuseum | 10GB | Yes | +| cf-builder-0 | /var/lib/docker for builder | 100GB | No*** | +| cf-runner-0 | /var/lib/docker for composition runner | 100GB | No*** | + +{% raw %} + + (*) Possibility to use external service + + (**) Running on netfs (nfs, cifs) is not recommended by product admin guide + + (***) Docker daemon can be run on block device only + +{% endraw %} + +StatefulSets (`cf-builder` and `cf-runner`) process their data on separate physical volumes (PVs) and can be claimed using Persistent Volume Claims (PVCs) with default initial sizes of 100Gi. Also, those StatefulSets have the ability to connect to existing pre-defined PVCs. + +The default initial volume size (100 Gi) can be overridden in the custom `config.yaml` file. Values descriptions are in the `config.yaml` file. +The registry’s initial volume size is 100Gi. It also can be overridden in a custom `config.yaml` file. There is a possibility to use a customer-defined registry configuration file (`config.yaml`) that allows using different registry storage back-ends (S3, Azure Blob, GCS, etc.) and other parameters. More details can be found in the [Docker documentation](https://docs.docker.com/registry/configuration/). + +Depending on the customer’s Kubernetes version we can assist with PV resizing. Details are can be found in this [Kubernetes blog post](https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/). + +#### Automatic Volume Provisioning + +Codefresh installation supports automatic storage provisioning based on the standard Kubernetes dynamic provisioner Storage Classes and Persistent Volume Claims. All required installation volumes will be provisioned automatically using the default Storage Class or custom Storage Class that can be specified as a parameter in `config.yaml` under `storageClass: my-storage-class`. + + + +### Retention policy for Codefresh builds +Define a retention policy to manage Codefresh builds. The retention settings are controlled through `cf-api` deployment environment variables, all of which have default settings which you can retain or customize. By default, Codefresh deletes builds older than six months, including offline logs. + +The retention mechanism, implemented as a Cron Job, removes data from collections such as: +* workflowproccesses +* workflowrequests +* workflowrevisions + +{: .table .table-bordered .table-hover} +| Env Variable | Description | Default | +|---------------|--------------------------- |---------------------- | +|`RETENTION_POLICY_IS_ENABLED` | Determines if automatic build deletion through the Cron job is enabled. | `true` | +|`RETENTION_POLICY_BUILDS_TO_DELETE`| The maximum number of builds to delete by a single Cron job. To avoid database issues, especially when there are large numbers of old builds, we recommend deleting them in small chunks. You can gradually increase the number after verifying that performance is not affected. | `50` | +|`RETENTION_POLICY_DAYS` | The number of days for which to retain builds. Builds older than the defined retention period are deleted. | `180` | +|`RUNTIME_MONGO_URI` | Optional. The URI of the Mongo database from which to remove MongoDB logs (in addition to the builds). | | + + +### Managing Codefresh backups + +Codefresh on-premises backups can be automated by installing a specific service as an addon to your Codefresh on-premises installation. It is based on the [mgob](https://github.com/stefanprodan/mgob){:target="\_blank"} open source project, and can run scheduled backups with retention, S3 & SFTP upload, notifications, instrumentation with Prometheus and more. + +#### Configure and deploy the Backup Manager + +Backup Manager is installed as an addon and therefore it needs an existing Codefresh on-premises installation. +Before installing it, please make sure you have selected a proper kube config pointing to the cluster, where you have Codefresh installed on. + +1. Go to the staging directory of your Codefresh installation, and open the config file: `your-CF-stage-dir/addons/backup-manager/config.yaml`. +1. Retain or customize the values of these configuration parameters: + * `metadada`: Various CF-installer-specific parameters, which should not be changed in this case + * `kubernetes`: Specify a kube context, kube config file, and a namespace for the backup manager + * `storage`: Storage class, storage size and read modes for persistent volumes to store backups locally within your cluster + * Backup plan configuration parameters under `jobConfigs.cfBackupPlan`: + * `target.uri` - target mongo URI. It is recommended to leave the mongo uri value blank - it will be taken automatically from the Codefresh release installed in your cluster + * `scheduler` - here you can specify cron expression for your backups schedule, backups retention and timeout values + +For more advanced backup plan settings, such as specifying various remote cloud-based storage providers for your backups, configuring notifications and other, please refer to [this](https://github.com/stefanprodan/mgob#configure) page + +To **deploy the backup manager** service, please select a correct kube context, where you have Codefresh on-premises installed and deploy backup-manager with the following command: + +``` +kcfi deploy -c `your-CF-stage-dir/addons/backup-manager/config.yaml` +``` + +#### On-demand/ad-hoc backup +``` +kubectl port-forward cf-backup-manager-0 8090 +curl -X POST http://localhost:8090/backup/cfBackupPlan +``` + +#### Restore from backup +``` +kubectl exec -it cf-backup-manager-0 bash +mongorestore --gzip --archive=/storage/cfBackupPlan/backup-archive-name.gz --uri mongodb://root:password@mongodb:27017 --drop +``` + +### Configuring AWS Load Balancers + +By default Codefresh deploys the [ingress-nginx](https://github.com/kubernetes/ingress-nginx/) controller and [Classic Load Balancer](https://docs.aws.amazon.com/eks/latest/userguide/load-balancing.html) as a controller service. + +#### NLB + +To use a **Network Load Balancer** - deploy a regular Codefresh installation with the following ingress config for the the `cf-ingress-controller` controller service. + +`config.yaml` +```yaml +ingress-nginx: + controller: + service: + annotations: + service.beta.kubernetes.io/aws-load-balancer-type: nlb + service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp + service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '60' + service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true' + +tls: + selfSigned: false + cert: certs/certificate.crt + key: certs/private.key +``` +This annotation will create a new Load Balancer - Network Load Balancer, which you should use in the Codefresh UI DNS record. +Update the DNS record according to the new service. + +#### L7 ELB with SSL Termination + +When a **Classic Load Balancer** is used, some Codefresh features that (for example `OfflineLogging`), will use a websocket to connect with Codefresh API and they will require secure TCP (SSL) protocol enabled on the Load Balancer listener instead of HTTPS. + +To use either a certificate from a third party issuer that was uploaded to IAM or a certificate [requested](https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-public.html) within AWS Certificate Manager see the followning config example: + + +`config.yaml` +```yaml +ingress-nginx: + controller: + service: + annotations: + service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp" + service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443" + service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '3600' + service.beta.kubernetes.io/aws-load-balancer-ssl-cert: < CERTIFICATE ARN > + targetPorts: + http: http + https: http + +tls: + selfSigned: true +``` + +- both http and https target port should be set to **80**. +- update your AWS Load Balancer listener for port 443 from HTTPS protocol to SSL. + +#### ALB + +To use the **Application Load Balancer** the [ALB Ingress Controller](https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html) should be deployed to the cluster. + +To support ALB: + +- First disable Nginx controller in the Codefresh init config file - __config.yaml__: + +```yaml +ingress-nginx: #disables creation of Nginx controller deployment + enabled: false + +ingress: #disables creation of Ingress object + enabled: false +``` + +- [deploy](https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html) the ALB controller; +- create a new **ingress** resource: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + annotations: + alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]' + alb.ingress.kubernetes.io/scheme: internet-facing + alb.ingress.kubernetes.io/target-type: ip + kubernetes.io/ingress.class: alb + meta.helm.sh/release-name: cf + meta.helm.sh/release-namespace: codefresh + labels: + app: cf-codefresh + release: cf + name: cf-codefresh-ingress + namespace: codefresh +spec: + defaultBackend: + service: + name: cf-cfui + port: + number: 80 + rules: + - host: myonprem.domain.com + http: + paths: + - backend: + service: + name: cf-cfapi + port: + number: 80 + path: /api/* + pathType: ImplementationSpecific + - backend: + service: + name: cf-cfapi + port: + number: 80 + path: /ws/* + pathType: ImplementationSpecific + - backend: + service: + name: cf-cfui + port: + number: 80 + path: / + pathType: ImplementationSpecific +``` + +### Configure CSP (Content Security Policy) +Add CSP environment variables to `config.yaml`, and define the values to be returned in the CSP HTTP headers. +```yaml +cfui: + env: + CONTENT_SECURITY_POLICY: " " + CONTENT_SECURITY_POLICY_REPORT_ONLY: "default-src 'self'; font-src 'self' + https://fonts.gstatic.com; script-src 'self' https://unpkg.com https://js.stripe.com; + style-src 'self' https://fonts.googleapis.com; 'unsafe-eval' 'unsafe-inline'" + CONTENT_SECURITY_POLICY_REPORT_TO: " - "
+```
+`CONTENT_SECURITY_POLICY` is the string describing content policies. Use semi-colons to separate between policies.
+`CONTENT_SECURITY_POLICY_REPORT_TO` is a comma-separated list of JSON objects. Each object must have a name and an array of endpoints that receive the incoming CSP reports.
+
+For detailed information, see the [Content Security Policy article on MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP).
+
+### Enable x-hub-signature-256 signature for GitHub AE
+Add the `USE_SHA256_GITHUB_SIGNATURE` environment variable to **cfapi** deployment in `config.yaml`.
+```yaml
+cfapi:
+ env:
+ USE_SHA256_GITHUB_SIGNATURE: "true"
+```
+
+For detailed information, see the [Securing your webhooks](https://docs.github.com/en/developers/webhooks-and-events/webhooks/securing-your-webhooks) and [Webhooks](https://docs.github.com/en/github-ae@latest/rest/webhooks).
+
+
+## Using existing external services for data storage/messaging
+
+Normally the Codefresh installer, is taking care of all needed dependencies internally by deploying the respective services (mongo, redis etc) on its own.
+
+You might want however to use your own existing options if you already have those services up and running externally.
+
+### Configuring an external Postgres database
+
+It is possible to configure Codefresh to work with your existing Postgres database service, if you don't want to use the default one as provided by the Codefresh installer.
+
+#### Configuration steps
+
+All the configuration comes down to putting a set of correct values into your Codefresh configuration file `config.yaml`, which is present in `your/stage-dir/codefresh` directory. During the installation, Codefresh will run a seed job, using the values described in the following steps:
+
+1. Specify a user name `global.postgresSeedJob.user` and password `global.postgresSeedJob.password` for a seed job. This must be a privileged user allowed to create databases and roles. It will be used only by the seed job to create the needed database and a user.
+2. Specify a user name `global.postgresUser` and password `global.postgresPassword` to be used by Codefresh installation. A user with the name and password will be created by the seed job and granted with required privileges to access the created database.
+3. Specify a database name `global.postgresDatabase` to be created by the seed job and used by Codefresh installation.
+4. Specify `global.postgresHostname` and optionally `global.postgresPort` (`5432` is a default value).
+5. Disable the postgres subchart installation with the `postgresql.enabled: false` value, because it is not needed in this case.
+
+
+Below is an example of the relevant piece of `config.yaml`:
+
+```yaml
+global:
+ postgresSeedJob:
+ user: postgres
+ password: zDyGp79XyZEqLq7V
+ postgresUser: cf_user
+ postgresPassword: fJTFJMGV7sg5E4Bj
+ postgresDatabase: codefresh
+ postgresHostname: my-postgres.ccjog7pqzunf.us-west-2.rds.amazonaws.com
+ postgresPort: 5432
+
+postgresql:
+ enabled: false #disable default postgresql subchart installation
+```
+#### Running the seed job manually
+
+If you prefer running the seed job manually, you can do it by using a script present in `your/stage-dir/codefresh/addons/seed-scripts` directory named `postgres-seed.sh`. The script takes the following set of variables that you need to have set before running it:
+
+```shell
+export POSTGRES_SEED_USER="postgres"
+export POSTGRES_SEED_PASSWORD="zDyGp79XyZEqLq7V"
+export POSTGRES_USER="cf_user"
+export POSTGRES_PASSWORD="fJTFJMGV7sg5E4Bj"
+export POSTGRES_DATABASE="codefresh"
+export POSTGRES_HOST="my-postgres.ccjog7pqzunf.us-west-2.rds.amazonaws.com"
+export POSTGRES_PORT="5432"
+```
+The variables have the same meaning as the configuration values described in the previous section about Postgres.
+
+However you **still need to specify a set of values** in the Codefresh config file as described in the section above, but with the whole **`postgresSeedJob` section omitted**, like this:
+
+```yaml
+global:
+ postgresUser: cf_user
+ postgresPassword: fJTFJMGV7sg5E4Bj
+ postgresDatabase: codefresh
+ postgresHostname: my-postgresql.prod.svc.cluster.local
+ postgresPort: 5432
+
+postgresql:
+ enabled: false #disable default postgresql subchart installation
+```
+
+### Configuring an external MongoDB
+
+Codefresh recommends to use the Bitnami MongoDB [chart](https://github.com/bitnami/charts/tree/master/bitnami/mongodb) as a Mongo database. The supported version of Mongo is 4.2.x
+
+To configure Codefresh on-premises to use an external Mongo service one needs to provide the following values in `config.yaml`:
+
+- **mongo connection string** - `mongoURI`. This string will be used by all of the services to communicate with mongo. Codefresh will automatically create and add a user with "ReadWrite" permissions to all of the created databases with the username and password from the URI. Optionally, automatic user addition can be disabled - `mongoSkipUserCreation`, in order to use already existing user. In such a case the existing user must have **ReadWrite** permissions to all of newly created databases
+Codefresh does not support [DNS Seedlist Connection Format](https://docs.mongodb.com/manual/reference/connection-string/#connections-dns-seedlist) at the moment, use the [Standard Connection Format](https://docs.mongodb.com/manual/reference/connection-string/#connections-standard-connection-string-format) instead.
+- mongo **root user** name and **password** - `mongodbRootUser`, `mongodbRootPassword`. The privileged user will be used by Codefresh only during installation for seed jobs and for automatic user addition. After installation, credentials from the provided mongo URI will be used. Mongo root user must have permissions to create users.
+
+See the [Mongo required Access](https://docs.mongodb.com/manual/reference/method/db.createUser/#required-access) for more details.
+
+Here is an example of all the related values:
+
+```yaml
+global:
+ mongodbRootUser:
+ mongodbRootPassword: + mongoURI: + mongoSkipUserCreation: true + mongoDeploy: false # disables deployment of internal mongo service + +mongo: + enabled: false + ``` + +#### MongoDB with Mutual TLS + +>The option available in kcfi **v0.5.10** + +Codefresh supports enabling SSL/TLS between cf microservices and MongoDB. To enable this option specify in `config.yaml` the following parameters: + + `global.mongoTLS: true`
+ `global.mongoCaCert` - CA certificate file path (in kcfi init directory)
+ `global.mongoCaKey` - CA certificate private key file path (in kcfi init directory) + +`config.yaml` example: +```yaml +global: + mongodbRootUser: root + mongodbRootPassword: WOIqcSwr0y + mongoURI: mongodb://my-mongodb.prod.svc.cluster.local/?ssl=true&authMechanism=MONGODB-X509&authSource=$external + mongoSkipUserCreation: true + mongoDeploy: false # disables deployment of internal mongo service + + mongoTLS: true #enable MongoDB TLS support + mongoCaCert: mongodb-ca/ca-cert.pem + mongoCaKey: mongodb-ca/ca-key.pem + + ### for OfflineLogging feature + runtimeMongoURI: mongodb://my-mongodb.prod.svc.cluster.local/?ssl=true&authMechanism=MONGODB-X509&authSource=$external + +### for OfflineLogging feature +cfapi: + env: + RUNTIME_MONGO_TLS: "true" + RUNTIME_MONGO_TLS_VALIDATE: "true" # 'false' if self-signed certificate to avoid x509 errors + +## set MONGO_MTLS_VALIDATE to `false` if self-signed certificate to avoid x509 errors +cluster-providers: + env: + MONGO_MTLS_VALIDATE: "false" + +k8s-monitor: + env: + MONGO_MTLS_VALIDATE: "false" + +mongo: + enabled: false #disable default mongodb subchart installation + ``` + + >Perform an upgarde:
+ >`kcfi deploy -c config.yaml --debug` + +### Configure an external Redis service +Codefresh recommends to use the Bitnami Redis [chart](https://github.com/bitnami/charts/tree/master/bitnami/redis) as a Redis store. + +**Limitations** + +Codefresh does not support secure connection to Redis (TLS) and AUTH username extension. + +**Configuration** + +To configure Codefresh to use an external Redis service, add the following parameters to your `config.yaml`: + +`config.yaml` example: +```yaml +global: + redisUrl: my-redis.prod.svc.cluster.local + redisPort: 6379 + redisPassword: 6oOhHI8fI5 + + runtimeRedisHost: my-redis.prod.svc.cluster.local + runtimeRedisPassword: 6oOhHI8fI5 + runtimeRedisPort: 6379 + runtimeRedisDb: 2 + +redis: + enabled: false #disable default redis subchart installation +``` + +Where `redis*` - are for the main Redis storage, and `runtimeRedis*` - for storage is used to store pipeline logs in case of `OfflineLogging` feature is turned on. In most cases the host value is the same for these two values. + + +### Configuring an external RabbitMQ service + +Codefresh recommends to use the Bitnami RabbitMQ [chart](https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq) as a RabbitMQ service. + +To use an external RabbitMQ service instead of the local helm chart, add the following values to the __config.yaml__: + +```yaml +rabbitmq: + enabled: false + +global: + rabbitmqUsername:+ rabbitmqPassword: + rabbitmqHostname: +``` + +### Configuring an external Consul service + + +Notice that at the moment Codefresh supports only the deprecated Consul API (image __consul:1.0.0__), and does not support connection via HTTPS and any authentication. +The Consul host must expose port `8500`. + +>In general, we don't recommend to take the Consul service outside the cluster. + + +To configure Codefresh to use your external Consul service, add the following values to the __config.yaml__: + +```yaml +global: + consulHost: + +consul: + enabled: false +``` + +## App Cluster Autoscaling + +Autoscaling in Kubernetes is implemented as an interaction between Cluster Autoscaler and Horizontal Pod Autoscaler + +{: .table .table-bordered .table-hover} +| | Scaling Target| Trigger | Controller | How it Works | +| ----------- | ------------- | ------- | --------- | --------- | +| [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler)| Nodes | **Up:** Pending pod
**Down:** Node resource allocations is low | On GKE we can turn on/off autoscaler and configure min/max per node group can be also installed separately | Listens on pending pods for scale up and node allocations for scaledown. Should have permissions to call cloud api. Considers pod affinity, pdb, storage, special annotations | +| [Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) | replicas on deployments or StatefulSets | metrics value thresholds defined in HPA object | part of Kubernetes controller | Controller gets metrics from "metrics.k8s.io/v1beta1" , "custom.metrics.k8s.io/v1beta1", "external.metrics.k8s.io/v1beta1" requires [metrics-server](https://github.com/kubernetes-sigs/metrics-server) and custom metrics adapters ([prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter), [stackdriver-adapter](https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/custom-metrics-stackdriver-adapter)) to listen on this API (see note (1) below) and adjusts deployment or sts replicas according to definitions in HorizontalPodAutocaler
There are v1 and beta api versions for HorizontalPodAutocaler:
[v1](https://github.com/kubernetes/api/blob/master/autoscaling/v1/types.go) - supports for resource metrics (cpu, memory) - `kubect get hpa`
[v2beta2](https://github.com/kubernetes/api/blob/master/autoscaling/v2beta2/types.go) and [v2beta1](https://github.com/kubernetes/api/blob/master/autoscaling/v2beta1/types.go) - supports for both resource and custom metrics - `kubectl get hpa.v2beta2.autoscaling` **The metric value should decrease on adding new pods.**
*Wrong metrics Example:* request rate
*Right metrics Example:* average request rate per pod | + +Note (1) +``` +kubectl get apiservices | awk 'NR==1 || $1 ~ "metrics"' +NAME SERVICE AVAILABLE AGE +v1beta1.custom.metrics.k8s.io monitoring/prom-adapter-prometheus-adapter True 60d +v1beta1.metrics.k8s.io kube-system/metrics-server True 84d +``` + + +**Implementation in Codefresh** + +* Default “Enable Autoscaling” settings for GKE +* Using [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter) with custom metrics + +We define HPA for cfapi and pipeline-manager services + +**CFapi HPA object** + +It's based on three metrics (HPA controller scales of only one of the targetValue reached): + +``` +kubectl get hpa.v2beta1.autoscaling cf-cfapi -oyaml +``` + +{% highlight yaml %} +{% raw %} +apiVersion: autoscaling/v2beta1 +kind: HorizontalPodAutoscaler +metadata: + annotations: + meta.helm.sh/release-name: cf + meta.helm.sh/release-namespace: default + labels: + app.kubernetes.io/managed-by: Helm + name: cf-cfapi + namespace: default +spec: + maxReplicas: 16 + metrics: + - object: + metricName: requests_per_pod + target: + apiVersion: v1 + kind: Service + name: cf-cfapi + targetValue: "10" + type: Object + - object: + metricName: cpu_usage_avg + target: + apiVersion: apps/v1 + kind: Deployment + name: cf-cfapi-base + targetValue: "1" + type: Object + - object: + metricName: memory_working_set_bytes_avg + target: + apiVersion: apps/v1 + kind: Deployment + name: cf-cfapi-base + targetValue: 3G + type: Object + minReplicas: 2 + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: cf-cfapi-base +{% endraw%} +{% endhighlight %} + +* `requests_per_pod` is based on `rate(nginx_ingress_controller_requests)` metric ingested from nginx-ingress-controller +* `cpu_usage_avg` based on cadvisor (from kubelet) rate `(rate(container_cpu_user_seconds_total)` +* `memory_working_set_bytes_avg` based on cadvisor `container_memory_working_set_bytes` + +**pipeline-manager HPA** + +based on `cpu_usage_avg` + +{% highlight yaml %} +{% raw %} +apiVersion: autoscaling/v2beta1 +kind: HorizontalPodAutoscaler +metadata: + annotations: + meta.helm.sh/release-name: cf + meta.helm.sh/release-namespace: default + labels: + app.kubernetes.io/managed-by: Helm + name: cf-pipeline-manager +spec: + maxReplicas: 8 + metrics: + - object: + metricName: cpu_usage_avg + target: + apiVersion: apps/v1 + kind: Deployment + name: cf-pipeline-manager-base + targetValue: 400m + type: Object + minReplicas: 2 + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: cf-pipeline-manager-base +{% endraw%} +{% endhighlight %} + +**prometheus-adapter configuration** + +Reference: [https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/config.md](https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/config.md +) + +{% highlight yaml %} +{% raw %} +Rules: + - metricsQuery: | + kube_service_info{<<.LabelMatchers>>} * on() group_right(service) + (sum(rate(nginx_ingress_controller_requests{<<.LabelMatchers>>}[2m])) + / on() kube_deployment_spec_replicas{deployment='<>-base',namespace='< >'}) + name: + as: requests_per_pod + matches: ^(.*)$ + resources: + overrides: + namespace: + resource: namespace + service: + resource: service + seriesQuery: kube_service_info{service=~".*cfapi.*"} + - metricsQuery: | + kube_deployment_labels{<<.LabelMatchers>>} * on(label_app) group_right(deployment) + (label_replace( + avg by (container) (rate(container_cpu_user_seconds_total{container=~"cf-(tasker-kubernetes|cfapi.*|pipeline-manager.*)", job="kubelet", namespace='< >'}[15m])) + , "label_app", "$1", "container", "(.*)")) + name: + as: cpu_usage_avg + matches: ^(.*)$ + resources: + overrides: + deployment: + group: apps + resource: deployment + namespace: + resource: namespace + seriesQuery: kube_deployment_labels{label_app=~"cf-(tasker-kubernetes|cfapi.*|pipeline-manager.*)"} + - metricsQuery: "kube_deployment_labels{<<.LabelMatchers>>} * on(label_app) group_right(deployment)\n + \ (label_replace(\n avg by (container) (avg_over_time (container_memory_working_set_bytes{container=~\"cf-.*\", + job=\"kubelet\", namespace='< >'}[15m]))\n + \ , \"label_app\", \"$1\", \"container\", \"(.*)\"))\n \n" + name: + as: memory_working_set_bytes_avg + matches: ^(.*)$ + resources: + overrides: + deployment: + group: apps + resource: deployment + namespace: + resource: namespace + seriesQuery: kube_deployment_labels{label_app=~"cf-.*"} + - metricsQuery: | + kube_deployment_labels{<<.LabelMatchers>>} * on(label_app) group_right(deployment) + label_replace(label_replace(avg_over_time(newrelic_apdex_score[15m]), "label_app", "cf-$1", "exported_app", '(cf-api.*|pipeline-manager|tasker-kuberentes)\\[kubernetes\\]'), "label_app", "$1cfapi$3", "label_app", '(cf-)(cf-api)(.*)') + name: + as: newrelic_apdex + matches: ^(.*)$ + resources: + overrides: + deployment: + group: apps + resource: deployment + namespace: + resource: namespace + seriesQuery: kube_deployment_labels{label_app=~"cf-(tasker-kubernetes|cfapi.*|pipeline-manager)"} +{% endraw%} +{% endhighlight %} + +**How to define HPA in Codefresh installer (kcfi) config** + +Most of Codefresh's Microservices subcharts contain `templates/hpa.yaml`: + +{% highlight yaml %} +{% raw %} +{{- if .Values.HorizontalPodAutoscaler }} +apiVersion: autoscaling/v2beta1 +kind: HorizontalPodAutoscaler +metadata: + name: {{ template "cfapi.fullname" . }} +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: {{ template "cfapi.fullname" . }}-{{ .version | default "base" }} + minReplicas: {{ coalesce .Values.HorizontalPodAutoscaler.minReplicas .Values.replicaCount 1 }} + maxReplicas: {{ coalesce .Values.HorizontalPodAutoscaler.maxReplicas .Values.replicaCount 2 }} + metrics: +{{- if .Values.HorizontalPodAutoscaler.metrics }} +{{ toYaml .Values.HorizontalPodAutoscaler.metrics | indent 4 }} +{{- else }} + - type: Resource + resource: + name: cpu + targetAverageUtilization: 60 +{{- end }} +{{- end }} +{% endraw%} +{% endhighlight %} + +To configure HPA for CFapi add `HorizontalPodAutoscaler` values to config.yaml, for example: + +(assuming that we already have prometheus adapter configured for metrics `requests_per_pod`, `cpu_usage_avg`, `memory_working_set_bytes_avg`) + +{% highlight yaml %} +{% raw %} +cfapi: + replicaCount: 4 + resources: + requests: + memory: "4096Mi" + cpu: "1100m" + limits: + memory: "4096Mi" + cpu: "2200m" + HorizontalPodAutoscaler: + minReplicas: 2 + maxReplicas: 16 + metrics: + - type: Object + object: + metricName: requests_per_pod + target: + apiVersion: "v1" + kind: Service + name: cf-cfapi + targetValue: 10 + - type: Object + object: + metricName: cpu_usage_avg + target: + apiVersion: "apps/v1" + kind: Deployment + name: cf-cfapi-base + targetValue: 1 + - type: Object + object: + metricName: memory_working_set_bytes_avg + target: + apiVersion: "apps/v1" + kind: Deployment + name: cf-cfapi-base + targetValue: 3G +{% endraw%} +{% endhighlight %} + +**Querying metrics (for debugging)** + +CPU Metric API Call + +``` +kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/codefresh/pods/cf-cfapi-base-****-/ | jq +``` + +Custom Metrics Call + +``` +kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/codefresh/services/cf-cfapi/requests_per_pod | jq +``` + + +## Common Problems, Solutions, and Dependencies + +### Dependencies + +#### Mongo + +All services using the MongoDB are dependent on the `mongo` pod being up and running. If the `mongo` pod is down, the following dependencies will not work: + +- `runtime-environment-manager` +- `pipeline-manager` +- `cf-api` +- `cf-broadcaster` +- `context-manager` +- `nomios` +- `cronius` +- `cluster-promoters` +- `k8s-monitor` +- `charts-manager` +- `tasker-kubernetes` + +#### Logs + +There is a dependency between the `cf-broadcaster` pod and the `cf-api` pod. If your pipeline runs, but does not show any logs, try restarting the broadcaster pod. + +### Problems and Solutions + +**Problem:** installer fails because `codefresh` database does not exist. + +**Solution:** If you are using an external PostgresSQL database (instead of the internal one that the installer provides), you will first need to manually create a new database named `codefresh` inside your PostgresSQL database before running the installer. + + diff --git a/_docs/installation/codefresh-runner.md b/_docs/installation/codefresh-runner.md new file mode 100644 index 00000000..a5fde797 --- /dev/null +++ b/_docs/installation/codefresh-runner.md @@ -0,0 +1,2072 @@ +--- +title: "Codefresh Runner installation" +description: "Run Codefresh pipelines on your private Kubernetes cluster" +group: installation +redirect_from: + - /docs/enterprise/codefresh-runner/ +toc: true +--- + +Install the Codefresh Runner on your Kubernetes cluster to run pipelines and access secure internal services without compromising on-premises security requirements. These pipelines run on your infrastructure, even behind the firewall, and keep code on your Kubernetes cluster secure. + +[Skip to quick installation →](#installation-with-the-quick-start-wizard) + +>Important: + You must install the Codefresh Runner on _each cluster running Codefresh pipelines_. + The Runner is **not** needed in clusters used for _deployment_. You can deploy applications on clusters other than the ones the runner is deployed on. + +The installation process takes care of all Runner components and other required resources (config-maps, secrets, volumes). + +## Prerequisites + +To use the Codefresh runner the following is required: + +1. A Kubernetes cluster with outgoing internet access (versions 1.10 to 1.23). Each node should have 50GB disk size. +2. A container runtime, such as [docker](https://kubernetes.io/blog/2020/12/02/dockershim-faq/), [containerd](https://containerd.io/) or [cri-o](https://cri-o.io/). Note that the runner is **not** dependent on any special dockershim features, so any compliant container runtime is acceptable. The docker socket/daemon used by Codefresh pipelines is **NOT** the one on the host node (as it might not exist at all in the case of containerd or cri-o), but instead an internal docker daemon created/managed by the pipeline itself. +3. A [Codefresh account]({{site.baseurl}}/docs/getting-started/create-a-codefresh-account/) with the Hybrid feature enabled. +4. A [Codefresh CLI token]({{site.baseurl}}/docs/integrations/codefresh-api/#authentication-instructions) that will be used to authenticate your Codefresh account. + +The runner can be installed from any workstation or laptop with access (i.e. via `kubectl`) to the Kubernetes cluster running Codefresh builds. The Codefresh runner will authenticate to your Codefresh account by using the Codefresh CLI token. + +## System Requirements + +Once installed the runner uses the following pods: + +* `runner` - responsible for picking tasks (builds) from the Codefresh API +* `engine` - responsible for running pipelines +* `dind` - responsible for building and using Docker images +* `dind-volume-provisioner` - responsible for provisioning volumes (PV) for dind +* `dind-lv-monitor` - responsible for cleaning **local** volumes + +**CPU/Memory** + +The following table shows **MINIMUM** resources for each component: + +{: .table .table-bordered .table-hover} +| Component | CPU requests| RAM requests | Storage | Type | Always on | +| -------------- | --------------|------------- |-------------------------|-------|-------| +| `runner` | 100m | 100Mi | Doesn't need PV | Deployment | Yes | +| `engine` | 100m | 500Mi | Doesn't need PV | Pod | No | +| `dind` | 400m | 800Mi | 16GB PV | Pod | No | +| `dind-volume-provisioner` | 300m | 400Mi | Doesn't need PV | Deployment | Yes | +| `dind-lv-monitor` | 300m | 400Mi | Doesn't need PV | DaemonSet | Yes | + +Components that are always on consume resources all the time. Components that are not always on only consume resources when pipelines are running (they are created and destroyed automatically for each pipeline). + +Node size and count will depend entirely on how many pipelines you want to be “ready” for and how many will use “burst” capacity. + +* Ready (nodes): Lower initialization time and faster build times. +* Burst (nodes): High initialization time and slower build times. (Not recommended) + +The size of your nodes directly relates to the size required for your pipelines and thus it is dynamic. If you find that only a few larger pipelines require larger nodes you may want to have two Codefresh Runners associated to different node pools. + + +**Storage** + +For the storage options needed by the `dind` pod we suggest: + +* [Local Volumes](https://kubernetes.io/docs/concepts/storage/volumes/#local) `/var/lib/codefresh/dind-volumes` on the K8S nodes filesystem (**default**) +* [EBS](https://aws.amazon.com/ebs/) in the case of AWS. See also the [notes](#installing-on-aws) about getting caching working. +* [Local SSD](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd) or [GCE Disks](https://cloud.google.com/compute/docs/disks#pdspecs) in the case of GCP. See [notes](#installing-on-google-kubernetes-engine) about configuration. + + +**Networking Requirements** + +* `dind` - this pod will create an internal network in the cluster to run all the pipeline steps; needs outgoing/egress access to Dockerhub and `quay.io` +* `runner` - this pod needs outgoing/egress access to `g.codefresh.io`; needs network access to [app-proxy]({{site.baseurl}}/docs/administration/codefresh-runner/#optional-installation-of-the-app-proxy) (if app-proxy is used) +* `engine` - this pod needs outgoing/egress access to `g.codefresh.io`, `*.firebaseio.com` and `quay.io`; needs network access to `dind` pod + +All CNI providers/plugins are compatible with the runner components. + +## Installation with the Quick-start Wizard + +Install the Codefresh CLI + +```shell +npm install -g codefresh +``` + +[Alternative install methods](https://codefresh-io.github.io/cli/installation/) + +Authenticate the CLI + +```shell +codefresh auth create-context --api-key {API_KEY} +``` + +You can obtain an API Key from your [user settings page](https://g.codefresh.io/user/settings). +>**Note:** Make sure when you generate the token used to authenticate with the CLI, you generate it with *all scopes*. + +>**Note:** access to the Codefresh CLI is only needed once during the Runner installation. After that, the Runner will authenticate on it own using the details provided. You do NOT need to install the Codefresh CLI on the cluster that is running Codefresh pipelines. + +Then run the wizard with the following command: + +```shell +codefresh runner init +``` + +or + +```shell +codefresh runner init --token +``` + +Brefore proceeding with installation, the wizard asks you some basic questions. + +{% include image.html + lightbox="true" + file="/images/administration/runner/installation-wizard.png" + url="/images/administration/runner/installation-wizard.png" + alt="Codefresh Runner wizard" + caption="Codefresh Runner wizard" + max-width="100%" + %} + +The wizard also creates and runs a sample pipeline that you can see in your Codefresh UI. + +{% include image.html + lightbox="true" + file="/images/administration/runner/sample-pipeline.png" + url="/images/administration/runner/sample-pipeline.png" + alt="Codefresh Runner example pipeline" + caption="Codefresh Runner example pipeline" + max-width="90%" + %} + +That's it! You can now start using the Runner. + +You can also verify your installation with: + +```shell +codefresh runner info +``` + +During installation you can see which API token will be used by the runner (if you don't provide one). The printed token is used by the runner to talk to the Codefresh platform carrying permissions that allow the runner to run pipelines. If you save the token, it can later be used to restore the runner's permissions without creating a new runner installation, if the deployment is deleted. + +**Customizing the Wizard Installation** + +You can customize the wizard installation by passing your own values in the `init` command. +To inspect all available options run `init` with the `--help` flag: + +```shell +codefresh runner init --help +``` + +**Inspecting the Manifests Before they are Installed** + +If you want to see what manifests are used by the installation wizard you can supply the `--dry-run` parameter in the installation process. + +```shell +codefresh runner init --dry-run +``` + +This will execute the wizard in a special mode that will not actually install anything in your cluster. After all configuration questions are asked, all Kubernetes manifests used by the installer will be instead saved locally in a folder `./codefresh_manifests`. + +## Install Codefresh Runner with values file + +To install the Codefresh Runner with pre-defined values file use `--values` flag: + +```shell +codefresh runner init --values values.yaml +``` + +Use [this example](https://github.com/codefresh-io/venona/blob/release-1.0/venonactl/example/values-example.yaml) as a starting point for your values file. + +## Install Codefresh Runner with Helm + +To install the Codefresh Runner using Helm, follow these steps: + +1. Download the Codefresh CLI and authenticate it with your Codefresh account. Click [here](https://codefresh-io.github.io/cli/getting-started/) for more detailed instructions. +2. Run the following command to create all of the necessary entities in Codefresh: + + ```shell + codefresh runner init --generate-helm-values-file + ``` + + * This will not install anything on your cluster, except for running cluster acceptance tests, (which may be skipped using the `--skip-cluster-test` option). Please note, that the Runner Agent and the Runtime Environment are still created in your Codefresh account. + * This command will also generate a `generated_values.yaml` file in your current directory, which you will need to provide to the `helm install` command later. If you want to install several Codefresh Runners, you will need a separate `generated_values.yaml` file for each Runner. + +3. Now run the following to complete the installation: + + ```shell + helm repo add cf-runtime https://chartmuseum.codefresh.io/cf-runtime + + helm install cf-runtime cf-runtime/cf-runtime -f ./generated_values.yaml --create-namespace --namespace codefresh + ``` + * Here is the link to a repository with the chart for reference: [https://github.com/codefresh-io/venona/tree/release-1.0/.deploy/cf-runtime](https://github.com/codefresh-io/venona/tree/release-1.0/.deploy/cf-runtime) + +4. At this point you should have a working Codefresh Runner. You can verify the installation by running: + + ```shell + codefresh runner execute-test-pipeline --runtime-name + ``` +>**Note!**
+Runtime components' (engine and dind) configuration is determined by the `runner init` command.
+The `helm install` command can only control the configuration of `runner`, `dind-volume-provisioner` and `lv-monitor` components. + +## Using the Codefresh Runner + +Once installed, the Runner is fully automated. It polls the Codefresh SAAS (by default every 3 seconds) on its own and automatically creates all resources needed for running pipelines. + +Once installation is complete, you should see the cluster of the runner as a new [Runtime environment](https://g.codefresh.io/account-admin/account-conf/runtime-environments) in Codefresh in your *Account Settings*, in the respective tab. + +{% include image.html + lightbox="true" + file="/images/administration/runner/runtime-environments.png" + url="/images/administration/runner/runtime-environments.png" + alt="Available runtime environments" + caption="Available runtime environments" + max-width="60%" + %} + +If you have multiple environments available, you can change the default (shown with a thin blue border) by clicking on the 3 dot menu on the right of each environment. The Codefresh runner installer comes with a `set-default` option that is automatically set by default in the new runtime environment. + +You can even override the runtime environment for a specific pipeline by specifying in the respective section in the [pipeline settings]({{site.baseurl}}/docs/configure-ci-cd-pipeline/pipelines/). + +{% include image.html + lightbox="true" + file="/images/administration/runner/environment-per-pipeline.png" + url="/images/administration/runner/environment-per-pipeline.png" + alt="Running a pipeline on a specific environment" + caption="Running a pipeline on a specific environment" + max-width="60%" + %} + +## Checking the Runner + +Once installed, the runner is a normal Kubernetes application like all other applications. You can use your existing tools to monitor it. + +Only the runner pod is long living inside your cluster. All other components (such as the engine) are short lived and exist only during pipeline builds. +You can always see what the Runner is doing by listing the resources inside the namespace you chose during installation: + +```shell +$ kubectl get pods -n codefresh-runtime +NAME READY STATUS RESTARTS AGE +dind-5ee7577017ef40908b784388 1/1 Running 0 22s +dind-lv-monitor-runner-hn64g 1/1 Running 0 3d +dind-lv-monitor-runner-pj84r 1/1 Running 0 3d +dind-lv-monitor-runner-v2lhc 1/1 Running 0 3d +dind-volume-provisioner-runner-64994bbb84-lgg7v 1/1 Running 0 3d +engine-5ee7577017ef40908b784388 1/1 Running 0 22s +monitor-648b4778bd-tvzcr 1/1 Running 0 3d +runner-5d549f8bc5-7h5rc 1/1 Running 0 3d +``` + +In the same manner you can list secrets, config-maps, logs, volumes etc. for the Codefresh builds. + +## Uninstall the Codefresh Runner + +You can uninstall the Codefresh runner from your cluster by running: + +```shell +codefresh runner delete +``` + +A wizard, similar to the installation wizard, will ask you questions regarding your cluster before finishing with the removal. + +Like the installation wizard, you can pass the additional options in advance as command line parameters (see `--help` output): +```shell +codefresh runner delete --help +``` + + + +## Runner architecture overview + +{% include image.html + lightbox="true" + file="/images/administration/runner/codefresh_runner.png" + url="/images/administration/runner/codefresh_runner.png" + alt="Codefresh Runner architecture overview" + caption="Codefresh Runner architecture overview" + max-width="100%" + %} + + +1. [Runtime-Environment specification]({{site.baseurl}}/docs/administration/codefresh-runner/) defines engine and dind pods spec and PVC parameters. +2. Runner pod (Agent) pulls tasks (Builds) from Codefresh API every 3 seconds. +3. Once the agent receives build task (either Manual run build or Webhook triggered build) it calls k8s API to create engine/dind pods and PVC object. +4. Volume Provisioner listens for PVC events (create) and based on StorageClass definition it creates PV object with the corresponding underlying volume backend (ebs/gcedisk/local). +5. During the build, each step (clone/build/push/freestyle/composition) is represented as docker container inside dind (docker-in-docker) pod. Shared Volume (`/codefresh/volume`) is represented as docker volume and mounted to every step (docker containers). PV mount point inside dind pod is `/var/lib/docker`. +6. Engine pod controls dind pod. It deserializes pipeline yaml to docker API calls, terminates dind after build has been finished or per user request (sigterm). +7. `dind-lv-monitor` DaemonSet OR `dind-volume-cleanup` CronJob are part of [Runtime Cleaner]({{site.baseurl}}/docs/administration/codefresh-runner/#runtime-cleaners), `app-proxy` Deployment and Ingress are described in the [next section]({{site.baseurl}}/docs/administration/codefresh-runner/#app-proxy-installation), `monitor` Deployment is for [Kubernetes Dashboard]({{site.baseurl}}/docs/deploy-to-kubernetes/manage-kubernetes/). + +## App Proxy installation + +The App Proxy is an **optional** component of the runner that is mainly used when the git provider server is installed on-premises behind the firewall. The App Proxy provides the following features once installed: + +* Enables you to automatically create webhooks for Git in the Codefresh UI (same as the SAAS experience) +* Sends commit status information back to your Git provider (same as the SAAS experience) +* Makes all Git Operations in the GUI work exactly like the SAAS installation of Codefresh + +The requirements for the App proxy is a Kubernetes cluster that: + +1. has already the Codefresh runner installed +1. has an active [ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress/) +1. allows incoming connections from the VPC/VPN where users are browsing the Codefresh UI. The ingress connection **must** have a hostname assigned for this route and **must** be configured to perform SSL termination + +>Currently the App-proxy works only for Github (SAAS and on-prem versions), Gitlab (SAAS and on-prem versions) and Bitbucket server. + +Here is the architecture of the app-proxy: + +{% include image.html + lightbox="true" + file="/images/administration/runner/app-proxy-architecture.png" + url="/images/administration/runner/app-proxy-architecture.png" + alt="How App Proxy and the Codefresh runner work together" + caption="How App Proxy and the Codefresh runner work together" + max-width="80%" + %} + +Basically when a Git GET operation takes place, the Codefresh UI will contact the app-proxy (if it is present) and it will route the request to the backing Git provider. The confidential Git information never leaves the firewall premises and the connection between the browser and the ingress is SSL/HTTPS. + +The app-proxy has to work over HTTPS and by default it will use the ingress controller to do its SSL termination. Therefore, the ingress controller will need to be configured to perform SSL termination. Check the documentation of your ingress controller (for example [nginx ingress](https://kubernetes.github.io/ingress-nginx/examples/tls-termination/)). This means that the app-proxy does not compromise security in any way. + +To install the app-proxy on a Kubernetes cluster that already has a Codefresh runner use the following command: + +```shell +codefresh install app-proxy --host=+``` + +If you want to install the Codefresh runner and app-proxy in a single command use the following: + +```shell +codefresh runner init --app-proxy --app-proxy-host= +``` + +If you have multiple ingress controllers in the Kubernetes cluster you can use the `--app-proxy-ingress-class` parameter to define which ingress will be used. For additional security you can also define an allowlist for IPs/ranges that are allowed to use the ingress (to further limit the web browsers that can access the Ingress). Check the documentation of your ingress controller for the exact details. + +By default the app-proxy ingress will use the path `hostname/app-proxy`. You can change that default by using the values file in the installation with the flag `--values values.yaml`. + +See the `AppProxy` section in the example [values.yaml](https://github.com/codefresh-io/venona/blob/release-1.0/venonactl/example/values-example.yaml#L231-L253). + +```shell +codefresh install app-proxy --values values.yaml +``` + +## Manual Installation of Runner Components + +If you don't want to use the wizard, you can also install the components of the runner yourself. + +The Codefresh runner consists of the following: + +* Runner - responsible for getting tasks from the platform and executing them. One per account. Can handle multiple runtimes +* Runtime - the components that are responsible on runtime for the workflow execution : + * Volume provisioner - (pod’s name prefix dind-volume-provisioner-runner) - responsible for volume provisioning for dind pod + * lv-monitor - (pod’s name prefix dind-lv-monitor-runner) - daemonset - responsible for cleaning volumes + +To install the runner on a single cluster with both the runtime and the agent, execute the following: + +```shell +kubectl create namespace codefresh +codefresh install agent --agent-kube-namespace codefresh --install-runtime +``` + +You can then follow the instructions for [using the runner](#using-the-codefresh-runner). + +### Installing Multiple runtimes with a Single Agent + +It is also possible, for advanced users to install a single agent that can manage multiple runtime environments. + +>NOTE: Please make sure that the cluster where the agent is installed has network access to the other clusters of the runtimes + +```shell +# 1. Create namespace for the agent: +kubectl create namespace codefresh-agent + +# 2. Install the agent on the namespace ( give your agent a unique name as $NAME): +# Note down the token and use it in the second command. +codefresh create agent $NAME +codefresh install agent --token $TOKEN --kube-namespace codefresh-agent +codefresh get agents + +# 3. Create namespace for the first runtime: +kubectl create namespace codefresh-runtime-1 + +# 4. Install the first runtime on the namespace +# 5. the runtime name is printed +codefresh install runtime --runtime-kube-namespace codefresh-runtime-1 + +# 6. Attach the first runtime to agent: +codefresh attach runtime --agent-name $AGENT_NAME --agent-kube-namespace codefresh-agent --runtime-name $RUNTIME_NAME --runtime-kube-namespace codefresh-runtime-1 + +# 7. Restart the runner pod in namespace `codefresh-agent` +kubectl delete pods $RUNNER_POD + +# 8. Create namespace for the second runtime +kubectl create namespace codefresh-runtime-2 + +# 9. Install the second runtime on the namespace +codefresh install runtime --runtime-kube-namespace codefresh-runtime-2 + +# 10. Attach the second runtime to agent and restart the Venona pod automatically +codefresh attach runtime --agent-name $AGENT_NAME --agent-kube-namespace codefresh-agent --runtime-name $RUNTIME_NAME --runtime-kube-namespace codefresh-runtime-2 --restart-agent +``` + +## Configuration Options + +You can fine tune the installation of the runner to better match your environment and cloud provider. + +### Installing on AWS + +If you've installed the Codefresh runner on [EKS](https://aws.amazon.com/eks/) or any other custom cluster (e.g. with kops) in Amazon you need to configure it properly to work with EBS volumes in order to gain [caching]({{site.baseurl}}/docs/configure-ci-cd-pipeline/pipeline-caching/). + +> This section assumes you already installed the Runner with default options: `codefresh runner init` + +**Prerequisites** + +`dind-volume-provisioner` deployment should have permissions to create/attach/detach/delete/get ebs volumes. + +There are 3 options: +* running `dind-volume-provisioner` pod on the node (node-group) with iam role +* k8s secret with [aws credentials format](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) mounted to ~/.aws/credentials (or `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` env vars passed) to the `dind-volume-provisioner` pod +* using [Aws Identity for Service Account](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) iam role assigned to `volume-provisioner-runner` service account + +Minimal policy for `dind-volume-provisioner`: +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ec2:AttachVolume", + "ec2:CreateSnapshot", + "ec2:CreateTags", + "ec2:CreateVolume", + "ec2:DeleteSnapshot", + "ec2:DeleteTags", + "ec2:DeleteVolume", + "ec2:DescribeInstances", + "ec2:DescribeSnapshots", + "ec2:DescribeTags", + "ec2:DescribeVolumes", + "ec2:DetachVolume" + ], + "Resource": "*" + } + ] +} +``` + +Create Storage Class for EBS volumes: +>Choose **one** of the Availability Zones you want to be used for your pipeline builds. Multi AZ configuration is not supported. + +**Storage Class (gp2)** + +```yaml +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: dind-ebs +### Specify name of provisioner +provisioner: codefresh.io/dind-volume-provisioner-runner-<-NAMESPACE-> # <---- rename <-NAMESPACE-> with the runner namespace +volumeBindingMode: Immediate +parameters: + # ebs or ebs-csi + volumeBackend: ebs + # Valid zone + AvailabilityZone: us-central1-a # <---- change it to your AZ + # gp2, gp3 or io1 + VolumeType: gp2 + # in case of io1 you can set iops + # iops: 1000 + # ext4 or xfs (default to xfs, ensure that there is xfstools ) + fsType: xfs +``` +**Storage Class (gp3)** + +```yaml +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: dind-ebs +### Specify name of provisioner +provisioner: codefresh.io/dind-volume-provisioner-runner-<-NAMESPACE-> # <---- rename <-NAMESPACE-> with the runner namespace +volumeBindingMode: Immediate +parameters: + # ebs or ebs-csi + volumeBackend: ebs + # Valid zone + AvailabilityZone: us-central1-a # <---- change it to your AZ + # gp2, gp3 or io1 + VolumeType: gp3 + # ext4 or xfs (default to xfs, ensure that there is xfstools ) + fsType: xfs + # I/O operations per second. Only effetive when gp3 volume type is specified. + # Default value - 3000. + # Max - 16,000 + iops: "5000" + # Throughput in MiB/s. Only effective when gp3 volume type is specified. + # Default value - 125. + # Max - 1000. + throughput: "500" +``` + +Apply storage class manifest: +```shell +kubectl apply -f dind-ebs.yaml +``` + +Change your [runtime environment]({{site.baseurl}}/docs/administration/codefresh-runner/#full-runtime-environment-specification) configuration: + +The same AZ you selected before should be used in nodeSelector inside Runtime Configuration: + +To get a list of all available runtimes execute: + +```shell +codefresh get runtime-environments +``` + +Choose the runtime you have just added and get its yaml representation: + +```shell +codefresh get runtime-environments my-eks-cluster/codefresh -o yaml > runtime.yaml +``` + + Under `dockerDaemonScheduler.cluster` block add the nodeSelector `topology.kubernetes.io/zone: `. It should be at the same level as `clusterProvider` and `namespace`. Also, the `pvcs.dind` block should be modified to use the Storage Class you created above (`dind-ebs`). + +`runtime.yaml` example: + +```yaml +version: 1 +metadata: + ... +runtimeScheduler: + cluster: + clusterProvider: + accountId: 5f048d85eb107d52b16c53ea + selector: my-eks-cluster + namespace: codefresh + serviceAccount: codefresh-engine + annotations: {} +dockerDaemonScheduler: + cluster: + clusterProvider: + accountId: 5f048d85eb107d52b16c53ea + selector: my-eks-cluster + namespace: codefresh + nodeSelector: + topology.kubernetes.io/zone: us-central1-a + serviceAccount: codefresh-engine + annotations: {} + userAccess: true + defaultDindResources: + requests: '' + pvcs: + dind: + volumeSize: 30Gi + storageClassName: dind-ebs + reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName' +extends: + - system/default/hybrid/k8s_low_limits +description: '...' +accountId: 5f048d85eb107d52b16c53ea +``` + +Update your runtime environment with the [patch command](https://codefresh-io.github.io/cli/operate-on-resources/patch/): + +```shell +codefresh patch runtime-environment my-eks-cluster/codefresh -f runtime.yaml +``` + +If necessary, delete all existing PV and PVC objects left from default local provisioner: +``` +kubectl delete pvc -l codefresh-app=dind -n +kubectl delete pv -l codefresh-app=dind -n +``` + +>You can define all these options above for clean Runner installation with [values.yaml](https://github.com/codefresh-io/venona/blob/release-1.0/venonactl/example/values-example.yaml) file: + +`values-ebs.yaml` example: + +```yaml +### Storage parameter example for aws ebs disks +Storage: + Backend: ebs + AvailabilityZone: us-east-1d + VolumeType: gp3 + #AwsAccessKeyId: ABCDF + #AwsSecretAccessKey: ZYXWV + Encrypted: # encrypt volume, default is false + VolumeProvisioner: + ServiceAccount: + Annotations: + eks.amazonaws.com/role-arn: arn:aws:iam:: :role/ +NodeSelector: topology.kubernetes.io/zone=us-east-1d +... + Runtime: + NodeSelector: # dind and engine pods node-selector (--build-node-selector) + topology.kubernetes.io/zone: us-east-1d +``` + +```shell +codefresh runner init --values values-ebs.yaml --exec-demo-pipeline false --skip-cluster-integration true +``` + +### Installing to EKS with Autoscaling + +#### Step 1- EKS Cluster Creation + +See below is a content of cluster.yaml file. We define separate node pools for dind, engine and other services(like runner, cluster-autoscaler etc). + +Before creating the cluster we have created two separate IAM policies: + +* one for our volume-provisioner controller(policy/runner-ebs) that should create and delete volumes +* one for dind pods(policy/dind-ebs) that should be able to attach/detach those volumes to the appropriate nodes using [iam attachPolicyARNs options](https://eksctl.io/usage/iam-policies/). + +`policy/dind-ebs:` + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ec2:DescribeVolumes" + ], + "Resource": [ + "*" + ] + }, + { + "Effect": "Allow", + "Action": [ + "ec2:DetachVolume", + "ec2:AttachVolume" + ], + "Resource": [ + "*" + ] + } + ] +} +``` + +`policy/runner-ebs:` + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ec2:AttachVolume", + "ec2:CreateSnapshot", + "ec2:CreateTags", + "ec2:CreateVolume", + "ec2:DeleteSnapshot", + "ec2:DeleteTags", + "ec2:DeleteVolume", + "ec2:DescribeInstances", + "ec2:DescribeSnapshots", + "ec2:DescribeTags", + "ec2:DescribeVolumes", + "ec2:DetachVolume" + ], + "Resource": "*" + } + ] +} +``` + +`my-eks-cluster.yaml` + +```yaml +apiVersion: eksctl.io/v1alpha5 +kind: ClusterConfig +metadata: + name: my-eks + region: us-west-2 + version: "1.15" + +nodeGroups: + - name: dind + instanceType: m5.2xlarge + desiredCapacity: 1 + iam: + attachPolicyARNs: + - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy + - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy + - arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess + - arn:aws:iam::XXXXXXXXXXXX:policy/dind-ebs + withAddonPolicies: + autoScaler: true + ssh: # import public key from file + publicKeyPath: ~/.ssh/id_rsa.pub + minSize: 1 + maxSize: 50 + volumeSize: 50 + volumeType: gp2 + ebsOptimized: true + availabilityZones: ["us-west-2a"] + kubeletExtraConfig: + enableControllerAttachDetach: false + labels: + node-type: dind + taints: + codefresh.io: "dinds:NoSchedule" + + - name: engine + instanceType: m5.large + desiredCapacity: 1 + iam: + withAddonPolicies: + autoScaler: true + minSize: 1 + maxSize: 10 + volumeSize: 50 + volumeType: gp2 + availabilityZones: ["us-west-2a"] + labels: + node-type: engine + taints: + codefresh.io: "engine:NoSchedule" + + - name: addons + instanceType: m5.2xlarge + desiredCapacity: 1 + ssh: # import public key from file + publicKeyPath: ~/.ssh/id_rsa.pub + minSize: 1 + maxSize: 10 + volumeSize: 50 + volumeType: gp2 + ebsOptimized: true + availabilityZones: ["us-west-2a"] + labels: + node-type: addons + iam: + attachPolicyARNs: + - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy + - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy + - arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess + - arn:aws:iam::XXXXXXXXXXXX:policy/runner-ebs + withAddonPolicies: + autoScaler: true +availabilityZones: ["us-west-2a", "us-west-2b", "us-west-2c"] +``` + +Execute: + +```shell +eksctl create cluster -f my-eks-cluster.yaml +``` + +The config above will leverage [Amazon Linux 2](https://aws.amazon.com/amazon-linux-2/) as the default operating system for the nodes in the nodegroup. To leverage [Bottlerocket-based nodes](https://aws.amazon.com/bottlerocket/), specify the AMI Family using `amiFamily: Bottlerocket` and add the following additional IAM Policies: `arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly` and `arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore`. + +>Bottlerocket is an open source Linux based Operating System specifically built to run containers. It focuses on security, simplicity and easy updates via transactions. Find more information in the [official repository](https://github.com/bottlerocket-os/bottlerocket). + +#### Step 2 - Autoscaler + +Once the cluster is up and running we need to install the [cluster autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html): + +We used iam AddonPolicies `"autoScaler: true"` in the cluster.yaml file so there is no need to create a separate IAM policy or add Auto Scaling group tags, everything is done automatically. + +Deploy the Cluster Autoscaler: + +```shell +kubectl apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml +``` + +Add the `cluster-autoscaler.kubernetes.io/safe-to-evict` annotation + +```shell +kubectl -n kube-system annotate deployment.apps/cluster-autoscaler cluster-autoscaler.kubernetes.io/safe-to-evict="false" +``` + +Edit the cluster-autoscaler container command to replace ` ` with *my-eks*(name of the cluster from cluster.yaml file), and add the following options: + `--balance-similar-node-groups` and `--skip-nodes-with-system-pods=false` + +```shell +kubectl -n kube-system edit deployment.apps/cluster-autoscaler +``` + +```yaml +spec: + containers: + - command: + - ./cluster-autoscaler + - --v=4 + - --stderrthreshold=info + - --cloud-provider=aws + - --skip-nodes-with-local-storage=false + - --expander=least-waste + - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-eks + - --balance-similar-node-groups + - --skip-nodes-with-system-pods=false +``` + +We created our EKS cluster with 1.15 version so the appropriate cluster autoscaler version from [https://github.com/kubernetes/autoscaler/releases](https://github.com/kubernetes/autoscaler/releases) is 1.15.6 + +```shell +kubectl -n kube-system set image deployment.apps/cluster-autoscaler cluster-autoscaler=us.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v1.15.6 +``` + +Check your own version to make sure that the autoscaler version is appropriate. + +#### Step 3 - Optional: We also advise to configure overprovisioning with Cluster Autoscaler + +See details at the [FAQ]( +https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-configure-overprovisioning-with-cluster-autoscaler). + +#### Step 4 - Adding an EKS cluster as a runner to the Codefresh platform with EBS support + +Make sure that you are targeting the correct cluster + +```shell +$ kubectl config current-context +my-aws-runner +``` + +Install the runner passing additional options: + +```shell +codefresh runner init \ +--name my-aws-runner \ +--kube-node-selector=topology.kubernetes.io/zone=us-west-2a \ +--build-node-selector=topology.kubernetes.io/zone=us-west-2a \ +--kube-namespace cf --kube-context-name my-aws-runner \ +--set-value Storage.VolumeProvisioner.NodeSelector=node-type=addons \ +--set-value=Storage.Backend=ebs \ +--set-value=Storage.AvailabilityZone=us-west-2a +``` + +* You should specify the zone in which you want your volumes to be created, example: `--set-value=Storage.AvailabilityZone=us-west-2a` +* (Optional) - if you want to assign the volume-provisioner to a specific node, for example a specific node group what has an IAM role which allows to create EBS volumes, example: `--set-value Storage.VolumeProvisioner.NodeSelector=node-type=addons` + +If you want to use [encrypted EBS volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html#EBSEncryption_key_mgmt) (they are unencrypted by default) - add the custom value `--set-value=Storage.Encrypted=true` +If you already have a key - add its ARN via `--set-value=Storage.KmsKeyId= value`, otherwise a key is generated by AWS. Here is the full command: + +```shell +codefresh runner init \ +--name my-aws-runner \ +--kube-node-selector=topology.kubernetes.io/zone=us-west-2a \ +--build-node-selector=topology.kubernetes.io/zone=us-west-2a \ +--kube-namespace cf --kube-context-name my-aws-runner \ +--set-value Storage.VolumeProvisioner.NodeSelector=node-type=addons \ +--set-value=Storage.Backend=ebs \ +--set-value=Storage.AvailabilityZone=us-west-2a\ +--set-value=Storage.Encrypted=[false|true] \ +--set-value=Storage.KmsKeyId= +``` + +For an explanation of all other options run `codefresh runner init --help` ([global parameter table](#customizing-the-wizard-installation)). + +At this point the quick start wizard will start the installation. + +Once that is done we need to modify the runtime environment of `my-aws-runner` to specify the necessary toleration, nodeSelector and disk size: + +```shell +codefresh get re --limit=100 my-aws-runner/cf -o yaml > my-runtime.yml +``` + +Modify the file my-runtime.yml as shown below: + +```yaml +version: null +metadata: + agent: true + trial: + endingAt: 1593596844167 + reason: Codefresh hybrid runtime + started: 1592387244207 + name: my-aws-runner/cf + changedBy: ivan-codefresh + creationTime: '2020/06/17 09:47:24' +runtimeScheduler: + cluster: + clusterProvider: + accountId: 5cb563d0506083262ba1f327 + selector: my-aws-runner + namespace: cf + nodeSelector: + node-type: engine + tolerations: + - effect: NoSchedule + key: codefresh.io + operator: Equal + value: engine + annotations: {} +dockerDaemonScheduler: + cluster: + clusterProvider: + accountId: 5cb563d0506083262ba1f327 + selector: my-aws-runner + namespace: cf + nodeSelector: + node-type: dind + annotations: {} + defaultDindResources: + requests: '' + tolerations: + - effect: NoSchedule + key: codefresh.io + operator: Equal + value: dinds + pvcs: + dind: + volumeSize: 30Gi + reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName' + storageClassName: dind-local-volumes-runner-cf + userAccess: true +extends: + - system/default/hybrid/k8s_low_limits +description: 'Runtime environment configure to cluster: my-aws-runner and namespace: cf' +accountId: 5cb563d0506083262ba1f327 +``` + +Apply changes. + +```shell +codefresh patch re my-aws-runner/cf -f my-runtime.yml +``` + +That's all. Now you can go to UI and try to run a pipeline on RE my-aws-runner/cf + +### Injecting AWS arn roles into the cluster + +**Step 1** - Make sure the OIDC provider is connected to the cluster + +See: + +* [https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) +* [https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/) + +**Step 2** - Create IAM role and policy as explained in [https://docs.aws.amazon.com/eks/latest/userguide/create-service-account-iam-policy-and-role.html](https://docs.aws.amazon.com/eks/latest/userguide/create-service-account-iam-policy-and-role.html) + +Here, in addition to the policy explained, you need a Trust Relationship established between this role and the OIDC entity. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { + "${OIDC_PROVIDER}:sub": "system:serviceaccount:${CODEFRESH_NAMESPACE}:codefresh-engine" + } + } + } + ] +} +``` + +**Step 3** - Annotate the `codefresh-engine` Kubernetes Service Account in the namespace where the Codefresh Runner is installed with the proper IAM role. + +```shell +kubectl annotate -n ${CODEFRESH_NAMESPACE} sa codefresh-engine eks.amazonaws.com/role-arn=${ROLE_ARN} +``` + +Once the annotation is added, you should see it when you describe the Service Account. + +```shell +kubectl describe -n ${CODEFRESH_NAMESPACE} sa codefresh-engine + +Name: codefresh-engine +Namespace: codefresh +Labels: app=app-proxy + version=1.6.8 +Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/Codefresh +Image pull secrets: +Mountable secrets: codefresh-engine-token-msj8d +Tokens: codefresh-engine-token-msj8d +Events: +``` + +**Step 4** - Using the AWS assumed role identity + +After annotating the Service Account, run a pipeline to test the AWS resource access: + +```yaml +RunAwsCli: + title : Communication with AWS + image : mesosphere/aws-cli + stage: "build" + commands : + - apk update + - apk add jq + - env + - cat /codefresh/volume/sensitive/.kube/web_id_token + - aws sts assume-role-with-web-identity --role-arn $AWS_ROLE_ARN --role-session-name mh9test --web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE --duration-seconds 1000 > /tmp/irp-cred.txt + - export AWS_ACCESS_KEY_ID="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.AccessKeyId")" + - export AWS_SECRET_ACCESS_KEY="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SecretAccessKey")" + - export AWS_SESSION_TOKEN="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SessionToken")" + - rm /tmp/irp-cred.txt + - aws s3api get-object --bucket jags-cf-eks-pod-secrets-bucket --key eks-pod2019-12-10-21-18-32-560931EEF8561BC4 getObjectNotWorks.txt +``` + +### Installing behind a proxy + +If you want to deploy the Codefresh runner on a Kubernetes cluster that doesn’t have direct access to `g.codefresh.io`, and has to go trough a proxy server to access `g.codefresh.io`, you will need to follow these additional steps: + +**Step 1** - Follow the installation instructions of the previous section + +**Step 2** - Run `kubectl edit deployment runner -n codefresh-runtime` and add the proxy variables like this + +```yaml +spec: + containers: + - env: + - name: HTTP_PROXY + value: http:// :port + - name: HTTPS_PROXY + value: http:// :port + - name: http_proxy + value: http:// :port + - name: https_proxy + value: http:// :port + - name: no_proxy + value: localhost,127.0.0.1, + - name: NO_PROXY + value: localhost,127.0.0.1, +``` + +**Step 3** - Add the following variables to your runtime.yaml, both under the `runtimeScheduler:` and under `dockerDaemonScheduler:` blocks inside the `envVars:` section + +```yaml +HTTP_PROXY: http:// :port +http_proxy: http:// :port +HTTPS_PROXY: http:// :port +https_proxy: http:// :port +No_proxy: localhost, 127.0.0.1, +NO_PROXY: localhost, 127.0.0.1, +``` + +**Step 4** - Add `.firebaseio.com` to the allowed-sites of the proxy server + +**Step 5** - Exec into the `dind` pod and run `ifconfig` + +If the MTU value for `docker0` is higher than the MTU value of `eth0` (sometimes the `docker0` MTU is 1500, while `eth0` MTU is 1440) - you need to change this, the `docker0` MTU should be lower than `eth0` MTU + +To fix this, edit the configmap in the codefresh-runtime namespace: + +```shell +kubectl edit cm codefresh-dind-config -n codefresh-runtime +``` + +And add this after one of the commas: +`\"mtu\":1440,` + +### Installing on Rancher RKE 2.X + +#### Step 1 - Configure the kubelet to work with the runner's StorageClass + +The runner's default StorageClass creates the persistent cache volume from local storage on each node. We need to edit the cluster config to allow this. + +In the Rancher UI (v2.5.9 and earlier), drill into the target cluster and then click the Edit Cluster button at the top-right. +{% include image.html + lightbox="true" + file="/images/administration/runner/rancher-cluster.png" + url="/images/administration/runner/rancher-cluster.png" + alt="Drill into your cluster and click Edit Cluster on the right" + caption="Drill into your cluster and click Edit Cluster on the right" + max-width="100%" + %} + +In Rancher v2.6+ with the updated UI, open the Cluster Management in the left panel, then click the three-dot menu near the corresponding cluster and select 'Edit Config'. +{% include image.html + lightbox="true" + file="/images/administration/runner/rancher-cluster-2.png" + url="/images/administration/runner/rancher-cluster-2.png" + alt="Click Edit Cluster on the right in your cluster list" + caption="Click Edit Cluster on the right in your cluster list" + max-width="100%" + %} + +On the edit cluster page, scroll down to the Cluster Options section and click its **Edit as YAML** button +{% include image.html + lightbox="true" + file="/images/administration/runner/rancher-edit-as-yaml.png" + url="/images/administration/runner/rancher-edit-as-yaml.png" + alt="Cluster Options -> Edit as YAML" + caption="Cluster Options -> Edit as YAML" + max-width="100%" + %} +Edit the YAML to include an extra mount in the kubelet service: + +```yaml +rancher_kubernetes_engine_config: + ... + services: + ... + kubelet: + extra_binds: + - '/var/lib/codefresh:/var/lib/codefresh:rshared' +``` + +{% include image.html + lightbox="true" + file="/images/administration/runner/rancher-kublet.png" + url="/images/administration/runner/rancher-kublet.png" + alt="Add volume to rancher_kubernetes_engine_config.services.kublet.extra_binds" + caption="Add volume to rancher_kubernetes_engine_config.services.kublet.extra_binds" + max-width="100%" + %} + +#### Step 2 - Make sure your kubeconfig user is a ClusterAdmin + +The user in your kubeconfig must be a cluster admin in order to install the runner. If you plan to have your pipelines connect to this cluster as a cluster admin, then you can go ahead and create a Codefresh user for this purpose in the Rancher UI with a **non-expiring** kubeconfig token. This is the easiest way to do the installation. + +However, if you want your pipelines to connect to this cluster with less privileges, then you can use your personal user account with Cluster Admin privileges for the installation, and then we'll create a Codefresh account with lesser privileges later (in Step 5). In that case, you can now move on to Step 3. + +Follow these steps to create a Codefresh user with Cluster Admin rights, from the Rancher UI: + +* Click Security at the top, and then choose Users + {% include image.html lightbox="true" file="/images/administration/runner/rancher-security.png" url="/images/administration/runner/rancher-security.png" alt="Create a cluster admin user for Codefresh" caption="Create a cluster admin ser for Codefresh" max-width="100%" %} +* Click the Add User button, and under Global Permissions check the box for **Restricted Administrstor** +* Log out of the Rancher UI, and then log back in as the new user +* Click your user icon at the top-right, and then choose **API & Keys** +* Click the **Add Key** button and create a kubeconfig token with Expires set to Never +* Copy the Bearer Token field (combines Access Key and Secret Key) +* Edit your kubeconfig and put the Bearer Token you copied in the `token` field of your user + +#### Step 3 - Install the Runner + +If you've created your kubeconfig from the Rancher UI, then it will contain an API endpoint that is not reachable internally, from within the cluster. To work around this, we need to tell the runner to instead use Kubernetes' generic internal API endpoint. Also, if you didn't create a Codefresh user in step 2 and your kubeconfig contains your personal user account, then you should also add the `--skip-cluster-integration` option. + +Install the runner with a Codefresh user (ClusterAdmin, non-expiring token): + +```shell +codefresh runner init \ + --set-value KubernetesHost=https://kubernetes.default.svc.cluster.local +``` + +Or install the runner with your personal user account: + +```shell +codefresh runner init \ + --set-value KubernetesHost=https://kubernetes.default.svc.cluster.local \ + --skip-cluster-integration +``` + +The wizard will then ask you some basic questions. + +#### Step 4 - Update the runner's Docker MTU + +By default, RKE nodes use the [Canal CNI](https://rancher.com/docs/rancher/v2.x/en/faq/networking/cni-providers/#canal), which combines elements of Flannel and Calico, and uses VXLAN encapsulation. This VXLAN encapsulation has a 50-byte overhead, thus reducing the MTU of its virtual interfaces from the standard 1500 to 1450. For example, when running `ifconfig` on an RKE 2.5.5 node, you might see several interfaces like this. Note the `MTU:1450`. + +```shell +cali0f8ac592086 Link encap:Ethernet HWaddr ee:ee:ee:ee:ee:ee + inet6 addr: fe80::ecee:eeff:feee:eeee/64 Scope:Link + UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 + RX packets:11106 errors:0 dropped:0 overruns:0 frame:0 + TX packets:10908 errors:0 dropped:0 overruns:0 carrier:0 + collisions:0 txqueuelen:0 + RX bytes:922373 (922.3 KB) TX bytes:9825590 (9.8 MB) +``` + +We must reduce the Docker MTU used by the runner's Docker in Docker (dind) pods to fit within this lower MTU. This is stored in a configmap in the namespace where the runner is installed. Assuming that you installed the runner into the `codefresh` namespace, you would edit the configmap like this: + +```shell +kubectl edit cm codefresh-dind-config -n codefresh +``` + +In the editor, update the **daemon.json** field - add `,\"mtu\":1440` just before the last curley brace. + {% include image.html + lightbox="true" + file="/images/administration/runner/rancher-mtu.png" + url="/images/administration/runner/rancher-mtu.png" + alt="Update the runner's Docker MTU" + caption="Update the runner's Docker MTU" + max-width="100%" + %} + +#### Step 5 - Create the Cluster Integration + +If you created a user in Step 2 and used it to install the runner in Step 3, then you can skip this step - your installation is complete! + +However, if you installed the runner with the `--skip-cluster-integration` option then you should follow the documentaion to [Add a Rancher Cluster]({{site.baseurl}}/docs/deploy-to-kubernetes/add-kubernetes-cluster/#adding-a-rancher-cluster) to your Kubernetes Integrations. + +Once complete, you can go to the Codefresh UI and run a pipeline on the new runtime, including steps that deploy to the Kubernetes Integration. + +#### Troubleshooting TLS Errors + +Depending on your Rancher configuration, you may need to allow insecure HTTPS/TLS connections. You can do this by adding an environment variable to the runner deployment. + +Assuming that you installed the runner into the `codefresh` namespace, you would edit the runner deployment like this: + +```shell +kubectl edit deploy runner -n codefresh +``` + +In the editor, add this environment variable under spec.containers.env[]: + +```yaml +- name: NODE_TLS_REJECT_UNAUTHORIZED + value: "0" +``` + +### Installing on Google Kubernetes Engine + +If you are installing Codefresh runner on the Kubernetes cluster on [GKE](https://cloud.google.com/kubernetes-engine/) + +* make sure your user has `Kubernetes Engine Cluster Admin` role in google console and +* bind your user with `cluster-admin` Kubernetes cluster role. + +```shell +kubectl create clusterrolebinding cluster-admin-binding \ + --clusterrole cluster-admin \ + --user $(gcloud config get-value account) +``` + + +#### Storage options on GKE + +**Local SSD** + +If you want to use *LocalSSD* in GKE: + +*Prerequisites:* [GKE cluster with local SSD](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd) + +Install Runner with the Wizard: + +```shell +codefresh runner init [options] --set-value=Storage.LocalVolumeParentDir=/mnt/disks/ssd0/codefresh-volumes \ + --build-node-selector=cloud.google.com/gke-local-ssd=true +``` + +Or with `values-example.yaml` values file: + +```yaml +... +### Storage parameters example for gke-local-ssd + Storage: + Backend: local + LocalVolumeParentDir: /mnt/disks/ssd0/codefresh-volumes + NodeSelector: cloud.google.com/gke-local-ssd=true +... + Runtime: + NodeSelector: # dind and engine pods node-selector (--build-node-selector) + cloud.google.com/gke-local-ssd: "true" +... +``` +```shell +codefresh runner init [options] --values values-example.yaml +``` + +To configure existing Runner with Local SSDs follow this article: + +[How-to: Configuring an existing Runtime Environment with Local SSDs (GKE only)](https://support.codefresh.io/hc/en-us/articles/360016652920-How-to-Configuring-an-existing-Runtime-Environment-with-Local-SSDs-GKE-only-) + + +**GCE Disks** + +If you want to use *GCE Disks*: + +*Prerequisites:* volume provisioner (dind-volume-provisioner) should have permissions to create/delete/get GCE disks + +There are 3 options to provide cloud credentials: + +* run `dind-volume-provisioner-runner` pod on a node with IAM role which is allowed to create/delete/get GCE disks +* create Google Service Account with `ComputeEngine.StorageAdmin` role, download its key in JSON format and pass it to `codefresh runner init` with `--set-file=Storage.GooogleServiceAccount=/path/to/google-service-account.json` +* use [Google Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) to assign IAM role to `volume-provisioner-runner` service account + +Notice that builds will be running in a single availability zone, so you must specify AvailabilityZone parameters. + + +##### Runner installation with GCE Disks (Google SA JSON key) + +Using the Wizard: + +```shell +codefresh runner init [options] \ + --set-value=Storage.Backend=gcedisk \ + --set-value=Storage.AvailabilityZone=us-central1-c \ + --kube-node-selector=topology.kubernetes.io/zone=us-central1-c \ + --build-node-selector=topology.kubernetes.io/zone=us-central1-c \ + --set-file=Storage.GoogleServiceAccount=/path/to/google-service-account.json +``` + +Using the values `values-example.yaml` file: +```yaml +... +### Storage parameter example for GCE disks + Storage: + Backend: gcedisk + AvailabilityZone: us-central1-c + GoogleServiceAccount: > #serviceAccount.json content + { + "type": "service_account", + "project_id": "...", + "private_key_id": "...", + "private_key": "...", + "client_email": "...", + "client_id": "...", + "auth_uri": "...", + "token_uri": "...", + "auth_provider_x509_cert_url": "...", + "client_x509_cert_url": "..." + } + NodeSelector: topology.kubernetes.io/zone=us-central1-c +... + Runtime: + NodeSelector: # dind and engine pods node-selector (--build-node-selector) + topology.kubernetes.io/zone: us-central1-c +... +``` +```shell +codefresh runner init [options] --values values-example.yaml +``` + + +##### Runner installation with GCE Disks (Workload Identity with IAM role) + +Using the values `values-example.yaml` file: + +```yaml +... +### Storage parameter example for GCE disks + Storage: + Backend: gcedisk + AvailabilityZone: us-central1-c + VolumeProvisioner: + ServiceAccount: + Annotations: #annotation to the volume-provisioner service account, using the email address of the Google service account + iam.gke.io/gcp-service-account: @ .iam.gserviceaccount.com + NodeSelector: topology.kubernetes.io/zone=us-central1-c +... + Runtime: + NodeSelector: # dind and engine pods node-selector (--build-node-selector) + topology.kubernetes.io/zone: us-central1-c +... +``` +```shell +codefresh runner init [options] --values values-example.yaml +``` + +Create the binding between Kubernetes service account and Google service account: + +```shell +export K8S_NAMESPACE=codefresh +export KSA_NAME=volume-provisioner-runner +export GSA_NAME= +export PROJECT_ID= + +gcloud iam service-accounts add-iam-policy-binding \ + --role roles/iam.workloadIdentityUser \ + --member "serviceAccount:${PROJECT_ID}.svc.id.goog[${K8S_NAMESPACE}/${KSA_NAME}]" \ + ${GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com +``` + +To configure existing Runner with GCE Disks follow this article: + +[How-to: Configuring an existing Runtime Environment with GCE disks](https://support.codefresh.io/hc/en-us/articles/360016652900-How-to-Configuring-an-existing-Runtime-Environment-with-GCE-disks) + + +##### Using multiple Availability Zones + +Currently, to support effective caching with GCE disks, the builds/pods need to be scheduled in a single AZ (this is more related to a GCP limitation than a Codefresh runner issue). + +If you have Kubernetes nodes running in multiple Availability Zones and wish to use the Codefresh runner we suggest the following: + +**Option A** - Provision a new Kubernetes cluster: a cluster that runs in a single AZ only. - The cluster should be dedicated for usage with the Codefresh runner. This is the preferred solution and avoids extra complexity. + +**Option B** - Install Codefresh runner in your multi-zone cluster, and let it run in the default Node Pool: - in this case, you must specify `--build-node-selector= ` (e.g.: `--build-node-selector=topology.kubernetes.io/zone=us-central1-c`) or simply modify the Runtime environment as below: + +```shell +codefresh get re $RUNTIME_NAME -o yaml > re.yaml +``` + +Edit the yaml: + +```yaml +version: 2 +metadata: + ... +runtimeScheduler: + cluster: + nodeSelector: #schedule engine pod onto a node whose labels match the nodeSelector + topology.kubernetes.io/zone: us-central1-c + ... +dockerDaemonScheduler: + cluster: + nodeSelector: #schedule dind pod onto a node whose labels match the nodeSelector + topology.kubernetes.io/zone: us-central1-c + ... + pvcs: + dind: + ... +``` + +Apply changes with: + +```shell +codefresh patch re -f re.yaml +``` + +**Option C** - Like option B, but with a dedicated Node Pool + +**Option D** - Have 2 separate Codefresh runner Runtimes, one for zone A, and the other for zone B, and so on: this technically works, but it will require you to manually set the RE to use for the pipelines that won't use the default Codefresh runner RE. To distribute the pipeline's builds across the Codefresh runner REs. + +For example, let's say Venona-zoneA is the default RE, then, that means that for the pipelines that you want to run in Venona-zoneB, then you'll need to modify their RE settings, and explicitly set Venona-zoneB as the one to use. + +Regarding [Regional Persistent Disks](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/regional-pd), their support is not currently implemented in the Codefresh runner. + + +### Installing on AKS + +**Azure Disks** + +*Prerequisite:* volume provisioner (`dind-volume-provisioner`) should have permissions to create/delete/get Azure Disks + +Minimal IAM Role for dind-volume-provisioner:
+`dind-volume-provisioner-role.json` +```json +{ + "Name": "CodefreshDindVolumeProvisioner", + "Description": "Perform create/delete/get disks", + "IsCustom": true, + "Actions": [ + "Microsoft.Compute/disks/read", + "Microsoft.Compute/disks/write", + "Microsoft.Compute/disks/delete" + + ], + "AssignableScopes": ["/subscriptions/"] +} +``` + +If you use AKS with managed [identities for node group](https://docs.microsoft.com/en-us/azure/aks/use-managed-identity), you can run the script below to assign `CodefreshDindVolumeProvisioner` role to aks node identity: + +```shell +export ROLE_DEFINITIN_FILE=dind-volume-provisioner-role.json +export SUBSCRIPTION_ID=$(az account show --query "id" | xargs echo ) +export RESOURCE_GROUP=codefresh-rt1 +export AKS_NAME=codefresh-rt1 +export LOCATION=$(az aks show -g $RESOURCE_GROUP -n $AKS_NAME --query location | xargs echo) +export NODES_RESOURCE_GROUP=MC_${RESOURCE_GROUP}_${AKS_NAME}_${LOCATION} +export NODE_SERVICE_PRINCIPAL=$(az aks show -g $RESOURCE_GROUP -n $AKS_NAME --query identityProfile.kubeletidentity.objectId | xargs echo) + +az role definition create --role-definition @${ROLE_DEFINITIN_FILE} +az role assignment create --assignee $NODE_SERVICE_PRINCIPAL --scope /subscriptions/$SUBSCRIPTION_ID/resourceGroups/$NODES_RESOURCE_GROUP --role CodefreshDindVolumeProvisioner +``` + +Now install Codefresh Runner with cli wizard: +```shell +codefresh runner init --set-value Storage.Backend=azuredisk --set Storage.VolumeProvisioner.MountAzureJson=true +``` +Or using [values-example.yaml](https://github.com/codefresh-io/venona/blob/release-1.0/venonactl/example/values-example.yaml): +```yaml +Storage: + Backend: azuredisk + VolumeProvisioner: + MountAzureJson: true +``` +```shell +codefresh runner init --values values-example.yaml +``` +Or with helm chart [values.yaml](https://github.com/codefresh-io/venona/blob/release-1.0/charts/cf-runtime/values.yaml): +```yaml +storage: + backend: azuredisk + azuredisk: + skuName: Premium_LRS + +volumeProvisioner: + mountAzureJson: true +``` +```shell +helm install cf-runtime cf-runtime/cf-runtime -f ./generated_values.yaml -f values.yaml --create-namespace --namespace codefresh +``` + + +### Internal Registry Mirror + +You can configure your Codefresh Runner to use an internal registry as a mirror for any container images that are mentioned in your pipelines. + +First setup an internal registry as described in [https://docs.docker.com/registry/recipes/mirror/](https://docs.docker.com/registry/recipes/mirror/). + +Then locate the `codefresh-dind-config` config map in the namespace that houses the runner and edit it. + +```shell +kubectl -n codefresh edit configmap codefresh-dind-config +``` + +Change the `data` field from: + +```yaml +data: + daemon.json: "{\n \"hosts\": [ \"unix:///var/run/docker.sock\",\n \"tcp://0.0.0.0:1300\"],\n + \ \"storage-driver\": \"overlay2\",\n \"tlsverify\": true, \n \"tls\": true,\n + \ \"tlscacert\": \"/etc/ssl/cf-client/ca.pem\",\n \"tlscert\": \"/etc/ssl/cf/server-cert.pem\",\n + \ \"tlskey\": \"/etc/ssl/cf/server-key.pem\",\n \"insecure-registries\" : [\"192.168.99.100:5000\"],\n + \ \"metrics-addr\" : \"0.0.0.0:9323\",\n \"experimental\" : true\n}\n" +``` + +to + +```yaml +data: + daemon.json: "{\n \"hosts\": [ \"unix:///var/run/docker.sock\",\n \"tcp://0.0.0.0:1300\"],\n + \ \"storage-driver\": \"overlay2\",\n \"tlsverify\": true, \n \"tls\": true,\n + \ \"tlscacert\": \"/etc/ssl/cf-client/ca.pem\",\n \"tlscert\": \"/etc/ssl/cf/server-cert.pem\",\n + \ \"tlskey\": \"/etc/ssl/cf/server-key.pem\",\n \"insecure-registries\" : [\"192.168.99.100:5000\"],\n + \ \"registry-mirrors\": [ \"https:// \" ], \n + \ \"metrics-addr\" : \"0.0.0.0:9323\",\n \"experimental\" : true\n}\n" +``` + +This adds the line `\ \"registry-mirrors\": [ \"https:// \" ], \n` which contains a single registry to use as a mirror. Quit and Save by typing `:wq`. + +Now any container image that is used in your pipeline and isn't fully qualified, will be pulled through the Docker registry that is configured as a mirror. + + +### Installing the monitoring component + +If your cluster is located [behind the firewall](https://codefresh.io/docs/docs/administration/behind-the-firewall/) you might want to use the runner monitoring component to get valuable information about the cluster resources to Codefresh, for example, to [Kubernetes](https://g.codefresh.io/kubernetes/services/) and [Helm Releases](https://g.codefresh.io/helm/releases/releasesNew/) dashboards. + +To install the monitoring component you can use `--install-monitor` flag in the `runner init` command: + +```shell +codefresh runner init --install-monitor +``` + +Please note, that the monitoring component will not be installed if you use `--install-monitor` with `--skip-cluster-integration` flag. In case you want to skip adding the cluster integration during the runner installation, but still want to get the cluster resources to Codefresh dashboards, you can install the monitoring component separately: + +```shell +codefresh install monitor --kube-context-name --kube-namespace --cluster-id --token +``` + + + +## Full runtime environment specification + +The following section contains an explanation of runtime environment specification and possible options to modify it. Notice that there are additional and hidden fields that are autogenerated by Codefresh that complete a full runtime spec. You can't directly see or edit them (unless you run your own [Codefresh On-Premises Installation]({{site.baseurl}}/docs/administration/codefresh-on-prem/) ) + + +To get a list of all available runtimes execute: +```shell +codefresh get runtime-environments +#or +codefresh get re +``` + +Choose the runtime that you want to inspect or modify and get its yaml/json representation: +```shell +codefresh get re my-eks-cluster/codefresh -o yaml > runtime.yaml +#or +codefresh get re my-eks-cluster/codefresh -o json > runtime.json +``` + +Update your runtime environment with the [patch command](https://codefresh-io.github.io/cli/operate-on-resources/patch/): +```shell +codefresh patch re my-eks-cluster/codefresh -f runtime.yaml +``` + +Below is the example for the default and basic runtime spec after you've installed the Runner: + +{% highlight yaml %} +{% raw %} +version: 1 +metadata: + ... +runtimeScheduler: + cluster: + clusterProvider: + accountId: 5f048d85eb107d52b16c53ea + selector: my-eks-cluster + namespace: codefresh + serviceAccount: codefresh-engine + annotations: {} +dockerDaemonScheduler: + cluster: + clusterProvider: + accountId: 5f048d85eb107d52b16c53ea + selector: my-eks-cluster + namespace: codefresh + serviceAccount: codefresh-engine + annotations: {} + userAccess: true + defaultDindResources: + requests: '' + pvcs: + dind: + storageClassName: dind-local-volumes-runner-codefresh +extends: + - system/default/hybrid/k8s_low_limits +description: '...' +accountId: 5f048d85eb107d52b16c53ea +{% endraw %} +{% endhighlight %} + +### Top level fields + +{: .table .table-bordered .table-hover} +| Field name | Type | Value | +| -------------- |-------------------------| -------------------------| +| `version` | string | Runtime environment version | +| `metadata` | object | Meta-information | +| `runtimeScheduler` | object | Engine pod definition | +| `dockerDaemonScheduler` | object | Dind pod definition | +| `extends` | array | System field (links to full runtime spec from Codefresh API) | +| `description` | string | Runtime environment description (k8s context name and namespace) | +| `accountId` | string | Account to which this runtime belongs | +| `appProxy` | object | Optional filed for [app-proxy]({{site.baseurl}}/docs/administration/codefresh-runner/#optional-installation-of-the-app-proxy) | + +### runtimeScheduler fields (engine) + +{: .table .table-bordered .table-hover} +| Field name | Type | Value | +| -------------- |-------------------------| -------------------------| +| `image` | string | Override default engine image | +| `imagePullPolicy` | string | Override image pull policy (default `IfNotPresent`) | +| `type` | string | `KubernetesPod` | +| `envVars` | object | Override or add environment variables passed into the engine pod | +| `userEnvVars` | object | Add external env var(s) to the pipeline. See [Custom Global Environment Variables]({{site.baseurl}}/docs/administration/codefresh-runner/#custom-global-environment-variables) | +| `cluster` | object | k8s related information (`namespace`, `serviceAccount`, `nodeSelector`) | +| `resources` | object | Specify non-default `requests` and `limits` for engine pod | +| `tolerations` | array | Add tolerations to engine pod | +| `annotations` | object | Add custom annotations to engine pod (empty by default `{}`) | +| `labels` | object | Add custom labels to engine pod (empty by default `{}`) | +| `dnsPolicy` | string | Engine pod's [DNS policy](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy) | +| `dnsConfig` | object | Engine pod's [DNS config](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config) | + +`runtimeScheduler` example: +{% highlight yaml %} +{% raw %} +runtimeScheduler: + imagePullPolicy: Always + cluster: + clusterProvider: + accountId: 5f048d85eb107d52b16c53ea + selector: my-eks-cluster + nodeSelector: #schedule engine pod onto a node whose labels match the nodeSelector + node-type: engine + namespace: codefresh + serviceAccount: codefresh-engine + annotations: {} + labels: + spotinst.io/restrict-scale-down: "true" #optional label to prevent node scaling down when the runner is deployed on spot instances using spot.io + envVars: + NODE_TLS_REJECT_UNAUTHORIZED: '0' #disable certificate validation for TLS connections (e.g. to g.codefresh.io) + METRICS_PROMETHEUS_ENABLED: 'true' #enable /metrics on engine pod + DEBUGGER_TIMEOUT: '30' #debug mode timeout duration (in minutes) + userEnvVars: + - name: GITHUB_TOKEN + valueFrom: + secretKeyRef: + name: github-token + key: token + resources: + requests: + cpu: 60m + memory: 500Mi + limits: + cpu: 1000m + memory: 2048Mi + tolerations: + - effect: NoSchedule + key: codefresh.io + operator: Equal + value: engine +{% endraw %} +{% endhighlight %} + +### dockerDaemonScheduler fields (dind) + +| Field name | Type | Value | +| -------------- |-------------------------| -------------------------| +| `dindImage` | string | Override default dind image | +| `type` | string | `DindPodPvc` | +| `envVars` | object | Override or add environment variables passed into the dind pod. See [IN-DIND cleaner]({{site.baseurl}}/docs/administration/codefresh-runner/#cleaners) | +| `userVolumeMounts` with `userVolumes` | object | Add volume mounts to the pipeline See [Custom Volume Mounts]({{site.baseurl}}/docs/administration/codefresh-runner/#custom-volume-mounts) | +| `cluster` | object | k8s related information (`namespace`, `serviceAccount`, `nodeSelector`) | +| `defaultDindResources` | object | Override `requests` and `limits` for dind pod (defaults are `cpu: 400m` and `memory:800Mi` ) | +| `tolerations` | array | Add tolerations to dind pod | +| `annotations` | object | Add custom annotations to dind pod (empty by default `{}`) | +| `labels` | object | Add custom labels to dind pod (empty by default `{}`) | +| `pvc` | object | Override default storage configuration for PersistentVolumeClaim (PVC) with `storageClassName`, `volumeSize`, `reuseVolumeSelector`. See [Volume Reusage Policy]({{site.baseurl}}/docs/administration/codefresh-runner/#volume-reusage-policy) | +| `dnsPolicy` | string | Dind pod's [DNS policy](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy) | +| `dnsConfig` | object | Dind pod's [DNS config](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config) | + +`dockerDaemonScheduler` example: +{% highlight yaml %} +{% raw %} +dockerDaemonScheduler: + cluster: + clusterProvider: + accountId: 5f048d85eb107d52b16c53ea + selector: my-eks-cluster + nodeSelector: #schedule dind pod onto a node whose labels match the nodeSelector + node-type: dind + namespace: codefresh + serviceAccount: codefresh-engine + annotations: {} + labels: + spotinst.io/restrict-scale-down: "true" #optional label to prevent node scaling down when the runner is deployed on spot instances using spot.io + userAccess: true + defaultDindResources: + requests: '' + limits: + cpu: 1000m + memory: 2048Mi + userVolumeMounts: + my-cert: + name: cert + mountPath: /etc/ssl/cert + readOnly: true + userVolumes: + my-cert: + name: cert + secret: + secretName: tls-secret + pvcs: + dind: + storageClassName: dind-local-volumes-runner-codefresh + volumeSize: 30Gi + reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id' + tolerations: + - key: codefresh.io + operator: Equal + value: dinds + effect: NoSchedule +{% endraw %} +{% endhighlight %} + +### Custom Global Environment Variables +You can add your own environment variables in the runtime environment, so that all pipeline steps will have access to it. A typical example would be a shared secret that you want to pass to the pipeline. + +Under the `runtimeScheduler` block you can add an additional element with named `userEnvVars` that follows the same syntax as [secret/environment variables](https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-environment-variables). + +`runtime.yaml` +{% highlight yaml %} +{% raw %} +... +runtimeScheduler: + userEnvVars: + - name: GITHUB_TOKEN + valueFrom: + secretKeyRef: + name: github-token + key: token +... +{% endraw %} +{% endhighlight %} + +### Custom Volume Mounts +You can add your own volume mounts in the runtime environment, so that all pipeline steps have access to the same set of external files. A typical example of this scenario is when you want to make a set of SSL certificates available to all your pipelines. Rather than manually download the certificates in each pipeline, you can provide them centrally on the runtime level. + +Under the `dockerDaemonScheduler` block you can add two additional elements with names `userVolumeMounts` and `userVolumes` (they follow the same syntax as normal k8s `volumes` and `volumeMounts`) and define your own global volumes. + +`runtime.yaml` +{% highlight yaml %} +{% raw %} +... +dockerDaemonScheduler: + userVolumeMounts: + my-cert: + name: cert + mountPath: /etc/ssl/cert + readOnly: true + userVolumes: + my-cert: + name: cert + secret: + secretName: tls-secret +... +{% endraw %} +{% endhighlight %} + +### Debug Timeout Duration + +The default timeout for [debug mode]({{site.baseurl}}/docs/configure-ci-cd-pipeline/debugging-pipelines/) is 14 minutes, and even if the user is actively working, it is still 14 minutes. To change the duration of the debugger, you will need to update your Runtime Spec for the runtime you would like to change. To change the default duration, you will need to add `DEBUGGER_TIMEOUT` to the environment variable. The value you pass is a string value that will define the timeout in minutes. For example, you can pass '30', which will be 30 minutes. + +Under `.runtimeScheduler`, add an `envVars` section, then add `DEBUGGER_TIMEOUT` under `envVars` with the value you want. + +```yaml +... +runtimeScheduler: + envVars: + DEBUGGER_TIMEOUT: '30' +... +``` + +### Volume Reusage Policy + +The behavior of how the volumes are reused depends on volume selector configuration. +`reuseVolumeSelector` option is configurable in runtime environment spec. + +The following options are available: + +* `reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName'` - determined PV can be used by **ANY** pipeline of your account (it's a **default** volume selector). +* `reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id'` - determined PV can be used only by a **single pipeline**. +* `reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id,io.codefresh.branch_name'` - determined PV can be used only by **single pipeline AND single branch**. +* `reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id,trigger'` - determined PV can be used only by **single pipeline AND single trigger**. + +For approach `codefresh-app,io.codefresh.accountName`: + +* Benefit: less PVs --> lower cost (since any PV can be used by any pipeline, then, the cluster would need to keep less PVs in its pool of PVs for Codefresh) +* Downside: since the PV can be used by any pipeline, then, the PVs could have assets and info from different pipelines, thus reducing the probability of cache, + +For approach `codefresh-app,io.codefresh.accountName,pipeline_id`: + +* Benefit: more probability of cache (no "spam" from other pipelines) +* Downside: more PVs to keep (higher cost) + + +To change volume selector get runtime yaml spec and under `dockerDaemonScheduler.pvcs.dind` block specify `reuseVolumeSelector`: + +```yaml + pvcs: + dind: + volumeSize: 30Gi + reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id' +``` + +## Runtime Cleaners + +### Key points + +* Codefresh pipelines require disk space for: + * [Pipeline Shared Volume](https://codefresh.io/docs/docs/yaml-examples/examples/shared-volumes-between-builds/) (`/codefresh/volume`, implemented as [docker volume](https://docs.docker.com/storage/volumes/)) + * Docker containers - running and stopped + * Docker images and cached layers +* To improve performance, `volume-provisioner` is able to provision previously used disk with docker images and pipeline volume from previously running builds. It improves performance by using docker cache and decreasing I/O rate. +* Least recently docker images and volumes should be cleaned to avoid out-of-space errors. +* There are several places where pipeline volume cleanup is required, so there are several kinds of cleaner. + +### Cleaners + +* [IN-DIND cleaner](https://github.com/codefresh-io/dind/tree/master/cleaner) - deletes extra docker containers, volumes, images in **dind pod** +* [External volumes cleaner](https://github.com/codefresh-io/runtime-cluster-monitor/blob/master/charts/cf-monitoring/templates/dind-volume-cleanup.yaml) - deletes unused **external** PVs (EBS, GCE/Azure disks) +* [Local volumes cleaner](https://github.com/codefresh-io/dind-volume-utils/blob/master/local-volumes/lv-cleaner.sh) - deletes **local** volumes in case node disk space is close to the threshold + +*** + +#### IN-DIND cleaner + +**Purpose:** Removes unneeded *docker containers, images, volumes* inside kubernetes volume mounted to the dind pod + +**Where it runs:** Running inside each dind pod as script + +**Triggered by:** SIGTERM and also during the run when disk usage (cleaner-agent ) > 90% (configurable) + +**Configured by:** Environment Variables which can be set in Runtime Environment configuration + +**Configuration/Logic:** [README.md](https://github.com/codefresh-io/dind/tree/master/cleaner#readme) + +Override `dockerDaemonScheduler.envVars` on Runtime Environment if necessary (the following are **defaults**): + +```yaml +dockerDaemonScheduler: + envVars: + CLEAN_PERIOD_SECONDS: '21600' # launch clean if last clean was more than CLEAN_PERIOD_SECONDS seconds ago + CLEAN_PERIOD_BUILDS: '5' # launch clean if last clean was more CLEAN_PERIOD_BUILDS builds since last build + IMAGE_RETAIN_PERIOD: '14400' # do not delete docker images if they have events since current_timestamp - IMAGE_RETAIN_PERIOD + VOLUMES_RETAIN_PERIOD: '14400' # do not delete docker volumes if they have events since current_timestamp - VOLUMES_RETAIN_PERIOD + DISK_USAGE_THRESHOLD: '0.8' # launch clean based on current disk usage DISK_USAGE_THRESHOLD + INODES_USAGE_THRESHOLD: '0.8' # launch clean based on current inodes usage INODES_USAGE_THRESHOLD +``` + +*** + +#### External volumes cleaner + +**Purpose:** Removes unused *kubernetes volumes and related backend volumes* + +**Where it runs:** On Runtime Cluster as CronJob +(`kubectl get cronjobs -n codefresh -l app=dind-volume-cleanup`). Installed in case the Runner uses non-local volumes (`Storage.Backend != local`) + +**Triggered by:** CronJob every 10min (configurable), part of [runtime-cluster-monitor](https://github.com/codefresh-io/runtime-cluster-monitor/blob/master/charts/cf-monitoring/templates/dind-volume-cleanup.yaml) and runner deployment + +**Configuration:** + +Set `codefresh.io/volume-retention` annotation on Runtime Environment: + +```yaml +dockerDaemonScheduler: + pvcs: + dind: + storageClassName: dind-ebs-volumes-runner-codefresh + reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id' + volumeSize: 32Gi + annotations: + codefresh.io/volume-retention: 7d +``` + +Override environment variables for `dind-volume-cleanup` cronjob if necessary: + +* `RETENTION_DAYS` (defaults to 4) +* `MOUNT_MIN` (defaults to 3) +* `PROVISIONED_BY` (defaults to `codefresh.io/dind-volume-provisioner`) + +About *optional* `-m` argument: + +* `dind-volume-cleanup` to clean volumes that were last used more than `RETENTION_DAYS` ago +* `dind-volume-cleanup-m` to clean volumes that were used more than a day ago, but mounted less than `MOUNT_MIN` times + +*** + +#### Local volumes cleaner + +**Purpose:** Deletes local volumes in case node disk space is close to the threshold + +**Where it runs:** On each node on runtime cluster as DaemonSet `dind-lv-monitor`. Installed in case the Runner use local volumes (`Storage.Backend == local`) + +**Triggered by:** Starts clean if disk space usage or inodes usage is more than thresholds (configurable) + +**Configuration:** + +Override environment variables for `dind-lv-monitor` daemonset if necessary: + +* `VOLUME_PARENT_DIR` - default `/var/lib/codefresh/dind-volumes` +* `KB_USAGE_THRESHOLD` - default 80 (percentage) +* `INODE_USAGE_THRESHOLD` - default 80 + +## ARM Builds + +With hybrid runner it's possibe to run native ARM64v8 builds. + +>**Note:** Running both amd64 and arm64 images within the same pipeline - it is not possible. We do not support multi-architecture builds. One runtime configuration - one architecture. Considering one pipeline can map only to one runtime, it is possible to run either amd64 or arm64, but not both within a one pipeline + +The following scenario is an example of how to set up ARM Runner on existing EKS cluster: + +**Step 1 - Preparing nodes** + +Create new ARM nodegroup: + +```shell +eksctl utils update-coredns --cluster +eksctl utils update-kube-proxy --cluster --approve +eksctl utils update-aws-node --cluster --approve + +eksctl create nodegroup \ +--cluster \ +--region \ +--name \ +--node-type \ +--nodes <3>\ +--nodes-min <2>\ +--nodes-max <4>\ +--managed +``` + +Check nodes status: + +```shell +kubectl get nodes -l kubernetes.io/arch=arm64 +``` + +Also it's recommeded to label and taint the required ARM nodes: + +```shell +kubectl taint nodes arch=aarch64:NoSchedule +kubectl label nodes arch=arm +``` + +**Step 2 - Runner installation** + +Use [values.yaml](https://github.com/codefresh-io/venona/blob/release-1.0/venonactl/example/values-example.yaml) to inject `tolerations`, `kube-node-selector`, `build-node-selector` into the Runtime Environment spec. + +`values-arm.yaml` + +```yaml +... +Namespace: codefresh + +### NodeSelector --kube-node-selector: controls runner and dind-volume-provisioner pods +NodeSelector: arch=arm + +### Tolerations --tolerations: controls runner, dind-volume-provisioner and dind-lv-monitor +Tolerations: +- key: arch + operator: Equal + value: aarch64 + effect: NoSchedule +... +######################################################## +### Codefresh Runtime ### +### ### +### configure engine and dind pods ### +######################################################## +Runtime: +### NodeSelector --build-node-selector: controls engine and dind pods + NodeSelector: + arch: arm +### Tolerations for engine and dind pods + tolerations: + - key: arch + operator: Equal + value: aarch64 + effect: NoSchedule +... +``` + +Install the Runner with: + +```shell +codefresh runner init --values values-arm.yaml --exec-demo-pipeline false --skip-cluster-integration true +``` + +**Step 3 - Post-installation fixes** + +Change `engine` image version in Runtime Environment specification: + +```shell +# get the latest engine ARM64 tag +curl -X GET "https://quay.io/api/v1/repository/codefresh/engine/tag/?limit=100" --silent | jq -r '.tags[].name' | grep "^1.*arm64$" +1.136.1-arm64 +``` + +```shell +# get runtime spec +codefresh get re $RUNTIME_NAME -o yaml > runtime.yaml +``` + +under `runtimeScheduler.image` change image tag: + +```yaml +runtimeScheduler: + image: 'quay.io/codefresh/engine:1.136.1-arm64' +``` + +```shell +# patch runtime spec +codefresh patch re -f runtime.yaml +``` + +For `local` storage patch `dind-lv-monitor-runner` DaemonSet and add `nodeSelector`: + +```shell +kubectl edit ds dind-lv-monitor-runner +``` + +```yaml + spec: + nodeSelector: + arch: arm +``` + +**Step 4 - Run Demo pipeline** + +Run a modified version of the *CF_Runner_Demo* pipeline: + +```yaml +version: '1.0' +stages: + - test +steps: + test: + stage: test + title: test + image: 'arm64v8/alpine' + commands: + - echo hello Codefresh Runner! +``` + +## Troubleshooting + +For troubleshooting refer to the [Knowledge Base](https://support.codefresh.io/hc/en-us/sections/4416999487762-Hybrid-Runner) + +## What to read next + +* [Codefresh installation options]({{site.baseurl}}/docs/installation/installation-options/) +* [Codefresh On-Premises]({{site.baseurl}}/docs/administration/codefresh-on-prem/) +* [Codefresh API]({{site.baseurl}}/docs/integrations/codefresh-api/) diff --git a/_docs/runtime/git-sources.md b/_docs/installation/git-sources.md similarity index 68% rename from _docs/runtime/git-sources.md rename to _docs/installation/git-sources.md index 2b95dc54..b51913a8 100644 --- a/_docs/runtime/git-sources.md +++ b/_docs/installation/git-sources.md @@ -1,23 +1,23 @@ --- -title: "Add Git Sources to runtimes" +title: "Add Git Sources to GitOps Runtimes" description: "" -group: runtime +group: installation toc: true --- -A Git Source is the equivalent of an Argo CD application that tracks a Git repository and syncs the desired state of the repo to the destination K8s cluster. In addition to application resources, the Git Source can store resources for Codefresh runtimes, and CI/CD entities such as delivery pipelines, Workflow Templates, workflows, and applications. +A Git Source is the equivalent of an Argo CD application that tracks a Git repository and syncs the desired state of the repo to the destination K8s cluster. In addition to application resources, the Git Source can store resources for GitOps Runtimes, and CI/CD entities such as delivery pipelines, Workflow Templates, workflows, and applications. -Provisioning a runtime automatically creates a Git Source that stores resources for the runtime and for the demo CI pipelines that are optionally installed with the runtime. Every Git Source is associated with a Codefresh runtime. A runtime can have one or more Git Sources. You can add Git Sources at any time, to the same or to different runtimes. +Provisioning a Runtime automatically creates a Git Source that stores resources for the Runtime and for the demo CI pipelines that are optionally installed with the Runtime. Every Git Source is associated with a Runtime. You can add Git Sources at any time, to the same or to different Runtimes. -Once you create a Git Source for a runtime, you can store resources for CI/CD entities associated with that runtime. For example, when creating pipelines or applications, you can select the Git Source to which to store manifest definitions. +Once you create a Git Source for a Runtime, you can store resources for CI/CD entities associated with it. For example, when creating pipelines or applications, you can select the Git Source to which to store manifest definitions. ### View Git Sources and definitions Drill down on a runtime in List View to see its Git Sources. -1. In the Codefresh UI, go to the [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"} page. -1. From the **List View** (the default), select a runtime name, and then select the **Git Sources** tab. +1. In the Codefresh UI, go to the [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"} page. +1. From the **List View** (the default), select a Runtime name, and then select the **Git Sources** tab. {% include image.html @@ -34,12 +34,12 @@ Drill down on a runtime in List View to see its Git Sources. 1. To see the definitions for the Git Source, select the three dots at the end of the row. ### Create a Git Source -Create Git Sources for any provisioned runtime. The Git Sources are available to store resources for pipelines or applications when you create them. +Create Git Sources for any provisioned Runtime. The Git Sources are available to store resources for pipelines or applications when you create them. >Make sure you are in the List View to create Git Sources. -1. In the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes**){:target="\_blank"}. -1. In the List View, select the runtime for which to add a Git Source, and then select the **Git Sources** tab. +1. In the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes**){:target="\_blank"}. +1. In the List View, select the Runtime for which to add a Git Source, and then select the **Git Sources** tab. 1. Select **Create Git Sources**, and in the Create Git Source panel, define the definitions for the Git Source: {% include @@ -56,7 +56,7 @@ Create Git Sources for any provisioned runtime. The Git Sources are available t * **Source**: The Git repo with the desired state, tracked by the Git Source, and synced to the destination cluster. * **Repository**: Mandatory. The URL to the Git repo. * **Branch**: Optional. The specific branch within the repo to track. - * **Path**: Optional. The specific path within the repo, and branch, if one is specified, to track. + * **Path**: Optional. The specific path within the repo, and branch if one is specified, to track. * **Destination**: The destination cluster with the actual state to which to apply the changes from the **Source**. * **Namespace**: The namespace in the destination cluster to which to sync the changes. @@ -73,8 +73,8 @@ Create Git Sources for any provisioned runtime. The Git Sources are available t Edit an existing Git Source by changing the source and destination definitions. > You cannot change the name of the Git Source. -1. In the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes**){:target="\_blank"}. -1. From the **List View** (the default), select the runtime with the Git Source, and then select the **Git Sources** tab. +1. In the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes**){:target="\_blank"}. +1. From the **List View** (the default), select the Runtime with the Git Source, and then select the **Git Sources** tab. 1. In the row with the Git Source to edit, select the three dots, and then select **Edit** in the panel that appears. {% include @@ -90,12 +90,12 @@ Edit an existing Git Source by changing the source and destination definitions. 1. Change the **Source** and **Destination** definitions for the Git Source, and select **Save**. ### View/download logs for a Git Source -View online logs for any Git Source associated with a runtime, and if needed, download the log file for offline viewing and analysis. -Online logs show up to 1000 of the most recent events (lines), updated in real time. Downloaded logs include all the events from the application launch to the date and time of download. +View online logs for any Git Source associated with a Runtime, and if needed, download the log file for offline viewing and analysis. +Online logs show up to 1000 of the most recent events (lines), updated in real time. Downloaded logs include all the events, from the application launch to the date and time of download. -1. In the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes**){:target="\_blank"}. -1. From the **List View** (the default), select the runtime with the Git Source, and then select the **Git Sources** tab. -1. In the row with the Git Source foe which to view/download logs, select the three dots, and then select **View Logs**. +1. In the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes**){:target="\_blank"}. +1. From the **List View** (the default), select the Runtime with the Git Source, and then select the **Git Sources** tab. +1. In the row with the Git Source for which to view/download logs, select the three dots, and then select **View Logs**. {% include image.html @@ -127,6 +127,7 @@ Online logs show up to 1000 of the most recent events (lines), updated in real t The file is downloaded with `.log` extension. ### What to read next -[Manage runtimes]({{site.baseurl}}/docs/runtime/monitor-manage-runtimes/) -[Recover runtimes]({{site.baseurl}}/docs/runtime/runtime-recovery/) +[Monitoring & managing GitOps Runtimes]({{site.baseurl}}/docs/installation/monitor-manage-runtimes/) +[Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration) + diff --git a/_docs/runtime/hosted-runtime.md b/_docs/installation/hosted-runtime.md similarity index 66% rename from _docs/runtime/hosted-runtime.md rename to _docs/installation/hosted-runtime.md index 0a08ba3b..cfc64c7e 100644 --- a/_docs/runtime/hosted-runtime.md +++ b/_docs/installation/hosted-runtime.md @@ -1,18 +1,28 @@ --- -title: "Set up a hosted runtime environment" -description: "" -group: runtime +title: "Hosted GitOps Runtime setup" +description: "Provision Hosted GitOps environment" +group: installation toc: true --- -If you have Codefresh's Hosted GitOps, set up your hosted environment, and you are all ready to leverage extensive CD Ops capabilities. -Read about [Hosted GitOps]({{site.baseurl}}/docs/incubation/intro-hosted-runtime/). +Set up your hosted environment with the Hosted GitOps Runtime to leverage extensive CD capabilities. + -### Where to start with Hosted GitOps -If you have not provisioned a hosted runtime, Codefresh presents you with the setup instructions in the **Home** dashboard. +## System requirements for Hosted GitOps Runtimes +{: .table .table-bordered .table-hover} +| Item | Requirement | +| -------------- | -------------- | +|Kubernetes cluster | Server version 1.18 and higher to which to deploy applications| +|Git provider | {::nomarkdown}
Tip: To check the server version, run:
kubectl version --short.{:/}| +| Ingress controller| Configured on Kubernetes cluster and exposed from the cluster. {::nomarkdown}
Supported and tested ingress controllers include: