-
- Steps 2 to 7 occur in the background and are transparent to the user.
-
-
-1. A user logs in to CDSP
-2. The user is redirected to Codefresh Service Provider (SP) to initiate SSO
-3. The user’s browser is then redirected to the customer IdP
-4. Once authenticated by the corporate side, a SAML token is sent to the user’s browser
-5. The SAML assertion is then forwarded to Codefresh SP
-6. If you are a valid Codefresh user for this SSO connection, an SSO token is returned to the user’s browser
-7. The user’s browser then returns a token to Codefresh and access is granted for your account
-
-### Configure SAML SSO settings in Codefresh
-
-1. In Codefresh, select **Account settings**.
-1. From the sidebar expand **Collaboration**, and select **Single Sign-on**.
- OR
- Go directly to [https://g.codefresh.io/account-admin/sso](https://g.codefresh.io/account-admin/sso))
-
-
- {% include image.html
- lightbox="true"
-file="/images/administration/sso/add-sso-dropdown.png"
-url="/images/administration/sso/add-sso-dropdown.png"
-alt="SSO provider settings"
-caption="SSO provider settings"
-max-width="70%"
-%}
-
-{:start="3"}
-1. Select **Add single-sign-on**, and then select **SAML**.
-1. Enter the following:
-
- * **Client Name**: For auto-generation, leave empty. Codefresh generates the client name once you save the settings.
- * **Display Name**: The name you want to give to this integration.
- * **IDP Entry**: The SSO endpoint of your Identity Provider. For Azure SAML, for example, this is the Login URL.
- * **Application Certificate**: The security certificate of your Identity Provider. Paste the value directly in the field. Do not convert to base64 or any other encoding by hand. (For Azure SAML, this will be Certificate (Base64) and the value needed is between the -----BEGIN ... and -----END... from the downloaded cert)
- * **Assertion URL**: `https://g.codefresh.io/api/auth/+ +Here is another example where two different applications use Node.js 11 and Node.js 9 in the same pipeline. + +`codefresh.yml` +{% highlight yaml %} +version: '1.0' +stages: + - packaging + - deploying +steps: + PackageMyNode1App: + title: Packaging Node application 1 + stage: packaging + image: node:11.1 + working_directory: ./brand-new-project + commands: + - echo "My Node version is" + - node --version + - npm install + PackageMyNode2App: + title: Packaging Node application 2 + stage: packaging + image: node:9.3.0-slim + working_directory: ./legacy-project + commands: + - echo "My Node version is" + - node --version + - npm install +{% endhighlight %} + +> These versions are per pipeline. So each team can use the versions they need for their projects without affecting the other teams. + +So one team in your company might use Terraform 0.10 in their pipelines: + + +{% highlight yaml %} + PlanWithTerraform: + image: hashicorp/terraform:0.10.0 + title: Deploying Terraform plan + stage: deploy + commands: + - terraform plan +{% endhighlight %} + +Another team can use Terraform 0.12 just by changing the YAML of their `codefresh.yml`: + +{% highlight yaml %} + DeployWithTerraform: + image: hashicorp/terraform:0.12.0 + title: Deploying Terraform plan + stage: deploy + commands: + - terraform apply -auto-approve +{% endhighlight %} + + +To summarize, you can easily use any version of any programming tool in a Codefresh pipeline without the fear of breaking +another unrelated pipeline. + + +## Related articles +[Introduction to Codefresh pipelines]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) +[Creating Codefresh pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) + + + + + + diff --git a/_docs/ci-cd-guides/preview-environments.md b/_docs/ci-cd-guides/preview-environments.md new file mode 100644 index 000000000..15e6ee2d4 --- /dev/null +++ b/_docs/ci-cd-guides/preview-environments.md @@ -0,0 +1,347 @@ +--- +title: "Previewing dynamic environments" +description: "Deploy pull requests to cluster namespaces" +group: ci-cd-guides +toc: true +--- + + +In addition to deploying to [predefined environments]({{site.baseurl}}/docs/ci-cd-guides/environment-deployments/), for each pull request (PR), you may also need to deploy to dynamic environments, which are temporary, testing environments. For these types of environments, it is best to dynamically create an environment when a PR is created, and tear it down when the same PR is closed. + + +{% include image.html +lightbox="true" +file="/images/guides/preview-environments/dynamic-environments.png" +url="/images/guides/preview-environments/dynamic-environments.png" +alt="Dynamic Test environments" +caption="Dynamic Test environments" +max-width="90%" +%} + +Each developer works in isolation to test their features. This pattern contrasts with the traditional way of reusing static preexisting environments. + +{% include image.html +lightbox="true" +file="/images/guides/preview-environments/static-environments.png" +url="/images/guides/preview-environments/static-environments.png" +alt="Traditional static environments" +caption="Traditional static environments" +max-width="90%" +%} + +With Kubernetes you don't need to book and release specific test environments any more. Testing environments should +be handled in a transient way. + +## Preview environments with Kubernetes + +There are many options to create temporary environments with Kubernetes. + +* Namespaces for each PR + The simplest option is to use different namespaces, one for each PR. So, a PR with name `fix-db-query` is deployed to a namespace called `fix-db-query`, and a PR with name `JIRA-1434`, is deployed to a namespace called `JIRA-1434` and so on. + +* Expose the environment URL + The second option is to expose the environment URL so that developers and testers can actually preview the application +deployment either manually or via automated tests. + The two major approaches here are with host-based and path-based URLs: + * For host-based URLs, the test environments are named `pr1.example.com`, `pr2.example.com` and so on + * For path-based URLs, the test environments are named `example.com/pr1`, `example.com/pr2` and so on + + Both approaches have advantages and disadvantages. Path-based URLs are easier to set up, but may not work with all applications, as they change the web context. Host-based URLs are more robust but need extra DNS configuration for the full effect. + + In Kubernetes clusters, you can set up types of URLs via [an Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/){:target="\_blank"}. + +## Example application + +You can find the application we will use at [https://github.com/codefresh-contrib/unlimited-test-environments-source-code](https://github.com/codefresh-contrib/unlimited-test-environments-source-code){:target="\_blank"}. +It is a standard Java/Spring boot application, that includes the following characteristics: + +* It has [integration tests]({{site.baseurl}}/docs/testing/integration-tests/) that can be targeted at any host/port. We will use those tests as smoke test that will verify the preview environment after it is deployed +* It comes bundled in [a Helm chart](https://github.com/codefresh-contrib/unlimited-test-environments-manifests){:target="\_blank"} +* It has an ingress configuration ready for path-based URLs + +We are using [the Ambassador gateway](https://www.getambassador.io/){:target="\_blank"} as an ingress for this example, but you can use any Kubernetes-compliant ingress. + +Here is the [ingress manifest](https://github.com/codefresh-contrib/unlimited-test-environments-manifests/blob/main/simple-java-app/templates/ingress.yaml){:target="\_blank"}. + +{% highlight yaml %} +{% raw %} +kind: Ingress +apiVersion: extensions/v1beta1 +metadata: + name: "simple-java-app-ing" + annotations: + kubernetes.io/ingress.class: {{ .Values.ingress.class }} + +spec: + rules: + - http: + paths: + - path: {{ .Values.ingress.path }} + backend: + serviceName: simple-service + servicePort: 80 +{% endraw %} +{% endhighlight %} + +The path of the application is configurable and can be set at deploy time. + +## Creating preview environments for each PR + +Each time a PR is created, we want to perform the following tasks: + +1. Compile the application and run unit tests. +1. Run security scans, quality checks, and everything else we need to decide if the PR is valid. +1. Create a namespace with the same name as the PR branch. Deploy the PR and expose it as a URL that has the same name as the branch. + +Here is an example pipeline that does all these tasks: + +{% include image.html +lightbox="true" +file="/images/guides/preview-environments/pull-request-preview-pipeline.png" +url="/images/guides/preview-environments/pull-request-preview-pipeline.png" +alt="Pull Request preview pipeline" +caption="Pull Request preview pipeline" +max-width="100%" +%} + +This pipeline has the following steps: + +1. A [clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/) to fetch the source code of the application. +1. A [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/) that runs Maven for compilation and unit tests. +1. A [build step]({{site.baseurl}}/docs/pipelines/steps/build/) to create the Docker image of the application. +1. A step that scans the source code for security issues with [Snyk](https://snyk.io/){:target="\_blank"}. +1. A step that scans the container image [for security issues]({{site.baseurl}}/docs/testing/security-scanning/) with [trivy](https://github.com/aquasecurity/trivy){:target="\_blank"}. +1. A step that runs [integration tests]({{site.baseurl}}/docs/testing/integration-tests/) by launching the app in a [service container]({{site.baseurl}}/docs/pipelines/service-containers/). +1. A step for [Sonar analysis]({{site.baseurl}}/docs/testing/sonarqube-integration/). +1. A step that clones [a second Git repository](https://github.com/codefresh-contrib/unlimited-test-environments-manifests){:target="\_blank"} with the [Helm chart]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/) of the application. +1. A step that deploys the source code to a new namespace. +1. A step that [adds a comment on the PR](https://codefresh.io/steps/step/kostis-codefresh%2Fgithub-pr-comment){:target="\_blank"} with the URL of the temporary environment. +1. A step that runs smoke tests against the temporary test environment. + +Note that the integration tests and security scans are just examples of what you can do before the PR is deployed. You can insert your own steps that check the content of a PR. + +Here is the complete YAML definition: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +stages: + - "prepare" + - "verify" + - "deploy" + +steps: + main_clone: + title: "Cloning repository" + type: "git-clone" + repo: "codefresh-contrib/unlimited-test-environments-source-code" + revision: "${{CF_REVISION}}" + stage: "prepare" + + run_unit_tests: + title: Compile/Unit test + stage: prepare + image: 'maven:3.5.2-jdk-8-alpine' + commands: + - mvn -Dmaven.repo.local=/codefresh/volume/m2_repository package + build_app_image: + title: Building Docker Image + type: build + stage: prepare + image_name: kostiscodefresh/spring-actuator-sample-app + working_directory: ./ + tag: '${{CF_BRANCH}}' + dockerfile: Dockerfile + scan_code: + title: Source security scan + stage: verify + image: 'snyk/snyk-cli:maven-3.6.3_java11' + commands: + - snyk monitor + scan_image: + title: Container security scan + stage: verify + image: 'aquasec/trivy' + commands: + - trivy image docker.io/kostiscodefresh/spring-actuator-sample-app:${{CF_BRANCH}} + run_integration_tests: + title: Integration tests + stage: verify + image: maven:3.5.2-jdk-8-alpine + commands: + - mvn -Dmaven.repo.local=/codefresh/volume/m2_repository verify -Dserver.host=http://my-spring-app -Dsonar.organization=kostis-codefresh-github + services: + composition: + my-spring-app: + image: '${{build_app_image}}' + ports: + - 8080 + readiness: + timeoutSeconds: 30 + periodSeconds: 15 + image: byrnedo/alpine-curl + commands: + - "curl http://my-spring-app:8080/" + sonar_scan: + title: Sonar Scan + stage: verify + image: 'maven:3.8.1-jdk-11-slim' + commands: + - mvn -Dmaven.repo.local=/codefresh/volume/m2_repository sonar:sonar -Dsonar.login=${{SONAR_TOKEN}} -Dsonar.host.url=https://sonarcloud.io -Dsonar.organization=kostis-codefresh-github + clone: + title: "Cloning repository" + type: "git-clone" + repo: "codefresh-contrib/unlimited-test-environments-manifests" + revision: main + stage: "deploy" + deploy: + title: Deploying Helm Chart + type: helm + stage: deploy + working_directory: ./unlimited-test-environments-manifests + arguments: + action: install + chart_name: simple-java-app + release_name: my-spring-app + helm_version: 3.2.4 + kube_context: myawscluster + namespace: ${{CF_BRANCH_TAG_NORMALIZED}} + cmd_ps: '--create-namespace --wait --timeout 5m' + custom_values: + - 'image_tag=${{CF_BRANCH_TAG_NORMALIZED}}' + - 'replicaCount=3' + - 'ingress_path=/${{CF_BRANCH_TAG_NORMALIZED}}/' + add_pr_comment: + title: Adding comment on PR + stage: deploy + type: kostis-codefresh/github-pr-comment + fail_fast: false + arguments: + PR_COMMENT_TEXT: "[CI] Staging environment is at https://kostis.sales-dev.codefresh.io/${{CF_BRANCH_TAG_NORMALIZED}}/" + GIT_PROVIDER_NAME: 'github-1' + run_smoke_tests: + title: Smoke tests + stage: deploy + image: maven:3.5.2-jdk-8-alpine + working_directory: "${{main_clone}}" + fail_fast: false + commands: + - mvn -Dmaven.repo.local=/codefresh/volume/m2_repository verify -Dserver.host=https://kostis.sales-dev.codefresh.io/${{CF_BRANCH_TAG_NORMALIZED}}/ -Dserver.port=443 +{% endraw %} +{% endhighlight %} + +The end result of the pipeline is a deployment to the path that has the same name as the PR branch. For +example, if my branch is named `demo`, then a `demo` namespace is created on the cluster and the application +is exposed on the `/demo/` context: + +{% include image.html +lightbox="true" +file="/images/guides/preview-environments/demo-path.png" +url="/images/guides/preview-environments/demo-path.png" +alt="Temporary environment" +caption="Temporary environment" +max-width="100%" +%} + +The environment is also mentioned as a comment in the PR UI in GitHub: + +{% include image.html +lightbox="true" +file="/images/guides/preview-environments/pull-request-comment.png" +url="/images/guides/preview-environments/pull-request-comment.png" +alt="Pull Request comment" +caption="Pull Request comment" +max-width="100%" +%} + +As explained in [pull Requests]({{site.baseurl}}/docs/ci-cd-guides/pull-request-branches/), we want to make this pipeline applicable only +to a PR-open event and PR-sync events that capture commits on an existing pull request. + +{% include image.html +lightbox="true" +file="/images/guides/preview-environments/pr-events.png" +url="/images/guides/preview-environments/pr-events.png" +alt="Git events for a Pull Request preview pipeline" +caption="Git events for a Pull Request preview pipeline" +max-width="100%" +%} + +Therefore, you need to set up your [pipeline triggers]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/) with the same options selected as shown in the picture above. + +## Cleaning up temporary environments + +Creating temporary environments is very convenient for developers, but can be very costly for your infrastructure if you use a cloud +provider for your cluster. For cost reasons and better resource utilization, it is best to destroy temporary environments that are no longer used. + +While you can run a batch job that automatically deletes old temporary environments, the optimal approach is to delete them as soon as the respective PR is closed. + +We can do that with a very simple pipeline that has only one step: + +{% include image.html +lightbox="true" +file="/images/guides/preview-environments/pull-request-closed-pipeline.png" +url="/images/guides/preview-environments/pull-request-closed-pipeline.png" +alt="Pipeline when a Pull Request is closed" +caption="Pipeline when a Pull Request is closed" +max-width="100%" +%} + +Here is the pipeline definition: + + `codefresh-close.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +steps: + delete_app: + title: Delete app + type: helm + arguments: + action: auth + helm_version: 3.2.4 + kube_context: myawscluster + namespace: ${{CF_BRANCH_TAG_NORMALIZED}} + commands: + - helm delete my-spring-app --namespace ${{CF_BRANCH_TAG_NORMALIZED}} + - kubectl delete namespace ${{CF_BRANCH_TAG_NORMALIZED}} +{% endraw %} +{% endhighlight %} + +The pipeline just uninstalls the Helm release for that namespace, and then deletes the namespace itself. + +To have this pipeline run only when a PR is closed, here are the [triggers]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/) to select: + +{% include image.html +lightbox="true" +file="/images/guides/preview-environments/close-events.png" +url="/images/guides/preview-environments/close-events.png" +alt="Git events for a Pull Request close pipeline" +caption="Git events for a Pull Request close pipeline" +max-width="100%" +%} + +With this setup, the pipeline runs when the PR is closed, regardless of whether it was merged or not (which is exactly what you want as in both cases the test environment is not needed anymore). + +## Viewing all environments in the Codefresh UI + +You can combine the pipeline above with any Codefresh UI dashboard if you want to see all your temporary environments in a single view. + +For more information, see: +* [Environment dashboard]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/) +* [Helm promotion dashboard]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion/) +* [GitOps dashboard]({{site.baseurl}}/docs/ci-cd-guides/gitops-deployments/#working-with-the-gitops-dashboard) + + + +## Related articles +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Working with Docker registries]({{site.baseurl}}/docs/integrations/docker-registries/) + + + + + + diff --git a/_docs/ci-cd-guides/progressive-delivery.md b/_docs/ci-cd-guides/progressive-delivery.md new file mode 100644 index 000000000..11780fdce --- /dev/null +++ b/_docs/ci-cd-guides/progressive-delivery.md @@ -0,0 +1,958 @@ +--- +title: "Progressive Delivery" +description: "Perform zero downtime deployments with Argo Rollouts" +group: ci-cd-guides +toc: true +--- + +Progressive Delivery is the practice of deploying an application in a gradual manner, allowing for minimum downtime and easy rollbacks. There are several forms of progressive delivery such as blue/green, canary, a/b, and feature flags. + +Codefresh can easily integrate with [Argo Rollouts](https://argoproj.github.io/argo-rollouts/){:target="\_blank"}, a Kubernetes operator that natively supports deployment practices for progressive delivery. + +## Installing the Argo Rollouts operator to your cluster + +To install Argo Rollouts, follow the [installation instructions](https://argoproj.github.io/argo-rollouts/installation/){:target="\_blank"}. Essentially, you need a terminal with `kubectl` access to your cluster. + +``` +kubectl create namespace argo-rollouts +kubectl apply -n argo-rollouts -f https://raw.githubusercontent.com/argoproj/argo-rollouts/stable/manifests/install.yaml +``` + +You can optionally install the [CLI locally](https://github.com/argoproj/argo-rollouts/releases/latest){:target="\_blank"}, if you want to have more visibility in your deployments. + + +## Blue/Green deployments + +Blue/Green deployments are one of the simplest ways to minimize deployment downtime. Blue/Green deployments are not specific to Kubernetes, and can be used even for traditional applications that reside on Virtual Machines. + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/how-blue-green-works.png" +url="/images/guides/progressive-delivery/how-blue-green-works.png" +alt="Blue/Green Deployments" +caption="Blue/Green Deployments" +max-width="50%" +%} + +1. At first all users of the application are routed to the current version (shown in blue). A key point is that all traffic passes from a load balancer. +1. A new version is deployed (shown in green). As this version does not receive any live traffic, all users are still served by the previous/stable version. +1. Developers can test internally the new green version, and verify its validity. If it is valid, traffic is switched to that new version. +1. If everything goes well, the old version is completely discarded. We are back to the initial state (order of colors does not matter). + +The major benefit of this pattern is that if at any point in time the new version has issues, all users can be switched back to the previous version (via the load balancer). Switching via the load balancer is much faster than redeploying a new version, resulting in minimum disruption for existing users. + +There are several variations of this pattern. In some cases, the old color is never destroyed but keeps running in the background. You can also retain even older versions online, maybe with a smaller footprint, allowing for easy switching to any previous application revision. + +### Blue/Green Kubernetes Deployment with Argo Rollouts + +Even though Argo Rollouts supports the basic blue/green pattern described in the previous section, it also offers a wealth of [customization options](https://argoproj.github.io/argo-rollouts/features/bluegreen/){:target="\_blank"}. +One of the most important additions is the ability to "test" the upcoming color by introducing a "preview" [Kubernetes service](https://kubernetes.io/docs/concepts/services-networking/service/){:target="\_blank"}, in addition to the service used for live traffic. +This preview service can be used by the team that performs the deployment to verify the new version before actually switching the traffic. + + +Here is the initial state of a deployment. The example uses two pods (shown as `xnsdx` and `jftql` in the diagram). + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/01_initial.png" +url="/images/guides/progressive-delivery/01_initial.png" +alt="Initial deployment. All services point to active version" +caption="Initial deployment. All services point to active version" +max-width="90%" +%} + +There are two Kubernetes services: +* A `rollout-blue-gree-active` service that captures all live traffic from actual users of the application (internet traffic coming from `51.141.221.40`). +* A secondary service called `rollout-bluegreen-preview`. Under normal circumstances it also points to the same live version. + + +Once a deployment starts, a new "color" is created. In the example we have two new pods that represent the next version of the application to be deployed (shown as `9t67t` and `7vs2m`). + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/02_two_colors.png" +url="/images/guides/progressive-delivery/02_two_colors.png" +alt="Deployment in progress. Active users see old version. Internal users preview new version" +caption="Deployment in progress. Active users see old version. Internal users preview new version" +max-width="90%" +%} + +The important point here is the fact that the normal "active" service still points to the old version, while the "preview" service points to the new pods. This means that all active users are still on the old/stable deployment, while internal teams can use the "preview" service to test the new deployment. + +If everything goes well, the next version is promoted to be the active version. + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/03_switch_traffic.png" +url="/images/guides/progressive-delivery/03_switch_traffic.png" +alt="Next application version is promoted. All users see new version" +caption="Next application version is promoted. All users see new version" +max-width="90%" +%} + +Here both services point to the new version. This is also the critical moment for all actual users of the application, as they are now switched to use the new version of the application. The old version is still around but no traffic is sent to it. + +Having the old version around is a great failsafe, as one can abort the deployment process and switch back all active users to the old deployment in the fastest way possible. + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/04_scale_down.png" +url="/images/guides/progressive-delivery/04_scale_down.png" +alt="Old application version is discarded. Only new version remains." +caption="Old application version is discarded. Only new version remains." +max-width="90%" +%} + +After the configured duration, as [defined in Argo Rollouts](https://argoproj.github.io/argo-rollouts/features/bluegreen/#scaledowndelayseconds){:target="\_blank"}, the old version is scaled down completely to preserve resources. We are now back +to the same configuration as the initial state, and the next deployment will follow the same sequence of events. + + +### Example application + +You can find an example application at [https://github.com/codefresh-contrib/argo-rollout-blue-green-sample-app](https://github.com/codefresh-contrib/argo-rollout-blue-green-sample-app){:target="\_blank"}, that also includes simple integration tests. + +Notice that the first deployment of your application will NOT follow the blue/green deployment process as there is no "previous" color. So you need to deploy it at least once. + +``` +git clone https://github.com/codefresh-contrib/argo-rollout-blue-green-sample-app.git +cd argo-rollout-blue-green-sample-app +kubectl create ns blue-green +kubectl apply -f ./blue-green-manual-approval -n blue-green +``` + +You can then monitor what Argo Rollouts is doing with the following command: + +``` +kubectl argo rollouts get rollout spring-sample-app-deployment --watch -n blue-green +``` + +### Blue/Green deployment with manual approval + +A quick way to use blue/green deployments is by simply inserting [an approval step]({{site.baseurl}}/docs/pipelines/steps/approval/) before the traffic switch step. +This will pause the pipeline and the developers or QA can test the next version on their own before any live users are redirected to it. + +Here is an example pipeline: + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/approval-pipeline.png" +url="/images/guides/progressive-delivery/approval-pipeline.png" +alt="Manual approval before traffic switch" +caption="Manual approval before traffic switch" +max-width="100%" +%} + +This pipeline does the following: + +1. [Clones]({{site.baseurl}}/docs/example-catalog/ci-examples/git-checkout/) the source code of the application. +1. [Builds]({{site.baseurl}}/docs/ci-cd-guides/building-docker-images/) a Docker image. +1. [Deploys]({{site.baseurl}}/docs/deployments/kubernetes/kubernetes-templating/) the application by updating the Kubernetes manifests. Argo Rollouts sees the new manifest, and creates a new "color" for the next version +1. The pipeline is paused and waits for an [approval/rejection]({{site.baseurl}}/docs/pipelines/steps/approval/#getting-the-approval-result) by a human user. +1. If the pipeline is approved, the new color is promoted, and becomes the new active version. +1. If the pipeline is rejected, the new color is discarded, and all live users are not affected in any way. + +Here is the [pipeline definition]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/): + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +stages: + - prepare + - build + - deploy + - finish +steps: + clone: + type: "git-clone" + stage: prepare + description: "Cloning main repository..." + repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}' + revision: "${{CF_BRANCH}}" + build_app_image: + title: Building Docker Image + type: build + stage: build + image_name: kostiscodefresh/argo-rollouts-blue-green-sample-app + working_directory: "${{clone}}" + tags: + - "latest" + - '${{CF_SHORT_REVISION}}' + dockerfile: Dockerfile + start_deployment: + title: Deploying new color + stage: deploy + image: codefresh/cf-deploy-kubernetes:master + working_directory: "${{clone}}" + commands: + - /cf-deploy-kubernetes ./blue-green-manual-approval/service.yaml + - /cf-deploy-kubernetes ./blue-green-manual-approval/service-preview.yaml + - /cf-deploy-kubernetes ./blue-green-manual-approval/rollout.yaml + environment: + - KUBECONTEXT=mydemoAkscluster@BizSpark Plus + - KUBERNETES_NAMESPACE=blue-green + wait_for_new_color: + fail_fast: false + type: pending-approval + title: Is the new color ok? + stage: deploy + promote_color: + title: Switching traffic to new color + stage: finish + image: kostiscodefresh/kubectl-argo-rollouts:latest + commands: + - /app/kubectl-argo-rollouts-linux-amd64 promote spring-sample-app-deployment -n blue-green --context "mydemoAkscluster@BizSpark Plus" + when: + steps: + - name: wait_for_new_color + on: + - approved + abort_deployment: + title: Keeping the existing color + stage: finish + image: kostiscodefresh/kubectl-argo-rollouts:latest + commands: + - /app/kubectl-argo-rollouts-linux-amd64 undo spring-sample-app-deployment -n blue-green --context "mydemoAkscluster@BizSpark Plus" + when: + steps: + - name: wait_for_new_color + on: + - denied +{% endraw %} +{% endhighlight %} + +Just before the approval, you can optionally execute the Argo Rollouts CLI to see what is happening behind the scenes: + +``` +kubectl argo rollouts get rollout spring-sample-app-deployment --watch -n blue-green +``` + +It should show the new color come up, but not accepting any traffic. + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/monitor-argo-rollouts.png" +url="/images/guides/progressive-delivery/monitor-argo-rollouts.png" +alt="Argo Rollouts CLI" +caption="Argo Rollouts CLI" +max-width="100%" +%} + +Once the deployment is complete, the old pods are destroyed after 30 seconds (this is the default value of Argo Rollouts). + + + +### Blue/Green deployment with smoke tests + +Using manual approval before promoting the new version is a great starting point. To truly achieve continuous deployment, one should automate the testing process and eliminate the human approval. + +There are many approaches on testing a release, and each organization will have a different set of "tests" that verify the next version of the software. Argo Rollouts +has [several integrations](https://argoproj.github.io/argo-rollouts/features/analysis/){:target="\_blank"} either with metric providers or [simple Kubernetes jobs](https://argoproj.github.io/argo-rollouts/analysis/job/){:target="\_blank"} that can run integration tests or collect metrics and decide if the next color should be promoted or not. + +Another alternative is to simply execute [integration tests]({{site.baseurl}}/docs/testing/integration-tests/) from within Codefresh. This is great if your integration tests need access to the source code or other external services that are accessible only to Codefresh. + +We can modify the previous pipeline to include automated smoke tests that are already part of the [example application](https://github.com/codefresh-contrib/argo-rollout-blue-green-sample-app/blob/main/src/test/java/sample/actuator/HealthIT.java){:target="\_blank"}. + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/smoke-tests-pipeline.png" +url="/images/guides/progressive-delivery/smoke-tests-pipeline.png" +alt="Smoke tests before traffic switch" +caption="Smoke tests before traffic switch" +max-width="100%" +%} + +This pipeline does the following: + +1. [Clones]({{site.baseurl}}/docs/examples/example-catalog/ci-examples/git-checkout/) the source code of the application. +1. [Builds]({{site.baseurl}}/docs/ci-cd-guides/building-docker-images/) a Docker image +1. [Deploys]({{site.baseurl}}/docs/deployments/kubernetes/kubernetes-templating/) the application by updating the Kubernetes manifests. Argo Rollouts sees the new manifest and creates a new "color" for the next version. +1. Runs integration tests against the "preview" service created by Argo Rollouts. Live users are still on the previous/stable version of the application. +1. If smoke tests pass, the new color is promoted and becomes the new active version. +1. If smoke tests fail, the new color is discarded and all live users are not affected in any way. + +Here is the [pipeline definition]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/): + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +stages: + - prepare + - build + - deploy + - finish +steps: + clone: + type: "git-clone" + stage: prepare + description: "Cloning main repository..." + repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}' + revision: "${{CF_BRANCH}}" + build_app_image: + title: Building Docker Image + type: build + stage: build + image_name: kostiscodefresh/argo-rollouts-blue-green-sample-app + working_directory: "${{clone}}" + tags: + - "latest" + - '${{CF_SHORT_REVISION}}' + dockerfile: Dockerfile + start_deployment: + title: Deploying new color + stage: deploy + image: codefresh/cf-deploy-kubernetes:master + working_directory: "${{clone}}" + commands: + - /cf-deploy-kubernetes ./blue-green-manual-approval/service.yaml + - /cf-deploy-kubernetes ./blue-green-manual-approval/service-preview.yaml + - /cf-deploy-kubernetes ./blue-green-manual-approval/rollout.yaml + environment: + - KUBECONTEXT=mydemoAkscluster@BizSpark Plus + - KUBERNETES_NAMESPACE=blue-green + run_integration_tests: + title: Smoke tests + stage: deploy + image: maven:3.5.2-jdk-8-alpine + working_directory: "${{clone}}" + fail_fast: false + commands: + - mvn -Dmaven.repo.local=/codefresh/volume/m2_repository verify -Dserver.host=http://13.86.102.74 -Dserver.port=80 + promote_color: + title: Switching traffic to new color + stage: finish + image: kostiscodefresh/kubectl-argo-rollouts:latest + commands: + - /app/kubectl-argo-rollouts-linux-amd64 promote spring-sample-app-deployment -n blue-green --context "mydemoAkscluster@BizSpark Plus" + when: + steps: + - name: run_integration_tests + on: + - success + abort_deployment: + title: Keeping the existing color + stage: finish + image: kostiscodefresh/kubectl-argo-rollouts:latest + commands: + - /app/kubectl-argo-rollouts-linux-amd64 undo spring-sample-app-deployment -n blue-green --context "mydemoAkscluster@BizSpark Plus" + when: + steps: + - name: run_integration_tests + on: + - failure +{% endraw %} +{% endhighlight %} + +You can optionally execute the Argo Rollouts CLI to see what is happening behind the scenes: + +``` +kubectl argo rollouts get rollout spring-sample-app-deployment --watch -n blue-green +``` + +>For the sake of simplicity, we have hardcoded the load balancer for the preview service at 13.86.102.74. For an actual application, you would have a DNS name such as `preview.example.com`, or use another `kubectl command` to fetch the endpoint of the load balancer dynamically. Also, our integration tests assume that the application is already deployed, before they run. If your application takes too much time to deploy, you need to make sure that it is up before the tests actually run. + + +The end result is a continuous deployment pipeline, where all release candidates that don't pass tests never reach production. + +## Canary deployments + +Blue/Green deployments are great for minimizing downtime after a deployment, but they are not perfect. If your new version has a hidden issue that manifests itself only after some time (i.e. it is not detected by your smoke tests), then **all** your users will be affected, because the traffic switch is all or nothing. + +An improved deployment method is canary deployments. These function similar to blue/green, but instead of switching 100% of live traffic all at once to the new version, you can instead move only a subset of users. + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/how-canary-deployments-work.png" +url="/images/guides/progressive-delivery/how-canary-deployments-work.png" +alt="Canary Deployments" +caption="Canary Deployments" +max-width="50%" +%} + +1. At the start, all users of the application are routed to the current version (shown in blue). A key point is that all traffic passes from a load balancer. +1. A new version is deployed (shown in green). This version gets only a very small amount of live traffic (for example 10%). +1. Developers can test internally and monitor their metrics to verify the new release. If they are confident, they can redirect more traffic to the new version (for example 33%). +1. If everything goes well, the old version is completely discarded. All traffic is now redirected to the new version. We are back to initial state (order of colors does not matter). + +The major benefit of this pattern is that if at any point in time the new version has issues, only a small subset of live users are affected. And like blue/green deployments, performing a rollback is as easy as resetting the load balancer to send no traffic to the canary version. Switching the load balancer is much faster than redeploying a new version, resulting in minimum disruption for existing users. + +There are several variations of this pattern. The amount of live traffic that you send to the canary at each step as well as the number of steps are user configurable. A simple approach would have just two steps (10%, 100%), while a more complex one could move traffic in a gradual way (10%, 30%, 60%, 90%, 100%). + +>Canary deployments are more advanced than blue/green deployments, and are also more complex to set up. The load balancer is now much smarter as it can handle two streams of traffic at the same time with different destinations of different weights. You also need a way (usually an API) to instruct the loadbalancer to change the weight distribution of the traffic streams. If you are just getting started with progressive delivery, we suggest you master blue/green deployments first, before adopting canaries. + +### Canary Deployment with Argo Rollouts + +Argo Rollouts supports the basic canary pattern described in the previous section, and also offers a wealth of [customization options](https://argoproj.github.io/argo-rollouts/features/canary/){:target="\_blank"}. +One of the most important +additions is the ability to "test" the upcoming version by introducing a "preview" [Kubernetes service](https://kubernetes.io/docs/concepts/services-networking/service/){:target="\_blank"}, in addition to the service used for live traffic. +This preview service can be used by the team that performs the deployment to verify the new version as it gets used by the subset of live users. + + +Here is the initial state of a deployment. The example uses four pods (shown as `22nqx`, `nqksq`, `8bzwh` and `jtdcc` in the diagram). + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/01_canary_initial_state.png" +url="/images/guides/progressive-delivery/01_canary_initial_state.png" +alt="Initial deployment. All services point to active version" +caption="Initial deployment. All services point to active version" +max-width="90%" +%} + +There are now three Kubernetes services: +* The `rollout-canary-all-traffic` that captures all live traffic from actual users of the application (internet traffic coming from `20.37.135.240`). +* A secondary service, `rollout-canary-active`, that always points to the stable/previous version of the software. +* A third service, `rollout-canary-preview`, that only routes traffic to the canary/new versions. + +In normal circumstances all there services point to the same version. + + +Once a deployment starts, a new "version" is created. In the example we have one new pod that represents the next version of the application to be deployed (shown as `9wx8w` at the top of the diagram). + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/02_canary_10.png" +url="/images/guides/progressive-delivery/02_canary_10.png" +alt="Deployment in progress. 10% of users are sent to the canary version" +caption="Deployment in progress. 10% of users are sent to the canary version" +max-width="90%" +%} + +The important point here is the fact that the service used by live users (called `rollout-canary-all-traffic`) routes traffic to **both** the canary and the previous version. It is not visible in the diagram, but only 10% of traffic is sent to the single pod that hosts the new version, while 90% of traffic goes to the four pods of the old version. + +The `rollout-canary-preview` service goes only to the canary pod. You can use this service to examine metrics from the canary or even give it to users who always want to try the new version first (e.g. your internal developers). On the other hand, the `rollout-canary-active` service always goes to the stable version. You can use that for users who never want to try the new version first or for verifying how something worked in the previous version. + + + +If everything goes well, and you are happy with how the canary works, we can redirect some more traffic to it. + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/03_canary_33.png" +url="/images/guides/progressive-delivery/03_canary_33.png" +alt="Deployment in progress. 33% of users are sent to the canary version" +caption="Deployment in progress. 33% of users are sent to the canary version" +max-width="90%" +%} + +We are now sending 33% of live traffic to the canary (the traffic weights are not visible in the picture). To accommodate the extra traffic, the canary version now has two pods instead of one. This is also another great feature of Argo Rollouts. The amount of pods you have in the canary is completely unrelated to the amount of traffic that you send to it. You can have all possible combinations that you can think of (e.g. 10% of traffic to five pods, or 50% of traffic to three pods and so on). It all depends on the resources used by your application. + +It makes sense of course to gradually increase the number of pods in the canary as you shift more traffic to it. + +Having the old version around is a great failsafe, as one can abort the deployment process and switch back all active users to the old deployment in the fastest way possible +by simply telling the load balancer to move 100% of traffic back to the previous version. + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/04_canary_finished.png" +url="/images/guides/progressive-delivery/04_canary_finished" +alt="Old application version is discarded. Only new version remains." +caption="Old application version is discarded. Only new version remains." +max-width="90%" +%} + +Two more pods are launched for the canary (for a total of four), and finally we can shift 100% of live traffic to it. After some time,the old version is scaled down completely to preserve resources. We are now back +to the same configuration as the initial state, and the next deployment will follow the same sequence of events. + +### Example application + +You can find an example application at [https://github.com/codefresh-contrib/argo-rollout-canary-sample-app](https://github.com/codefresh-contrib/argo-rollout-canary-sample-app){:target="\_blank"} that also includes simple metrics (we will use them in the second example with canaries). + +Notice that the first deployment of your application will NOT follow the canary deployment process as there is no "previous" version. So you need to deploy it at least +once. + +``` +git clone https://github.com/codefresh-contrib/argo-rollout-canary-sample-app.git +cd argo-rollout-canary-sample-app +kubectl create ns canary +kubectl apply -f ./canary-manual-approval -n canary +``` + +You can then monitor what argo rollouts is doing with the following command: + +``` +kubectl argo rollouts get rollout golang-sample-app-deployment --watch -n canary +``` + +### Choosing a solution for Traffic Management + +Unlike Blue/Green deployments, canary deployments require a smarter way to handle incoming traffic to your application. Specifically for Kubernetes, you need a networking solution that can split traffic according to percentages. Kubernetes on its own performs simple load balancing where the number of pods affects the traffic they get. But that is not enough for canary deployments. + +Argo Rollouts has [several integrations](https://argoproj.github.io/argo-rollouts/features/traffic-management/){:target="\_blank"} with Service Meshes and ingresses that can be used for Traffic Splits. + +Apart from the platforms that are supported natively by Argo Rollouts, you can also use any solution that implements the [Service Mesh Interface (SMI)](https://smi-spec.io/){:target="\_blank"}, a common +standard for service mesh implementations. Argo Rollouts [adheres to the SMI spec](https://argoproj.github.io/argo-rollouts/features/traffic-management/smi/){:target="\_blank"}, and can instruct any compliant solution for the traffic split process during canaries. + +In our example we are using [LinkerD](https://linkerd.io/){:target="\_blank"}, an open source service mesh solution for Kubernetes that also implements SMI. +You can install LinkerD by following [the official documentation](https://linkerd.io/2.10/getting-started/){:target="\_blank"} in your cluster and then making sure that your application is [meshed](https://linkerd.io/2.10/tasks/adding-your-service/){:target="\_blank"} (i.e. it is managed by LinkerD) by adding the special annotation [linkerd.io/inject:enabled](https://github.com/codefresh-contrib/argo-rollout-canary-sample-app/blob/main/canary-manual-approval/rollout.yaml#L36){:target="\_blank"} in the rollout YAML. + + +### Canary deployment with manual approval + +As with Blue/Green deployments, the easiest way to use canaries is by simply inserting [an approval step]({{site.baseurl}}/docs/pipelines/steps/approval/) before each subsequent traffic switch step. +This will pause the pipeline and the developers or QA team can evaluate the canary stability. + +Here is the [Canary setup](https://github.com/codefresh-contrib/argo-rollout-canary-sample-app/blob/main/canary-manual-approval/rollout.yaml#L8){:target="\_blank"}: + +`rollout.yaml` (excerpt) +```yaml +spec: + replicas: 4 + strategy: + canary: + canaryService: rollout-canary-preview + stableService: rollout-canary-active + trafficRouting: + smi: + trafficSplitName: rollout-example-traffic-split + rootService: rollout-canary-all-traffic + steps: + - setWeight: 10 + - setCanaryScale: + weight: 25 + - pause: {} + - setWeight: 33 + - setCanaryScale: + weight: 50 + - pause: {} +``` + +The canary has essentially three stages. At the beginning, it gets only 10% of the traffic and then it stops. At this point it creates 1/4 of pods. Then +if we promote it, it gets 33% of the traffic and is now scaled up to 1/2 the number of pods constituting a full deployment. We pause again and then finally it gets 100% of +live traffic. + + +Here is the pipeline with canary steps: + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/canary-manual-approval-pipeline.png" +url="/images/guides/progressive-delivery/canary-manual-approval-pipeline.png" +alt="Manual approval with two intermediate canary steps" +caption="Manual approval with two intermediate canary steps" +max-width="100%" +%} + +This pipeline does the following: + +1. [Clones]({{site.baseurl}}/docs/example-catalog/examples/git-checkout/) the source code of the application. +1. [Builds]({{site.baseurl}}/docs/ci-cd-guides/building-docker-images/) a Docker image +1. [Deploys]({{site.baseurl}}/docs/deployments/kubernetes/kubernetes-templating/) the application by updating the Kubernetes manifests. Argo Rollouts sees the new manifest and creates a new version. 10% of live traffic is redirected to it. +1. The pipeline is paused and waits for an [approval/rejection]({{site.baseurl}}/docs/pipelines/steps/approval/#getting-the-approval-result) by a human user. +1. If the pipeline is approved, 33% of traffic is now sent to the canary. If the pipeline is rejected, the canary is discarded and all traffic goes back to the stable version. +1. In the next pause, the pipeline waits for a second approval. +1. If the pipeline is approved, all traffic is now sent to the canary. If the pipeline is rejected, the canary is discarded and all traffic goes back to the stable version. + +Here is the [pipeline definition]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/): + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +stages: + - prepare + - build + - 'canary 10%' + - 'canary 33%' + - finish +steps: + clone: + type: "git-clone" + stage: prepare + description: "Cloning main repository..." + repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}' + revision: "${{CF_BRANCH}}" + build_app_image: + title: Building Docker Image + type: build + stage: build + image_name: kostiscodefresh/argo-rollouts-canary-sample-app + working_directory: "${{clone}}" + tags: + - "latest" + - '${{CF_SHORT_REVISION}}' + dockerfile: Dockerfile + start_deployment: + title: Deploy to 10% of live traffic + stage: 'canary 10%' + image: codefresh/cf-deploy-kubernetes:master + working_directory: "${{clone}}" + commands: + - /cf-deploy-kubernetes ./canary-manual-approval/service.yaml + - /cf-deploy-kubernetes ./canary-manual-approval/service-preview.yaml + - /cf-deploy-kubernetes ./canary-manual-approval/service-all.yaml + - /cf-deploy-kubernetes ./canary-manual-approval/rollout.yaml + environment: + - KUBECONTEXT=mydemoAkscluster@BizSpark Plus + - KUBERNETES_NAMESPACE=canary + check_canary_10: + fail_fast: false + type: pending-approval + title: Is canary ok? + stage: 'canary 10%' + promote_canary_33: + title: Switching 33% traffic to canary + stage: 'canary 33%' + image: kostiscodefresh/kubectl-argo-rollouts:latest + commands: + - /app/kubectl-argo-rollouts-linux-amd64 promote golang-sample-app-deployment -n canary --context "mydemoAkscluster@BizSpark Plus" + when: + steps: + - name: check_canary_10 + on: + - approved + abort_deployment_10: + title: Discarding canary at 10% + stage: 'canary 10%' + image: kostiscodefresh/kubectl-argo-rollouts:latest + commands: + - /app/kubectl-argo-rollouts-linux-amd64 undo golang-sample-app-deployment -n canary --context "mydemoAkscluster@BizSpark Plus" + when: + steps: + - name: check_canary_10 + on: + - denied + exit_10: + title: Stopping pipeline + stage: 'canary 10%' + image: alpine:39 + commands: + - echo "Canary failed" + - exit 1 + when: + steps: + - name: check_canary_10 + on: + - denied + check_canary_33: + fail_fast: false + type: pending-approval + title: Is canary ok? + stage: 'canary 33%' + promote_canary_full: + title: Switching all traffic to canary + stage: finish + image: kostiscodefresh/kubectl-argo-rollouts:latest + commands: + - /app/kubectl-argo-rollouts-linux-amd64 promote golang-sample-app-deployment -n canary --context "mydemoAkscluster@BizSpark Plus" + when: + steps: + - name: check_canary_33 + on: + - approved + abort_deployment_33: + title: Discarding canary at 33% + stage: 'canary 33%' + image: kostiscodefresh/kubectl-argo-rollouts:latest + commands: + - /app/kubectl-argo-rollouts-linux-amd64 undo golang-sample-app-deployment -n canary --context "mydemoAkscluster@BizSpark Plus" + when: + steps: + - name: check_canary_33 + on: + - denied + exit_33: + title: Stopping pipeline + stage: 'canary 33%' + image: alpine:39 + commands: + - echo "Canary failed" + - exit 1 + when: + steps: + - name: check_canary_33 + on: + - denied +{% endraw %} +{% endhighlight %} + +Just before the approval, you can optionally execute the Argo Rollouts CLI to see what is happening behind the scenes: + +``` +kubectl argo rollouts get rollout golang-sample-app-deployment --watch -n canary +``` + +It should show the status of the canary pods with amount of traffic that is redirected to it. + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/canary-watch.png" +url="/images/guides/progressive-delivery/canary-watch.png" +alt="Argo Rollouts CLI" +caption="Argo Rollouts CLI" +max-width="100%" +%} + +In the above picture, the canary deployment has just started. There is only one pod for the canary that gets 10% of live traffic. The four pods of the previous version still receive 90% percent of live traffic. + +You can also see the traffic split in the [LinkerD Dashboard](https://linkerd.io/2.10/reference/architecture/#dashboard){:target="\_blank"}: + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/canary-traffic-split.png" +url="/images/guides/progressive-delivery/canary-traffic-split.png" +alt="Linkerd Traffic split details" +caption="Linkerd Traffic split details" +max-width="80%" +%} + +The screenshot above is from the second stage of the canary where 33% of live traffic is redirected to the canary pods. +You can also get the same information from the command line with `kubectl get trafficsplit`. + +### Choosing a solution for automated metric analysis + +Canary deployments with manual pauses are great to get started but can quickly become cumbersome and error-prone. Ideally, the canary should automatically promote itself if the application "looks good". One of the most straightforward ways to examine application health is by reading its metrics and decide on the progress of the canary in a completely automated way. + +There are two main sources for metrics that you can use + +1. Application-specific metrics. This requires instrumentation in your application but is very powerful as you can query exactly what you want. +1. Cluster-level metrics (i.e. from the service mesh). These are very easy to set up, but are generic and deal mostly with the traffic the application receives. + + +Argo Rollouts has native integration for [several metric providers](https://argoproj.github.io/argo-rollouts/features/analysis/){:target="\_blank"}. We will use Prometheus in our example. +The example application [is already instrumented](https://github.com/codefresh-contrib/argo-rollout-canary-sample-app/blob/main/main.go#L51){:target="\_blank"} to expose some basic metrics. + +First, you need to install Prometheus by following [the official documentation](https://prometheus.io/docs/prometheus/latest/installation/){:target="\_blank"}. Then you need to make sure that Prometheus will actually scrape your application. Prometheus has [native service discovery for Kubernetes](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config){:target="\_blank"} but you need to enable it in the configuration. + +If you [install Prometheus with the Helm chart](https://github.com/prometheus-community/helm-charts){:target="\_blank"}, Kubernetes service discovery is already enabled. The only thing to set up is to add the `prometheus.io/scrape: "true"` annotation in your rollout so that Prometheus does not ignore your application. + +You can optionally install [Graphana](https://grafana.com/){:target="\_blank"} so that you can inspect your application metrics before using them in the canary process. The example application has an [basic dashboard](https://github.com/codefresh-contrib/argo-rollout-canary-sample-app/blob/main/graphana/graphana-dashboard.json){:target="\_blank"} +that you can import: + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/graphana-dashboard.png" +url="/images/guides/progressive-delivery/graphana-dashboard.png" +alt="Prometheus metrics from the application" +caption="Prometheus metrics from the application" +max-width="90%" +%} + +Next you need a way to filter your metrics so that you can query only those from the canary pods and not the stable pods. There are many ways to do this, but the easiest one is to simply have Argo Rollouts put special labels/tags in the canary pods. Then you can write any Prometheus Query and focus only on the canary instances: + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/canary-metrics.png" +url="/images/guides/progressive-delivery/canary-metrics.png" +alt="Canary metrics during a deployment" +caption="Canary metrics during a deployment" +max-width="100%" +%} + +For the decision on how to promote the canary, you need to examine your application and decide which metrics you consider representative for the health of the application. +For our example we have a simple query that checks number of successful calls (i.e. that return HTTP code 200) vs the number of all calls. Every number below 100% means that the application has calls that return with error. + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/canary-decision.png" +url="/images/guides/progressive-delivery/canary-decision.png" +alt="The query that will promote or cancel the canary" +caption="The query that will promote or cancel the canary" +max-width="100%" +%} + +Note that Argo Rollouts can evaluate multiple queries when deciding if the canary is health or not. You are not constrained to a single query. + + +### Canary deployment with metric evaluation + +Once your have your metric solution in place we need to instruct Argo Rollouts to use it during a deployment. + +This happens with an [Analysis CRD](https://github.com/codefresh-contrib/argo-rollout-canary-sample-app/blob/main/canary-with-metrics/analysis.yaml){:target="\_blank"}. + +`analysis.yaml` +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: AnalysisTemplate +metadata: + name: success-rate +spec: + args: + - name: service-name + metrics: + - name: success-rate + interval: 2m + count: 2 + # NOTE: prometheus queries return results in the form of a vector. + # So it is common to access the index 0 of the returned array to obtain the value + successCondition: result[0] >= 0.95 + provider: + prometheus: + address: http://prom-release-prometheus-server.prom.svc.cluster.local:80 + query: sum(response_status{app="{{args.service-name}}",role="canary",status=~"2.*"})/sum(response_status{app="{{args.service-name}}",role="canary"}) +``` + +This Analysis template instructs Argo Rollouts to contact the internal Prometheus server every two minutes for a query that checks the successful HTTP calls +to the application. If the percentage of HTTP calls that return 200 is more than 95% then the canary will be promoted. Otherwise the canary will fail. + +The Analysis can be reused by multiple deployments as the name of the service is configurable. The parameter is filled in the Rollout definition. + +`rollout.yaml` (excerpt) +```yaml +spec: + replicas: 4 + strategy: + canary: + canaryService: rollout-canary-preview + stableService: rollout-canary-active + canaryMetadata: + annotations: + role: canary + labels: + role: canary + trafficRouting: + smi: + trafficSplitName: rollout-example-traffic-split + rootService: rollout-canary-all-traffic + steps: + - setWeight: 10 + - setCanaryScale: + weight: 25 + - pause: {duration: 5m} + - setWeight: 33 + - setCanaryScale: + weight: 50 + - pause: {duration: 5m} + analysis: + templates: + - templateName: success-rate + startingStep: 4 # delay starting analysis run until setWeight: 10% + args: + - name: service-name + value: golang-sample-app +``` + +Here you can see that instead of waiting for ever after each canary step, we instead wait for 5 minutes at 10% of traffic and 5 more minutes at 50% of traffic. During that time the Prometheus Analysis is running automatically behind the scenes. + +The Codefresh pipeline is now very simple: + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/canary-metrics-pipeline.png" +url="/images/guides/progressive-delivery/canary-metrics-pipeline.png" +alt="Fully automated Canary pipeline" +caption="Fully automated Canary pipeline" +max-width="100%" +%} + +This pipeline does the following: + +1. [Clones]({{site.baseurl}}/docs/example-catalog/ci-examples/git-checkout/) the source code of the application. +1. [Builds]({{site.baseurl}}/docs/ci-cd-guides/building-docker-images/) a Docker image. +1. [Deploys]({{site.baseurl}}/docs/deployments/kubernetes/kubernetes-templating/) the application by updating the Kubernetes manifests. Argo Rollouts sees the new manifest and creates a new version and starts the canary process. + +Here is the pipeline definition: For more information, see [What is the Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/): + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +stages: + - prepare + - build + - deploy +steps: + clone: + type: "git-clone" + stage: prepare + description: "Cloning main repository..." + repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}' + revision: "${{CF_BRANCH}}" + build_app_image: + title: Building Docker Image + type: build + stage: build + image_name: kostiscodefresh/argo-rollouts-canary-sample-app + working_directory: "${{clone}}" + tags: + - "latest" + - '${{CF_SHORT_REVISION}}' + dockerfile: Dockerfile + build_arguments: + - git_hash=${{CF_SHORT_REVISION}} + start_deployment: + title: Start canary + stage: deploy + image: codefresh/cf-deploy-kubernetes:master + working_directory: "${{clone}}" + commands: + - /cf-deploy-kubernetes ./canary-with-metrics/service.yaml + - /cf-deploy-kubernetes ./canary-with-metrics/service-preview.yaml + - /cf-deploy-kubernetes ./canary-with-metrics/service-all.yaml + - /cf-deploy-kubernetes ./canary-with-metrics/analysis.yaml + - /cf-deploy-kubernetes ./canary-with-metrics/rollout.yaml + environment: + - KUBECONTEXT=mydemoAkscluster@BizSpark Plus + - KUBERNETES_NAMESPACE=canary +{% endraw %} +{% endhighlight %} + +The pipeline is very simple because Argo Rollouts does all the heavy lifting behind the scenes. + +You can see the Analysis running with + +``` +kubectl argo rollouts get rollout golang-sample-app-deployment --watch -n canary +``` + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/canary-watch-metrics.png" +url="/images/guides/progressive-delivery/canary-watch-metrics.png" +alt="Running the Analysis in the background" +caption="Running the Analysis in the background" +max-width="100%" +%} + +For each deployment you can also see the result of the Analysis along with the canary pods. The number next to the checkmark shows how many times the analysis will run (this is defined by the `count` property in the Analysis file). See the [Canary specification](https://argoproj.github.io/argo-rollouts/features/canary/) for more parameters. + +## Monitoring the Argo Rollouts controller + +Regardless of whether you use metric evaluation for your own applications, Argo Rollouts itself exposes Prometheus metrics +for its internal functionality. You can ingest those metrics like any other Prometheus application +and create your own dashboards if you want to get some insights on what the controller is doing. + +You can find an example dashboard at [https://github.com/argoproj/argo-rollouts/blob/master/examples/dashboard.json](https://github.com/argoproj/argo-rollouts/blob/master/examples/dashboard.json){:target="\_blank"} that can be used as a starting point. + +{% include image.html +lightbox="true" +file="/images/guides/progressive-delivery/monitor-rollout.png" +url="/images/guides/progressive-delivery/monitor-rollout.png" +alt="Integrated metrics from the Argo Rollouts controller" +caption="Integrated metrics from the Argo Rollouts controller" +max-width="80%" +%} + + +For more details, see the [metrics documentation page](https://argoproj.github.io/argo-rollouts/features/controller-metrics/){:target="\_blank"}. + +## Using Argo Rollouts with GitOps + +For simplicity reasons, we covered progressive delivery in this page using Argo Rollouts on its own. Argo Rollouts integrates seamlessly with Argo CD bringing together GitOps and Progressive delivery. + +If you use Argo CD and Argo Rollouts together you will also have access to the Codefresh GitOps dashboard to manage your deployments: + +{% include image.html + lightbox="true" + file="/images/guides/gitops/gitops-dashboard.png" + url="/images/guides/gitops/gitops-dashboard.png" + alt="The Codefresh GitOps dashboard" + caption="The Codefresh GitOps dashboard" + max-width="60%" + %} + + +See our [GitOps page]({{site.baseurl}}/docs/ci-cd-guides/gitops-deployments/) for more details. + + + +## Related articles +[Deploying to predefined environments]({{site.baseurl}}/docs/ci-cd-guides/environment-deployments/) +[GitOps Deployments]({{site.baseurl}}/docs/ci-cd-guides/gitops-deployments/) +[Pipelines for microservices]({{site.baseurl}}/docs/ci-cd-guides/microservices/) + + + + + diff --git a/_docs/ci-cd-guides/pull-request-branches.md b/_docs/ci-cd-guides/pull-request-branches.md new file mode 100644 index 000000000..1414d7337 --- /dev/null +++ b/_docs/ci-cd-guides/pull-request-branches.md @@ -0,0 +1,354 @@ +--- +title: "Pull requests and branches" +description: "Handle builds for pull requests or other branches" +group: ci-cd-guides +toc: true +--- + +Codefresh has native support for working with different branches and building pull requests. In particular, it has a very rich trigger model that allows you to handle specific events (such as opening a pull request or adding a comment). + +The possible actions can be seen in the trigger dialog of your pipeline: + +{% include image.html +lightbox="true" +file="/images/pipeline/triggers/add-git-trigger.png" +url="/images/pipeline/triggers/add-git-trigger.png" +alt="Adding GIT Trigger" +max-width="50%" +%} + +Notice however that Codefresh capabilities are always based on what your Git provider is offering. If your Git provider does not support webhooks for specific events, then these will not be available in the trigger dialog. + +## Building branches automatically + +By default, Codefresh connects to your Git provider and does the following: + +1. Auto-builds every new commit that happens in master or any other branch +2. Auto-builds every new branch when it is created + +You can change the default behavior so that it matches your own workflow using extra [Git triggers]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/). + +You don't have to do anything special to set up this communication between Codefresh and your Git provider. It was set up automatically when you connected your Codefresh account to your Git provider. + +Codefresh also creates a default Git trigger the first time you create a project. + +{% include +image.html +lightbox="true" +file="/images/pipeline/triggers/default-git-trigger.png" +url="/images/pipeline/triggers/default-git-trigger.png" +alt="Default GIT trigger" +caption="Default GIT trigger" +max-width="50%" +%} + +If you create a new branch in your repository, Codefresh automatically builds it and also stores the resulting Docker image. + +``` +git checkout -b another-branch +[..make changes...] +git commit -a -m "My changes" +git push -u origin another-branch +``` + +The build will clearly define its source branch: + +{% include image.html +lightbox="true" +file="/images/guides/branches-pull-requests/auto-branch-build.png" +url="/images/guides/branches-pull-requests/auto-branch-build.png" +alt="Building automatically new branches" +caption="Building automatically new branches" +max-width="100%" +%} + +When you commit to a Pull Request (PR), Codefresh auto-builds the PR, and you can also see the build request in the GitHub UI as well: + +{% include +image.html +lightbox="true" +file="/images/getting-started/quick-start-test-pr/auto-build-pr.png" +url="/images/getting-started/quick-start-test-pr/auto-build-pr.png" +alt="Pull Request Status" +caption="Pull Request Status (click image to enlarge)" +max-width="50%" +%} + +## Building specific branches manually + +Sometimes you want to run an ad-hoc build on a specific branch without actually committing anything. You can do that in the [run dialog of a pipeline]({{site.baseurl}}/docs/pipelines/pipelines/#creating-new-pipelines) by selecting a branch from the dropdown menu. + +{% include image.html +lightbox="true" +file="/images/guides/branches-pull-requests/build-specific-branch.png" +url="/images/guides/branches-pull-requests/build-specific-branch.png" +alt="Building a specific branch" +caption="Building a specific branch" +max-width="50%" +%} + +From the same dialog, you can also select a specific trigger to "emulate" for this branch if you have connected multiple triggers on the same pipeline. + +## Restricting which branches to build + +The auto-build nature of Codefresh for all branches is what you want most times. For larger projects, you might wish to restrict pipelines running only on specific branches. + +This is performed by defining [the branch field]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/#pull-request-target-branch-and-branch) in the trigger dialog with a regular expression. + +{% include image.html +lightbox="true" +file="/images/guides/branches-pull-requests/restrict-branch.png" +url="/images/guides/branches-pull-requests/restrict-branch.png" +alt="Restrict a pipeline to a single branch" +caption="Restrict a pipeline to a single branch" +max-width="50%" +%} + +The trigger above will only be activated for the `production` branch, so if a developer creates a new branch this pipeline will not run for it. Remember also that this field is actually a regular expression, so you can restrict a pipeline to a specific naming pattern (i.e. a group of branch names). + +Another popular filtering mechanism is to keep the auto-build nature of Codefresh, but enable/disable specific pipeline steps according to the branch being built. This is performed by using [step conditions]({{site.baseurl}}/docs/pipelines/conditional-execution-of-steps/). +Here is an example: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - build + - deploy +steps: + main_clone: + title: Cloning main repository... + stage: prepare + type: git-clone + repo: 'codefresh-contrib/spring-boot-2-sample-app' + revision: master + git: github + build_app_image: + title: Building Docker Image + type: build + stage: build + image_name: spring-boot-2-sample-app + working_directory: ./ + tag: 'multistage' + dockerfile: Dockerfile + deploy_production: + title: Deploying to production + type: deploy + stage: deploy + kind: kubernetes + cluster: 'my-prod-cluster' + namespace: default + service: my-prod-app + candidate: + image: '${{build_app_image}}' + registry: 'dockerhub' + when: + branch: + only: + - master + deploy_staging: + title: Deploying to staging + type: deploy + stage: deploy + kind: kubernetes + cluster: 'my-staging-cluster' + namespace: development + service: my-staging-app + candidate: + image: '${{build_app_image}}' + registry: 'dockerhub' + when: + branch: + only: + - /^JIRA-FEATURE-.*/i +{% endraw %} +{% endhighlight %} + +This pipeline will execute for **ALL** branches and pull requests, but: + +1. If the branch is `master` it will deploy the Docker image to the production cluster and namespace `default` +1. If the branch starts with `JIRA-FEATURE-` (e.g. JIRA-FEATURE-1234, JIRA-FEATURE-testing, JIRA-FEATURE-fixbbug), it will deploy to a staging cluster to namespace `development` +1. In all other cases of branches or pull requests, it will just build the Docker image without deploying it anywhere + +You can see that if a developer creates an unrelated branch (that doesn't match the expected name), no deployment will take place: + +{% include image.html +lightbox="true" +file="/images/guides/branches-pull-requests/branch-step-condition.png" +url="/images/guides/branches-pull-requests/branch-step-condition.png" +alt="Restrict pipeline steps according to branch" +caption="Restrict pipeline steps according to branch" +max-width="80%" +%} + +This is a more granular way to control how your branch affects your pipeline. + +>We recommend you follow the first method of having multiple simple pipelines with different branch expressions in the trigger dialog, instead of having a single complex pipeline using step conditions. Remember that in Codefresh you can create as many pipelines as you want for a single project instead of being limited to one pipeline per project. + +## Handling pull request events + +The power of Codefresh becomes evident when you realize that you can have extra pipelines that respond to specific PR events. For example, you can have a specific pipeline that runs **only** when a PR is opened for the first time or when a PR is closed. + +You can see all supported PR events in the trigger dialog. + +{% include image.html +lightbox="true" +file="/images/guides/branches-pull-requests/choosing-pr-events.png" +url="/images/guides/branches-pull-requests/choosing-pr-events.png" +alt="Choosing PR events for a pipeline" +caption="Choosing PR events for a pipeline" +max-width="80%" +%} + +>Remember that the events shown are those supported by your Git provider. Not all Git providers support all possible pull request events. + +You can select multiple pull request events for a single pipeline, or have multiple pipelines that respond to individual pull request events. There is no right or wrong answer as it mostly depends on how your team handles pull requests. + +The most useful events are: + +* Pull request open +* Pull request sync (when a commit happens to a PR) +* Pull request closed +* Comment added on a pull request + +There is also the shortcut checkbox for *any PR event* if you don't care about which specific event happened. + +## Trunk Based Development + +One of the most popular Git workflows is [Trunk Based development](https://trunkbaseddevelopment.com/){:target="\_blank"} with short-lived feature branches. + +{% include image.html +lightbox="true" +file="/images/guides/branches-pull-requests/trunk-based-development.png" +url="/images/guides/branches-pull-requests/trunk-based-development.png" +alt="Trunk Based Development" +caption="Trunk Based Development" +max-width="100%" +%} + +In this process, the master branch is always ready for production. The feature branches are created from the master and can have several commits before being merged back to master. + +This process can be easily created in Codefresh with two separate pipelines: + +* The "main" pipeline that deploys master to the production environment +* The feature pipeline that checks each feature as it is developed (and optionally deploys it to a staging environment) + +As an example, here is a minimal pipeline for the master branch: + +{% include image.html +lightbox="true" +file="/images/guides/branches-pull-requests/production-pipeline.png" +url="/images/guides/branches-pull-requests/production-pipeline.png" +alt="Pipeline that deploys to production" +caption="Pipeline that deploys to production" +max-width="100%" +%} + +The pipeline: + +1. Checks out the source code +1. Builds a Docker image +1. Creates and stores a Helm chart +1. Deploys the chart to Kubernetes + +The pipeline for feature branches is different: + +{% include image.html +lightbox="true" +file="/images/guides/branches-pull-requests/feature-pipeline.png" +url="/images/guides/branches-pull-requests/feature-pipeline.png" +alt="Pipeline for feature branches" +caption="Pipeline for feature branches" +max-width="100%" +%} + +For each feature branch: + +1. We check out the code +1. Run linters on the source code +1. Build the Docker image +1. Run some unit tests to verify the Docker image (possible with [service containers]({{site.baseurl}}/docs/pipelines/service-containers/)) + +To implement trunk-based development, we create two triggers for these pipelines. For the production pipeline, we just make sure that the trigger is only launched when commits land on master (and only there). + +{% include image.html +lightbox="true" +file="/images/guides/branches-pull-requests/trigger-for-production-pipeline.png" +url="/images/guides/branches-pull-requests/trigger-for-production-pipeline.png" +alt="Trigger for production pipeline" +caption="Trigger for production pipeline" +max-width="50%" +%} + +For the feature branch pipeline, we check the events for: + +* PR (pull request) Open +* PR Sync (when a commit happens on the PR) + +For the [branch specifications]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/#pull-request-target-branch-and-branch) we make sure that we look only for Pull Requests that are targeted **AT** `master`. + +{% include image.html +lightbox="true" +file="/images/guides/branches-pull-requests/trigger-for-features.png" +url="/images/guides/branches-pull-requests/trigger-for-features.png" +alt="Trigger for pull request pipeline" +caption="Trigger for pull request pipeline" +max-width="50%" +%} + +With this configuration, the whole process is as follows: + +1. A developer creates a new branch from master. Nothing really happens at this point. +1. The developer opens a new PR for this branch. The feature pipeline runs (because of the PR open checkbox). +1. The developer makes one or more commits to the branch. The feature pipeline runs again for each commit (because of the PR sync checkbox). +1. The developer commits the branch back to master. The main pipeline runs and deploys to production. + +You can fine-tune this workflow according to your needs. For example, you might also specify a naming pattern on the branches for the PR (e.g. feature-xxx) to further restrict which branches are considered ready for production. + +> We didn't need to handle the PR close/merge events. As soon as a PR is merged back to master, the Git provider sends anyway an event that a commit has happened in master, which means that the main production pipeline will take care of releasing the contents of master. + +## Git-flow + +[Git Flow](https://nvie.com/posts/a-successful-git-branching-model/){:target="\_blank"} is another popular management process for Git branches. For brevity reasons, we will not list all the details for all branch types, but it should be obvious that you can recreate all aspects of Git flow with Codefresh triggers. + +For example, to run a pipeline only for pull requests from branches named `feature-XXX` that will be merged back to `develop` branch, you can create a trigger like this: + +{% include image.html +lightbox="true" +file="/images/guides/branches-pull-requests/git-flow-feature-trigger.png" +url="/images/guides/branches-pull-requests/git-flow-feature-trigger.png" +alt="Git flow feature branch trigger" +caption="Git flow feature branch trigger" +max-width="50%" +%} + +To launch a pipeline that will only run when a commit happens on a release branch named `release-XXX`, you can create a trigger like this: + +{% include image.html +lightbox="true" +file="/images/guides/branches-pull-requests/git-flow-release-pipeline-trigger.png" +url="/images/guides/branches-pull-requests/git-flow-release-pipeline-trigger.png" +alt="Git flow release branch trigger" +caption="Git flow release branch trigger" +max-width="50%" +%} + +In a similar manner, you can create the triggers for all other branch types in Git flow. + +## Create your own workflow + +Trunk-based development and Git-flow are only some examples of what a Git workflow can look like. Your organization might follow a completely different process. Using the basic building blocks of Codefresh triggers (branch field, PR checkboxes, etc) you should be able to model your own workflow according to your own pipelines. + +## Related articles +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Git triggers in pipelines]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/) +[YAML examples]({{site.baseurl}}/docs/example-catalog/examples/) +[Preview environments]({{site.baseurl}}/docs/ci-cd-guides/preview-environments/) + + + + + diff --git a/_docs/ci-cd-guides/working-with-docker-registries.md b/_docs/ci-cd-guides/working-with-docker-registries.md new file mode 100644 index 000000000..28ab8c91b --- /dev/null +++ b/_docs/ci-cd-guides/working-with-docker-registries.md @@ -0,0 +1,573 @@ +--- +title: "Work with Docker Registries" +description: "Push, pull, and tag Docker images in Codefresh pipelines" +group: ci-cd-guides +redirect_from: + - /docs/build-specific-revision-image/ + - /docs/image-management/build-specific-revision-image/ + - /docs/docker-registries/working-with-docker-registries/ +toc: true +--- + +Codefresh contains first-class Docker registry support. This means that you don't need to manually write `docker login` and `docker pull/push` commands within pipelines. You can use declarative YAML, and all credentials are configured in a central location once. + +## Viewing Docker images + +To see all images from [all connected registries]({{site.baseurl}}/docs/integrations/docker-registry/docker-registries/): + +* In the Codefresh UI, from the Artifacts section in the sidebar, select [**Images**](https://g.codefresh.io/images/){:target="\_blank"}. + +{% + include image.html + lightbox="true" + file="/images/guides/working-with-images/docker-registry-list.png" + url="/images/guides/working-with-images/docker-registry-list.png" + alt="Codefresh Registry Image List" + caption="Codefresh Registry Image List" + max-width="70%" +%} + +Each image displays basic details such as the Git branch, commit message, hash that created it, creation date, as well as all tags. +* To view image metadata, click on the image. For details, see [Docker image metadata]({{site.baseurl}}/docs/pipelines/docker-image-metadata/). + + +**Filters for Docker images** +The top left of the Images page has several filters that allow you to search for a specific subset of Docker images. +Filters include: +* Tagged/untagged images +* Base image name +* Git branch +* Tag +* Pipeline volumes + +Multiple filters work in an `AND` manner. + +{% + include image.html + lightbox="true" + file="/images/guides/working-with-images/docker-registry-filters.png" + url="/images/guides/working-with-images/docker-registry-filters.png" + alt="Codefresh Registry Image filters" + caption="Codefresh Registry Image filters" + max-width="40%" +%} + + +**Actions for Docker images** +On the right are the actions available foreach Docker image. +You can: +* Launch a Docker image as a [test environment]({{site.baseurl}}/docs/getting-started/on-demand-environments/) +* Promote a Docker image (explained in the following sections) +* Pull the image locally on your workstation with different commands +* Re-run the pipeline that created the image + + +## Pulling Docker images + +Pulling Docker images in Codefresh is completely automatic. You only need to mention a Docker image by name, and Codefresh automatically pulls it for you and uses it in a pipeline. + +### Pulling public images + +To pull a public image from Docker Hub or other public registries: + +* Specify the name of the image and tag that you want to use. + +For example: + +```yaml +CollectAllMyDeps: + title: Install dependencies + image: python:3.6.4-alpine3.6 + commands: + - pip install . +``` + +This [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/) pulls the image `python:3.6.4-alpine3.6` from Docker Hub, and then runs the command `pip install .` inside it. +You can see the images that get pulled in the [Codefresh pipeline log]({{site.baseurl}}/docs/pipelines/monitoring-pipelines/). + +{% + include image.html + lightbox="true" + file="/images/guides/working-with-images/pull-public-image.png" + url="/images/guides/working-with-images/pull-public-image.png" + alt="Pulling a public image" + caption="Pulling a public image" + max-width="70%" +%} + +The image is also automatically cached in the [image cache]({{site.baseurl}}/docs/pipelines/pipeline-caching/#distributed-docker-image-caching). + +Codefresh also automatically pull for you any images mentioned in Dockerfiles (i.e. the `FROM` directive) as well as [service containers]({{site.baseurl}}/docs/pipelines/service-containers/). + + +### Pulling private images + +To pull a private image from one of your connected registries, in addition to specifying the image by name and tag, you must also prepend the appropriate prefix of the registry domain. The registry domain prefix is required for Codefresh to understand that it is a private image. + +For example, in the case of ACR (Azure Container Registry): + +``` +registry-name.azurecr.io/my-docker-repo/my-image-name:tag +``` + +Get the full name of a Docker image: +* In the Codefresh UI, from the Artifacts section in the sidebar, select [**Images**](https://g.codefresh.io/images/){:target="\_blank"}. +* Click on the image and copy the image name from the Activity column, **Image promoted** label. + +{% + include image.html + lightbox="true" + file="/images/guides/working-with-images/image-dashboard-tag.png" + url="/images/guides/working-with-images/image-dashboard-tag.png" + alt="Looking at tag of a private image" + caption="Looking at tag of a private image" + max-width="65%" +%} + +The exact format of the image name depends on the type of registry you use. Codefresh uses the domain prefix of each image to understand which integration to use, and then takes care of all `docker login` and `docker pull` commands on its own behind the scenes. + +For example, if you have connected [Azure]({{site.baseurl}}/docs/integrations/docker-registries/azure-docker-registry/){:target="\_blank"}, [AWS]({{site.baseurl}}/docs/integrations/docker-registries/amazon-ec2-container-registry/){:target="\_blank"}, and [Google]({{site.baseurl}}/docs/integrations/docker-registries/google-container-registry/){:target="\_blank"} registries, you can pull three images for each in a pipeline like this: + + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + my_go_unit_tests: + title: Running Go Unit tests + image: 'us.gcr.io/project-k8s-sample-123454/my-golang-app:prod' + commands: + - go test -v + my_mvn_unit_tests: + title: Running Maven Unit tests + image: '123456789012.dkr.ecr.us-west-2.amazonaws.com/my-java-app:latest' + commands: + - mvn test + my_python_unit_tests: + title: Running Python Unit tests + image: 'my-azure-registry.azurecr.io/kostis-codefresh/my-python-app:master' + commands: + - python setup.py test +{% endraw %} +{% endhighlight %} + +Codefresh automatically logs in to each registry using the credentials you have defined centrally, and pulls all the images. The same thing will happen with Dockerfiles that mention any valid Docker image in their `FROM` directive. + + +### Pulling images created in the same pipeline + +Codefresh allows you to create a Docker image on demand and use it in the same pipeline that created it. In several scenarios (such as [unit tests]({{site.baseurl}}/docs/testing/unit-tests/)), it is very common to use a Docker image right after it is built. + +In that case, as a shortcut, Codefresh allows you to simply [specify the name]({{site.baseurl}}/docs/pipelines/variables/#context-related-variables) of the respective [build step]({{site.baseurl}}/docs/pipelines/steps/build/). + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'codefresh-contrib/python-flask-sample-app' + revision: 'master' + git: github + MyAppDockerImage: + title: Building Docker Image + type: build + image_name: my-app-image + working_directory: ./ + tag: 'master' + dockerfile: Dockerfile + MyUnitTests: + title: Running Unit tests + image: '${{MyAppDockerImage}}' + commands: + - python setup.py test + +{% endraw %} +{% endhighlight %} + +In the above pipeline, Codefresh: + +1. Checks out source code through a [git-clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +1. Builds a Docker image, named `my-app-image:master`. Notice the lack of `docker push` commands. +1. In the next step, automatically uses that image and runs `python setup.py test` inside it. Again, notice the lack of `docker pull` commands. + +The important line here is the following: + +{% highlight yaml %} +{% raw %} + image: ${{MyAppDockerImage}} +{% endraw %} +{% endhighlight %} + +This says to Codefresh "in this step please use the Docker image that was built in the step named `MyAppDockerImage`". + +You can see the automatic pull inside the Codefresh logs. + +{% + include image.html + lightbox="true" + file="/images/guides/working-with-images/pull-private-image.png" + url="/images/guides/working-with-images/pull-private-image.png" + alt="Auto-Pulling a private image" + caption="Auto-Pulling a private image" + max-width="70%" +%} + +The image is still pushed to your default Docker registry. If you don't want this behavior, add the `disable_push` property in the build step. + + +## Pushing Docker images + +Pushing to your default Docker registry is completely automatic. All successful [build steps]({{site.baseurl}}/docs/pipelines/steps/build/) automatically push to the default Docker registry of your Codefresh account without any extra configuration. + +To push to another registry, you only need to know how this registry is [connected to Codefresh]({{site.baseurl}}/docs/docker-registries/external-docker-registries/), and more specifically, what is the unique name of the integration. You can see the name from your [Docker Registry integrations](https://g.codefresh.io/account-admin/account-conf/integration/registryNew), or asking your Codefresh administrator. + + +{% + include image.html + lightbox="true" + file="/images/guides/working-with-images/linked-docker-registries.png" + url="/images/guides/working-with-images/linked-docker-registries.png" + alt="Name of linked Docker Registries" + caption="Name of linked Docker Registries" + max-width="50%" +%} + +Once you know the registry identifier, you can use it in any [push step]({{site.baseurl}}/docs/pipelines/steps/push/) or [build step]({{site.baseurl}}/docs/pipelines/steps/build/) by specifying the registry with that name: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} + build_image: + title: Building my app image + type: build + image_name: my-app-image + dockerfile: Dockerfile + tag: 'master' + push_to_registry: + title: Pushing to Docker Registry + type: push + #Name of the build step that is building the image + candidate: '${{build_image}}' + tag: '1.2.3' + # Unique registry name + registry: azure-demo +{% endraw %} +{% endhighlight %} + +Notice that + * the `candidate` field of the push step mentions the name of the build step (`build_image`) that will be used for the image to be pushed. + * The registry is only identified by name (`azure-demo` in the example). The domain and credentials are not part of the pipeline as they are already known to Codefresh through the Docker registry integration. + + You can also override the name of the image with any custom name. This way the push step can choose any image name regardless of what was used in the build step. + + `codefresh.yml` +{% highlight yaml %} +{% raw %} + build_image: + title: Building my app image + type: build + image_name: my-app-image + dockerfile: Dockerfile + tag: 'master' + push_to_registry: + title: Pushing to Docker Registry + type: push + #Name of the build step that is building the image + candidate: '${{build_image}}' + tag: '1.2.3' + # Unique registry name + registry: azure-demo + image_name: my-company/web-app +{% endraw %} +{% endhighlight %} + +Here the build step creates an image named `my-app-image:master`, but the push step actually pushes it as `my-company/web-app:1.2.3`. + +For more examples, such as using multiple tags, or pushing in parallel, see the [push examples]({{site.baseurl}}/docs/pipelines/steps/push/#examples) + +### Pushing images with an optional prefix + +There are some registry providers that require a specific prefix for all your Docker images. This is often the name of an organization, account, or other top-level construct defined by the registry. + + `codefresh.yml` +{% highlight yaml %} +{% raw %} + build: + title: "Building Docker image" + type: "build" + image_name: "acme-company/trivial-go-web" + working_directory: "${{clone}}" + tag: "latest" + dockerfile: "Dockerfile.multistage" + stage: "build" + registry: azure +{% endraw %} +{% endhighlight %} + +The example above will push the final Docker image as `kostisazureregistry.azurecr.io/acme-company/trivial-go-web:latest`. + +However, you can also set up the prefix globally once in the [Docker registry integrations]({{site.baseurl}}/docs/integrations/docker-registries/). This way you can simplify your pipelines and have them mention only the final image name. + +{% + include image.html + lightbox="true" + file="/images/guides/working-with-images/registry-prefix.png" + url="/images/guides/working-with-images/registry-prefix.png" + alt="Global registry prefix" + caption="Global registry prefix" + max-width="70%" +%} + +Using the repository prefix in the example above, automatically prefixes all your Docker images with `acme-company`. + +Now you can simplify your build/push step as below: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} + build: + title: "Building Docker image" + type: "build" + image_name: "trivial-go-web" + working_directory: "${{clone}}" + tag: "latest" + dockerfile: "Dockerfile.multistage" + stage: "build" + registry: azure +{% endraw %} +{% endhighlight %} + +The final Docker image will still be `kostisazureregistry.azurecr.io/acme-company/trivial-go-web:latest`. + +## Working with multiple registries with the same domain + +With Codefresh, you can [connect multiple registries on a global level]({{site.baseurl}}/docs/integrations/docker-registries/). This allows you to create pipelines that push/pull images to different registries without having to deal with Docker credentials within the pipeline itself. + +However, there are several times where you have multiple registries that have the same domain. For example, you might have two Docker Hub accounts connected to Codefresh (so both of them can resolve images for the `docker.io` domain). + +This means that when you reference an image by domain name, as in a freestyle step for example, Codefresh might not know which Docker registry account to use for the pull action. + +> This is not a Codefresh limitation, but a Docker one. Even with vanilla Docker you cannot log in to multiple registries at the same time if they share the same domain. + +To solve this problem, Codefresh automatically detects connected registries that have the same domain and allow you to designate the primary one. The primary registry is used for image resolution when pulling Docker images. + +{% + include image.html + lightbox="true" + file="/images/guides/working-with-images/primary-dockerhub.png" + url="/images/guides/working-with-images/primary-dockerhub.png" + alt="Choosing a Docker registry as the primary one if they have the same domain" + caption="Choosing a Docker registry as the primary one if they have the same domain" + max-width="90%" +%} + +In the example above, even though two Docker Hub integrations are connected to Codefresh, only the primary one is used to pull images from the `docker.io` domain. You can still use the second one in push/build steps using the `registry` property. + +You can override the default behavior in each pipeline, by adding the optional `registry_context` property to instruct Codefresh on how to use a specific registry for pulling Docker images (if you have more than one for the same domain). + + + +You can use the `registry_context` property in [build]({{site.baseurl}}/docs/pipelines/steps/build/), [push]({{site.baseurl}}/docs/pipelines/steps/push/), [freestyle]({{site.baseurl}}/docs/pipelines/steps/freestyle/), and [composition]({{site.baseurl}}/docs/pipelines/steps/composition/) steps. + +The `registry_context` property takes as value the name of an external connected registry. Build and composition steps accept an array of values as `registry_contexts`. In all cases, by using this optional property you instruct Codefresh to use a specific registry for pulling images. + +> The optional `registry_context` and `registry_contexts` properties only affect the **pulling** of Docker images. The registry used for *pushing* images is still declared explicitly in build and push pipeline steps. + +The syntax for the freestyle step is the following: + +{% highlight yaml %} +{% raw %} + test: + title: "Running test" + type: "freestyle" + image: "gcr.io/my-google-project/my-image:latest" + registry_context: my-second-gcr-registry # define what registry will be used for pulling the image + working_directory: "${{clone}}" + commands: + - "ls" +{% endraw %} +{% endhighlight %} + +The syntax for the build step is the following: + +{% highlight yaml %} +{% raw %} + build: + title: "Building Docker image" + type: "build" + image_name: "trivial-go-web" + working_directory: "${{clone}}" + tag: "latest" + dockerfile: "Dockerfile.multistage" + stage: "build" + registry_contexts: # define what registries will be used for pulling images + - second-dockerhub + - production-azure + registry: azure +{% endraw %} +{% endhighlight %} + + +The syntax for the push step is the following: + +{% highlight yaml %} +{% raw %} + push: + title: "Pushing 1st Docker image" + type: push + image_name: "kostiscodefresh/trivial-go-web" + tag: "latest" + stage: "push" + registry: dockerhub # Docker registry to push to + registry_context: second-dockerhub # Docker registry to pull images from + candidate: ${{build}} +{% endraw %} +{% endhighlight %} + +The syntax for the composition step is the following: + +{% highlight yaml %} +{% raw %} + my-composition: + title: Running Composition + type: composition + registry_contexts: + - first-gcr + - second-gcr + arguments: + composition: + version: '2' + services: + db: + image: postgres + composition_candidates: + test_service: + image: 'alpine:3.9' + command: printenv + working_dir: /app + environment: + - key=value +{% endraw %} +{% endhighlight %} + +Let's look at an example. We assume that we have two GCR integrations: + +{% + include image.html + lightbox="true" + file="/images/guides/working-with-images/two-gcr-integrations.png" + url="/images/guides/working-with-images/two-gcr-integrations.png" + alt="Two GCR integrations" + caption="Two GCR integrations" + max-width="90%" +%} + +The first integration is the "production" one, and the second integration is the "staging" one. The production one is designated as primary. This means that by default even though both integrations work for the `gcr.io` domain, only the primary one is used in the Codefresh pipeline for pulling images. + +Let's say however that you want to build a Docker image that has a `FROM` statement from an image that exists in the staging registry. The image should be pushed to the production registry. You can use the `registry_context` property as shown below: + + +{% highlight yaml %} +{% raw %} + build: + title: "Building Docker image" + type: "build" + image_name: "gcr.io/production-project/my-image" + working_directory: "${{clone}}" + tag: "latest" + dockerfile: "Dockerfile" + stage: "build" + registry: production-gcr + registry_contexts: # define what registries will be used for pulling images + - staging-gcr +{% endraw %} +{% endhighlight %} + +Behind the scenes Codefresh will: + +1. First log in to the "staging" Docker registry using the "staging" credentials. +1. Build the Docker image, by resolving the `FROM` statements with "staging" images, pulling them as needed using the staging credentials. +1. Tag the Docker image. +1. Log in to the "production" Docker registry. +1. Push the final Docker image to the "production" registry. + +If your primary Docker registry is also the one that is used in your pipelines, you don't need the `registry_context` property at all, as this is the default behavior. When searching for an image to pull Codefresh will look at all your Docker registries (if they manage only a single domain), plus your "primary" Docker registries in case you have multiple Docker registries for the same domain. + +## Promoting Docker images + +Apart from building and pushing a brand new Docker image, you can also "promote" a Docker image by copying it from one registry to another. +You can perform this action either from the Codefresh UI or automatically from pipelines. + + +### Promoting images via the Codefresh UI + +You have the capability to "promote" any image of your choosing and push it to an external registry you have integrated into Codefresh (such as Azure, Google, Bintray etc.). + + +1. In the Codefresh UI, from the Artifacts section in the sidebar, select [**Images**](https://g.codefresh.io/images/){:target="\_blank"}. +1. To promote an image, in the row with the image, click the **Promote Image** icon on the right. + +{% + include image.html + lightbox="true" + file="/images/guides/working-with-images/docker-image-promotion.png" + url="/images/guides/working-with-images/docker-image-promotion.png" + alt="Promoting a Docker image" + caption="Promoting a Docker image" + max-width="50%" +%} + +1. From the list of connected registries, select the target registry, and define the tag that you want to push. +1. To "copy" this image from the existing registry to the target registry, click **Promote**. + +### Promoting images in pipelines + +You can also copy images from one registry to the other within a pipeline. +This is accomplished by specifying an existing image in the `candidate` field of the push step. + +For example: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} + promote_to_production_registry: + title: Promoting to Azure registry + type: push + candidate: us.gcr.io/project-k8s-sample-123454/my-golang-app + tag: '1.2.3' + # Unique registry name + registry: azure-demo +{% endraw %} +{% endhighlight %} + +In the example above, we promote an image from [GCR]({{site.baseurl}}/docs/integrations/docker-registries/google-container-registry/) to [ACR]({{site.baseurl}}/docs/integrations/docker-registries/azure-docker-registry/), which is already set up as `azure-demo`. + +You can even "promote" Docker images within the same registry by simply creating new tags. +For example: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} + promote_to_production: + title: Marking image with prod tag + type: push + candidate: my-azure-registry.azurecr.io/kostis-codefresh/my-python-app:1.2.3 + tag: 'production' + # Unique registry name + registry: azure-demo +{% endraw %} +{% endhighlight %} + +In the example above, the image `my-azure-registry.azurecr.io/kostis-codefresh/my-python-app:1.2.3` is re-tagged as `my-azure-registry.azurecr.io/kostis-codefresh/my-python-app:prod`. A very common pattern is to mark images with a special tag such as `prod` **after** the image has been deployed successfully. + + +## Related articles +[Push pipeline step]({{site.baseurl}}/docs/pipelines/steps/push/) +[External Docker registries]({{site.baseurl}}/docs/integrations/docker-registries/) +[Accessing a Docker registry from your Kubernetes cluster]({{site.baseurl}}/docs/deployments/kubernetes/access-docker-registry-from-kubernetes/) +[Build and push an image example]({{site.baseurl}}/docs/example-catalog/ci-examples/build-and-push-an-image/) + diff --git a/_docs/clients/csdp-cli.md b/_docs/clients/csdp-cli.md deleted file mode 100644 index 2882c3672..000000000 --- a/_docs/clients/csdp-cli.md +++ /dev/null @@ -1,48 +0,0 @@ ---- -title: "Download CLI" -description: "" -group: clients -toc: true ---- - -You need the Codefresh CLI to install Codefresh runtimes. -* For the initial download, you also need to generate the API key and create the API authentication context, all from the UI. -* Subsequent downloads for upgrade purposes require you to only run the download command, using existing API credentials. - -### Download Codefresh CLI -Downloading the Codefresh CLI requires you to select the download mode and OS, generate an API key, and authentication context. -1. Do one of the following: - * For first-time installation, go to the Welcome page, select **+ Install Runtime**. - * If you have provisioned a hybrid/hosted runtime, in the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}, and select **+ Add Runtime**. -1. Download the Codefresh CLI: - * Select one of the methods. - * Generate the API key and create the authentication context. - {% include - image.html - lightbox="true" - file="/images/getting-started/quick-start/quick-start-download-cli.png" - url="/images/getting-started/quick-start/quick-start-download-cli.png" - alt="Download CLI to install runtime" - caption="Download CLI to install runtime" - max-width="30%" - %} - -### Upgrade Codefresh CLI -Upgrade the CLI to the latest version to prevent installation errors. -1. Check the version of the CLI you have installed: - `cf version` -1. Compare with the [latest version](https://github.com/codefresh-io/cli-v2/releases){:target="\_blank"} released by Codefresh. -1. Select and run the appropriate command: - -{: .table .table-bordered .table-hover} -| Download mode | OS | Commands | -| -------------- | ----------| ----------| -| `curl` | MacOS-x64 | `curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-darwin-amd64.tar.gz | tar zx && mv ./cf-darwin-amd64 /usr/local/bin/cf && cf version`| -| | MacOS-m1 |`curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-darwin-arm64.tar.gz | tar zx && mv ./cf-darwin-arm64 /usr/local/bin/cf && cf version` | -| | Linux - X64 |`curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-linux-amd64.tar.gz | tar zx && mv ./cf-linux-amd64 /usr/local/bin/cf && cf version` | -| | Linux - ARM | `curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-linux-arm64.tar.gz | tar zx && mv ./cf-linux-arm64 /usr/local/bin/cf && cf version`| -| `brew` | N/A| `brew tap codefresh-io/cli && brew install cf2`| - -### Related articles -[Set up hosted (Hosted GitOps) environment]({{site.baseurl}}/docs/runtime/hosted-runtime) -[Install hybrid runtimes]({{site.baseurl}}/docs/runtime/installation) diff --git a/_docs/clients/upgrade-gitops-cli.md b/_docs/clients/upgrade-gitops-cli.md new file mode 100644 index 000000000..4a5e1a1e5 --- /dev/null +++ b/_docs/clients/upgrade-gitops-cli.md @@ -0,0 +1,88 @@ +--- +title: "Download/upgrade GitOps CLI" +description: "Have the latest version of the GitOps CLI" +group: installation +sub_group: gitops +toc: true +--- + +You need the Codefresh CLI to install Hybrid GitOps Runtimes, and have access to all the newest features. +For the initial download, you need to generate an API key and create the API authentication context, which you do from the UI. +When newer versions are available, the CLI automatically notifies you through a banner. You can use the existing API credentials for the upgrade. + + +## GitOps CLI installation modes +The table lists the modes available to install the GitOps CLI. + +{: .table .table-bordered .table-hover} +| Install mode | OS | Commands | +| -------------- | ----------| ----------| +| `curl` | MacOS-x64 | `curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-darwin-amd64.tar.gz | tar zx && mv ./cf-darwin-amd64 /usr/local/bin/cf && cf version`| +| | MacOS-m1 |`curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-darwin-arm64.tar.gz | tar zx && mv ./cf-darwin-arm64 /usr/local/bin/cf && cf version` | +| | Linux - X64 |`curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-linux-amd64.tar.gz | tar zx && mv ./cf-linux-amd64 /usr/local/bin/cf && cf version` | +| | Linux - ARM | `curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-linux-arm64.tar.gz | tar zx && mv ./cf-linux-arm64 /usr/local/bin/cf && cf version`| +| `brew` | N/A| `brew tap codefresh-io/cli && brew install cf2`|```` + +## Install the GitOps CLI +Install the GitOps CLI using the option that best suits you: `curl`, `brew`, or standard download. +If you are not sure which OS to select for `curl`, simply select one, and Codefresh automatically identifies and selects the right OS for CLI installation. + +1. Do one of the following: + * For first-time installation, go to the Welcome page, select **+ Install Runtime**. + * If you have provisioned a GitOps Runtime, in the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}, and select **+ Add Runtime**. +1. Install the Codefresh CLI: + * Select one of the installation modes. + * Generate the API key. + * Create the authentication context: + `cf config create-context codefresh --api-key
{:/} in DORA metrics, displays DORA metrics only for those applications.
+
+
+## Metric totals
+As the title indicates, the Totals bar shows the total numbers, based on the filters defined, or for the last 90 days, if there are no filters:
+
+* Deployments
+* Rollbacks
+* Commits/Pull Requests
+* Failure Rate: The number of failed deployments divided by the total number of deployments
+
+## Metric graphs
+The metric graphs are key to performance insights with DORA metrics. The metrics are again based on the filters defined, or for the last 90 days if there are no filters.
+
+In addition, you can select the granularity for each graph:
+
+* Daily
+* Weekly
+* Monthly
+
+>Tip:
+ Remember that the graphs for the DORA metrics reflect metrics of application deployments, not workflows.
+
+**Deployment Frequency**
+ The frequency at which applications are deployed to production, including both successful (Healthy) and failed (Degraded), deployments. A deployment is considered an Argo CD sync where there was a change in the application source code that resulted in a new deployment of the application to production.
+ The X-axis charts the time based on the granularity selected, and the Y-axis charts the number of deployments. The number shown on the top right is the average deployment frequency based on granularity.
+
+**Lead Time for Changes**
+ The average number of days from the first commit for a PR (pull request) until the deployment date for the same PR. The key term here is _deployment_. Lead Time for Changes considers only those changes to workflows that result in a deployment. Making a change to a repo that does not result in a deployment is not included when calculating Lead Time for Changes.
+ The X-axis charts the time based on the granularity selected, and the Y-axis charts the time in minutes until the deployment. The number shown on the top right is the average number of days for a commit to reach production.
+
+**Change Failure Rate**
+ The failure or rollback rate in percentage for applications whose health status changed to Degraded on deployment. The key term here is _on deployment_. For example, bumping an image tag with one that does not exist, results in the application being Degraded on deployment, and designated as failed.
+ The Change Failure Rate is derived by dividing the number of Degraded (failed/rollback) deployments with the total number of deployments.
+ The X-axis charts the time based on the granularity selected, and the Y-axis charts the failure rate. The number shown on the top right is the average failure rate based on granularity, and therefore may not be equal to the Total Failure Rate.
+
+**Time to Restore Service**
+ The average number of hours taken for the status of Degraded deployments to return to Healthy. Again, similar to the Change Failure Rate, Time to Restore Service includes only deployments that became Degraded. It is derived by dividing the total number of hours for all Degraded deployments to return to Healthy by the total number of Degraded deployments.
+ The X-axis charts the time based on the granularity, and the Y-axis charts the time in hours. The number shown on the top right is the average number of hours between the previous deployment and rollback for the same application.
+
+## Related articles
+[Global analytics dashboard]({{site.baseurl}}/docs/dashboards/home-dashboard)
+[Monitoring applications]({{site.baseurl}}/docs/deployments/gitops/applications-dashboard/)
+
diff --git a/_docs/reporting/home-dashboard.md b/_docs/dashboards/home-dashboard.md
similarity index 96%
rename from _docs/reporting/home-dashboard.md
rename to _docs/dashboards/home-dashboard.md
index c3a402c05..be0e4e505 100644
--- a/_docs/reporting/home-dashboard.md
+++ b/_docs/dashboards/home-dashboard.md
@@ -1,7 +1,7 @@
---
-title: "Home dashboard"
+title: "Global analytics dashboard"
description: ""
-group: reporting
+group: dashboards
toc: true
---
@@ -135,8 +135,8 @@ Analytics are derived by comparing the selected date range to the corresponding
|**Longest Delivery Pipelines** | Up to ten pipelines with the longest duration. The same KPIs are displayed, and compared to those in the reference period. |
### Related articles
-[DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics/)
-[Monitoring applications]({{site.baseurl}}/docs/deployment/applications-dashboard/)
-[Images in Codefresh]({{site.baseurl}}/docs/deployment/images/)
+[DORA metrics]({{site.baseurl}}/docs/dashboards/dora-metrics/)
+[Monitoring applications]({{site.baseurl}}/docs/deployments/gitops/applications-dashboard/)
+[Images in Codefresh]({{site.baseurl}}/docs/deployments/gitops/images/)
diff --git a/_docs/deployment/applications-dashboard.md b/_docs/deployments/gitops/applications-dashboard.md
similarity index 89%
rename from _docs/deployment/applications-dashboard.md
rename to _docs/deployments/gitops/applications-dashboard.md
index 1e9609760..51907f811 100644
--- a/_docs/deployment/applications-dashboard.md
+++ b/_docs/deployments/gitops/applications-dashboard.md
@@ -1,7 +1,8 @@
---
-title: "Monitoring applications"
+title: "Monitoring GitOps applications"
description: ""
-group: deployment
+group: deployments
+sub_group: gitops
toc: true
---
@@ -27,15 +28,15 @@ Monitor the current [health and sync status of applications](#identify-applicati
* [Monitor deployments for selected application](#monitor-deployments-for-selected-application)
* [Monitor services for selected application](#monitor-services-for-selected-application)
->For information on creating and managing applications and application resources, see [Creating applications]({{site.baseurl}}/docs/deployment/create-application/) and [Managing applications]({{site.baseurl}}/docs/deployment/manage-application/).
+>For information on creating and managing applications and application resources, see [Creating applications]({{site.baseurl}}/docs/deployments/gitops/create-application/) and [Managing applications]({{site.baseurl}}/docs/deployments/gitops/manage-application/).
-### Select view mode for the Applications dashboard
+## Select view mode for the Applications dashboard
View deployed applications in either List (the default) or Card views. Both views are sorted by the most recent deployments.
1. In the Codefresh UI, go to the [Applications dashboard](https://g.codefresh.io/2.0/applications-dashboard/list){:target="\_blank"}.
1. Select **List** or **Cards**.
-#### Applications List view
+### Applications List view
Here is an example of the Applications dashboard in List view mode.
@@ -49,7 +50,7 @@ caption="Applications Dashboard: List view"
max-width="60%"
%}
-#### Applications Card view
+### Applications Card view
Here is an example of the Applications dashboard in Card view mode. The Card view provides a scannable view of application data and the actions to manage applications.
{% include
@@ -62,18 +63,18 @@ caption="Applications Dashboard: Card view"
max-width="60%"
%}
-### Applications dashboard information
+## Applications dashboard information
Here's a description of the information and actions in the Applications dashboard.
{: .table .table-bordered .table-hover}
| Item | Description |
| -------------- | -------------- |
|Application filters | Filter by a range of attributes to customize the information in the dashboard to bring you what you need. {::nomarkdown} - Application state
A snapshot that displays a breakdown of the deployed applications by their health status.
Click a status to filter by applications that match it.
Codefresh tracks Argo CD's set of health statuses. See the official documentation on Health sets. . - Application attributes
Attribute filters support multi-selection, and results are based on an OR relationship within the same filter with multiple options, and an AND relationship between filters.
Clicking More Filters gives you options to filter by Health status, Cluster names, Namespace, and Type.- Application Type: Can be any of the following
- Applications: Standalone applications. See the official documentation on Applications.
- ApplicationSet: Applications created using the ApplicationSet Custom Resource (CR) template. An ApplicationSet can generate single or multiple applications. See the official documentation on Generating Applications with ApplicationSet.
- Git Source: Applications created by Codefresh that includes other applications and CI resources. See Git Sources.
- Application Type: Can be any of the following
- Labels:The K8s labels defined for the applications. The list displays labels of all the applications, even if you have applied filters.
To see the available labels, select Add, and then select the required label and one or more values.
To filter by the labels, select Add and then Apply.
See the official documentation on Labels and selectors.
{:/}| Star applications as favorites and view only the starred applications.{::nomarkdown}Select the
to star the application as a favorite.To filter by favorite applications, on the filters bar, select
.{:/} TIP: If you star applications as favorites in the Applications dashboard, you can filter by the same applications in the [DORA metrics dashboard]({{site.baseurl}}/docs/reporting/dora-metrics/#metrics-for-favorite-applications). | -|Application actions| Options to monitor/manage applications through the application's context menu. {::nomarkdown}
- Quick view
A comprehensive read-only view of the deployment and definition information for the application. {:/}See [Application Quick View](#view-deployment-and-configuration-info-for-selected-application) in this article.{::nomarkdown} - Synchronize/Sync
Manually synchronize the application. {:/}See [Manually sync applications]({{site.baseurl}}/docs/deployment/manage-application/#manually-synchronize-an-application).{::nomarkdown} - Edit
Modify application definitions. {:/}See [Edit application definitions]({{site.baseurl}}/docs/deployment/manage-application/#edit-application-definitions).{::nomarkdown} - Refresh and Hard Refresh: Available in Card view only. In List view, you must first select the application.
- Refresh: Retrieve desired (Git) state, compare with the live (cluster) state, and refresh the application to sync with the desired state.
- Hard Refresh: Refresh the application to sync with the Git state, while removing the cache.
{:/}| Star applications as favorites and view only the starred applications.{::nomarkdown}
Select the to star the application as a favorite.
To filter by favorite applications, on the filters bar, select
.
{:/} TIP: If you star applications as favorites in the Applications dashboard, you can filter by the same applications in the [DORA metrics dashboard]({{site.baseurl}}/docs/reporting/dora-metrics/#metrics-for-favorite-applications). | +|Application actions| Options to monitor/manage applications through the application's context menu. {::nomarkdown}- Quick view
A comprehensive read-only view of the deployment and definition information for the application. {:/}See [Application Quick View](#view-deployment-and-configuration-info-for-selected-application) in this article.{::nomarkdown} - Synchronize/Sync
Manually synchronize the application. {:/}See [Manually sync applications]({{site.baseurl}}/docs/deployments/gitops/manage-application/#manually-synchronize-an-application).{::nomarkdown} - Edit
Modify application definitions. {:/}See [Edit application definitions]({{site.baseurl}}/docs/deployments/gitops/manage-application/#edit-application-definitions).{::nomarkdown} - Refresh and Hard Refresh: Available in Card view only. In List view, you must first select the application.
- Refresh: Retrieve desired (Git) state, compare with the live (cluster) state, and refresh the application to sync with the desired state.
- Hard Refresh: Refresh the application to sync with the Git state, while removing the cache.
{:/} -#### Warning: Missing Rollouts reporter in cluster +### Warning: Missing Rollouts reporter in cluster **Reason**: Codefresh has detected that Argo Rollouts is not installed on the target cluster. Rollout instructions are therefore not executed and the application is not deployed. Applications with `rollout` resources need Argo Rollouts on the target cluster, both to visualize rollouts in the Applications dashboard and control rollout steps with the Rollout Player. @@ -108,7 +109,7 @@ Applications with `rollout` resources need Argo Rollouts on the target cluster,
{:/} -#### Warning: Long sync +### Warning: Long sync **Reason**: Ongoing sync for application exceeds 30 minutes (Argo CD's default duration for a sync operation). **Corrective Action**: @@ -119,7 +120,7 @@ Applications with `rollout` resources need Argo Rollouts on the target cluster, * Drill down into the application to investigate the issue and make changes. -### View deployment and configuration info for selected application +## View deployment and configuration info for selected application View deployment, definition, and event information for the selected application in a centralized location through the Quick View. A read-only view, the Quick View displays information on the application state and location, labels and annotations, parameters, sync options, manifest, status and sync events. @@ -165,7 +166,8 @@ max-width="50%" -##### Quick View: Summary +### Quick View: Summary + Displays health, sync status, and source and destination definitions. {% include @@ -178,7 +180,9 @@ caption="Application Quick View: Summary" max-width="30%" %} -##### Quick View: Metadata + +### Quick View: Metadata + Displays labels and annotations for the application. {% include @@ -191,7 +195,9 @@ caption="Application Quick View: Metadata" max-width="30%" %} -##### Quick View: Parameters + +### Quick View: Parameters + Displays parameters configured for the application, based on the tool used to create the application's manifests. The parameters displayed differ according to the tool: `directory` (as in the screenshot below), `Helm` charts, or `Kustomize` manifests, or the specific plugin. @@ -205,7 +211,8 @@ caption="Application Quick View: Parameters" max-width="30%" %} -##### Quick View: Sync Options +### Quick View: Sync Options + Displays sync options enabled for the application. {% include @@ -218,7 +225,8 @@ caption="Application Quick View: Parameters" max-width="30%" %} -##### Quick View: Manifest +### Quick View: Manifest + Displays the YAML version of the application manifest. {% include @@ -231,7 +239,8 @@ caption="Application Quick View: Manifest" max-width="30%" %} -##### Quick View: Events +### Quick View: Events + Displays status and sync events for the application. {% include @@ -244,7 +253,7 @@ caption="Application Quick View: Events" max-width="30%" %} -### Monitor health and sync statuses for selected application +## Monitor health and sync statuses for selected application Monitor the health status of the selected application, the current sync status, and the result of the previous sync operation. Once you select an application, the quickest option to monitor statuses is through the application header which is always displayed, no matter what tab you navigate to. @@ -274,7 +283,7 @@ max-width="40%" You can also view the current health and sync status for the application as a resource in the Current State tab. -### Monitor resources for selected application +## Monitor resources for selected application Monitor the resources deployed in the current version of the selected application in the Current State tab. Selecting an application from the Applications dashboard takes you to the Current State tab, which as its title indicates, displays the @@ -302,7 +311,7 @@ You can view application resources in [List or Tree views](#view-modes-for-appli > To quickly see which resources have been added, modified, or removed for the current or for a specific deployment, switch to the Timeline tab and expand the deployment record to show Updated Resources. See [Monitor resource updates for deployments](#monitor-resource-updates-for-deployments). -#### View modes for application resources +### View modes for application resources The Current State tab supports Tree and List view formats. * Tree view (default): A hierarchical, interactive visualization of the application and its resources. Useful for complex deployments with multiple clusters and large numbers of resources. See also [Working with resources in Tree view](#working-with-resources-in-tree-view). @@ -335,7 +344,7 @@ max-width="50%" -##### Working with resources in Tree view +#### Working with resources in Tree view The Tree view is designed to impart key information at a glance. Review the sections that follow for pointers to quickly get to what you need in the Tree view. **Context menu** @@ -415,7 +424,7 @@ max-width="50%" %} -#### Filters for application resources +### Filters for application resources Filters are common to both Tree and List views, and when applied are retained when switching between views. `IgnoreExtraneous` is a filter in [Argo CD](https://argo-cd.readthedocs.io/en/stable/user-guide/compare-options){:target="\_blank"} that allows you to hide specific resources from the Current State views. These resources are usually generated by a tool and their sync statuses have no impact on the sync status of the application. For example, `ConfigMap` and `pods`. The application remains in-sync even when such resources are syncing or out-of-sync. @@ -450,7 +459,7 @@ max-width="50%" %} -#### Health status for application resources +### Health status for application resources View and monitor health status of the selected application's resources in the Current State tab, in Tree or List views. Identify the health of an application resource through the color-coded border and the resource-type icon (Tree view), or the textual labels at the right of the resource (List view). @@ -458,18 +467,20 @@ Identify the health of an application resource through the color-coded border an {: .table .table-bordered .table-hover} | Health status | Description | Display in Tree view | | -------------- | ------------| ------------------| -| **Healthy** | Resource is functioning as required. | {::nomarkdown}
{:/} |
-| **Progressing** | Resource is not healthy but can become healthy before the timeout occurs.| {::nomarkdown}
{:/} |
-| **Suspended** | Resource is not functioning, and is either suspended or paused. For example, Cron job or a canary rollout.| {::nomarkdown}
{:/} |
+
+| **Healthy** | Resource is functioning as required. | {::nomarkdown}
{:/} |
+| **Progressing** | Resource is not healthy but can become healthy before the timeout occurs.| {::nomarkdown}
{:/} |
+| **Suspended** | Resource is not functioning, and is either suspended or paused. For example, Cron job or a canary rollout.| {::nomarkdown}
{:/} |
| **Missing** | Resource is not present on the cluster. |{::nomarkdown}
{:/} |
-| **Degraded** | Resource is not healthy, or a timeout occurred before it could reach a healthy status.| {::nomarkdown}
{:/} |
-| **Unknown** | Resource does not have a health status, or the health status is not tracked in Argo CD. For example,`ConfigMaps` resource types. | {::nomarkdown}
{:/} |
+| **Degraded** | Resource is not healthy, or a timeout occurred before it could reach a healthy status.| {::nomarkdown}
{:/} |
+| **Unknown** | Resource does not have a health status, or the health status is not tracked in Argo CD. For example,`ConfigMaps` resource types. | {::nomarkdown}
{:/} |
+
See also [Argo CD's set of health checks](https://argo-cd.readthedocs.io/en/stable/operator-manual/health/){:target="\_blank"}.
-#### Sync status for application resources
+### Sync status for application resources
Similar to the health status, the Current State also tracks the sync status of all application resources. The sync status identifies if the live state of the application resource on the cluster is synced with its desired state in Git.
Identify the sync status through the icon on the left of the resource name and the color of the resource name (Tree view), or the textual labels at the right of the resource (List view).
@@ -479,15 +490,17 @@ The table describes the possible sync statuses for an application resource, and
{: .table .table-bordered .table-hover}
| Sync state | Description |Display in Tree view |
| -------------- | ---------- | ---------- |
-| **Synced** | The live state of the resource on the cluster is identical to the desired state in Git.| {::nomarkdown}
{:/} |
-| **Syncing** | The live state of the resource was not identical to the desired state, and is currently being synced.| {::nomarkdown}
{:/} |
-| **Out-of-Sync** | {::nomarkdown}The live state is not identical to the desired state.
To sync a resource, select the Sync option from the resource's context menu in Tree view. {:/}| {::nomarkdown}
{:/} |
-| **Unknown** | The sync status could not be determined. | {::nomarkdown}
{:/} |
+
+| **Synced** | The live state of the resource on the cluster is identical to the desired state in Git.| {::nomarkdown}
{:/} |
+| **Syncing** | The live state of the resource was not identical to the desired state, and is currently being synced.| {::nomarkdown}
{:/} |
+| **Out-of-Sync** | {::nomarkdown}The live state is not identical to the desired state.
To sync a resource, select the Sync option from the resource's context menu in Tree view. {:/}| {::nomarkdown}
{:/} |
+| **Unknown** | The sync status could not be determined. | {::nomarkdown}
{:/} |
+
> The application header displays the statuses of the current and previous sync operations. Clicking **More** opens the Sync panels with Sync Info, Sync Result and Commit Info.
The Application Warnings/Errors panel surfaces sync errors on exceeding the maximum number of retries and when a sync operation extends beyond 30 minutes.
-#### Manifests for application resources
+### Manifests for application resources
In either Tree or List views, double-click an application resource to see its manifests. The manifests are displayed in the Summary tab.
> Based on the selected resource type, you can also view logs, and events. Endpoints for example show only manifests, while pods show manifests, logs, and events.
@@ -519,7 +532,7 @@ Here's what you can see and do in the Summary tab:
{:/} -#### Logs for application resources +### Logs for application resources In either Tree or List views, double-click an application resource to see its logs. Logs are available only for resource types such as pods. {% include @@ -541,7 +554,7 @@ max-width="50%"
{:/} -#### Events for application resources +### Events for application resources In either Tree or List views, double-click an application resource to see events in the Events tab. > If your runtime is lower than the version required to view events, you are notified to upgrade to the required version. @@ -563,7 +576,7 @@ max-width="50%" -### Monitor deployments for selected application +## Monitor deployments for selected application Monitor an ongoing deployment for the selected application, and review its historical deployments. The Timeline tab displays the history of deployments for the selected application, sorted by the most recent deployment (default), labeled **Current Version** at the top. @@ -600,7 +613,7 @@ caption="Applications Dashboard: Deployment chart" max-width="30%" %} -#### Monitor CI details by deployment +### Monitor CI details by deployment Each deployment record displays the complete CI history for that deployment. @@ -611,7 +624,7 @@ Each deployment record displays the complete CI history for that deployment. * The **Committer** who made the changes. -#### Monitor updated resources by deployment +### Monitor updated resources by deployment Each deployment record also identifies the resources that were changed (created, updated, or removed) as part of that deployment in **Updated Resources**. You can trace the history of a resource, from the original to their final versions. For each version, you can see the actual change or changes through the Diff view. The Full View shows the complete resource manifest, with the diff view of the changes, while the Compact View shows only those lines with the changes. > For detailed information on the current state of a resource, switch to the Current State tab and click the resource node. See [Monitoring application resources](#monitoring-application-resources). @@ -657,15 +670,15 @@ max-width="70%" -#### Monitor rollouts by deployment +### Monitor rollouts by deployment A rollout is initiated when there is an Argo CD sync due to a change in the desired state. Visualize ongoing and completed rollouts by deployments in **Services**. -> To view and manage a rollout, you must have an Argo `rollout` resource defined for your application, and [install Argo Rollouts in the cluster]({site.baseurl}}/docs/_docs/deployment/install-argo-rollouts). +> To view and manage a rollout, you must have an Argo `rollout` resource defined for your application, and [install Argo Rollouts in the cluster]({site.baseurl}}/docs/deployments/gitops/install-argo-rollouts). For detailed information on Argo Rollouts, see [Argo Rollouts documentation](https://argoproj.github.io/argo-rollouts/){:target="\_blank"}. -##### Rollout progress +#### Rollout progress For an ongoing rollout, the rollout bar displays the progress of the rollout. You can also visualize the steps in the rollout, and control the rollout using the options in the Rollout Player. Here is an example of an ongoing rollout for a canary deployment in Updated Services. The rollout comprising four steps has not started, and no traffic has not been routed as yet to the new version of the application. @@ -693,7 +706,7 @@ caption="Rollout completed for deployment" max-width="50%" %} -##### Manage ongoing rollout +#### Manage ongoing rollout Click the rollout name to visualize its steps. Manually manage the rollout through the controls in the Rollout Player. Here you can see that two out of four steps have been completed, 25% of the traffic has been routed, and the rollout has been paused for the defined length of time. @@ -720,7 +733,7 @@ The table lists the controls in the Rollout Player to manage an ongoing rollout. -##### View analysis run +#### View analysis run If you have defined an analysis template for the rollout, you can check the run results and the manifest. The result of an analysis run determines if the rollout is completed, paused, or aborted. For detailed information, see the [Analysis section in Argo Rollouts](https://argoproj.github.io/argo-rollouts/features/analysis/){:target="\_blank"}. @@ -749,7 +762,7 @@ max-width="50%" %} -### Monitor services for selected application +## Monitor services for selected application The Services tab shows the K8s services for each deployment of the application. Each service shows the number of replicas, the endpoint IP, the labels that reference the application, and the health status. @@ -765,6 +778,11 @@ caption="Applications Dashboard: Services tab" max-width="50%" %} +## Related articles +[Creating GitOps applications]({{site.baseurl}}/docs/deployments/gitops/create-application) +[Managing GitOps applications]({{site.baseurl}}/docs/deployments/gitops/manage-applications) +[Home dashboard]({{site.baseurl}}/docs/reporting/home-dashboard) +[DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics/) diff --git a/_docs/deployment/create-application.md b/_docs/deployments/gitops/create-application.md similarity index 96% rename from _docs/deployment/create-application.md rename to _docs/deployments/gitops/create-application.md index 47aea2d77..274251725 100644 --- a/_docs/deployment/create-application.md +++ b/_docs/deployments/gitops/create-application.md @@ -1,7 +1,8 @@ --- -title: "Creating applications" +title: "Creating GitOps applications" description: "" -group: deployment +group: deployments +sub_group: gitops toc: true --- @@ -19,7 +20,7 @@ Codefresh provides all the options and functionality to create and manage Argo C * Edit and delete applications Once the application is created and synced to the cluster, it is displayed in the Applications dashboard. Here, you can select an application to update the application's configuration settings, or delete it. - To monitor the health and sync status, deployments, and resources for the application, see [Monitoring applications]({{site.baseurl}}/docs/deployment/applications-dashboard/). + To monitor the health and sync status, deployments, and resources for the application, see [Monitoring GitOps applications]({{site.baseurl}}/docs/deployments/gitops/applications-dashboard/). ### Application: Definitions Application definitions include the name, runtime, and the name of the YAML manifest. By default, the YAML manifest has the same name as that of the application. @@ -225,7 +226,7 @@ Track the application in the [Applications dashboard](https://g.codefresh.io/2.0 ### Related articles -[Monitoring applications]({{site.baseurl}})/docs/deployment/applications-dashboard) -[Managing applications]({{site.baseurl}})/docs/deployment/manage-applications) -[Home dashboard]({{site.baseurl}})docs/reporting/home-dashboard) -[DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics/) \ No newline at end of file +[Monitoring GitOps applications]({{site.baseurl}})/docs/deployments/gitops/applications-dashboard) +[Managing GitOps applications]({{site.baseurl}})/docs/deployments/gitops/manage-applications) +[Home dashboard]({{site.baseurl}}/docs/reporting/home-dashboard) +[DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics/) \ No newline at end of file diff --git a/_docs/deployment/images.md b/_docs/deployments/gitops/images.md similarity index 92% rename from _docs/deployment/images.md rename to _docs/deployments/gitops/images.md index d4538a528..ebe6af780 100644 --- a/_docs/deployment/images.md +++ b/_docs/deployments/gitops/images.md @@ -1,11 +1,12 @@ --- title: "Images in Codefresh" description: "" -group: deployment +group: deployments +sub_group: gitops toc: true --- -Building Docker images is one of the most basic requirements for creating Delivery Pipelines. +Building Docker images is one of the most basic requirements for creating Codefresh pipelines and Argo Workflows. Once you create an image, push the image to a registry, and report it to Codefresh, image information is continually updated in the Images page. ### Requirements for Images in Codefresh @@ -18,7 +19,7 @@ Complete the mandatory steps to see your Images in the Codefresh UI. Each step h 1. (Mandatory) Report image information to Codefresh. See the [report-image-info](https://github.com/codefresh-io/argo-hub/blob/main/workflows/codefresh-csdp/versions/0.0.6/docs/report-image-info.md){:target="\_blank"} example. -> If you are using an external GitHub Actions-based pipeline, we have a new template that combines image reporting and enrichment. See [Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/). +> If you are using an external GitHub Actions-based pipeline, we have a new template that combines image reporting and enrichment. See [Image enrichment with integrations]({{site.baseurl}}/docs/integrations/gitops/image-enrichment-overview/). ### Image views in Codefresh * In the Codefresh UI, go to [Images](https://g.codefresh.io/2.0/images){:target="\_blank"}. @@ -111,3 +112,10 @@ Selecting **more details** for an image tag. | **3** | The Git details for this image tag, such as the Git hash, the Jira issue number, Git Pull Request, commit information, the name of the user who performed the commit. | | **4** | The workflow for the image step. Select to go to the Workflow.| | **5** | The log information for the build image step in the relevant workflow. Select to view Logs panel. | + +## Related articles + +[Creating GitOps applications]({{site.baseurl}}/docs/deployments/gitops/create-application) +[Managing GitOps applications]({{site.baseurl}}/docs/deployments/gitops/manage-applications) +[Image enrichment with integrations]({{site.baseurl}}/integrations/image-enrichment-overview) +[Home dashboard]({{site.baseurl}}/docs/reporting/home-dashboard) diff --git a/_docs/deployment/install-argo-rollouts.md b/_docs/deployments/gitops/install-argo-rollouts.md similarity index 74% rename from _docs/deployment/install-argo-rollouts.md rename to _docs/deployments/gitops/install-argo-rollouts.md index 22f6afb73..0847b58c3 100644 --- a/_docs/deployment/install-argo-rollouts.md +++ b/_docs/deployments/gitops/install-argo-rollouts.md @@ -1,12 +1,13 @@ --- -title: "Install Argo Rollouts" +title: "Progressive delivery with GitOps" description: "" -group: deployment +group: deployments +sub_group: gitops toc: true --- -Install Argo Rollouts on managed clusters with a single click. With Argo Rollouts installed on your cluster, you can visualize rollout progress for deployed applications in the [Applications dashboard]({{site.baseurl}}/docs/deployment/applications-dashboard/#rollout-progress-visualization). +Install Argo Rollouts on managed clusters with a single click. With Argo Rollouts installed on your cluster, you can visualize rollout progress for deployed applications in the [Applications dashboard]({{site.baseurl}}/docs/deployments/gitops/applications-dashboard/#rollout-progress-visualization). If Argo Rollouts has not been installed, an **Install Argo Rollouts** button is displayed on selecting the managed cluster. 1. In the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. @@ -24,4 +25,4 @@ If Argo Rollouts has not been installed, an **Install Argo Rollouts** button is %} ### Related articles -[Add external clusters to runtimes]({{site.baseurl}}/docs/runtime/managed-cluster/) \ No newline at end of file +[Add external clusters to runtimes]({{site.baseurl}}/docs/installation/managed-cluster/) \ No newline at end of file diff --git a/_docs/deployment/manage-application.md b/_docs/deployments/gitops/manage-application.md similarity index 96% rename from _docs/deployment/manage-application.md rename to _docs/deployments/gitops/manage-application.md index 2ccd3e16e..15a0049d9 100644 --- a/_docs/deployment/manage-application.md +++ b/_docs/deployments/gitops/manage-application.md @@ -1,7 +1,8 @@ --- -title: "Managing applications" +title: "Managing GitOps applications" description: "" -group: deployment +group: deployments +sub_group: gitops toc: true --- @@ -49,8 +50,8 @@ Update General or Advanced configuration settings for a deployed application thr {:start="3"} 1. Update the **General** or **Advanced** configuration settings as needed: - [General configuration]({{site.baseurl}}/docs/deployment/create-application/#application-general-configuration-settings) - [Advanced configuration]({{site.baseurl}}/docs/deployment/create-application/#application-advanced-configuration-settings) + [General configuration]({{site.baseurl}}/docs/deployments/gitops/create-application/#application-general-configuration-settings) + [Advanced configuration]({{site.baseurl}}/docs/deployments/gitops/create-application/#application-advanced-configuration-settings) When you change a setting, the Commit and Discard Changes buttons are displayed. {% include @@ -218,7 +219,7 @@ For example, if you made changes to `api` resources or `audit` resources, type ` Delete an application from Codefresh. Deleting an application deletes the manifest from the Git repository, and then from the cluster where it is deployed. When deleted from the cluster, the application is removed from the Applications dashboard in Codefresh. >**Prune resources** in the application's General settings determines the scope of the delete action. -When selected, both the application and its resources are deleted. When cleared, only the application is deleted. For more information, review [Sync settings]({{site.baseurl}}/docs/deployment/create-application/#sync-settings). +When selected, both the application and its resources are deleted. When cleared, only the application is deleted. For more information, review [Sync settings]({{site.baseurl}}/docs/deployments/gitops/create-application/#sync-settings). Codefresh warns you of the implication of deleting the selected application in the Delete form. 1. In the Codefresh UI, go to the [Applications dashboard](https://g.codefresh.io/2.0/applications-dashboard/list){:target="\_blank"}. @@ -351,10 +352,9 @@ The table describes the options for the `Rollout` resource. ### Related articles -[Creating applications]({{site.baseurl}}/docs/deployment/create-application) +[Creating GitOps applications]({{site.baseurl}}/docs/deployments/gitops/create-application) [Home dashboard]({{site.baseurl}}/docs/reporting/home-dashboard) -[DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics) - +[DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics) diff --git a/_docs/deployments/helm/custom-helm-uploads.md b/_docs/deployments/helm/custom-helm-uploads.md new file mode 100644 index 000000000..33aa06d54 --- /dev/null +++ b/_docs/deployments/helm/custom-helm-uploads.md @@ -0,0 +1,125 @@ +--- +title: "Creating and uploading Helm packages" +description: "Manually create and upload Helm packages" +group: deployments +sub_group: helm +redirect_from: + - /docs/create-helm-artifacts-using-codefresh-pipeline/ +toc: true +--- + +Helm packages are just TAR files. Helm repositories are simple file hierarchies with an extra [index.yaml](https://helm.sh/docs/developing_charts/#the-chart-repository-structure){:target="\_blank"}. +You can run custom commands and manually upload indexes and packages to a Helm repo. + +>This articles shows some non-standard Helm examples. + For the basic use cases, or if you are just getting started with Helm, see our [Helm quick start guide]({{site.baseurl}}/docs/quick-start/ci-quickstart/deploy-with-helm/) and [Using Helm in pipelines]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/). + +## Package a Helm chart +Below is an example of a freestyle step in a Codefresh pipeline that packages the Helm chart and then extracts the chart name from the command output. It also saves that package name in an environment variable for later use. + + `YAML` +{% highlight yaml %} +{% raw %} +helm_package: + image: devth/helm + commands: + - cf_export PACKAGE=$(helm package| cut -d " " -f 8) +{% endraw %} +{% endhighlight %} + +The `helm package` command expects a path to an unpacked chart. Replace ` ` in the example with the directory that holds your chart files. Note that this directory must have the same name as the chart name, as per Helm requirements.
+See [Helm package docs](https://github.com/kubernetes/helm/blob/master/docs/helm/helm_package.md){:target="_blank"} and [Helm charts overview](https://github.com/kubernetes/helm/blob/master/docs/charts.md){:target="_blank"} for more information. + +{{site.data.callout.callout_info}} +To use `cf_export`and make the variable available to other steps in the pipeline, see [Variables in pipelines]({{site.baseurl}}/docs/pipelines/variables). +{{site.data.callout.end}} + +## Example 1: Push the chart to GCS based Helm Repository +The first example pushes the packaged chart into a public cloud storage service, like AWS S3, Azure Storage, or Google Cloud Storage. We chose Google Cloud Storage (GCS) for this example. +Our pipeline has three steps: + +{:start="1"} +1. download_index: download the Helm `index.yaml` file from GCS, or create one of it's not there. + +{:start="2"} +2. helm_package_merge: package the chart as described earlier, and also merge the new package into the downloaded `index.yaml` file, using the `helm repo index --merge` command. + +{:start="3"} +3. push_gcs: upload the updated `index.yaml` file and the newly created package to GCS. + + `YAML` +{% highlight yaml %} +{% raw %} +steps: + download_index: + image: appropriate/curl:latest + commands: + - 'curl https://storage.googleapis.com/$GOOGLE_BUCKET_NAME/index.yaml --output ./index.yaml --fail || :' + - '[ ! -f ./index.yaml ] && echo "apiVersion: v1">./index.yaml' + helm_package_merge: + image: devth/helm + commands: + - cf_export PACKAGE=$(helm package| cut -d " " -f 8) + - helm repo index --merge ./index.yaml . + push_gcs: + image: camil/gsutil + commands: + - echo -E $GOOGLE_CREDENTIALS > /gcs-creds.json + - echo -e "[Credentials]\ngs_service_key_file = /gcs-creds.json\n[GSUtil]\ndefault_project_id = $GOOGLE_PROJECT_ID" > /root/.boto + - gsutil cp ./index.yaml gs://$GOOGLE_BUCKET_NAME + - gsutil cp $PACKAGE gs://$GOOGLE_BUCKET_NAME +{% endraw %} +{% endhighlight %} + + +### Environment setup + +This pipeline references some predefined environment variables such as `GOOGLE_BUCKET_NAME`, `GOOGLE_PROJECT_ID` and `GOOGLE_CREDENTIALS`. +For this example, we created a service account with appropriate permissions in Google Cloud, and saved the credentials into `GOOGLE_CREDENTIALS` as a Codefresh Secret.
+For more information, see: +[Authenticating with Google services](https://cloud.google.com/storage/docs/authentication#service_accounts){:target="_blank"}.
+[Codefresh pipeline configuration and secrets](https://codefresh.io/docs/docs/codefresh-yaml/variables/#user-provided-variables){:target="_blank"}. + +## Example 2: Push the chart to Chart Museum +Chart Museum is a Helm repository *server* that has an HTTP API, pluggable backends, authentication, and more. +Read more about [Chart Museum](https://github.com/kubernetes-helm/chartmuseum){:target="_blank"}. + +In this example, we already have a Chart Museum server running, so we'll push the packaged chart to it. + +The steps will be: + +{:start="1"} +1. helm_package: package the chart as described earlier. + +{:start="2"} +2. get_repo_url: In order to avoid hard-coding the repository URL into the pipeline, we will retrieve it from the Codefresh Helm integration. +In this case, we have added our repository with Codefresh as described in [Using external Helml repos in Codefresh pipelines]({{site.baseurl}}/docs/deployments/helm/helm-charts-and-repositories). +Replace `` in the example with the name you gave to your repository when you added it to Codefresh. + +{:start="3"} +3. helm_push: call the Chart Museum HTTP api to just upload the package. Chart Museum will take care of the rest. + + `YAML` +{% highlight yaml %} +{% raw %} +steps: + helm_package: + image: devth/helm + commands: + - cf_export PACKAGE=$(helm package | cut -d " " -f 8) + get_repo_url: + image: codefresh/cli:latest + commands: + - cf_export HELM_URL=$(codefresh get ctx -o=yaml | grep repositoryUrl | cut -d "'" -f 2) + helm_push: + image: appropriate/curl + commands: + - curl --data-binary "@$PACKAGE" $HELM_URL/api/charts +{% endraw %} +{% endhighlight %} + + +## Related articles +[Using Helm in a Codefresh pipeline]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/) +[Using a managed Helm repository]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/) +[Helm environment promotion]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion) diff --git a/_docs/deployments/helm/helm-charts-and-repositories.md b/_docs/deployments/helm/helm-charts-and-repositories.md new file mode 100644 index 000000000..5973f4147 --- /dev/null +++ b/_docs/deployments/helm/helm-charts-and-repositories.md @@ -0,0 +1,111 @@ +--- +title: "Using external Helm repos in Codefresh pipelines" +description: "Use external Helm Charts and repositories in Codefresh pipelines" +group: deployments +sub_group: helm +toc: true +--- +Codefresh allows you to integrate with external Helm repositories and Helm charts in the Helm Charts page. +It is optional to use external Helm repositories as all Codefresh accounts already include a [built-in Helm repository]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/). + +## Add an external Helm repository + +Easily add your own Helm charts. +By default, we show charts from the [official Helm repository](https://github.com/kubernetes/charts){:target="_blank"}. + +1. In the Codefresh UI, from the Artifacts section in the sidebar, select [**Helm Charts**](https://g.codefresh.io/helm/releases/releasesNew/){:target="\_blank"}. +1. On the top right, click **Add Existing Helm Repository**. + You are taken to Pipeline Integrations. +1. In the Integrations page, click **Add Helm Repository**, and then select the type of Helm repo to add from the list. +1. Enter the **Helm repository name** and **URL**. + Do not include the specific path to `index.yaml` in the URL. + +{% include image.html +lightbox="true" +file="/images/deployments/helm/quick-helm-integration.png" +url="/images/deployments/helm/quick-helm-integration.png" +alt="Adding a Helm repository" +caption="Adding a Helm repository" +max-width="70%" +%} + +1. If your repository doesn't require authentication, to complete the process, click **Save**. + +For more details on adding Helm repositories, see [Helm integrations]({{site.baseurl}}/docs/integrations/helm/). + +## Use a Helm repository in a Codefresh pipeline + +Once connected, inject any Helm repository context into Codefresh pipelines. + +1. From the Pipelines page, select the pipeline into which to import the Helm configuation. +1. In the Workflows tab, do one of the following: + * Click **Variables** on the right, and then click the **Settings** (gear) icon. + * Click the context menu next to the settings icon. +1. Click on **Import from/Add shared configuration**, and select the name of the repository. + The repository settings are injected as environment variables into the pipeline. + +{% include image.html +lightbox="true" +file="/images/deployments/helm/connect-helm-repo.png" +url="/images/deployments/helm/connect-helm-repo.png" +alt="Connecting a Helm repository in the pipeline" +caption="Connecting a Helm repository in the pipeline" +max-width="70%" +%} + +1. If you are using the Helm step, the step uses these settings to connect to your authenticated repository automatically. For details, see [Using Helm in Codefresh pipelines]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/). + +## Install a chart from your Helm repository +Install a chart from a Helm repository to your cluster. + +* Values in the Chart Install wizard are provided in the following order: + 1. Chart Default Values (implicitly part of the chart). + 2. Overridden default values (provided as values file, provided only if edited by the user). + 3. Supplied values files from Yaml Shared Configuration. + 4. Override variables are provided as `--set` arguments. +* Variables available for custom pipelines: + If you select a custom pipeline, the following variables are available: + * `CF_HELM_RELEASE` - name of release + * `CF_HELM_KUBE_CONTEXT` - kubectl context name of target cluster (cluster name from [dashboard]({{site.baseurl}}/docs/deploy-to-kubernetes/manage-kubernetes/#work-with-your-services)) + * `CF_HELM_INSTALLATION_NAMESPACE` - desired namespace for the release + * `CF_HELM_CHART_VERSION` - Chart Version, + * `CF_HELM_CHART_NAME` - Chart Name + * `CF_HELM_CONTEXTS` - values from [shared configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/#using-shared-helm-values) + * `CF_HELM_VALUES` - extra values + * `CF_HELM_SET` - extra values, + * `CF_HELM_CHART_REPO_URL` - URL of Chart repository + * `CF_HELM_COMMIT_MESSAGE` - Message to show in Helm GUI, + +
+ +**Before you begin** +* Make sure tht you have a Kubernetes integration with the cluster and namespace, as described [here]({{site.baseurl}}/docs/deploy-to-kubernetes/add-kubernetes-cluster/) + +**How to** +1. In the Codefresh UI, from the Artifacts section in the sidebar, select [**Helm Charts**](https://g.codefresh.io/helm/releases/releasesNew/){:target="\_blank"}. +1. In the row with the chart to install, click **Install**. +1. Enter the **Release name** for the chart, and select the **Chart version** to install. +1. From Cluster Information, select a Kubernetes **Cluster** and the **Namespace** to install to. +1. Select the **Pipeline** to install to. +1. If required, edit the **Default Chart Values** to view and override them. + When the default values yaml is changed, it is provided to Helm install as a values file. You can revert to the original values cby clicking Revert. +1. To provide additional values files, do the following: + * From the **Import from configuration** list, select **Add new context of type: YAML**. + * Enter the **Context name**. + * Insert your values YAML, and click **Save**. + The YAML is saved and added to the list of configuration files that you can import from. +1. To override variable values, click **+Add variable**, and then enter the Key and Value. + > The order of value configurations matter for Helm: most recently provided values override earlier ones. +1. Click **Install**. You can observe the newly installed release in Helm Releases. + +You can also install Helm releases from [any Helm environment board]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion). + + +## Related articles +[Using Helm in a Codefresh pipeline]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/) +[Helm integrations]({{site.baseurl}}/docs/integrations/helm/) +[Helm Dashboard]({{site.baseurl}}/docs/deployments/helm/helm-releases-management) +[Helm Promotion boards]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion) +[Helm best practices]({{site.baseurl}}/docs/ci-cd-guides/helm-best-practices/) + + diff --git a/_docs/deployments/helm/helm-environment-promotion.md b/_docs/deployments/helm/helm-environment-promotion.md new file mode 100644 index 000000000..21466e5dc --- /dev/null +++ b/_docs/deployments/helm/helm-environment-promotion.md @@ -0,0 +1,290 @@ +--- + +title: "Promoting Helm Environments" +description: "Manage your Helm Environments with the Codefresh UI" +group: deployments +sub_group: helm +toc: true +--- +Apart from the [Helm Releases]({{site.baseurl}}/docs/deployments/helm/helm-releases-management) that show your Kubernetes clusters at the application level, Codefresh also comes with a special environment board that allows you to track one or more applications as they move within your infrastructure (example, Dev, QA, Prod). + +The environment board can function both as an overview of the whole lifecycle of the application, as well as a tool to shift-left/right Helm releases between environments. + +Here is an example board: + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/board.png" +url="/images/deployments/helm/promotion/board.png" +alt="Helm Environment Dashboard" +caption="Helm Environment Dashboard" +max-width="80%" +%} + +This board has three environments that correspond to Kubernetes clusters: + * A Load-testing environment where applications are stress-tested + * A Staging environment where smoke tests are performed + * The Production environment where applications go live + +You can see that a Python example app at version 0.2.0 is already in production. Version 0.3.0 is waiting in the staging environment for smoke tests. Once it is tested it can be dragged to the production column therefore *promoting* it to production status. + + +## Using the Helm Environment Board + +You can create and manage as many Helm promotion boards as you want. +For each board, you define how many columns it will contain, where each column is a Helm-enabled Kubernetes cluster. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/helm-environments.png" +url="/images/deployments/helm/promotion/helm-environments.png" +alt="Helm environments column structure" +caption="Helm environments column structure" +max-width="80%" +%} + +You can use different clusters for each column or different namespaces from the same cluster. You can even mix and match both approaches. +As an example, you could create a Helm board with the following environments: + +* Column 1, dev cluster showing all namespaces (DEV) +* Column 2, namespace qa from cluster staging (QA) +* Column 3, namespace staging from cluster staging (STAGING) +* Column 4, namespace production from cluster prod (PRODUCTION) + +Once you have your columns in place, you can move Helm releases between clusters/namespaces by drag-n-drop. Each Helm release can be dragged to any other column either promoting it, for example, from QA to Production, or shifting it left, for example, from Production to QA. + +## Creating a custom Helm Board + +Create your own Helm board with a single or multiple Helm applications. You can create as many boards as you want. + +1. In the Codefresh UI, from the DevOps Insights section in the sidebar, select [**Helm Boards**](https://g.codefresh.io/helm/helm-kanban/){:target="\_blank"}. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/board-selection.png" +url="/images/deployments/helm/promotion/board-selection.png" +alt="Helm board selection" +caption="Helm board selection" +max-width="80%" +%} + +{:start="2"} +1. On the top-right, click **Add board**. +1. Enter the title of your board as the **Board Name**. +1. Optional. In the **Release name regex expression** field, enter the Regex expression for this board to filter all its environments to show only Helm releases that match this regular expression. + Regex expressions are very helpful if you want your environment board to focus only on a single or set of Helm applications. + To see all Helm releases of your clusters, leave empty. + +You can edit both options for an existing board if you change your mind later. + +### Define Clusters/Namespaces for each Environment + +Once you create your Helm environment board, you are ready to define its columns. + +* To add a column, on the top-right, click **Add environment***. + You will see the environment details dialog: + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/edit-helm-environment.png" +url="/images/deployments/helm/promotion/edit-helm-environment.png" +alt="Edit Helm environment" +caption="Edit Helm environment" +max-width="50%" +%} + + For each environment you can select: + * A name for that column + * The Kubernetes cluster it corresponds to + * One or more namespaces that define this environment (You can even toggle the switch for a regex match) + * A custom pipeline that will be used when a Helm release is installed for the first time in this column + * A custom pipeline that will be used when a Helm release is dragged in this column (promoted from another column) + * Optional. One or more charts to use for the environment. Defining charts for the environment saves you from having to search through all the charts in your Helm repository. When you install an application from the install graphical dialog, only the selected chart(s) are displayed. + * A presentation color to easily identify the environment on the board (For example, a "production" environment should have a red color) + +You can also select no namespace at all. In that case, the column will show Helm releases for all namespaces in that cluster. +You can change all these options after creation, so feel free to change your mind. + +Repeat the same process for additional environments. Remember that you can name your environment as you want and define any combination of cluster/namespace for any of the columns. This gives you a lot of power to define a Helm environment board that matches exactly your own process. + +You don't have to define the environments in order. You can drag-n-drop columns to change their order after the initial creation. + + +### Installing Helm Releases on each Environment + +If you already have [pipelines that deploy Helm releases]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/), your columns are populated automatically with information. + +For each Helm release, you will get some basic details such as the chart version and the name of the release. You can expand a release by clicking on the arrow button to get additional information such as the docker images and the replicas of each pod that are contained in the release. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/expand.png" +url="/images/deployments/helm/promotion/expand.png" +alt="Helm release details" +caption="Helm release details" +max-width="50%" +%} + +You can even install manually a Helm release from any external repository by clicking on the *PLUS* button at the header of each column. In that case you will see a list of possible Helm applications to choose from. + +You will be able to select the target cluster and namespace as well as the chart values [as any other Helm release]({{site.baseurl}}/docs/deployments/helm/helm-charts-and-repositories/#install-chart-from-your-helm-repository). + + +## Moving Releases between Environments + +A Helm environment board can be used by different stakeholders in order to get the detailed status of all defined environments. In that aspect it can act as a read-only tool that simply shows the results of Codefresh pipelines that deploy Helm applications. + +### Promoting Helm Releases with the UI + +You can also use the board as an action tool in order to promote/demote a Helm release between individual environments. To move a Helm release between environments just drag-n-drop it to a different column. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/shift-right.png" +url="/images/deployments/helm/promotion/shift-right.png" +alt="Promoting a Helm release" +caption="Promoting a Helm release" +max-width="80%" +%} + +Once you drop the release you will also see the promotion dialog. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/promote-settings.png" +url="/images/deployments/helm/promotion/promote-settings.png" +alt="Promotion Settings" +caption="Promotion Settings" +max-width="40%" +%} + +All fields here will be auto-filled according to the Helm release that you dragged. You can also choose a custom pipeline (see below) for the promotion if you don't want to use the default one. + +By clicking the *Variables* button you can override the chart values, import a specific shared configuration or add new values. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/value-options.png" +url="/images/deployments/helm/promotion/value-options.png" +alt="Changing deployment values" +caption="Changing deployment values" +max-width="40%" +%} + +By default Codefresh will use a built-in install/upgrade pipeline for performing the promotion. You can choose your own pipeline from the promotion dialog. That pipeline will be automatically provided with the following [environment variables]({{site.baseurl}}/docs/deployments/helm/helm-releases-management/#overriding-the-default-helm-actions): + +* `CF_HELM_RELEASE` - name of release +* `CF_HELM_KUBE_CONTEXT` - `kubectl` context name of target cluster (cluster name from [dashboard]({{site.baseurl}}/docs/deploy-to-kubernetes/manage-kubernetes/#work-with-your-services)) +* `CF_HELM_NAMESPACE` - Tiller Namespace if you use Helm 2 +* `CF_HELM_INSTALLATION_NAMESPACE` - namespace where release is promoted to +* `CF_HELM_CONTEXTS` - [shared configuration]({{site.baseurl}}/docs/configure-ci-cd-pipeline/shared-configuration) Helm contexts +* `CF_HELM_VALUES` - Helm chart values +* `CF_HELM_SET` - Additional values there were overriden +* `CF_HELM_CHART_JSON_GZIP` - Gzipped JSON of Helm chart (only for Helm 3) +* `CF_HELM_CHART_JSON` - JSON of Helm chart (only for Helm 2) +* `CF_HELM_BOARD` - Name of the board that is used for the drag-n-drop-action +* `CF_HELM_TARGET_SECTION` - Name of the Source Environment that you are promoting from +* `CF_HELM_SOURCE_SECTION` - Name of the Target Environment that you are promoting to + + +Note that the variable `CF_HELM_CHART_JSON_GZIP` is both compressed and base64 encoded. To get the raw value you need a command like `echo $CF_HELM_CHART_JSON_GZIP | base64 -d | gunzip` + +>Overriding the default pipeline can only be done by [Codefresh admin users]({{site.baseurl}}/docs/administration/access-control/#users-and-administrators). + +Once you click the *update* button, a new build will run that will perform the deployment. + +Note that you can move releases to any column both on the right and on the left of the current column. This is helpful if for example you find a bug in your production environment and you want to bring it back to a staging environment for debugging. + +### Promoting Helm releases programmatically + +You can also promote Helm releases with the [Codefresh CLI](https://codefresh-io.github.io/cli/predefined-pipelines/promote-helm-release/){:target="\_blank"}. + +Once you have [installed the CLI](https://codefresh-io.github.io/cli/getting-started/){:target="\_blank"}, you can use it from an external script or terminal with the `helm-promotion` parameter: + +{% highlight shell %} +{% raw %} +codefresh helm-promotion --board MySampleBoard --source Staging --target Production --source-release my-app --set myenv=prod +{% endraw %} +{% endhighlight %} + +Here we promote the Helm release `my-app` to the *Production* column overriding also the `myenv` value. + +Remember that the Codefresh CLI can also run in a Codefresh pipeline with a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). +Here is an example of a Helm promotion from within a Codefresh pipeline. + + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + triggerstep: + title: trigger + image: codefresh/cli + commands: + - 'codefresh helm-promotion --board MySampleBoard --source Staging --target Production --source-release my-app --namespace my-namespace --set myenv=prod' +{% endraw %} +{% endhighlight %} + +## Viewing the promotion pipeline + +When you promote a Helm Release for a Board, you can view the pipeline for that release. + +1. Click on Boards under the Helm section on the left-hand side +2. Select the board you want to view +3. Select the Builds tab on the top +4. Here, you can see the Promotion Pipelines / builds for promoting a Release + +## Editing your Helm Boards + +For any existing Helm board, you have the following options: + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/board-management.png" +url="/images/deployments/helm/promotion/board-management.png" +alt="Editing a Helm environment" +caption="Editing a Helm environment" +max-width="80%" +%} + + +1. The refresh button will update the board with the current state of the clusters +1. The filtering menu can be used to further constrain the Helm releases shown in each column. +1. The *edit properties* button allows you to change again the title of the board as well as a global filter for Helm releases +1. The *remove board* completely deletes the present board from the Codefresh UI +1. The environment details on the environment header are: +* The edit button to change again the options for this column (shown on mouse hover) +* The delete button to remove this column from the board (shown on mouse hover) +* The plus button to install a new chart. If you selected one or more charts when you defined your environment, only the selected charts are displayed. +* A numeric value that shows how many releases are contained on this environment +1. The delete button allows you to uninstall a Helm release for an environment + +The filtering options allow you to further constrain the Helm release shown for the whole board. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/promotion/filter.png" +url="/images/deployments/helm/promotion/filter.png" +alt="Filtering options" +caption="Filtering options" +max-width="50%" +%} + +The filters are especially helpful in Helm boards with large numbers of environments and/or releases. + +## Related articles +[Using Helm in a Codefresh pipeline]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/) +[Using external Helml repos in Codefresh pipelines]({{site.baseurl}}/docs/deployments/helm/helm-charts-and-repositories/#add-helm-repository) +[Managing Helm releases]({{site.baseurl}}/docs/deployments/helm/helm-releases-management) +[Environment Dashboard]({{site.baseurl}}/docs/deploy-to-kubernetes/environment-dashboard/) diff --git a/_docs/deployments/helm/helm-releases-management.md b/_docs/deployments/helm/helm-releases-management.md new file mode 100644 index 000000000..7f0154acd --- /dev/null +++ b/_docs/deployments/helm/helm-releases-management.md @@ -0,0 +1,263 @@ +--- +title: "Managing Helm releases" +description: "Manage Helm deployments from the Codefresh UI" +group: deployments +sub_group: helm +redirect_from: + - /docs/helm-releases-management/ + - /docs/deployments/helm/helm3/ +toc: true +--- +Codefresh has built-in integration for Helm that provides a unique view into your production Kubernetes cluster. +In Helm Releases, you can see the current status of your cluster, including the currently deployed releases, their previous revisions including change tracking, and even roll back to a previous release. + +Codefresh also offers [an environment view for Helm releases]({{site.baseurl}}/docs/deploy-to-kubernetes/environment-dashboard/) as well as [a promotion dashboard]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion). + + +## View Helm releases and release information + +View all the Helm releases in your cluster, and drill down into a specific release to see its services, deployed versions, manifests and more. + +> Make sure you have [connected your Kubernetes cluster]({{site.baseurl}}/docs/integrations/kubernetes/adding-non-gke-kubernetes-cluster/) to Codefresh. + +1. In the Codefresh UI, from the DevOps Insights section in the sidebar, select [**Helm Releases**](https://g.codefresh.io/helm/releases/releasesNew/){:target="\_blank"}. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/helm-release-dashboard.png" +url="/images/deployments/helm/helm-release-dashboard.png" +alt="Helm Releases" +caption="Helm Releases" +max-width="90%" +%} + + + + +{:start="2"} +1. To see details for a specific release, click the release name. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/services.png" +url="/images/deployments/helm/services.png" +alt="Kubernetes Services" +caption="Kubernetes Services" +max-width="70%" +%} + +The History tab shows all previous releases. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/history.png" +url="/images/deployments/helm/history.png" +alt="Helm History" +caption="Helm History" +max-width="60%" +%} + +You can further expand a release revision to see exactly what files were changed in this release. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/diff.png" +url="/images/deployments/helm/diff.png" +alt="Helm diff" +caption="Helm diff" +max-width="60%" +%} + +There are other tabs that show you the chart used, the values as well as the final manifests that were actually deployed. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/manifests.png" +url="/images/deployments/helm/manifests.png" +alt="Final rendered manifests" +caption="Final rendered manifests" +max-width="50%" +%} + +## Add labels to Kubernetes services + +For better visibility into services, add the [recommended labels](https://helm.sh/docs/topics/chart_best_practices/labels/){:target="\_blank"} to your Kubernetes service. + +{% highlight yaml %} +{% raw %} + apiVersion: v1 +kind: Service +metadata: + name: {{ template "fullname" . }} + labels: + app.kubernetes.io/name: "{{ template "name" . }}" + helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}" + app.kubernetes.io/managed-by: "{{ .Release.Service }}" + app.kubernetes.io/instance: "{{ .Release.Name }}" +{% endraw %} +{% endhighlight %} + +To use the instance label for something different, you can also use a release label instead: + +{% highlight yaml %} +{% raw %} +release: {{ .Release.Name }} +{% endraw %} +{% endhighlight %} + + + +## Add an upgrade message + +Codefresh allows you to display a meaningful description for each release in the release history. This message +can help show the main reason behind each release, or any other message that is convenient for you. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/helm-commit-message.png" +url="/images/deployments/helm/helm-commit-message.png" +alt="Helm release message" +caption="Helm release message" +max-width="70%" +%} + +You can set this message for your Helm release in three ways: + +1. When you manually install a Helm release from the [Helm charts screen]({{site.baseurl}}/docs/deployments/helm/helm-charts-and-repositories/#install-chart-from-your-helm-repository), there is a field for this message. +1. Set the property `commit_message` inside the [notes.txt](https://helm.sh/docs/chart_template_guide/notes_files/){:target="\_blank"} file of your chart. +1. By providing an environment variable called `COMMIT_MESSAGE` within your [pipeline Helm step]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/). + + +## Roll back a Helm release + +You can rollback to a previous revision of a release in the History tab. + +1. Click the Helm release for which to perform a rollback, and then click the **History** tab. +1. To rollback to a specific release, click **Rollback** in the row. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/rollback.png" +url="/images/deployments/helm/rollback.png" +alt="Rolling back to a previous release" +caption="Rolling back to a previous release" +max-width="50%" +%} + +>It takes time to complete rollback for a release, and the change in the cluster is not instantly updated in the Codefresh UI. If you also use a [custom rollback pipeline](#overriding-the-default-helm-actions), the delay between the cluster update and the UI refresh is even longer. + +## Helm UI actions + +From the main release screen, you have some additional actions. + +You can issue a [Helm test](https://github.com/kubernetes/helm/blob/master/docs/chart_tests.md) by clicking on the 'Run Test' button on the desired chart row. + +You can delete a release by clicking on the 'Delete' button on the desired chart row. +For deletion options, see the [helm delete documentation](https://github.com/kubernetes/helm/blob/master/docs/helm/helm_delete.md){:target="\_blank"}, for example, *purge* will remove the revision from the release history. + +## Helm deployment badge + +Similar to a [build badge]({{site.baseurl}}/docs/pipelines/build-status/#using-the-build-badge), you can also get a deployment badge for a Helm release. + +1. In the Codefresh UI, from the DevOps Insights section in the sidebar, select [**Helm Releases**](https://g.codefresh.io/helm/releases/releasesNew/){:target="\_blank"}. +1. In the row with the Helm release for which to add a deployment badge, click the **Settings** (gear) icon. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/helm-badge.png" +url="/images/deployments/helm/helm-badge.png" +alt="Helm Deployment badge" +caption="Helm Deployment badge" +max-width="60%" +%} + +{:start="3"} +1. To get deployment information, click **Badge**. + Codefresh provides the Markdown/HTML/Link segment that you can embed in README or other documents to show deployment information. + +## Overriding default Helm actions for releases + +By default, when you take an action in the UI, Codefresh executes the native Helm command corresponding to that action: + +* `helm test` for testing a chart +* `helm rollback` for rollbacks +* `helm delete` or `helm uninstall --keep-history` for delete +* `helm delete --purge ` or `helm uninstall ` for purging a release + +You can override these actions for a specific Helm release by defining custom pipelines for each action. This way you can add your extra logic on top of these actions. For example your own Helm uninstall pipeline might also have a notification step that posts a message to a Slack channel after a release is removed. + +>Only [Codefresh admin users]({{site.baseurl}}/docs/administration/access-control/#users-and-administrators) can override the default pipelines defined for a Helm release. + +1. In the Codefresh UI, from the DevOps Insights section in the sidebar, select [**Helm Releases**](https://g.codefresh.io/helm/releases/releasesNew/){:target="\_blank"}. +1. In the row with the Helm release for which to override default actions, click the **Settings** (gear) icon. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/override-helm-actions.png" +url="/images/deployments/helm/override-helm-actions.png" +alt="Changing default Helm actions" +caption="Changing default Helm actions" +max-width="50%" +%} + +{:start="3"} +1. Select the pipeline to use for the respective actions. + +### Environment variables for custom Helm commands +If you do override any of these actions, the following [environment variables]({{site.baseurl}}/docs/codefresh-yaml/variables/) are available in the respective pipeline, so that you can use your own custom Helm command. + +**Helm Test pipeline** +* `CF_HELM_RELEASE`: Name of release +* `CF_HELM_KUBE_CONTEXT`: `kubectl` context name of target cluster (cluster name from [dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/#work-with-your-services)) +* `CF_HELM_NAMESPACE`: Namespace where release is stored +* `CF_HELM_TIMEOUT`: Time in seconds to wait for any individual Kubernetes operation +* `CF_HELM_CLEANUP`: Delete test pods upon completion + + + +**Helm Rollback pipeline** +* `CF_HELM_VERSION`: Helm version, ex.: 3.0.1, 2.7.0 +* `CF_HELM_RELEASE`: Name of release on cluster +* `CF_HELM_REVISION`: Revision to use for rollback +* `CF_HELM_KUBE_CONTEXT`: `kubectl` context name of target cluster (cluster name from [dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/#work-with-your-services)) +* `CF_HELM_NAMESPACE`: Namespace where release is stored + + +**Helm Delete pipeline** +* `CF_HELM_PURGE`: Boolean, delete release from store +* `CF_HELM_RELEASE`: Name of release +* `CF_HELM_TIMEOUT`: Time in seconds to wait for any individual Kubernetes operation +* `CF_HELM_HOOKS`: Prevent hooks from running during install +* `CF_HELM_KUBE_CONTEXT`: `kubectl` context name of target cluster (cluster name from [dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/#work-with-your-services)) +* `CF_HELM_VERSION`: Helm version, ex.: 3.0.1, 2.7.0 +* `CF_HELM_NAMESPACE`: Namespace where release is stored + + +## Related articles +[Using Helm in a Codefresh pipeline]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/) +[Helm charts and repositories]({{site.baseurl}}/docs/deployments/helm/helm-charts-and-repositories/) +[Codefresh-managed Helm Repositories]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/) +[Helm promotion boards]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion) \ No newline at end of file diff --git a/_docs/deployments/helm/managed-helm-repository.md b/_docs/deployments/helm/managed-helm-repository.md new file mode 100644 index 000000000..6b27d3ade --- /dev/null +++ b/_docs/deployments/helm/managed-helm-repository.md @@ -0,0 +1,137 @@ +--- +title: "Using a managed Helm repository" +description: "Use the Codefresh integrated Helm repository" +group: deployments +sub_group: helm +toc: true +--- + +Codefresh provides fully managed, hosted Helm repositories for users. +While we automatically create a default managed repo for every Codefresh account, you can also add [external Helm repositories]({{site.baseurl}}/docs/deployments/helm/helm-charts-and-repositories/). + +The built-in Helm repo that Codefresh creates, is private by default, allowing access only via Codefresh or via a Codefresh API token. + +> Tip: + You may be familiar with the popular open source Helm repository implementation called 'ChartMuseum', that Codefresh sponsors. Codefresh-managed repositories are based on, and therefore compatible with, ChartMuseum and its unique features. For details, see [ChartMuseum](https://github.com/kubernetes-helm/chartmuseum){:target="\_blank"}. + +## View Helm repository integrations + +The Codefresh-managed Helm repo is displayed with other Helm repositories you have added to Helm integrations. + +>You cannot delete the built-in Helm repo that Codefresh creates for you. + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select **Pipeline Integrations**. +1. Scroll to **Helm Repositories**, and then click **Configure**. + All the Helm integrations you set up are displayed. + +{% include +image.html +lightbox="true" +file="/images/deployments/helm/managed-helm-repo.png" +url="/images/deployments/helm/managed-helm-repo.png" +alt="Codefresh built-in Helm repository" +caption="Codefresh built-in Helm repository" +max-width="50%" +%} + + +## Get the chart repository URL +Get the chart repository URL for any Helm integration. +The URL is in the format: `cm://h.cfcr.io// `, where the default repo is `default`. + +* From the list of Helm integrations, select the integration and then click the **Edit** icon on the left. + The Helm Repository URL field displays the chart URL. + +## Codefresh Helm dashboards + +The Helm Charts and Helm Releases dashboards are automatically configured to work with your default managed repo to easily install charts and manage releases. +For more information, see [install chart from a Helm repository]({{site.baseurl}}/docs/deployments/helm/helm-charts-and-repositories/#install-chart-from-your-helm-repository) and [Managing Helm releases]({{site.baseurl}}/docs/deployments/helm/helm-releases-management/). + +## Use Codefresh CLI for advanced Helm repo management + +The Codefresh CLI supports advanced management options for your managed repository, without having to log in to the Codefresh UI. +For more information on CLI support for Helm repos, see the [CLI documentation on Helm Repos](https://codefresh-io.github.io/cli/helm-repos/){:target="\_blank"}. + + +## Set access level for managed repo + +The managed Helm repository supports two modes of access: +* Private +* Public + +By default, the managed Helm repo is created with `Private` access, meaning that read/write access is protected by Codefresh authentication. + +You can switch the access level to `Public`, which will make the repository accessible to anonymous users only *for read operations*. Write operations, even in public access mode, always require authentication. +Be very careful when you make your repo public, as the whole world will be able to access your charts. We recommend this setting only for quick demos and POCs. + +**How to** + +* Use the Codefresh CLI to toggle access level on a managed repo: + +{% highlight bash %} +codefresh patch helm-repo mycfrepo -public +{% endhighlight %} + +For more info, see the relevant section in the [Codefresh CLI documentation](https://codefresh-io.github.io/cli/helm-repos/update-helm-repo/){:target="\_blank"}. + +## Working with Helm CLI + +The private Helm repository offered by Codefresh is a standard Helm repo and will work with the vanilla Helm executable even outside of the Codefresh UI. +We suggest using the private [Helm repo from Codefresh pipelines]({{site.baseurl}}/docs/example-catalog/cd-examples/helm/), but you can also use it from your workstation. + +### Add a Public repo to Helm + +If your repo is set to `public` access mode, you can use it just like any other HTTP Helm repository. +You can: + +{% highlight bash %} +helm repo add mycfrepo https://h.cfcr.io/ / +{% endhighlight %} + +### Add a Private repo to Helm + +If your repo is set to `private` access mode, the default, then the Helm client needs to authenticate with Codefresh. +To authenticate, you can use ChartMuseum's 'Helm Push' CLI plugin which adds support for authentication and chart manipulation on top of the basic Helm CLI functionality. + +We highly recommend that you familiarize yourself with the [Helm Push plugin](https://github.com/chartmuseum/helm-push){:target="\_blank"}. + +#### Install the Helm Push plugin + +{% highlight bash %} +helm plugin install https://github.com/chartmuseum/helm-push +{% endhighlight %} + +#### Configure the Helm Push plugin + +If you have the Codefresh CLI installed and configured, there's nothing you need to do. The Helm Push plugin picks up your settings automatically. +To learn about getting started with Codefresh CLI, see [CLI getting started](https://codefresh-io.github.io/cli/getting-started/). +To learn about manual authentication without depending on the Codefresh CLI, see [here](https://github.com/chartmuseum/helm-push#token). + +#### Add the private repo + +{% highlight bash %} +helm repo add mycfrepo cm://h.cfcr.io/kostis-codefresh/default +{% endhighlight %} + +Notice the protocol is `cm://` instead of `https://`. This indicates the custom authentication scheme supported by ChartMuseum Helm Push plugin. + +## Using in a Codefresh pipeline + +The Codefresh Helm plugin automatically handles authentication for managed repositories. You can use the plugin as you usually would. For more information, see the [Codefresh Helm plugin]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/). + +## Removing a Helm chart from a private Codefresh repository + +You can delete a Helm chart from your own Helm repository with the following HTTP call. + +{% highlight bash %} +curl -X DELETE -v -H "Authorization: Bearer " https://h.cfcr.io/api/ / /charts/ / +{% endhighlight %} + +Replace values in `<>` with your own (also removing `<>` in the process). + +Generate an api key from [https://g.codefresh.io/user/settings](https://g.codefresh.io/user/settings) as explained in the [API page]({{site.baseurl}}/docs/integrations/codefresh-api/). + +## Related articles +[Using Helm in a Codefresh pipeline]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/) +[Helm integration]({{site.baseurl}}/docs/integrations/helm/) +[Managing Helm releases]({{site.baseurl}}/docs/deployments/helm/helm-releases-management) diff --git a/_docs/deployments/helm/using-helm-in-codefresh-pipeline.md b/_docs/deployments/helm/using-helm-in-codefresh-pipeline.md new file mode 100644 index 000000000..f8e0b33bc --- /dev/null +++ b/_docs/deployments/helm/using-helm-in-codefresh-pipeline.md @@ -0,0 +1,346 @@ +--- +title: "Using Helm in a Codefresh pipeline" +description: "Deploy and push Helm charts with Codefresh" +group: deployments +sub_group: helm +redirect_from: + - /docs/deployments/helm/create-helm-artifacts-using-codefresh-pipeline/ + - /docs/install-helm-chart-using-codefresh-pipeline/ +toc: true +--- + +We created a [special Helm step](https://codefresh.io/steps/step/helm){:target="\_blank"} for easy integration of Helm in Codefresh pipelines. The Helm step facilitates authentication, configuration, and execution of Helm commands. + +> If you have a special use case that is not covered by the Codefresh Helm step, you can always use the regular `helm` cli in a freestyle step. + In this case, you can use the simpler container `codefresh/kube-helm` which includes only Kubectl and helm tools. `kube-helm` is available on DockerHub: [https://hub.docker.com/r/codefresh/kube-helm/](https://hub.docker.com/r/codefresh/kube-helm/){:target="\_blank"}. + +If you are just starting with Helm, refer to our [Helm quick start guide]({{site.baseurl}}/docs/quick-start/ci-quickstart/deploy-with-helm/) . And, if you prefer to work directly with code, see our [full Helm example]({{site.baseurl}}/docs/example-catalog/cd-examples/helm/). + +## Helm setup + + + +To use Helm in your Codefresh pipeline you must do the following: + +1. Make sure that your application has a [Helm chart](https://helm.sh/docs/chart_template_guide/getting_started/) +1. Create a Helm package for your application from the chart +1. [Add a Kubernetes cluster]({{site.baseurl}}/docs/deploy-to-kubernetes/add-kubernetes-cluster/) in Codefresh +1. Define a Helm repository or use the [one offered by Codefresh to all accounts]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/) +1. Import the Helm [configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/) into your pipeline variables +1. Use the Helm step in your [yml build definition]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) + +Let's see these steps in order. + +### Step 1: Create a Helm chart for your application + +Helm applications are bundled in special archives called *Charts*. You can create a Helm +chart for your application by following [the official documentation on charts](https://helm.sh/docs/chart_template_guide/getting_started/){:target="\_blank"}. + +The example Codefresh application includes a [sample chart](https://github.com/codefresh-contrib/python-flask-sample-app/tree/with-helm/charts/python){:target="\_blank"}, used in our Helm quick start guide, mentioned earlier in this article. + +You can create the chart manually or by using the [helm create](https://helm.sh/docs/helm/#helm-create){:target="\_blank"} command on your workstation. There are also several third-party tools that can create Helm packages for you such as [Draft](https://draft.sh/){:target="\_blank"}. + +Once your Helm chart is ready, commit it to a folder called `charts`, in the same Git repository that contains the source code of your application. Codefresh can also work with Helm charts that are in different Git repositories. We suggest however that you keep both the source code and the Helm chart of an application in the same Git repository to make chart management much easier. + + +### Step 2: Select Kubernetes cluster for deployment + +The Helm pipeline step requires the configuration of a `kube_context` variable that determines the Kubernetes cluster used for the deployment. + +1. Connect your Kubernetes cluster with Codefresh, as described [here]({{site.baseurl}}/docs/deploy-to-kubernetes/add-kubernetes-cluster/). + +1. Provide the cluster to the Helm step by adding the `KUBE_CONTEXT` variable, where the value is the connection *name* entered when you created the connection. +> The connection name also appears as the title of the cluster in Kubernetes integration settings (Account Settings >Integrations > Kubernetes). + +{% include image.html +lightbox="true" +file="/images/deployments/helm/k8s-name.png" +url="/images/deployments/helm/k8s-name.png" +alt="Name of Kubernetes cluster" +caption="Name of Kubernetes cluster" +max-width="70%" +%} + +1. Verify that your cluster is set up for Helm, from the sidebar, below DevOps Insights, select **Helm Releases**. + The [Helm releases]({{site.baseurl}}/docs/deployments/helm/helm-releases-management/) in your cluster are displayed. If you have just started using Helm, the release page may be empty. + +### Step 3: Define a Helm repository + +To push your chart to a Helm repository, configure the target repository to work with. +Always a good practice to save Helm charts in Helm repositories, Codefresh supports a variety of private, authenticated Helm repositories +in addition to public HTTP repositories. Codefresh also provides a free, managed Helm repository for every account. + +* Either [connect your repository with Codefresh]({{site.baseurl}}/docs/deployments/helm/add-helm-repository/) +OR +* Obtain your [managed Helm repository URL]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/#chart-repository-url) + + +### Step 4: (Optional) Import the Helm configuration into your pipeline definition + +Once you have a connected to a Helm repository, attach it to the pipeline. + +1. Frpm the Pipelines page, select the pipeline into which to import the Helm configuation. +1. In the Workflows tab, do one of the following: + * Click **Variables** on the right, and then click the Settings (gear) icon in the variables section on the right. + * Click the context menu next to the settings icon. +1. Click on **Import from/Add shared configuration**, and from the list, select `CF_HELM_DEFAULT`. See [shared configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/). + +{% include image.html +lightbox="true" +file="/images/deployments/helm/import-helm-configuration.png" +url="/images/deployments/helm/import-helm-configuration.png" +alt="Connecting a Helm repository in the pipeline" +caption="Connecting a Helm repository in the pipeline" +max-width="50%" +%} + + +### Step 5: Use the Helm freestyle step in the pipeline + +You can now use the Helm freestyle step in the `codefresh.yml` file. This step is only needed in pipelines that actually upload/fetch Helm charts to/from Helm repositories. If your pipeline directly installs a Helm chart from the Git filesystem, there is no need to import a Helm configuration. + +>Currently, you can use only one Helm configuration in the same pipeline. We are aware +of this limitation and will soon improve the way Codefresh works with multiple Helm configurations. + + + +* Use the Helm typed step from the [Step Marketplace](https://codefresh.io/steps/step/helm){:target="\_blank"}. +* Configure the Helm step using environment variables, as described [here]({{site.baseurl}}/docs/codefresh-yaml/variables/#user-provided-variables). + +The example below illustrates how to provide variables as part of the Helm step definition: + +```yaml +deploy: + type: helm + arguments: + action: install + chart_name: test_chart + release_name: first + helm_version: 3.0.3 + kube_context: my-kubernetes-context + custom_values: + - 'pat.arr="{one,two,three}"' + - 'STR_WITH_COMAS="one\,two\,three"' +``` + + + +#### Helm step action modes + +The Helm step can operate in one of three modes, as defined by the `action` field: + +1. `install`: Installs the chart into a Kubernetes cluster. This is the default mode if not explicitly set. +2. `push`: Packages the chart and pushes it to the repository. +3. `auth`: Authenticate only. Only sets up authentication and adds the repo to the Helm. This mode is useful to write your own Helm commands using the freestyle step's `commands` property, but still allow the step to handle authentication. + + +#### Helm values + +* To supply a value file, add to the Helm step, `custom_values_file`, with the value pointing to an existing values file. +* To override specific values, add to the Helm step, `custom_values` followed by the path to the value to set. For example, `myservice_imageTag`. Note that `.` (dot) should be replaced with `_` (underscore). The value of the variable is used to override or set the templated property. + +Examples: +```yaml +... + custom_values: + - 'myimage_pullPolicy=Always' +... +``` +results in: +`--set myimage.pullPolicy=Always` + +```yaml +... + custom_value_files: + - 'values-prod.yaml' +... +``` +results in: +`--values values-prod.yaml` + +If a variable already contains a `_` (underscore) in its name, replace it with `__` (double underscore). + +## Helm usage examples + +The following sections illustrate all three modes of Helm usage. + +You can also look at the [GitHub repository](https://github.com/codefresh-contrib/helm-sample-app){:target="\_blank"} of [our Helm example]({{site.baseurl}}/docs/example-catalog/cd-examples/helm/) for full pipelines: + +* Pipeline YAML for [deploying a chart](https://github.com/codefresh-contrib/helm-sample-app/blob/master/codefresh-do-not-store.yml){:target="\_blank"} +* Pipeline YAML for [both storing and deploying a chart](https://github.com/codefresh-contrib/helm-sample-app/blob/master/codefresh.yml){:target="\_blank"} + +### Helm usage example: Installing a Helm Chart + +The following example includes the minimum configuration to install a Helm chart from a repository. For more configuration options, see the [Arguments reference](https://codefresh.io/steps/step/helm){:target="\_blank"}. + +```yaml +deploy: + type: helm + arguments: + action: install + chart_name: path/to/charts + release_name: first + helm_version: 3.0.3 + kube_context: my-kubernetes-context +``` + +### Helm usage example: Pushing a Helm Chart + +The following example illustrates how to package and push a Helm chart into a repository. + +```yaml +deploy: + type: helm + arguments: + action: push + chart_name: /codefresh/volume/repo/chart + chart_repo_url: 'cm://h.cfcr.io/useraccount/default' +``` + +> **Notes**: + - Assumes that a Git repository with the Helm chart files was cloned as a part of the pipeline. + - The Git repository contains the chart files in the `chart` directory. + - `chart_repo_url` is optional. If a [Helm repository configuration](#step-4-optional-import-the-helm-configuration-in-your-pipeline-definition) is attached to the pipeline, this setting is ignored. + +### Helm usage example: Authenticating only + +The following example illustrates the Helm mode for authentication only. + +```yaml +deploy: + type: helm + arguments: + action: auth + kube_context: my-kubernetes-context + commands: + - helm list +``` + +### Helm usage example: Custom Helm commands + +The following example illustrates executing custom Helm commands. + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +my_custom_helm_command: + type: helm + arguments: + action: auth + kube_context: my-kubernetes-context + commands: + - source /opt/bin/release_chart + - helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/ + - helm repo add stable https://kubernetes-charts.storage.googleapis.com + - helm repo list + - helm repo update + - helm list +{% endraw %} +{% endhighlight %} + +> Notes: +- The directory that contains a chart MUST have the same name as the chart. Thus, a chart named `my-chart` MUST be created in a directory called `my-chart/`. This is a requirement of the [Helm Chart format](https://helm.sh/docs/chart_template_guide/). + +## Helm configuration fields + +Name|Required|Description +---|---|--- +action|Defaults to 'install'|Operation mode: `install`/`push`/`auth` +chart_name|required for install/push|Chart reference to use, adhering to Helm's lookup rules (path to chart folder, or name of packaged chart). There's no need to prefix with `/reponame` if referencing a chart in a repository, this is handled automatically. a.k.a `CHART_NAME` but `CHART_NAME` shouldn't be used anymore. +chart_repo_url|optional|Helm chart repository URL. If a [Helm repository configuration](#step-4-optional---import-the-helm-configuration-in-your-pipeline-definition) is attached to the pipeline, this setting is ignored. +chart_version|optional|Override or set the chart version. +cmd_ps|optional|When defined, Command Postscript is appended as is to the generated Helm command string. Can be used to set additional parameters supported by the command but not exposed as configuration options.| +commands|optional|Commands to execute in plugin after `auth` action. +custom_value_files|optional|Values file to provide to Helm as `--values` or `-f`.| +custom_values|optional|Values to provide to Helm as `--set` +helm_version|optional|Version of [cfstep-helm image](https://hub.docker.com/r/codefresh/cfstep-helm/tags){:target="\_blank"} +kube_context|required for install|Kubernetes context to use. The name of the cluster as [configured in Codefresh]({{site.baseurl}}/docs/deploy-to-kubernetes/add-kubernetes-cluster/). +namespace|optional|Target Kubernetes namespace to deploy to. +release_name|required for install|Helm release name. If the release exists, it is upgraded. +repos|optional|Array of custom repositories. + + +## Full Helm pipeline example + +The pipeline in this example builds a docker image, runs unit tests, stores the Helm chart in the Codefresh private Helm repository and finally deploys the Helm chart to a cluster. + +{% include image.html +lightbox="true" +file="/images/deployments/helm/full-helm-pipeline.png" +url="/images/deployments/helm/full-helm-pipeline.png" +alt="Helm pipeline" +caption="Helm pipeline" +max-width="90%" +%} + +This is the pipeline definition: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - checkout + - build + - test +steps: + clone: + title: Cloning main repository... + stage: checkout + type: git-clone + arguments: + repo: 'codefresh-contrib/python-flask-sample-app' + revision: with-helm + git: github + MyAppDockerImage: + title: Building Docker Image + stage: build + type: build + working_directory: '${{clone}}' + arguments: + image_name: kostis-codefresh/python-flask-sample-app + tag: 'master' + dockerfile: Dockerfile + MyUnitTests: + title: Running Unit tests + stage: test + type: freestyle + working_directory: '${{clone}}' + arguments: + image: ${{MyAppDockerImage}} + commands: + - python setup.py test + StoreChart: + title: Storing Helm Chart + type: helm + stage: store + working_directory: ./python-flask-sample-app + arguments: + action: push + chart_name: charts/python + kube_context: kostis-demo@FirstKubernetes + DeployMyChart: + type: helm + stage: deploy + working_directory: ./python-flask-sample-app + arguments: + action: install + chart_name: charts/python + release_name: my-python-chart + helm_version: 3.0.2 + kube_context: kostis-demo@FirstKubernetes + custom_values: + - 'buildID=${{CF_BUILD_ID}}' + - 'image_pullPolicy=Always' + - 'image_tag=master' + - 'image_pullSecret=codefresh-generated-r.cfcr.io-cfcr-default' +{% endraw %} +{% endhighlight %} + +You can see the source code in our [example section]({{site.baseurl}}/docs/example-catalog/cd-examples/helm/). + + +## Related articles +[Helm Charts and repositories]({{site.baseurl}}/docs/deployments/helm/add-helm-repository/) +[Using managed Helm repositories]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/) +[Helm Promotion boards]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion) diff --git a/_docs/deployments/kubernetes/custom-kubectl-commands.md b/_docs/deployments/kubernetes/custom-kubectl-commands.md new file mode 100644 index 000000000..77bc7411d --- /dev/null +++ b/_docs/deployments/kubernetes/custom-kubectl-commands.md @@ -0,0 +1,184 @@ +--- +title: "Custom kubectl commands" +description: "Use kubectl in your Codefresh pipelines" +group: deployments +sub_group: kubernetes +toc: true +--- + +As explained in [Kubernetes deployment options]({{site.baseurl}}/docs/deployments/kubernetes/deployment-options-to-kubernetes/), Codefresh has built-in functionality for deploying to Kubernetes clusters. + +For maximum flexibility with cluster deployments, you can run your own custom `kubectl` commands in a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). +[Kubectl](https://kubernetes.io/docs/reference/kubectl/overview/){:target="\_blank"} is the command line interface for managing kubernetes clusters. + +Codefresh automatically sets up your [config context](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/){:target="\_blank"} with your connected clusters. + +The config context is automatically placed for you at the path of the [variable]({{site.baseurl}}/docs/pipelines/variables/) `$CF_KUBECONFIG_PATH`. +In the current Codefresh implementation, this expands to `/codefresh/volume/sensitive/.kube/config`, within the [shared step volume]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/#sharing-the-workspace-between-build-steps). + +When you use custom `kubectl` commands, it is your responsibility to template your manifests using any of the available options. To employ Codefresh for templating, it is better to use the dedicated [cf-deploy-kubernetes step]({{site.baseurl}}/docs/deployments/ci-cd-guides/kubernetes-templating/), which provides simple templating capabilities. + +## Using the Codefresh kubectl image + +Codefresh already offers a public Docker image with `kubectl` at [https://hub.docker.com/r/codefresh/kubectl/tags](https://hub.docker.com/r/codefresh/kubectl/tags){:target="\_blank"}. You can choose a specific version of `kubectl` with the appropriate tag or just select `latest` for the most up-to-date version. + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + MyCustomKubectlCommands: + title: Running Kubectl + image: codefresh/kubectl:1.13.3 + commands: + - echo $CF_KUBECONFIG_PATH + - kubectl help +{% endraw %} +{% endhighlight %} + +If you run the pipeline, you can see the help options for `kubectl`. + +## Getting a config context + +The important thing to know when running custom `kubectl` commands is that Codefresh automatically sets up +your [kubeconfig files](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/){:target="\_blank"} for you with the cluster information present in [integrations]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster). + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/kube-context.png" +url="/images/deployments/kubernetes/kube-context.png" +alt="Codefresh cluster names" +caption="Codefresh cluster names" +max-width="50%" +%} + +If you run this pipeline, you will see the names of all your connected clusters: + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + MyCustomKubectlCommands: + title: Running Kubectl + image: codefresh/kubectl + commands: + - kubectl config get-contexts +{% endraw %} +{% endhighlight %} + +With two sample clusters, the output of this pipeline is the following: + +``` +Running freestyle step: Running Kubectl +Pulling image codefresh/kubectl:latest +Status: Image is up to date for codefresh/kubectl:latest +NAME CLUSTER AUTHINFO NAMESPACE +gke-kostisdemo-codefresh-kostis gke-kostisdemo-codefresh-kostis gke-kostisdemo-codefresh-kostis default +kostis-demo@FirstKubernetes kostis-demo@FirstKubernetes kostis-demo@FirstKubernetes default + +``` + +You can modify the current config context and run any `kubectl` command you want applied to that context. The next pipeline will print all the nodes of the first cluster: + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + MyCustomKubectlCommands: + title: Running Kubectl + image: codefresh/kubectl + commands: + - kubectl config get-contexts + - kubectl config use-context "gke-kostisdemo-codefresh-kostis" + - kubectl get nodes +{% endraw %} +{% endhighlight %} + +## Example of parallel deployment with kubectl + +Let's see a full example. In this pipeline, we will create two Docker images and deploy them on two separate clusters, using custom `kubectl` commands. We will also use the [parallel capability]({{site.baseurl}}/docs/pipelines/advanced-workflows/) of Codefresh pipelines. + +Here is the pipeline: + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/parallel-kubectl.png" +url="/images/deployments/kubernetes/parallel-kubectl.png" +alt="Parallel kubectl deployment" +caption="Parallel kubectl deployment" +max-width="100%" +%} + +And here is the complete `codefresh.yml`: + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' + +stages: +- build +- deploy + +steps: + BuildingApps: + type: parallel + stage: 'build' + steps: + BuildingApp1: + title: Building App 1 + type: build + stage: build + image_name: nestjs-app + working_directory: ./my-nestjs-project/ + dockerfile: Dockerfile + BuildingApp2: + title: Building App 2 + type: build + stage: build + image_name: rails + working_directory: ./my-rails-project/ + dockerfile: Dockerfile + DeployingApps: + type: parallel + stage: 'deploy' + steps: + DeployApp1: + title: Deploying App 1 + stage: deploy + image: codefresh/kubectl + working_directory: ./my-nestjs-project/ + commands: + - kubectl config get-contexts + - kubectl config use-context "gke-kostisdemo-codefresh-kostis" + - kubectl apply -f service.yml deployment.yml + DeployApp2: + title: Deploying App 2 + stage: deploy + image: codefresh/kubectl + working_directory: ./my-rails-project/ + commands: + - kubectl config get-contexts + - kubectl config use-context "kostis-demo@FirstKubernetes" + - kubectl apply -f service.yml deployment.yml configmap.yml +{% endraw %} +{% endhighlight %} + +In the example above, we select one of the clusters in each deployment step, and then apply several Kubernetes manifests that constitute an application. + +## Related articles +[Managing your cluster]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) +[Accessing a Docker registry]({{site.baseurl}}/docs/ci-cd-guides/access-docker-registry-from-kubernetes/) + + + + + + + + + + + \ No newline at end of file diff --git a/_docs/deployments/kubernetes/deployment-options-to-kubernetes.md b/_docs/deployments/kubernetes/deployment-options-to-kubernetes.md new file mode 100644 index 000000000..3442e983f --- /dev/null +++ b/_docs/deployments/kubernetes/deployment-options-to-kubernetes.md @@ -0,0 +1,141 @@ +--- +title: "Deployment options for Kubernetes" +description: "Deploy to Kubernetes with the declarative deploy step" +group: deployments +sub_group: kubernetes +redirect_from: + - /docs/deploy-to-kubernetes/ + - /docs/deployment-to-kubernetes-quick-start-guide/ + - /docs/deploy-to-kubernetes/deployment-to-kubernetes-quick-start-guide/ + - /docs/deploy-to-kubernetes/get-ready-to-deploy/ +toc: true +--- + +Codefresh offers several options when it comes to Kubernetes deployments: + +1. Codefresh UI for on demand deployments + This is the easiest deployment option for Kubernetes. See our [Kubernetes quick start guide]({{site.baseurl}}/docs/getting-started/deployment-to-kubernetes-quick-start-guide/). +1. Through a dedicated [deploy step]({{site.baseurl}}/docs/pipelines/steps/deploy/) in a pipeline + Described in this article. +1. Through the [cf-deploy-kubernetes step]({{site.baseurl}}/docs/ci-cd-guides/kubernetes-templating/) in a pipeline + Use this to also perform simple templating on Kubernetes manifests. +1. Through a [freestyle]({{site.baseurl}}/docs/pipelines/steps/freestyle/) step with [Kustomize](https://kustomize.io){:target="\_blank"}. + See [Deployment with Kustomize]({{site.baseurl}}/docs/example-catalog/cd-examples/deploy-with-kustomize). +1. Using a [freestyle]({{site.baseurl}}/docs/codefresh-yaml/steps/freestyle/) step with your own `kubectl` commands + This deployment option gives you great flexibility, but assumes that you know how to work with `kubectl`. See [Custom kubectl commands]({{site.baseurl}}/docs/deployments/kubernetes/custom-kubectl-commands/). +1. Using Helm as a package manager + See our [Helm quick start guide]({{site.baseurl}}/docs/quick-start/ci-quickstart/deploy-with-helm/) . + +## Prerequisites + +* A K8s cluster in Codefresh (see [Connecting a Kubernetes cluster]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster/) +* Familiarity with the [Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/), basic [pipeline steps ]({{site.baseurl}}/docs/pipelines/steps/), and how to describe them +* [Integrate your Docker registry]({{site.baseurl}}/docs/integrations/docker-registries/) with Codefresh + +## Build and push your image +Here is a basic Codefresh pipeline scenario to build and push your image to Dockerhub registry. + + `YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + BuildImage: + type: build + image_name: ' / ' #specify your future image reference here + dockerfile: Dockerfile + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + + PushToDockerRegistry: + type: push + candidate: '${{BuildImage}}' + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + registry: 'dockerhub' #the name of the registry you added to Codefresh +{% endraw %} +{% endhighlight %} + +Using this YAML example, we'll add an additional step to deploy the image in Dockerhub to Kubernetes. + +## Describe your deployment +The following instructions describe how to create a new service in your Kubernetes cluster in order to deploy to it. +>If you're deploying to an existing service in your Kubernetes cluster, please skip to the [next step]({{site.baseurl}}/docs/getting-started/deployment-to-kubernetes-quick-start-guide/#add-a-deployment-step). + + + 1. Go to the **`Kubernetes` → `Services page`**. + 1. Click the button **“Add Service”**. + 1. Select the **cluster**. + 1. Select the **namespace**. + 1. Type an arbitrary **service name**. + 1. Specify the **number of replicas**. + 1. Type the name of your **pushed image**. + 1. In the **“Internal Ports”** field specify the port which your application listens to. + 1. In the **“Expose port”** field specify the port to be exposed to the Internet and check the checkbox. + 1. Click the button **“Deploy”** to deploy the application. + +Wait until the deployment is completed, and you can open the deployed application in your browser by clicking on the "endpoint" link. + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/describe-k8s-deployment.png" +url="/images/deployments/kubernetes/describe-k8s-deployment.png" +alt="Describe Kubernetes deployment" +caption="Describe Kubernetes deployment" +max-width="60%" +%} + +## Add a Deployment step +So now you have deployed your image manually, which is great. But how to trigger the deployment within your pipeline? For that you will need to add a step of a “Deploy” type to the Codefresh YAML manifest file: + + `YAML` +{% highlight yaml %} +{% raw %} +RunningDeployScript: + title: Running Deploy Script + type: deploy + kind: kubernetes + cluster: ' ' #the name specified when you added the cluster + namespace: #the namespace you wish to deploy into + service: #the service you would like to update the deployment in + candidate: + image: '${{BuildImage}}' + registry: 'dockerhub' +{% endraw %} +{% endhighlight %} + +The full Codefresh YAML looks like this: + + `YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + BuildImage: + type: build + image_name: ' / ' + dockerfile: Dockerfile + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + + PushToDockerRegistry: + type: push + candidate: '${{BuildImage}}' + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + registry: 'dockerhub' #the name of the registry you added to Codefresh + + RunningDeployScript: + title: Running Deploy Script + type: deploy + kind: kubernetes + cluster: ' ' #the name specified when you added the cluster + namespace: #the namespace you wish to deploy into + service: #the service you would like to update the deployment in + candidate: + image: '${{BuildImage}}' + registry: 'dockerhub' +{% endraw %} +{% endhighlight %} + +You can now run the whole pipeline that builds your application from source to a docker image, pushes it to a docker registry and deploys it to your Kubernetes cluster. + +## Related articles +[Manage your Kubernetes cluster]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) +[Environment dashboard]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/) diff --git a/_docs/deployments/kubernetes/manage-kubernetes.md b/_docs/deployments/kubernetes/manage-kubernetes.md new file mode 100644 index 000000000..0b9081093 --- /dev/null +++ b/_docs/deployments/kubernetes/manage-kubernetes.md @@ -0,0 +1,169 @@ +--- +title: "Managing Kubernetes clusters" +description: "Use the graphical Kubernetes dashboard in Codefresh" +group: deployments +sub_group: kubernetes +redirect_from: + - /docs/deploy-to-kubernetes/codefresh-kubernetes-integration-beta/ + - /docs/codefresh-kubernetes-integration-beta/ +toc: true +--- + +Codefresh includes a built-in Kubernetes Dashboard that allows you to see the state of your clusters, and even make changes if you have the appropriate access privileges. + +## Accessing the Kubernetes Dashboard + +After [adding a cluster]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster), you will be able to manage your Kubernetes assets via the *Kubernetes tab* on the left pane. Clicking on the Kubernetes icon will take you to your services dashboard. + +{% include image.html +lightbox="true" +file="/images/integrations/kubernetes/kubernetes-dashboard.png" +url="/images/integrations/kubernetes/kubernetes-dashboard.png" +alt="Codefresh Kubernetes Dashboard" +caption="Codefresh Kubernetes Dashboard" +max-width="80%" + %} + +With the graphical dashboard it is very easy to locate problematic services or deploy new ones quickly. If there are clusters that are not accessible to your user you can hide them by enabling the *Hide inaccessible clusters* option at the top right of the window in order to simplify the view. + +## Viewing your Kubernetes services + +If you have too many clusters you can choose the *add filter* button at the top of the window to hide specific clusters or namespaces. + +You will be able to see the following parameters for each service: +* Name +* Cluster +* Namespace +* Replica count +* Docker image +* Selector +* A status check + +You can also switch to a Grid view if you prefer that over the default List view: + + +{% include image.html +lightbox="true" +file="/images/kubernetes/dashboard/grid-view.png" +url="/images/kubernetes/dashboard/grid-view.png" +alt="Kubernetes Dashboard grid view" +caption="Kubernetes Dashboard grid view" +max-width="80%" + %} + + If there are clusters that are not accessible to your user you can hide them by enabling the *Hide inaccessible clusters* option at the top right of the window in order to simplify the view. + + +## Work with your services + +In this view, you will be able to perform the following actions: + +* Add new service +* Edit/Update existing services +* Remove service + + +## Deploying a new service + +The Kubernetes dashboard provides a GUI dialog to quickly deploy new services in your cluster. + +### Choose a Docker image + +To add a service, click the "Add Service" button on the top or the "plus" button on a specific namespace. Then fill in the details for your new service. + +You can add images built in Codefresh which were pushed to Codefresh registry or provide a name for Docker image that will be pulled from an [external Docker registry]({{site.baseurl}}/docs/integrations/docker-registries/). Notice that images which are not from Dockerhub must be mentioned with their full domain name. + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/quick-ui-deploy.png" +url="/images/deployments/kubernetes/quick-ui-deploy.png" +alt="Deploying with the quick UI dialog" +caption="Deploying with the quick UI dialog" +max-width="60%" +%} + + +Use the following steps in order to add Image and pull secrets from the [connected Docker Registry]({{site.baseurl}}/docs/docker-registries/external-docker-registries/): +* Specify the image name in the format ` / / : ` +* Provide and image pull secret - this will be done for each namespace + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/deploying-private-cf-registry.png" +url="/images/deployments/kubernetes/deploying-private-cf-registry.png" +alt="Deploying from the private Codefresh registry" +caption="Deploying from the private Codefresh registry" +max-width="60%" +%} + + +From this screen you can also [create Kubernetes image secrets]({{site.baseurl}}/docs/ci-cd-guides/access-docker-registry-from-kubernetes/) without actually deploying anything. + + +### Set environment variables and resources + +You can add extra environment variables that will passed to the deployment image. + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/environment-variables-deployment.png" +url="/images/deployments/kubernetes/environment-variables-deployment.png" +alt="Environment variables for the deployment" +caption="Environment variables for the deployment" +max-width="60%" +%} + + + +You can also define resource limits for your pods. +It is a good practice to place maximum limits so that your services do not experience resource starvation. + + +### Adding a service with a manifest file + +If you are an advanced Kubernetes user, toggle the Deployment option button to the `YAML` position on the top right corner of the screen. +In this mode you can define exactly the contents for the service and deployment Kubernetes resources. + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/define-k8s-service-resource.png" +url="/images/deployments/kubernetes/define-k8s-service-resource.png" +alt="Define a Kubernetes Service Resource" +caption="Define a Kubernetes Service Resource" +max-width="60%" +%} + +You can type directly in the browser window or paste content from a text editor. + +{% include image.html +lightbox="true" +file="/images/deployments/kubernetes/define-k8s-deployment-resource.png" +url="/images/deployments/kubernetes/define-k8s-deployment-resource.png" +alt="Define a Kubernetes Deployment Resource" +caption="Define a Kubernetes Deployment Resource" +max-width="60%" +%} + + +Congratulations! Your service is now deployed to your Kubernetes cluster. + +You can update an existing service in a similar manner from your Kubernetes services window - Just hit the "edit" icon and update your service using the same steps as in "Add new service" section. + +## Automate your deployment + +After your service is deployed to your Kubernetes cluster, you can automate image deployment using Codefresh pipelines. + +Some of the possible options are: + +1. The dedicated [deploy step]({{site.baseurl}}/docs/pipelines/steps/deploy/) in a pipeline. +1. The [cf-deploy-kubernetes step]({{site.baseurl}}/docs/ci-cd-guides/kubernetes-templating/) in a pipeline. This can also perform simple templating on Kubernetes manifests. + +See more choices in the [Deployment options page]({{site.baseurl}}/docs/deployments/kubernetes/deployment-options-to-kubernetes/). + +## Related articles +[Environment dashboard]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/) +[Add Config Maps]({{site.baseurl}}/docs/ci-cd-guides/add-config-maps-to-your-namespaces/) +[Kubernetes deployment quick start]({{site.baseurl}}/docs/getting-started/deployment-to-kubernetes-quick-start-guide/) + + + diff --git a/_docs/example-catalog/cd-examples/amazon-ecs.md b/_docs/example-catalog/cd-examples/amazon-ecs.md new file mode 100644 index 000000000..89905c22e --- /dev/null +++ b/_docs/example-catalog/cd-examples/amazon-ecs.md @@ -0,0 +1,155 @@ +--- +title: "Amazon ECS/Fargate" +description: "Use Codefresh to deploy Docker containers to ECS/Fargate" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/amazon-ecs/ + - /docs/deploy-your-containers/ + - /docs/deploy-your-containers/amazon-ecs/ +toc: true +--- +Codefresh can deploy to any ECS or Fargate cluster created in Amazon. + +{% include image.html +lightbox="true" +file="/images/examples/amazon-ecs/ecs-pipeline-deployment.png" +url="/images/examples/amazon-ecs/ecs-pipeline-deployment.png" +alt="Deploying to Amazon ECS" +caption="Deploying to Amazon ECS" +max-width="100%" +%} + +## Prerequisites + + +1. Configure an ECS (or Fargate) Cluster with at least one running instance. +1. Configure an ECS Service and Task Definition with a reference to **the image that you are going to build and push.** See [the official amazon docs](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html) for more details. +1. Connect your [ECR to Codefresh]({{site.baseurl}}/docs/docker-registries/external-docker-registries/amazon-ec2-container-registry/) so that it can be used by name in Codefresh pipelines. +1. Verify you have AWS Credentials (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`), with the following privileges: + + `JSON` +{% highlight json %} +{% raw %} +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "Stmt1479146904000", + "Effect": "Allow", + "Action": [ + "ecs:DescribeServices", + "ecs:DescribeTaskDefinition", + "ecs:DescribeTasks", + "ecs:ListClusters", + "ecs:ListServices", + "ecs:ListTasks", + "ecs:RegisterTaskDefinition", + "ecs:UpdateService" + ], + "Resource": [ + "*" + ] + } + ] +} +{% endraw %} +{% endhighlight %} + + + +## Create a CI/CD pipeline for ECS/Fargate + +Here is the complete pipeline: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - "clone" + - "build" + - "deploy" +steps: + main_clone: + type: "git-clone" + description: "Cloning main repository..." + repo: "${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}" + revision: "${{CF_BRANCH}}" + stage: "clone" + git: github + BuildingDockerImage: + stage: "build" + title: Building Docker Image + type: build + image_name: ${{IMAGE}} + tag: '${{CF_SHORT_REVISION}}' + dockerfile: Dockerfile.multistage + Push: + title: "Pushing image to ECR" + stage: "deploy" + type: "push" + tag: '${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}' + registry: "ecr" + candidate: "${{BuildingDockerImage}}" + DeployToFargate: + stage: "deploy" + image: codefreshplugins/cf-deploy-ecs + commands: + - cfecs-update ${{REGION}} ${{ECS_CLUSTER_NAME}} ${{ECS_SERVICE_NAME}} --image-name ${{IMAGE_PREFIX}}/${{IMAGE}} --image-tag '${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}' + environment: + - AWS_ACCESS_KEY_ID=${{AWS_ACCESS_KEY_ID}} + - AWS_SECRET_ACCESS_KEY=${{AWS_SECRET_ACCESS_KEY}} + +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code with a [Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/) +1. Uses a [build step]({{site.baseurl}}/docs/pipelines/steps/build/) to create a Docker image +1. Uses a [push step]({{site.baseurl}}/docs/cpipelines/steps/push/) to push the docker image to ECR. The registry was previously [connected in Codefresh]({{site.baseurl}}/docs/docker-registries/external-docker-registries/) with the `ecr` identifier. +1. Runs `codefreshplugins/cf-deploy-ecs` to perform the actual deployment + + +The pipeline needs [environment variables]({{site.baseurl}}/docs/pipelines/pipelines/#pipeline-settings) that hold all the required parameters. + +{% include image.html +lightbox="true" +file="/images/examples/amazon-ecs/ecs-variables.png" +url="/images/examples/amazon-ecs/ecs-variables.png" +alt="ECS environment variables" +caption="ECS environment variables" +max-width="80%" +%} + + + + +Note that the **`--image-name`** and **`--image-tag`** pair should comprise the **full name** of the image that was pushed to the registry (including the registry name) in order to be correctly referred by the corresponding Task Definition. + + + +## Deployment Flow + +The `codefreshplugins/cf-deploy-ecs` step performs the following: + + +1. Gets the ECS service by specified `aws-region`, `ecs-cluster`, and `service-names`. +1. Creates a new revision from the current task definition of the service. If `--image-name` and `--image-tag` are provided, it replaces the image tag. +1. Runs the `update-service` command with the new task definition revision. +1. Waits for the deployment to complete. + * Deployment is successfully completed if `runningCount == desiredCount` for PRIMARY deployment - see `aws ecs describe-services` + * The `cfecs-update` command exits with a timeout error if after --timeout (default = 900s) `runningCount` does not equal `desiredCount` + * The `cfecs-update` exits with an error if --max-failed (default = 2) or more ECS tasks were stopped with error for the task definition that you are deploying. ECS continuously retries failed tasks. + +You can also find the same step in the form of a [Codefresh plugin](https://codefresh.io/steps/step/ecs-deploy). + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[External Registries]({{site.baseurl}}/docs/integration/docker-registries/) + + diff --git a/_docs/example-catalog/cd-examples/deploy-to-heroku.md b/_docs/example-catalog/cd-examples/deploy-to-heroku.md new file mode 100644 index 000000000..100a56eca --- /dev/null +++ b/_docs/example-catalog/cd-examples/deploy-to-heroku.md @@ -0,0 +1,212 @@ +--- +title: "Deploy to Heroku" +description: "Deploy your application or image to Heroku" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +Heroku is a container-based cloud PaaS (Platform as a Service) software that allows you to deploy, run, and manage your applications. Built on top of AWS, it supports Ruby, Node.js, Java, Python, Clojure, Scala, Go and PHP. + +This tutorial will cover two examples, depending on your use case. If you are not using containers, your use case is covered using the Codefresh heroku-deployer plugin ([Example #1](#pipeline-example-1-deploying-source-code-to-heroku-using-the-codefresh-heroku-plugin)). If you are using containers, you can achieve deployment by using a combination of build, push, and freestyle steps ([Example #2](#pipeline-example-2-deploy-a-docker-image-to-heroku)). + +## Example Django Application + +You can find the example project on [GitHub](https://github.com/codefresh-contrib/heroku-python-django-sample-app). + +The repository contains a Django starter project with the following commands: + +- `pip install -r requirements.txt` to install dependencies. +- `python -m unittest composeexample.utils` runs unit tests. +- `python manage.py runserver 0.0.0.0:8000` to start the application locally. + +Once launched the application presents the Django starter page at localhost:8000. + +## Pipeline Example #1: Deploying Source Code to Heroku Using the Codefresh Heroku Plugin + +### Prerequisites + +- A [free Codefresh account]({{site.baseurl}}/docs/administration/account-user-management/#create-a-codefresh-account/) +- A [free Heroku account](https://signup.heroku.com){:target="\_blank"} +- A Heroku API token (you can find this under **Account Settings** and then scrolling down, you will find the API Key) + +### Create the pipeline + +This pipeline has three stages: clone, test, and deploy. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/heroku-deployer-pipeline.png" +url="/images/examples/deployments/heroku-deployer-pipeline.png" +alt="Codefresh UI Pipeline View" +caption="Codefresh UI Pipeline View" +max-width="100%" +%} + +You should be able to copy and paste this YAML in the in-line editor of the Codefresh UI. It will automatically clone the project for you. + +Note that you need to change the environment variables in the deploy stage to your respective values. You can do this directly [in the YAML itself]({{site.baseurl}}/docs/how-to-guides/migrating-from-travis-ci/#environment-variables), or through the Codefresh UI. Navigate to the in-line editor, and to the right you will find a tab lebeled **Variables**. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/heroku-deployer-variables2.png" +url="/images/examples/deployments/heroku-deployer-variables2.png" +alt="Codefresh UI Pipeline Variables View" +caption="Codefresh UI Pipeline Variables View" +max-width="100%" +%} + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - clone + - test + - deploy +steps: + clone: + title: "Cloning main repository..." + stage: "clone" + type: "git-clone" + arguments: + repo: "codefresh-contrib/heroku-python-django-sample-app" + revision: "master" + git: "github" + run_unit_tests: + title: "Running unit tests..." + stage: "test" + type: "freestyle" + working_directory: "${{clone}}" + arguments: + image: "python:3.6-slim" + commands: + - "pip install -r requirements.txt --cache-dir=/codefresh/volume/pip-cache" + - "python -m unittest composeexample.utils" + deploy_to_heroku: + title: "Deploying to Heroku..." + stage: "deploy" + type: "heroku-deployer" + arguments: + APP_NAME: $APP_NAME + EMAIL: $EMAIL + API_TOKEN: $API_TOKEN +{% endraw %} +{% endhighlight %} + +The above pipeline has the following steps: + +1. A [git-clone]({{site.baseurl}}/docs/pipelines/steps/git-clone/) step that clones the main repository +2. A [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/) that installs dependencies and runs the unit tests +3. A freestyle step that deploys the application to Heroku using the heroku-deployer plugin from the [Step Marketplace](https://codefresh.io/steps/step/heroku-deployer) + +## Pipeline Example #2: Deploy a Docker Image to Heroku + +This example differs from the plugin usage, as it deploys a built Docker image to Heroku. + +Note that you need to change the environment variables to your respective values. You can do this directly [in the YAML itself]({{site.baseurl}}/docs/how-to-guides/migrating-from-travis-ci/#environment-variables), or through the Codefresh UI. Navigate to the in-line editor, and to the right you will find a tab lebeled **Variables**. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/heroku-deployer-variables.png" +url="/images/examples/deployments/heroku-deployer-variables.png" +alt="Codefresh UI Pipeline Variables View" +caption="Codefresh UI Pipeline Variables View" +max-width="100%" +%} + +## Prerequisites + +- A [free Codefresh account]({{site.baseurl}}/docs/administration/account-user-management/#create-a-codefresh-account/) +- A [free Heroku account](https://signup.heroku.com){:target="\_blank"} +- An empty repository already created in Heroku using the `heroku create ` command +- A Heroku registry [connected to Codefresh]({{site.baseurl}}/docs/docker-registries/external-docker-registries/other-registries/#heroku-registries) +- A Heroku API token (you can find this under **Account Settings** and then scrolling down, you will find the API Key) + +### Create the pipeline + +This pipeline has five stages: clone, build, test, push, and release. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/heroku-vanilla-push-pipeline.png" +url="/images/examples/deployments/heroku-vanilla-push-pipeline.png" +alt="Codefresh UI Pipeline View" +caption="Codefresh UI Pipeline View" +max-width="100%" +%} + +You should be able to copy and paste this YAML in the in-line editor of the Codefresh UI. It will automatically clone the project for you. + +`codefresh-heroku-push-image.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +version: '1.0' +stages: + - clone + - build + - test + - push + - release +steps: + clone: + title: "Cloning main repository..." + stage: "clone" + type: "git-clone" + arguments: + repo: "codefresh-contrib/heroku-python-django-sample-app" + revision: "master" + git: "github" + build: + title: "Building Docker Image..." + stage: "build" + type: "build" + working_directory: "./heroku-python-django-sample-app" + arguments: + image_name: "${{IMAGE_NAME}}" + tag: "master" + dockerfile: "Dockerfile" + run_unit_tests: + title: "Running unit tests..." + stage: "test" + type: "freestyle" + working_directory: "./heroku-python-django-sample-app" + arguments: + image: '${{build}}' + commands: + - "python -m unittest composeexample.utils" + push_image: + title: "Pushing image to Heroku..." + stage: "push" + type: "push" + arguments: + candidate: '${{build}}' + image_name: "${{IMAGE_NAME}}/web" + registry: "heroku" + release_image: + title: "Releasing image..." + stage: "release" + type: "freestyle" + arguments: + image: "nazarcodefresh/heroku-cli:alpine" + commands: + - >- + printf "machine api.heroku.com\n login $EMAIL\n password + $API_TOKEN\nmachine git.heroku.com\n login $EMAIL\n password + $API_TOKEN\n" > ~/.netrc + - "heroku container:release --app $IMAGE_NAME web" +{% endraw %} +{% endhighlight %} + +The pipeline does the following: +1. Clones the main repository through the [git-clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +1. Builds our Docker image through a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). +1. Runs unit tests on our Docker image through another freestyle step. +1. Pushes to the Heroku registry through a [push step]({{site.baseurl}}/docs/pipelines/steps/push/). +1. Releases the Docker image through another freestyle step. + + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) diff --git a/_docs/example-catalog/cd-examples/deploy-to-tomcat-via-scp.md b/_docs/example-catalog/cd-examples/deploy-to-tomcat-via-scp.md new file mode 100644 index 000000000..fa66c1c25 --- /dev/null +++ b/_docs/example-catalog/cd-examples/deploy-to-tomcat-via-scp.md @@ -0,0 +1,122 @@ +--- +title: "Deploy to a VM via SCP" +description: "Deploy your application to Tomcat using SCP" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +## Prerequisites + +- A [free Codefresh account]({{site.baseurl}}/docs/administration/account-user-management/create-codefresh-account) +- A distribution of [Tomcat](https://tomcat.apache.org/download-90.cgi){:target="\_blank"} setup on a remote server (running with port 8080 exposed) + +## The example Java Application + +You can find the example project on [GitHub](https://github.com/codefresh-contrib/scp-war-app){:target="\_blank"}. + +The example application is a simple Hello World Java application using the [Spark Java framework](http://sparkjava.com/){:target="\_blank"}: + +{% include image.html +lightbox="true" +file="/images/examples/deployments/scp-hello-world.png" +url="/images/examples/deployments/scp-hello-world.png" +alt="Hello World!" +caption="Hello World!" +max-width="100%" +%} + + +```java + @Override + public void init() { + get("/hello", (req, res) -> "Hello World"); + } +``` + +## Create the pipeline + +Our pipeline has three stages: clone, package, and transfer. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/scp-pipeline.png" +url="/images/examples/deployments/scp-pipeline.png" +alt="SCP pipeline" +caption="Codefresh UI Pipeline View" +max-width="100%" +%} + +You should be able to copy and paste this YAML in the in-line editor of the Codefresh UI. It will automatically clone the project for you. + +Note that you need to change the environment variables under the `transfer` step to your respective values. + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +# More examples of Codefresh YAML can be found at +# https://codefresh.io/docs/docs/example-catalog/example + +version: "1.0" +# Stages can help you organize your steps in stages +stages: + - "clone" + - "package" + - "transfer" + +steps: + clone: + title: "Cloning repository..." + type: "git-clone" + stage: "clone" + arguments: + repo: "codefresh-contrib/scp-war-app" + + package: + title: "Packaging war..." + type: "freestyle" + stage: "package" + arguments: + image: "maven:3.5.2-jdk-8-alpine" + working_directory: "${{clone}}" + commands: + - "mvn -Dmaven.repo.local=/codefresh/volume/m2_repository clean package" + + transfer: + title: "Transferring war to Tomcat..." + type: "freestyle" + stage: "transfer" + arguments: + image: "ictu/sshpass:latest" + working_directory: "${{package}}/target" + environment: + - USER= + - HOST= + - PASSWORD= + - TOMCAT_DIR= + commands: + - "echo | ssh-keygen -P '' -t rsa" + - "sshpass -p $PASSWORD ssh-copy-id -i /root/.ssh/id_rsa.pub -o StrictHostKeyChecking=no $USER@$HOST" + - "scp sparkjava-hello-world-1.0.war $USER@$HOST:$TOMCAT_DIR" +{% endraw %} +{% endhighlight %} + +The above pipeline does the following: + +1. Clones the main repository through the [git-clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +2. Installs the dependencies via Maven and packages our `war` file through a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). +3. Transfers our application via scp to a Tomcat server through another freestyle step. + +Note that you will need to change the listed environment variables accordingly, either through the YAML itself, or through your pipeline settings: + +{% include image.html +lightbox="true" +file="/images/examples/deployments/scp-variables.png" +url="/images/examples/deployments/scp-variables.png" +alt="Pipeline variables" +caption="Pipeline variables" +max-width="100%" +%} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) diff --git a/_docs/example-catalog/cd-examples/deploy-with-kustomize.md b/_docs/example-catalog/cd-examples/deploy-with-kustomize.md new file mode 100644 index 000000000..21b3089fa --- /dev/null +++ b/_docs/example-catalog/cd-examples/deploy-with-kustomize.md @@ -0,0 +1,244 @@ +--- +title: "Deploy with Kustomize" +description: "Deploy your services to Kubernetes using Kustomize" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +[Kustomize](https://kustomize.io) is a tool included with kubectl 1.14 that "lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is." + +Kustomize is more of an overlay engine, as opposed to a templating engine. You create a base configuration and overlays. Your overlays contain a *kustomization.yaml* file, and any variants/changes are applied over top of the base configuration. Kustomize does not use templates at all. + +While it is good for simple scenarios, we suggest that you use Helm for managing your Kubernetes applications. Helm is a full package manager for Kubernetes manifests that also provides templating capabilities. See [this example]({{site.baseurl}}/docs/example-catalog/cd-examples/helm/){:target="\_blank"} for more information. + +## The example application + +You can find the example project on [GitHub](https://github.com/codefresh-contrib/kustomize-sample-app){:target="\_blank"}. + +The sample application is a simple Spring Boot web app, that displays an environment variable, `MY_MYSQL_DB` on the page: + +```java +public class HelloController { + + String my_sql_db = System.getenv("MY_MYSQL_DB"); + + @RequestMapping("/") + public String index() { + return my_sql_db; + } +``` + +The project contains a [base](https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#base){:target="\_blank"} and two [overlays](https://github.com/kubernetes-sigs/kustomize/blob/master/docs/glossary.md#overlay){:target="\_blank"}, one for a staging environment and one for production. + +The base manifest holds a dummy variable for `MY_MYSQL_DB` which will be overlayed once we call the kustomize command in our pipeline. + +`base/deployment.yaml` +```yaml +... + env: + - name: MY_MYSQL_DB + valueFrom: + configMapKeyRef: + name: the-map + key: mysqlDB +``` + +We will overlay on top of the manifests a different value for `MY_MYSQL_DB` for the staging environment and production environment. + +`overlays/staging/config-map.yaml` +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: the-map +data: + mysqlDB: "staging-mysql.example.com:3306" +``` + +`overlays/production/config-map.yaml` +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: the-map +data: + mysqlDB: "prod-mysql.example.com:3306" +``` + +In addition, for the production environment, the number of replicas will be overlayed to 3 instead of 1 (as [defined in the base deployment](https://github.com/codefresh-contrib/kustomize-sample-app/blob/32e683f82940de0bf2de2da40fa6b150e2b24b23/base/deployment.yaml#L8)){:target="\_blank"}. + +`overlays/production/deployment.yaml` +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: the-deployment +spec: + replicas: 3 +``` + +## Prerequisites + +- A [free Codefresh account]({{site.baseurl}}/docs/administration/account-user-management/create-codefresh-account) + +- A Kubernetes cluster [connected to your Codefresh account](https://codefresh.io/docs/docs/integrations/kubernetes/#connect-a-kubernetes-cluster) + +## Create the staging environment pipeline + +This pipeline will have two stages: clone and deploy. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/k8s-kustomize-staging-pipeline.png" +url="/images/examples/deployments/k8s-kustomize-staging-pipeline.png" +alt="Codefresh UI Pipeline View" +caption="Codefresh UI Pipeline View" +max-width="100%" +%} + +You should be able to copy and paste this YAML in the in-line pipeline editor of the Codefresh UI. However, make sure to replace cluster context for the kubectl command under the arguments section with your own that you integrated with Codefresh. It will automatically clone the project for you and deploy. + +`staging-codefresh.yml` +{% highlight yaml %} +{% raw %} +# More examples of Codefresh YAML can be found at +# https://codefresh.io/docs/docs/example-catalog/ + +version: "1.0" +# Stages can help you organize your steps in stages + +stages: + - clone + - deploy + +steps: + clone: + title: Cloning main repository... + type: git-clone + stage: clone + arguments: + repo: https://github.com/codefresh-contrib/kustomize-sample-app.git + git: github + revision: master + + deploy: + title: Deploying to Staging using Kustomize... + type: freestyle + stage: deploy + working_directory: ${{clone}} + arguments: + image: codefresh/kubectl:1.14.9 + commands: + - kubectl config use-context anna-sandbox@codefresh-support + - kubectl apply -k overlays/staging +{% endraw %} +{% endhighlight %} + +The above pipeline does the following: +1. Clones the main repository through a [git-clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +2. Connects to our Kubernetes cluster we have integrated with Codefresh using `kubectl`, and deploys the application as a staging environment with the appropriate value for `MY_MYSQL_DB` as defined in our configMap using Kustomize (the `-k` flag), through a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). + +>If you are using `kubectl` prior to 1.14, you can use the following command to deploy with Kustomize: + `kustomize build overlays/production | kubectl apply -f` + +## Create the production environment pipeline + +Likewise, this pipeline will have two stages: clone and deploy. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/k8s-kustomize-prod-pipeline.png" +url="/images/examples/deployments/k8s-kustomize-prod-pipeline.png" +alt="Codefresh UI Pipeline View" +caption="Codefresh UI Pipeline View" +max-width="100%" +%} + +You should be able to copy and paste this YAML in the in-line editor of the Codefresh UI and remember to replace cluster context for the kubectl command again with your own. Click Save and Run and it will automatically clone the project for you. + +`prod-codefresh.yml` +{% highlight yaml %} +{% raw %} +# More examples of Codefresh YAML can be found at +# https://codefresh.io/docs/docs/example-catalog/ + +version: "1.0" +# Stages can help you organize your steps in stages + +stages: + - clone + - deploy + +steps: + clone: + title: Cloning main repository... + type: git-clone + stage: clone + arguments: + repo: https://github.com/codefresh-contrib/kustomize-sample-app.git + git: github + revision: master + + deploy: + title: Deploying to Production using Kustomize... + type: freestyle + stage: deploy + working_directory: ${{clone}} + arguments: + image: codefresh/kubectl:1.14.9 + commands: + - kubectl config use-context anna-sandbox@codefresh-support + - kubectl apply -k overlays/production +{% endraw %} +{% endhighlight %} + +The above pipeline does the following: + +1. Clones the main repository through a [git-clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +1. Connects to our Kubernetes cluster we have integrated with Codefresh using `kubectl`, and deploys the application as a staging environment with the appropriate value for `MY_MYSQL_DB` as defined in our configMap using Kustomize (the `-k` flag), through a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). + + +>Note that if you are using kubectl prior to 1.14, you can use the following command to deploy with Kustomize: +>`kustomize build overlays/production | kubectl apply -f` + +## Verification + +After you run these pipelines, your deployments are displayed in the [Kubernetes dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/#accessing-the-kubernetes-dashboard). + +{% include image.html +lightbox="true" +file="/images/examples/deployments/k8s-kustomize-dashboard.png" +url="/images/examples/deployments/k8s-kustomize-dashboard.png" +alt="Codefresh Kubernetes Deployments" +caption="Codefresh Kubernetes Deployments" +max-width="100%" +%} + +You can test that the application deployed correctly to both environments by accessing the endpoints: + +{% include image.html +lightbox="true" +file="/images/examples/deployments/k8s-kustomize-staging-endpoint.png" +url="/images/examples/deployments/k8s-kustomize-staging-endpoint.png" +alt="Staging endpoint" +caption="Staging endpoint" +max-width="100%" +%} + +{% include image.html +lightbox="true" +file="/images/examples/deployments/k8s-kustomize-prod-endpoint.png" +url="/images/examples/deployments/k8s-kustomize-prod-endpoint.png" +alt="Production endpoint" +caption="Production endpoint" +max-width="100%" +%} + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Deployment options to Kubernetes]({{site.baseurl}}/docs/deployments/kubernetes/deployment-options-to-kubernetes) +[Running custom kubectl commands]({{site.baseurl}}/docs/deployments/kubernetes/custom-kubectl-commands/) +[Deploy with Helm]({{site.baseurl}}/docs/example-catalog/cd-examples/helm/) + diff --git a/_docs/example-catalog/cd-examples/docker-swarm.md b/_docs/example-catalog/cd-examples/docker-swarm.md new file mode 100644 index 000000000..ad6dfbe1f --- /dev/null +++ b/_docs/example-catalog/cd-examples/docker-swarm.md @@ -0,0 +1,221 @@ +--- +title: "Deploy to Docker SWARM" +description: "Deploy to Docker Swarm with Codefresh" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/docker-swarm/ + - /docs/deploy-to-docker-swarm/ + - /docs/deploy-your-containers/docker-swarm/ +toc: true +--- + +Codefresh can easily deploy your application to [Docker Swarm](https://docs.docker.com/engine/swarm/){:target="\_blank"} using [Codefresh pipelines]({{site.baseurl}}/docs/pipelines/pipelines/). + +You will need to provide: + +1. The `docker-stack.yml` that contains the definition of the application +1. The host where your Docker Swarm is running +1. An SSH key that Codefresh can use to access remotely the Docker Swarm host +1. The stack name that will be used once the application is deployed + +All this information will be passed to the pipeline in the form of build parameters. + + +## Example application + +For an example Docker Swarm application, see [https://github.com/codefreshdemo/example-voting-app](https://github.com/codefreshdemo/example-voting-app){:target="\_blank"} + +To launch it locally you need to download [Docker](https://www.docker.com/products/overview){:target="\_blank"}. +If you are on Mac or Windows, [Docker Compose](https://docs.docker.com/compose){:target="\_blank"} is automatically installed. +On Linux, make sure you have the latest version of [Compose](https://docs.docker.com/compose/install/){:target="\_blank"}. + + +Run in this root directory: + +{% highlight bash %} +{% raw %} + +docker-compose up + +{% endraw %} +{% endhighlight %} + +The app runs at [http://localhost:5000](http://localhost:5000), and the results are at [http://localhost:5001](http://localhost:5001). + +Alternately, if you want to run it on a Docker Swarm, first make sure you have a Swarm. +If you don't, run: + +{% highlight bash %} +{% raw %} + +docker swarm init + +{% endraw %} +{% endhighlight %} + +Once you have your swarm, in this directory run: + +{% highlight bash %} +{% raw %} + +docker stack deploy --compose-file docker-stack.yml vote + +{% endraw %} +{% endhighlight %} + +{{site.data.callout.callout_warning}} +The swarm master must have Python installed. +{{site.data.callout.end}} + +## Deploy to Remote Swarm with Codefresh + +First you need to set up the following environment variables in your Codefresh pipeline: + +{: .table .table-bordered .table-hover} +| `RDOCKER_HOST` | remote Docker Swarm master machine, accessible over SSH (for example, ubuntu@ec2-public-ip) | +| `STACK_NAME` | is new Docker stack name (use \"vote\", for example) | +| `SSH_KEY` | private SSH key, used to access Docker Swarm master machine | +| `SPLIT_CHAR` | split character, you've used to replace `newline` in SSH key. Recommendation: use `,` (`comma` character). | + +The `SSH_KEY` variable has the contents of the [SSH key](https://www.ssh.com/ssh/public-key-authentication){:target="\_blank"} that can access the Docker Swarm host. Currently, in order to pass SSH key through Codefresh UI, you need to convert it to single line string (replacing `newline` with `comma`), like this: + +{% highlight bash %} +{% raw %} +SSH_KEY=$(cat ~/.ssh/my_ssh_key_file | tr '\n' ',') +{% endraw %} +{% endhighlight %} + +The `SPLIT_CHAR` variable should hold the replacement character that was used for the SSH key (in the example above it is the comma character) + +{% include image.html +lightbox="true" +file="/images/2f1884a-codefresh_env_vars.png" +url="/images/2f1884a-codefresh_env_vars.png" +alt="Docker Swarm build parameters" +caption="Docker Swarm build parameters" +max-width="70%" +%} + + +## Deploy to Docker Swarm with a YAML step + +Once you have defined all the variables, deploy to your cluster using the following [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). + + `codefresh.yml` +{% highlight yaml %} +{% raw %} + +deploy_to_swarm: + image: codefresh/remote-docker + working_directory: ${{main_clone}} + commands: + - rdocker ${{RDOCKER_HOST}} docker stack deploy --compose-file docker-stack.yml ${{STACK_NAME}} + environment: + - SSH_KEY=${{SSH_KEY}} + when: + branch: + only: + - master + +{% endraw %} +{% endhighlight %} + +You can also pass custom credentials like this: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} + +deploy_to_swarm: + image: codefresh/remote-docker + working_directory: ${{main_clone}} + commands: + - rdocker ${{RDOCKER_HOST}} docker login ${{MY_REGISTRY}} -u ${{MY_REGISTRY_USER}} -p ${{MY_REGISTRY_PASSWORD}} \&\& docker stack deploy --compose-file docker-compose.yml --with-registry-auth ${{STACK_NAME}} + environment: + - SSH_KEY=${{SSH_KEY}} + when: + branch: + only: + - master +{% endraw %} +{% endhighlight %} + + + +## Create a CI/CD pipeine for Docker Swarm + +Here is the complete pipeline: + +{% include +image.html +lightbox="true" +file="/images/examples/docker-swarm/docker-swarm-pipeline.png" +url="/images/examples/docker-swarm/docker-swarm-pipeline.png" +alt="Docker Swarm pipeline" +caption="Docker Swarm pipeline" +max-width="100%" +%} + +And here is the pipeline definition: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - build + - deploy +steps: + main_clone: + title: Cloning main repository... + stage: prepare + type: git-clone + repo: 'codefreshdemo/example-voting-app' + revision: master + git: github-1 + MyResultDockerImage: + title: Building Result Docker Image + stage: build + type: build + image_name: resultApp + working_directory: ./result/ + tag: master + dockerfile: Dockerfile + MyVoteDockerImage: + title: Building Vote Docker Image + stage: build + type: build + image_name: voteApp + working_directory: ./vote/ + tag: master + dockerfile: Dockerfile + MyWorkerDockerImage: + title: Building Worker Docker Image + stage: build + type: build + image_name: workedApp + working_directory: ./worker/ + tag: master + dockerfile: Dockerfile + DeployToSwarmNow: + image: codefresh/remote-docker + working_directory: ${{main_clone}} + stage: deploy + commands: + - rdocker ${{RDOCKER_HOST}} docker login ${{MY_REGISTRY}} -u ${{MY_REGISTRY_USER}} -p ${{MY_REGISTRY_PASSWORD}} \&\& docker stack deploy --compose-file docker-compose.yml --with-registry-auth ${{STACK_NAME}} + environment: + - SSH_KEY=${{SSH_KEY}} +{% endraw %} +{% endhighlight %} + +The values of `MY_REGISTRY`, `MY_REGISTRY_USER` and `MY_REGISTRY_PASSWORD` depend upon the type of [your connected registry]({{site.baseurl}}/docs/integration/docker-registries/). + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) + diff --git a/_docs/example-catalog/cd-examples/elastic-beanstalk.md b/_docs/example-catalog/cd-examples/elastic-beanstalk.md new file mode 100644 index 000000000..cd0b6949d --- /dev/null +++ b/_docs/example-catalog/cd-examples/elastic-beanstalk.md @@ -0,0 +1,136 @@ +--- +title: "Deploy to Elastic Beanstalk" +description: "" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/elastic-beanstalk/ + - /docs/deploy-your-containers/elastic-beanstalk/ +toc: true +--- + + +## Prerequisites + +- Configured Application in Elastic Beanstalk service
+ See: [http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/GettingStarted.html](http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/GettingStarted.html){:target="_blank"} + + +## Deployment with Codefresh +- Add encrypted environment variables for AWS credentials: + * `AWS_ACCESS_KEY_ID` + * `AWS_SECRET_ACCESS_KEY` + +- Provide the following environment variables: + * `AWS_REGION` + * `AWS_ENV_NAME` + * `AWS_VERSION` + * `AWS_BRANCH` + +{% include +image.html +lightbox="true" +file="/images/examples/elastic-beanstalk/codefresh_eb_env_vars.png" +url="/images/examples/elastic-beanstalk/codefresh_eb_env_vars.png" +alt="codefresh_eb_env_vars.png" +max-width="40%" +%} + +{{site.data.callout.callout_info}} +{% raw %} +The ``${{AWS_VERSION}}`` of application you can find in the Elastic Beanstalk service. +{% endraw %} +{{site.data.callout.end}} + +{% include +image.html +lightbox="true" +file="/images/examples/elastic-beanstalk/codefresh_eb_version_label.png" +url="/images/examples/elastic-beanstalk/codefresh_eb_version_label.png" +alt="codefresh_eb_version_label.png" +max-width="40%" +%} + +{{site.data.callout.callout_info}} +{% raw %} +The ``${{AWS_ENV_NAME}}`` of application you can find in the Elastic Beanstalk service. +{% endraw %} +{{site.data.callout.end}} + +{% include +image.html +lightbox="true" +file="/images/examples/elastic-beanstalk/codefresh_eb_environment.png" +url="/images/examples/elastic-beanstalk/codefresh_eb_environment.png" +alt="codefresh_eb_environment.png" +max-width="40%" +%} + +Add the following step to codefresh.yml: + + `deploy_step` +{% highlight yaml %} +{% raw %} +deploy-elastic-beanstalk: + fail-fast: false + image: garland/aws-cli-docker:latest + commands: + - sh -c "aws configure set region '${{AWS_REGION}}' && aws elasticbeanstalk update-environment --environment-name '${{AWS_ENV_NAME}}' --version-label '${{AWS_VERSION}}' " + when: + condition: + all: + masterBranch: "'${{CF_BRANCH}}' == '${{AWS_BRANCH}}'" +{% endraw %} +{% endhighlight %} + +{:.text-secondary} +## Deployment Flow +- Go to the Elastic Beanstalk service and create an application and environment. + + +{% include +image.html +lightbox="true" +file="/images/examples/elastic-beanstalk/codefresh_eb_environment-deploy.png" +url="/images/examples/elastic-beanstalk/codefresh_eb_environment-deploy.png" +alt="codefresh_eb_environment.png" +max-width="40%" +%} + +- Perform the following commands from root of your project: + * eb init + * eb create {% raw %}`${{AWS_ENV_NAME}}`{% endraw %} + + + +>Note: + If you don't have awsebcli - install EB CLI [http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-install.html](http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-install.html){:target="_blank"}. + + +{% include +image.html +lightbox="true" +file="/images/examples/elastic-beanstalk/codefresh_eb_health.png" +url="/images/examples/elastic-beanstalk/codefresh_eb_health.png" +alt="codefresh_eb_health.png" +max-width="40%" +%} + +- Add this repository to Codefresh, provide the necessary environments variables and build this service + +{% include +image.html +lightbox="true" +file="/images/examples/elastic-beanstalk/codefresh_eb_cf_step_deploy.png" +url="/images/examples/elastic-beanstalk/codefresh_eb_cf_step_deploy.png" +alt="codefresh_eb_cf_step_deploy.png" +max-width="40%" +%} + +## Example + +* [cf-example-deploy-elasticbeanstalk](https://github.com/codefreshdemo/cf-example-deploy-elasticbeanstalk){:target="_blank"} + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) diff --git a/_docs/example-catalog/cd-examples/helm.md b/_docs/example-catalog/cd-examples/helm.md new file mode 100644 index 000000000..1b104663f --- /dev/null +++ b/_docs/example-catalog/cd-examples/helm.md @@ -0,0 +1,225 @@ +--- +title: "Deploy with Helm" +description: "Use Helm in a Codefresh pipeline" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +[Helm](https://helm.sh/){:target=\_blank"} is the package manager for Kubernetes. +Codefresh has comprehensive support for Helm: + +* Free [built-in Helm repository]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/) with each Codefresh account +* [Helm chart dashboard]({{site.baseurl}}/docs/docs/deployments/add-helm-repository/) to track your charts +* [Helm Release dashboard]({{site.baseurl}}/docs/docs/deployments/helm-releases-management/) to view your deployments +* [Environment dashsboard]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/) to view Helm releases +* [Helm promotion dashboard]({{site.baseurl}}/docs/deployments/helm-environment-promotion/) to promote Helm releases +* Add any external Helm repository on any other cloud provider + +Codefresh also provides a [pipeline step]({{site.baseurl}}/docs/new-helm/using-helm-in-codefresh-pipeline/) for deploying with Helm. + +For more insights on Helm charts see also our [Helm best practices]({{site.baseurl}}/docs/new-helm/helm-best-practices/) guide. + + +## The example Helm project + +You can see the example project at [https://github.com/codefresh-contrib/helm-sample-app](https://github.com/codefresh-contrib/helm-sample-app){:target=\_blank"}. The repository contains a simple Go application, a Dockerfile and an example chart. + + +## Prerequisites + +[At least one Kubernetes cluster]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster/) in your Codefresh account. + +>Notice that if you still use Helm 2 you should also have installed the server side of Helm 2 (Tiller) using `helm init`. This command is best run from the cloud console of your cluster. The respective pipelines of this guide are in the [helm-2 branch](https://github.com/codefresh-contrib/helm-sample-app/tree/helm-2){:target=\_blank"}. + + + +## CI/CD pipeline with Helm deployment + +It is possible to deploy directly a Helm chart as it exists on the filesystem. This is not the recommended way to use Helm, because you are bypassing the Helm chart repository, but it is certainly the simplest Helm pipeline possible. + +{% include image.html +lightbox="true" +file="/images/examples/helm/helm-deploy-pipeline.png" +url="/images/examples/helm/helm-deploy-pipeline.png" +alt="Pipeline for Helm deployment" +caption="Pipeline for Helm deployment" +max-width="100%" +%} + +Here is the whole pipeline: + + `codefresh-do-not-store.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - build + - deploy +steps: + clone: + title: Cloning main repository... + stage: prepare + type: git-clone + arguments: + repo: codefresh-contrib/helm-sample-app + revision: master + git: github + build: + title: Building Docker Image + stage: build + type: build + working_directory: ./helm-sample-app + arguments: + image_name: helm-sample-app-go + tag: multi-stage + dockerfile: Dockerfile + deploy: + title: Deploying Helm Chart + type: helm + stage: deploy + working_directory: ./helm-sample-app + arguments: + action: install + chart_name: charts/helm-example + release_name: my-go-chart-prod + helm_version: 3.0.2 + kube_context: my-demo-k8s-cluster + custom_values: + - 'buildID=${{CF_BUILD_ID}}' + - 'image_pullPolicy=Always' + - 'image_tag=multi-stage' + - 'replicaCount=3' + - 'image_pullSecret=codefresh-generated-r.cfcr.io-cfcr-default' +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code through a [Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/) +1. Builds a docker image through a [build step]({{site.baseurl}}/docs/pipelines/steps/build/) +1. Deploys the Helm chart to a cluster named `my-demo-k8s-cluster` using the Helm step [from the Step Marketplace](https://codefresh.io/steps/step/helm){:target=\_blank"}. + +In this example, `charts/helm-example` refers to the [filesystem location in the code](https://github.com/codefresh-contrib/helm-sample-app/tree/master/charts/helm-example){:target=\_blank"} that was just checked out. + +The deployment will be visible in the [Helm releases dashboard]({{site.baseurl}}/docs/new-helm/helm-releases-management/). + +{% include image.html +lightbox="true" +file="/images/examples/helm/helm-release.png" +url="/images/examples/helm/helm-release.png" +alt="Helm release view" +caption="Helm release view" +max-width="100%" +%} + +If you want to run this example yourself, make sure to edit the chart and put your own values there for the Docker image. + +## CI/CD pipeline with Helm deployment that also stores the chart + +It is recommended to use a Helm repository to store your chart before deploying it. This way you know what is deployed in your clusters +and you can also reuse charts in other installations. + +First of all you need to import in your pipeline from the [shared configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/) the settings for the internal Helm repository (or any other external repository that you have setup in Codefresh). + This will make available the internal Helm repository to your pipeline so that it can push/pull Helm charts from it. + + {% include image.html + lightbox="true" + file="/images/examples/helm/import-helm-configuration.png" + url="/images/examples/helm/import-helm-configuration.png" + alt="Using the default Helm repository in a Pipeline" + caption="Using the default Helm repository in a Pipeline" + max-width="40%" + %} + +Once that is done you can change your pipeline to also store the chart first and *then* deploy it. + + +{% include image.html +lightbox="true" +file="/images/examples/helm/helm-push-and-deploy-pipeline.png" +url="/images/examples/helm/helm-push-and-deploy-pipeline.png" +alt="Pipeline for Helm deployment that stores chart" +caption="Pipeline for Helm deployment that stores chart" +max-width="100%" +%} + +Here is the whole pipeline: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - build + - store + - deploy +steps: + clone: + title: Cloning main repository... + stage: prepare + type: git-clone + arguments: + repo: codefresh-contrib/helm-sample-app + revision: master + git: github + build: + title: Building Docker Image + stage: build + type: build + working_directory: ./helm-sample-app + arguments: + image_name: helm-sample-app-go + tag: multi-stage + dockerfile: Dockerfile + store: + title: Storing Helm Chart + type: helm + stage: store + working_directory: ./helm-sample-app + arguments: + action: push + chart_name: charts/helm-example + kube_context: my-demo-k8s-cluster + deploy: + type: helm + stage: deploy + working_directory: ./helm-sample-app + arguments: + action: install + chart_name: charts/helm-example + release_name: my-go-chart-prod + helm_version: 3.0.2 + kube_context: my-demo-k8s-cluster + custom_values: + - 'buildID=${{CF_BUILD_ID}}' + - 'image_pullPolicy=Always' + - 'image_tag=multi-stage' + - 'replicaCount=3' + - 'image_pullSecret=codefresh-generated-r.cfcr.io-cfcr-default' +{% endraw %} +{% endhighlight %} + + +After you finish running your pipeline, not only the deployment will take place, but you will also see your chart in your [Helm Chart dashboard]({{site.baseurl}}/docs/new-helm/add-helm-repository/): + +{% include image.html +lightbox="true" +file="/images/examples/helm/helm-chart.png" +url="/images/examples/helm/helm-chart.png" +alt="Stored Helm chart" +caption="Stored Helm chart" +max-width="80%" +%} + +It is also possible to [run your own Helm commands]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/#example-custom-helm-commands) in a Codefresh pipeline. + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) diff --git a/_docs/example-catalog/cd-examples/import-data-to-mongodb.md b/_docs/example-catalog/cd-examples/import-data-to-mongodb.md new file mode 100644 index 000000000..68a6c79a3 --- /dev/null +++ b/_docs/example-catalog/cd-examples/import-data-to-mongodb.md @@ -0,0 +1,60 @@ +--- + +title: "Import data to MongoDB" +description: "" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/import-data-to-mongodb-in-composition/ + - /docs/on-demand-test-environment/example-compositions/import-data-to-mongodb/ +toc: true +--- + +If you want to import/restore or to do something else before using MongoDB in your application, you can look at the following example. + +You just need to create Dockerfile for mongo seed service and provide the command to prepare MongoDB. In this case it's command `mongoimport` + + `Dockerfile mongo_seed` +{% highlight docker %} +FROM mongo +COPY init.json /init.json +CMD mongoimport --host mongodb --db exampleDb --collection contacts --type json --file /init.json --jsonArray +{% endhighlight %} + +## Looking around +In the root of this repository you'll find a file named `docker-compose.yml`. +Let's quickly review the contents of this file: + + `docker-compose.yml` +{% highlight yaml %} +{% raw %} +version: '3' +services: + mongodb: + image: mongo + command: mongod --smallfiles + ports: + - 27017 + + mongo_seed: + image: ${{mongo_seed}} + links: + - mongodb + + client: + image: ${{build_prj}} + links: + - mongodb + ports: + - 9000 + environment: + - MONGO_URI=mongodb:27017/exampleDb +{% endraw %} +{% endhighlight %} + +{{site.data.callout.callout_info}} +You can add the following example to your GitHub or Bitbucket account, and build the [example](https://github.com/codefreshdemo/cf-example-manage-mongodb){:target="_blank"}. +{{site.data.callout.end}} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) diff --git a/_docs/example-catalog/cd-examples/nodejs-angular2-mongodb.md b/_docs/example-catalog/cd-examples/nodejs-angular2-mongodb.md new file mode 100644 index 000000000..f4e698393 --- /dev/null +++ b/_docs/example-catalog/cd-examples/nodejs-angular2-mongodb.md @@ -0,0 +1,52 @@ +--- +title: "NodeJS + Angular2 + MongoDB" +description: "" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/nodejs-angular2-mongodb/ + - /docs/on-demand-test-environment/example-compositions/nodejs-angular2-mongodb/ +toc: true +--- +This tutorial will walk you through the process of adding the following: + +- Build client +- Build server +- Launch composition + +## Looking around +In the root of this repository you'll find a file named `docker-compose.yml`. +Let's quickly review the contents of this file: + + `docker-compose.yml` +{% highlight yaml %} +{% raw %} +version: '3' +services: + mongodb: + image: mongo + ports: + - 28017 + server: + image: ${{build_server}} + environment: + - MONGO_URI=mongodb://mongodb/exampleDb + links: + - mongodb + ports: + - 9000 + client: + image: ${{build_client}} + ports: + - 3000 +{% endraw %} +{% endhighlight %} + +{{site.data.callout.callout_info}} +##### Example + +Just head over to the example [__repository__](https://github.com/codefreshdemo/nodejs-angular2-mongo){:target="_blank"} in GitHub and follow the instructions there. +{{site.data.callout.end}} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) diff --git a/_docs/example-catalog/cd-examples/nomad.md b/_docs/example-catalog/cd-examples/nomad.md new file mode 100644 index 000000000..a7e78d797 --- /dev/null +++ b/_docs/example-catalog/cd-examples/nomad.md @@ -0,0 +1,225 @@ +--- +title: "Deploy to Nomad" +description: "Deploy Docker images to a Nomad cluster with Codefresh" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +Even though Codefresh has great support for Kubernetes and Helm deployments, there is no lock-in on using just Kubernetes. Codefresh can deploy on any infrastructure. + + +[Nomad](https://www.nomadproject.io/){:target=\_blank"} is an alternative scheduling platform from Hashicorp. It supports docker containers (like Kubernetes), but you can also use Nomad to schedule VMs, Java apps, Go apps or any other standalone executable. + +There are several public Docker Images with Nomad, so it is very easy to use Codefresh pipelines to deploy to a Nomad cluster. + + +{% include image.html +lightbox="true" +file="/images/examples/nomad/nomad-ci-pipeline.png" +url="/images/examples/nomad/nomad-ci-pipeline.png" +alt="Deploying to Nomad with Codefresh" +caption="Deploying to Nomad with Codefresh" +max-width="80%" +%} + +In this example, we will use the image at [https://hub.docker.com/r/djenriquez/nomad](https://hub.docker.com/r/djenriquez/nomad){:target=\_blank"}. + +## The example Nomad project + +You can see the example project at [https://github.com/codefresh-contrib/nomad-sample-app](https://github.com/codefresh-contrib/nomad-sample-app){:target=\_blank"}. The repository contains a simple job specification that deploys a docker container on nomad cluster. + + +Here is the whole job file: + + `docker-job.hcl` +{% highlight hcl %} +{% raw %} +job "example-job" { + # Specify this job should run in the region named "us". Regions + # are defined by the Nomad servers' configuration. + #region = "us" + + # Spread the tasks in this job between us-west-1 and us-east-1. + datacenters = ["dc1"] + + # Run this job as a "service" type. Each job type has different + # properties. See the documentation below for more examples. + type = "service" + + # Specify this job to have rolling updates, two-at-a-time, with + # 30 second intervals. + update { + stagger = "30s" + max_parallel = 1 + } + + # A group defines a series of tasks that should be co-located + # on the same client (host). All tasks within a group will be + # placed on the same host. + group "example-group" { + # Specify the number of these tasks we want. + count = 3 + + # Create an individual task (unit of work). This particular + # task utilizes a Docker container to front a web application. + task "example-task" { + # Specify the driver to be "docker". Nomad supports + # multiple drivers. + driver = "docker" + + # Configuration is specific to each driver. + config { + image = "r.cfcr.io/$CF_ACCOUNT/$CF_REPO_NAME:$CF_BRANCH_TAG_NORMALIZED" + + auth { + username = "$CF_ACCOUNT" + password = "$CFCR_LOGIN_TOKEN" + server_address = "r.cfcr.io" + } + + port_map { + http = 8080 + } + } + + # The service block tells Nomad how to register this service + # with Consul for service discovery and monitoring. + service { + # This tells Consul to monitor the service on the port + # labelled "http". Since Nomad allocates high dynamic port + # numbers, we use labels to refer to them. + port = "http" + + check { + type = "http" + path = "/" + interval = "10s" + timeout = "2s" + } + } + + # Specify the maximum resources required to run the task, + # include CPU, memory, and bandwidth. + resources { + cpu = 500 # MHz + memory = 128 # MB + + network { + mbits = 100 + + + port "http" {} + + + + } + } + } + } +} + +{% endraw %} +{% endhighlight %} + +Notice that the job specification has several [Codefresh variables]({{site.baseurl}}/docs/pipelines/variables/) embedded. We will use [envsubst](https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html){:target=\_blank"} in our pipeline to replace +them with the correct values. + +## Prerequisites + +You need to create a Codefresh account and have a Nomad cluster running. You need to decide on how Codefresh will communicate +with the nomad cluster. In this simple example we just use the `NOMAD_ADDR` variable to point the nomad client to our cluster. In a production environment you should use proper [ACL](https://www.nomadproject.io/guides/security/acl.html){:target=\_blank"} and [certificate](https://www.nomadproject.io/guides/security/securing-nomad.html){:target=\_blank"} variables as well. + + +In this example the Nomad cluster is already setup on a VM at Google cloud. + +You also need to create a [token for the Docker registry]({{site.baseurl}}/docs/integrations/docker-registries/) so that Nomad can pull your private images on the cluster. + +## Create a CI/CD pipeline for Nomad deployments + +Here is the whole pipeline: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +stages: + - "clone" + - "build" + - "deploy" +steps: + main_clone: + type: "git-clone" + title: "Clone main repository..." + repo: "codefresh-contrib/nomad-sample-app" + revision: "${{CF_BRANCH}}" + stage: "clone" + build: + title: "Building Docker Image" + type: "build" + image_name: "nomad-sample-app" + tag: "${{CF_BRANCH_TAG_NORMALIZED}}" + dockerfile: "Dockerfile" + stage: "build" + prepareJob: + title: "Preparing Nomad job" + image: bhgedigital/envsubst + stage: deploy + commands: + - envsubst < docker-job.hcl > docker-job-export.hcl + - cat docker-job-export.hcl + runJob: + title: "Deploying Nomad job" + image: djenriquez/nomad + stage: deploy + commands: + - nomad run docker-job-export.hcl +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code through a [Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +1. Creates a Docker image for a simple Go application through a [build step]({{site.baseurl}}/docs/pipelines/steps/build/). The image is automatically pushed to the default Docker registry. +1. Replaces all variables in the job spec by running `envsubst`. These include: + * The Registry token so that Nomad can access the default Docker registry + * The docker image name and tag to be deployed +1. Runs the job to deploy the image to Nomad through a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). + + +Run the pipeline and see your deployment succeed. + +Here are the environment variables defined for this pipeline. + +{% include image.html +lightbox="true" +file="/images/examples/nomad/nomad-variables.png" +url="/images/examples/nomad/nomad-variables.png" +alt="Pipeline variables for Nomad deployments" +caption="Pipeline variables for Nomad deployments" +max-width="50%" +%} + + +The `NOMAD_ADDR` variable is holding the URL of the cluster. The `CFCR_LOGIN_TOKEN` variable holds authentication for the Codefresh Docker registry. + +## Verify the deployment + +Nomad also comes with its own UI that can show you the result of a deployment. + +{% include image.html +lightbox="true" +file="/images/examples/nomad/nomad-ui-deployment.png" +url="/images/examples/nomad/nomad-ui-deployment.png" +alt="Nomad UI deployment" +caption="Nomad UI deployment" +max-width="80%" +%} + +You can also use [Terraform]({{site.baseurl}}/docs/example-catalog/cd-examples/terraform/) in Codefresh pipelines. + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/cd-examples/packer-gcloud.md b/_docs/example-catalog/cd-examples/packer-gcloud.md new file mode 100644 index 000000000..58d12aded --- /dev/null +++ b/_docs/example-catalog/cd-examples/packer-gcloud.md @@ -0,0 +1,132 @@ +--- +title: "Deploy to a Virtual Machine" +description: "Deploy to Google Cloud in a Codefresh pipeline with Packer" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +Even though Codefresh is Kubernetes-native and designed for containers, it can still deploy traditional applications in the form of Virtual Machines to any Cloud provider. + +In this example, we will use [Packer](http://www.packer.io/){:target="\_blank"} to package an application into a VM disk image that will then be launched in Google Cloud. +Because Packer itself is already offered [in a Docker container](https://hub.docker.com/r/hashicorp/packer/){:target="\_blank"}, it is very easy to run Packer in a Codefresh pipeline. + +Google also offers a [Docker image for GCloud](https://hub.docker.com/r/google/cloud-sdk/){:target="\_blank"} making the launching of the VM straightforward in a Codefresh pipeline. + + +{% include image.html +lightbox="true" +file="/images/examples/packer-gcloud/packer-codefresh-pipeline.png" +url="/images/examples/packer-gcloud/packer-codefresh-pipeline.png" +alt="Running Packer inside Codefresh" +caption="Running Packer inside Codefresh" +max-width="80%" +%} + +This Codefresh pipeline creates a VM image and then uses it to launch a Google Compute instance. + + +## The example Packer/Gcloud project + +You can see the example project at [https://github.com/codefresh-contrib/vm-packer-sample-app](https://github.com/codefresh-contrib/vm-packer-sample-app){:target="\_blank"}. The repository contains a simple Go application as well as a packer template. + +You can play with it locally after installing the `packer` and `gcloud` executables. + +## Prerequisites + +You need to create a Codefresh account and a Google account first. Then you need to create a [Service account Key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys){:target="\_blank"} which will allow `packer` and `gcloud` to communicate with Google cloud. + + +Add your service account json as a pipeline variable called `SERVICE_ACCOUNT`. The content of this variable will be used +in order to authenticate to Google cloud. + +{% include image.html +lightbox="true" +file="/images/examples/packer-gcloud/service-account-variable.png" +url="/images/examples/packer-gcloud/service-account-variable.png" +alt="Using a Service Account JSON in Codefresh" +caption="Using a Service Account JSON in Codefresh" +max-width="50%" +%} + +## Create a CI/CD pipeline for Packer/GCloud + +Here is the whole pipeline: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - build + - deploy +steps: + main_clone: + title: 'Cloning main repository...' + type: git-clone + repo: 'codefresh-contrib/vm-packer-sample-app' + git: github + revision: 'master' + stage: prepare + SetupAuth: + title: 'Setup GCloud Auth' + image: 'alpine' + stage: prepare + commands: + - echo $SERVICE_ACCOUNT > account.json + BuildMyApp: + title: Compiling App code + stage: build + image: 'golang:1.12' + commands: + - go build -o sample src/sample/trivial-web-server.go + CreatePackerImage: + title: Baking VM image + stage: build + image: 'hashicorp/packer' + commands: + - packer validate my-google-cloud-example.json + - packer build -force my-google-cloud-example.json + DeployToVM: + title: Deploying to VM + stage: deploy + image: 'google/cloud-sdk' + commands: + - gcloud auth activate-service-account --key-file=account.json + - gcloud config set project firstkubernetes-176201 + - gcloud compute instances create packer-demo-codefresh --image codefresh-simple-ubuntu-vm --zone europe-west1-b --metadata-from-file startup-script=startup.sh --tags http-server --preemptible --quiet + +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code through a [Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +1. Saves the content of the variable that holds the Google account as a file called `account.json`. +1. Compiles the Go application through a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). +1. Runs `packer` to create a VM image based on Ubuntu that also contains the simple Go application. +1. Runs `gcloud` to launch a VM with the image that was just created. + + +Run the pipeline and see your deployment succeed. You can customize the image by editing the [Packer template](https://github.com/codefresh-contrib/vm-packer-sample-app/blob/master/my-google-cloud-example.json){:target="\_blank"}. + +Once the VM has finished launching you can access it with your web browser. + +{% include image.html +lightbox="true" +file="/images/examples/packer-gcloud/web-app-url.png" +url="/images/examples/packer-gcloud/web-app-url.png" +alt="Accessing the VM application" +caption="Accessing the VM application" +max-width="70%" +%} + + +You can follow the same procedure for any other cloud that has an API/CLI (such as AWS, Azure, Digital Ocean etc). + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/cd-examples/pulumi.md b/_docs/example-catalog/cd-examples/pulumi.md new file mode 100644 index 000000000..74c1f3f74 --- /dev/null +++ b/_docs/example-catalog/cd-examples/pulumi.md @@ -0,0 +1,116 @@ +--- +title: "Deploy with Pulumi" +description: "Use Pulumi in a Codefresh pipeline with Docker" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +[Pulumi](https://pulumi.io/){:target="\_blank"} is a platform for *Infrastructure as Code*. It works like Terraform but allows you to use a proper programming language (TypeScript, Python, Go) to describe your infrastructure (instead of a configuration language). + +You can use Pulumi to deploy to Kubernetes or any other supported cloud platform. Because Pulumi itself is already offered [in a Docker container](https://hub.docker.com/r/pulumi/pulumi), it is very easy to run Pulumi in a Codefresh pipeline. + + +{% include image.html +lightbox="true" +file="/images/examples/pulumi/pulumi-pipeline.png" +url="/images/examples/pulumi/pulumi-pipeline.png" +alt="Running Pulumi inside Codefresh" +caption="Running Pulumi inside Codefresh" +max-width="80%" +%} + +## The example Pulumi project + +You can see the example project at [https://github.com/codefresh-contrib/pulumi-sample-app](https://github.com/codefresh-contrib/pulumi-sample-app){:target="\_blank"}. The repository contains a simple Pulumi stack based on Kubernetes and TypeScript. + +You can play with it locally after installing the `pulumi` executable. + +## Prerequisites + +You need to create a Codefresh account and a Pulumi account first. Then you need to create a [Pulumi token](https://app.pulumi.com/account/tokens){:target="\_blank"} which will allows Codefresh to communicate with Pulumi. + +[Add a Kubernetes cluster]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster/) in your Codefresh account from any cloud provider. + +Codefresh automatically creates a kubeconfig in any [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/) with all your clusters. This is the same way that Pulumi communicated with Kubernetes, so the integration between Codefresh and Pulumi is ready out of the box. + +Create a [stack](https://pulumi.io/reference/stack.html){:target="\_blank"} in Pulumi or use the one provided in the example. + +Finally add you Pulumi token as a pipeline variable called `PULUMI_ACCESS_TOKEN`. All freestyle steps have automatic access to all pipeline variables, and Pulumi will search for a token by default with this name when logging in. + + +## Create a CI/CD pipeline for Pulumi + +Here is the whole pipeline: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - build + - deploy +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}' + revision: '${{CF_REVISION}}' + stage: prepare + git: github-1 + BuildProject: + title: Build project + stage: build + image: pulumi/pulumi + commands: + - yarn install + SelectMyCluster: + title: Select K8s cluster + stage: deploy + image: codefresh/kubectl:1.13.3 + commands: + - kubectl config get-contexts + - kubectl config use-context "kostis-demo@FirstKubernetes" + RunPulumi: + title: Deploying + stage: deploy + image: pulumi/pulumi + commands: + - pulumi stack select dev --non-interactive + - pulumi stack --non-interactive + - pulumi up --non-interactive +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code through a [Git clone step]({{site.baseurl}}/docs/pipelines/pipelines/git-clone/). +1. Runs `yarn install` to download dependencies. In this example we use TypeScript, but Go and Python would work as well (or any other language supported by Pulumi). +1. Chooses the cluster that will be used for deployments, if you have more than one. Use your own cluster name as seen in the [Kubernetes dashboard]({{site.baseurl}}/docs/deploy-to-kubernetes/manage-kubernetes/) of Codefresh. +1. Runs `pulumi up` with the same target cluster. + +The pipeline needs a [single environment variable]({{site.baseurl}}/docs/pipelines/pipelines/#pipeline-settings) that holds the content of your Pulumi Token. + + +{% include image.html +lightbox="true" +file="/images/examples/pulumi/pulumi-access-token.png" +url="/images/examples/pulumi/pulumi-access-token.png" +alt="Passing the Pulumi Token in the pipeline parameters" +caption="Passing the Pulumi Token in the pipeline parameters" +max-width="60%" +%} + +Run the pipeline and see your deployment succeed. + +## Handling Pull requests + +You can easily use the same pipeline or a different one for pull requests. In this case replace the `pulumi up` command with `pulumi preview`. Even better you can add an [approval step]({{site.baseurl}}/docs/pipelines/steps/approval/) to allows humans to inspect the pipeline first. + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) diff --git a/_docs/example-catalog/cd-examples/secure-a-docker-container-using-http-basic-auth.md b/_docs/example-catalog/cd-examples/secure-a-docker-container-using-http-basic-auth.md new file mode 100644 index 000000000..b7e2884cd --- /dev/null +++ b/_docs/example-catalog/cd-examples/secure-a-docker-container-using-http-basic-auth.md @@ -0,0 +1,92 @@ +--- +title: "Secure a Docker Container using HTTP Basic Auth" +description: "" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/securing-docker-container-with-http-basic-auth/ + - /docs/on-demand-test-environment/examples-compositions/securing-docker-container-with-http-basic-auth/ + - /docs/on-demand-test-environment/example-compositions/secure-a-docker-container-using-http-basic-auth/ +toc: true +--- +Before making a product publicly available, you might want to restrict access to certain users. These are some options to accomplish this goal: + + - Implement custom authentication within the system + - Configure the server to act as a proxy between the user and the application + - Limit access to specific IP addresses + +This article explains how to secure a container by exposing public ports, using an extra NGINX container to act as a proxy. + +## Expose Web App Public Port + + `webapp` +{% highlight yaml %} +{% raw %} +version: '3' +services: + web: + image: codefreshio/webapp + ports: + - "3000" +{% endraw %} +{% endhighlight %} + +The architecture for this step is displayed in the diagram below. In this step example, Docker is forwarding an internal 3000 port to the host 80 port. + +{% include +image.html +lightbox="true" +file="/images/examples/docker-https/codefresh_webapp_container.png" +url="/images/examples/docker-https/codefresh_webapp_container.png" +alt="codefresh_webapp_container.png" +max-width="40%" +%} + +## Add NGINX Proxy +To secure the web-app we are going to specify these commands in the ```docker-compose.yml``` file. + +1. Remove the port that maps from the web-app (it won't be directly accessible) +2. Add an extra NGINX container with custom configuration (proxy all traffic) +3. Configure NGINX to communicate with the web-app + + `docker-compose.yml` +{% highlight yaml %} +{% raw %} +version: '3' +services: + web: + image: ${{build-prj}} + auth: + image: ${{build-nginx}} + ports: + - 80 + links: + - web + environment: + USER: ${{USERNAME}} + PASS: ${{PASSWORD}} +{% endraw %} +{% endhighlight %} + +The architecture for the ```docker-compose.yml``` file is displayed in the diagram below. + +{% include +image.html +lightbox="true" +file="/images/examples/docker-https/codefresh_nginx_container.png" +url="/images/examples/docker-https/codefresh_nginx_container.png" +alt="codefresh_nginx_container.png" +max-width="40%" +%} + +{{site.data.callout.callout_info}} +##### Example + +Just head over to the example [__repository__](https://github.com/codefreshdemo/cf-example-basic-auth-container){:target="_blank"} in GitHub and follow the instructions there. +{{site.data.callout.end}} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/cd-examples/spring-boot-kafka-zookeeper.md b/_docs/example-catalog/cd-examples/spring-boot-kafka-zookeeper.md new file mode 100644 index 000000000..2134ff171 --- /dev/null +++ b/_docs/example-catalog/cd-examples/spring-boot-kafka-zookeeper.md @@ -0,0 +1,203 @@ +--- +title: "Spring Boot + Kafka + Zookeeper" +description: "" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/spring-boot-kafka-zookeeper/ + - /docs/on-demand-test-environment/example-compositions/spring-boot-kafka-zookeeper/ +toc: true +--- +This project uses `Java, Spring Boot, Kafka, Zookeeper` to show you how to integrate these services in the composition. + +{{site.data.callout.callout_info}} +##### Example + +Just head over to the example [__repository__](https://github.com/codefreshdemo/example-springboot-kafka){:target="_blank"} in GitHub and follow the instructions there. +{{site.data.callout.end}} + +## Zookeeper Docker image + +Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one + + `docker-compose.yml` +{% highlight yaml %} +{% raw %} + zookeeper: + image: wurstmeister/zookeeper + ports: + - "2181:2181" +{% endraw %} +{% endhighlight %} + +## Kafka Docker image +Now start the Kafka server. In the `docker-compose.yml` it can be something like this + + `docker-compose.yml` +{% highlight yaml %} +{% raw %} + kafka: + build: + context: kafka + dockerfile: Dockerfile + links: + - zookeeper:zk + ports: + - "9092:9092" + environment: + KAFKA_ADVERTISED_HOST_NAME: $CF_HOST_IP + KAFKA_ZOOKEEPER_CONNECT: zk:2181 + KAFKA_MESSAGE_MAX_BYTES: 2000000 + KAFKA_CREATE_TOPICS: "Topic1:1:1" + volumes: + - /var/run/docker.sock:/var/run/docker.sock + depends_on: + - zookeeper +{% endraw %} +{% endhighlight %} + +To start the Kafka server with the certain per-configuration, you need to use Environment variables. Below, you can see which Environment variables are available for this service. + +__Broker IDs__ + +You can configure the broker id in different ways: + +1. Explicitly, using ```KAFKA_BROKER_ID``` +2. Via a command, using ```BROKER_ID_COMMAND```, e.g. ```BROKER_ID_COMMAND: "hostname | awk -F'-' '{print $2}'"``` + +If you don't specify a broker id in your docker-compose file, it will automatically be generated (see [https://issues.apache.org/jira/browse/KAFKA-1070](https://issues.apache.org/jira/browse/KAFKA-1070){:target="_blank"}. This allows scaling up and down. In this case it is recommended to use the ```--no-recreate``` option of docker-compose to ensure that containers are not re-created and thus keep their names and ids. + + +__Automatically create topics__ + +If you want to have kafka-docker automatically create topics in Kafka during +creation, a ```KAFKA_CREATE_TOPICS``` environment variable can be +added in ```docker-compose.yml```. + +Here is an example snippet from ```docker-compose.yml```: + + environment: + KAFKA_CREATE_TOPICS: "Topic1:1:3,Topic2:1:1:compact" + +```Topic 1``` will have 1 partition and 3 replicas, ```Topic 2``` will have 1 partition, 1 replica and a `cleanup.policy` set to `compact`. + +__Advertised hostname__ + +You can configure the advertised hostname in different ways: + +1. Explicitly, using ```KAFKA_ADVERTISED_HOST_NAME``` +2. Via a command, using ```HOSTNAME_COMMAND```, e.g. ```HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"``` + +When using commands, make sure you review the "Variable Substitution" section in [https://docs.docker.com/compose/compose-file/](https://docs.docker.com/compose/compose-file/){:target="_blank"} + +If ```KAFKA_ADVERTISED_HOST_NAME``` is specified, it takes precedence over ```HOSTNAME_COMMAND``` + +For AWS deployment, you can use the Metadata service to get the container host's IP: +``` +HOSTNAME_COMMAND=wget -t3 -T2 -qO- http://169.254.169.254/latest/meta-data/local-ipv4 +``` +Reference: [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html){:target="_blank"} + +__JMX__ + +For monitoring purposes, you may wish to configure JMX. Additional to the standard JMX parameters, problems could arise from the underlying RMI protocol used to connect + +* java.rmi.server.hostname - interface to bind listening port. +* com.sun.management.jmxremote.rmi.port - the port to service RMI requests. + +For example, to connect to a kafka running locally (assumes exposing port 1099) + + KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=1099" + JMX_PORT: 1099 + +## Spring Boot + Kafka +Then grab the spring-kafka JAR and all of its dependencies - the easiest way to do that is to declare a dependency in your build tool, e.g. for Maven: + + `Text` +{% highlight xml %} +{% raw %} ++ +org.springframework.kafka +spring-kafka +${spring-kafka.version} ++ +{% endraw %} +{% endhighlight %} + +Using plain Java to send and receive a message: + + `Java` +{% highlight java %} +{% raw %} +private static String BOOT_TOPIC = "boot.t"; + +@Autowired +private Sender sender; + +@Autowired +private Receiver receiver; + +@ClassRule +public static KafkaEmbedded embeddedKafka = new KafkaEmbedded(1, true, BOOT_TOPIC); + +@BeforeClass +public static void setUpBeforeClass() throws Exception { + System.setProperty("spring.kafka.bootstrap-servers", embeddedKafka.getBrokersAsString()); +} + +@Test +public void testReceive() throws Exception { + sender.send(BOOT_TOPIC, "Hello Boot!"); + + receiver.getLatch().await(10000, TimeUnit.MILLISECONDS); + assertThat(receiver.getLatch().getCount()).isEqualTo(0); +} +{% endraw %} +{% endhighlight %} + +Maven will download the needed dependencies, compile the code and run the unit test case. The result should be a successful build during which following logs are generated: + + `Java` +{% highlight java %} +{% raw %} +. ____ _ __ _ _ + /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ +( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ + \\/ ___)| |_)| | | | | || (_| | ) ) ) ) + ' |____| .__|_| |_|_| |_\__, | / / / / + =========|_|==============|___/=/_/_/_/ + :: Spring Boot :: (v1.5.2.RELEASE) + +08:36:56.175 [main] INFO c.c.kafka.SpringKafkaApplicationTest - Starting SpringKafkaApplicationTest on cnf-pc with PID 700 (started by CodeNotFound in c:\code\st\spring-kafka\spring-kafka-avro) +08:36:56.175 [main] INFO c.c.kafka.SpringKafkaApplicationTest - No active profile set, falling back to default profiles: default +08:36:56.889 [main] INFO c.c.kafka.SpringKafkaApplicationTest - Started SpringKafkaApplicationTest in 1.068 seconds (JVM running for 5.293) +08:36:58.223 [main] INFO c.codenotfound.kafka.producer.Sender - sending user='{"name": "John Doe", "favorite_number": null, "favorite_color": "green"}' +08:36:58.271 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-L-1] INFO c.c.kafka.consumer.Receiver - received user='{"name": "John Doe", "favorite_number": null, "favorite_color": "green"}' +08:37:00.240 [main] ERROR o.a.zookeeper.server.ZooKeeperServer - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes +Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.871 sec - in com.codenotfound.kafka.SpringKafkaApplicationTest + +Results: + +Tests run: 3, Failures: 0, Errors: 0, Skipped: 0 + +[INFO] ------------------------------------------------------------------------ +[INFO] BUILD SUCCESS +[INFO] ------------------------------------------------------------------------ +[INFO] Total time: 41.632 s +[INFO] Finished at: 2017-04-17T08:37:31+02:00 +[INFO] Final Memory: 18M/212M +[INFO] ------------------------------------------------------------------------ +{% endraw %} +{% endhighlight %} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/cd-examples/terraform.md b/_docs/example-catalog/cd-examples/terraform.md new file mode 100644 index 000000000..0dd05f466 --- /dev/null +++ b/_docs/example-catalog/cd-examples/terraform.md @@ -0,0 +1,113 @@ +--- +title: "Deploy with Terraform" +description: "Use Terraform in a Codefresh pipeline with Docker" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +[Terraform](https://www.terraform.io/){:target="\_blank"} is a platform for *Infrastructure as Code*. It allows you to describe your cloud infrastructure in a declarative manner. + +You can use Terraform to deploy to Kubernetes or any other supported cloud platform. Because Terraform itself is already offered [in a Docker container](https://hub.docker.com/r/hashicorp/terraform/){:target="\_blank"}, it is very easy to run Terraform in a Codefresh pipeline. + + +{% include image.html +lightbox="true" +file="/images/examples/terraform/terraform-pipeline.png" +url="/images/examples/terraform/terraform-pipeline.png" +alt="Running Terraform inside Codefresh" +caption="Running Terraform inside Codefresh" +max-width="80%" +%} + +## The example Terraform project + +You can see the example project at [https://github.com/codefresh-contrib/terraform-sample-app](https://github.com/codefresh-contrib/terraform-sample-app){:target="\_blank"}. The repository contains a simple Terraform definition that creates a VM on Google cloud. + +You can play with it locally after installing the `terraform` executable. + +## Prerequisites + +You need to [create a Codefresh account]({{site.baseurl}}/docs/administration/create-a-codefresh-account/) and a Google account first. Then you need to create a [Service account Key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys){:target="\_blank"} which will allow terraform to communicate with Google cloud. + + +Add your service account json as a pipeline variable called `ACCOUNT_JSON_CONTENT`. The content of this variable will be used +in order to authenticate to Google cloud. + +## Create a CI/CD pipeline for Terraform + +Here is the whole pipeline: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - checkout + - prepare + - deploy +steps: + main_clone: + title: Cloning main repository... + stage: checkout + type: git-clone + repo: 'codefresh-contrib/terraform-sample-app' + revision: master + git: github + SetupAuth: + image: alpine:3.9 + title: Setting up Google cloud auth + stage: prepare + commands: + - echo $ACCOUNT_JSON_CONTENT > /codefresh/volume/account.json + - cf_export GOOGLE_CLOUD_KEYFILE_JSON=/codefresh/volume/account.json + DeployWithTerraform: + image: hashicorp/terraform:0.12.0 + title: Deploying Terraform plan + stage: deploy + commands: + - terraform init + - terraform apply -auto-approve + +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code through a [Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +1. Creates a pipeline variable with the path of the Google service account by running [cf_export]({{site.baseurl}}/docs/pipelines/variables/#exporting-environment-variables-from-a-freestyle-step). +1. Creates the VM on Google cloud by running `terraform init/apply`. + +>For simplicity, we auto-approve the Terraform plan in the example pipeline. In a production pipeline, you would instead use an [approval step]({{site.baseurl}}/docs/pipelines/steps/approval/) to inspect the plan before actually applying it. + +The pipeline needs a [single environment variable]({{site.baseurl}}/docs/pipelines/pipelines/#pipeline-settings) that holds the content of the service account. + + +{% include image.html +lightbox="true" +file="/images/examples/terraform/google_cloud_json.png" +url="/images/examples/terraform/google_cloud_json.png" +alt="Passing the Google account in the pipeline parameters" +caption="Passing the Google account in the pipeline parameters" +max-width="60%" +%} + + +Run the pipeline and see your deployment succeed. + + +Note that in a production pipeline you should also handle the [Terraform state](https://www.terraform.io/docs/state/){:target="\_blank"} in a proper manner. The example provided is using a file for [state storage](https://www.terraform.io/docs/backends/index.html){:target="\_blank"} which is not appropriate when using Terraform in a team environment. Instead you should use one of the [storage backends](https://www.terraform.io/docs/backends/types/index.html){:target="\_blank"} that support High Availability and Locking. + + + + +## Handling Pull requests + +You can easily use the same pipeline or a different one for pull requests. In this case replace the `terraform apply` command with `terraform plan`. Even better, you can add an [approval step]({{site.baseurl}}/docs/pipelines/steps/approval/) to allow humans to inspect the pipeline first. + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) diff --git a/_docs/example-catalog/cd-examples/transferring-php-ftp.md b/_docs/example-catalog/cd-examples/transferring-php-ftp.md new file mode 100644 index 000000000..56aa1d270 --- /dev/null +++ b/_docs/example-catalog/cd-examples/transferring-php-ftp.md @@ -0,0 +1,118 @@ +--- +title: "Deploy to VM via FTP" +description: "Deploying a PHP application to a VM using FTP" +group: example-catalog +sub_group: cd-examples +toc: true +redirect_from: + - /docs//learn-by-example/java/spring-mvc-jdbc-template/ +--- + +## Prerequisites + +- A [free Codefresh account]({{site.baseurl}}/docs/administration/account-management/create-a-codefresh-account/){:target="\_blank"} +- A remote machine with an FTP server and SSH setup (ensure that your FTP directory, I.e., `/srv/ftp/pub` has the proper write permissions for the FTP user). + +>Note that as you may already know, FTP is extremely insecure as it relies on plain-text passwords and usernames, making data very vulnerable to sniffing. A more secure solution would be to use SFTP or SCP. + +## Example PHP project + +The example project can be found on [GitHub](https://github.com/codefresh-contrib/ftp-php-app){:target="\_blank"}. The application is a simple PHP application that displays an example timer. + +{% include image.html +lightbox="true" +file="/images/examples/php-file-transfer/test-environment.png" +url="/images/examples/php-file-transfer/test-environment.png" +alt="Example PHP Application" +caption="Example PHP Application" +max-width="90%" +%} + +## Create the pipeline + +Our pipeline includes four stages: + +- A stage for cloning +- A stage for packaging +- A stage for transferring files + +{% include image.html +lightbox="true" +file="/images/examples/php-file-transfer/pipeline.png" +url="/images/examples/php-file-transfer/pipeline.png" +alt="Codefresh UI Pipeline View" +caption="Codefresh UI Pipeline View" +max-width="90%" +%} + +Here is the entire pipeline: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +# More examples of Codefresh YAML can be found at +# https://codefresh.io/docs/docs/example-catalog/ + +version: "1.0" +# Stages can help you organize your steps in stages +stages: + - "clone" + - "install" + - "transfer" +steps: + clone: + title: "Cloning main repository..." + type: "git-clone" + arguments: + repo: "codefresh-contrib/ftp-php-app" + git: "github" + stage: "clone" + install_dependencies: + title: "Collecting Php dependencies..." + type: "freestyle" + working_directory: "./ftp-php-app" + arguments: + image: "composer:1.9.3" + commands: + - "composer install --ignore-platform-reqs --no-interaction --no-plugins --no-scripts --prefer-dist" + stage: "install" + steps: + ftp_transfer: + title: "Transferring application to VM via ftp..." + type: "freestyle" + working_directory: "./ftp-php-app" + arguments: + image: "dockito/lftp-client:latest" + environment: + - USER=org.springframework.kafka +spring-kafka-test +${spring-kafka.version} +test ++ - PASSWORD= + - HOST= + - PUB_FTP_DIR= + commands: + - lftp -e "set ftp:use-site-utime2 false; mirror -x ^\.git/$ -X flat-logo.png -p -R ftp-php-ap $PUB_FTP_DIR/ftp-php-app; exit" -u $USER,$PASSWORD $HOST + stage: "transfer" +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the main repository through a [Git-clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +2. Installs the necessary PHP dependencies for our application through a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). +3. Transfers our application via FTP through another freestyle step. Note that you will need to change the environment variables to your respective values, either in the YAML itself (above), or through the pipeline settings: + +{% include image.html +lightbox="true" +file="/images/examples/php-file-transfer/variables.png" +url="/images/examples/php-file-transfer/variables.png" +alt="Codefresh Environment Variables" +caption="Codefresh Environment Variables" +max-width="90%" +%} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) + + diff --git a/_docs/example-catalog/cd-examples/trigger-a-k8s-deployment-from-docker-registry.md b/_docs/example-catalog/cd-examples/trigger-a-k8s-deployment-from-docker-registry.md new file mode 100644 index 000000000..c15084dd7 --- /dev/null +++ b/_docs/example-catalog/cd-examples/trigger-a-k8s-deployment-from-docker-registry.md @@ -0,0 +1,135 @@ +--- +title: "Trigger a Kubernetes Deployment from a Docker Hub Push Event" +description: "Learn how to trigger a Kubernetes deployment when an image is updated" +group: example-catalog +sub_group: cd-examples +toc: true +--- + +In this example, we will cover how to trigger a Kubernetes deployment from a Dockerhub Push event using a Dockerhub [registry trigger]({{site.baseurl}}/docs/pipelines/triggers/dockerhub-triggers/#create-a-new-dockerhub-trigger). + +Our example has two pipelines: one for packaging code (CI), and the second for deploying code (CD). + +## Prerequisites + +- A [free Codefresh account](https://codefresh.io/docs/docs/getting-started/create-a-codefresh-account/) +- A DockerHub registry [connected to your Codefresh account]({{site.baseurl}}/docs/integrations/docker-registries/#docker-hub) +- A Kubernetes cluster [connected to your Codefresh account]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster) +- A service for your application [deployed to your cluster]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/#viewing-your-kubernetes-services) + +## Example Project + +You can see the example project on [GitHub](https://github.com/codefresh-contrib/registry-trigger-sample-app/tree/master){:target=\_blank"}. The repository contains a simple Hello World NodeJs app as well as 2 pipelines. + +## Create the CI Pipeline + +As mentioned before, our first pipeline will handle the CI process. +The pipeline has three stages: + +- A stage for cloning +- A stage for building the image +- A stage for pushing the image to DockerHub + +{% include image.html +lightbox="true" +file="/images/examples/deployments/k8s-deployment-ci-pipeline.png" +url="/images/examples/deployments/k8s-deployment-ci-pipeline.png" +alt="Codefresh UI CI Pipeline View" +caption="Codefresh UI CI Pipeline View" +max-width="90%" +%} + + `codefresh-CI-pipeline.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' + +stages: +- checkout +- build +- push + +steps: + clone: + title: Cloning main repository... + type: git-clone + stage: checkout + arguments: + repo: 'codefresh-contrib/registry-trigger-sample-app' + revision: 'master' + git: github + build_my_app: + title: Building image... + type: build + stage: build + arguments: + image_name: registry-trigger-sample-app + working_directory: ${{clone}} + tag: 'master' + dockerfile: Dockerfile + push_to_my_registry: + stage: 'push' + type: push + title: Pushing to Dockerhub... + arguments: + candidate: ${{build_my_app}} + tag: 'latest' + registry: dockerhub + image_name: annabaker/registry-trigger-sample-app +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code through a [Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +2. Builds a docker image tagged with the Application version through a [build step]({{site.baseurl}}/docs/pipelines/steps/build/). +3. Pushes the Docker image through a [push step](https://codefresh.io/docs/docs/pipelines/steps/push/) to the Docker Hub registry you have integrated with Codefresh. + +## Create the CD Pipeline + +This pipeline contains one stage/step, for deploying. + +{% include image.html +lightbox="true" +file="/images/examples/deployments/k8s-deployment-CD-pipeline.png" +url="/images/examples/deployments/k8s-deployment-CD-pipeline.png" +alt="Codefresh UI CD Pipeline View" +caption="Codefresh UI CD Pipeline View" +max-width="90%" +%} + +Note that for the trigger mechanism to take place, you will need to [add a Docker Hub registry trigger]({{site.baseurl}}/docs/pipelines/triggers/dockerhub-triggers/#create-a-new-dockerhub-trigger) to the pipeline. + + `codefresh-CD-pipeline.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" + +stages: + - "deploy" + +steps: + deploy_to_k8s: + title: Running Deploy Script... + type: deploy + kind: kubernetes + arguments: + cluster: anna-demo@FirstKubernetes + namespace: default + service: registry-trigger-sample-app + candidate: + image: annabaker/registry-trigger-sample-app:latest + registry: 'dockerhub' +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Deploys the image to Kubernetes through a [deploy step]({{site.baseurl}}/docs/pipelines/steps/deploy/). The deploy step uses a [Registry trigger]({{site.baseurl}}/docs/pipelines/triggers/dockerhub-triggers/#create-a-new-dockerhub-trigger) to kick off the pipeline when the updated image is pushed to the registry. + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) +[Triggers in pipelines]({{site.baseurl}}/docs/pipelines/triggers/) diff --git a/_docs/example-catalog/cd-examples/use-kubectl-as-part-of-freestyle-step.md b/_docs/example-catalog/cd-examples/use-kubectl-as-part-of-freestyle-step.md new file mode 100644 index 000000000..b228a895f --- /dev/null +++ b/_docs/example-catalog/cd-examples/use-kubectl-as-part-of-freestyle-step.md @@ -0,0 +1,42 @@ +--- +title: "Use kubectl as part of freestyle step" +description: "How to run manually kubectl in a Codefresh pipeline" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/use-kubectl-as-part-of-freestyle-step/ +toc: true +--- + + +Running Kubernetes commands in Codefresh as part of the workflow is very easy. + + +Codefresh is adding all your clusters into the workflow ready to be used as part of your CI/CD pipeline. +The context remains the same as it appears in the [Codefresh Kubernetes dashboard]({{site.baseurl}}/docs/deploy-to-kubernetes/manage-kubernetes/). + +>If your cluster name includes spaces then make sure that you use quotes in the `kubectl` command. + +* Use image: `codefresh/kubectl` +* Add your commands: + * `kubectl config get-contexts`. Will print the cluster that we added to the workflow + * `kubectl config use-context "my-cluster-name"`. The name is the same as in `Account settings` → `Integrations` → `Kubernetes` + * `kubectl get po -owide` + * `kubectl get nodes` + + +## Follow the example + +* Add this [Git repo](https://github.com/Codefresh-Examples/kubectl-in-freestyle-step){:target="_blank"} to your account +* Change the pipeline configuration to use `codefresh.yml`. +* Build. + +## Running parallel steps with kubectl + +More complex examples can be found in the [custom kubectl commands]({{site.baseurl}}/docs/deploy-to-kubernetes/custom-kubectl-commands/) documentation page. + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/cd-examples/web-terminal.md b/_docs/example-catalog/cd-examples/web-terminal.md new file mode 100644 index 000000000..371515283 --- /dev/null +++ b/_docs/example-catalog/cd-examples/web-terminal.md @@ -0,0 +1,48 @@ +--- +title: "Web terminal" +description: "" +group: example-catalog +sub_group: cd-examples +redirect_from: + - /docs/web-terminal/ + - /docs/on-demand-test-environment/example-compositions/web-terminal/ +toc: true +--- +This example shows you how to access containers running in a Codefresh standup environment. + +## Looking around +In the root of this repository you'll find a file named `docker-compose.yml`. +Here are the contents of this file: + + `Composition.yml` +{% highlight yaml %} +version: '3' +services: + my-service: + image: 'containers101/whomi:master' + volumes: + - my-service:/app + ports: + - '1337' + terminal: + image: 'containers101/cfterminal:master' + ports: + - '8000' + volumes_from: + - my-service +volumes: + my-service: + driver: local +{% endhighlight %} + +{{site.data.callout.callout_info}} +##### Example + +Just head over to the example [__repository__](https://github.com/codefreshdemo/cf-example-web-termial){:target="_blank"} in GitHub and follow the instructions there. +{{site.data.callout.end}} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#cd-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/ci-examples/android.md b/_docs/example-catalog/ci-examples/android.md new file mode 100644 index 000000000..a02d66b17 --- /dev/null +++ b/_docs/example-catalog/ci-examples/android.md @@ -0,0 +1,80 @@ +--- +title: "Compile and package an Android application" +description: "Using Codefresh pipelines" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +Android applications use Java/Gradle for their build system. Because Codefresh already supports [Gradle]({{site.baseurl}}/docs/example-catalog/ci-examples/gradle/), it is also very easy to build Android projects. + +Any Gradle command can run inside a Docker image that contains the Android SDK. As an example, we will use a [Nextcloud](https://hub.docker.com/r/nextcloudci/android){:target="\_blank"} image from Dockerhub. + + +## The example project + +You can see the example project at [https://github.com/codefresh-contrib/android-sample-app](https://github.com/codefresh-contrib/android-sample-app){:target="\_blank"}. The repository contains a Hello World Android project with the following tasks: + +* `./gradlew test` runs unit tests +* `./gradlew build` builds the application + + +## Create a CI pipeline that compiles/releases Android + +In most cases you would create a similar pipeline to a Gradle project. + +{% include image.html +lightbox="true" +file="/images/learn-by-example/mobile/android-ci-pipeline.png" +url="/images/learn-by-example/mobile/android-ci-pipeline.png" +alt="Building and Testing an Android app" +caption="Building and Testing an Android app" +max-width="80%" +%} + +Here is the [full pipeline](https://github.com/codefresh-contrib/android-sample-app/blob/master/codefresh.yml){:target="\_blank"} that uses a Docker image with the Android SDK in order to run Gradle. + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - test + - build +steps: + main_clone: + title: Cloning main repository... + stage: prepare + type: git-clone + repo: 'codefresh-contrib/android-sample-app' + revision: master + git: github + TestIt: + title: Running Tests + stage: test + image: nextcloudci/android:android-48 + commands: + - chmod +x ./gradlew + - ./gradlew test --no-daemon --gradle-user-home=/codefresh/volume/.gradle + BuildIt: + title: Packaging Android App + stage: build + image: nextcloudci/android:android-48 + commands: + - ./gradlew build --no-daemon --gradle-user-home=/codefresh/volume/.gradle +{% endraw %} +{% endhighlight %} + +This pipeline clones the source code, runs unit tests and finally builds the Android application. + +Codefresh is smart enough that [caches automatically]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/#how-caching-works-in-codefresh) for us the workspace of a build (`/codefresh/volume`). This works great for build tools that keep their cache in the project folder, but not for Maven/Gradle which keep their cache externally. By changing the location of the Gradle cache we make sure that Codefresh will cache automatically the Gradle libraries resulting in much faster builds. + + + +## Related articles +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) + diff --git a/_docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository.md b/_docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository.md new file mode 100644 index 000000000..d81e5363a --- /dev/null +++ b/_docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository.md @@ -0,0 +1,94 @@ +--- +title: "Build an Image from a different Git repository" +description: "Build microservices from other repositories" +group: example-catalog +sub_group: ci-examples +redirect_from: + - /docs/build-an-image-from-a-different-git-repository/ +toc: true +--- + +In most cases, your Codefresh pipeline checks out a single Git repository. Codefresh has great support also for [monorepos]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/#using-the-modified-files-field-to-constrain-triggers-to-specific-folderfiles) if you have placed all your applications in a single repository. + +A Codefresh pipeline is not really tied to a specific Git repository, which means that by [checking out multiple Git repositories]({{site.baseurl}}/docs/example-catalog/ci-examples/git-checkout/#cloning-multiple-repositories) you can build Docker images from other unrelated repositories in a single pipeline if you wish to do so. + +## Building Docker images from other Git repositories + + +Here is a Codefresh pipeline that checks out two microservices from two different Git repositories. + +{% include image.html +lightbox="true" +file="/images/examples/docker-build/build-from-other-git-repo.png" +url="/images/examples/docker-build/build-from-other-git-repo.png" +alt="Checkout and build docker images" +caption="Checkout and build docker images" +max-width="100%" +%} + +And here is the [pipeline definition]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/). + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - 'clone phase' + - 'build phase' +steps: + checkoutApp1: + title: 'Cloning first repository...' + type: git-clone + repo: kostis-codefresh/example_nodejs_postgres + revision: experiment1 + git: github + stage: 'clone phase' + checkoutApp2: + title: 'Cloning second repository...' + type: git-clone + repo: kostis-codefresh/trivial-go-web + revision: master + git: github + stage: 'clone phase' + myFirstDockerImage: + title: 'Building Microservice A' + type: build + dockerfile: Dockerfile + image_name: my-nodejs-image + tag: from-develop-branch + working_directory: './example_nodejs_postgres' + stage: 'build phase' + mySecondDockerImage: + title: 'Building Microservice B' + type: build + dockerfile: Dockerfile + working_directory: './trivial-go-web' + image_name: my-app-image + tag: from-master-branch + stage: 'build phase' +{% endraw %} +{% endhighlight %} + +The pipeline first checks out two different Git repositories, which themselves contain Dockerfiles. Then it creates a Docker image for each one using the respective Dockerfile. + +You can see both images in the [Docker image dashboard]({{site.baseurl}}/docs/docker-registries/#viewing-docker-images) . + +{% include image.html +lightbox="true" +file="/images/examples/docker-build/two-docker-images.png" +url="/images/examples/docker-build/two-docker-images.png" +alt="Docker images from other Git repos" +caption="Docker images from other Git repos" +max-width="100%" +%} + + +Notice that there are no explicit push steps in the pipeline, as all successful Codefresh pipelines automatically push to the private Docker registry. + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/) +[Build step in pipelines in pipelines]({{site.baseurl}}/docs/pipelines/steps/build/) +[Build and Push an image]({{site.baseurl}}/docs/pipelines/examples/build-and-push-an-image/) +[Parallel pipelines]({{site.baseurl}}/docs/pipelines/advanced-workflows/) diff --git a/_docs/example-catalog/ci-examples/build-an-image-specify-dockerfile-location.md b/_docs/example-catalog/ci-examples/build-an-image-specify-dockerfile-location.md new file mode 100644 index 000000000..75d5b67f5 --- /dev/null +++ b/_docs/example-catalog/ci-examples/build-an-image-specify-dockerfile-location.md @@ -0,0 +1,74 @@ +--- +title: "Build an Image by specifying a Dockerfile location" +description: "How to choose a Dockerfile to build with Codefresh pipelines" +group: example-catalog +sub_group: ci-examples +redirect_from: + - /docs/build-an-image-specify-dockerfile-location/ +toc: true +--- + +You may have a project where the Dockerfile is **not** in the root folder of the project. Maybe the repository has multiple projects inside, each with its own Dockerfile, or you simply want to use a different folder for the Docker context. + +>The source code of the repository is at [https://github.com/codefreshdemo/cf-example-dockerfile-other-location](https://github.com/codefreshdemo/cf-example-dockerfile-other-location){:target="\_blank"}. Feel free to fork it if you want to follow along. + +If you don't have a Codefresh account already, you can easily create a free one from the [sign-up page]({{site.baseurl}}/docs/administration/create-a-codefresh-account/). + + +## Building a Dockerfile from a different folder + +By default, if you run a single command like the one below, Docker uses the Dockerfile of the current folder: + +``` +docker build . -t my-web-app +``` + +If your Dockerfile is in a different folder, specify it explicitly with: + +``` +docker build . -t my-web-app -f subfolder/Dockerfile +``` + +Codefresh supports a similar syntax as well. The `dockerfile` property of the [build step]({{site.baseurl}}/docs/pipelines/steps/build/) can accept a full path. + +Here is the full pipeline: + + `codefresh.yml` +{% highlight yaml %} +version: '1.0' +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'codefreshdemo/cf-example-dockerfile-other-location' + revision: 'master' + git: github + build_my_app: + title: Building Node.Js Docker Image + type: build + image_name: my-app + working_directory: '.' + tag: 'master' + dockerfile: docker/Dockerfile +{% endhighlight %} + +This pipeline checks out the source code of the repository and then builds a Dockerfile found at the subfolder `docker` while still keeping as Docker context the root directory. + +{% include image.html +lightbox="true" +file="/images/examples/docker-build/build-spefify-dockerfile.png" +url="/images/examples/docker-build/build-spefify-dockerfile.png" +alt="Building a Docker image with specific Dockerfile" +caption="Building a Docker image with specific Dockerfile" +max-width="100%" +%} + +You could also change the Docker build context by editing the `working_directory` property. By default, it looks at the root folder of the project, but any subfolder path is also valid. + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Build step in pipelines]({{site.baseurl}}/docs/pipelines/steps/build/) +[Build an Image with the Dockerfile in root directory]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-dockerfile-in-root-directory/) +[Build an Image from a different Git repository]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository) +[Build and push an Image]({{site.baseurl}}/docs/yaml-examples/example-catalog/ci-examples/build-and-push-an-image) +[Build an Image With build arguments]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-with-build-arguments) \ No newline at end of file diff --git a/_docs/example-catalog/ci-examples/build-an-image-with-build-arguments.md b/_docs/example-catalog/ci-examples/build-an-image-with-build-arguments.md new file mode 100644 index 000000000..a7a623566 --- /dev/null +++ b/_docs/example-catalog/ci-examples/build-an-image-with-build-arguments.md @@ -0,0 +1,133 @@ +--- +title: "Build an Image with build arguments" +description: "Use Docker arguments in Codefresh pipelines" +group: example-catalog +sub_group: ci-examples +redirect_from: + - /docs/build-an-image-with-build-arguments/ +toc: true +--- + +Building a Docker image that requires build arguments is very easy with Codefresh pipelines. + +The source code of the repository is at [https://github.com/codefreshdemo/cf-example-build-arguments](https://github.com/codefreshdemo/cf-example-build-arguments){:target="\_blank"}. Feel free to fork it if you want to follow along. + +If you don't have a Codefresh account already, you can easily create a free one from the [sign-up page]({{site.baseurl}}/docs/administration/create-a-codefresh-account/). + +## Using Docker build arguments + +The example application is a very simple NodeJS application with the following DYouockerfile: + +`Dockerfile` +{% highlight docker %} +{% raw %} +ARG NODE_VERSION +FROM node:$NODE_VERSION + +ARG APP_DIR + +RUN mkdir -p $APP_DIR + +WORKDIR $APP_DIR + +COPY package.json . +RUN npm install --silent +COPY . . +EXPOSE 3000 + +ENV PORT 3000 + +CMD [ "npm", "start" ] +{% endraw %} +{% endhighlight %} + +This Dockerfile expects two [build arguments](https://docs.docker.com/engine/reference/builder/#/arg){:target="\_blank"}: + +* `NODE_VERSION` is the version of Node image to use as base +* `APP_DIR` is the source directory to be used inside the container + +## Building a Dockerfile passing values for build arguments + +When you build an image locally on your workstation, you can define build arguments with the `--build-arg` syntax: + +``` +docker build . -t my-node-app --build-arg NODE_VERSION=8 --build-arg APP_DIR=/usr/src/app +``` + +You can get the same result within a Codefresh pipeline: + + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'codefreshdemo/cf-example-build-arguments' + revision: 'master' + git: github + build_my_app: + title: Building Node.Js Docker Image + type: build + image_name: my-app + working_directory: '.' + tag: 'master' + dockerfile: Dockerfile + build_arguments: + - NODE_VERSION=8 + - APP_DIR=/usr/src/app +{% endraw %} +{% endhighlight %} + +This pipeline checks out the source code of the repository and then builds the Dockerfile by passing the values `8` and `/usr/src/app` to the two arguments. + +{% include image.html +lightbox="true" +file="/images/examples/docker-build/docker-build-arguments.png" +url="/images/examples/docker-build/docker-build-arguments.png" +alt="Using Docker build arguments in a pipeline" +caption="Using Docker build arguments in a pipeline" +max-width="100%" +%} + +## Using Codefresh variables as build arguments + +In the previous pipeline, the Docker build arguments are defined in the pipeline itself, but you can also use [pipeline variables]({{site.baseurl}}/docs/pipelines/pipelines/#creating-new-pipelines), [shared configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/), or any other standard mechanism you already have in place. + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'codefreshdemo/cf-example-build-arguments' + revision: 'master' + git: github + build_my_app: + title: Building Node.Js Docker Image + type: build + image_name: my-app + working_directory: '.' + tag: 'master' + dockerfile: Dockerfile + build_arguments: + - NODE_VERSION=${{NODE_VERSION_FROM_SHARED_CONFIG}} + - APP_DIR=${{APP_DIR_PIPELINE_VARIABLE}} +{% endraw %} +{% endhighlight %} + +In this case, you can also use any of the built-in [Codefresh variables]({{site.baseurl}}/docs/pipelines/variables/). + + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Build step in pipelines]({{site.baseurl}}/docs/pipelines/steps/build/) +[Build an Image with the Dockerfile in root directory]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-dockerfile-in-root-directory/) +[Build an Image by specifying the Dockerfile location]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-specify-dockerfile-location) +[Build an Image from a different Git repository]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository) +[Build and push an Image]({{site.baseurl}}/docs/yaml-examples/example-catalog/ci-examples/build-and-push-an-image) diff --git a/_docs/example-catalog/ci-examples/build-an-image-with-the-dockerfile-in-root-directory.md b/_docs/example-catalog/ci-examples/build-an-image-with-the-dockerfile-in-root-directory.md new file mode 100644 index 000000000..a9c5cb2e2 --- /dev/null +++ b/_docs/example-catalog/ci-examples/build-an-image-with-the-dockerfile-in-root-directory.md @@ -0,0 +1,67 @@ +--- +title: "Build an Image with the Dockerfile in root directory" +description: "Get started quickly with building Docker images" +group: example-catalog +sub_group: ci-examples +toc: true +--- +Building a Docker image is one of the basic operations in Codefresh pipelines. + +>The source code of the repository is at [https://github.com/codefreshdemo/cf-yml-example-build-dockerfile-inroot](https://github.com/codefreshdemo/cf-yml-example-build-dockerfile-inroot){:target="\_blank"}. Feel free to fork it if you want to follow along. + +If you don't have a Codefresh account already, you can easily create a free one from the [sign-up page]({{site.baseurl}}/docs/administration/create-a-codefresh-account/). + + +## Building a Dockerfile from the root folder + +By default, if you run a single command like the one below, Docker uses the Dockerfile of the current folder: + +``` +docker build . -t my-web-app +``` + +You can get the same result within a Codefresh pipeline: + + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'codefreshdemo/cf-yml-example-build-dockerfile-inroot' + revision: 'master' + git: github + build_my_app: + title: Building Node.Js Docker Image + type: build + image_name: my-app + working_directory: '${{main_clone}}' + tag: 'master' + dockerfile: Dockerfile +{% endraw %} +{% endhighlight %} + +This pipeline checks out the source code of the repository and then builds a dockerfile found at the root folder of the project. + +{% include image.html +lightbox="true" +file="/images/examples/docker-build/build-dockerfile-root.png" +url="/images/examples/docker-build/build-dockerfile-root.png" +alt="Building a Docker image with a default Dockerfile" +caption="Building a Docker image with a default Dockerfile" +max-width="100%" +%} + +You can also change the Docker build context by editing the `working_directory` property. By default, it looks at the root folder of the project, but any subfolder path is also valid. + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Build step in pipelines]({{site.baseurl}}/docs/pipelines/steps/build/) +[Build an Image by specifying the Dockerfile location]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-specify-dockerfile-location) +[Build an Image from a different Git repository]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository) +[Build and push an Image]({{site.baseurl}}/docs/yaml-examples/example-catalog/ci-examples/build-and-push-an-image) +[Build an Image With build arguments]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-with-build-arguments) diff --git a/_docs/example-catalog/ci-examples/build-and-push-an-image.md b/_docs/example-catalog/ci-examples/build-and-push-an-image.md new file mode 100644 index 000000000..33ebac637 --- /dev/null +++ b/_docs/example-catalog/ci-examples/build-and-push-an-image.md @@ -0,0 +1,137 @@ +--- +title: "Build and push an Image" +description: "Build Docker images and push them to registries with Codefresh" +group: example-catalog +sub_group: ci-examples +redirect_from: + - /docs/build-and-push-an-image/ + - /docs/docker-registries/push-image-to-a-docker-registry/ +toc: true +--- + +Building a Docker image and then pushing it to a registry is one of the most basic scenarios for creating a pipeline. +In this example we will use a demo Node.js application that will be packaged in a Docker image. + +The source code of the repository is at [https://github.com/codefreshdemo/cf-example-build-and-push](https://github.com/codefreshdemo/cf-example-build-and-push){:target="\_blank"}. Feel free to fork it if you want to follow along. + +If you don't have a Codefresh account already, you can easily create a free one from the [sign-up page]({{site.baseurl}}/docs/administration/create-a-codefresh-account/). + + +## Building and push Docker image to default registry + +Building a Docker image with Codefresh is easy, and only requires a simple step. In addition, all successful pipelines in Codefresh automatically push to [your default Docker registry]({{site.baseurl}}/docs/docker-registries/#the-default-registry), without additional configuration, if you have one. + +Here is the most basic pipeline that clones a repo and builds an image: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: +- checkout +- build +steps: + clone: + title: Cloning main repository... + type: git-clone + stage: checkout + repo: 'codefreshdemo/cf-example-build-and-push' + revision: 'master' + git: github + build_my_app: + title: Building Node.Js Docker Image + type: build + stage: build + image_name: my-node-js-app + working_directory: {{clone}} + tag: 'master' + dockerfile: Dockerfile +{% endraw %} +{% endhighlight %} + +## Building and pushing Docker image to _any registry_. + +You can push your image to any [registry]({{site.baseurl}}/docs/docker-registries/). + +* First you need to connect your external registry in the integrations page. Here are the instructions for: + + * [Docker Hub]({{site.baseurl}}/docs/integrations/docker-registries/docker-hub/) + * [Google Container Registry]({{site.baseurl}}/docs/integrations/docker-registries/google-container-registry/) + * [Amazon EC2 Container Registry]({{site.baseurl}}/docs/integrations/docker-registries/amazon-ec2-container-registry/) + * [Bintray.io]({{site.baseurl}}/docs/integrations/docker-registries/bintray-io/) + * [Quay.io]({{site.baseurl}}/docs/integrations/docker-registries/quay-io/) + * [Other Registries]({{site.baseurl}}/docs/integrations/docker-registries/other-registries/) + +* Then add a [push step]({{site.baseurl}}/docs/pipelines/steps/push/) in your pipeline and use the registry name of your integration. + +Here is the full example: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: +- checkout +- build +- push +steps: + clone: + title: Cloning main repository... + type: git-clone + stage: checkout + repo: 'codefreshdemo/cf-example-build-and-push' + revision: 'master' + git: github + build_my_app: + title: Building Node.Js Docker Image + type: build + stage: build + image_name: my-node-js-app + working_directory: {{clone}} + tag: 'master' + dockerfile: Dockerfile + push_to_my_registry: + stage: 'push' + type: push + title: Pushing to a registry + candidate: ${{build_my_app}} + tag: 'v1.0.0' + registry: dockerhub + image_name: kkapelon/my-node-js-app +{% endraw %} +{% endhighlight %} + +Here we use a specific tag - `v1.0.0` but +Codefresh has several variables that you can use to tag images. Common examples are `CF_BRANCH_TAG_NORMALIZED`, `CF_SHORT_REVISION` or `CF_BUILD_ID`. Read more on [variables]({{site.baseurl}}/docs/pipelines/variables/). + +{% include image.html + lightbox="true" + file="/images/examples/docker-build/build-and-push-pipeline.png" + url="/images/examples/docker-build/build-and-push-pipeline.png" + alt="Pushing image to external registry" + caption="Pushing image to external registry" + max-width="100%" + %} + + +If you run the pipeline, the Docker image is pushed *both* to the private Docker regisry (by the build step) *and* the external docker registry (by the push step). + + +## More options for pushing images + +Codefresh has several options when it comes to pushing images: + +* You can specify multiple tags to be pushed +* You can use directly ECR registries +* You can embed credentials in the push steps + +Read more in [push steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/push/). + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Build step in pipelines]({{site.baseurl}}/docs/pipelines/steps/build/) +[Build an Image with the Dockerfile in root directory]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-dockerfile-in-root-directory/) +[Build an Image by specifying the Dockerfile location]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-specify-dockerfile-location) +[Build an Image from a different Git repository]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository) +[Build an Image With Build arguments]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-with-build-arguments) diff --git a/_docs/example-catalog/ci-examples/c-make.md b/_docs/example-catalog/ci-examples/c-make.md new file mode 100644 index 000000000..06b95d76d --- /dev/null +++ b/_docs/example-catalog/ci-examples/c-make.md @@ -0,0 +1,74 @@ +--- +title: "Compile and test a C application" +description: "Using Codefresh pipelines" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +Codefresh can work with any C/C++ application very easily as both `gcc` and `g++` are already offered in Dockerhub. There is also another example available with [C++ and cmake]({{site.baseurl}}/docs/example-catalog/ci-examples/cpp-cmake). + +## The example C project + +You can see the example project at [https://github.com/codefresh-contrib/c-sample-app](https://github.com/codefresh-contrib/c-sample-app){:target="\_blank"}. The repository contains a C starter project with a `Makefile` and several targets: + +* `make` compiles the code. +* `make test` runs unit tests +* `make clean` removes artifacts and binaries. + +There are also extra targets for `tags` and `etags`. + +## Create a CI pipeline for C applications + +Creating a CI/CD pipeline for C is very easy, because Codefresh can run any [gcc image](https://hub.docker.com/_/gcc/){:target="\_blank"} that you wish. Gcc docker images already contain the `make` utility. + +{% include image.html +lightbox="true" +file="/images/learn-by-example/cc/c-make-pipeline.png" +url="/images/learn-by-example/cc/c-make-pipeline.png" +alt="Compiling a C application in a pipeline" +caption="Compiling a C application in a pipeline" +max-width="80%" +%} + +Here is the [full pipeline](https://github.com/codefresh-contrib/c-sample-app/blob/master/codefresh.yml){:target="\_blank"} that compiles the application after checking out the code. + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - checkout + - build +steps: + main_clone: + title: Cloning main repository... + stage: checkout + type: git-clone + repo: 'codefresh-contrib/c-sample-app' + revision: master + git: github + compile_my_sources: + title: Compile + stage: build + image: gcc + commands: + - make + run_my_tests: + title: Test + stage: build + image: gcc + commands: + - make test +{% endraw %} +{% endhighlight %} + +This pipeline clones the source code, compiles the code and runs unit tests. In all cases we use the public Docker image of Gcc that also contains `make`. + + +## Related articles +[C++ example]({{site.baseurl}}/docs/example-catalog/ci-examples/cpp-cmake/) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/ci-examples/call-child-pipelines.md b/_docs/example-catalog/ci-examples/call-child-pipelines.md new file mode 100644 index 000000000..fd83b5b7c --- /dev/null +++ b/_docs/example-catalog/ci-examples/call-child-pipelines.md @@ -0,0 +1,108 @@ +--- +title: "Call a CD pipeline from a CI pipeline" +description: "How to call child pipelines from a parent pipeline" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +In Codefresh you can easily create nested pipelines by calling other pipelines from within an existing pipeline. The [codefresh-run plugin](https://codefresh.io/steps/step/codefresh-run){:target="\_blank"} allows you to launch another pipeline, and optionally wait for its completion. + +{% include image.html +lightbox="true" +file="/images/examples/nested-pipelines/call-other-pipeline.png" +url="/images/examples/nested-pipelines/call-other-pipeline.png" +alt="Parent and child pipelines" +caption="Parent and child pipelines" +max-width="80%" +%} + +A very common pattern in Codefresh is to have a parent pipeline responsible for Continuous Integration (packaging code), that calls a child pipeline for Continuous Delivery (taking care of deployment). + +## Example project + +You can see the example project at [https://github.com/codefresh-contrib/call-child-pipeline-sample-app](https://github.com/codefresh-contrib/call-child-pipeline-sample-app){:target="\_blank"}. The repository contains a NodeJs app as well as three - one parent and two child pipelines. + +## Create a pipeline that calls other pipelines + +Here is the definition of the parent pipeline: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - package + - deploy +steps: + main_clone: + title: 'Cloning main repository...' + type: git-clone + repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}' + revision: '${{CF_REVISION}}' + git: github + stage: prepare + read_my_app_version: + title: Reading Application version + stage: prepare + image: node:latest + commands: + - export PACKAGE_VERSION=$(node -p "require('./package.json').version") + - cf_export PACKAGE_VERSION + build_my_docker_image: + title: 'Building My Docker Image' + stage: package + type: build + dockerfile: Dockerfile + image_name: my-app-image + tag: ${{PACKAGE_VERSION}} + call_qa_pipeline: + title: Deploy to QA + stage: deploy + type: codefresh-run + arguments: + PIPELINE_ID: child-pipelines/qa-pipeline + VARIABLE: + - CF_BRANCH=${{CF_BRANCH}} + - CF_REVISION=${{CF_REVISION}} + - APP_VERSION=${{PACKAGE_VERSION}} + when: + branch: + only: + - develop + call_prod_pipeline: + title: Deploy to Prod + stage: deploy + type: codefresh-run + arguments: + PIPELINE_ID: child-pipelines/prod-pipeline + VARIABLE: + - CF_BRANCH=${{CF_BRANCH}} + - CF_REVISION=${{CF_REVISION}} + - APP_VERSION=${{PACKAGE_VERSION}} + when: + branch: + only: + - /^release.*/i + + +{% endraw %} +{% endhighlight %} + +This pipeline does the following: + +1. Clones the source code through a [Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/). +1. Creates a variable that contains the Application version as specified in `package.json` through [cf_export]({{site.baseurl}}/docs/pipelines/variables/#exporting-environment-variables-from-a-freestyle-step). +1. Builds a docker image tagged with the Application version through a [build step]({{site.baseurl}}/docs/pipelines/steps/build/). +1. Optionally runs the downstream QA pipeline if the branch is named `develop`. It also passes several environment variables to the child pipeline (including the Application version). +1. Optionally runs the downstream Prod pipeline if the branch name starts with `release`. It also passes several environment variables to the child pipeline (including the Application version). + +The last two steps use [conditions]({{site.baseurl}}/docs/pipelines/conditional-execution-of-steps/) to decide if they will run or not. + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[Pipeline plugins](https://codefresh.io/steps/){:target="\_blank"} \ No newline at end of file diff --git a/_docs/example-catalog/ci-examples/cc.md b/_docs/example-catalog/ci-examples/cc.md new file mode 100644 index 000000000..c23c08fcd --- /dev/null +++ b/_docs/example-catalog/ci-examples/cc.md @@ -0,0 +1,10 @@ +--- +title: "C/C++" +description: "How to build C/C++ applications with Codefresh CI/CD pipelines" +group: example-catalog +toc: true +--- +This section contains Codefresh examples based on C and C++. + +- [C Example with make]({{site.baseurl}}/docs/learn-by-example/cc/c-make) +- [C++ Example with cmake]({{site.baseurl}}/docs/learn-by-example/cc/cpp-cmake) \ No newline at end of file diff --git a/_docs/example-catalog/ci-examples/codacy-testing.md b/_docs/example-catalog/ci-examples/codacy-testing.md new file mode 100644 index 000000000..bd2e437b6 --- /dev/null +++ b/_docs/example-catalog/ci-examples/codacy-testing.md @@ -0,0 +1,174 @@ +--- +title: "Codacy coverage reports" +description: "How to forward coverage reports to Codacy" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +[Codacy](https://www.codacy.com/){:target="\_blank"} is a code review tool that allows automatic analysis, code coverage tracking, and extensive reports, for you and your team to improve your code quality over time. + +Analysis reports displayed within Codacy dashboard: +{% include image.html +lightbox="true" +file="/images/testing/codacy/codacy-report.png" +url="/images/testing/codacy/codacy-report.png" +alt="Codacy UI with coverage reports" +max-width="100%" +%} + +## Prerequisites for using Codacy + +* A simple [Codefresh pipeline, up and running]({{site.baseurl}}/docs/getting-started/create-a-basic-pipeline/) +* A [Codacy account](https://www.codacy.com/){:target="\_blank"} (free, pro or enterprise) +* A testing tool added to your project that produces coverage reports + +Codacy supports over [30 different language integrations](https://docs.codacy.com/getting-started/supported-languages-and-tools/){:target="\_blank"}. Depending on the programming language used, it requires little to no set-up. + +You could try it out by cloning our [node example application](https://github.com/codefresh-contrib/codacy-sample-app){:target="\_blank"} that utilises [jest](https://jestjs.io/){:target="\_blank"}. + +## Create an account with Codacy +Codacy has a free version, a pro version, and an on-premises version. The latter two have a free trial, which allows you to test all features over the course of two weeks. You can sign-up via GitHub, Bitbucket, or GitLab. + +When you log into Codacy for the first time, it will ask you to provide access to a repository. At this stage, Codacy will not download any code from your repository but merely access its names. You can then either provide access to selective repositories or your entire git account. + +{% include image.html +lightbox="true" +file="/images/testing/codacy/codacy-add-repo.png" +url="/images/testing/codacy/codacy-add-repo.png" +alt="Add repository to codacy" +max-width="80%" +%} + +## Generate Project API token +To use Codacy, we need a project API token. To generate the token, select your project => go to settings => integrations => add integration => select “Project API”. Make sure that you select the API token from here and not your general project settings. + +{% include image.html +lightbox="true" +file="/images/testing/codacy/create-api-token.png" +url="/images/testing/codacy/create-api-token.png" +alt="Create Project API token" +max-width="80%" +%} + +## Codefresh pipeline + +In case the project that you want to use Codacy in does not have a pipeline, [create a new pipeline]({{site.baseurl}}/docs/getting-started/create-a-basic-pipeline/). + +{% include image.html +lightbox="true" +file="/images/testing/codacy/create-codacy-pipeline.png" +url="/images/testing/codacy/create-codacy-pipeline.png" +alt="Create Codacy Pipeline" +max-width="80%" +%} + +**Setting-up step** + +This step is based on our [TypeScript application](https://github.com/codefresh-contrib/codacy-sample-app){:target="\_blank"}. Before we set up our pipeline, we will add our Project API token as our environment variable. Note that we have specified our token in the variables section on the right, as displayed in the following screenshot. + +{% include image.html +lightbox="true" +file="/images/testing/codacy/codacy-variable.png" +url="/images/testing/codacy/codacy-variable.png" +alt="Provide Codacy ENV variable" +max-width="80%" +%} + +Once the variable is called through the [Codefresh yml syntax]({{site.baseurl}}/docs/pipelines/variables/), it automatically uses the value provided within the variables section. If you are using this example as your pipeline, please delete anything in your pipeline. We can then add the following pipeline to our Inline YAML within the Workflow section in our UI: + +{% highlight yaml %} +{% raw %} +version: "1.0" +# Stages can help you organize your steps in stages +stages: + - "clone" + - "build" + - "test" + +steps: + clone: + title: "Cloning repository" + type: "git-clone" + repo: "anais-codefresh/codacy-sample-app" + # CF_BRANCH value is auto set when pipeline is triggered + # Learn more at codefresh.io/docs/docs/pipelines/variables/ + revision: "${{CF_BRANCH}}" + git: "github" + stage: "clone" + + build: + title: "Building Docker image" + type: "build" + image_name: "anaisurlichs/codacy-sample-app" + working_directory: "${{clone}}" + tag: "${{CF_BRANCH_TAG_NORMALIZED}}" + dockerfile: "Dockerfile" + stage: "build" + registry: "dockerhub" + + tests: + title: "Running test" + type: "freestyle" + working_directory: '${{clone}}' + arguments: + image: 'node:15.2' + commands: + - "npm install --save-dev jest" + - "npm run test" + stage: "test" + + codacy: + title: "Pushing reports to codacy" + type: "freestyle" + working_directory: '${{clone}}' + arguments: + image: 'alpine:3.8' + commands: + - "export CODACY_PROJECT_TOKEN=${{CODACY_PROJECT_TOKEN}}" + - "wget -qO - https://coverage.codacy.com/get.sh | sh" + stage: "test" +{% endraw %} +{% endhighlight %} + +The last two steps, ’tests’ and ’codacy’, are used to run our tests, create our coverage reports and forward those to Codacy. If you are using your own project and existing pipeline, add those two steps to your pipeline. In case you are using your own application, make sure to adapt the commands within the test step to run the tests of your application. Additionally, ensure that both the ’repo’ and the ’image_name’ point to your integrations. + +Once you run the pipeline, the steps will create the coverage report and forwards it to Codacy. + +{% include image.html +lightbox="true" +file="/images/testing/codacy/codacy-pipeline.png" +url="/images/testing/codacy/codacy-pipeline.png" +alt="Pipeline with Codacy step" +max-width="80%" +%} + +## View reports + +You can view the updated coverage reports within Codacy's UI every time you make a commit and/or run the Codefresh pipeline directly. + +{% include image.html +lightbox="true" +file="/images/testing/codacy/codacy-report.png" +url="/images/testing/codacy/codacy-report.png" +alt="Codacy UI Analysis Dashboard" +max-width="80%" +%} + +You can access further information on the coverage report by opening the file tab and accessing a specific file from your repository. + +{% include image.html +lightbox="true" +file="/images/testing/codacy/file-analysis.png" +url="/images/testing/codacy/file-analysis.png" +alt="Codacy report details" +max-width="90%" +%} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Unit tests]({{site.baseurl}}/docs/testing/unit-tests/) +[Integration tests]({{site.baseurl}}/docs/testing/integration-tests/) +[Sonarqube Integration]({{site.baseurl}}/docs/testing/sonarqube-integration/) diff --git a/_docs/example-catalog/ci-examples/codecov-testing.md b/_docs/example-catalog/ci-examples/codecov-testing.md new file mode 100644 index 000000000..82e06f88a --- /dev/null +++ b/_docs/example-catalog/ci-examples/codecov-testing.md @@ -0,0 +1,128 @@ +--- +title: "Codecov coverage reports" +description: "How to forward coverage reports to Codecov" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +[Codecov account](https://codecov.io/){:target="\_blank"} is a code analysis tool with which users can group, merge, archive, and compare coverage reports. Code coverage describes which lines of code were executed by the test suite and which ones were not. However, this is not to be confused with a testing tool. + +Analysis reports displayed within the Codecov dashboard: +{% include image.html +lightbox="true" +file="/images/testing/codecov/analysis-report.png" +url="/images/testing/codecov/analysis-report.png" +alt="Codecov UI Analysis reports" +max-width="50%" +%} + +## Prerequisites for using Codecov + +* A simple [Codefresh pipeline up and running](https://codefresh.io/docs/docs/getting-started/create-a-codefresh-account/) +* A [Codecov account](https://codecov.io/){:target="\_blank"} (free or enterprise) +* A testing tool added to your project that produces coverage reports + +Note that reports should ideally be written in .json, .xml, or txt. To be sure, please double check that your coverage [report format](https://docs.codecov.io/docs/supported-report-formats){:target="\_blank"} is supported. You can find a variety of examples for different programming languages and suggestions for respective testing tools in the [Codecov docs](https://docs.codecov.io/docs/supported-languages){:target="\_blank"}. + +To test Codecov and follow along with the next section, you can clone our [Codecov sample app](https://github.com/codefresh-contrib/codecov-sample-app){:target="\_blank"}. + +## Create a Codecov account + +Once you sign up to Codecov, you can add a new repository. The UI will then provide you with an access token to the repository. While it is recommended that you take note of the token, you will still be able to access it within the **Settings** tap. + +{% include image.html +lightbox="true" +file="/images/testing/codecov/codecov-interface.png" +url="/images/testing/codecov/codecov-interface.png" +alt="Codecov Project Repository UI" +max-width="50%" +%} + +## Codefresh pipeline + +In this case, we divided testing and connecting Codefresh to Codecov into two different steps. If they can be run within the same image, you could also connect them. + +**Testing step** +Runs the command(s) for our testing tool. This will generate the code coverage report upon running the pipeline. Please refer to the Codecov documentation for [supported testing frameworks](https://docs.codecov.io/docs/supported-report-formats){:target="\_blank"}. The [README of each example](https://docs.codecov.io/docs/supported-languages){:target="\_blank"} refers to possible frameworks that can be used. + +In general, ensure that the framework you use for testing and generating code coverage reports: +* Produce code coverage reports in the supported file format +* Is compatible with the programming language that your program is written in + +{% highlight yaml %} +{% raw %} + test: + title: "Running test" + type: "freestyle" # Run any command + image: "node:14.19.0" # The image in which command will be executed + working_directory: "${{clone}}" # Running command where code cloned + commands: + - "npm install --save-dev jest" + - "npx jest --coverage" + stage: "test" +{% endraw %} +{% endhighlight %} + +**Codecov step** + +{% highlight yaml %} +{% raw %} +upload: + title: "Running test" + type: "freestyle" # Run any command + image: "node:14.19.0" # The image in which command will be executed + working_directory: "${{clone}}" # Running command where code cloned + commands: + - "ci_env=`curl -s https://codecov.io/env`" + - "npm install codecov -g" + - "codecov -t ${{CODECOV_TOKEN}} -f ./coverage/clover.xml" + stage: "upload" +{% endraw %} +{% endhighlight %} + +The commands run inside of the node Docker image: +* `ci_env= curl -s https://codecov.io/env`: Sets the CI environment variable to take note that we are using Codefresh +* `npm install codecov -g`: Installs the odecov CLI +* `codecov -t ${{CODECOV_TOKEN}} -f ./coverage/clover.xml`: Sets the Codevoc access token provided in the UI when we connect to a new Git repository and point to the file that contains our coverage report. + +Once you run the pipeline, the steps will create the coverage report and forward it to Codecov. + +{% include image.html +lightbox="true" +file="/images/testing/codecov/codecov-pipeline.png" +url="/images/testing/codecov/codecov-pipeline.png" +alt="Pipeline with codecov step" +max-width="50%" +%} + +## View reports + +You can view the updated coverage reports within the Codecov UI every time you make a commit and/or run the Codefresh pipeline directly. + +{% include image.html +lightbox="true" +file="/images/testing/codecov/codecov-report.png" +url="/images/testing/codecov/codecov-report.png" +alt="Pipeline with codecov step" +max-width="50%" +%} + +You can access further information on the coverage report by opening the link to the file displayed in the table. + +{% include image.html +lightbox="true" +file="/images/testing/codecov/codecov-report-details.png" +url="/images/testing/codecov/codecov-report-details.png" +alt="Codecov report details" +max-width="50%" +%} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Unit tests]({{site.baseurl}}/docs/testing/unit-tests/) +[Integration tests]({{site.baseurl}}/docs/testing/integration-tests/) +[Sonarqube Integration]({{site.baseurl}}/docs/testing/sonarqube-integration/) + diff --git a/_docs/example-catalog/ci-examples/coveralls-testing.md b/_docs/example-catalog/ci-examples/coveralls-testing.md new file mode 100644 index 000000000..dd060c20b --- /dev/null +++ b/_docs/example-catalog/ci-examples/coveralls-testing.md @@ -0,0 +1,221 @@ +--- +title: "Coveralls coverage reports" +description: "How to forward coverage reports to Coveralls" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +[Coveralls](https://coveralls.io/){:target="\_blank"} is a web service that allows users to track the code coverage of their application over time in order to optimize the effectiveness of their unit tests. This section details how coverage reports can be generated and forwarded to Coveralls with every Codefresh build. + +Analysis reports displayed within Coveralls dashboard: +{% include image.html +lightbox="true" +file="/images/testing/coveralls/coveralls-sample-app.png" +url="/images/testing/coveralls/coveralls-sample-app.png" +alt="Coveralls UI Analysis reports" +max-width="80%" +%} + +## Prerequisites for using Coveralls + +* A simple [Codefresh pipeline up and running](https://codefresh.io/docs/docs/getting-started/create-a-codefresh-account/) +* A [Coveralls account](https://coveralls.io/) (free or enterprise) -- Note that all open-source projects are free on Coveralls +* A testing tool added to your project that produces coverage reports + +Coveralls supports [22 different language integrations](https://docs.coveralls.io/about-coveralls){:target="\_blank"}. Each example provided in the official documentation suggests several coverage report tools that can be used in combination with Coveralls. + +You could try it out by cloning our [node example application](https://github.com/codefresh-contrib/coveralls-sample-app){:target="\_blank"} that utilises [jest](https://jestjs.io/){:target="\_blank"}. + +## Prepare your repository + +If you are using your own application as an example, you have to make a few modifications to the repository. Please have a look at the Coveralls example section for other languages. + +First, install Coveralls in your project: +{% highlight yaml %} +{% raw %} +npm install coveralls --save-dev +{% endraw %} +{% endhighlight %} + +Coveralls requires a [script](https://github.com/nickmerwin/node-coveralls){:target="\_blank"} that takes standard input and sends it to coveralls.io to report your code coverage. Depending on the framework that you are using, you will have to add a different script to your application. + +Any coverage reports can be forwarded that are within a [lcov data format](http://ltp.sourceforge.net/coverage/lcov/geninfo.1.php){:target="\_blank"} (including [mocha's LCOV reporter](https://www.npmjs.com/package/mocha-lcov-reporter){:target="\_blank"}). For this, we are going to set-up a “bin” folder, and within the folder a coveralls.js file that contains the following content: + +{% highlight yaml %} +{% raw %} +#!/usr/bin/env node + +'use strict'; + +const { handleInput } = require('..'); + +process.stdin.resume(); +process.stdin.setEncoding('utf8'); + +let input = ''; + +process.stdin.on('data', chunk => { + input += chunk; +}); + +process.stdin.on('end', () => { + handleInput(input, err => { + if (err) { + throw err; + } + }); +}); +{% endraw %} +{% endhighlight %} + +## Create a Coveralls account + +Once you sign-up to Coveralls, you can add a new repository. The UI will then provide you with an access token to the repository. Take note of the token since it will be required in the next sections. + +{% include image.html +lightbox="true" +file="/images/testing/coveralls/add-repository.png" +url="/images/testing/coveralls/add-repository.png" +alt="Coveralls repository" +max-width="80%" +%} + +## Codefresh pipeline + + +In case the project that you want to use Coveralls in does not have a pipeline, [create a new pipeline]({{site.baseurl}}/docs/getting-started/create-a-basic-pipeline/). + +{% include image.html +lightbox="true" +file="/images/testing/coveralls/create-coveralls-pipeline.png" +url="/images/testing/coveralls/create-coveralls-pipeline.png" +alt="Create Coveralls Pipeline" +max-width="80%" +%} + +Once you ’create’ the pipeline, a standard codefresh.yml file is generated with three steps: +* The first step will clone your repository; +* The second step will both, build and push your repository to the container registry that you have connected with Codefresh; +* And the third step currently does not do much. +In the next section, we will modify the testing step. + +**Testing step** + +The testing step requires three different environment variables to connect to Coveralls: +* `export COVERALLS_SERVICE_NAME="codefresh"` +* `export COVERALLS_GIT_BRANCH="insert the branch that you will be using with your application"` +* `export COVERALLS_REPO_TOKEN="insert the secret repo token from coveralls.io"` + +{% highlight yaml %} +{% raw %} + test: + title: "Running test" + type: "freestyle" # Run any command + image: "node:15.2" # The image in which command will be executed + working_directory: "${{clone}}" # Running command where code cloned + commands: + - "export COVERALLS_SERVICE_NAME=${{COVERALLS_SERVICE_NAME}}" + - "export COVERALLS_GIT_BRANCH=${{CF_BRANCH}}" + - "export COVERALLS_REPO_TOKEN=${{COVERALLS_REPO_TOKEN}}" + - "npm install --save-dev jest" + - "npm run test" + stage: "test" +{% endraw %} +{% endhighlight %} + +We specify several variables within this step. Those, which start with ’CF’ are [Codefresh-specific steps]({{site.baseurl}}/docs/pipelines/variables/) and the value is automatically provided by Codefresh once you run the pipeline. Our entire codefresh.yml will look as such: + +{% highlight yaml %} +{% raw %} +version: "1.0" +stages: + - "clone" + - "build" + - "test" + +steps: + clone: + title: "Cloning repository" + type: "git-clone" + repo: "anais-codefresh/coveralls-sample-app" + # CF_BRANCH value is auto set when pipeline is triggered + # Learn more at codefresh.io/docs/docs/pipelines/variables/ + revision: "${{CF_BRANCH}}" + git: "github" + stage: "clone" + + build: + title: "Building Docker image" + type: "build" + image_name: "anaisurlichs/coveralls-sample-app" + working_directory: "${{clone}}" + tag: "${{CF_BRANCH_TAG_NORMALIZED}}" + dockerfile: "Dockerfile" + stage: "build" + registry: "dockerhub" + + test: + title: "Running test" + type: "freestyle" # Run any command + image: "node:15.2" # The image in which command will be executed + working_directory: "${{clone}}" # Running command where code cloned + commands: + - "export COVERALLS_SERVICE_NAME=${{COVERALLS_SERVICE_NAME}}" + - "export COVERALLS_GIT_BRANCH=${{CF_BRANCH}}" + - "export COVERALLS_REPO_TOKEN=${{COVERALLS_REPO_TOKEN}}" + - "npm install --save-dev jest" + - "npm run test" + stage: "test" +{% endraw %} +{% endhighlight %} + +Once you run the pipeline the steps will create the coverage report and forward it to Coveralls. + +{% include image.html +lightbox="true" +file="/images/testing/coveralls/coveralls-pipeline.png" +url="/images/testing/coveralls/coveralls-pipeline.png" +alt="Pipeline with Coveralls step" +max-width="80%" +%} + +## View reports + +You can view the updated coverage reports within Coveralls UI every time you make a commit and/or run the Codefresh pipeline directly. + +{% include image.html +lightbox="true" +file="/images/testing/coveralls/coveralls-sample-app.png" +url="/images/testing/coveralls/coveralls-sample-app.png" +alt="Coveralls UI Analysis reports" +max-width="80%" +%} + +You can access further information on the coverage report by opening the link to the file displayed in the table. + +{% include image.html +lightbox="true" +file="/images/testing/coveralls/coveralls-specific-report.png" +url="/images/testing/coveralls/coveralls-specific-report.png" +alt="Coveralls report details" +max-width="80%" +%} + +And view a the code coverage of a specific file: +{% include image.html +lightbox="true" +file="/images/testing/coveralls/coveralls-coverage.png" +url="/images/testing/coveralls/coveralls-coverage.png" +alt="Coveralls report details" +max-width="80%" +%} + + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/#ci-examples) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Unit tests]({{site.baseurl}}/docs/testing/unit-tests/) +[Integration tests]({{site.baseurl}}/docs/testing/integration-tests/) +[Sonarqube Integration]({{site.baseurl}}/docs/testing/sonarqube-integration/) diff --git a/_docs/example-catalog/ci-examples/cpp-cmake.md b/_docs/example-catalog/ci-examples/cpp-cmake.md new file mode 100644 index 000000000..11f8e963a --- /dev/null +++ b/_docs/example-catalog/ci-examples/cpp-cmake.md @@ -0,0 +1,125 @@ +--- +title: "Compile and test a C++ application" +description: "Using Codefresh pipelines" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +Codefresh can work with any C/C++ application very easily as both `gcc` and `g++` are already offered in Dockerhub. There is also another example available with [C and make]({{site.baseurl}}/docs/example-catalog/ci-examples/c-make). + +## The example C++ project + +You can see the example project at [https://github.com/codefresh-contrib/cpp-sample-app](https://github.com/codefresh-contrib/cpp-sample-app){:target="\_blank"}. The repository contains a C++ starter project with a `CMakeLists.txt` file: + +* `cmake .` creates the makefiles. +* `make test` runs unit tests +* `make` compiles the code + +The project is also using the [boost testing libraries](https://www.boost.org/){:target="\_blank"}. + +## Cmake, g++ and Docker + +Creating a CI/CD pipeline for C is very easy, because Codefresh can run any [gcc image](https://hub.docker.com/_/gcc/){:target="\_blank"} that you wish. Gcc docker images already contain the `make` utility but not the the `cmake` one. Therefore we will first create a Dockerfile that has `g++`, cmake and the boost libraries. You can follow the same pattern for other development tools that you use. + + +Here is the Dockerfile: + + `Dockerfile` +{% highlight docker %} +{% raw %} +FROM gcc:9.2 + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && apt-get install -y cmake libgtest-dev libboost-test-dev && rm -rf /var/lib/apt/lists/* + +CMD ["cmake"] + +{% endraw %} +{% endhighlight %} + +This docker build does the following: + +1. Starts from the GCC image +1. Installs cmake and boost +1. Sets cmake as the default command + +## Create a CI pipeline for C++ applications + +We can now use the custom Docker image in order to compile/test the C++ application: + +{% include image.html +lightbox="true" +file="/images/learn-by-example/cc/cpp-cmake-pipeline.png" +url="/images/learn-by-example/cc/cpp-cmake-pipeline.png" +alt="Compiling a C++ application in a pipeline" +caption="Compiling a C++ application in a pipeline" +max-width="80%" +%} + +Here is the [full pipeline](https://github.com/codefresh-contrib/cpp-sample-app/blob/master/codefresh.yml){:target="\_blank"} that compiles the application after checking out the code. + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - checkout + - prepare + - build +steps: + main_clone: + title: Cloning main repository... + stage: checkout + type: git-clone + repo: 'codefresh-contrib/cpp-sample-app' + revision: master + git: github + build_dev_image: + title: Building Dev Image + stage: prepare + type: build + image_name: cmake + working_directory: ./dev/ + tag: 'latest' + dockerfile: Dockerfile + create_makefiles: + title: Create Makefiles + stage: prepare + image: ${{build_dev_image}} + commands: + - cmake . + compile_my_sources: + title: Compile + stage: build + image: ${{build_dev_image}} + commands: + - make + run_my_tests: + title: Test + stage: build + image: ${{build_dev_image}} + commands: + - make test +{% endraw %} +{% endhighlight %} + +This pipeline: + +1. clones the source code +1. Creates a development docker image that has g++, cmake and boost +1. Runs cmake on the source code to create the make files +1. Compiles the source code +1. Runs unit tests + +You can add additional tools in the pipeline by extending the Dockerfile mentioned in the previous section. You can also +change the version of Gcc/g++ by starting from a different public or private Docker image. + + +## Related articles +[C example]({{site.baseurl}}/docs/example-catalog/ci-examples/c-make/) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in pipelines]({{site.baseurl}}/docs/codefresh-yaml/steps/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[How Codefresh pipelines work]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) \ No newline at end of file diff --git a/_docs/example-catalog/ci-examples/decryption-with-mozilla-sops.md b/_docs/example-catalog/ci-examples/decryption-with-mozilla-sops.md new file mode 100644 index 000000000..d091ca245 --- /dev/null +++ b/_docs/example-catalog/ci-examples/decryption-with-mozilla-sops.md @@ -0,0 +1,177 @@ +--- +title: "Decrypt with Mozilla SOPS" +description: "Store secrets in your repository and decrypt them using Mozilla SOPS" +group: example-catalog +sub_group: ci-examples +toc: true +--- + +## Prerequisites + +- A [free Codefresh account]({{site.baseurl}}/docs/administration/create-a-codefresh-account/) +- A public and private GnuGP key pair +- A credentials yaml, that is encrypted using Mozilla SOPS, and stored in your repository + +## Example Java application + +You can find the example project on [GitHub](https://github.com/codefresh-contrib/mozilla-sops-app){:target="\_blank"}. + +The example application retrieves the system variable "password," from the pipeline and uses it to authenticate to a Redis database, but you are free to use any type of database of your choosing. + +```java + String password = System.getenv("password"); + String host = System.getProperty("server.host"); + + RedisClient redisClient = new RedisClient( + RedisURI.create("redis://" + password + "@" + host + ":6379")); + RedisConnection : ' + VAULT_PATH: 'path/to/secret' + VAULT_AUTH_TOKEN: ' ' + clone: + title: Cloning main repository... + type: git-clone + arguments: + repo: 'codefresh-contrib/vault-sample-app' + git: github + stage: clone + package_jar: + title: Packaging jar and running unit tests... + stage: package + working_directory: ${{clone}} + arguments: + image: maven:3.5.2-jdk-8-alpine + commands: + - mvn -Dmaven.repo.local=/codefresh/volume/m2_repository -Dserver.host=my-redis-db-host clean package + services: + composition: + my-redis-db-host: + image: 'redis:4-alpine' + command: 'redis-server --requirepass $password' + ports: + - 6379 +``` + +The pipeline does the following: + +1. Imports the key-value pairs from the Vault server and exports them into the pipeline under `/meta/env_vars_to_export`. +2. Clones the main repository (note the special use of naming the step `main_clone`). This ensures that all subsequent commands are run [inside the project that was checked out]({{site.baseurl}}/docs/pipelines/steps/git-clone/#basic-clone-step-project-based-pipeline). +3. The `package_jar`, does a few special things to take note of: + - Spins up a [Service Container]({{site.baseurl}}/docs/pipelines/service-containers/) running Redis on port 6379 , and sets the password to the database using our exported environment variable + - Sets `maven.repo.local` to cache Maven dependencies into the local codefresh volume to [speed up builds]({{site.baseurl}}/docs/example-catalog/ci-examples/spring-boot-2/#caching-the-maven-dependencies) + - Runs unit tests and packages the jar. Note how you can directly refer to the service container's name (`my-redis-db-host`) when we set `server.host` + +You will see that the variable was correctly exported to the pipeline by running a simple `echo` command: + {% include image.html + lightbox="true" + file="/images/examples/secrets/vault-pipeline2.png" + url="/images/examples/secrets/vault-pipeline2.png" + alt="Vault pipeline variable" + caption="Vault pipeline variable" + max-width="100%" + %} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/ci-examples/) +[Steps in pipelines]({{site.baseurl}}/docs/pipelines/steps/) + diff --git a/_docs/example-catalog/ci-examples/voting-app.md b/_docs/example-catalog/ci-examples/voting-app.md new file mode 100644 index 000000000..08cb4a5cf --- /dev/null +++ b/_docs/example-catalog/ci-examples/voting-app.md @@ -0,0 +1,93 @@ +--- +title: "Voting app" +description: "" +excerpt: "" +group: example-catalog +sub_group: ci-examples +redirect_from: + - /docs/voting-app-1/ + - /docs/python/voting-app/ +toc: true +--- +This voting application is a demo with which you can build an advanced composition that uses `Python, Redis, Postgres, Node.js, and .Net`. + +## Looking around +In the root of this repository you'll find a file named codefresh.yml, this is our build descriptor and it describes the different steps that comprise our process. Let's quickly review the contents of this file: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + unit-tests: + image: codefresh/buildpacks:nodejs-5 + working-directory : ${{initial-clone}} + commands: + - echo Installing npm modules silent + - npm install + - gulp test + - echo $(date) + + build-step: + #title: Build My Image #Display name for the step + type: build + image-name: containers101/cf-example-result + tag: ${{CF_BRANCH}} + build_arguments: + - OPTION_A=${{OPTION_A}} + - OPTION_B=${{OPTION_B}} + + push-to-registry: + type: push + #candidate: the image from the build step + candidate: ${{build-step}} + tag: ${{CF_BRANCH}} + + integration-tests-step: + type: composition + #location of the compostion on the filesystem of the cloned image + composition: './cf-compositions/voting-app-full.yml' + #run integration only when pushing to master + when: + branch: + only: + - master #can also be regex + composition-candidates: + #this will be the image that we will test + integ-test: + image: containers101/cf-example-tests:master + command: ./tests.sh + composition-variables: + - VOTING_OPTION_A=${{OPTION_A}} + - VOTING_OPTION_B=${{OPTION_B}} + + launch-composition: + type: launch-composition + environment-name: 'Test composition after build' + composition: './cf-compositions/voting-app-full.yml' + composition-variables: + - VOTING_OPTION_A=${{OPTION_A}} + - VOTING_OPTION_B=${{OPTION_B}} + + deploy to ecs: + image: codefresh/cf-deploy-ecs + commands: + - cfecs-update --image-name containers101/cf-example-result --image-tag ${{CF_BRANCH}} eu-west-1 vote-app result + environment: + - AWS_ACCESS_KEY_ID=${{AWS_ACCESS_KEY_ID}} + - AWS_SECRET_ACCESS_KEY=${{AWS_SECRET_ACCESS_KEY}} + when: + condition: + all: + pushCommit: 'includes(lower("${{CF_COMMIT_MESSAGE}}"), "[deploy]") == true' +{% endraw %} +{% endhighlight %} + +{{site.data.callout.callout_info}} +##### Example + +Just head over to the example [__repository__](https://github.com/containers101/cf-example-result){:target="_blank"} in GitHub and follow the instructions there. +{{site.data.callout.end}} + +## Related articles +[CI/CD pipeline examples]({{site.baseurl}}/docs/example-catalog/ci-examples/) diff --git a/_docs/example-catalog/examples.md b/_docs/example-catalog/examples.md new file mode 100644 index 000000000..9dee45256 --- /dev/null +++ b/_docs/example-catalog/examples.md @@ -0,0 +1,127 @@ +--- +title: "CI/CD pipeline examples" +description: "A collection of examples for Codefresh pipelines" +group: example-catalog +redirect_from: + - /docs/examples-v01/ + - examples.html + - /docs/catalog-examples/ + - /docs/examples/ + - /docs/pipelines-examples/ + - /docs/pipelines/pipelines-examples/ +toc: true +--- +Codefresh enables you to define the steps of your pipeline in a [YAML file]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/). By default, the file is named `codefresh.yml`, and is located in the root directory of the repository. + +## CI examples + +### Programming-language specific examples + +Codefresh is agnostic as far as programming languages are concerned. All major programming languages are supported: + +- [Go Web App]({{site.baseurl}}/docs/example-catalog/ci-examples/golang-hello-world/) or [Go CLI]({{site.baseurl}}/docs/example-catalog/golang/goreleaser) +- [Spring Java app with Maven]({{site.baseurl}}/docs/example-catalog/ci-examples/spring-boot-2/) or [Gradle]({{site.baseurl}}/docs/example-catalog/ci-examples/gradle/). Also how to [upload JAR to Nexus/Artifactory]({{site.baseurl}}/docs/example-catalog/ci-examples/publish-jar/) +- Node [Express.js App]({{site.baseurl}}/docs/example-catalog/ci-examples/lets-chat/) or [React.js App]({{site.baseurl}}/docs/example-catalog/ci-examples/react/) +- [Php App]({{site.baseurl}}/docs/example-catalog/ci-examples/php) +- [Python Django App]({{site.baseurl}}/docs/example-catalog/ci-examples/django/) +- [Ruby On Rails App]({{site.baseurl}}/docs/example-catalog/ci-examples/ruby) +- [C]({{site.baseurl}}/docs/example-catalog/ci-examples/c-make/) or [C++]({{site.baseurl}}/docs/example-catalog/ci-examples/cpp-cmake) +- [Rust]({{site.baseurl}}/docs/example-catalog/ci-examples/rust/) +- [C# .NET core]({{site.baseurl}}/docs/example-catalog/ci-examples/dotnet/) +- [Scala App]({{site.baseurl}}/docs/example-catalog/ci-examples/scala-hello-world/) +- [Android (Mobile)]({{site.baseurl}}/docs/example-catalog/ci-examples/android/) + +### Source code checkout examples + +You can check out code from one or more repositories in any pipeline phase. Codefresh includes [built-in GIT integration]({{site.baseurl}}/docs/integrations/git-providers/) with all the popular GIT providers and can be used with [git-clone]({{site.baseurl}}/docs/pipelines/steps/git-clone/) steps. + +- [Cloning Git repositories using the built-in integration]({{site.baseurl}}/docs/example-catalog/ci-examples/git-checkout/) +- [Cloning Git repositories using manual Git commands]({{site.baseurl}}/docs/example-catalog/ci-examples/git-checkout-custom/) +- [Checking out from Subversion, Perforce, Mercurial, etc ]({{site.baseurl}}/docs/example-catalog/ci-examples/non-git-checkout/) + +### Build/push examples + +Codefresh has native support for [building]({{site.baseurl}}/docs/pipelines/steps/build/) and [pushing]({{site.baseurl}}/docs/pipelines/steps/push/) Docker containers. +You can also compile traditional applications that are not Dockerized yet. + +- [Build an Image with the Dockerfile in root directory]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-dockerfile-in-root-directory/) +- [Build an Image by specifying the Dockerfile location]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-specify-dockerfile-location) +- [Build an Image from a different Git repository]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-from-a-different-git-repository) +- [Build and Push an Image]({{site.baseurl}}/docs/example-catalog/ci-examples/build-and-push-an-image) +- [Build an Image with build arguments]({{site.baseurl}}/docs/example-catalog/ci-examples/build-an-image-with-build-arguments) +- [Share data between steps]({{site.baseurl}}/docs/example-catalog/ci-examples/shared-volumes-between-builds) +- [Upload or download from a Google Storage Bucket]({{site.baseurl}}/docs/example-catalog/ci-examples/uploading-or-downloading-from-gs/) +- [Get Short SHA ID and use it in a CI process]({{site.baseurl}}/docs/example-catalog/ci-examples/get-short-sha-id-and-use-it-in-a-ci-process) +- [Call a CD pipeline from a CI pipeline]({{site.baseurl}}/docs/example-catalog/ci-examples/call-child-pipelines) +- [Trigger a Kubernetes Deployment from a Dockerhub Push Event]({{site.baseurl}}/docs/example-catalog/ci-examples/trigger-a-k8s-deployment-from-docker-registry/) + + +### Unit and integration test examples + +Codefresh has support for both [unit]({{site.baseurl}}/docs/testing/unit-tests/) and [integration tests]({{site.baseurl}}/docs/testing/integration-tests/) as well as [test reporting]({{site.baseurl}}/docs/testing/test-reports/). + +- [Run unit tests]({{site.baseurl}}/docs/example-catalog/ci-examples/run-unit-tests) +- [Run integration tests]({{site.baseurl}}/docs/example-catalog/ci-examples/run-integration-tests/) +- [Run integration tests with MongoDB]({{site.baseurl}}/docs/example-catalog/ci-examples/integration-tests-with-mongo/) +- [Run integration tests with MySQL]({{site.baseurl}}/docs/example-catalog/ci-examples/integration-tests-with-mysql/) +- [Run integration tests with PostgreSQL]({{site.baseurl}}/docs/example-catalog/ci-examples/integration-tests-with-postgres/) +- [Run integration tests with Redis]({{site.baseurl}}/docs/example-catalog/ci-examples/integration-tests-with-redis/) +- [Populate a database with existing data]({{site.baseurl}}/docs/example-catalog/populate-a-database-with-existing-data) + +- [Shared volumes of service from composition step for other yml steps]({{site.baseurl}}/docs/example-catalog/shared-volumes-of-service-from-composition-step-for-other-yml-steps) +- [Launch Composition]({{site.baseurl}}/docs/example-catalog/ci-examples/launch-composition) +- [Launch Composition and define Service Environment variables using a file]({{site.baseurl}}/docs/example-catalog/ci-examples/launching-a-composition-and-defining-a-service-environment-variables-using-a-file) +- [Run multiple kinds of unit tests using fan-in-fan-out parallel pipeline]({{site.baseurl}}/docs/example-catalog/fan-in-fan-out) + +### Code coverage examples + +- [Run coverage reports with Codecov]({{site.baseurl}}/docs/example-catalog/ci-examples/codecov-testing) +- [Run coverage reports with Coveralls]({{site.baseurl}}/docs/example-catalog/ci-examples/coveralls-testing) +- [Run coverage reports with Codacy]({{site.baseurl}}/docs/example-catalog/ci-examples/codacy-testing) + +### Secrets examples + +Codefresh can automatically export secret key-value pairs using the Vault plugin from the [Step Marketplace](https://codefresh.io/steps/step/vault). + +- [Vault secrets in the Pipeline]({{site.baseurl}}/docs/example-catalog/ci-examples/vault-secrets-in-the-pipeline) +- [Decryption with Mozilla SOPS]({{site.baseurl}}/docs/example-catalog/ci-examples/ci-examples/decryption-with-mozilla-sops) +- [GitOps with Bitnami sealed secrets]({{site.baseurl}}/docs/example-catalog/ci-examples/gitops-secrets) + +### Notification examples + +- [Send notification to Slack]({{site.baseurl}}/docs/example-catalog/ci-examples/sending-the-notification-to-slack) +- [Send notification to Jira]({{site.baseurl}}/docs/example-catalog/ci-examples/sending-the-notification-to-jira) + + +## CD examples + +### Preview environment examples + +Codefresh can automatically launch environments (powered by Docker swarm) to [preview a Pull Reqest or feature]({{site.baseurl}}/docs/getting-started/on-demand-environments/). The definition of the environment can come from an [existing composition]({{site.baseurl}}/docs/testing/create-composition/), a docker-compose file or an inline YAML. Preview environments can be launched manually or [automatically from pipelines]({{site.baseurl}}/docs/pipelines/steps/launch-composition/). + +- [MongoDB preload data]({{site.baseurl}}/docs/example-catalog/cd-examples/import-data-to-mongodb/) +- [NodeJS + Angular2 + MongoDB]({{site.baseurl}}/docs/example-catalog/cd-examples/nodejs-angular2-mongodb/) +- [NGINX Basic Auth]({{site.baseurl}}/docs/example-catalog/cd-examples/secure-a-docker-container-using-http-basic-auth/) +- [Spring Boot + Kafka + Zookeeper]({{site.baseurl}}/docs/example-catalog/cd-examples/spring-boot-kafka-zookeeper/) +- [Web terminal]({{site.baseurl}}/docs/example-catalog/cd-examples/web-terminal/) + +### Deployment examples + +Codefresh can deploy to any platform such as VMs, FTP/SSH/S3 sites, app servers, but of course it has great support for [Kubernetes clusters]({{site.baseurl}}/docs/deploy-to-kubernetes/deployment-options-to-kubernetes/) and [Helm releases]({{site.baseurl}}/docs/new-helm/helm-releases-management/): + +- [Deploy to a VM with packer]({{site.baseurl}}/docs/example-catalog/cd-examples/packer-gcloud/) +- [Deploy to a VM with FTP]({{site.baseurl}}/docs/example-catalog/cd-examples/transferring-php-ftp) +- [Deploy to Tomcat using SCP]({{site.baseurl}}/docs/example-catalog/cd-examples/deploy-to-tomcat-via-scp) +- [Deploy Demochat to a Kubernetes cluster]({{site.baseurl}}/docs/cd-examples/deploy-to-kubernetes/codefresh-kubernetes-integration-demochat-example/) +- [Use kubectl as part of freestyle step]({{site.baseurl}}/docs/example-catalog/cd-examples/use-kubectl-as-part-of-freestyle-step) +- [Deploy with Kustomize]({{site.baseurl}}/docs/example-catalog/cd-examples/deploy-with-kustomize) +- [Deploy with Helm]({{site.baseurl}}/docs/example-catalog/cd-examples/helm) +- [Deploy with Terraform]({{site.baseurl}}/docs/example-catalog/cd-examples/terraform) +- [Deploy with Pulumi]({{site.baseurl}}/docs/example-catalog/cd-examples/pulumi) +- [Deploy to Nomad]({{site.baseurl}}/docs/example-catalog/cd-examples/nomad) +- [Deploy to Heroku]({{site.baseurl}}/docs/example-catalog/cd-examples/deploy-to-heroku/) +- [Deploy to Docker swarm]({{site.baseurl}}/docs/example-catalog/cd-examples/docker-swarm/) +- [Deploy to Elastic Beanstalk]({{site.baseurl}}/docs/example-catalog/cd-examples/elastic-beanstalk/) +- [Deploy to Amazon ECS/Fargate]({{site.baseurl}}/docs/example-catalog/cd-examples/amazon-ecs/) + + diff --git a/_docs/example-catalog/gitops-example.md b/_docs/example-catalog/gitops-example.md new file mode 100644 index 000000000..ba1c727d1 --- /dev/null +++ b/_docs/example-catalog/gitops-example.md @@ -0,0 +1,9 @@ +--- +title: "GitOps examples" +description: "A collection of examples for GitOps deployments" +group: example-catalog +sub_group: gitops-examples +toc: true +--- + +TBD \ No newline at end of file diff --git a/_docs/getting-started/architecture.md b/_docs/getting-started/architecture.md deleted file mode 100644 index 584fd5070..000000000 --- a/_docs/getting-started/architecture.md +++ /dev/null @@ -1,199 +0,0 @@ ---- -title: "Architecture" -description: "" -group: getting-started -toc: true ---- - -Codefresh GitOps is built around an enterprise version of the Argo Ecosystem, fully compliant with the GitOps paradigm, with industry-standard security. -To cater to differing requirements and degrees of enterprise security, Codefresh supports hosted and hybrid installation environments for Codefresh runtimes. - -The sections that follow illustrate the architecture of the different installation environments, starting with a high-level overview of the Codefresh Platform. - -### Codefresh architecture - -The diagram shows a high-level view of the Codefresh Platform and its core components, the Codefresh Control Plane, the Codefresh Runtime, and the Codefresh Clients. - -{% include - image.html - lightbox="true" - file="/images/getting-started/architecture/arch-codefresh-simple.png" - url="/images/getting-started/architecture/arch-codefresh-simple.png" - alt="Codefresh Platform architecture" - caption="Codefresh Platform architecture" - max-width="100%" -%} - -{::nomarkdown} -
-{:/} - -#### Codefresh Control Plane -The Codefresh Control Plane is the SaaS component in the platform. External to the enterprise firewall, it does not have direct communication with the Codefresh Runtime, Codefresh Clients, or the customer's organizational systems. The Codefresh Runtime and the Codefresh Clients communicate with the Codefresh Control Plane to retrieve the required information. - - -{::nomarkdown} -
-{:/} - -#### Codefresh Runtime -The Codefresh Runtime is installed on a Kubernetes cluster, and houses the enterprise distribution of the Codefresh Application Proxy and the Argo Project. -Depending on the type of installation environment, the Codefresh Runtime is installed either in the Codefresh platform (hosted), or in the customer environment (hybrid). Read more in [Codefresh runtime architecture](#codefresh-runtime-architecture). - - -{::nomarkdown} -
-{:/} - -#### Codefresh Clients - -Codefresh Clients include the Codefresh UI and the Codefresh CLI. -The Codefresh UI provides a unified, enterprise-wide view of deployments (runtimes and clusters), and CI/CD operations (Delivery Pipelines, workflows, and deployments) in the same location. -The Codefresh CLI includes commands to install hybrid runtimes, add external clusters, and manage runtimes and clusters. - -### Codefresh runtime architecture -The sections that follow show detailed views of runtime architecture in the different installation environments, and descriptions of the Codefresh Runtime components. - -* [Hosted GitOps runtime architecture](#hosted-gitops-runtime-architecture) - In this installation environment, the Codefresh Runtime is installed on a _Codefresh-managed cluster_ in the Codefresh platform. -* Hybrid runtime architecture: - In this installation environment, the Codefresh Runtime is installed on a _customer-managed cluster_ in the customer environment. The Codefresh Runtime with or without ingress controllers: - * [Ingress controller](#ingress-controller-hybrid-runtime-architecture) - * [Ingress-less](#ingress-less-hybrid-runtime-architecture) -* Runtime components - * [Codefresh Application Proxy](#codefresh-application-proxy) - * [Argo Project](#argo-project) - * [Request Routing Service](#request-routing-service) - * [Tunnel Server](#codefresh-tunnel-server) - * [Tunnel Client](#codefresh-tunnel-client) - - -#### Hosted GitOps runtime architecture -In the hosted environment, the Codefresh Runtime is installed on a K8s cluster managed by Codefresh. - -{% include - image.html - lightbox="true" - file="/images/getting-started/architecture/arch-hosted.png" - url="/images/getting-started/architecture/arch-hosted.png" - alt="Hosted runtime architecture" - caption="Hosted runtime architecture" - max-width="100%" -%} - -#### Ingress controller hybrid runtime architecture -Runtimes with ingress use an ingress controller to control communication between the Codefresh Runtime in the customer cluster and the Codefresh Platform. Ingress controllers are optimal when the cluster with the Codefresh Runtime is exposed to the internet. - - - -{% include - image.html - lightbox="true" - file="/images/getting-started/architecture/arch-hybrid-ingress.png" - url="/images/getting-started/architecture/arch-hybrid-ingress.png" - alt="Ingress-based hybrid runtime architecture" - caption="Ingress-based hybrid runtime architecture" - max-width="100%" -%} - -#### Ingress-less hybrid runtime architecture -Ingress-less runtimes uses tunneling to control communication between the Codefresh Runtime in the customer cluster and the Codefresh Platform. Ingress-less runtimes are optimal when the cluster with the Codefresh Runtime is not exposed to the internet. - -{% include - image.html - lightbox="true" - file="/images/getting-started/architecture/arch-hybrid-ingressless.png" - url="/images/getting-started/architecture/arch-hybrid-ingressless.png" - alt="Ingress-less hybrid runtime architecture" - caption="Ingress-less hybrid runtime architecture" - max-width="100%" -%} - - - -#### Codefresh Application Proxy -The Codefresh Application Proxy (App-Proxy) functions as the Codefresh agent, and is deployed as a service in the Codefresh Runtime. -For hybrid runtimes with ingress, the App-Proxy is the single point-of-contact between the Codefresh Runtime, and the Codefresh Clients, the Codefresh Platform, and any organizational systems in the customer environment. -For ingress-less hybrid runtimes, the Tunnel Client forwards the incoming traffic from the Tunnel Server using internal reverse proxy to the App-Proxy. - -The App-Proxy: -* Accepts and serves requests from Codefresh Clients either via the Codefresh UI or CLI -* Retrieves a list of Git repositories for visualization in Codefresh Clients -* Retrieves permissions from the Codefresh Control Plane to authenticate and authorize users for the required operations. -* Implements commits for GitOps-controlled entities, such as Delivery Pipelines and other CI resources -* Implements state-change operations for non-GitOps controlled entities, such as terminating Argo Workflows - -{::nomarkdown} -
-{:/} - -#### Argo Project - -The Argo Project includes: -* Argo CD for declarative continuous deployment -* Argo Rollouts for progressive delivery -* Argo Workflows as the workflow engine -* Argo Events for event-driven workflow automation framework - - -{::nomarkdown} -
-{:/} - -#### Request Routing Service -The Request Routing Service is installed on the same cluster as the Codefresh Runtime in the customer environment. -It receives requests from the ingress controller (ingress) or the Tunnel Client (ingress-less), and forwards the request URLs to the Application Proxy, and webhooks directly to the Event Sources. - ->Important: - The Request Routing Service is available from runtime version 0.0.543 and higher. - Older runtime versions are not affected as there is complete backward compatibility, and the ingress controller continues to route incoming requests. - -#### Tunnel Server -Applies only to _ingress-less_ runtimes in hybrid installation environments. -The Codefresh Tunnel Server is installed in the Codefresh platform. It communicates with the enterprise cluster located behind a NAT or firewall. - -The Tunnel Server: -* Forwards traffic from Codefresh Clients to the client (customer) cluster. -* Manages the lifecycle of the Codefresh Tunnel Client. -* Authenticates requests from the Codefresh Tunnel Client to open tunneling connections. - -{::nomarkdown} -
-{:/} - -#### Tunnel Client -Applies only to _ingress-less_ runtimes in hybrid installation environments. - -Installed on the same cluster as the Codefresh Runtime, the Codefresh Tunnel Client establishes the tunneling connection to the Codefresh Tunnel Server via the WebSocket Secure (WSS) protocol. -A single Codefresh Runtime can have a single Tunnel Client. - -The Codefresh Tunnel Client: -* Initiates the connection with the Codefresh Tunnel Server. -* Forwards the incoming traffic from the Tunnel Server through the Request Routing Service to App-Proxy, and other services. - -{::nomarkdown} -
-{:/} - - -### Customer environment -The customer environment that communicates with the Codefresh Runtime and the Codefresh Platform, generally includes: -* Ingress controller for ingress hybrid runtimes - The ingress controller is configured on the same Kubernetes cluster as the Codefresh Runtime, and implements the ingress traffic rules for the Codefresh Runtime. - See [Ingress controller requirements]({{site.baseurl}}/docs/runtime/requirements/#ingress-controller). -* Managed clusters - Managed clusters are external clusters registered to provisioned hosted or hybrid runtimes for application deployment. - Hosted runtimes requires you to connect at least one external K8s cluster as part of setting up the Hosted GitOps environment. - Hybrid runtimes allow you to add external clusters after provisioning the runtimes. - See [Add external clusters to runtimes]({{site.baseurl}}/docs/runtime/managed-cluster/). -* Organizational systems - Organizational Systems include the customer's tracking, monitoring, notification, container registries, Git providers, and other systems. They can be entirely on-premises or in the public cloud. - Either the ingress controller (ingress hybrid environments), or the Tunnel Client (ingress-less hybrid environments), forwards incoming events to the Codefresh Application Proxy. - -### Related articles -[Set up a hosted runtime environment]({{site.baseurl}}/docs/runtime/hosted-runtime/) -[Install a hybrid runtime]({{site.baseurl}}/docs/runtime/installation/) - - - - diff --git a/_docs/getting-started/cd-codefresh.md b/_docs/getting-started/cd-codefresh.md new file mode 100644 index 000000000..2dc6a70fc --- /dev/null +++ b/_docs/getting-started/cd-codefresh.md @@ -0,0 +1,91 @@ +--- +title: "Codefresh for CD" +description: "Continuous deployment (CD) with Codefresh pipelines" +group: getting-started +toc: true +--- + +Work in progress TBD + + + + +## Connecting to Kubernetes +Continuous deployment starts with Kubernetes clusters, and Codefresh integrates with any known cluster provider for Kubernetes through a few simple steps. Connect your Google, Azure, Amazon Kubernetes cluster to Codefresh through simple integration steps. +For those Kubernetes clusters that are not in our list of cluster providers, you can manually enter your cluster settings to add any generic Kubernetes cluster. + +See [Connecting a Kubernetes cluster]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster). + +## Deploying to Kubernetes +Codefresh offers a variety of options for you to choose from when deploying to Kubernetes. +Deploy to Kubernetes from the Codefresh UI, or programmatically through dedicated steps in pipelines, avoiding the need for `kubectl` commands. + +**On-demand deployment** +For quick and easy deployment, deploy on-demand from the Codefresh UI. + +**Dedicated steps in pipelines** +We have the `deploy` step, and the more advanced `cf-deploy-kubernetes`step that enables simple templating on Kubernetes manifests. + +Codefresh pipelines also support Kustomize and Helm for deployments through freestyle steps. + +Finally, if you are familiar with and want to work with `kubectl`, run your own custom `kubectl` commands in a freestyle step. Read more in [kubectl](#kubectl). + +See [Deployment options for Kubernetes]({{site.baseurl}}/docs/deployments/kubernetes/deployment-options-to-kubernetes/). + +## kubectl +`kubectl` is the command line interface for managing kubernetes clusters. Running custom `kubectl` commands in a freestyle step gives your maximum flexibility with cluster deployments. +Codefresh automatically sets up your config context with your connected clusters. The config context is the value of the `$CF_KUBECONFIG_PATH` variable, which expands to `/codefresh/volume/sensitive/.kube/config` within the shared step volume. + +Codefresh has a public Docker image for kubectl at [Docker Hub](https://hub.docker.com/r/codefresh/kubectl/tags){:target="\_blank"} that you can use. + +Because Codefresh automatically sets up your `kubeconfig` files with the information from your cluster integrations, you can modify the current config context and run any `kubectl` command you want applied to that context. For example, leverage the parallel capability of Codefresh pipelines to create two Docker images and deploy them to two different clusters with custom `kubectl` commands. + +See [Running custom kubectl commands]({{site.baseurl}}/docs/deployments/kubernetes/custom-kubectl-commands/). + +## Helm and Codefresh +Codefresh supplies a built-in Helm repository with every Codefresh account. And supports, besides public HTTP repositories, several private, authenticated Helm repositories. + +**Connect to Helm repositories** +In addition to the official Helm repositories which are displayed in the Helm charts, Codefresh allows you connect with any external Helm repository through simple integrations. You can then inject the Helm repository context into your pipelines by selecting the repository name. + +**Build Helm charts** +Install Helm charts from Helm repositories, or build a new one. + + + +Deploy Helm charts +Deploy the Helm chart to a Kubernetes cluster, Helm repo, or both. + + +## Dashboards + +TBD + \ No newline at end of file diff --git a/_docs/getting-started/ci-codefresh.md b/_docs/getting-started/ci-codefresh.md new file mode 100644 index 000000000..6d0015b27 --- /dev/null +++ b/_docs/getting-started/ci-codefresh.md @@ -0,0 +1,98 @@ +--- +title: "Codefresh for CI" +description: "Continuous integration (CI) with Codefresh pipelines" +group: getting-started +toc: true +--- + + + +Work in progress + + +## Docker images +WBuilding a Docker image from the source code is probably the most common and basic requirement for a CI pipeline. In Codefresh you can build, push, and promote Docker images, using declarative YAML and credentials that are defined once stored centrally. + +**Build and push image** +Building a Dockerfile in a pipeline works in the same way as building the Dockerfile locally on your workstation. The `build` step in Codefresh enables you to build a Docker image in a completely declarative manner, and to automatically push it to your default Docker registry without any configuration. + +See: +[Build and push Docker images]({{site.baseurl}}/docs/example-catalog/ci-examples/build-and-push-an-image/) + + +**View image** +The Images dashboard displays images from all registries connected to Codefresh. Every image is enriched with Git branch, Git hash and commit message, and any tags defined for the image. + +See: +[Viewing Docker images]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/#viewing-docker-images) + + + +**Promote image** +Promote an image by copying it from one registry to another. You can promote images either from the Codefresh UI, or automatically from pipelines by specifying an existing image in the pipeline step. + +See: +[Promoting Docker images]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/#viewing-docker-images) + + +## Code compilation +TBD + +## Unit testing +Codefresh supports all testing frameworks, including mocking frameworks, for all popular programming languages. Easily run unit tests on the source code of the application for every commit or pull request (PR) through our freestyle step in pipelines. + +Run any type of unit tests in Codefresh pipelines, from smoke tests in a dockerfile, to tests with external or application images for simple applications, and evenrun them on a special testing image for complex applications. +You can create test reports and view them whenever you need. + +See: +[Run unit tests example]({{site.baseurl}}/docs/example-catalog/ci-examples/run-unit-tests/) + + +## Integration testing +Compared to unit tests that run on the source code, integration tests run on the application itself. You need to either launch the application itself, or one or more external services such as a database. +In Codefresh, you can launch these sidecar containers within the pipeline through compositions and service containers. + + +See: +[Run integration tests example]({{site.baseurl}}/docs/example-catalog/ci-examples/run-integrations-tests/) + +## Security scanning +Security scans are critical to deploying quality code. With Codefresh, in addition you can control when to implement the security scan, and then view the scan results in the Codefresh UI, without having to go to the security platform. + +**Security scan platforms** +Codefresh can integrate with any security scanning platform that scans source code or Docker images for vulnerabilities. We already have ready-to-use Docker images for several security platforms such as Anchore, Aqua Security, Clair, Twistlock and WhiteSource. For the full list, visit our [Plug-ins library](https://codefresh.io/steps/){:target="\_blank"}. + +**Scan timing in pipeline step** +The security scan is implemented through a freestyle step, inserted anywhere in the pipeline. The fact that you can insert the step anywhere allows you to control when the scan is executed, for example, before the source code is packaged in a container, or before the container is stored in a registry or deployed to production, or any combination of these. + +**View scan results** +As with any scan, the final step is viewing the scan results. Make the scan results available in Codefresh release dashboards (Test Report button) by attaching analysis reports to the pipeline build. + +**Security annotations** +Correlate the Docker images in Codefresh with the results of the security scanning platform by adding annotations for custom metatdata. For example, you can add annotations such as the number of issues or the URL of the full report. + +See: +[Security scanning tests]({{site.baseurl}}/docs/testing/security-scanning/) +[Test reporting modes]({{site.baseurl}}/docs/testing/test-reports/) +[Metadata in Docker images]({{site.baseurl}}/docs/pipelines//docker-image-metadata/) + + + +## Code quality coverage +Good quality code is central to any CI platform or tool. Codefresh integrates with the top code quality platforms/tools in the market to track code coverage, inspect code quailty, and generate code-coverage analysis reports. + +Three steps to +* Set up integrations with the platforms/tools (Coverall, SonarQube, Codecov, for example). +* Copy and paste the ready-to-use step for your platform/tool into your pipeline from our [Plug-ins library](https://codefresh.io/steps/){:target="\_blank"}. +* Reference them by name in the pipeline step, and view the updated reports in the respective UIs. + +See: +[Code coverage examples]({{site.baseurl}}/docs/example-catalog/examples/#code-coverage-examples) + + diff --git a/_docs/getting-started/concepts.md b/_docs/getting-started/concepts.md new file mode 100644 index 000000000..21943d500 --- /dev/null +++ b/_docs/getting-started/concepts.md @@ -0,0 +1,113 @@ +--- +title: "Concepts in Codefresh" +description: "Understand terminology and nuances in Codefresh" +group: getting-started +toc: true +--- +Work in progress + + +### Runtime +A Runtime in Codefresh is a GitOps installation in your Codefresh account, in either a Hosted or Hybrid installation environment. Hosted Runtimes are installed on a Codefresh cluster and managed by Codefresh. Hybrid Runtimes are installed on customer clusters, and managed by the customers. +You can install a single Hosted runtime, and multiple Hybrid Runtines in a Codefresh account. + + + +A single Runtime can connect to and manage multiple remote clusters. + + +See: +[GitOps runtime architecture]({{site.baseurl}}/docs/installation/runtime-architecture) +[Hybrid GitOps Runtime installation]({{site.baseurl}}/docs/installation/gitops/hybrid-gitops) +[Hosted GitOps Runtime installation]({{site.baseurl}}/docs/installation/gitops/hosted-runtime) + + + +### Runner +The Runner is the hybrid installation option for CI/CD pipelines in your Codefresh account. The Runner is installed as a Kubernetes native application on any Kubernetes-compliant cluster. It allows you to run pipelines on your own Kubernetes cluster, including private clusters behind company firewalls. + +Codefresh Runner gives you: +* Access to secure services (such as Git repositories or databases) that are behind the firewall and normally not accessible to the public cloud. +* The ability to use special resources in your Codefresh pipeline that are unique to your application, GPU nodes or other special hardware only present in your data center. +* Complete control over the build environment in addition to resources for pipelines. + +Every Runner installation creates a runtime enviroment in your account. Assign the Runner to any pipeline to automatically run the pipeline in your own cluster. External integrations (such as Docker registry or Helm repositories) are also available to the Runner making pipelines exactly the same regardless of their runtime environment. + +You can have multiple Runner installations in the same Codefresh account. A Runner can also manage multiple remote clusters in your account. + +See: +[Codefresh Runner installation]({{site.baseurl}}/docs/installation/codefresh-runner) +[Runner installation behind firewalls]({{site.baseurl}}/docs/reference/behind-the-firewall) + + +### Project +A project is a top-level entity in Codefresh for grouping related pipelines. Projects can group pipelines according to any criteria that is relevant to your enterprise. The criteria can be logical and based on teams, departments, or location for example, or funtional, and based microservices in applications. +Projects centralize viewing and configuration settings for the pipelines that belong to them: +* Selecting a pipeline shows the other pipelines in the same project. +* Define access control and user-defined variables for the project, and they are inherited by all the pipelines assigned to the project + +There are no limits to the number of projects you can create in your account. You can also create standalone pipelines and assign them later to a project, or detach a pipeline assigned to a project. + +See: +[Projects in pipelines]({{site.baseurl}}/docs/pipelines/pipelines/#pipeline-concepts) + +### Pipeline +The pipeline is the central component in Codefresh that implements CI/CD processes. Everything for CI/CD in Codefresh starts and ends with pipelines. a pipeline can do only CI, only CD, both CI and CD, or run any custom action, such as unit and integration tests. + +A CI pipeline can compile and package code, build and push Docker images. A CD pipeline can deploy applications/artifacts to VMs, Kubernetes clusters, FTP sites, S3 buckets, and more. And a CI/CD pipeline can combine code compilation, integration, and deployment for full CI/CD. + +More to be added... + + + + +### Applications +An application is a deployment to a Kubernetes, or any Kubernetes-compatiale cluster or clusters. +Codefresh supports two types of applications: +* Containerized applications packaged as Docker images or +* Argo CD applications + +**Containerized applications** +Containerized applications are compiled, packaged, and deployed through Codefresh pipelines. Codefresh has native support for Docker artifacts, and also supports non-Dockerized applications that don’t use a Dockerfile for the actual build. + +Deploy an application directly to Kubernetes through the Codefresh UI, or use Helm as a package manager to deploy to Kubernetes, again from Codefresh. +Codefresh offers several levels of visibility into your deployments : +* The Kubernetes dashboard displays the status of pods and Docker images. +* The Helm dashabord displays the applications deployed to the cluster through Helm packages. +* The Environment dashbaord displays both Helm and Kubernetes releases, the status of the cluster, and most importantly that of the builds that affect it. + +See: + + +**Agro CD applications** +Argo CD applications conform to Argo CD's application definition CRD (Custom Resource Definition). Argo CD supports several types of Kubernetes manifests, including Jsonnet, Kustomize applications, Helm charts, and YAML/json files, and supports webhook notifications from Git. + +Create Argo CD applications that are fully GitOps-compliant from the Codefresh UI. Work in form mode or directly in YAML in the Create Application wizard. Built-in validation makes it easy to identify and fix errors before commit. The application manifest is generated, committed to Git, and synced to your cluster. +After creation, you can edit and optimize the application, + +Just as with Dockerized applications, you get full visibily into the applications and their deployment thorugh the global Analytics, DORA metrics, and the Application dashboards. The Applications dashboard displays the individual deployments across your enterprise. Drill down shows the current state of all the resources in the application with actions and detilaed information for each resource. + + + +### Triggers +TBD + +### Events +TBD + diff --git a/_docs/getting-started/csdp-introduction.md b/_docs/getting-started/csdp-introduction.md deleted file mode 100644 index 6d48105f0..000000000 --- a/_docs/getting-started/csdp-introduction.md +++ /dev/null @@ -1,208 +0,0 @@ ---- -title: "Introducing Codefresh" -description: "" -group: getting-started -toc: true ---- - -Codefresh is a full-featured, turn-key solution for application deployments and releases. Powered by Argo, Codefresh uses Argo CD, Argo Workflows, Argo Events, and Argo Rollouts, extended with unique functionality and features essential for enterprise deployments. - -Codefresh offers security, maintainability, traceability, and most importantly, a single control plane for all stakeholders, be they developers, operators, product owners or project managers. - -With Codefresh, you can: - -* Deliver software at scale by managing hundreds or thousands of deployment targets and applications -* Get a secure, enterprise-ready distribution of Argo with built-in identity, RBAC (role-based access control), and secrets -* Gain clear visibility across all deployments and trace changes and regressions from code to cloud in seconds -* Get enterprise-level dedicated support for Argo deployments -* Get insights into every aspect of your CI/CD with smart dashboards -* Manage multiple runtimes and multiple clusters in a single pane of glass - - -### Codefresh deployment models - -Codefresh supports hosted and hybrid deployments: - -* **Hosted** deployment or Hosted GitOps, a hosted and managed version of Argo CD. The SaaS version of Codefresh, the runtime is hosted on a Codefresh cluster (easy setup) and managed by Codefresh (zero maintenance overhead). -Click once to provision the hosted runtime, and start deploying applications to clusters without having to install and maintain Argo CD. - - -* **Hybrid** deployment, with the runtime hosted on the customer cluster and managed by the customer. -The hybrid offering retains runtimes within the customer infrastructure while giving you the power of Argo CD with Codefresh's CI and CD tools, to help achieve continuous integration and continuous delivery goals. - -For details, see [Codefresh architecture]({{site.baseurl}}/docs/getting-started/architecture). - - -### Codefresh and open source Argo -Codefresh brings the power of the Argo project to your Kubernetes deployments: - -* Argo CD for declarative continuous deployment -* Argo Rollouts for progressive delivery -* Argo Workflows as the workflow engine -* Argo Events for event-driven workflow automation framework - -Codefresh creates a conformed fork of the Argo project, providing an enterprise-supported version of the same, enhanced with unique functionality. - - - -### Codefresh and GitOps -Codefresh is GitOps-centric, and supports GitOps from the ground up. Codefresh leverages Argo components to have the entire desired state applied from Git to your Kubernetes cluster, and then reported back to Codefresh. -In addition: - -* Every state change operation in Codefresh is made via Git -* Codefresh audit log is derived from the Git changelog -* Codefresh access control is derived from Git permissions - -For details, see [entity model]({{site.baseurl}}/docs/getting-started/entity-model) and [access control]({{site.baseurl}}/docs/administration/access-control). - - -### Insights in Codefresh -Codefresh makes it easy to both access and visualize critical information for any CI/CD resource at any stage, at any level, and for anyone, from managers to DevOps engineers. - -{::nomarkdown} -
- {:/} - -#### Global deployment analytics - -The Home dashboard presents system-wide highlights in real-time, making it an ideal tool for management. -Get insights into important KPIs for entities across runtimes and clusters, in the same location. View status of runtimes and managed clusters, deployments, failed deployments with rollbacks, most active applications, and Delivery Pipelines. - -{% include - image.html - lightbox="true" - file="/images/incubation/home-dashboard.png" - url="/images/incubation/home-dashboard.png" - alt="Global deployment analytics" - caption="Global deployment analytics" - max-width="70%" -%} - -{::nomarkdown} -
- {:/} - -#### DORA metrics - -DORA metrics has become integral to enterprises wanting to quantify DevOps performance, and Codefresh has out-of-the-box support for it. - - -Apart from the metrics themselves, the DORA dashboard in Codefresh has several features such as the Totals bar with key metrics, filters that allow you to pinpoint just which applications or runtimes are contributing to problematic metrics, and the ability to set a different view granularity for each DORA metric. - -See [DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics/). - -{% include - image.html - lightbox="true" - file="/images/incubation/intro-dora-metrics.png" - url="/images/incubation/intro-dora-metrics.png" - alt="DORA metrics" - caption="DORA metrics" - max-width="60%" -%} - -{::nomarkdown} -
- {:/} - -#### Application analytics and analysis - -The Applications dashboard displays a unified view of applications across runtimes and clusters. No matter what the volume and frequency of your deployments, the Applications dashboard makes it easy to track them. Search for Jira issues, commit messages, committers, and see exactly when and if the change was applied to a specific application. - -See [Monitoring applications]({{site.baseurl}}/docs/deployment/applications-dashboard/). - -{::nomarkdown} -
- {:/} - -#### Delivery Pipelines -The Delivery Pipelines dashboard displays aggregated performance analytics based on the pipeline’s workflows, including step analytics across all the workflows in the pipeline. - -{::nomarkdown} -
- {:/} - -#### Workflows -View and monitor submitted workflows across all pipelines in the Workflows dashboard. Select a time range, or view up to fifty of the most recent workflows for all the pipelines in the runtime. Drill down to any workflow for further analysis. -{::nomarkdown} -
- {:/} - -### CI/CD resources in Codefresh -Wizards make it easy to create delivery pipelines and applications. Smart views and options make it easier to monitor and manage them. -{::nomarkdown} -
- {:/} - -#### Delivery Pipelines - -Delivery Pipelines are where the CI magic happens in Codefresh. Our pipeline creation wizard removes the complexity from creating, validating, and maintaining pipelines. Every stage has multi-layered views of all the related Git change information for the pipeline. -See [Create delivery pipelines]({{site.baseurl}}/docs/pipelines/create-pipeline/). - -{::nomarkdown} -
- {:/} - -#### Workflows -Drill down into a workflow to visualize the connections between the steps in the workflow. -A unique feature is the incorporation of Argo Events into the workflow visualization. You get a unified view of Argo Events and Argo Workflows in the same location, the events that triggered the workflow combined with the workflow itself. - -{::nomarkdown} -
- {:/} - -#### Workflow Templates -Select from ready-to-use Workflow Templates in the Codefresh Hub for Argo or create your own custom template. The **Run** option allows you to test a new Workflow Template, or changes to an existing template, without needing to first commit the changes. - - {% include - image.html - lightbox="true" - file="/images/whats-new/wrkflow-template-main.png" - url="/images/whats-new/wrkflow-template-main.png" - alt="Workflow Templates" - caption="Workflow Templates" - max-width="70%" - %} - -{::nomarkdown} -
- {:/} - -#### Applications -Create GitOps-compliant applications, and manage the application lifecycle in the Codefresh UI. - -Define all application settings in a single location through the intuitive Form mode or directly in YAML, and commit all changes to Git. -For easy access, after commit, the configuration settings are available in the Applications dashboard along with the deployment and resource information. - -See [Applications]({{site.baseurl}}/docs/deployment/create-application/). - -{% include - image.html - lightbox="true" - file="/images/applications/add-app-general-settings.png" - url="/images/applications/add-app-general-settings.png" - alt="Application creation in Codefresh" - caption="Application creation in Codefresh" - max-width="60%" -%} - -### GitOps CI integrations - -If you have Hosted GitOps, and your own CI tools for pipelines and workflows, enrich your deployments with CI information without disrupting existing processes. -Simply connect your CI tools to Codefresh, and our new report image template retrieves the information. For example, add the report image step in your GitHub Actions pipeline and reference the different integrations for Codefresh to retrieve and enrich the image with Jira ticket information. - -See [Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/). - -{% include - image.html - lightbox="true" - file="/images/incubation/github-action-int-settings.png" - url="/images/incubation/github-action-int-settings.png" - alt="Image enrichment with GitHub Actions integration" - caption="Image enrichment with GitHub Actions integration" - max-width="60%" -%} - - -### What to read next -[Quick start tutorials]({{site.baseurl}}/docs/getting-started/quick-start) \ No newline at end of file diff --git a/_docs/getting-started/entity-model.md b/_docs/getting-started/entity-model.md deleted file mode 100644 index 302940c17..000000000 --- a/_docs/getting-started/entity-model.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -title: "Entity model" -description: "" -group: getting-started -toc: true ---- - -The Codefresh entity model is derived from these entity types: -* Codefresh account/user management entities -* Argo ecosystem entities -* Workflow, runtime, and Git Source entities -* Codefresh-specific entities such as pipelines, images, and applications - - - -### Codefresh account/user management entities -The account/user management entity types includes entities that do not share a direct relationship to the Codefresh domain. These are enterprise-specific entities in standard SAAS solutions. - -#### Account -Every user who signs in to Codefresh gets a private administrator user account. - -If you received an invitation to Codefresh, instead of a private user account, you are added as a collaborator to the main account. Your permissions are based on those explicitly assigned to you. - -The number of collaborators in an account is defined by the current plan associated with it. - -#### User -A user in Codefresh is one who has completed the sign-up process, and can log in using authorized third-party systems such as: -* GitHub -* Bitbucket -* GitLab -* Azure -* Google - -> If you configure SSO (Single Sign-On) for the account, the user can log in using only the configured SSO. - -#### Billing -For details, please contact [Sales](mailto:sales@codefresh.io?subject=[Codefresh] Codefresh billing inquiry). - -#### Single Sign-On (SSO) -Enterprise accounts can configure SSO. For details, see [Federated Single Sign-On (SSO) overview]({{site.baseurl}}/docs/administration/single-sign-on/). - -#### Security configuration -Security settings include: -* Inactivity timeout per collaborator account -* Domain restriction for invitations - -### Argo ecosystem entities -Codefresh is built on top of the successful open source Argo project, and as such, supports all the native Argo project-entities. -You can apply every supported entity that exists in the open source projects to your Codefresh account. - -### Workflow -Codefresh shows all the workflows executed with Argo Workflows. -Workflows with pipelines display links to the pipelines. Users can terminate or retry a workflow, and view its logs. - -### Runtime -A runtime represents an installation of Codefresh on the customer's K8s cluster, and contains all the components required to perform all tasks on the cluster. - -Review [Codefresh architecture]({{site.baseurl}}/docs/getting-started/architecture/), and [runtime installation ]({{site.baseurl}}/docs/runtime/installation/). - -### Git Source -A Git Source is a link to a Git repository that stores GitOps-controlled entities. You can create as many as Git Sources as you require. - -To understand how to control Git Sources using GitOps, see [access control]({{site.baseurl}}/docs/administration/access-control/). - -### Codefresh high-level entities -Codefresh creates high-level views that better represents, abstracts, and connects all the different entities in the Argo ecosystem. - -#### CI/CD pipeline -A pipeline is a Codefresh-representation of Argo Events, comprising an Argo Events Sensor and Argo Events Triggers. Every trigger in a sensor becomes a different pipeline in Codefresh. The same sensor can be associated with multiple pipelines, if it has different trigger conditions. - -A pipeline links to the following Argo Events entities: -* Sensor -* Event Source -* Workflow Template (or a cluster-level Workflow Template) - -A pipeline also shows all the workflows created from the triggered event associated with that pipeline. - -#### Image -An image represents a built artifact of a Docker image, reported to Codefresh using a dedicated interface. - -Users can use a predefined [Argo Workflow Template](https://codefresh.io/argohub/workflow-template/codefresh-csdp) to help with transferring the image information to Codefresh. - -#### Application -A holistic view of all your Argo CD and Argo Rollouts deployments that link to the underlying artifacts and workflows. diff --git a/_docs/getting-started/faq.md b/_docs/getting-started/faq.md deleted file mode 100644 index bd189aab4..000000000 --- a/_docs/getting-started/faq.md +++ /dev/null @@ -1,52 +0,0 @@ ---- -title: "Frequently asked questions" -description: "" -group: getting-started ---- -We have collected a few of the common questions on the Codefresh solution. - -For questions on Codefresh Classic, navigate to our [FAQs for Codefresh Classic](https://codefresh.io/docs/docs/getting-started/faq/){:target="\_blank"}. - - -**Q. What is the Codefresh platform?** - -A. The Codefresh platform is a full-featured, turn-key solution for application deployments and releases. Powered by the Argo Project, Codefresh uses Argo CD, Argo Workflows, Argo Events, and Argo Rollouts, extended with unique functionality and features essential for enterprise deployments. - -**Q. Which deployment environments does Codefresh support?** - -A. The current release of Codefresh supports hosted and hybrid deployment environments. Stay tuned for our announcement on support for on-premises deployments. - -**Q. How does Codefresh relate to Open Source Argo?** - -A. Codefresh creates a conformed fork of the Argo Project. You get an enterprise-supported version of the Argo Project comprising Argo Workflows, Argo Events, Argo CD, and Argo Rollouts. You can take advantage of the Argo Project offering, with the extended functionality that Codefresh brings to it. - -**Q. I already have a Kubernetes cluster with Argo CD. Can I install Codefresh on the same cluster?** - -A. Hybrid runtimes must be installed on a clean Kubernetes cluster without any Argo Project components. Because we create a conformed fork of the Argo Project in Codefresh, installing it on a cluster with Argo components creates a conflict that will cause the installation to fail. - -**Q. I have resources on my Kubernetes cluster that I want to use in Codefresh. What should I do?** - -A. We will be giving detailed instructions on migrating resources from Kubernetes clusters to Codefresh-based Kubernetes clusters. - -**Q. Does Codefresh support all Git providers?** -A. At the time of writing, Codefresh supports GitHub. We are working to quickly extend support to GitLab and Bitbucket. Stay tuned. - -**Q. What are the browser requirements for the Codefresh UI?** - -A. Officially, we support the latest version of the Chrome browser. Any browser released in the last couple of years should work without major issues. -The following browser versions are **NOT** supported: - -{: .table .table-bordered .table-hover} -| Browser | Version | Date released | -| -------------- | ---------------------------- |-------------------------| -| Chrome | < 51 | May 2016 | -| Firefox | < 54 | Jun 2017 | -| Edge | < 14 | Aug 2016 | -| Safari | < 10 | Sep 2016 | - - -## Migration from Codefresh Classic - -**Q. I have Codefresh Classic. Can I migrate to Codefresh?** -A. At the time of writing, we are working on making the migration from Codefresh Classic to Codefresh as seamless as possible. Stay tuned for the migration announcement. - diff --git a/_docs/getting-started/gitops-codefresh.md b/_docs/getting-started/gitops-codefresh.md new file mode 100644 index 000000000..059a2136e --- /dev/null +++ b/_docs/getting-started/gitops-codefresh.md @@ -0,0 +1,8 @@ +--- +title: "Codefresh for GitOps" +description: "Argo CD with Codefresh GitOps" +group: getting-started +toc: true +--- + +Work in progress TBD \ No newline at end of file diff --git a/_docs/getting-started/gitops.md b/_docs/getting-started/gitops.md deleted file mode 100644 index d7542fbbb..000000000 --- a/_docs/getting-started/gitops.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: "GitOps approach" -description: "" -group: getting-started -toc: true ---- - -> In the documentation, Kubernetes and K8s are used interchangeably. - -### GitOps - -The Codefresh platform is built entirely around the concept of GitOps, a set of best practices where the entire code delivery process is controlled via Git, including infrastructure and application definition, and automation to complete updates and rollbacks. - -To fully understand the benefits of Codefresh, let's briefly recap GitOps, and how it can help: - -#### Infrastructure as code, the entire system described declaratively - Infrastructure as code is a modern approach that "declaratively" describes the state of a system as code, while having that single source of truth applied to an end-system. The end-systems in most cases are modern cloud native tools. - - Declarative means that configuration is guaranteed by a set of facts, instead of by a set of instructions. With your end system's declarations versioned in Git, you have a single source of truth. You can then both easily deploy and roll back your end system according to the state changes in Git. And more important, if and when disaster strikes, you can also reproduce your cluster’s infrastructure reliably and quickly. - - GitOps is just a specific case of infrastructure as code where the end system is a Kubernetes cluster. - -#### Desired system state versioned in Git - With the declaration of your system stored in a version control system, and serving as your canonical source of truth, you have a single place from which everything is derived and driven. Now not only your application code is in Git, but also all the information required to install and manage your application, including service definition, deployment information, and more. - - Developers can continue with the familiar and convenient approaches they are already using for their applicative code. In addition, Git makes complicated tasks like collaboration (via pull requests), security (via signed commits), permissions (repository permissions), and rollback, as trivial as they can get. - - -#### Use dedicated tools to implement transfer of desired state into the end system - Once the state of your end-system is declared and kept under version control, you need a tool and process to apply the updated desired state into the end system. - - One of the tools for implementing infrastructure as code in the realm of DevOps is [Terraform](https://www.terraform.io/), for example. - - While you can implement GitOps (infrastructure as code for Kubernetes), using a battle-ready tool like Terraform which has a plugin system that also supports Kubernetes, K8s has many nuances that differ from a traditional sync process to a cloud system or some other standard REST API end system. - - To address the specific use cases of Kubernetes, there are new tools dedicated to implementing GitOps (infrastructure as code for k8s), such as [ArgoCD](https://github.com/argoproj/argo-cd). - - diff --git a/_docs/getting-started/intro-to-codefresh.md b/_docs/getting-started/intro-to-codefresh.md new file mode 100644 index 000000000..850aad11c --- /dev/null +++ b/_docs/getting-started/intro-to-codefresh.md @@ -0,0 +1,113 @@ +--- +title: "Introduction to Codefresh" +description: "What is Codefresh?" +group: getting-started +toc: true +--- + +TBD + + + \ No newline at end of file diff --git a/_docs/getting-started/main-concepts.md b/_docs/getting-started/main-concepts.md deleted file mode 100644 index c2d4f4180..000000000 --- a/_docs/getting-started/main-concepts.md +++ /dev/null @@ -1,37 +0,0 @@ ---- -title: "Main concepts" -description: "" -group: getting-started -toc: true ---- - -### Built on top of the open source Argo -Codefresh maintains a [conformed](https://github.com/argoproj/argo-conformance-program) fork of the following Argo components, providing an enterprise-supported version of them: -* [Argo CD](https://github.com/argoproj/argo-cd): Declarative continuous deployment for Kubernetes. -* [Argo Rollouts](https://argoproj.github.io/argo-rollouts/): Progressive Delivery for Kubernetes. -* [Argo Workflows](https://github.com/argoproj/argo-workflows): Workflow engine for Kubernetes. -* [Argo Events](https://github.com/argoproj/argo-events): Event-driven workflow automation framework. - -For details, see [Codefresh architecture]({{site.baseurl}}/docs/getting-started/architecture/). - -### Hybrid behind firewall model -Codefresh performs an installation, called a Runtime, on the user's K8s cluster. The Runtime contains all required components for the Codefresh experience. - -For details, see [Codefresh architecture]({{site.baseurl}}/docs/getting-started/architecture/). - -### GitOps native approach -Codefresh is built entirely on the heavily-adopted concept of GitOps. Read the detailed explanation on our [GitOps approach]({{site.baseurl}}/docs/getting-started/gitops/).
-Codefresh leverages Argo components (Argo CD and Argo Events), to have the entire desired state applied from Git to the user's K8s cluster, and also reported back to Codefresh platform. - -### Every state change operation in Codefresh is made via Git -Codefresh has taken the GitOps approach a step forward by making our entire entity model fully controlled by GitOps via Codefresh, meaning that the entire state of your account is maintained in Git. For details, see [entity model]({{site.baseurl}}/docs/getting-started/entity-model/). - -Codefresh provides a full front-end experience powered by a strong API layer (GraphQL), and every state change (via GraphQL mutation) actually performs a commit on behalf of the user to Git. - -### Audit log derived from Git changelog -Codefresh has built its sophisticated but simple audit log on all operations to the system, for both the Git change and the log of API calls that have been made to the system. -For details, see [audit]({{site.baseurl}}/docs/administration/audit/). - -### Access control derived from Git permissions -Codefresh has built its sophisticated but simple access control model on top of the existing Git operations that are defined externally to the system.
-For details, see [access control]({{site.baseurl}}/docs/administration/access-control/). diff --git a/_docs/integrations/ci-integrations.md b/_docs/gitops-integrations/ci-integrations.md similarity index 89% rename from _docs/integrations/ci-integrations.md rename to _docs/gitops-integrations/ci-integrations.md index f43ad8e40..f1779d19b 100644 --- a/_docs/integrations/ci-integrations.md +++ b/_docs/gitops-integrations/ci-integrations.md @@ -1,18 +1,18 @@ --- -title: "CI integrations" +title: "GitOps CI integrations" description: "" -group: integrations +group: gitops-integrations toc: true --- -Use Codefresh's Hosted GitOps with any popular Continuous Integration (CI) solution, not just with Codefresh CI. +Use Codefresh Hosted GitOps with any popular Continuous Integration (CI) solution, not just with Codefresh CI. You can connect a third-party CI solution to Codefresh, such as GitHub Actions for example, to take care of common CI tasks such as building/testing/scanning source code, and have Codefresh Hosted GitOps still responsible for the deployment, including image enrichment and reporting. The integration brings in all the CI information to your images which you can see in the Images dashboard. -See [Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/). +See [Image enrichment with GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/image-enrichment-overview/). -### Codefresh image reporting and enrichment action +## Codefresh image reporting and enrichment action To support the integration between Codefresh and third-party CI platforms and tools, we have created dedicated actions for supported CI tools in the Codefresh Marketplace. These actions combine image enrichment and reporting through integrations with issue tracking and container registry tools. >You can also configure the integration directly in the Codefresh UI, as described in [Connect a third-party CI platform/tool to Codefresh](#connect-a-third-party-ci-platformtool-to-codefresh). @@ -29,17 +29,17 @@ Use the action as follows: 1. When the pipeline completes execution, Codefresh retrieves the information on the image that was built and its metadata through the integration names specified (essentially the same data that Codefresh CI would send automatically). 1. View the image in Codefresh's [Images dashboard]({{site.baseurl}}/docs/deployment/images/), and in any [application]({{site.baseurl}}/docs/deployment/applications-dashboard/) in which it is used. -### Connect a third-party CI platform/tool to Codefresh -Connecting the CI platform/tool to Codefresh from the UI includes configuring the required arguments, and then generating and copying the YAML manifest for the report image to your pipeline. +## Connect a third-party CI platform/tool to GitOps +Connecting the CI platform/tool to GitOps from the UI includes configuring the required arguments, and then generating and copying the YAML manifest for the report image to your pipeline. -1. In the Codefresh UI, go to [Integrations](https://g.codefresh.io/2.0/account-settings/integrations){:target="\_blank"}. +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**GitOps Integrations**](https://g.codefresh.io/2.0/account-settings/integrations){:target="\_blank"}. 1. Filter by **CI tools**, then select the CI tool and click **Add**. 1. Define the arguments for the CI tool: - [Codefresh Classic]({{site.baseurl}}/docs/integrations/ci-integrations/codefresh-classic/) - [GitHub Action]({{site.baseurl}}/docs/integrations/ci-integrations/github-actions/) - [Jenkins]({{site.baseurl}}/docs/integrations/ci-integrations/jenkins/) + [Codefresh Classic]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/codefresh-classic/) + [GitHub Action]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/github-actions/) + [Jenkins]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/jenkins/) - For the complete list of arguments you can use, see [CI integration argument reference](#ci-integration-argument-reference) in this article. + For the complete list of arguments you can use, see [CI integration for GitOps argument reference](#ci-integration-argument-reference) in this article. 1. To generate a YAML snippet with the arguments, on the top-right, click **Generate Manifest**. Codefresh validates the generated manifest, and alerts you to undefined arguments that are required, and other errors. @@ -79,9 +79,9 @@ The table describes _all_ the arguments required for CI integrations in general. | `CF_RUNTIME_NAME` | The runtime to use for the integration. If you have more than one runtime, select the runtime from the list. | Required | | `CF_PLATFORM_URL` | The root URL of the Codefresh application. The default value is `https://g.codefresh.io`. | Optional | | `CF_API_KEY` | The API key for authentication. Generate the key for the integration. | Required | -| `CF_CONTAINER_REGISTRY_INTEGRATION` | The name of the container registry integration created in Codefresh where the image is stored. See [Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/). | Optional | +| `CF_CONTAINER_REGISTRY_INTEGRATION` | The name of the container registry integration created in Codefresh where the image is stored. See [Container registry integrations]({{site.baseurl}}/docs/gitops-integrations/container-registries/). | Optional | | `CF_JIRA_INTEGRATION` | _Deprecated from version 0.0.565 and higher._ Replaced by `CF_ISSUE_TRACKING_INTEGRATION`. | _Deprecated_ -| `CF_ISSUE_TRACKING_INTEGRATION` | The name of the issue tracking integration created in Codefresh to use for image enrichment. Relevant only if Jira enrichment is required for the image. If you don't have a Jira integration, click **Create Atlassian Jira Integration** and configure settings. See [Jira integration]({{site.baseurl}}/docs/integrations/issue-tracking/jira/). | Optional | +| `CF_ISSUE_TRACKING_INTEGRATION` | The name of the issue tracking integration created in Codefresh to use for image enrichment. Relevant only if Jira enrichment is required for the image. If you don't have a Jira integration, click **Create Atlassian Jira Integration** and configure settings. See [Jira integration]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/jira/). | Optional | | `CF_IMAGE` | The image to be enriched and reported in Codefresh. Pass the `[account-name]/[image-name]:[tag]` built in your CI. | Required | | `CF_WORKFLOW_NAME` | The name assigned to the workflow that builds the image. When defined, the name is displayed in the Codefresh platform. Example, `Staging step` | Optional | | `CF_GIT_BRANCH` | The Git branch with the commit and PR (pull request) data to add to the image. Pass the Branch from the event payload used to trigger your action. | Required | @@ -96,9 +96,9 @@ The table describes _all_ the arguments required for CI integrations in general. | `CF_JIRA_MESSAGE` | Relevant only when `CF_ISSUE_TRACKING_INTEGRATION` is defined. The Jira issue IDs matching the string to associate with the image. | Required | | `CF_JIRA_FAIL_ON_NOT_FOUND` | Relevant only when `CF_ISSUE_TRACKING_INTEGRATION` is defined. The report image action when the `CF_JIRA_MESSAGE` is not found. When set to `true`, the report image action is failed. | Required | -### Related articles -[Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/) -[Issue tracking intergrations]({{site.baseurl}}/docs/integrations/issue-tracking/) +## Related articles +[Container registry GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/container-registries/) +[Issue tracking GitOps intergrations]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/) diff --git a/_docs/integrations/ci-integrations/codefresh-classic.md b/_docs/gitops-integrations/ci-integrations/codefresh-classic.md similarity index 83% rename from _docs/integrations/ci-integrations/codefresh-classic.md rename to _docs/gitops-integrations/ci-integrations/codefresh-classic.md index e15ab68ec..73ab5e94a 100644 --- a/_docs/integrations/ci-integrations/codefresh-classic.md +++ b/_docs/gitops-integrations/ci-integrations/codefresh-classic.md @@ -1,16 +1,16 @@ --- -title: "Codefresh Classic" +title: "Codefresh CI pipeline GitOps integration" description: "" -group: integrations +group: gitops-integrations sub_group: ci-integrations toc: true --- - Use Hosted GitOps with any popular Continuous Integration (CI) solution, not just with Codefresh CI. Codefresh Classic is one of the third-party CI platform/tools that you can connect to Codefresh for deployment with image enrichment and reporting. + Use Hosted GitOps with any popular Continuous Integration (CI) solution, not just with Codefresh CI. If you have Hosted or Hybrid GitOps, you can connect your CI pipelines to Hosted GitOps for deployment with image enrichment and reporting. - Connecting Codefresh Classic, adds the CI information to images which are displayed in the Images dashboard, as in the example below. + Connecting your CI pipeline, adds the CI information to images which are displayed in the Images dashboard, as in the example below. {% include image.html @@ -24,53 +24,65 @@ toc: true -For information on how to use the image reporting action in your Codefresh Classic pipeline and how to configure the integration, see [CI Integrations]({{site.baseurl}}/docs/integrations/ci-integrations/). +For information on how to use the image reporting action in your Codefresh pipeline and how to configure the integration, see [CI Integrations]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/). -### Example of Codefresh Classic pipeline with report image step +## Example of Codefresh pipeline with report image step {% highlight yaml %} {% raw %} -reportImage: - title: Report image to Codefresh CD - type: codefresh-report-image - working_directory: /code - arguments: - # The URL to the cluster with the Codefresh runtime to integrate with. - CF_HOST: '[runtime-host-url]' - - # Codefresh API key !! Committing a plain text token is a security risk. We highly recommend using encrypted secrets !! - # Documentation - https://codefresh.io/docs/docs/configure-ci-cd-pipeline/secrets-store/ - CF_API_KEY: ${{API_KEY}} - - # Image path to enrich - CF_IMAGE: '[full image path here, including tag]' - - # Name of Container registry integration - CF_CONTAINER_REGISTRY_INTEGRATION: 'v2' - - # The git branch which is related for the commit - CF_GIT_BRANCH: '[name-of-your-git-branch]' - - # Name of Jira integration - CF_JIRA_INTEGRATION: 'jira' - - # Jira project filter - CF_JIRA_PROJECT_PREFIX: '[jira-project-prefix]' - - # String starting with the issue ID to associate with image - CF_JIRA_MESSAGE: '[issue-id]' +version: "1.0" +stages: + - "clone" + - "build" + - "report" + +steps: + clone: + title: "Cloning repository" + type: "git-clone" + repo: "${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}" + revision: "${{CF_BRANCH}}" + stage: "clone" + + build: + title: "Building Docker image" + type: "build" + image_name: "${{CF_REPO_OWNER}}/color" + working_directory: "${{clone}}" + tag: "${{CF_SHORT_REVISION}}" + dockerfile: "Dockerfile" + registry: docker-lr + stage: "build" + + ReportImageMetadataAll: + title: Report image to Codefresh CD + type: codefresh-report-image + working_directory: /code + stage: "report" + arguments: + CF_API_KEY: '${{CF_API_KEY}}' + CF_IMAGE: 'docker.io/${{CF_REPO_OWNER}}/color:${{CF_SHORT_REVISION}}' + CF_CONTAINER_REGISTRY_INTEGRATION: docker + CF_RUNTIME_NAME: "codefresh-hosted" + CF_GITHUB_TOKEN: '${{GITHUB_TOKEN}}' + CF_GIT_PROVIDER: github + CF_GIT_REPO: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}' + CF_GIT_BRANCH: '${{CF_BRANCH}}' + CF_ISSUE_TRACKING_INTEGRATION: jira + CF_JIRA_MESSAGE: "${{CF_COMMIT_MESSAGE}}" + CF_JIRA_PROJECT_PREFIX: CR {% endraw %} {% endhighlight yaml %} -### Codefresh Classic-Codefresh integration arguments +## CI pipeline-GitOps integration settings The table describes the arguments required to connect Codefresh Classic to Codefresh. >Except for Git branch and Git repo which are required, you can omit other Git provider arguments. Codefresh retrieves the required values from the runtime context selected for the integration. -For the complete argument reference, see [CI integration argument reference]({{site.baseurl}}/docs/integrations/ci-integrations/#ci-integration-argument-reference). +For the complete argument reference, see [CI integration for GitOps argument reference]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/#ci-integration-argument-reference). {: .table .table-bordered .table-hover} @@ -79,9 +91,9 @@ For the complete argument reference, see [CI integration argument reference]({{s | `CF_RUNTIME_NAME` | The runtime to use for the integration. If you have more than one runtime, select the runtime from the list. | Required | | `CF_PLATFORM_URL` | The root URL of the Codefresh application. The default value is `https://g.codefresh.io`. | Optional | | `CF_API_KEY` | The API key to authenticate the Codefresh Classic user to Codefresh. Generate the key for the integration. | Required | -| `CF_CONTAINER_REGISTRY_INTEGRATION` | The name of the container registry integration created in Codefresh where the image is stored. To create a container registry integration if you don't have one, click **Create Container Registry Integration**, and then configure the settings. See [Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/). | Optional | +| `CF_CONTAINER_REGISTRY_INTEGRATION` | The name of the container registry integration created in Codefresh where the image is stored. To create a container registry integration if you don't have one, click **Create Container Registry Integration**, and then configure the settings. See [Container registry integrations]({{site.baseurl}}/docs/gitops-integrations/container-registries/). | Optional | | `CF_JIRA_INTEGRATION` | Deprecated from version 0.0.565. Replaced by `CF_ISSUE_TRACKING_INTEGRATION`. | _Deprecated_ -| `CF_ISSUE_TRACKING_INTEGRATION` | The name of the issue tracking integration created in Codefresh to use to enrich the image. Relevant only if Jira enrichment is required for the image. If you don't have a Jira integration, click **Create Atlassian Jira Integration** and configure settings. See [Jira integration]({{site.baseurl}}/docs/integrations/issue-tracking/jira/). | Optional | +| `CF_ISSUE_TRACKING_INTEGRATION` | The name of the issue tracking integration created in Codefresh to use to enrich the image. Relevant only if Jira enrichment is required for the image. If you don't have a Jira integration, click **Create Atlassian Jira Integration** and configure settings. See [Jira integration]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/jira/). | Optional | | `CF_IMAGE` | The image to be enriched and reported in Codefresh. Pass the `[account-name]/[image-name]:[tag]` built in your CI. | Required | | `CF_WORKFLOW_NAME` | The name assigned to the workflow that builds the image. When defined, the name is displayed in the Codefresh platform. Example, `Staging step` | Optional | | `CF_GIT_BRANCH` | The Git branch with the commit and PR (pull request) data to add to the image. Pass the Branch from the event payload used to trigger your action. | Required | @@ -96,19 +108,19 @@ For the complete argument reference, see [CI integration argument reference]({{s | `CF_JIRA_MESSAGE` | Relevant only when `CF_ISSUE_TRACKING_INTEGRATION` is defined. The Jira issue IDs matching the string to associate with the image. | Required | | `CF_JIRA_FAIL_ON_NOT_FOUND` | Relevant only when `CF_ISSUE_TRACKING_INTEGRATION` is defined. The report image action when the `CF_JIRA_MESSAGE` is not found. When set to `true`, the report image action is failed. | Required | -For how-to instructions, see [Connect a third-party CI platform/tool to Codefresh]({{site.baseurl}}/docs/integrations/ci-integrations/#connect-a-third-party-ci-platformtool-to-codefresh/). +For how-to instructions, see [Connect a third-party CI platform/tool to Codefresh GitOps]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/#connect-a-third-party-ci-platformtool-to-codefresh/). -### Templatization examples for CF arguments +## Templatization examples for CF arguments -Arguments such as `CF_IMAGE`, `CF_GIT_BRANCH`, and `CF_JIRA_MESSAGE` are populated dynamically when the Codefresh Classic integration pipeline is triggered. You can templatize the values of these arguments to ensure that the required information is included in the reported image. +Arguments such as `CF_IMAGE`, `CF_GIT_BRANCH`, and `CF_JIRA_MESSAGE` are populated dynamically when the Codefresh integration pipeline is triggered. You can templatize the values of these arguments to ensure that the required information is included in the reported image. -Codefresh Classic offers [system variables](https://codefresh.io/docs/docs/codefresh-yaml/variables/#system-provided-variables) you can use to templatize argument values. +Codefresh pipelines have [system variables]({{site.baseurl}}/docs/pipelines/variables/#system-provided-variables) you can use to templatize argument values. {::nomarkdown}
{:/} -#### CF_IMAGE examples +### CF_IMAGE examples **Example: Report full repo and branch information** This example illustrates how to define the value for `CF_IMAGE` to report the repo owner, name, and branch, with the Git hash. @@ -146,7 +158,7 @@ where:
{:/} -#### CF_GIT_BRANCH examples +### CF_GIT_BRANCH examples **Example: Report Git branch or tag with committer and commit message** @@ -195,14 +207,14 @@ Value:
{:/} -#### CF_JIRA_MESSAGE examples +### CF_JIRA_MESSAGE examples The Jira message represents an existing Jira issue, and must be a literal string. Value: `CR-1246` -### Codefresh Classic integration logs -View and analyze logs for Codefresh Classic workflows through the Logs tab. When a Codefresh Classic pipeline is run, it is added to the Logs tab. +## Codefresh pipeline integration logs +View and analyze logs for Codefresh pipelines through the Logs tab. When a Codefresh pipeline is run, it is added to the Logs tab. You can: * Filter by status or by date range to view a subset of actions * Navigate to the build file in Codefresh Classic, and view the Codefresh report image step @@ -216,7 +228,7 @@ caption="Codefresh Classic: Logs tab" max-width="50%" %} -**Build in Codefresh Classic** +**Build in Codefresh** The Run column includes the link to the pipeline in Codefresh Classic. @@ -240,8 +252,8 @@ caption="Logs for Codefresh report image step in Codefresh Classic build" max-width="50%" %} -### Related articles +## Related articles [Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration/) -[Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/) -[Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/) -[Issue-tracking integrations]({{site.baseurl}}/docs/integrations/issue-tracking/) \ No newline at end of file +[Image enrichment with GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/image-enrichment-overview/) +[Container registry GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/container-registries/) +[Issue-tracking GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/) \ No newline at end of file diff --git a/_docs/integrations/ci-integrations/github-actions.md b/_docs/gitops-integrations/ci-integrations/github-actions.md similarity index 91% rename from _docs/integrations/ci-integrations/github-actions.md rename to _docs/gitops-integrations/ci-integrations/github-actions.md index fa6dd1c89..2d6246cae 100644 --- a/_docs/integrations/ci-integrations/github-actions.md +++ b/_docs/gitops-integrations/ci-integrations/github-actions.md @@ -1,13 +1,13 @@ --- title: "GitHub Actions" description: "" -group: integrations +group: gitops-integrations sub_group: ci-integrations toc: true --- -Use Codefresh Hosted GitOps with any popular Continuous Integration (CI) solution, not just with Codefresh CI. -GitHub Actions is one of the third-party CI solutions that you can connect to Codefresh for deployment with image reporting and enrichment. +Use Hosted GitOps with any popular Continuous Integration (CI) solution, not just with Codefresh CI. +GitHub Actions is one of the third-party CI solutions that you can connect to Hosted GitOps for deployment with image reporting and enrichment. Connecting a GitHub Action, adds the CI information to images which are displayed in the Images dashboard, as in the example below. @@ -21,10 +21,10 @@ GitHub Actions is one of the third-party CI solutions that you can connect to Co max-width="70%" %} -For information on how to use the image reporting action in your GitHub Action pipeline and how to configure the integration, see [CI Integrations]({{site.baseurl}}/docs/integrations/ci-integrations/). +For information on how to use the image reporting action in your GitHub Action pipeline and how to configure the integration, see [CI Integrations]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/). -### Example of GitHub Actions pipeline with Codefresh report image action +## Example of GitHub Actions pipeline with Codefresh report image action Here is an example pipeline that uses GitHub Actions to build a container image, and the Codefresh action to enrich and report the resulting image to Codefresh. @@ -100,7 +100,7 @@ jobs: {% endraw %} {% endhighlight yaml %} -### GitHub Action-Codefresh integration arguments +## GitHub Action-GitOps integration settings The table describes the arguments required to connect a GitHub Action to Codefresh. @@ -111,10 +111,10 @@ The table describes the arguments required to connect a GitHub Action to Codefre | `CF_RUNTIME_NAME` | The runtime to use for the integration. If you have more than one runtime, select the runtime from the list. | Required | | `CF_PLATFORM_URL` | The root URL of the Codefresh application. The default value is `https://g.codefresh.io`. | Optional | | `CF_API_KEY` | The API key to authenticate the GitHub Actions user to Codefresh. Generate the key for the GitHub Action. {::nomarkdown}
Enter this token in GitHub Actions as a secret with the name CF_API_KEY. You can then reference it in all GitHub pipelines as you would any other secret.{:/}| Required | -| `CF_CONTAINER_REGISTRY_INTEGRATION` | The name of the container registry integration created in Codefresh where the image is stored. {::nomarkdown}- For a GitHub Container registry, select GHCR_GITHUB_TOKEN_AUTHENTICATION even if you have not created an integration in Codefresh.
Codefresh retrieves and provides the explicit credentials for the container registry on generating the integration manifest. - To create a container registry integration if you don't have one, click Create Container Registry Integration, and then configure the settings.
See Container registry integrations.
- For a GitHub Container registry, select GHCR_GITHUB_TOKEN_AUTHENTICATION even if you have not created an integration in Codefresh.
Codefresh retrieves and provides the explicit credentials for the container registry on generating the integration manifest. - To create a container registry integration if you don't have one, click Create Container Registry Integration, and then configure the settings.
See Container registry integrations.
{:/} -#### CF_IMAGE +### CF_IMAGE **Example: Report full repo and branch information** This example illustrates how to define the value for `CF_IMAGE` to report the repo owner, name, and short branch, with the Git hash. @@ -176,7 +177,7 @@ where:
{:/} -#### CF_GIT_BRANCH +### CF_GIT_BRANCH **Example: Report fully-formed reference of the branch or tag** This example illustrates how to define the value for `CF_GIT_BRANCH` to report the fully-formed reference of the branch or tag that triggered the workflow run. @@ -203,13 +204,13 @@ where:
{:/} -#### CF_JIRA_MESSAGE +### CF_JIRA_MESSAGE The Jira message represents an existing Jira issue, and must be a literal string. Value: `CR-1246` -### GitHub Action logs +## GitHub Action logs View and analyze logs for GitHub Action workflows through the Logs tab. When a GitHub Action is run, it is added to the Logs tab. You can: * Filter by status or by date range to view a subset of actions @@ -249,10 +250,10 @@ max-width="50%" %} -### Related articles +## Related articles [Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration/) -[Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/) -[Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/) -[Issue-tracking integrations]({{site.baseurl}}/docs/integrations/issue-tracking/) +[Image enrichment with GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/image-enrichment-overview/) +[Container registry GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/container-registries/) +[Issue-tracking GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/) diff --git a/_docs/integrations/ci-integrations/jenkins.md b/_docs/gitops-integrations/ci-integrations/jenkins.md similarity index 94% rename from _docs/integrations/ci-integrations/jenkins.md rename to _docs/gitops-integrations/ci-integrations/jenkins.md index ddfa1eba4..842a47a17 100644 --- a/_docs/integrations/ci-integrations/jenkins.md +++ b/_docs/gitops-integrations/ci-integrations/jenkins.md @@ -1,7 +1,7 @@ --- -title: "Jenkins" +title: "Jenkins GitOps integration" description: "" -group: integrations +group: gitops-integrations sub_group: ci-integrations toc: true --- @@ -21,9 +21,9 @@ toc: true %} -For information on how to use the image reporting action in your Jenkins pipeline and how to configure the integration, see [CI Integrations]({{site.baseurl}}/docs/integrations/ci-integrations/). +For information on how to use the image reporting action in your Jenkins pipeline and how to configure the integration, see [CI Integrations]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/). -### Example of Jenkins pipeline with report image step +## Example of Jenkins pipeline with report image step {% highlight yaml %} {% raw %} @@ -110,7 +110,7 @@ pipeline { {% endraw %} {% endhighlight yaml %} -### Jenkins-Codefresh integration arguments +## Jenkins-GitOps integration settings The table describes the arguments to connect Codefresh Classic to Codefresh. {: .table .table-bordered .table-hover} @@ -119,9 +119,9 @@ The table describes the arguments to connect Codefresh Classic to Codefresh. | `CF_RUNTIME_NAME` | The runtime to use for the integration. If you have more than one runtime, select the runtime from the list. | Required | | `CF_PLATFORM_URL` | The root URL of the Codefresh application. The default value is `https://g.codefresh.io`. | Optional | | `CF_API_KEY` | The API key to authenticate the Codefresh Classic user to Codefresh. Generate the key for the integration. | Required | -| `CF_CONTAINER_REGISTRY_INTEGRATION` | The name of the container registry integration created in Codefresh where the image is stored. To create a container registry integration if you don't have one, click **Create Container Registry Integration**, and then configure the settings. See [Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/). | Optional | +| `CF_CONTAINER_REGISTRY_INTEGRATION` | The name of the container registry integration created in Codefresh where the image is stored. To create a container registry integration if you don't have one, click **Create Container Registry Integration**, and then configure the settings. See [Container registry integrations]({{site.baseurl}}/docs/gitops-integrations/container-registries/). | Optional | | `CF_JIRA_INTEGRATION` | Deprecated from version 0.0.565. Replaced by `CF_ISSUE_TRACKING_INTEGRATION`. | _Deprecated_ -| `CF_ISSUE_TRACKING_INTEGRATION` | The name of the issue tracking integration created in Codefresh to use to enrich the image. Relevant only if Jira enrichment is required for the image. If you don't have a Jira integration, click **Create Atlassian Jira Integration** and configure settings. See [Jira integration]({{site.baseurl}}/docs/integrations/issue-tracking/jira/). | Optional | +| `CF_ISSUE_TRACKING_INTEGRATION` | The name of the issue tracking integration created in Codefresh to use to enrich the image. Relevant only if Jira enrichment is required for the image. If you don't have a Jira integration, click **Create Atlassian Jira Integration** and configure settings. See [Jira integration]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/jira/). | Optional | | `CF_IMAGE` | The image to be enriched and reported in Codefresh. Pass the `[account-name]/[image-name]:[tag]` built in your CI. | Required | | `CF_GIT_BRANCH` | The Git branch with the commit and PR (pull request) data to add to the image. Pass the Branch from the event payload used to trigger your action. | Required | | `CF_GIT_REPO` | The Git repository with the configuration and code used to build the image. | Required | @@ -136,9 +136,9 @@ The table describes the arguments to connect Codefresh Classic to Codefresh. | `CF_JIRA_FAIL_ON_NOT_FOUND` | Relevant only when `CF_ISSUE_TRACKING_INTEGRATION` is defined. The report image action when the `CF_JIRA_MESSAGE` is not found. When set to `true`, the report image action is failed. | Required | -For how-to instructions, see [Connect a third-party CI platform/tool to Codefresh]({{site.baseurl}}/docs/integrations/ci-integrations/#connect-a-third-party-ci-platform-tool-to-codefresh). +For how-to instructions, see [Connect a third-party CI platform/tool to Codefresh]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/#connect-a-third-party-ci-platform-tool-to-codefresh). -### Templatization examples for CF arguments +## Templatization examples for CF arguments Arguments such as `CF_IMAGE`, `CF_GIT_BRANCH`, and `CF_JIRA_MESSAGE` are populated dynamically when the Jenkins pipeline is triggered. You can templatize the values of these arguments in the pipeline to ensure that the required information is included in the reported image. @@ -148,7 +148,7 @@ Jenkins offers a Git plugin with [environment variables](https://plugins.jenkins
{:/} -#### CF_IMAGE +### CF_IMAGE **Example: Report repo, branch with Git hash** This example illustrates how to define the value for `CF_IMAGE` to report Git repo, branch, committer, and Git hash information. @@ -188,7 +188,7 @@ This example illustrates how to define the value for `CF_IMAGE` value to report
{:/} -#### CF_GIT_BRANCH +### CF_GIT_BRANCH **Example: Report the fully-formed Git branch** This example illustrates how to define the value for `CF_GIT_BRANCH` value to report the fully-formed Git branch. @@ -215,13 +215,13 @@ This example illustrates how to define the value for `CF_GIT_BRANCH` value to re
{:/} -#### CF_JIRA_MESSAGE +### CF_JIRA_MESSAGE The Jira message represents an existing Jira issue, and must be a literal string. Value: `CR-1246` -### Jenkins integration logs +## Jenkins integration logs View and analyze logs for Jenkins through the Logs tab. When a Jenkins pipeline is run, it is added to the Logs tab. You can: * Filter by status or by date range to view a subset of actions @@ -243,8 +243,8 @@ caption="Logs for Codefresh report image step in Jenkins build" max-width="50%" %} -### Related articles +## Related articles [Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration/) -[Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/) -[Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/) -[Issue-tracking integrations]({{site.baseurl}}/docs/integrations/issue-tracking/) \ No newline at end of file +[Image enrichment with GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/image-enrichment-overview/) +[Container registry GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/container-registries/) +[Issue-tracking GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/) diff --git a/_docs/integrations/container-registries.md b/_docs/gitops-integrations/container-registries.md similarity index 69% rename from _docs/integrations/container-registries.md rename to _docs/gitops-integrations/container-registries.md index 2aef4ac1e..69a7a87bd 100644 --- a/_docs/integrations/container-registries.md +++ b/_docs/gitops-integrations/container-registries.md @@ -1,13 +1,13 @@ --- -title: "Container registry integrations" +title: "GitOps container registry integrations" description: "" -group: integrations +group: gitops-integrations toc: true --- Codefresh can integrate with popular container registries such as Docker Hub, JFrog Artifactory, and more. -Adding a container registry integration in Codefresh allows you to reference the integration in third-party CI platforms/tools such as GitHub Actions and Codefresh Classic by the name of the registry integration, instead of explicit credentials. See [Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/) and [CI integrations]({{site.baseurl}}/docs/integrations/ci-integrations/). +Adding a container registry integration in Codefresh allows you to reference the integration in third-party CI platforms/tools such as GitHub Actions and Codefresh Classic by the name of the registry integration, instead of explicit credentials. See [Image enrichment with integrations]({{site.baseurl}}/docs/gitops-integrations/image-enrichment-overview/) and [CI integrations]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/). You add a container registry integration in Codefresh by: * Defining the integration name @@ -21,29 +21,29 @@ You can add more than one integration for the same registry. Once added, Codefre -### Configure container registry integrations in Codefresh -Configure the settings for a container registry integration in Codefresh. +## Configure container registry integrations for GitOps in Codefresh +Configure the settings for a GitOps container registry integration in Codefresh. -1. In the Codefresh UI, go to [Integrations](https://g.codefresh.io/2.0/account-settings/integrations){:target="\_blank"}. +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**GitOps Integrations**](https://g.codefresh.io/2.0/account-settings/integrations){:target="\_blank"}. 1. Filter by **Container Registry**, select the container registry, and click **Configure**. 1. If you already have integrations, click **Add**. 1. Define the arguments for the container registry: - [Amazon ECR]({{site.baseurl}}/docs/integrations/container-registries/amazon-ecr/) - [Docker Hub]({{site.baseurl}}/docs/integrations/container-registries/dockerhub/) - [GitHub Container Registry]({{site.baseurl}}/docs/integrations/container-registries/github-cr/) - [JFrog Artifactory]({{site.baseurl}}/docs/integrations/container-registries/jfrog/) - [Quay]({{site.baseurl}}/docs/integrations/container-registries/quay/) + [Amazon ECR]({{site.baseurl}}/docs/gitops-integrations/container-registries/amazon-ecr/) + [Docker Hub]({{site.baseurl}}/docs/gitops-integrations/container-registries/dockerhub/) + [GitHub Container Registry]({{site.baseurl}}/docs/gitops-integrations/container-registries/github-cr/) + [JFrog Artifactory]({{site.baseurl}}/docs/gitops-integrations/container-registries/jfrog/) + [Quay]({{site.baseurl}}/docs/gitops-integrations/container-registries/quay/) 1. To test the connection to the container registry before committing the changes, click **Test Connection**. 1. To confirm, click **Commit**. It may take a few moments for the new integration to be synced to the cluster before it appears in the list. -### Integration resource in shared configuration repo +## Integration resource in shared configuration repo The integration resource for the container registry is created in the Git repository with the shared configuration, within `resources`. The exact location depends on whether the integration is shared with all or specific runtimes: * All runtimes: Created in `resources/all-runtimes-all-clusters/` * Selected runtimes: Created in `resources/runtimes//` -### View container registry integrations +## View container registry integrations for GitOps Selecting a container registry integration displays the existing integrations for that registry in Codefresh. The example below shows integrations for JFrog Artifactory. @@ -61,12 +61,12 @@ Every container registry integration displays the following information: * Runtime or runtimes it is shared with * Sync status -### Edit/delete container registry integrations +### Edit/delete container registry integrations for GitOps If you have existing integrations, you can change the connection details, or delete an integration. >Deleting an integration deletes the integration resource from the shared configuration Git repo, its secrets, the CI workflows that use it. -1. In the Codefresh UI, go to [Integrations](https://g.codefresh.io/2.0/account-settings/integrations){:target="\_blank"}. +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**GitOps Integrations**](https://g.codefresh.io/2.0/account-settings/integrations){:target="\_blank"}. 1. Filter by **Container Registry**, and select the specific container registry integration. 1. In the row with the integration to edit or delete, click the three dots and select **Edit** or **Delete**. 1. To edit, update the **Username** and **Password** fields, and click **Test Connection** to verify the account credentials. @@ -83,8 +83,8 @@ use it. %} ### Related articles -[CI integrations]({{site.baseurl}}/docs/integrations/ci-integrations/) -[Issue-tracking integrations]({{site.baseurl}}/docs/integrations/issue-tracking/) +[CI GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/) +[Issue-tracking GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/) [Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration/) [Images]({{site.baseurl}}/docs/deployment/images/) [Monitoring applications]({{site.baseurl}}/docs/deployment/applications-dashboard/) diff --git a/_docs/integrations/container-registries/amazon-ecr.md b/_docs/gitops-integrations/container-registries/amazon-ecr.md similarity index 73% rename from _docs/integrations/container-registries/amazon-ecr.md rename to _docs/gitops-integrations/container-registries/amazon-ecr.md index 17aee8611..37d4baba2 100644 --- a/_docs/integrations/container-registries/amazon-ecr.md +++ b/_docs/gitops-integrations/container-registries/amazon-ecr.md @@ -1,17 +1,17 @@ --- -title: "Amazon ECR" +title: "Amazon ECR GitOps integration" description: "" -group: integrations +group: gitops-integrations sub_group: container-registries toc: true --- Codefresh has native support for interacting with Amazon ECR (Elastic Container Registry), to push, pull, and deploy images. -For information on adding an Amazon ECR integration in Codefresh, see [Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/). +For information on adding an Amazon ECR integration for GitOps in Codefresh, see [Container registry integrations]({{site.baseurl}}/docs/gitops-integrations/container-registries/). ->Amazon ECR integration is supported only in hybrid runtimes. +>Amazon ECR integration is supported only for Hybrid GitOps. -### Prerequisites +## Prerequisites Before you configure settings in Codefresh to integrate Amazon ECR: * [Create an IAM (Identity and Access Management) role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html){:target="\_blank"} @@ -29,8 +29,8 @@ For example: ``` For detailed information, see [How Amazon Elastic Container Registry Works with IAM](https://docs.aws.amazon.com/AmazonECR/latest/userguide/security_iam_service-with-iam.html){:target="\_blank"} and the [AWS security blog](https://aws.amazon.com/blogs/security/how-to-use-trust-policies-with-iam-roles/){:target="\_blank"}. -### Amazon ECR integration settings in Codefresh -The table describes the arguments required to integrate Amazon ECR in Codefresh. +## Amazon ECR-GitOps integration settings in Codefresh +The table describes the arguments required for GitOps integrations with Amazon ECR in Codefresh. {: .table .table-bordered .table-hover} | Setting | Description | @@ -52,10 +52,11 @@ The table describes the arguments required to integrate Amazon ECR in Codefresh max-width="50%" %} -For how-to instructions, see [Configure container registry integrations in Codefresh]({{site.baseurl}}/docs/integrations/container-registries/#configure-container-registry-integrations-in-codefresh) and [Edit/delete container registry integrations in Codefresh]({{site.baseurl}}/docs/integrations/container-registries/#editdelete-container-registry-integrations). +For how-to instructions, see [Configure container registry integrations for GitOps in Codefresh]({{site.baseurl}}/docs/gitops-integrations/container-registries/#configure-container-registry-integrations-in-codefresh) and [Edit/delete container registry integrations for GitOps in Codefresh]({{site.baseurl}}/docs/gitops-integrations/container-registries/#editdelete-container-registry-integrations). -### Related articles + +## Related articles [Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration/) -[Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/) -[CI integrations]({{site.baseurl}}/docs/integrations/ci-integrations/) -[Issue-tracking]({{site.baseurl}}/docs/integrations/issue-tracking/) \ No newline at end of file +[Image enrichment with GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/image-enrichment-overview/) +[CI GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/) +[Issue-tracking GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/) \ No newline at end of file diff --git a/_docs/integrations/container-registries/dockerhub.md b/_docs/gitops-integrations/container-registries/dockerhub.md similarity index 74% rename from _docs/integrations/container-registries/dockerhub.md rename to _docs/gitops-integrations/container-registries/dockerhub.md index d74025c95..e6ee7a7eb 100644 --- a/_docs/integrations/container-registries/dockerhub.md +++ b/_docs/gitops-integrations/container-registries/dockerhub.md @@ -1,23 +1,23 @@ --- -title: "Docker Hub" +title: "Docker Hub GitOps integration" description: "" -group: integrations +group: gitops-integrations sub_group: container-registries toc: true --- Codefresh has native support for interacting with Docker Hub registries, to push, pull, and deploy images. -For information on adding a Docker Hub integration in Codefresh, see [Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/). +For information on adding a Docker Hub integration in Codefresh, see [Container registry integrations]({{site.baseurl}}/docs/gitops-integrations/container-registries/). -### Prerequisites +## Prerequisites Before you configure settings in Codefresh to integrate Docker Hub registry, do the following: * [Create an account or sign in to your account at Docker Hub](https://hub.docker.com/signup){:target="\_blank"} * (Optional) [Enable 2FA (Two-Factor Authentication)](https://docs.docker.com/docker-hub/2fa/){:target="\_blank"} * [Create a personal account token](https://docs.docker.com/docker-hub/access-tokens/){:target="\_blank"} -### Docker Hub integration settings in Codefresh -The table describes the arguments required to integrate Docker Hub to Codefresh. +## Docker Hub-GitOps integration settings in Codefresh +The table describes the arguments required for Docker Hub GitOps integration in Codefresh. {: .table .table-bordered .table-hover} | Setting | Description | @@ -39,11 +39,11 @@ The table describes the arguments required to integrate Docker Hub to Codefresh. max-width="50%" %} -For how-to instructions, see [Configure container registry integrations in Codefresh]({{site.baseurl}}/docs/integrations/container-registries/#configure-container-registry-integrations-in-codefresh) and [Edit/delete container registry integrations in Codefresh]({{site.baseurl}}/docs/integrations/container-registries/#editdelete-container-registry-integrations). +For how-to instructions, see [Configure container registry integrations for GitOps in Codefresh]({{site.baseurl}}/docs/gitops-integrations/container-registries/#configure-container-registry-integrations-in-codefresh) and [Edit/delete container registry integrations for GitOps in Codefresh]({{site.baseurl}}/docs/gitops-integrations/container-registries/#editdelete-container-registry-integrations). -### Related articles +## Related articles [Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration/) -[Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/) -[CI integrations]({{site.baseurl}}/docs/integrations/ci-integrations/) -[Issue-tracking integrations]({{site.baseurl}}/docs/integrations/issue-tracking/) +[Image enrichment with GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/image-enrichment-overview/) +[CI GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/) +[Issue-tracking GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/) diff --git a/_docs/integrations/container-registries/github-cr.md b/_docs/gitops-integrations/container-registries/github-cr.md similarity index 75% rename from _docs/integrations/container-registries/github-cr.md rename to _docs/gitops-integrations/container-registries/github-cr.md index f8aa77a79..48baa862c 100644 --- a/_docs/integrations/container-registries/github-cr.md +++ b/_docs/gitops-integrations/container-registries/github-cr.md @@ -1,15 +1,15 @@ --- -title: "GitHub Container Registry (GHCR)" +title: "GitHub Container Registry (GHCR) GitOps integration" description: "" -group: integrations +group: gitops-integrations sub_group: container-registries toc: true --- The GitHub Container registry allows you to host and manage your Docker container images in your personal or organisation account on GitHub. One of the benefits is that permissions can be defined for the Docker image independent from any repository. Thus, your repository could be private and your Docker image public. -For information on adding a GitHub Container registry integration in Codefresh, see [Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/). +For information on adding a GitHub Container registry integration in Codefresh, see [Container registry GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/container-registries/). -### Prerequisites +## Prerequisites Before you configure settings in Codefresh to integrate GitHub container registry: * Make sure you have a personal access token with the correct scopes or create one. You need at least the following scopes: @@ -21,7 +21,7 @@ Before you configure settings in Codefresh to integrate GitHub container registr For detailed information, see the [Authenticating to the Container registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-to-the-container-registry){:target="\_blank"}. -### GitHub Container registry (GHCR) integration settings in Codefresh +### GitHub Container registry (GHCR)-GitOps integration settings in Codefresh {: .table .table-bordered .table-hover} | Setting | Description | @@ -44,10 +44,10 @@ Before you configure settings in Codefresh to integrate GitHub container registr max-width="50%" %} -For how-to instructions, see [Configure container registry integrations in Codefresh]({{site.baseurl}}/docs/integrations/container-registries/#configure-container-registry-integrations-in-codefresh) and [Edit/delete container registry integrations in Codefresh]({{site.baseurl}}/docs/integrations/container-registries/#editdelete-container-registry-integrations). +For how-to instructions, see [Configure container registry integrations for GitOps in Codefresh]({{site.baseurl}}/docs/gitops-integrations/container-registries/#configure-container-registry-integrations-in-codefresh) and [Edit/delete container registry integrations for GitOps in Codefresh]({{site.baseurl}}/docs/gitops-integrations/container-registries/#editdelete-container-registry-integrations). -### Related articles +## Related articles [Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration/) -[Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/) -[CI integrations]({{site.baseurl}}/docs/integrations/ci-integrations/) -[Issue-tracking]({{site.baseurl}}/docs/integrations/issue-tracking/) +[Image enrichment with GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/image-enrichment-overview/) +[CI GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/) +[Issue-tracking GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/) diff --git a/_docs/integrations/container-registries/jfrog.md b/_docs/gitops-integrations/container-registries/jfrog.md similarity index 71% rename from _docs/integrations/container-registries/jfrog.md rename to _docs/gitops-integrations/container-registries/jfrog.md index ccb22dd0d..3aa108e7f 100644 --- a/_docs/integrations/container-registries/jfrog.md +++ b/_docs/gitops-integrations/container-registries/jfrog.md @@ -1,17 +1,17 @@ --- -title: "JFrog Artifactory" +title: "JFrog Artifactory GitOps integration" description: "" -group: integrations +group: gitops-integrations sub_group: container-registries toc: true --- Codefresh has native support for interacting with JFrog Artifactory. -For information on adding a JFrog Artifactory integration in Codefresh, see [Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/). +For information on adding a JFrog Artifactory integration in Codefresh, see [Container registry integrations]({{site.baseurl}}/docs/gitops-integrations/container-registries/). -### JFrog Artifactory integration settings in Codefresh +## JFrog Artifactory-GitOps integration settings in Codefresh {: .table .table-bordered .table-hover} | Setting | Description | @@ -34,10 +34,10 @@ For information on adding a JFrog Artifactory integration in Codefresh, see [Con max-width="50%" %} -For how-to instructions, see [Configure container registry integrations in Codefresh]({{site.baseurl}}/docs/integrations/container-registries/#configure-container-registry-integrations-in-codefresh) and [Edit/delete container registry integrations in Codefresh]({{site.baseurl}}/docs/integrations/container-registries/#editdelete-container-registry-integrations). +For how-to instructions, see [Configure container registry integrations for GitOps in Codefresh]({{site.baseurl}}/docs/gitops-integrations/container-registries/#configure-container-registry-integrations-in-codefresh) and [Edit/delete container registry integrations for GitOps in Codefresh]({{site.baseurl}}/docs/gitops-integrations/container-registries/#editdelete-container-registry-integrations). -### Related articles +## Related articles [Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration/) -[Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/) -[CI integrations]({{site.baseurl}}/docs/integrations/ci-integrations/) -[Issue-tracking]({{site.baseurl}}/docs/integrations/issue-tracking/) +[Image enrichment with GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/image-enrichment-overview/) +[CI GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/) +[Issue-tracking GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/) diff --git a/_docs/integrations/container-registries/quay.md b/_docs/gitops-integrations/container-registries/quay.md similarity index 71% rename from _docs/integrations/container-registries/quay.md rename to _docs/gitops-integrations/container-registries/quay.md index dfe9cf817..80d50d5f0 100644 --- a/_docs/integrations/container-registries/quay.md +++ b/_docs/gitops-integrations/container-registries/quay.md @@ -1,23 +1,22 @@ --- -title: "Quay" +title: "Quay GitOps integration" description: "" -group: integrations +group: gitops-integrations sub_group: container-registries toc: true --- Codefresh has native support for interacting with Quay registries, from where you can push, pull, and deploy images. -Adding a Quay integration allows you to reference the integration in external CI tools such as GitHub Actions by the name of the integration account, instead of adding explicit credentials. See [Image enrichment overview]({{site.baseurl}}/docs/integrations/image-enrichment-overview/) and [CI integrations]({{site.baseurl}}/docs/integrations/ci-integrations/). +Adding a Quay integration allows you to reference the integration in external CI tools such as GitHub Actions by the name of the integration account, instead of adding explicit credentials. See [Image enrichment with GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/image-enrichment-overview/) and [CI integrations for GitOps]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/). -### Prerequisites +## Prerequisites 1. [Create a Redhat/Quay account at Quay](https://quay.io/){:target="\_blank"}. 1. Optional. For Codefresh integration, [create a robot account](https://docs.quay.io/glossary/robot-accounts.html){:target="\_blank"}. -### Quay integration settings in Codefresh +## Quay-GitOps integration settings in Codefresh -The table describes the arguments required to integrate Quay in Codefresh. {: .table .table-bordered .table-hover} | Setting | Description | @@ -37,15 +36,15 @@ The table describes the arguments required to integrate Quay in Codefresh. max-width="50%" %} -For how-to instructions, see [Configure container registry integrations in Codefresh]({{site.baseurl}}/docs/integrations/container-registries/#configure-container-registry-integrations-in-codefresh) and [Edit/delete container registry integrations in Codefresh]({{site.baseurl}}/docs/integrations/container-registries/#editdelete-container-registry-integrations). +For how-to instructions, see [Configure container registry integrations for GitOps in Codefresh]({{site.baseurl}}/docs/gitops-integrations/container-registries/#configure-container-registry-integrations-in-codefresh) and [Edit/delete container registry integrations for GitOps in Codefresh]({{site.baseurl}}/docs/gitops-integrations/container-registries/#editdelete-container-registry-integrations). Make sure you have the: * Quay domain username * Quay domain-encrypted password or that of the robot account -### Related articles +## Related articles [Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration/) -[Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/) -[CI integrations]({{site.baseurl}}/docs/integrations/ci-integrations/) -[Issue-tracking integrations]({{site.baseurl}}/docs/integrations/issue-tracking/) +[Image enrichment with GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/image-enrichment-overview/) +[CI GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/) +[Issue-tracking GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/) diff --git a/_docs/integrations/image-enrichment-overview.md b/_docs/gitops-integrations/image-enrichment-overview.md similarity index 65% rename from _docs/integrations/image-enrichment-overview.md rename to _docs/gitops-integrations/image-enrichment-overview.md index cf2d9ca29..a19ec3239 100644 --- a/_docs/integrations/image-enrichment-overview.md +++ b/_docs/gitops-integrations/image-enrichment-overview.md @@ -1,7 +1,7 @@ --- -title: "Image enrichment with integrations" +title: "GitOps image enrichment with integrations" description: "" -group: integration +group: gitops-integrations toc: true --- @@ -16,38 +16,38 @@ Codefresh has new report images templates, optimized to work with third-party CI -### CI integration flow for image enrichment +## CI integration flow for image enrichment Integrate Codefresh with your CI platform/tool account with a unique name per integration account. -#### 1. Add/configure integration +### 1. Add/configure integration Add/configure the integration account for the third-party tools. You can set up multiple integration accounts for the same tool. When you add an integration, Codefresh creates a Sealed Secret with the integration credentials, and a ConfigMap that references the secret. See: * Issue tracking - [JIRA]({{site.baseurl}}/docs/integrations/issue-tracking/jira/) + [JIRA]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/jira/) * Container registries - [Amazon ECR]({{site.baseurl}}/docs/integrations/container-registries/amazon-ecr/) - [DockerHub]({{site.baseurl}}/docs/integrations/container-registries/dockerhub/) - [JFrog Artifactory]({{site.baseurl}}/docs/integrations/container-registries/jfrog/) - [Quay]({{site.baseurl}}/docs/integrations/container-registries/quay/) + [Amazon ECR]({{site.baseurl}}/docs/gitops-integrations/container-registries/amazon-ecr/) + [DockerHub]({{site.baseurl}}/docs/gitops-integrations/container-registries/dockerhub/) + [JFrog Artifactory]({{site.baseurl}}/docs/gitops-integrations/container-registries/jfrog/) + [Quay]({{site.baseurl}}/docs/gitops-integrations/container-registries/quay/) We are working on supporting integrations for more tools. Stay tuned for the release announcements. For image enrichment with a tool that is as yet unsupported, you must define the explicit credentials. -#### 2. Connect CI platform/tool to Codefresh +### 2. Connect CI platform/tool to GitOps -Connect a CI platform/tool to Codefresh with an API token for the runtime cluster, the integration accounts, and image information for enrichment and reporting. +Connect a CI platform/tool to Codefresh GitOps with an API token for the runtime cluster, the integration accounts, and image information for enrichment and reporting. -[Codefresh Classic]({{site.baseurl}}/docs/integrations/ci-integrations/codefresh-classic/) -[GitHub Actions]({{site.baseurl}}/docs/integrations/ci-integrations/github-actions/) -[Jenkins]({{site.baseurl}}/docs/integrations/ci-integrations/jenkins/) +[Codefresh Classic]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/codefresh-classic/) +[GitHub Actions]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/github-actions/) +[Jenkins]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/jenkins/) -#### 3. Add the enrichment step for the CI platform/tool to your GitHub Actions pipeline +### 3. Add the enrichment step for the CI platform/tool to your GitHub Actions pipeline Finally, add the enrichment step to your CI pipeline with the API token and integration information. Codefresh uses the integration name to get the corresponding Sealed Secret to securely access and retrieve the information for image enrichment. @@ -55,7 +55,7 @@ Finally, add the enrichment step to your CI pipeline with the API token and inte [Codefresh Classic Codefresh report image](https://codefresh.io/steps/step/codefresh-report-image){:target="\_blank"}. -#### 4. View enriched image information +### 4. View enriched image information Once deployed, view enriched information in the Codefresh UI: * Go to [Images](https://g.codefresh.io/2.0/images){:target="\_blank"} * Go to the [Applications dashboard](https://g.codefresh.io/2.0/applications-dashboard){:target="\_blank"} @@ -69,7 +69,7 @@ View: * Jira issues, status and details for each deployment -### Related articles +## Related articles [Images]({{site.baseurl}}/docs/deployment/images/) -[Applications dashboard]({{site.baseurl}}/docs/deployment/applications-dashboard/) +[Applications dashboard]({{site.baseurl}}/docs/gitops-deployment/applications-dashboard/) diff --git a/_docs/integrations/issue-tracking.md b/_docs/gitops-integrations/issue-tracking.md similarity index 72% rename from _docs/integrations/issue-tracking.md rename to _docs/gitops-integrations/issue-tracking.md index 60ea2d5a1..508dce8b4 100644 --- a/_docs/integrations/issue-tracking.md +++ b/_docs/gitops-integrations/issue-tracking.md @@ -1,14 +1,14 @@ --- -title: "Issue tracking integrations" +title: "GitOps issue tracking integrations" description: "" -group: integrations +group: gitops-integrations toc: true --- One of the major highlights of the Codefresh platform is the ability to automatically correlate software features with their deployment (where and when). While the software version of a component is easily identifiable, what is likely more interesting and important is to know which features are included in a release. -Adding an issue-tracking integration in Codefresh allows you to reference the integration in third-party CI platforms/tools such as GitHub Actions and Codefresh Classic by the name of the integration, instead of explicit credentials. See [Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/) and [CI integrations]({{site.baseurl}}/docs/integrations/ci-integrations/). +Adding an issue-tracking integration in Codefresh allows you to reference the integration in third-party CI platforms/tools such as GitHub Actions and Codefresh Classic by the name of the integration, instead of explicit credentials. See [Image enrichment with GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/image-enrichment-overview/) and [CI integrations for GitOps]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/). You add an issue-tracking integration in Codefresh by: * Defining the integration name @@ -17,15 +17,16 @@ You add an issue-tracking integration in Codefresh by: * Committing the changes Once added, Codefresh displays the list of existing integrations with their sync status. You can edit or delete any integration. -### Configure container registry integrations in Codefresh -Configure the settings for a container registry integration in Codefresh. -1. In the Codefresh UI, go to [Integrations](https://g.codefresh.io/2.0/account-settings/integrations){:target="\_blank"}. +## Configure issue tracking integrations for GitOps in Codefresh +Configure the settings for an issue tracking integration for GitOps in Codefresh. + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**GitOps Integrations**](https://g.codefresh.io/2.0/account-settings/integrations){:target="\_blank"}. 1. Filter by **Issue Tracking**, select the issue tracking tool to integrate, and click **Configure**. 1. Jira integrations only: For a new Jira integration, from the **Add Integration** dropdown, select the type of integration, as either **Deployment reporting** or **Image enrichment**. 1. If you already have integrations, click **Add**. 1. Define the arguments for the issue tracking tool: - [Jira]({{site.baseurl}}/docs/integrations/issue-tracking/jira/) + [Jira]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/jira/) 1. To confirm, click **Commit**. It may take a few moments for the new integration to be synced to the cluster before it appears in the list. @@ -35,7 +36,7 @@ The exact location depends on whether the integration is shared with all or spec * All runtimes: Created in `resources/all-runtimes-all-clusters/` * Selected runtimes: Created in `resources/runtimes/ /` -### View issue-tracking integrations +### View issue-tracking integrations for GitOps Selecting an issue tracking tool displays the existing integrations in Codefresh. @@ -44,12 +45,12 @@ Every issue tracking integration displays the following information: * Runtime or runtimes it is shared with * Sync status -### Edit/delete issue-tracking integrations in Codefresh +### Edit/delete issue-tracking integrations for GitOps in Codefresh If you have existing integrations, you can change the credentials, or delete an integration. >Deleting an integration deletes the integration resource from the shared configuration Git repo, its secrets, the CI workflows that use it. -1. In the Codefresh UI, go to [Integrations](https://g.codefresh.io/2.0/account-settings/integrations){:target="\_blank"}. +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**GitOps Integrations**](https://g.codefresh.io/2.0/account-settings/integrations){:target="\_blank"}. 1. Filter by **Issue Tracking**, and select the specific integration. 1. In the row with the integration to edit or delete, click the three dots and select **Edit** or **Delete**. 1. To edit, update the **Username** and **Password** fields, and click **Test Connection** to verify the account credentials. @@ -67,6 +68,6 @@ use it. ### Related articles [Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration/) -[CI integrations]({{site.baseurl}}/docs/integrations/ci-integrations/) -[Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/) +[CI GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/) +[Container registry GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/container-registries/) diff --git a/_docs/integrations/issue-tracking/jira.md b/_docs/gitops-integrations/issue-tracking/jira.md similarity index 81% rename from _docs/integrations/issue-tracking/jira.md rename to _docs/gitops-integrations/issue-tracking/jira.md index fd223281e..08475c334 100644 --- a/_docs/integrations/issue-tracking/jira.md +++ b/_docs/gitops-integrations/issue-tracking/jira.md @@ -1,7 +1,7 @@ --- title: "Jira" description: " " -group: integrations +group: gitops-integrations sub_group: issue-tracking toc: true --- @@ -9,10 +9,10 @@ toc: true Codefresh has native integration for Atlassian Jira, to enrich images with information from Jira. Codefresh can monitor a feature all the way from the ticket creation phase, up to when it is implemented and deployed to an environment. -For information on adding a Jira integration in Codefresh, see [Issue-tracking integrations]({{site.baseurl}}/docs/integrations/issue-tracking/). +For information on adding a Jira integration in Codefresh, see [Issue-tracking GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/). -### Prerequisites +## Prerequisites 1. Get your Jira instance credentials by following the [Atlassian documentation](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/){:target="\_blank"}. 1. Note down the following as you will need them to complete the integration with Codefresh: @@ -21,9 +21,8 @@ For information on adding a Jira integration in Codefresh, see [Issue-tracking i * Jira password/token created for this user -### Jira integration settings in Codefresh +## Jira-GitOps integration settings in Codefresh -The table describes the arguments required to integrate Jira in Codefresh. {: .table .table-bordered .table-hover} | Setting | Description | @@ -45,14 +44,14 @@ The table describes the arguments required to integrate Jira in Codefresh. max-width="60%" %} -For information on adding a Jira integration in Codefresh, see [Issue-tracking integrations]({{site.baseurl}}/docs/integrations/issue-tracking/). +For information on adding a Jira integration in Codefresh, see [Issue-tracking GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/issue-tracking/). -### Using Jira integration in pipelines +## Using Jira integration in pipelines For pipelines based on GitHub Actions, configure the Jira integration in Codefresh, and then connect your GitHub Action to Codefresh, referencing the Jira integration by name. Codefresh uses the Secret Key stored in the runtime cluster to securely access Jira and retrieve the information. -### Related articles +## Related articles [Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration/) -[Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/) -[CI integrations]({{site.baseurl}}/docs/integrations/ci-integrations/) -[Container registry integrations]({{site.baseurl}}/docs/integrations/container-registries/) +[Image enrichment with GitOps integrations]({{site.baseurl}}/docs/gitops-integrations/image-enrichment-overview/) +[CI integrations]({{site.baseurl}}/docs/gitops-integrations/ci-integrations/) +[Container registry integrations]({{site.baseurl}}/docs/gitops-integrations/container-registries/) diff --git a/_docs/incubation/intro-hosted-runtime.md b/_docs/incubation/intro-hosted-runtime.md deleted file mode 100644 index c2a66b694..000000000 --- a/_docs/incubation/intro-hosted-runtime.md +++ /dev/null @@ -1,146 +0,0 @@ ---- -title: "Hosted GitOps" -description: "" -group: incubation -toc: true ---- - - -Codefresh has enhanced our solution offering with Hosted GitOps, the SaaS version of Codefresh. - -What do you get with Hosted GitOps? -In a nutshell, a hosted and managed version of Argo CD. From application analytics, to application creation, rollout, and deployment, you get the best of both worlds: Argo CD with unique features and functionality from Codefresh to help achieve your CD goals. -What it also means is easy set up and zero maintenance overhead. - -Read on for more details. And check out our [blog](https://codefresh.io/blog/codefresh-upends-continuous-delivery-with-hosted-gitops-platform-featuring-dora-dashboards-and-first-class-integrations-for-ci/). - -### Hosted runtimes - -Setting up your hosted environment takes just a few clicks. All you need is a Codefresh account, a Git account, and a Kubernetes cluster to which to deploy your applications. -Codefresh guides you through the simple three-step process of provisioning your hosted runtime. From that point, Codefresh handles administration and maintenance of the hosted runtime, including version and security updates. - -See [Set up a hosted (Hosted GitOps) environment]({{site.baseurl}}/docs/runtime/hosted-runtime/). - -{% include - image.html - lightbox="true" - file="/images/runtime/intro-hosted-hosted-initial-view.png" - url="/images/runtime/intro-hosted-hosted-initial-view.png" - alt="Hosted runtime setup" - caption="Hosted runtime setup" - max-width="80%" -%} - -### Global deployment analytics - -The Home dashboard presents enterprise-wide deployment highlights, making it a useful management tool. - -Get insights into important KPIs and deployments, across runtimes and clusters, all in the same location. View status of runtimes and managed clusters, deployments, failed deployments with rollbacks, most active applications. Use filters to narrow the scope to focus on anything specific. - -{% include - image.html - lightbox="true" - file="/images/incubation/home-dashboard.png" - url="/images/incubation/home-dashboard.png" - alt="Global deployment analytics" - caption="Global deployment analytics" - max-width="80%" -%} - -### Application analytics and analysis - -The Applications dashboard displays applications across runtimes and clusters, from which you can select and analyze individual applications. Individual application information is grouped by current and historical deployments, enriched with Argo, Jira, and Git details, including rollout visualizations for ongoing deployments (Timeline tab), and an interactive tree view of application resources (Current State tab). - -See [Monitoring applications]({{site.baseurl}}/docs/deployment/applications-dashboard/). - -{% include - image.html - lightbox="true" - file="/images/applications/app-dashboard-main-view.png" - url="/images/applications/app-dashboard-main-view.png" - alt="Applications dashboard" - caption="Applications dashboard" - max-width="80%" -%} - -### DORA metrics - -DORA metrics has become integral to enterprises wanting to quantify DevOps performance, and Codefresh has out-of-the-box support for it. - -Apart from the metrics themselves, the DORA dashboard in Codefresh has several features such as the Totals bar with key metrics, filters that allow you to pinpoint just which applications or runtimes are contributing to problematic metrics, and the ability to set a different view granularity for each DORA metric. - -See [DORA metrics]({{site.baseurl}}/docs/reporting/dora-metrics/). - -{% include - image.html - lightbox="true" - file="/images/incubation/intro-dora-metrics.png" - url="/images/incubation/intro-dora-metrics.png" - alt="DORA metrics" - caption="DORA metrics" - max-width="60%" -%} - -### Application management - -Manage the application lifecycle in the Codefresh UI, from creating, editing, and deleting them. - -Define all application settings in a single location through the intuitive Form mode or directly in YAML, and commit all changes to Git. -For easy access, the configuration settings are available for editing in the Applications dashboard. - -See [Applications]({{site.baseurl}}/docs/deployment/create-application/). - -{% include - image.html - lightbox="true" - file="/images/applications/add-app-general-settings.png" - url="/images/applications/add-app-general-settings.png" - alt="Application creation in Codefresh" - caption="Application creation in Codefresh" - max-width="60%" -%} - -### Third-party CI integrations - -If you have your own tools for CI pipelines and workflows, Hosted GitOps gives you the option to connect them to Codefresh and enrich deployment information with our new report image template. For example, you can add the report image step in your GitHub Actions pipeline and reference the different integrations for Codefresh to retrieve and enrich the image information. - -* Git PRs (Pull Requests), Commits, and Committer information directly from the code repositories -* Jira ticket information for correlation with deployed features -* Docker Hub or Quay for image information - -See [Image enrichment with integrations]({{site.baseurl}}/docs/integrations/image-enrichment-overview/). - -{% include - image.html - lightbox="true" - file="/images/incubation/github-action-int-settings.png" - url="/images/incubation/github-action-int-settings.png" - alt="Image enrichment with GitHub Actions integration" - caption="Image enrichment with GitHub Actions integration" - max-width="60%" -%} - -### Hosted vs. hybrid environments - -The table below highlights the main differences between hosted and hybrid environments. - -{: .table .table-bordered .table-hover} -| Functionality |Feature | Hosted | Hybrid | -| -------------- | --------------|--------------- | --------------- | -| Runtime | Installation | Provisioned by Codefresh | Provisioned by customer | -| | Runtime cluster |Managed by Codefresh | Managed by customer | -| | Number per account | Only one runtime | Multiple runtimes | -| | Upgrade | Managed by Codefresh | Managed by customer | -| | External cluster | Managed by customer | Managed by customer | -| | Uninstall | Managed by customer | Managed by customer | -| Argo CD | | Codefresh cluster | Customer cluster | -| CI Ops | Delivery Pipelines |Not supported | Supported | -| |Workflows | Not supported | Supported | -| |Workflow Templates | Not supported | Supported | -| CD Ops |Applications | Supported | Supported | -| |Image enrichment | Supported | Supported | -| | Rollouts | Supported | Supported | -|Integrations | | Supported | Supported | -|Dashboards |Home Analytics | Hosted runtime and deployments|Runtimes, deployments, Delivery Pipelines | -| |DORA metrics | Supported |Supported | -| |Applications | Supported |Supported | diff --git a/_docs/installation/codefresh-on-prem-upgrade.md b/_docs/installation/codefresh-on-prem-upgrade.md new file mode 100644 index 000000000..335b30755 --- /dev/null +++ b/_docs/installation/codefresh-on-prem-upgrade.md @@ -0,0 +1,575 @@ +--- +title: "Codefresh On-Premises Upgrade" +description: "Use the Kubernetes Codefresh Installer to upgrade your Codefresh On-Premises platform " +group: installation +redirect_from: + - /docs/enterprise/codefresh-on-prem-upgrade/ +toc: true +--- +Upgrade the Codefresh On-premises platform to the latest version: +* Prepare for the upgrade: _Before_ the upgrade, based on the version you are upgrading to, complete the required tasks +* Upgrade On-premises +* Complete post-upgrade configuration: If needed, also based on the version you are upgrading to, complete the required tasks + + +## Upgrade to 1.1.1 +Prepare for the upgrade to v1.1.1 by performing the tasks listed below. + +### Maintain backward compatibility for infrastructure services +If you have Codefresh version 1.0.202 or lower installed, and are upgrading to v1.1.1, to retain the existing images for the services listed below, update the `config.yaml` for `kcfi`. + +* `cf-mongodb` +* `cf-redis` +* `cf-rabbitmq` +* `cf-postgresql` +* `cf-nats` +* `cf-consul` + +> In the `config.yaml`, as in the example below, if needed, replace the `bitnami` prefix with that of your private repo. + +```yaml +... + +global: + ### Codefresh App domain name. appUrl is manadatory parameter + appUrl: onprem.mydomain.com + appProtocol: https + + mongodbImage: bitnami/mongodb:3.6.13-r0 # (default `mongodbImage: bitnami/mongodb:4.2`) + +mongodb: + image: bitnami/mongodb:3.6.13-r0 # (default `image: bitnami/mongodb:4.2`) + podSecurityContext: + enabled: true + runAsUser: 0 + fsGroup: 0 + containerSecurityContext: + enabled: false + +redis: + image: bitnami/redis:3.2.9-r2 # (default `image: bitnami/redis:6.0.16`) + podSecurityContext: + enabled: false + containerSecurityContext: + enabled: false + +postgresql: + imageTag: 9.6.2 # (default `imageTag:13`) + +nats: + imageTag: 0.9.4 # (default `imageTag:2.7`) + +consul: + ImageTag: 1.0.0 # (default `imageTag:1.11`) +... +``` +## Upgrade to 1.2.0 and higher +This major release **deprecates** the following Codefresh managed charts: +* Ingress +* Rabbitmq +* Redis + +See the instructions below for each of the affected charts. + +> Before the upgrade remove any seed jobs left from previous release with: + `kubectl delete job --namespace ${CF_NAMESPACE} -l release=cf ` + +> Before the upgrade remove PDBs for Redis and RabbitMQ left from previous release with: + `kubectl delete pdb cf-rabbitmq --namespace ${CF_NAMESPACE}`
+ `kubectl delete pdb cf-redis --namespace ${CF_NAMESPACE}` + +### Update configuration for Ingress chart +From version **1.2.0 and higher**, we have deprecated support for `Codefresh-managed-ingress`. +Kubernetes community public `ingress-nginx` chart replaces `Codefresh-managed-ingress` chart. For more information on the `ingress-nginx`, see [kubernetes/ingress-nginx](https://github.com/kubernetes/ingress-nginx). + +> Parameter locations have changed as the ingress chart name was changed from `ingress` to `ingress-nginx`: + **NGINX controller** parameters are now defined under `ingress-nginx` + **Ingress object** parameters are now defined under `ingress` + +You must update `config.yaml`, if you are using: +* External ingress controllers, including ALB (Application Load Balancer) +* Codefresh-managed ingress controller with _custom_ values + +#### Update configuration for external ingress controllers + +For external ingress controllers, including ALB (Application Load Balancer), update the relevant sections in `config.yaml` to align with the new name for the ingress chart: + +* Replace `ingress` with `ingress-nginx` + +*v1.1.1 or lower* +```yaml +ingress: #disables creation of both Nginx controller deployment and Ingress objects + enabled: false +``` + +*v1.2.2 or higher* +```yaml +ingress-nginx: #disables creation of Nginx controller deployment + enabled: false + +ingress: #disables creation of Ingress objects (assuming you've manually created ingress resource before) + enabled: false +``` + +* Replace `annotations` that have been deprecated with `ingressClassName` + +*v1.1.1 or lower* +```yaml +ingress: + annotations: + kubernetes.io/ingress.class: my-non-codefresh-nginx +``` + +*v1.2.2 or higher* +```yaml +ingress-nginx: + enabled: false + +ingress: + ingressClassName: my-non-codefresh-nginx +### `kubernetes.io/ingress.class` annotation is deprecated from kubernetes v1.22+. +# annotations: +# kubernetes.io/ingress.class: my-non-codefresh-nginx +``` + +#### Update configuration for Codefresh-managed ingress with custom values + +If you were running `Codefresh-managed ingress` controller with _custom_ values refer to [values.yaml](https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml) from the official repo. If needed, update the `ingress-nginx` section in `config.yaml`. The example below shows the default values (already provided in Codefresh chart) for `ingress-nginx`: + +```yaml +ingress-nginx: + enabled: true + controller: + ## This section refers to the creation of the IngressClass resource + ## IngressClass resources are supported since k8s >= 1.18 and required since k8s >= 1.19 + ingressClassResource: + # -- Is this ingressClass enabled or not + enabled: true + # -- Is this the default ingressClass for the cluster + default: false + # -- Controller-value of the controller that is processing this ingressClass + controllerValue: "k8s.io/ingress-nginx-codefresh" + # -- Name of the ingressClass + name: nginx-codefresh + # -- For backwards compatibility with ingress.class annotation. + # Algorithm is as follows, first ingressClassName is considered, if not present, controller looks for ingress.class annotation + ingressClass: nginx-codefresh + # -- Process IngressClass per name (additionally as per spec.controller). + ingressClassByName: true + # Limit the scope of the controller to a specific namespace + scope: + # -- Enable 'scope' or not + enabled: true + admissionWebhooks: + enabled: false +``` +> New `ingress-nginx` subchart creates a new `cf-ingress-nginx-controller` service (`type: LoadBalancer`) instead of old `cf-ingress-controller` service. So make sure to update DNS record for `global.appUrl` to point to a new external load balancer IP. + You can get external load balancer IP with: + `kubectl get svc cf-ingress-nginx-controller -o jsonpath={.status.loadBalancer.ingress[0].ip` + + +### Update configuration for RabbitMQ chart +From version **1.2.2 and higher**, we have deprecated support for the `Codefresh-managed Rabbitmq` chart. Bitnami public `bitnami/rabbitmq` chart has replaced the `Codefresh-managed rabbitmq`. For more information, see [bitnami/rabbitmq](https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq). + +> Configuration updates are not required if you are running an **external** RabbitMQ service. + +> RabbitMQ chart was replaced so as a consequence values structure might be different for some parameters. + For the complete list of values, see [values.yaml](https://github.com/bitnami/charts/blob/master/bitnami/rabbitmq/values.yaml) + +**`existingPvc` changed to `existingClaim` and defined under `persistence`** + +*v1.1.1 or lower* +```yaml +rabbitmq: + existingPvc: my-rabbitmq-pvc + nodeSelector: + foo: bar + resources: + limits: + cpu: 2000m + memory: 2Gi + requests: + cpu: 500m + memory: 1Gi + tolerations: + - effect: NoSchedule + key:+ operator: Equal + value: +``` + +*v1.2.2 or higher* +```yaml +rabbitmq: + volumePermissions: ## Enable init container that changes the owner and group of the persistent volume from existing claim + enabled: true + persistence: + existingClaim: my-rabbitmq-pvc + nodeSelector: + foo: bar + resources: + limits: + cpu: 2000m + memory: 2Gi + requests: + cpu: 500m + memory: 1Gi + tolerations: + - effect: NoSchedule + key: + operator: Equal + value: +``` + +**`storageClass` and `size` defined under `persistence`** + +*v1.1.1 or lower* +```yaml +rabbitmq: + storageClass: my-storage-class + storageSize: 32Gi +``` + +*v1.2.2 or higher* +```yaml +rabbitmq: + persistence: + storageClass: my-storage-class + size: 32Gi +``` + +### Update configuration for Redis chart +From version **1.2.2 and higher**, we have deprecated support for the `Codefresh-managed Redis` chart. Bitnami public `bitnami/redis` chart has replaced the `Codefresh-managed Redis` chart. For more information, see [bitnami/redis](https://github.com/bitnami/charts/tree/master/bitnami/redis). + +Redis storage contains **CRON and Registry** typed triggers so you must migrate existing data from the old deployment to the new stateful set. +This is done by backing up the existing data before upgrade, and then restoring the backed up data after upgrade. + +> Configuration updates are not required: + * When running an **external** Redis service. + * If CRON and Registy triggers have not been configured. + +#### Verify existing Redis data for CRON and Registry triggers +Check if you have CRON and Registry triggers configured in Redis. + +* Run `codefresh get triggers` + OR + Directly from the K8s cluster where Codefresh is installed. + +```shell +NAMESPACE=codefresh +REDIS_PASSWORD=$(kubectl get secret --namespace $NAMESPACE cf-redis -o jsonpath="{.data.redis-password}" | base64 --decode) + +kubectl exec -it deploy/cf-redis -- env REDIS_PASSWORD=$REDIS_PASSWORD bash +#once inside cf-redis pod +REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli +info keyspace # list db +select 15 # select db 15 +keys * #show keys +``` + +* If there are results, continue with _Back up existing Redis data_. + +#### Back up existing Redis data +Back up the existing data before the upgrade: + +* Connect to the pod, run `redis-cli`, export AOF data from old `cf-redis-*` pod: + +```shell +NAMESPACE=codefresh +REDIS_PASSWORD=$(kubectl get secret --namespace $NAMESPACE cf-redis -o jsonpath="{.data.redis-password}" | base64 --decode) +REDIS_POD=$(kubectl get pods -l app=cf-redis -o custom-columns=:metadata.name --no-headers=true) +kubectl cp $REDIS_POD:/bitnami/redis/data/appendonly.aof appendonly.aof -c cf-redis +``` + +#### Restore backed-up Redis data +Restore the data after the upgrade: + +* Copy `appendonly.aof` to the new `cf-redis-master-0` pod: + + ```shell + kubectl cp appendonly.aof cf-redis-master-0:/data/appendonly.aof + ```` +* Restart `cf-redis-master-0` and `cf-api` pods: + + ```shell + kubectl delete pod cf-redis-master-0 + + kubectl scale deployment cf-cfapi-base --replicas=0 -n codefresh + kubectl scale deployment cf-cfapi-base --replicas=2 -n codefresh + ``` + +> Redis chart was replaced so as a consequence values structure might be different for some parameters. + For the complete list of values, see [values.yaml](https://github.com/bitnami/charts/blob/master/bitnami/redis/values.yaml). + +**`existingPvc` changed to `existingClaim` and defined under `master.persistence`** + +*v1.1.1 or lower* +```yaml +redis: + existingPvc: my-redis-pvc + nodeSelector: + foo: bar + resources: + limits: + cpu: 1000m + memory: 1Gi + requests: + cpu: 500m + memory: 500Mi + tolerations: + - effect: NoSchedule + key: + operator: Equal + value: +``` + +*v1.2.2 or higher* +```yaml +redis: + volumePermissions: ## Enable init container that changes the owner and group of the persistent volume from existing claim + enabled: true + master: + persistence: + existingClaim: my-redis-pvc + nodeSelector: + foo: bar + resources: + limits: + cpu: 1000m + memory: 1Gi + requests: + cpu: 500m + memory: 500Mi + tolerations: + - effect: NoSchedule + key: + operator: Equal + value: +``` + +**`storageClass` and `size` defined under `master.persistence`** + + +*v1.1.1 or lower* +```yaml +redis: + storageClass: my-storage-class + storageSize: 32Gi +``` + +*v1.2.2 or higher* +```yaml +redis: + master: + persistence: + storageClass: my-storage-class + size: 32Gi +``` + +> If you run the upgrade without redis backup and restore procedure, **Helm Releases Dashboard** page might be empty for a few minutes after the upgrade. + +## Upgrade to 1.3.0 and higher +This major release **deprecates** the following Codefresh managed charts: +* Consul +* Nats + +### Update configuration for Consul +From version **1.3.0 and higher**, we have deprecated the Codefresh-managed `consul` chart, in favor of Bitnami public `bitnami/consul` chart. For more information, see [bitnami/consul](https://github.com/bitnami/charts/tree/master/bitnami/consul). + +Consul storage contains data about **Windows** worker nodes, so if you had any Windows nodes connected to your OnPrem installation, see the following instruction: + +> Use `https:// /admin/nodes` to check for any existing Windows nodes. + +#### Back up existing consul data +_Before starting the upgrade_, back up existing data. + +> Because `cf-consul` is a StatefulSet and has some immutable fields in its spec with both old and new charts having the same names, you cannot perform a direct upgrade. + Direct upgrade will most likely fail with: + `helm.go:84: [debug] cannot patch "cf-consul" with kind StatefulSet: StatefulSet.apps "cf-consul" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy' and 'minReadySeconds' are forbidden` + After backing up existing data, you must delete the old StatefulSet. + + +1. Exec into the consul pod and create a snapshot: +```shell +kubectl exec -it cf-consul-0 -n codefresh -- consul snapshot save backup.snap +``` +1. Copy snapshot locally: +```shell +kubectl cp -n codefresh cf-consul-0:backup.snap backup.snap +``` +1. **Delete the old** `cf-consul` stateful set: + +```shell +kubectl delete statefulset cf-consul -n codefresh +``` + +#### Restore backed up data + +After completing the upgrade to the current version, restore the `consul` data that you backed up. + +1. Copy the snapshot back to the new pod: + +```shell +kubectl cp -n codefresh backup.snap cf-consul-0:/tmp/backup.snap +``` +1. Restore the data: +``` +kubectl exec -it cf-consul-0 -n codefresh -- consul snapshot restore /tmp/backup.snap +``` +> Consul chart was replaced, and values structure might be different for some parameters. + For the complete list of values, see [values.yaml](https://github.com/bitnami/charts/blob/master/bitnami/consul/values.yaml) + + +### Update Nats configuration +From version **1.3.0 and higher**, we have deprecated Codefresh-managed `nats` chart in favor of Bitnami public `bitnami/nats` chart. For more information, see [bitnami/nats](https://github.com/bitnami/charts/tree/master/bitnami/consul). + +> Because `cf-nats` is a StatefulSet and has some immutable fields in its spec, both the old and new charts have the same names, preventing a direct upgrade. + Direct upgrade will most likely fail with: + `helm.go:84: [debug] cannot patch "cf-nats" with kind StatefulSet: StatefulSet.apps "cf-nats" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy' and 'minReadySeconds' are forbidden` + After backing up existing data, you must delete the old StatefulSet. + + +* **Delete the old** `cf-nats` stateful set. + +```shell +kubectl delete statefulset cf-nats -n codefresh +``` + +> Nats chart was replaced, and values structure might be different for some parameters. + For the complete list of values, see [values.yaml](https://github.com/bitnami/charts/blob/master/bitnami/nats/values.yaml). + +### Upgrade to 1.3.1 and higher + +Chart **v1.3.1** fixes duplicated env vars `CLUSTER_PROVIDERS_URI` and `CLUSTER_PROVIDERS_PORT` in `cf-api` deployment. +```yaml +W1010 03:03:55.553842 280 warnings.go:70] spec.template.spec.containers[0].env[94].name: duplicate name "CLUSTER_PROVIDERS_URI" +W1010 03:03:55.553858 280 warnings.go:70] spec.template.spec.containers[0].env[95].name: duplicate name "CLUSTER_PROVIDERS_PORT" +``` + + +> Due to Helm issue [Removal of duplicate array entry removes completely from Kubernetes](https://github.com/helm/helm/issues/10741), you shoud run `kcfi deploy` or `helm upgrade` two times consecutively. + + +With chart **v1.3.1** [insecure registy](https://docs.docker.com/registry/insecure/) property has been moved under `builder` section: + +```yaml +builder: + insecureRegistries: + - "myregistrydomain.com:5000" +``` + +## Upgrade the Codefresh Platform with [kcfi](https://github.com/codefresh-io/kcfi) + +1. Locate the `config.yaml` file you used in the initial installation. +1. Change the release number inside it. + ```yaml + metadata: + kind: codefresh + installer: + type: helm + helm: + chart: codefresh + repoUrl: https://chartmuseum.codefresh.io/codefresh + version: 1.2.14 + ``` +1. Perform a dry run and verify that there are no errors: + `kcfi deploy --dry-run --debug -c codefresh/config.yaml` +1. Run the actual upgrade: + `kcfi deploy --debug -c codefresh/config.yaml` +1. Verify that all the pods are are in running state: + `kubectl -n codefresh get pods --watch` +1. Log in to the Codefresh UI, and check the new version. +1. If needed, enable/disable new feature flags. + +## Codefresh with Private Registry + +If you install/upgrade Codefresh on the air-gapped environment (without access to public registries or Codefresh Enterprise registry) you will have to copy the images to your organization container registry. + +**Obtain [image list](https://github.com/codefresh-io/onprem-images/tree/master/releases) for specific release** + +**Push images to private docker registry** + +There are 3 types of images: + +> localhost:5000 is your
Note: Maintenance support for Kubernetes v1.19 ended on Oct 28, 2021.{:/}| +|Operating systems|{::nomarkdown}- Windows 10/7
- Linux
- OSX {:/}| +|Node requirements| {::nomarkdown}
- Memory: 5000 MB
- CPU: 2
- GitHub: SaaS and on-premises versions
- Bitbucket: SaaS and Bitbucket server (on-premises) 5.4.0 version and above
- GitLab: SaaS and on-premise versions (API v4 only)
- Single node: 8 CPU core and 16GB RAM
- Multi node: master(s) + 3 nodes with 4 CPU core and 8GB RAM (24 GB in total)
- GitHub
- Bitbucket Cloud
- Ambassador {:/}(see [Ambassador ingress configuration](#ambassador-ingress-configuration)){::nomarkdown}
- AWS ALB (Application Load Balancer) {:/} (see [AWS ALB ingress configuration](#aws-alb-ingress-configuration)){::nomarkdown}
- Istio {:/} (see [Istio ingress configuration](#istio-ingress-configuration)){::nomarkdown}
- NGINX Enterprise (nginx.org/ingress-controller) {:/} (see [NGINX Enterprise ingress configuration](#nginx-enterprise-ingress-configuration)){::nomarkdown}
- NGINX Community (k8s.io/ingress-nginx) {:/} (see [NGINX Community ingress configuration](#nginx-community-version-ingress-configuration)){::nomarkdown}
- Trafik {:/}(see [Traefik ingress configuration](#traefik-ingress-configuration))| +|Node requirements| {::nomarkdown}
- Memory: 5000 MB
- CPU: 2
- GitHub
- GitHub Enterprise
- GitLab Cloud
- GitLab Server
- Bitbucket Cloud
- Bitbucket Server
- Valid expiration date
- Scopes:
- GitHub and GitHub Enterprise: repo, admin-repo.hook
- GitLab Cloud and GitLab Server: api, read_repository
- Bitbucket Cloud and Server: Permissions: Read, Workspace membership: Read, Webhooks: Read and write, Repositories: Write, Admin
--api-url ` + `cf integration git register default --runtime --token ` + +{::nomarkdown} + +{:/} + +### Istio ingress configuration +For detailed configuration information, see [Istio ingress controller documentation](https://istio.io/latest/docs/tasks/traffic-management/ingress/kubernetes-ingress){:target="\_blank}. + +The table below lists the specific configuration requirements for Codefresh. + +{: .table .table-bordered .table-hover} +| What to configure | When to configure | +| -------------- | -------------- | +|Valid external IP address |_Before_ installing hybrid runtime | +|Valid TLS certificate| | +|TCP support | | +|Cluster routing service | _After_ installing hybrid runtime | + +{::nomarkdown} + +{:/} + +#### Valid external IP address +Run `kubectl get svc -A` to get a list of services and verify that the `EXTERNAL-IP` column for your ingress controller shows a valid hostname. + +{::nomarkdown} + +{:/} + +#### Valid TLS certificate +For secure runtime installation, the ingress controller must have a valid TLS certificate. +> Use the FQDN (Fully Qualified Domain Name) of the ingress controller for the TLS certificate. + +{::nomarkdown} + +{:/} + +#### TCP support +Configure the ingress controller to handle TCP requests. + +{::nomarkdown} + +{:/} + + + +#### Cluster routing service +> The cluster routing service must be configured _after_ installing the hybrid runtime. + +Based on the runtime version, you need to configure a single or multiple `VirtualService` resources for the `app-proxy`, `webhook`, and `workflow` services. + +##### Runtime version 0.0.543 or higher +Configure a single `VirtualService` resource to route traffic to the `app-proxy`, `webhook`, and `workflow` services, as in the example below. + +```yaml +apiVersion: networking.istio.io/v1alpha3 +kind: VirtualService +metadata: + namespace: pov-codefresh-istio-runtime # replace with your runtime name + name: internal-router +spec: + hosts: + - pov-codefresh-istio-runtime.sales-dev.codefresh.io # replace with your host name + gateways: + - istio-system/internal-router # replace with your gateway name + http: + - match: + - uri: + prefix: /webhooks + route: + - destination: + host: internal-router + port: + number: 80 + - match: + - uri: + prefix: /app-proxy + route: + - destination: + host: internal-router + port: + number: 80 + - match: + - uri: + prefix: /workflows + route: + - destination: + host: internal-router + port: + number: 80 +``` + +##### Runtime version 0.0.542 or lower + +Configure two different `VirtualService` resources, one to route traffic to the `app-proxy`, and the second to route traffic to the `webhook` services, as in the examples below. + +{::nomarkdown} + +{:/} + +**`VirtualService` example for `app-proxy`:** + +```yaml +apiVersion: networking.istio.io/v1alpha3 +kind: VirtualService +metadata: + namespace: test-runtime3 # replace with your runtime name + name: cap-app-proxy +spec: + hosts: + - my.support.cf-cd.com # replace with your host name + gateways: + - my-gateway # replace with your host name + http: + - match: + - uri: + prefix: /app-proxy + route: + - destination: + host: cap-app-proxy + port: + number: 3017 +``` + +**`VirtualService` example for `webhook`:** + +> Configure a `uri.prefix` and `destination.host` for each event-source if you have more than one. + +```yaml +apiVersion: networking.istio.io/v1alpha3 +kind: VirtualService +metadata: + namespace: test-runtime3 # replace with your runtime name + name: csdp-default-git-source +spec: + hosts: + - my.support.cf-cd.com # replace with your host name + gateways: + - my-gateway # replace with your gateway name + http: + - match: + - uri: + prefix: /webhooks/test-runtime3/push-github # replace `test-runtime3` with your runtime name, and `push-github` with the name of your event source + route: + - destination: + host: push-github-eventsource-svc # replace `push-github' with the name of your event source + port: + number: 80 + - match: + - uri: + prefix: /webhooks/test-runtime3/cypress-docker-images-push # replace `test-runtime3` with your runtime name, and `cypress-docker-images-push` with the name of your event source + route: + - destination: + host: cypress-docker-images-push-eventsource-svc # replace `cypress-docker-images-push` with the name of your event source + port: + number: 80 +``` + +{::nomarkdown} + +{:/} + +### NGINX Enterprise ingress configuration + +For detailed configuration information, see [NGINX ingress controller documentation](https://docs.nginx.com/nginx-ingress-controller){:target="\_blank}. + +The table below lists the specific configuration requirements for Codefresh. + +{: .table .table-bordered .table-hover} +| What to configure | When to configure | +| -------------- | -------------- | +|Verify valid external IP address |_Before_ installing hybrid runtime | +|Valid TLS certificate | | +|TCP support| | +|NGINX Ingress: Enable report status to cluster | | +|NGINX Ingress Operator: Enable report status to cluster| | +|Patch certificate secret |_After_ installing hybrid runtime + +{::nomarkdown} + +{:/} + +#### Valid external IP address +Run `kubectl get svc -A` to get a list of services and verify that the `EXTERNAL-IP` column for your ingress controller shows a valid hostname. + +{::nomarkdown} + +{:/} + +#### Valid TLS certificate +For secure runtime installation, the ingress controller must have a valid TLS certificate. +> Use the FQDN (Fully Qualified Domain Name) of the ingress controller for the TLS certificate. + +{::nomarkdown} + +{:/} + +#### TCP support +Configure the ingress controller to handle TCP requests. + +{::nomarkdown} + +{:/} + +#### NGINX Ingress: Enable report status to cluster + +If the ingress controller is not configured to report its status to the cluster, Argo’s health check reports the health status as “progressing” resulting in a timeout error during installation. + +* Pass `--report-ingress-status` to `deployment`. + +```yaml +spec: + containers: + - args: + - --report-ingress-status +``` + +{::nomarkdown} + +{:/} + +#### NGINX Ingress Operator: Enable report status to cluster + +If the ingress controller is not configured to report its status to the cluster, Argo’s health check reports the health status as “progressing” resulting in a timeout error during installation. + +1. Add this to the `Nginxingresscontrollers` resource file: + + ```yaml + ... + spec: + reportIngressStatus: + enable: true + ... + ``` + +1. Make sure you have a certificate secret in the same namespace as the runtime. Copy an existing secret if you don't have one. +You will need to add this to the `ingress-master` when you have completed runtime installation. + +{::nomarkdown} + +{:/} + +#### Patch certificate secret +> The certificate secret must be configured _after_ installing the hybrid runtime. + +Patch the certificate secret in `spec.tls` of the `ingress-master` resource. +The secret must be in the same namespace as the runtime. + +1. Go to the runtime namespace with the NGINX ingress controller. +1. In `ingress-master`, add to `spec.tls`: + + ```yaml + tls: + - hosts: + - + secretName: + ``` + +{::nomarkdown} + +{:/} + +### NGINX Community version ingress configuration + +Codefresh has been tested with and supports implementations of the major providers. For your convenience, we have provided configuration instructions, both for supported and untested providers in [Provider-specific configuration](#provider-specific-configuration). + + +This section lists the specific configuration requirements for Codefresh to be completed _before_ installing the hybrid runtime. +* Verify valid external IP address +* Valid TLS certificate +* TCP support + +{::nomarkdown} + +{:/} + +#### Valid external IP address +Run `kubectl get svc -A` to get a list of services, and verify that the `EXTERNAL-IP` column for your ingress controller shows a valid hostname. + +{::nomarkdown} + +{:/} + +#### Valid TLS certificate +For secure runtime installation, the ingress controller must have a valid TLS certificate. +> Use the FQDN (Fully Qualified Domain Name) of the ingress controller for the TLS certificate. + +{::nomarkdown} + +{:/} + +#### TCP support +Configure the ingress controller to handle TCP requests. + +Here's an example of TCP configuration for NGINX Community on AWS. +Verify that the `ingress-nginx-controller` service manifest has either of the following annotations: + +`service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"` +OR +`service.beta.kubernetes.io/aws-load-balancer-type: nlb` + +{::nomarkdown} + +{:/} + +#### Provider-specific configuration + +> The instructions are valid for `k8s.io/ingress-nginx`, the community version of NGINX. + + ++AWS
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/aws/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ +Azure (AKS)
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ +Bare Metal Clusters
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/baremetal/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ +Digital Ocean
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/do/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ +Docker Desktop
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
+Note: By default, Docker Desktop services will provision with localhost as their external address. Triggers in delivery pipelines cannot reach this instance unless they originate from the same machine where Docker Desktop is being used. + +++ + +Exoscale
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/exoscale/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ + +Google (GKE)
+
+Add firewall rules +
+GKE by default limits outbound requests from nodes. For the runtime to communicate with the control-plane in Codefresh, add a firewall-specific rule. + +-
+
- Find your cluster's network:
+ gcloud container clusters describe [CLUSTER_NAME] --format=get"(network)" +
+ - Get the Cluster IPV4 CIDR:
+ gcloud container clusters describe [CLUSTER_NAME] --format=get"(clusterIpv4Cidr)" +
+ - Replace the `[CLUSTER_NAME]`, `[NETWORK]`, and `[CLUSTER_IPV4_CIDR]`, with the relevant values:
+ gcloud compute firewall-rules create "[CLUSTER_NAME]-to-all-vms-on-network"
+ + --network="[NETWORK]" \ +
+ + --source-ranges="[CLUSTER_IPV4_CIDR]" \ +
+ + --allow=tcp,udp,icmp,esp,ah,sctp +
+
+
+Use ingress-nginx
+-
+
- Create a `cluster-admin` role binding:
+ + kubectl create clusterrolebinding cluster-admin-binding \ +
+ + --clusterrole cluster-admin \ +
+ + --user $(gcloud config get-value account) +
+
+ - Apply:
+ + kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml + +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
+
++ + +MicroK8s
+-
+
- Install using Microk8s addon system:
+ microk8s enable ingress +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ + + +MiniKube
+-
+
- Install using MiniKube addon system:
+ minikube addons enable ingress +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ +Oracle Cloud Infrastructure
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
++ +{::nomarkdown} + +{:/} + +### Traefik ingress configuration +For detailed configuration information, see [Traefik ingress controller documentation](https://doc.traefik.io/traefik/providers/kubernetes-ingress){:target="\_blank}. + +The table below lists the specific configuration requirements for Codefresh. + +{: .table .table-bordered .table-hover} + +| What to configure | When to configure | +| -------------- | -------------- | +|Valid external IP address | _Before_ installing hybrid runtime | +|Valid SSL certificate | | +|TCP support | | +|Enable report status to cluster| | + +{::nomarkdown} + +{:/} + +#### Valid external IP address +Run `kubectl get svc -A` to get a list of services and verify that the `EXTERNAL-IP` column for your ingress controller shows a valid hostname. + +{::nomarkdown} + +{:/} + +#### Valid TLS certificate +For secure runtime installation, the ingress controller must have a valid TLS certificate. +> Use the FQDN (Fully Qualified Domain Name) of the ingress controller for the TLS certificate. + +{::nomarkdown} + +{:/} + +#### TCP support +Configure the ingress controller to handle TCP requests. + +{::nomarkdown} + +{:/} + +#### Enable report status to cluster +By default, the Traefik ingress controller is not configured to report its status to the cluster. If not configured, Argo’s health check reports the health status as “progressing”, resulting in a timeout error during installation. + +To enable reporting its status, add `publishedService` to `providers.kubernetesIngress.ingressEndpoint`. + +The value must be in the format `"Scaleway
+-
+
- Apply:
+ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/scw/deploy.yaml +
+ - Verify a valid external address exists:
+ kubectl get svc ingress-nginx-controller -n ingress-nginx +
+
/ "`, where: + ` ` is the Traefik service from which to copy the status + +```yaml +... +providers: + kubernetesIngress: + ingressEndpoint: + publishedService: " / " # Example, "codefresh/traefik-default" +... +``` + +{::nomarkdown} + +{:/} + +## GitOps CLI installation + +### GitOps CLI installation modes +The table lists the modes available to install the Codefresh CLI. + +{: .table .table-bordered .table-hover} +| Install mode | OS | Commands | +| -------------- | ----------| ----------| +| `curl` | MacOS-x64 | `curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-darwin-amd64.tar.gz | tar zx && mv ./cf-darwin-amd64 /usr/local/bin/cf && cf version`| +| | MacOS-m1 |`curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-darwin-arm64.tar.gz | tar zx && mv ./cf-darwin-arm64 /usr/local/bin/cf && cf version` | +| | Linux - X64 |`curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-linux-amd64.tar.gz | tar zx && mv ./cf-linux-amd64 /usr/local/bin/cf && cf version` | +| | Linux - ARM | `curl -L --output - https://github.com/codefresh-io/cli-v2/releases/latest/download/cf-linux-arm64.tar.gz | tar zx && mv ./cf-linux-arm64 /usr/local/bin/cf && cf version`| +| `brew` | N/A| `brew tap codefresh-io/cli && brew install cf2`| + +### Install the GitOps CLI +Install the GitOps CLI using the option that best suits you: `curl`, `brew`, or standard download. +If you are not sure which OS to select for `curl`, simply select one, and Codefresh automatically identifies and selects the right OS for the installation. + +1. Do one of the following: + * For first-time installation, go to the Welcome page, select **+ Install Runtime**. + * If you have provisioned a GitOps Runtime, in the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}, and select **+ Add Runtime**. +1. Install the Codefresh CLI: + * Select one of the installation modes. + * Generate the API key. + * Create the authentication context: + `cf config create-context codefresh --api-key ` + + + {% include + image.html + lightbox="true" + file="/images/getting-started/quick-start/quick-start-download-cli.png" + url="/images/getting-started/quick-start/quick-start-download-cli.png" + alt="Download CLI to install runtime" + caption="Download CLI to install runtime" + max-width="30%" + %} + + +{::nomarkdown} + +{:/} + +## Install Hybrid GitOps Runtime + +### Before you begin +* Make sure you meet the [minimum requirements]({{site.baseurl}}/docs/runtime/requirements/#minimum-requirements) for installation +* Make sure you have [Runtime token with the required scopes from your Git provider]({{site.baseurl}}/docs/reference/git-tokens) +* [Download or upgrade to the latest version of the CLI]({{site.baseurl}}/docs/installation/clients/upgrade-gitops-cli) +* Review [Hybrid Runtime installation flags](#hybrid-runtime-installation-flags) +* For ingress-based runtimes, make sure your ingress controller is configured correctly: + * [Ambasador ingress configuration]({{site.baseurl}}/docs/runtime/requirements/#ambassador-ingress-configuration) + * [AWS ALB ingress configuration]({{site.baseurl}}/docs/runtime/requirements/#alb-aws-ingress-configuration) + * [Istio ingress configuration]({{site.baseurl}}/docs/runtime/requirements/#istio-ingress-configuration) + * [NGINX Enterprise ingress configuration]({{site.baseurl}}/docs/runtime/requirements/#nginx-enterprise-ingress-configuration) + * [NGINX Community ingress configuration]({{site.baseurl}}/docs/runtime/requirements/#nginx-community-version-ingress-configuration) + * [Traefik ingress configuration]({{site.baseurl}}/docs/runtime/requirements/#traefik-ingress-configuration) + + +{::nomarkdown} + +{:/} + +### How to + +1. Do one of the following: + * If this is your first Hybrid GitOps installation, in the Welcome page, select **+ Install Runtime**. + * If you have already provisioned a Hybrid GitOps Runtime, to provision additional runtimes, in the Codefresh UI: + On the toolbar, click the **Settings** icon, and expand Runtimes in the sidebar, and select [**GitOps Runtimes**](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. Click **+ Add Runtimes**, and then select **Hybrid Runtimes**. +1. Do one of the following: + * CLI wizard: Run `cf runtime install`, and follow the prompts to enter the required values. + * Silent install: Pass the required flags in the install command: + `cf runtime install --repo --git-token --silent` + For the list of flags, see [Hybrid runtime installation flags](#hybrid-runtime-installation-flags). +1. If relevant, complete the configuration for these ingress controllers: + * [ALB AWS: Alias DNS record in route53 to load balancer]({{site.baseurl}}/docs/runtime/requirements/#alias-dns-record-in-route53-to-load-balancer) + * [Istio: Configure cluster routing service]({{site.baseurl}}/docs/runtime/requirements/#cluster-routing-service) + * [NGINX Enterprise ingress controller: Patch certificate secret]({{site.baseurl}}/docs/runtime/requirements/#patch-certificate-secret) +1. If you bypassed installing ingress resources with the `--skip-ingress` flag for ingress controllers not in the supported list, create and register Git integrations using these commands: + `cf integration git add default --runtime --api-url ` + `cf integration git register default --runtime --token ` + + +{::nomarkdown} + +{:/} + + + +## Hybrid GitOps Runtime installation flags +This section describes the required and optional flags to install a Hybrid GitOps Runtime. +For documentation purposes, the flags are grouped into: +* Runtime flags, relating to Runtime, cluster, and namespace requirements +* Ingress-less flags, for tunnel-based installation +* Ingress-controller flags, for ingress-based installation +* Git provider and repo flags +* Codefresh resource flags + +{::nomarkdown} + +{:/} + +### Runtime flags + +**Runtime name** +Required. +The Runtime name must start with a lower-case character, and can include up to 62 lower-case characters and numbers. +* CLI wizard: Add when prompted. +* Silent install: Add the `--runtime` flag and define the name. + +**Namespace resource labels** +Optional. +The label of the namespace resource to which you are installing the Hybrid Runtime. Labels are required to identify the networks that need access during installation, as is the case when using services meshes, such as Istio for example. + +* CLI wizard and Silent install: Add the `--namespace-labels` flag, and define the labels in `key=value` format. Separate multiple labels with `commas`. + +**Kube context** +Required. +The cluster defined as the default for `kubectl`. If you have more than one Kube context, the current context is selected by default. + +* CLI wizard: Select the Kube context from the list displayed. +* Silent install: Explicitly specify the Kube context with the `--context` flag. + +**Access mode** +The access mode for the runtime, which can be one of the following: +* [Tunnel-based]({{site.baseurl}}/docs/installation/runtime-architecture/#tunnel-based-hybrid-gitops-runtime-architecture), for runtimes without ingress controllers. This is the default. +* [Ingress-based]({{site.baseurl}}/docs/getting-started/architecture/#ingress-based-hybrid-gitops-runtime-architecture) for runtimes with ingress contollers. + + +* CLI wizard: Select the access mode from the list displayed. +* Silent install: + * For tunnel-based, see [Tunnel-based runtime flags](#tunnel-based-runtime-flags) + * For ingress-based, add the [Ingress controller flags](#ingress-controller-flags) + + >If you don't specify any flags, tunnel-based access is automatically selected. + +**Shared configuration repository** +The Git repository per Runtime account with shared configuration manifests. +* CLI wizard and Silent install: Add the `--shared-config-repo` flag and define the path to the shared repo. + +{::nomarkdown} + +{:/} + +### Tunnel-based runtime flags +These flags are required to install tunnel-based Hybrid Runtimes, without an ingress controller. + +**IP allowlist** + +Optional. + +The allowed list of IPs from which to forward requests to the internal customer cluster for ingress-less runtime installations. The allowlist can include IPv4 and IPv6 addresses, with/without subnet and subnet masks. Multiple IPs must be separated by commas. + +When omitted, all incoming requests are authenticated regardless of the IPs from which they originated. + +* CLI wizard and Silent install: Add the `--ips-allow-list` flag, followed by the IP address, or list of comma-separated IPs to define more than one. For example, `--ips-allow-list 77.126.94.70/16,192.168.0.0` + +{::nomarkdown} + +{:/} + +### Ingress controller flags + + +**Skip ingress** +Required, if you are using an unsupported ingress controller. +For unsupported ingress controllers, bypass installing ingress resources with the `--skip-ingress` flag. +In this case, after completing the installation, manually configure the cluster's routing service, and create and register Git integrations. See the last step in [Install the Hybrid GitOps Runtime](#install-hybrid-gitops-runtime). + +**Ingress class** +Required. + +* CLI wizard: Select the ingress class for Runtime installation from the list displayed. +* Silent install: Explicitly specify the ingress class through the `--ingress-class` flag. Otherwise, Runtime installation fails. + +**Ingress host** +Required. +The IP address or host name of the ingress controller component. + +* CLI wizard: Automatically selects and displays the host, either from the cluster or the ingress controller associated with the **Ingress class**. +* Silent install: Add the `--ingress-host` flag. If a value is not provided, takes the host from the ingress controller associated with the **Ingress class**. + > Important: For AWS ALB, the ingress host is created post-installation. However, when prompted, add the domain name you will create in `Route 53` as the ingress host. + +**Insecure ingress hosts** +TLS certificates for the ingress host: +If the ingress host does not have a valid TLS certificate, you can continue with the installation in insecure mode, which disables certificate validation. + +* CLI wizard: Automatically detects and prompts you to confirm continuing the installation in insecure mode. +* Silent install: To continue with the installation in insecure mode, add the `--insecure-ingress-host` flag. + +**Internal ingress host** +Optional. +Enforce separation between internal (app-proxy) and external (webhook) communication by adding an internal ingress host for the app-proxy service in the internal network. +For both CLI wizard and Silent install: + +* For new Runtime installations, add the `--internal-ingress-host` flag pointing to the ingress host for `app-proxy`. +* For existing installations, commit changes to the installation repository by modifying the `app-proxy ingress` and ` .yaml` + See [(Optional) Internal ingress host configuration for existing Hybrid Runtimes](#optional-internal-ingress-host-configuration-for-existing-hybrid-runtimes). + +{::nomarkdown} + +{:/} + + + +### Git provider and repo flags +The Git provider defined for the Runtime. + +>Because Codefresh creates a [shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration) for the Runtimes in your account, the Git provider defined for the first Runtime you install in your account is used for all the other Runtimes in the same account. + +You can define any of the following Git providers: +* GitHub: + * [GitHub](#github) (the default Git provider) + * [GitHub Enterprise](#github-enterprise) +* GitLab: + * [GitLab Cloud](#gitlab-cloud) + * [GitLab Server](#gitlab-server) +* Bitbucket: + * [Bitbucket Cloud](#bitbucket-cloud) + * [Bitbucket Server](#bitbucket-server) + +{::nomarkdown} + +{:/} + + + +#### GitHub +GitHub is the default Git provider for Hybrid Runtimes. Being the default provider, for both the CLI wizard and Silent install, you need to provide only the repository URL and the Git runtime token. + +> For the required scopes, see [GitHub and GitHub Enterprise Runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#github-and-github-enterprise-runtime-token-scopes). + +`--repo --git-token ` + +where: +* `--repo ` (required), is the `HTTPS` clone URL of the Git repository for the Runtime installation, including the `.git` suffix. Copy the clone URL from your GitHub website (see [Cloning with HTTPS URLs](https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls){:target="\_blank"}). + If the repo doesn't exist, copy an existing clone URL and change the name of the repo. Codefresh creates the repository during the installation. + + Repo URL format: + `https://github.com/ /reponame>.git[/subdirectory][?ref=branch]` + where: + * ` / ` is your username or organization name, followed by the name of the repo, identical to the HTTPS clone URL. For example, `https://github.com/nr-codefresh/codefresh.io.git`. + * `[/subdirectory]` (optional) is the path to a subdirectory within the repo. When omitted, the Runtime is installed in the root of the repository. For example, `/runtimes/defs`. + * `[?ref=branch]` (optional) is the `ref` queryParam to select a specific branch. When omitted, the Runtime is installed in the default branch. For example, `codefresh-prod`. + + Example: + `https://github.com/nr-codefresh/codefresh.io.git/runtimes/defs?ref=codefresh-prod` +* `--git-token ` (required), is the Git token authenticating access to the Runtime installation repository (see [GitHub runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#github-and-github-enterprise-runtime-token-scopes)). + +{::nomarkdown} + +{:/} + +#### GitHub Enterprise + +> For the required scopes, see [GitHub and GitHub Enterprise runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#github-and-github-enterprise-runtime-token-scopes). + + +`--provider github --repo --git-token ` + +where: +* `--provider github` (required), defines GitHub Enterprise as the Git provider for the Runtime and the account. +* `--repo ` (required), is the `HTTPS` clone URL of the Git repository for the Runtime installation, including the `.git` suffix. Copy the clone URL for HTTPS from your GitHub Enterprise website (see [Cloning with HTTPS URLs](https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls){:target="\_blank"}). + If the repo doesn't exist, copy an existing clone URL and change the name of the repo. Codefresh creates the repository during the installation. + Repo URL format: + + `https://ghe-trial.devops.cf-cd.com/ / .git[/subdirectory][?ref=branch]` + where: + * ` / ` is your username or organization name, followed by the name of the repo. For example, `codefresh-io/codefresh.io.git`. + * `[/subdirectory]` (optional) is the path to a subdirectory within the repo. When omitted, the Runtime is installed in the root of the repository. For example, `/runtimes/defs`. + * `[?ref=branch]` (optional) is the `ref` queryParam to select a specific branch. When omitted, the Runtime is installed in the default branch. For example, `codefresh-prod`. + + Example: + `https://ghe-trial.devops.cf-cd.com/codefresh-io/codefresh.io.git/runtimes/defs?ref=codefresh-prod` +* `--git-token ` (required), is the Git token authenticating access to the Runtime installation repository (see [GitHub runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#github-and-github-enterprise-runtime-token-scopes)). + + +{::nomarkdown} + +{:/} + +#### GitLab Cloud +> For the required scopes, see [GitLab Cloud and GitLab Server runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#gitlab-cloud-and-gitlab-server-runtime-token-scopes). + + +`--provider gitlab --repo --git-token ` + +where: +* `--provider gitlab` (required), defines GitLab Cloud as the Git provider for the Runtime and the account. +* `--repo ` (required), is the `HTTPS` clone URL of the Git project for the Runtime installation, including the `.git` suffix. Copy the clone URL for HTTPS from your GitLab website. + If the repo doesn't exist, copy an existing clone URL and change the name of the repo. Codefresh creates the repository during the installation. + + > Important: You must create the group with access to the project prior to the installation. + + Repo URL format: + + `https://gitlab.com/ / .git[/subdirectory][?ref=branch]` + where: + * ` ` is either your username, or if your project is within a group, the front-slash separated path to the project. For example, `nr-codefresh` (owner), or `parent-group/child-group` (group hierarchy) + * ` ` is the name of the project. For example, `codefresh`. + * `[/subdirectory]` (optional) is the path to a subdirectory within the repo. When omitted, the Runtime is installed in the root of the repository. For example, `/runtimes/defs`. + * `[?ref=branch]` (optional) is the `ref` queryParam to select a specific branch. When omitted, the Runtime is installed in the default branch. For example, `codefresh-prod`. + + Examples: + `https://gitlab.com/nr-codefresh/codefresh.git/runtimes/defs?ref=codefresh-prod` (owner) + + `https://gitlab.com/parent-group/child-group/codefresh.git/runtimes/defs?ref=codefresh-prod` (group hierarchy) + +* `--git-token ` (required), is the Git token authenticating access to the Runtime installation repository (see [GitLab runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#gitlab-cloud-and-gitlab-server-runtime-token-scopes)). + + +{::nomarkdown} + +{:/} + + +#### GitLab Server + +> For the required scopes, see [GitLab Cloud and GitLab Server runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#gitlab-cloud-and-gitlab-server-runtime-token-scopes). + +`--provider gitlab --repo --git-token ` + +where: +* `--provider gitlab` (required), defines GitLab Server as the Git provider for the Runtime and the account. +* `--repo ` (required), is the `HTTPS` clone URL of the Git repository for the Runtime installation, including the `.git` suffix. + If the project doesn't exist, copy an existing clone URL and change the name of the project. Codefresh creates the project during the installation. + + > Important: You must create the group with access to the project prior to the installation. + + Repo URL format: + `https://gitlab-onprem.devops.cf-cd.com/ / .git[/subdirectory][?ref=branch]` + where: + * ` ` is your username, or if the project is within a group or groups, the name of the group. For example, `nr-codefresh` (owner), or `parent-group/child-group` (group hierarchy) + * ` ` is the name of the project. For example, `codefresh`. + * `[/subdirectory]` (optional) is the path to a subdirectory within the repo. When omitted, the Runtime is installed in the root of the repository. For example, `/runtimes/defs`. + * `[?ref=branch]` (optional) is the `ref` queryParam to select a specific branch. When omitted, the Runtime is installed in the default branch. For example, `codefresh-prod`. + + Examples: + `https://gitlab-onprem.devops.cf-cd.com/nr-codefresh/codefresh.git/runtimes/defs?ref=codefresh-prod` (owner) + + `https://gitlab-onprem.devops.cf-cd.com/parent-group/child-group/codefresh.git/runtimes/defs?ref=codefresh-prod` (group hierarchy) + +* `--git-token ` (required), is the Git token authenticating access to the Runtime installation repository (see [GitLab runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#gitlab-cloud-and-gitlab-server-runtime-token-scopes)). + + +{::nomarkdown} + +{:/} + +#### Bitbucket Cloud +> For the required scopes, see [Bitbucket runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#bitbucket-cloud-and-bitbucket-server-runtime-token-scopes). + + +`--provider bitbucket --repo --git-user --git-token ` + +where: +* `--provider gitlab` (required), defines Bitbucket Cloud as the Git provider for the Runtime and the account. +* `--repo ` (required), is the `HTTPS` clone URL of the Git repository for the Runtime installation, including the `.git` suffix. + If the project doesn't exist, copy an existing clone URL and change the name of the project. Codefresh creates the project during Runtime installation. + >Important: Remove the username, including @ from the copied URL. + + Repo URL format: + + `https://bitbucket.org .git[/subdirectory][?ref=branch]` + where: + * ` ` is your workspace ID. For example, `nr-codefresh`. + * ` ` is the name of the repository. For example, `codefresh`. + * `[/subdirectory]` (optional) is the path to a subdirectory within the repo. When omitted, the Runtime is installed in the root of the repository. For example, `/runtimes/defs`. + * `[?ref=branch]` (optional) is the `ref` queryParam to select a specific branch. When omitted, the Runtime is installed in the default branch. For example, `codefresh-prod`. + + Example: + `https://bitbucket.org/nr-codefresh/codefresh.git/runtimes/defs?ref=codefresh-prod` +* `--git-user ` (required), is your username for the Bitbucket Cloud account. +* `--git-token ` (required), is the Git token authenticating access to the runtime installation repository (see [Bitbucket runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#bitbucket-cloud-and-bitbucket-server-runtime-token-scopes)). + + +{::nomarkdown} + +{:/} + +#### Bitbucket Server + +> For the required scopes, see [Bitbucket runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#bitbucket-cloud-and-bitbucket-server-runtime-token-scopes). + + +`--provider bitbucket-server --repo --git-user --git-token ` + +where: +* `--provider gitlab` (required), defines Bitbucket Cloud as the Git provider for the Runtime and the account. +* `--repo ` (required), is the `HTTPS` clone URL of the Git repository for the Runtime installation, including the `.git` suffix. + If the project doesn't exist, copy an existing clone URL and change the name of the project. Codefresh then creates the project during the installation. + >Important: Remove the username, including @ from the copied URL. + + Repo URL format: + + `https://bitbucket-server-8.2.devops.cf-cd.com:7990/scm/ / .git[/subdirectory][?ref=branch]` + where: + * ` ` is your username or organization name. For example, `codefresh-io.`. + * ` ` is the name of the repo. For example, `codefresh`. + * `[/subdirectory]` (optional) is the path to a subdirectory within the repo. When omitted, the Runtime is installed in the root of the repository. For example, `/runtimes/defs`. + * `[?ref=branch]` (optional) is the `ref` queryParam to select a specific branch. When omitted, the Runtime is installed in the default branch. For example, `codefresh-prod`. + + Example: + `https://bitbucket-server-8.2.devops.cf-cd.com:7990/scm/codefresh-io/codefresh.git/runtimes/defs?ref=codefresh-prod` +* `--git-user ` (required), is your username for the Bitbucket Server account. +* `--git-token ` (required), is the Git token authenticating access to the Runtime installation repository (see [Bitbucket runtime token scopes]({{site.baseurl}}/docs/reference/git-tokens/#bitbucket-cloud-and-bitbucket-server-runtime-token-scopes)). + +{::nomarkdown} + +{:/} + +### Codefresh resource flags +**Codefresh demo resources** +Optional. +Install demo pipelines to use as a starting point to create your own GitOps pipelines. We recommend installing the demo resources as these are used in our quick start tutorials. + +* Silent install: Add the `--demo-resources` flag, and define its value as `true` (default), or `false`. For example, `--demo-resources=true` + +**Insecure flag** +For _on-premises installations_, if the Ingress controller does not have a valid SSL certificate, to continue with the installation, add the `--insecure` flag to the installation command. + +{::nomarkdown} + +{:/} + + + + + + + +## (Optional) Internal ingress host configuration for existing Hybrid GitOps Runtimes +If you already have provisioned Hybrid GitOps Runtimes, to use an internal ingress host for app-proxy communication and an external ingress host to handle webhooks, change the specs for the `Ingress` and `Runtime` resources in the Runtime installation repository. Use the examples as guidelines. + +` /apps/app-proxy/overlays/ /ingress.yaml`: change `host` + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: codefresh-cap-app-proxy + namespace: codefresh #replace with your runtime name +spec: + ingressClassName: nginx + rules: + - host: my-internal-ingress-host # replace with the internal ingress host for app-proxy + http: + paths: + - backend: + service: + name: cap-app-proxy + port: + number: 3017 + path: /app-proxy/ + pathType: Prefix +``` + +`../ /bootstrap/ .yaml`: add `internalIngressHost` + +```yaml +apiVersion: v1 +data: + base-url: https://g.codefresh.io + runtime: | + apiVersion: codefresh.io/v1alpha1 + kind: Runtime + metadata: + creationTimestamp: null + name: codefresh #replace with your runtime name + namespace: codefresh #replace with your runtime name + spec: + bootstrapSpecifier: github.com/codefresh-io/cli-v2/manifests/argo-cd + cluster: https://7DD8390300DCEFDAF87DC5C587EC388C.gr7.us-east-1.eks.amazonaws.com + components: + - isInternal: false + name: events + type: kustomize + url: github.com/codefresh-io/cli-v2/manifests/argo-events + wait: true + - isInternal: false + name: rollouts + type: kustomize + url: github.com/codefresh-io/cli-v2/manifests/argo-rollouts + wait: false + - isInternal: false + name: workflows + type: kustomize + url: github.com/codefresh-io/cli-v2/manifests/argo-workflows + wait: false + - isInternal: false + name: app-proxy + type: kustomize + url: github.com/codefresh-io/cli-v2/manifests/app-proxy + wait: false + defVersion: 1.0.1 + ingressClassName: nginx + ingressController: k8s.io/ingress-nginx + ingressHost: https://support.cf.com/ + internalIngressHost: https://my-internal-ingress-host # add this line and replace my-internal-ingress-host with your internal ingress host + repo: https://github.com/NimRegev/my-codefresh.git + version: 99.99.99 +``` + + +## Related articles +[Add external clusters to Hybrid and Hosted Runtimes]({{site.baseurl}}/docs/installation/managed-cluster/) +[Monitoring & managing GitOps Runtimes]({{site.baseurl}}/docs/installation/monitor-manage-runtimes/) +[Add Git Sources to runtimes]({{site.baseurl}}/docs/installation/git-sources/) +[Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration) +[Troubleshoot Hybrid Runtime installation]({{site.baseurl}}/installation/troubleshooting/runtime-issues/) diff --git a/_docs/runtime/managed-cluster.md b/_docs/installation/gitops/managed-cluster.md similarity index 65% rename from _docs/runtime/managed-cluster.md rename to _docs/installation/gitops/managed-cluster.md index 25ae4546e..bb5a756b4 100644 --- a/_docs/runtime/managed-cluster.md +++ b/_docs/installation/gitops/managed-cluster.md @@ -1,42 +1,43 @@ --- -title: "Add external clusters to runtimes" -description: "" -group: runtime +title: "Add external clusters to GitOps Runtimes" +description: "Manage multiple remote clusters with single GitOps Runtime" +group: installation +sub_group: gitops toc: true --- -Register external clusters to provisioned hybrid or hosted runtimes in Codefresh. Once you add an external cluster, you can deploy applications to that cluster without having to install Argo CD in order to do so. External clusters allow you to manage multiple clusters through a single runtime. +Register external clusters to provisioned Hybrid or Hosted GitOps Runtimes in Codefresh. Once you add an external cluster, you can deploy applications to that cluster without having to install Argo CD on the clusters in order to do so. Manage multiple external clusters through a single Runtime. -When you add an external cluster to a provisioned runtime, the cluster is registered as a managed cluster. A managed cluster is treated as any other managed K8s resource, meaning that you can monitor its health and sync status, deploy applications on the cluster and view information in the Applications dashboard, and remove the cluster from the runtime's managed list. +When you add an external cluster to a provisioned GitOps Runtime, the cluster is registered as a managed cluster. A managed cluster is treated as any other managed K8s resource, meaning that you can monitor its health and sync status, deploy applications to it, view information in the Applications dashboard, and remove the cluster from the Runtime's managed list. Add managed clusters through: -* Codefresh CLI +* GitOps CLI * Kustomize -Adding a managed cluster via Codefresh ensures that Codefresh applies the required RBAC resources (`ServiceAccount`, `ClusterRole` and `ClusterRoleBinding`) to the target cluster, creates a `Job` that updates the selected runtime with the information, registers the cluster in Argo CD as a managed cluster, and updates the platform with the new cluster information. +Adding a managed cluster via Codefresh ensures that Codefresh applies the required RBAC resources (`ServiceAccount`, `ClusterRole` and `ClusterRoleBinding`) to the target cluster, creates a `Job` that updates the selected Runtime with the information, registers the cluster in Argo CD as a managed cluster, and updates the platform with the new cluster information. -### Add a managed cluster with Codefresh CLI -Add an external cluster to a provisioned runtime through the Codefresh CLI. When adding the cluster, you can also add labels and annotations to the cluster, which are added to the cluster secret created by Argo CD. +## Add a managed cluster with GitOps CLI +Add an external cluster to a provisioned GitOps Runtime through the GitOps CLI. When adding the cluster, you can also add labels and annotations to the cluster, which are added to the cluster secret created by Argo CD. Optionally, to first generate the YAML manifests, and then manually apply them, use the `dry-run` flag in the CLI. **Before you begin** - -* For _hosted_ runtimes: [Configure access to these IP addresses]({{site.baseurl}}/docs/administration/platform-ip-addresses/) +* For _Hosted GitOps_ Runtimes: [Configure access to these IP addresses]({{site.baseurl}}/docs/administration/platform-ip-addresses/) * Verify that: - * Your Git personal access token is valid and has the correct permissions - * You have installed the latest version of the Codefresh CLI + * Your Git personal access token is valid and has the [required scopes]({{site.baseurl}}/docs/reference/git-tokens) + * You have installed the [latest version of the Codefresh CLI]({{site.baseurl}}/docs/clients/#upgrade-gitops-cli) **How to** -1. In the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. -1. From either the **Topology** or **List** views, select the runtime to which to add the cluster. +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, expand Runtimes in the sidebar, and select [**GitOps Runtimes**](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. From either the **Topology** or **List** views, select the Runtime to which to add the cluster. 1. Topology View: Select {::nomarkdown} 
{:/}.
List View: Select the **Managed Clusters** tab, and then select **+ Add Cluster**.
1. In the Add Managed Cluster panel, copy and run the command:
- `cf cluster add [--labels label-key=label-value] [--annotations annotation-key=annotation-value][--dry-run]` + `cf cluster add [runtime-name] [--labels label-key=label-value] [--annotations annotation-key=annotation-value][--dry-run]` where: + * `runtime-name` is the name of the Runtime to which to add the cluster. * `--labels` is optional, and required to add labels to the cluster. When defined, add a label in the format `label-key=label-value`. Separate multiple labels with `commas`. * `--annotations` is optional, and required to add annotations to the cluster. When defined, add an annotation in the format `annotation-key=annotation-value`. Separate multiple annotations with `commas`. * `--dry-run` is optional, and required if you want to generate a list of YAML manifests that you can redirect and apply manually with `kubectl`. @@ -54,7 +55,7 @@ Optionally, to first generate the YAML manifests, and then manually apply them, {:start="5"} 1. If you used `dry-run`, apply the generated manifests to the same target cluster on which you ran the command. - Here is an example of the YAML manifest generated with the `--dry-run` flag. Note that there are placeholders in the example, which are replaced with the actual values with `--dry-run`. + Here is an example of the YAML manifest generated with the `--dry-run` flag. Note that the example has placeholders, which are replaced with the actual values during the `--dry-run`. ```yaml @@ -177,9 +178,9 @@ spec: ``` -The new cluster is registered to the runtime as a managed cluster. +The new cluster is registered to the GitOps Runtime as a managed cluster. -### Add a managed cluster with Kustomize +## Add a managed cluster with Kustomize Create a `kustomization.yaml` file with the information shown in the example below, and run `kustomize build` on it. ```yaml @@ -222,16 +223,20 @@ resources: ``` -### Work with managed clusters -Work with managed clusters in hybrid or hosted runtimes in either the Topology or List runtime views. For information on runtime views, see [Runtime views]({{site.baseurl}}/docs/runtime/runtime-views). -As the cluster is managed through the runtime, updates to the runtime automatically updates the components on all the managed clusters that include it. +## Work with managed clusters +Work with managed clusters in either the Topology or List Runtime views. For information on Runtime views, see [Runtime views]({{site.baseurl}}/docs/installation/monitor-manage-runtimes/#gitops-runtime-views). +As the cluster is managed through the Runtime, updates to the Runtime automatically updates the components on all the managed clusters that include it. View connection status for the managed cluster, and health and sync errors. Health and sync errors are flagged by the error notification in the toolbar, and visually flagged in the List and Topology views. -#### Install Argo Rollouts -Install Argo Rollouts directly from Codefresh with a single click to visualize rollout progress in the [Applications dashboard]({{site.baseurl}}/docs/deployment/applications-dashboard/). If Argo Rollouts has not been installed, an **Install Argo Rollouts** button is displayed on selecting the managed cluster. +### Install Argo Rollouts +Applications with `rollout` resources need Argo Rollouts on the target cluster, both to visualize rollouts in the Applications dashboard and control rollout steps with the Rollout Player. +If Argo Rollouts has not been installed on the target cluster, it displays **Install Argo Rollouts** button. + +Install Argo Rollouts with a single click to execute rollout instructions, deploy the application, and visualize rollout progress in the [Applications dashboard]({{site.baseurl}}/docs/deployment/gitops/applications-dashboard/). + -1. In the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, expand Runtimes in the sidebar, and select [**GitOps Runtimes**](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. 1. Select **Topology View**. 1. Select the target cluster, and then select **+ Install Argo Rollouts**. @@ -246,16 +251,16 @@ Install Argo Rollouts directly from Codefresh with a single click to visualize r %} -#### Remove a managed cluster from the Codefresh UI -Remove a cluster from the runtime's list of managed clusters from the Codefresh UI. +### Remove a managed cluster from the Codefresh UI +Remove a cluster from the Runtime's list of managed clusters from the Codefresh UI. > You can also remove it through the CLI. -1. In the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +In the Codefresh UI, on the toolbar, click the **Settings** icon, expand Runtimes in the sidebar, and select [**GitOps Runtimes**](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. 1. Select either the **Topology View** or the **List View** tabs. 1. Do one of the following: - * In the Topology View, select the cluster node from the runtime it is registered to. - * In the List View, select the runtime, and then select the **Managed Clusters** tab. + * In the Topology View, select the cluster node from the Runtime it is registered to. + * In the List View, select the Runtime, and then select the **Managed Clusters** tab. 1. Select the three dots next to the cluster name, and then select **Uninstall** (Topology View) or **Remove** (List View). {% include @@ -269,8 +274,8 @@ Remove a cluster from the runtime's list of managed clusters from the Codefresh %} -#### Remove a managed cluster through the Codefresh CLI -Remove a cluster from the list managed by the runtime, through the CLI. +### Remove a managed cluster through the GitOps CLI +Remove a cluster from the list managed by the GitOps Runtime, through the GitOps CLI. * Run: `cf cluster remove --server-url ` @@ -279,7 +284,6 @@ Remove a cluster from the list managed by the runtime, through the CLI. ` ` is the URL of the server on which the managed cluster is installed. -### Related articles -[Add Git Sources to runtimes]({{site.baseurl}}/docs/runtime/git-sources/) -[Manage provisioned hybrid runtimes]({{site.baseurl}}/docs/runtime/monitor-manage-runtimes/) -[(Hybrid) Monitor provisioned runtimes]({{site.baseurl}}/docs/runtime/monitoring-troubleshooting/) \ No newline at end of file +## Related articles +[Add Git Sources to GitOps Runtimes]({{site.baseurl}}/docs/installation/gitops/git-sources/) +[Monitoring & managing GitOps Runtimes]({{site.baseurl}}/docs/installation/gitops/monitor-manage-runtimes/) diff --git a/_docs/installation/gitops/monitor-manage-runtimes.md b/_docs/installation/gitops/monitor-manage-runtimes.md new file mode 100644 index 000000000..d67609286 --- /dev/null +++ b/_docs/installation/gitops/monitor-manage-runtimes.md @@ -0,0 +1,642 @@ +--- +title: "Monitoring & managing GitOps Runtimes" +description: "Optimize GitOps Runtimes" +group: runtime +sub_group: gitops +redirect_from: + - /monitor-manage-runtimes/ + - /monitor-manage-runtimes +toc: true +--- + + +The **Runtimes** page displays the provisioned GitOps Runtimes in your account, both Hybrid, and the Hosted Runtime if you have one. + +View Runtime components and information in List or Topology view formats to manage and monitor them. + +{% include + image.html + lightbox="true" + file="/images/runtime/runtime-list-view.png" + url="/images/runtime/runtime-list-view.png" + alt="Runtime List View" + caption="Runtime List View" + max-width="70%" +%} + +Manage provisioned GitOps Runtimes: +* [Add managed clusters to GitOps Runtimes]({{site.baseurl}}/docs/installation/gitops/managed-cluster/) +* [Add and manage Git Sources for GitOps Runtimes]({{site.baseurl}}/docs/installation/gitops/git-sources/) +* [Upgrade GitOps CLI](#hybrid-gitops-upgrade-provisioned-runtimes) +* Upgrade Hybrid GitOps Runtimes +* Uninstall GitOps Runtimes + + + +Monitor provisioned GitOps Runtimes for security, health, and sync errors: + +* (Hybrid and Hosted) View/download logs for Runtimes and for Runtime components +* (Hybrid) Restore provisioned Runtimes +* (Hybrid) Configure browsers to allow access to insecure Runtimes +* (Hybrid) Monitor notifications in the Activity Log + + +> Unless specified otherwise, all options are common to both types of GitOps Runtimes. If an option is valid only for Hybrid GitOps, it is indicated as such. + + +## GitOps Runtime views + +View provisioned GitOps Runtimes in List or Topology view formats. + +* List view: The default view, displays the list of provisioned Runtimes, the clusters managed by them, and Git Sources associated with them. +* Topology view: Displays a hierarchical view of Runtimes and the clusters managed by them, with health and sync status of each cluster. + +### List view + +The List view is a grid-view of the provisioned Runtimes. + +Here is an example of the List view for runtimes. +{% include + image.html + lightbox="true" + file="/images/runtime/runtime-list-view.png" + url="/images/runtime/runtime-list-view.png" + alt="Runtime List View" + caption="Runtime List View" + max-width="70%" +%} + +Here is a description of the information in the List View. + +{: .table .table-bordered .table-hover} +| List View Item| Description | +| -------------- | ---------------- | +|**Name**| The name of the provisioned GitOps Runtime. | +|**Type**| The type of GitOps Runtime provisioned, and can be **Hybrid** or **Hosted**. | +|**Cluster/Namespace**| The K8s API server endpoint, as well as the namespace with the cluster. | +|**Modules**| The modules installed based on the type of provisioned Runtime. Hybrid Runtimes include CI amnd CD Ops modules. Hosted runtimes include CD Ops. | +|**Managed Cluster**| The number of managed clusters if any, for the runtime. To view list of managed clusters, select the runtime, and then the **Managed Clusters** tab. To work with managed clusters, see [Adding external clusters to runtimes]({{site.baseurl}}/docs/runtime/managed-cluster).| +|**Version**| The version of the runtime currently installed. **Update Available!** indicates there are later versions of the runtime. To see all the commits to the runtime, mouse over **Update Available!**, and select **View Complete Change Log**. +|**Last Updated**| The most recent update information from the runtime to the Codefresh platform. Updates are sent to the platform typically every few minutes. Longer update intervals may indicate networking issues.| +|**Sync Status**| The health and sync status of the runtime or cluster. {::nomarkdown} -
indicates health or sync errors in the runtime, or a managed cluster if one was added to the runtime. The runtime name is colored red. 
indicates that the runtime is being synced to the cluster on which it is provisioned.
indicates the local cluster, always displayed as `in-cluster`. The in-cluster server URL is always set to `https://kubernetes.default.svc/`.
indicates a managed cluster.- select to add a new managed cluster.
- indicates health or sync errors in the Runtime, or a managed cluster if one was added to the runtime. The runtime or cluster node is bordered in red and the name is colored red.
- indicates that the Runtime is being synced to the cluster on which it is provisioned.
- Find a Runtime or its clusters by typing part of the Runtime/cluster name, and then navigate to the entries found.
- Topology view options: Resize to window, zoom in, zoom out, full screen view.
--git-token --silent` + where: + ` ` is a valid Git token with the correct scopes. + +**CLI wizard-based upgrade** + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, expand Runtimes in the sidebar, and select [**GitOps Runtimes**](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. Switch to either the **List View** or to the **Topology View**. +1. **List view**: + * Select the Runtime name. + * To see all the commits to the Runtime, in the Version column, mouse over **Update Available!**, and select **View Complete Change Log**. + * On the top-right, select **Upgrade**. + + {% include + image.html + lightbox="true" + file="/images/runtime/runtime-list-view-upgrade.png" + url="/images/runtime/runtime-list-view-upgrade.png" + alt="List View: Upgrade runtime option" + caption="List View: Upgrade runtime option" + max-width="30%" + %} + + **Topology view**: + Select the Runtime cluster, and from the panel, select the three dots and then select **Upgrade Runtime**. + {% include + image.html + lightbox="true" + file="/images/runtime/runtiime-topology-upgrade.png" + url="/images/runtime/runtiime-topology-upgrade.png" + alt="Topology View: Upgrade runtime option" + caption="Topology View: Upgrade runtime option" + max-width="30%" +%} + +{:start="4"} + +1. If you have already installed the GitOps CLI, in the Install Upgrades panel, copy the upgrade command. + + {% include + image.html + lightbox="true" + file="/images/runtime/install-upgrades.png" + url="/images/runtime/install-upgrades.png" + alt="Upgrade runtime" + caption="Upgrade runtime panel" + max-width="30%" +%} + +{:start="5"} +1. In your terminal, paste the command, and do the following: + * Update the Git token value. + * To manually define the shared configuration repo, add the `--shared-config-repo` flag with the path to the repo. +1. Confirm to start the upgrade. + + + + + + +### Uninstall provisioned GitOps Runtimes + +Uninstall provisioned GitOps Runtimes that are not in use, through a silent uninstall or through the GitOps CLI wizard. +> Uninstalling a Runtime removes the Git Sources and managed clusters associated with it. + +**Before you begin** +For both types of uninstalls, make sure you have: + +* The latest version of the GitOps CLI +* A valid runtime Git token +* The Kube context from which to uninstall the provisioned Runtime + +**Silent uninstall** +Pass the mandatory flags in the uninstall command: + `cf runtime uninstall --git-token --silent` + where: + `--git-token` is a valid runtime token with the `repo` and `admin-repo.hook` scopes. + +**GitOps CLI wizard uninstall** + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, expand Runtimes in the sidebar, and select [**GitOps Runtimes**](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. Switch to either the **List View** or to the **Topology View**. +1. **List view**: On the top-right, select the three dots and then select **Uninstall**. + + {% include + image.html + lightbox="true" + file="/images/runtime/uninstall-location.png" + url="/images/runtime/uninstall-location.png" + alt="List View: Uninstall runtime option" + caption="List View: Uninstall runtime option" + max-width="30%" +%} + +**Topology view**: Select the Runtime node, and from the panel, select the three dots and then select **Uninstall Runtime**. + {% include + image.html + lightbox="true" + file="/images/runtime/runtime-topology-uninstall.png" + url="/images/runtime/runtime-topology-uninstall.png" + alt="Topology View: Uninstall runtime option" + caption="Topology View: Uninstall runtime option" + max-width="30%" +%} + +{:start="4"} + +1. If you already have the latest version of the GitOps CLI, in the Uninstall Codefresh Runtime panel, copy the uninstall command. + + {% include + image.html + lightbox="true" + file="/images/runtime/uninstall.png" + url="/images/runtime/uninstall.png" + alt="Uninstall Codefresh runtime" + caption="Uninstall Codefresh runtime" + max-width="40%" +%} + +{:start="5"} + +1. In your terminal, paste the command, and update the Git token value. +1. Select the Kube context from which to uninstall the Runtime, and then confirm the uninstall. +1. If you get errors, run the uninstall command again, with the `--force` flag. + + + +### Update Git tokens for Runtimes + +Provisioned Runtimes require valid Git tokens at all times to authenticate Git actions by you as a user. +>These tokens are specific to the user, and the same can be used for multiple runtimes. + +There are two different situations when you need to update Git tokens: +* Update invalid, revoked, or expired tokens: Codefresh automatically flags Runtimes with such tokens. It is mandatory to update the Git tokens to continue working with the platform. +* Update valid tokens: Optional. You may want to update Git tokens, even valid ones, by deleting the existing token and replacing it with a new token. + +The methods for updating any Git token are the same regardless of the reason for the update: +* OAuth2 authorization, if your admin has registered an OAuth Application for Codefresh +* Git access token authentication, by generating a personal access token in your Git provider account with the correct scopes + +**Before you begin** +* To authenticate through a Git access token, make sure your token is valid and has [the required scopes]({{site.baseurl}}/docs/reference/git-tokens) + +**How to** +1. Do one of the following: + * If you see a notification in the Codefresh UI about invalid Runtime tokens, click **[Update Token]**. + The GitOps Runtimes page shows runtimes with invalid tokens prefixed by the key icon. Mouse over shows invalid token. + * To update an existing token, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. Select the GitOps Runtime for which to update the Git token. +1. From the context menu with the additional actions at the top-right, select **Update Git Runtime token**. + + {% include + image.html + lightbox="true" + file="/images/runtime/update-git-runtime-token.png" + url="/images/runtime/update-git-runtime-token.png" + alt="Update Git runtime token option" + caption="Update Git runtime token option" + max-width="40%" +%} + +{:start="4"} +1. Do one of the following: + * If your admin has set up OAuth access, click **Authorize Access to Git Provider**. Go to _step 5_. + * Alternatively, authenticate with an access token from your Git provider. Go to _step 6_. + +{:start="5"} +1. For OAuth2 authorization: + > If the application is not registered, you get an error. Contact your admin for help. + * Enter your credentials, and select **Sign In**. + * If required, as for example if two-factor authentication is configured, complete the verification. + + {% include + image.html + lightbox="true" + file="/images/administration/user-settings/oauth-user-authentication.png" + url="/images/administration/user-settings/oauth-user-authentication.png" + alt="Authorizing access with OAuth2" + caption="Authorizing access with OAuth2" + max-width="30%" + %} + +{:start="6"} +1. For Git token authentication, expand **Advanced authorization options**, and then paste the generated token in the **Git runtime token** field. + +1. Click **Update Token**. + +## Monitoring GitOps Runtimes +* [View/download logs to troubleshoot Runtimes](#viewdownload-logs-to-troubleshoot-runtimes) +* [(Hybrid GitOps) Restoring provisioned Runtimes](#hybrid-gitops-restoring-provisioned-runtimes) +* [(Hybrid GitOps) Configure browser to allow insecure Runtimes](#hybrid-gitops-configure-browser-to-allow-insecure-runtimes) +* [(Hybrid GitOps) View notifications in Activity Log](#hybrid-gitops-view-notifications-in-activity-log) +* [(Hybrid GitOps) Troubleshoot health and sync errors for Runtimes](#hybrid-gitops-troubleshoot-health-and-sync-errors-for-runtimes) + +### View/download logs to troubleshoot GitOps Runtimes +Logs are available for completed Runtimes, both for the runtime and for individual runtime components. Download log files for offline viewing and analysis, or view online logs for a Runtime component, and download if needed for offline analysis. Online logs support free-text search, search-result navigation, and line-wrap for enhanced readability. + +Log files include events from the date of the application launch, with the newest events listed first. + +{::nomarkdown} +
+{:/} + +#### Download logs for GitOps Runtimes +Download the log file for a Runtime. The Runtime log is downloaded as a `.tar.gz` file, which contains the individual log files for each runtime component. + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, expand Runtimes in the sidebar, and select [**GitOps Runtimes**](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. If needed, switch to **List View**, and then select the runtime for which to download logs. +1. From the context menu, select **Download All Logs**. + The log file is downloaded to the Downloads folder or the folder designated for downloads, with the filename, `.tar.gz`. For example, `codefreshv2-production2.tar.gz`. + + + {% include + image.html + lightbox="true" + file="/images/runtime/runtime-logs-download-all.png" + url="/images/runtime/runtime-logs-download-all.png" + alt="Download logs for selected runtime" + caption="Download logs for selected runtime" + max-width="40%" +%} + + +{:start="4"} +1. To view the log files of the individual components, unzip the file. + Here is an example of the folder with the individual logs. + + {% include + image.html + lightbox="true" + file="/images/runtime/runtime-logs-folder-view.png" + url="/images/runtime/runtime-logs-folder-view.png" + alt="Individual log files in folder" + caption="Individual log files in folder" + max-width="50%" +%} + +{:start="5"} +1. Open a log file with the text editor of your choice. + +{::nomarkdown} +
+{:/} + +#### View/download logs for Runtime components +View online logs for any Runtime component, and if needed, download the log file for offline viewing and analysis. + +Online logs show up to 1000 of the most recent events (lines), updated in real time. Downloaded logs include all the events, from the application launch to the date and time of download. + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, expand Runtimes in the sidebar, and select [**GitOps Runtimes**](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"}. +1. If needed, switch to **List View**, and then select the Runtime. +1. Select the Runtime component and then select **View Logs**. + + {% include + image.html + lightbox="true" + file="/images/runtime/runtime-logs-view-component.png" + url="/images/runtime/runtime-logs-view-component.png" + alt="View log option for individual runtime component" + caption="View log option for individual runtime component" + max-width="40%" +%} + + +{:start="4"} +1. Do the following: + * Search by free-text for any string, and click the next and previous buttons to navigate between the search results. + * To switch on line-wrap for readability, click **Wrap**. + + {% include + image.html + lightbox="true" + file="/images/runtime/runtime-logs-screen-view.png" + url="/images/runtime/runtime-logs-screen-view.png" + alt="Runtime component log example" + caption="Runtime component log example" + max-width="50%" +%} + +{:start="5"} +1. To download the log, click **Download**. + The file is downloaded as `.log`. + +### (Hybrid GitOps) Restoring provisioned Runtimes + +In case of cluster failure, restore the provisioned Hybrid Runtime from the existing runtime installation repository. +For partial or complete cluster failures, you can restore the Runtime to either the failed cluster or to a different cluster. +Restoring the provisioned Runtime reinstalls it, leveraging the resources in the existing Runtime repo. + +Restoring the runtime: +* Applies `argo-cd` from the installation manifests in your repo to your cluster +* Associates `argo-cd` with the existing installation repo +* Applies the Runtime and `argo-cd` secrets to the cluster +* Updates the Runtime config map (` .yaml` in the `bootstrap` directory) with the new cluster configuration for these fields: + `cluster` + `ingressClassName` + `ingressController` + `ingressHost` + +{::nomarkdown} +
+{:/} + +#### Restore a Hybrid Runtime +Reinstall the Hybrid Runtime from the existing installation repository to restore it to the same or a different cluster. + +**Before you begin** + +* Have the following information handy: + > All values must be the identical to the Runtime to be restored. + * Runtime name + * Repository URL + * Codefresh context + * Kube context: Required if you are restoring to the same cluster + +**How to** + +1. Run: + `cf runtime install --from-repo` +1. Provide the relevant values when prompted. +1. If you are performing the runtime recovery in a different cluster, verify the ingress resource configuration for `app-proxy`, `workflows`, and `default-git-source`. + If the health status remains as `Progressing`, do the following: + + * In the Runtime installation repo, check if the `ingress.yaml` files for the `app-proxy` and `workflows` are configured with the correct `host` and `ingressClassName`: + + `apps/app-proxy/overlays//ingress.yaml` + `apps/workflows/overlays/ /ingress.yaml` + + * In the Git Source repository, check the `host` and `ingressClassName` in `cdp-default-git-source.ingress.yaml`: + + `resources_ /cdp-default-git-source.ingress.yaml` + + See the [example](#ingress-example) below. + +{:start="4"} +1. If you have managed clusters registered to the hybrid runtime you are restoring, reconnect them. + Run the command and follow the instructions in the wizard: + `cf cluster add` + +1. Verify that you have a registered Git integration: + `cf integration git list --runtime ` + +1. If needed, create a new Git integration: + `cf integration git add default --runtime --provider github --api-url https://api.github.com` + +{::nomarkdown} +
+{:/} + +#### Ingress example +This is an example of the `ingress.yaml` for `workflows`. + + ```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + annotations: + ingress.kubernetes.io/protocol: https + ingress.kubernetes.io/rewrite-target: /$2 + nginx.ingress.kubernetes.io/backend-protocol: https + nginx.ingress.kubernetes.io/rewrite-target: /$2 + creationTimestamp: null + name: runtime-name-workflows-ingress + namespace: runtime-name +spec: + ingressClassName: nginx + rules: + - host: your-ingress-host.com + http: + paths: + - backend: + service: + name: argo-server + port: + number: 2746 + path: /workflows(/|$)(.*) + pathType: ImplementationSpecific +status: + loadBalancer: {} +``` + + +### (Hybrid GitOps) Configure browser to allow insecure Runtimes + +If at least one of your Hybrid Runtimes was installed in insecure mode (without an SSL certificate for the ingress controller from a CA), the UI alerts you that _At least one runtime was installed in insecure mode_. +{% include + image.html + lightbox="true" + file="/images/runtime/runtime-insecure-alert.png" + url="/images/runtime/runtime-insecure-alert.png" + alt="Insecure runtime installation alert" + caption="Insecure runtime installation alert" + max-width="100%" +%} + +All you need to do is to configure the browser to trust the URL and receive content. + +1. Select **View Runtimes** to the right of the alert. + You are taken to the Runtimes page, where you can see insecure Runtimes tagged as **Allow Insecure**. + {% include + image.html + lightbox="true" + file="/images/runtime/runtime-insecure-steps.png" + url="/images/runtime/runtime-insecure-steps.png" + alt="Insecure runtimes in Runtime page" + caption="Insecure runtimes in Runtime page" + max-width="40%" +%} +{:start="2"} +1. For _every_ insecure Runtime, select **Allow Insecure**, and when the browser prompts you to allow access, do as relevant: + +* Chrome: Click **Advanced** and then **Proceed to site**. +* Firefox: Click **Advanced** and then **Accept the risk and continue**. +* Safari: Click **Show Certificate**, and then select **Always allow content from site**. +* Edge: Click **Advanced**, and then select **Continue to site(unsafe)**. + +### (Hybrid GitOps) View notifications in Activity Log + +The Activity Log is a quick way to monitor notifications for Runtime events such as upgrades. A pull-down panel in the Codefresh toolbar, the Activity Log shows ongoing, success, and error notifications, sorted by date, starting with today's date. + +1. In the Codefresh UI, on the top-right of the toolbar, select  **Activity Log**. +1. To see notifications for provisioned Runtimes, filter by **Runtime**. + + {% include image.html + lightbox="true" + file="/images/runtime/runtime-activity-log.png" + url="/images/runtime/runtime-activity-log.png" + alt="Activity Log filtered by Runtime events" + caption="Activity Log filtered by Runtime events" + max-width="30%" + %} + +{:start="3"} + +1. To see more information on an error, select the **+** sign. + +### (Hybrid GitOps) Troubleshoot health and sync errors for Runtimes +The  icon with the Runtime in red indicates either health or sync errors. + +**Health errors** +Health errors are generated by Argo CD and by Codefresh for Runtime components. + +**Sync errors** +Runtimes with sync errors display an **Out of sync** status in Sync Status column. They are related to discrepancies between the desired and actual state of a Runtime component or one of the Git sources associated with the Runtime. + +**View errors** +For both views, select the Runtime, and then select **Errors Detected**. +Here is an example of health errors for a Runtime. + + {% include image.html + lightbox="true" + file="/images/runtime/runtime-health-sync-errors.png" + url="/images/runtime/runtime-health-sync-errors.png" + alt="Health errors for runtime example" + caption="Health errors for runtime example" + max-width="30%" + %} + + +## Related articles +[Add Git Sources to GitOps Runtimes]({{site.baseurl}}/docs/installation/gitops/git-sources/) +[Add external clusters to GitOps Runtimes]({{site.baseurl}}/docs/installation/gitops/managed-cluster/) +[Shared configuration repo for GitOps Runtimes]({{site.baseurl}}/docs/reference/shared-configuration) + + diff --git a/_docs/installation/installation-options.md b/_docs/installation/installation-options.md new file mode 100644 index 000000000..908c507ad --- /dev/null +++ b/_docs/installation/installation-options.md @@ -0,0 +1,221 @@ +--- +title: "Installation environments" +description: "Understand Runner and GitOps installation options" +group: installation +toc: true +--- +To be changed and updated for ProjectOne + +The Codefresh platform supports three different installation options, all compliant with Soc2. + +* Hybrid Runner + The Runner installation is the hybrid installation mode for Codefresh pipelines. The Codefresh UI runs in the Codefresh cloud, and the builds run on customer premises. + The Runner combines flexibility with security, and is Enterprise customers looking for a "behind-the-firewall" solution. For a detailed look, read [Runner installation behind firewalls]({{site.baseurl}}/docs/reference/behind-the-firewall). + Pipelines created in Codefresh fetch code from your Git repository, packages/compiles the code, and deploys the final artifact to a target environment. + +* On-premises + On-premises installation is for customers who want full control over their environments. Both the UI and builds run on the Kubernetes cluster in an environment fully managed by you as our customer. + + While Codefresh can still help with maintenance of the on-premises platform, we would recommend the Hybrid Runner as it combines both flexibility and high security. + + + + +* GitOps + GitOps installation is a full-featured solution for application deployments and releases. Powered by the Argo Project, Codefresh uses Argo CD, Argo Workflows, Argo Events, and Argo Rollouts, extended with unique functionality and features essential for enterprise deployments. + + GitOps installations support Hosted and Hybrid options. + + + + + +## Hybrid Runner + +The Hybrid Runner installation is for organizations who want their source code to live within their premises, or have other security constraints. For more about the theory and implementation, see [[Runner installation behind firewalls]({{site.baseurl}}/docs/reference/behind-the-firewall). +The UI runs on Codefresh infrastructure, while the builds happen in a Kubernetes cluster in the customer's premises. + +{% include image.html + lightbox="true" + file="/images/installation/hybrid-installation.png" + url="/images/installation/hybrid-installation.png" + alt="sso-diagram.png" + max-width="70%" + %} + + +Hybrid Runner installation strikes the perfect balance between security, flexibility, and ease of use. Codefresh still does the heavy lifting for maintaining most of the platform parts. Sensitive data such as source code and internal services never leave customer premises. +Codefresh can easily connect to internal [secure services]({{site.baseurl}}/docs/reference/behind-the-firewall/#using-secure-services-in-your-pipelines) that have no public presence. +The UI is still compliant with Soc2. + + +The table lists the security implications of Hybrid Runner installation. + +{: .table .table-bordered .table-hover} +| Company Asset | Flow/Storage of data | Comments | +| -------------- | ---------------------------- |-------------------------| +| Source code | Stays behind the firewall | | +| Binary artifacts | Stay behind the firewall | | +| Build logs | Also sent to Codefresh Web application | | +| Pipeline volumes | Stay behind the firewall | | +| Pipeline variables | Defined in Codefresh Web application | | +| Deployment docker images | Stay behind the firewall| Stored on your Docker registry | +| Development docker images | Stay behind the firewall | Stored on your Docker registry| +| Testing docker images | Stay behind the firewall| Stored on your Docker registry | +| Inline pipeline definition | Defined in Codefresh Web application | | +| Pipelines as YAML file | Stay behind the firewall | | +| Test results | Stay behind the firewall | | +| HTML Test reports | Shown on Web application | Stored in your S3 or Google bucket or Azure storage | +| Production database data | Stays behind the firewall | | +| Test database data | Stays behind the firewall | | +| Other services (e.g. Queue, ESB) | Stay behind the firewall | | +| Kubernetes deployment specs | Stay behind the firewall | | +| Helm charts | Stay behind the firewall | | +| Other deployment resources/script (e.g. terraform) | Stay behind the firewall | | +| Shared configuration variables | Defined in Codefresh Web application | | +| Deployment secrets (from git/Puppet/Vault etc) | Stay behind the firewall| | +| Audit logs | Managed via Codefresh Web application | | +| SSO/Idp Configuration | Managed via Codefresh Web application | | +| User emails | Managed via Codefresh Web application | | +| Access control rules | Managed via Codefresh Web application | | + + + +## Codefresh On-premises + +For customers who want full control, Codefresh also offers on-premises installation. Both the UI and builds run on a Kubernetes cluster fully managed by the customer. + +See [Codefresh On-Prem Installation & Configuration]({{site.baseurl}}/docs/installation/codefresh-on-prem). + + +## Codefresh GitOps installation + +Codefresh GitOps also supports SaaS and hybrid installation options: + + +### Hosted GitOps +The SaaS version of GitOps, Hosted GitOps has Argo CD installed in the Codefresh cluster. +Hosted GitOps Runtime is installed and provisioned in a Codefresh cluster, and managed by Codefresh. +Hosted enviroments are full-cloud environments, where all updates and improvements are managed by Codefresh, with zero-maintenance overhead for you as the customer. +Currently, you can add one Hosted GitOps Runtime per account. +For the architecture, see [Hosted GitOps Runtime architecture]({{site.baseurl}}/docs/installation/architecture/#hosted-gitops-runtime-architecture). + + +{% include + image.html + lightbox="true" + file="/images/runtime/intro-hosted-hosted-initial-view.png" + url="/images/runtime/intro-hosted-hosted-initial-view.png" + alt="Hosted runtime setup" + caption="Hosted runtime setup" + max-width="80%" +%} + + For more information on how to set up the hosted environment, including provisioning hosted runtimes, see [Set up Hosted GitOps]({{site.baseurl}}/docs/installation/gitops/hosted-runtime/). + +### Hybrid GitOps +The hybrid version of GitOps, has Argo CD installed in the customer's cluster. +Hybrid GitOps is installed in the customer's cluster, and managed by the customer. +The Hybrid GitOps Runtime is optimal for organizations with security constraints, wanting to manage CI/CD operations within their premises. Hybrid GitOps strikes the perfect balance between security, flexibility, and ease of use. Codefresh maintains and manages most aspects of the platform, apart from installing and upgrading Hybrid GitOps Runtimes which are managed by the customer. + + +{% include + image.html + lightbox="true" + file="/images/runtime/runtime-list-view.png" + url="/images/runtime/runtime-list-view.png" + alt="Runtime List View" + caption="Runtime List View" + max-width="70%" +%} + + For more information on Hybrid GitOps, see [Hybrid GitOps runtime requirements]({{site.baseurl}}/docs/installation/gitops/hybrid-gitops/#minimum-system-requirements) and [Installling Hybrid GitOps Runtimes]({{site.baseurl}}/docs/installation/gitops/hybrid-gitops/). + + + + + +### Hosted vs.Hybrid GitOps + +The table below highlights the main differences between Hosted and Hybrid GitOps. + +{: .table .table-bordered .table-hover} +| GitOps Functionality |Feature | Hosted | Hybrid | +| -------------- | -------------- |--------------- | --------------- | +| Runtime | Installation | Provisioned by Codefresh | Provisioned by customer | +| | Runtime cluster | Managed by Codefresh | Managed by customer | +| | Number per account | One runtime | Multiple runtimes | +| | External cluster | Managed by customer | Managed by customer | +| | Upgrade | Managed by Codefresh | Managed by customer | +| | Uninstall | Managed by customer | Managed by customer | +| Argo CD | | Codefresh cluster | Customer cluster | +| CI Ops | Delivery Pipelines |Not supported | Supported | +| |Workflows | Not supported | Supported | +| |Workflow Templates | Not supported | Supported | +| CD Ops |Applications | Supported | Supported | +| |Image enrichment | Supported | Supported | +| | Rollouts | Supported | Supported | +|Integrations | | Supported | Supported | +|Dashboards |Home Analytics | Hosted runtime and deployments|Runtimes, deployments, Delivery Pipelines | +| |DORA metrics | Supported |Supported | +| |Applications | Supported |Supported | + + +## Installation options comparison +Codefresh Runner and GitOps environments can co-exist giving you the best of both worlds. + +{: .table .table-bordered .table-hover} +| Characteristic | Hybrid Runner | On Premise | GitOps +| -------------- | ---------------------------- |-------------------------| ----------------| +| Managed by | Codefresh and customer | Customer | Codefresh and customer | +| UI runs on | Public cloud | Private cluster | Public cloud| +| Builds run on | Private cluster | Private cluster | Private cluster (Hybrid)/Codefresh cluster (Hosted)| +| Access to secure/private services | Yes | Yes | Yes | +| Customer maintenance effort | Some | Full | Some | +| Best for | Companies with security constraints | Large scale installations | Companies with security constraints | +| Available to |[Enterprise plans](https://codefresh.io/contact-us/){:target="\_blank"} | [Enterprise plans](https://codefresh.io/contact-us/) |[Enterprise plans](https://codefresh.io/contact-us/) | + + +## Related articles +[Architecture]({{site.baseurl}}/docs/installation/runtime-architecture/) +[Add Git Sources to GitOps Runtimes]({{site.baseurl}}/docs/installation/gitops/git-sources/) +[Shared configuration repository]({{site.baseurl}}/docs/reference/shared-configuration) + diff --git a/_docs/installation/runtime-architecture.md b/_docs/installation/runtime-architecture.md new file mode 100644 index 000000000..f6ec3650a --- /dev/null +++ b/_docs/installation/runtime-architecture.md @@ -0,0 +1,244 @@ +--- +title: "Runtime architecture" +description: "" +group: installation +toc: true +--- + +If you have familiarized yourself with the different installation, here's a deep dive into the architecture and components of Codefresh Runner and GitOps runtime architectures. + +## Runner architecture + +The most important components are the following: + +**Codefresh VPC:** All internal Codefresh services run in the VPC (analyzed in the next section). Codefresh uses Mongo and PostgreSQL to store user and authentication information. + +**Pipeline execution environment**: The Codefresh engine component is responsible for taking pipeline definitions and running them in managed Kubernetes clusters by automatically launching the Docker containers that each pipeline needs for its steps. + +**External actors**. Codefresh offers a [public API]({{site.baseurl}}/docs/integrations/ci-integrations/codefresh-api/) that is consumed both by the Web user interface and the [Codefresh CLI](https://codefresh-io.github.io/cli/){:target="\_blank"}. The API is also available for any custom integration with external tools or services. + +### Runner topology + +If we zoom into Hybrid Runner services, we will see the following: + +{% include image.html + lightbox="true" + file="/images/installation/topology-new.png" + url="/images/installation/topology-new.png" + alt="Topology diagram" + caption="Topology diagram" + max-width="100%" + %} + +### Runner core components + +{: .table .table-bordered .table-hover} +|Category | Component | Function | +| -------------- | ----------| ----------| +| Core | **pipeline-manager**| Manages all CRUD operations for CI pipelines.| +| | **cfsign** | Signs server TLS certificates for docker daemons, and generates client TLS certificates for hybrid pipelines. | +| | **cf-api** | Central back-end component that functions as an API gateway for other services, and handles authentication/authorization. | +| | **context-manager**| Manages the authentications/configurations used by Codefresh CI/CD and by the Codefresh engine. | +| | **runtime-environment-manager**| Manages the different runtime environments for CI pipelines. The runtime environment for CI/CD SaaS is fully managed by Codefresh. For CI/CD Hybrid, customers can add their own runtime environments using private Kubernetes clusters. | +| Trigger | **hermes**| Controls CI pipeline trigger management. See [triggers]({{site.baseurl}}/docs/pipelines/triggers/). | +| | **nomios**| Enables triggers from Docker Hub when a new image/tag is pushed.See [Triggers from Docker Hub]({{site.baseurl}}/docs/pipelines/triggers/dockerhub-triggers/). | +| | **cronus**| Enables defining Cron triggers for CI pipelines. See [Cron triggers]({{site.baseurl}}/docs/pipelines/triggers/cron-triggers/).| +| Log | **cf-broadcaster**| Stores build logs from CI pipelines. The UI and CLI stream logs by accessing the **cf-broadcaster** through a web socket. | +| Kubernetes | **cluster-providers** | Provides an interface to define cluster contexts to connect Kubernetes clusters in CI/CD installation environments. | +| | **helm-repo-manager** | Manages the Helm charts for CI/CD installation environments through the Helm repository admin API and ChartMuseum proxy. See [Helm charts in Codefresh]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/). | +| | **k8s-monitor** | The agent installed on every Kubernetes cluster, providing information for the Kubernetes dashboards. See [Kubernetes dashboards]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/). | +| |**charts-manager** | Models the Helm chart view in Codefresh. See [Helm chart view]({{site.baseurl}}/docs/deployments/helm/helm-releases-management/). | +| | **kube-integration** | Provides an interface to retrieve required information from a Kubernetes cluster, can be run either as an http server or an NPM module. | +| | **tasker-kubernetes** | Provides cache storage for Kubernetes dashboards. See [Kubernetes dashboards]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/). | + + +## GitOps architecture + +The diagram shows a high-level view of the GitOps environment, and its core components, the Codefresh Control Plane, the Codefresh Runtime, and the Codefresh Clients. + +{% include +image.html +lightbox="true" +file="/images/getting-started/architecture/arch-codefresh-simple.png" +url="/images/getting-started/architecture/arch-codefresh-simple.png" +alt="Codefresh GitOps Platform architecture" +caption="Codefresh GitOps Platform architecture" +max-width="100%" +%} + +{::nomarkdown} +
+{:/} + +### GitOps Control Plane +The Codefresh Control Plane is the SaaS component in the platform. External to the enterprise firewall, it does not have direct communication with the Codefresh Runtime, Codefresh Clients, or the customer's organizational systems. The Codefresh Runtime and the Codefresh Clients communicate with the Codefresh Control Plane to retrieve the required information. + + +{::nomarkdown} +
+{:/} + +### GitOps Runtime +The GitOps Runtime is installed on a Kubernetes cluster, and houses the enterprise distribution of the Codefresh Application Proxy and the Argo Project. +Depending on the type of GitOps installation, the GitOps Runtime is installed either in the Codefresh platform (Hosted GitOps), or in the customer environment (Hybrid GitOps). Read more in [Codefresh GitOps Runtime architecture](#codefresh-gitops-runtime-architecture). + + +{::nomarkdown} +
+{:/} + +### GitOps Clients + +GitOps Clients include the UI and the GitOps CLI. +The UI provides a unified, enterprise-wide view of deployments (runtimes and clusters), and CI/CD operations (Delivery Pipelines, workflows, and deployments) in the same location. +The Codefresh CLI includes commands to install hybrid runtimes, add external clusters, and manage runtimes and clusters. + +### GitOps Runtime architecture +The sections that follow show detailed views of the GitOps Runtime architecture for the different installation options, and descriptions of the GitOps Runtime components. + +* [Hosted GitOps runtime architecture](#hosted-gitops-runtime-architecture) + For Hosted GitOps, the GitOps Runtime is installed on a _Codefresh-managed cluster_ in the Codefresh platform. +* Hybrid GitOps runtime architecture: + For Hybrid GitOps, the GitOps Runtime is installed on a _customer-managed cluster_ in the customer environment. The Hybrid GitOps Runtime can be tunnel- or ingress-based: + * [Tunnel-based](#tunnel-based-hybrid-gitops-runtime-architecture) + * [Ingress-based](#ingress-based-hybrid-gitops-runtime-architecture) +* GitOps Runtime components + * [Application Proxy](#application-proxy) + * [Argo Project](#argo-project) + * [Request Routing Service](#request-routing-service) + * [Tunnel Server](#tunnel-server) + * [Tunnel Client](#tunnel-client) + + +#### Hosted GitOps runtime architecture +In the hosted environment, the Codefresh Runtime is installed on a K8s cluster managed by Codefresh. + +{% include + image.html + lightbox="true" + file="/images/getting-started/architecture/arch-hosted.png" + url="/images/getting-started/architecture/arch-hosted.png" + alt="Hosted runtime architecture" + caption="Hosted runtime architecture" + max-width="100%" +%} + +#### Tunnel-based Hybrid GitOps runtime architecture +Tunnel-based Hybrid GitOps runtimes use tunneling instead of ingress controllers to control communication between the GitOps Runtime in the customer cluster and the Codefresh GitOps Platform. Tunnel-based runtimes are optimal when the cluster with the GitOps Runtime is not exposed to the internet. + +{% include + image.html + lightbox="true" + file="/images/getting-started/architecture/arch-hybrid-ingressless.png" + url="/images/getting-started/architecture/arch-hybrid-ingressless.png" + alt="Tunnel-based hybrid runtime architecture" + caption="Tunnel-based hybrid runtime architecture" + max-width="100%" +%} + + +#### Ingress-based Hybrid GitOps runtime architecture +Ingress-based runtimes use ingress controllers to control communication between the GitOps Runtime in the customer cluster and the Codefresh GitOps Platform. Ingress-based runtimes are optimal when the cluster with the GitOps Runtime is exposed to the internet. + + + +{% include + image.html + lightbox="true" + file="/images/getting-started/architecture/arch-hybrid-ingress.png" + url="/images/getting-started/architecture/arch-hybrid-ingress.png" + alt="Ingress-based hybrid runtime architecture" + caption="Ingress-based hybrid runtime architecture" + max-width="100%" +%} + + +#### Application Proxy +The GitOps Application Proxy (App-Proxy) functions as the Codefresh agent, and is deployed as a service in the GitOps Runtime. + +For tunnel-based Hybrid GitOps Runtimes, the Tunnel Client forwards the incoming traffic from the Tunnel Server using the Request Routing Service to the GitOps App-Proxy. +For Hybrid GitOps Runtimes with ingress, the App-Proxy is the single point-of-contact between the GitOps Runtime, and the GitOps Clients, the GitOps Platform, and any organizational systems in the customer environment. + + +The GitOps App-Proxy: +* Accepts and serves requests from GitOps Clients either via the UI or CLI +* Retrieves a list of Git repositories for visualization in the Client interfaces +* Retrieves permissions from the GitOps Control Plane to authenticate and authorize users for the required operations. +* Implements commits for GitOps-controlled entities, such as Delivery Pipelines and other CI resources +* Implements state-change operations for non-GitOps controlled entities, such as terminating Argo Workflows + +{::nomarkdown} +
+{:/} + +#### Argo Project + +The Argo Project includes: +* Argo CD for declarative continuous deployment +* Argo Rollouts for progressive delivery +* Argo Workflows as the workflow engine +* Argo Events for event-driven workflow automation framework + +>Codefresh users rely on our platform to deliver software reliably, and predictably without interruption. + To maintain that high standard, we add several weeks of testing and bug fixes to new versions of Argo before making them available within Codefresh. + Typically, new versions of Argo are available within 30 days of release in Argo. + +{::nomarkdown} +
+{:/} + +#### Request Routing Service +The Request Routing Service is installed on the same cluster as the GitOps Runtime in the customer environment. +It receives requests from the the Tunnel Client (tunnel-based) or the ingress controller (ingress-based), and forwards the request URLs to the Application Proxy, and webhooks directly to the Event Sources. + +>Important: + The Request Routing Service is available from runtime version 0.0.543 and higher. + Older runtime versions are not affected as there is complete backward compatibility, and the ingress controller continues to route incoming requests. + +#### Tunnel Server +Applies only to _tunnel-based_ Hybrid GitOps Runtimes. +The Codefresh Tunnel Server is installed in the Codefresh platform. It communicates with the enterprise cluster located behind a NAT or firewall. + +The Tunnel Server: +* Forwards traffic from Codefresh Clients to the client (customer) cluster. +* Manages the lifecycle of the Tunnel Client. +* Authenticates requests from the Tunnel Client to open tunneling connections. + +{::nomarkdown} +
+{:/} + +#### Tunnel Client +Applies only to _tunnel-based_ Hybrid GitOps Runtimes. + +Installed on the same cluster as the Hybrid GitOps Runtime, the Tunnel Client establishes the tunneling connection to the Tunnel Server via the WebSocket Secure (WSS) protocol. +A single Hybrid GitOps Runtime can have a single Tunnel Client. + +The Tunnel Client: +* Initiates the connection with the Tunnel Server. +* Forwards the incoming traffic from the Tunnel Server through the Request Routing Service to App-Proxy, and other services. + +{::nomarkdown} +
+{:/} + + +#### Customer environment +The customer environment that communicates with the GitOps Runtime and Codefresh, generally includes: +* Ingress controller for ingress-based Hybrid runtimes + The ingress controller is configured on the same Kubernetes cluster as the GitOps Runtime, and implements the ingress traffic rules for the GitOps Runtime. + See [Ingress controller requirements]({{site.baseurl}}/docs/installation/gitops/monitor-manage-runtimes/#ingress-controller). +* Managed clusters + Managed clusters are external clusters registered to provisioned Hosted or Hybrid GitOps runtimes for application deployment. + Hosted GitOps requires you to connect at least one external K8s cluster as part of setting up the Hosted GitOps environment. + Hybrid GitOps allow you to add external clusters after provisioning the runtimes. + See [Add external clusters to runtimes]({{site.baseurl}}/docs/installation/gitops/managed-cluster/). +* Organizational systems + Organizational Systems include the customer's tracking, monitoring, notification, container registries, Git providers, and other systems. They can be entirely on-premises or in the public cloud. + Either the ingress controller (ingress hybrid environments), or the Tunnel Client (tunnel-based hybrid environments), forwards incoming events to the GitOps Application Proxy. + + ## Related articles +[Codefresh pricing](https://codefresh.io/pricing/){:target="\_blank"} +[Codefresh features](https://codefresh.io/features/){:target="\_blank"} + + \ No newline at end of file diff --git a/_docs/integrations/amazon-web-services.md b/_docs/integrations/amazon-web-services.md new file mode 100644 index 000000000..28494a1f2 --- /dev/null +++ b/_docs/integrations/amazon-web-services.md @@ -0,0 +1,93 @@ +--- +title: "Amazon Web Services (AWS) pipeline integration" +description: "How to use Codefresh with AWS" +group: integrations +toc: true +--- + +Codefresh has native support for AWS in the following areas: + +- [Connecting to Amazon registries]({{site.baseurl}}/docs/integrations/docker-registries/amazon-ec2-container-registry/) +- [Deploying to Amazon EKS]({{site.baseurl}}/docs/integrations/kubernetes/#adding-eks-cluster) +- [Using Amazon S3 for Test reports]({{site.baseurl}}/docs/testing/test-reports/#connecting-an-s3-bucket) +- [Using Amazon S3 for Helm charts]({{site.baseurl}}/docs/deployments/helm/add-helm-repository/#private-repository---s3) + + +## Using Amazon ECR + +Amazon Container Registries are fully compliant with the Docker registry API that Codefresh follows. Follow the instruction under [Amazon EC2 Container Registry]({{site.baseurl}}/docs/integrations/docker-registries/amazon-ec2-container-registry/) to connect. + +Once the registry is added, you can use the [standard push step]({{site.baseurl}}/docs/pipelines/steps/push/) in your pipelines. See [working with Docker registries]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/) for more information. + +## Deploying to Amazon Kubernetes + +Codefresh has native support for connecting an EKS cluster in the [cluster configuration screen]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster). + +{% + include image.html + lightbox="true" +file="/images/integrations/aws/aws-integration.png" +url="/images/integrations/aws/aws-integration.png" +alt="Connecting an Amazon cluster" +caption="Connecting a Amazon cluster" +max-width="40%" +%} + +Once the cluster is connected, you can use any of the [available deployment options]({{site.baseurl}}/docs/deployments/kubernetes/deployment-options-to-kubernetes/) for Kubernetes clusters. You also get access to all other Kubernetes dashboards such as the [cluster dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) and the [environment dashboard]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/). + +## Storing test reports in Amazon S3 bucket + +Codefresh has native support for test reports. You can store the reports on Amazon S3. + +{% include +image.html +lightbox="true" +file="/images/integrations/aws/amazon-storage.png" +url="/images/integrations/aws/amazon-storage.png" +alt="Amazon cloud storage" +caption="Amazon cloud storage" +max-width="60%" +%} + +See the full documentation for [test reports]({{site.baseurl}}/docs/testing/test-reports/). + +## Using Amazon S3 for storing Helm charts + +You can connect an Amazon S3 bucket as a Helm repository in the [integrations screen]({{site.baseurl}}/docs/deployments/helm/add-helm-repository/). + +{% include +image.html +lightbox="true" +file="/images/integrations/aws/amazon-s3-helm-repo.png" +url="/images/integrations/aws/amazon-s3-helm-repo.png" +alt="Using Amazon for Helm charts" +caption="Using Amazon for Helm charts" +max-width="80%" +%} + +Once you connect your Helm repository you can use it any [Codefresh pipeline with the Helm step]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/). + + +## Traditional Amazon deployments + +For any other Amazon deployment you can use the [Amazon CLI from a Docker image](https://hub.docker.com/r/amazon/aws-cli){:target="\_blank"} in a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/) + +`YAML` +{% highlight yaml %} +{% raw %} + create_a_vm: + title: "Creating a Virtual machine" + type: "freestyle" + arguments: + image: "amazon/aws-cli" + commands: + - aws ec2 run-instances --image-id ami-xxxxxxxx --count 1 --instance-type t2.micro --key-name MyKeyPair --security-group-ids sg-903004f8 --subnet-id subnet-6e7f829e +{% endraw %} +{% endhighlight %} + + +## Related articles +[Add your cluster]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster) +[Manage your Kubernetes cluster]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) +[Cloning Git repositories]({{site.baseurl}}/docs/example-catalog/ci-examples/git-checkout/) + diff --git a/_docs/integrations/argocd.md b/_docs/integrations/argocd.md new file mode 100644 index 000000000..2889c926e --- /dev/null +++ b/_docs/integrations/argocd.md @@ -0,0 +1,177 @@ +--- +title: "ArgoCD integration for CI pipelines" +description: "Connect Codefresh to your ArgoCD endpoint" +group: integrations +toc: true +--- + + +Before you can use Codefresh and ArgoCD together, you need to connect your ArgoCD installation in your Codefresh account. This way Codefresh will send and receive information from your ArgoCD instance. + +{% include image.html + lightbox="true" + file="/images/guides/gitops/gitops-environment.png" + url="/images/guides/gitops/gitops-environment.png" + alt="GitOps deployments with Codefresh" + caption="GitOps deployments with Codefresh" + max-width="100%" + %} + +>Important: + Codefresh has a --> + +## Set up ArgoCD integration in Codefresh + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **GitOps** and then click **Configure**. +1. From the **Add GitOps Provider** dropdown, select **ArgoCD**. +1. Follow the on-screen instructions to complete the integration. + +### Codefresh CLI +To connect to an existing ArgoCD installation: +1. Install the [Codefresh CLI](https://codefresh-io.github.io/cli/){:target="\_blank"} by following the [documentation](https://codefresh-io.github.io/cli/installation/){:target="\_blank"}. + The Codefresh CLI installs an agent in your cluster, in the same namespace that ArgoCD runs in. The agent handles all communication between ArgoCD and Codefresh. +1. Authenticate the CLI with your Codefresh account [by creating an API token]({{site.baseurl}}/docs/integrations/codefresh-api/#authentication-instructions). Make sure that you choose all scopes if this is the first time you are authenticating your CLI with Codefresh. +1. From a workstation that has a `kubeconfig` context pointing to the ArgoCD cluster, run the installation command: + +``` +codefresh install gitops argocd-agent +``` + +>You can also run it from your cloud console if you install codefresh CLI there. + +1. Answer the questions asked by the wizard. These include: + + * The name of the integration (user-defined) + * Your ArgoCD URL, username and password (you can also use [an auth token](https://argoproj.github.io/argo-cd/operator-manual/user-management/){:target="\_blank"} instead of password) + * The context and namespace in the cluster where ArgoCD is installed + * If you want to automatically import your ArgoCD applications to Codefresh + +``` +codefresh install gitops argocd-agent +This installer will guide you through the Codefresh ArgoCD installation agent to integrate your ArgoCD with Codefresh +? Select Kubernetes context mydemoAkscluster +? Codefresh integration name argocd +? Choose an authentication method Username and password +? Argo username admin +? Argo password ***************************** + +Testing requirements +-------------------- +√ checking argocd credentials... +√ checking argocd projects accessibility... +√ checking argocd applications accessibility... +-------------------- + +? Select Git/GithubApp context (Please create a dedicated context for the agent to avoid hitting the Github rate limits or use github app integration) github-1 +? Select argocd sync behavior please Import all existing Argo applications to Codefresh +? Enable auto-sync of applications, this will import all existing applications and update Codefresh in the future Yes + +Installation options summary: + 1. Kubernetes Context: + 2. Kubernetes Namespace: argocd + 3. Git Integration: github-1 + 4. Codefresh Host: https://g.codefresh.io + 5. ArgoCD Host: https://52.154.209.119 + 6. ArgoCD Username: admin + 7. ArgoCD Password: ****** + 8. Enable auto-sync of applications: Yes + 9. HTTP proxy: none + 10. HTTPS proxy: none + +Argo agent installation finished successfully to namespace "argocd" +Gitops view: "https://g.codefresh.io/gitops" +Documentation: "https://codefresh.io/docs/docs/ci-cd-guides/gitops-deployments/" +``` + +Once the installation is complete, you should see the agent's health status: + +{% include image.html + lightbox="true" + file="/images/integrations/argocd/argocd-agent-health.png" + url="/images/integrations/argocd/argocd-agent-health.png" + alt="ArgoCD agent health" + caption="ArgoCD agent health" + max-width="100%" + %} + + +This concludes the basic integration. You can repeat the procedure for different ArgoCD installations by choosing a different +name for the integration. + +## Creating ArgoCD applications + +In addition to the existing [Kubernetes/Helm environments]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/), you can now create ArgoCD applications via the Codefresh UI. + +Visit your GitOps dashboard by clicking on *GitOps* from the left sidebar. The click the *Add Application* button at the top right. + +If you already have an application set up in ArgoCD, you can enter its project and name and Codefresh will automatically retrieve all information from the ArgoCD instance. + +{% include image.html + lightbox="true" + file="/images/integrations/argocd/argocd-existing-app.png" + url="/images/integrations/argocd/argocd-existing-app.png" + alt="Using an existing ArgoCD application in a Codefresh environment" + caption="Using an existing ArgoCD application in a Codefresh environment" + max-width="60%" + %} + +You can also create a brand-new application with the *provision* option. In this dialog you can enter the exact same details that [ArgoCD asks when creating a new application](https://argoproj.github.io/argo-cd/getting_started/#6-create-an-application-from-a-git-repository). + +{% include image.html + lightbox="true" + file="/images/integrations/argocd/argocd-provision-app.png" + url="/images/integrations/argocd/argocd-provision-app.png" + alt="Creating a new ArgoCD application in a Codefresh environment" + caption="Creating a new ArgoCD application in a Codefresh environment" + max-width="60%" + %} + +The options are: + +* Name - User defined name of the Codefresh environment dashboard +* Project - A way to [group/secure applications](https://argoproj.github.io/argo-cd/user-guide/projects/). Choose default if you have only one project in ArgoCD. +* Application - name of application +* Manual/automatic sync - If automatic when a git commit happens, a deployment will automatically take place. +* Use schema - Kubernetes manifests will be checked for correctness before deployed to the cluster +* source repository - Git repository that holds your Kubernetes manifests +* revision - Revision to be checked out when a deployment happens +* path - folder inside the Git repository that should be searched for manifests (if your Git repo has multiple applications). Use `./` if all your manifests are in the root folder. +* cluster - Kubernetes cluster when deployment will take place +* namespace - Kubernetes namespace where the application will be deployed to +* directory recurse - whether to check all folders in the Git repository for manifests in a recursive way. + +For a sample application you can use the [https://github.com/codefresh-contrib/gitops-kubernetes-configuration](https://github.com/codefresh-contrib/gitops-kubernetes-configuration) repository (or even fork it on your own GitHub account first). + +Codefresh will communicate with ArgoCD via its API and pass all the relevant details. + +The end result is a new entry for your ArgoCD application will now appear in the dashboard along with the sync status. + +{% include image.html + lightbox="true" + file="/images/integrations/argocd/argocd-environment.png" + url="/images/integrations/argocd/argocd-environment.png" + alt="ArgoCD environment status" + caption="ArgoCD environment status" + max-width="80%" + %} + +To learn about the full GitOps support in Codefresh, see our [GitOps deployment guide]({{site.baseurl}}/docs/ci-cd-guides/gitops-deployments/). + +## Uninstall the gitops agent + +You can uninstall the gitops agent with : + +``` +codefresh uninstall gitops argocd-agent +``` + +Note this will only uninstall the Codefresh agent. Your Argo CD installation +will remain unaffected. + +## Related articles +[Environment Dashboard]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/) +[Kubernetes integration]({{site.baseurl}}/docs/integrations/kubernetes/) + + + diff --git a/_docs/integrations/codecov-integration.md b/_docs/integrations/codecov-integration.md new file mode 100644 index 000000000..4f7b2f303 --- /dev/null +++ b/_docs/integrations/codecov-integration.md @@ -0,0 +1,71 @@ +--- +title: "Codecov integration for CI pipelines" +description: "Create Code Coverage Reports with Codefresh and Codecov" +group: integrations +toc: true +--- + +Codefresh has native integration for [Codecov analysis](https://about.codecov.io/){:target="\_blank"}. +You need to first set up a new project in Codecov. + +## Set up a new project in Codecov + +* Sign up for a free account with Codecov. +* Add a new project. + +{% include image.html +lightbox="true" +file="/images/integrations/codecov-integration/codecovtoken.png" +url="/images/integrations/codecov-integration/codecovtoken.png" +max-width="70%" +caption="Getting a Token from Codecov" +alt="Getting a Token from Codecov" +%} + +* Note down the Token as you will need it to set up the Codecov integration in Codefresh. + +## Set up Codecov integration in Codefresh + + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Codecov** and then click **Configure**. +1. Click **Add Codecov**. +1. Define the following: + * **Integration Name**: Enter a name for the integration which is used to reference it in `codefresh.yaml`. + * **Token**: Paste the token that you copied when you created the new Codecov project for this integration. + * **Url**: The base URL for this integration. Do not add the trailing slash to the URL definition. For more information, see the [official Codecov documentation](https://docs.codecov.com/docs/configuration#codecov-url){:target="\_blank"}. + + +{% include image.html +lightbox="true" +file="/images/integrations/codecov-integration/codecovintegration.png" +url="/images/integrations/codecov-integration/codecovintegration.png" +max-width="70%" +caption="Enter Token" +alt="Enter Token" +%} + + + +## Using Codecov in a CI pipeline + +With the integration in place, you can reference it by name in any Codefresh pipeline by using the [Codecov reporter step](https://codefresh.io/steps/step/codecov-reporter){:target="\_blank"}. + +`codefresh.yml` +```yaml + codecov-report: + stage: "prepare" + title: Codecov report + type: codecov-reporter + arguments: + codecov_integration: my-codecov-integration +``` + +For more details see our [Codecov example](https://codefresh.io/docs/docs/example-catalog/ci-examples/codecov-testing/). + +## Related articles +[Integration Tests]({{site.baseurl}}/docs/testing/integration-tests/) +[Service Containers]({{site.baseurl}}/docs/pipelines/service-containers/) +[Coveralls Example]({{site.baseurl}}/docs/example-catalog/ci-examples/coveralls-testing/) +[Codacy Example]({{site.baseurl}}/docs/example-catalog/ci-examples/codacy-testing/) +[Test Reports]({{site.baseurl}}/docs/testing/test-reports/) \ No newline at end of file diff --git a/_docs/integrations/codefresh-api.md b/_docs/integrations/codefresh-api.md new file mode 100644 index 000000000..1ec768888 --- /dev/null +++ b/_docs/integrations/codefresh-api.md @@ -0,0 +1,527 @@ +--- +title: "Codefresh API pipeline integration" +description: "Integrate Codefresh CI pipelines with other systems" +group: integrations +redirect_from: + - /docs/codefresh-api/ +toc: true +old_url: /docs/codefresh-api +--- + +Codefresh offers a comprehensive API that you can use to integrate with any other application or solution you already have. + +The full details of the API are documented at [https://g.codefresh.io/api/](https://g.codefresh.io/api/){:target="\_blank"}. + +{% include image.html +lightbox="true" +file="/images/integrations/api/overview.png" +url="/images/integrations/api/overview.png" +alt="Using the Codefresh API" +max-width="70%" +%} + +You can use the API in various ways: + +* From your local workstation, with any tool that speaks HTTP (such as [postman](https://github.com/postmanlabs){:target="\_blank"}, [httpie](https://httpie.org/){:target="\_blank"}, [curl](https://curl.haxx.se/){:target="\_blank"} etc.). +* From another HTTP-enabled tool such as Jenkins. For example, you can trigger [Codefresh pipelines from Jenkins jobs]({{site.baseurl}}/docs/integrations/jenkins-integration/#calling-codefresh-pipelines-from-jenkins-jobs). +* Using the [Codefresh command line interface](https://codefresh-io.github.io/cli/){:target="\_blank"} which itself uses the API. +* Calling it programmatically from any other system. You can use your favorite programming language to make HTTP calls to Codefresh. + + +The Codefresh API is updated when new features are added in the Codefresh platform so you can expect any new functionality +to appear in the API as well. + +## Ways to use the Codefresh API + +There are several ways to use the API. Some of the most popular ones are: + + +1. Triggering builds from another system. You can start a Codefresh pipeline from any other internal system that you already have in your organization. +1. Getting the status of builds in another system. +1. Creating pipelines externally. You don't have to use the Codefresh UI to create pipelines. You can create them programmatically using your favorite template mechanism. You can reuse pipelines using your own custom implementation if you have special needs in your organization. + +You can browse the current API at [https://g.codefresh.io/api/](https://g.codefresh.io/api/){:target="\_blank"}. + +{% include image.html +lightbox="true" +file="/images/integrations/api/codefresh-api-example.png" +url="/images/integrations/api/codefresh-api-example.png" +alt="Browsing the Codefresh API" +caption="Browsing the Codefresh API" +max-width="70%" +%} + +For each call you will also see an example with `curl`. + +## Authentication instructions + + +1. Log in to your Codefresh account, and from your avatar dropdown, select [**User Settings**](https://g.codefresh.io/user/settings){:target="\_blank"}. +1. Scroll down to **API Keys**. +1. To create a new API key, click **Generate**, and do the following: + * **Key Name**: Enter the name of the key, preferable one that will help you remember its purpose. The token is tied to your Codefresh account and should be considered sensitive information. + * **Scopes**: Select the required [access scopes](#access-scopes). +1. Copy the token to your clipboard. +1. Click **Create**. + +{% include image.html +lightbox="true" +file="/images/integrations/api/generate-token.png" +url="/images/integrations/api/generate-token.png" +alt="Generating a key for the API" +caption="Generating a key for the API" +max-width="70%" +%} + + +From the same screen you can also revoke keys if you don't need them anymore. + +### Access scopes + +The following resources can be targeted with the API: + +* *Agent* - Used for [Codefresh Runner installation]({{site.baseurl}}/docs/reference/behind-the-firewall/) +* *Audit* - Read [Audit logs]({{site.baseurl}}/docs/administration/audit-logs/) +* *Build* - Get/change [build status]({{site.baseurl}}/docs/pipelines/monitoring-pipelines/) +* *Cluster* - [Access control]({{site.baseurl}}/docs/administration/access-control/) for [Kubernetes clusters]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) +* *Environments-v2* - Read/Write [Environment Dashboard]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/) information +* *GitHub Actions* - Run [GitHub Actions inside Codefresh pipelines]({{site.baseurl}}/docs/integrations/github-actions/) +* *Pipeline* - [Access control]({{site.baseurl}}/docs/administration/access-control/) for [pipelines]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/) +* *Repos* - Refers to [Git repositories]({{site.baseurl}}/docs/integrations/git-providers/) +* *Step Type* - Refers to [custom pipeline steps]({{site.baseurl}}/docs/pipelines/steps/#creating-a-typed-codefresh-plugin) + + +The scopes available for each resource differ according to the type of resource. + + +## Using the API Key with the Codefresh CLI + +Once you have the key, use it in the Codefresh CLI: + +{% highlight bash %} +codefresh auth create-context--api-key +{% endhighlight %} + +Now the Codefresh CLI is fully authenticated. The key is stored in `~/.cfconfig` so you only need to run this command once. The CLI +can also work with [multiple authentication contexts](https://codefresh-io.github.io/cli/authentication/){:target="\_blank"} so you can manage multiple Codefresh accounts at the same time. + +## Example: Triggering pipelines + +You can trigger any pipeline in Codefresh and even pass extra environment variables (even if they are not +declared in the UI). + +Triggering a pipeline via the Codefresh CLI: + +{% highlight bash %} +codefresh run kostis-codefresh/nestjs-example/ci-build -b master -t nestjs-example-trigger-name +{% endhighlight %} + +You can pass extra environment variables as well: +{% highlight bash %} +codefresh run kostis-codefresh/nestjs-example/ci-build -b master -t nestjs-example-trigger-name -v sample-var1=sample1 -v SAMPLE_VAR2=SAMPLE2 +{% endhighlight %} + +For the API, you can trigger a pipeline by finding its `serviceId` from the UI + +{% highlight bash %} +curl 'https://g.codefresh.io/api/builds/5b1a78d1bdbf074c8a9b3458' --compressed -H 'content-type:application/json; charset=utf-8' -H 'Authorization: ' --data-binary '{"serviceId":"5b1a78d1bdbf074c8a9b3458","type":"build","repoOwner":"kostis-codefresh","branch":"master","repoName":"nestjs-example"}' +{% endhighlight %} + +You can also pass extra environment variables using an array + +{% highlight bash %} +curl 'https://g.codefresh.io/api/builds/5b1a78d1bdbf074c8a9b3458' --compressed -H 'content-type:application/json; charset=utf-8' -H 'Authorization: ' --data-binary '{"serviceId":"5b1a78d1bdbf074c8a9b3458","type":"build","repoOwner":"kostis-codefresh","branch":"master","repoName":"nestjs-example","variables":{"sample-var1":"sample1","SAMPLE_VAR2":"SAMPLE2"}}' +{% endhighlight %} + +## Example: Getting status from builds + +You can get the status of a build from the CLI by using the build ID: + +{% highlight bash %} +codefresh get builds 5b4f927dc70d080001536fe3 +{% endhighlight %} + +Same thing with the API: + +{% highlight bash %} +curl -X GET --header "Accept: application/json" --header "Authorization: " "https://g.codefresh.io/api/builds/5b4f927dc70d080001536fe3" +{% endhighlight %} + +## Example: Creating Codefresh pipelines externally + +Codefresh has a great UI for creating pipelines for each of your projects. If you wish, you can also create pipelines +programmatically in an external manner. This allows you to use your own templating solution for re-using pipelines +and creating them from an external system. + +First you need a YAML file that defines the pipeline. This is a pipeline [specification](#full-pipeline-specification). + +>It is also very easy to create a a dummy pipeline in the Codefresh UI and then get its specification by running `codefresh get pipeline my-project/my-pipeline -o yaml > my-pipeline-spec.yml` + +Here is an example + +`Pipeline Spec` +{% highlight yaml %} +{% raw %} +version: '1.0' +kind: pipeline +metadata: + name: my-project/my-basic-pipeline + description: my description + labels: + key1: value1 + key2: value2 + deprecate: + applicationPort: '8080' + project: my-project +spec: + triggers: + - type: git + provider: github + name: my-trigger + repo: kostis-codefresh/nestjs-example + events: + - push + branchRegex: /./ + contexts: [] + variables: + - key: PORT + value: 5000 + encrypted: false + - key: SECRET + value: "secret-value" + encrypted: true + steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}' + revision: '${{CF_REVISION}}' + git: github-1 + PrintFileList: + title: Listing files + image: 'alpine:latest' + commands: + - ls -l + stages: [] +{% endraw %} +{% endhighlight %} + +Save this spec into a file with an arbitrary name like `my-pipeline-spec.yml`. First create the new project (if it doesn't exist already): + +{% highlight bash %} +codefresh create project my-project +{% endhighlight %} + +Then you can create the pipeline with the cli + +{% highlight bash %} +codefresh create pipeline -f my-pipeline-spec.yml +{% endhighlight %} + +And your pipeline will be available in the GUI + +{% include image.html +lightbox="true" +file="/images/integrations/api/creation-of-pipeline.png" +url="/images/integrations/api/creation-of-pipeline.png" +alt="Created Pipeline" +caption="New pipeline created" +max-width="70%" +%} + +Notice that you must prefix the name of the pipeline with your username and repository so that it becomes +visible in the GUI under the correct project. + +## Full pipeline specification + +If you don't want to create a pipeline from an existing one, you can also create your own YAML from scratch. +The following sections contain an explanation of the fields. +> Codefresh automatically generates additional fields, usually fields with dates and internal ID numbers. While you cannot edit these fields, you can view them by exporting the pipeline. + +### Top level fields + +{: .table .table-bordered .table-hover} +| Field name | Parent field | Type | Value | +| -------------- | ---------------------------- |-------------------------| -------------------------| +| `version` | | string | Always `'1.0'` | +| `kind` | | string | Always `pipeline` | +| `metadata` | | object | Holds various meta-information | +| `spec` | | object | Holds the pipeline definition and other related information | + + +### Metadata fields + +{: .table .table-bordered .table-hover} +| Field name | Parent field | Type | Value | +| -------------- | ---------------------------- |-------------------------| -------------------------| +| `name` | `metadata` | string | the full pipeline name should be formatted `project_name/pipeline_name` | +| `project` | `metadata` | string | the project that contains this pipeline | +| `originalYamlString` | `metadata` | string | the full contents of the pipeline editor. Only kept for archival purposes | +| `labels` | `metadata` | object | Holds the `tags` array | +| `tags` | `labels` | array | A list of [access control tags]({{site.baseurl}}/docs/administration/access-control/#marking-pipelines-with-policy-attributes) for this pipeline | +| `description` | `metadata` | string | Human readable description of the pipeline | +| `isPublic ` | `metadata` | boolean | If true the pipeline logs [will be public]({{site.baseurl}}/docs/configure-ci-cd-pipeline/build-status/) even for non-authenticated users | +| `template ` | `metadata` | boolean | If true, this pipeline will be listed as a template when creating a new pipeline | + +Example of metadata: + +`Pipeline Spec` +{% highlight yaml %} +{% raw %} +version: '1.0' +kind: pipeline +metadata: + name: project_name/pipeline_name + project: project_name + labels: + tags: + - tag1 + - tag2 + description: pipeline description here + isPublic: false + template: + isTemplate: false +{% endraw %} +{% endhighlight %} + +### Spec fields + +{: .table .table-bordered .table-hover} +| Field name | Parent field | Type | Value | +| -------------- | ---------------------------- |-------------------------| -------------------------| +| `steps` | `spec` | object | The [pipeline steps]({{site.baseurl}}/docs/codefresh-yaml/steps/) to be executed | +| `stages` | `spec` | array | The [pipeline stages]({{site.baseurl}}/docs/codefresh-yaml/stages/) for a better visual overview | +| `variables` | `spec` | array | List of variables defined in the pipeline itself | +| `contexts` | `spec` | array | Variable sets imported from [shared configuration]({{site.baseurl}}/docs/configure-ci-cd-pipeline/shared-configuration/) | +| `runtimeEnvironment` | `spec` | array | where to execute this pipeline | +| `terminationPolicy ` | `spec` | array | Termination settings of this pipeline | +| `concurrency ` | `spec` | number | How many instances of this pipeline [can run at the same time]({{site.baseurl}}/docs/configure-ci-cd-pipeline/pipelines/#policies) | +| `triggerConcurrency ` | `spec` | number | How many instances of this pipeline can run at the same time per trigger | +| `branchConcurrency ` | `spec` | number | How many instances of this pipeline can run at the same time per branch | +| `externalResources ` | `spec` | array | Optional external files available to this pipeline | +| `triggers` | `spec` | array | a list of [Git triggers]({{site.baseurl}}/docs/configure-ci-cd-pipeline/triggers/git-triggers/) that affect this pipeline | +| `options` | `spec` | object | Extra options for the pipeline | +| `enableNotifications` | `options` | boolean | if false the pipeline will not send notifications to [Slack]({{site.baseurl}}/docs/integrations/notifications/slack-integration/) and status updates back to the Git provider | + +### Pipeline variables + +The `variables` array has entries with the following fields: + +{: .table .table-bordered .table-hover} +| Field name | Parent field | Type | Value | +| -------------- | ---------------------------- |-------------------------| -------------------------| +| `key` | `variables` | string | Name of the variable | +| `value` | `variables` | string | Raw value | +| `encrypted` | `variables` | boolean | if true the value is stored encrypted | + +Example of variables: + +`Pipeline Spec` +{% highlight yaml %} +{% raw %} + variables: + - key: my-key + value: my-value + encrypted: false + - key: my-second-variable + value: '*****' + encrypted: true +{% endraw %} +{% endhighlight %} + +Encrypted variables cannot be read back by exporting the pipeline. + +### Runtime environment + +The `runtimeEnvironment` selects the cluster that will execute the pipeline (mostly useful for organizations using the [Codefresh Runner]({{site.baseurl}}/docs/installation/codefresh-runner/)) + +{: .table .table-bordered .table-hover} +| Field name | Parent field | Type | Value | +| -------------- | ---------------------------- |-------------------------| -------------------------| +| `name` | `runtimeEnvironment` | string | Name the environment as connected by the runner | +| `cpu` | `runtimeEnvironment` | string | CPU share using Kubernetes notation | +| `memory` | `runtimeEnvironment` | string | memory share using Kubernetes notation | +| `dindStorage` | `runtimeEnvironment` | string | storage size using Kubernetes notation | + + +Example of metadata: + +`Pipeline Spec` +{% highlight yaml %} +{% raw %} +runtimeEnvironment: + name: my-aws-runner/cf + cpu: 2000m + memory: 800Mi + dindStorage: nullGi +{% endraw %} +{% endhighlight %} + + + +### External resources + +The `externalResources` field is an array of objects that hold [external resource information]({{site.baseurl}}/docs/pipelines/pipelines/#external-resources). + +{: .table .table-bordered .table-hover} +| Field name | Parent field | Type | Value | +| -------------- | ---------------------------- |-------------------------| -------------------------| +| `type` | `externalResources` | string | Only `git` is supported | +| `source` | `externalResources` | string | Source folder or file path in Git repo | +| `context` | `externalResources` | string | Name of Git provider to be used | +| `destination` | `externalResources` | string | Target folder or file path to be copied to | +| `isFolder` | `externalResources` | boolean | if true path is a folder, else it is a single file | +| `repo` | `externalResources` | string | git repository name for the trigger. should be in format of `git_repo_owner/git_repo_name` | +| `revision` | `externalResources` | string | branch name or git hash to checkout | + + +`Pipeline Spec` +{% highlight yaml %} +{% raw %} +externalResources: + - type: git + source: /src/sample/venonalog.json + context: my-github-integration + destination: codefresh/volume/helm-sample-app/ + isFolder: false + repo: codefresh-contrib/helm-sample-app + revision: master +{% endraw %} +{% endhighlight %} + +### Git triggers + +The `triggers` field is an array of objects that hold [Git trigger information]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/) with the following fields. + +{: .table .table-bordered .table-hover} +| Field name | Parent field | Type | Value | +| -------------- | ---------------------------- |-------------------------| -------------------------| +| `name` | `triggers` | string | user defined trigger name | +| `type` | `triggers` | string | Always `git` | +| `repo` | `triggers` | string | git repository name for the trigger. should be in format of `git_repo_owner/git_repo_name` | +| `events` | `triggers` | array | All possible values are documented later. The possible values depend on Git provider | +| `pullRequestAllowForkEvents` | `triggers` | boolean | If this trigger is also applicable to Git forks | +| `commentRegex` | `triggers` | string | Only activate trigger if regex expression matches PR comment | +| `branchRegex ` | `triggers` | string | Only activate trigger if regex expression/string matches branch | +| `branchRegexInput ` | `triggers` | string | Defines what type of content is in `branchRegex`. Possible values are `regex`, `multiselect`, `multiselect-exclude` | +| `provider ` | `triggers` | string | Name of provider as found in Git integrations | +| `modifiedFilesGlob ` | `triggers` | string | Only activate trigger if changed files match glob expression | +| `disabled ` | `triggers` | boolean | if true, trigger will never be activated | +| `options ` | `triggers` | array | Choosing [caching behavior]({{site.baseurl}}/docs/configure-ci-cd-pipeline/pipeline-caching/) of this pipeline | +| `noCache ` | `options` | boolean | if true, docker layer cache is disabled | +| `noCfCache ` | `options` | boolean | if true, extra Codefresh caching is disabled | +| `resetVolume ` | `options` | boolean | if true, all files on volume will be deleted before each execution | +| `context ` | `triggers` | string | Name of git context to use | +| `contexts` | `spec` | array | Variable sets imported from [shared configuration]({{site.baseurl}}/docs/configure-ci-cd-pipeline/shared-configuration/) | +| `variables` | `triggers` | array | Override variables that were defined in the pipeline level | +| `runtimeEnvironment` | `triggers` | array | Override the runtime environment that was defined in the pipeline level | + +The possible values for the `events` array are the following (only those supported by the Git provider can be actually used) + + * `push.heads` - for push commit + * `push.tags` - for push tag event + * `pullrequest` - any pull request event + * `pullrequest.opened` - pull request opened + * `pullrequest.closed` - pull request closed + * `pullrequest.merged` - pull request merged + * `pullrequest.unmerged-closed` - pull request closed (not merged) + * `pullrequest.reopened` - pull request reopened + * `pullrequest.edited` - pull request edited + * `pullrequest.assigned` - pull request assigned + * `pullrequest.unassigned` - pull request unassigned + * `pullrequest.reviewRequested` - pull request review requested + * `pullrequest.reviewRequestRemoved` - pull request review request removed + * `pullrequest.labeled` - pull request labeled + * `pullrequest.unlabeled` - pull request unlabeled + * `pullrequest.synchronize` - pull request synchronized + * `pullrequest.commentAdded` - pull request comment added + * `release` - Git release event + +The `variables` and `runtimeEnvironment` fields have exactly the same format as in the parent pipeline fields but values defined in the trigger will take higher priority. + +Full example: + +`Pipeline Spec` +{% highlight yaml %} +{% raw %} +triggers: + - name: guysalton-codefresh/helm-sample-app + type: git + repo: guysalton-codefresh/helm-sample-app + events: + - push.heads + - pullrequest.commentAdded + pullRequestAllowForkEvents: true + commentRegex: /.*/gi + branchRegex: /^((master|develop)$).*/gi + branchRegexInput: regex + modifiedFilesGlob: /project1/** + provider: github + disabled: false + options: + noCache: false + noCfCache: false + resetVolume: false + verified: true + context: guyGithub + contexts: + - artifactory + variables: + - key: key + value: '*****' + encrypted: true + runtimeEnvironment: + name: docker-desktop/cf + cpu: 400m + memory: 800Mi + dindStorage: nullGi +{% endraw %} +{% endhighlight %} + + +## Using Codefresh from within Codefresh + +The Codefresh CLI is also packaged as a [Docker image on its own](https://hub.docker.com/r/codefresh/cli/){:target="\_blank"}. This makes it +very easy to use it from within Codefresh in a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). + +For example, you can easily call pipeline B from pipeline A +with the following step: + +`codefresh.yml` of pipeline A +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + myTriggerStep: + title: triggering another pipeline + image: codefresh/cli + commands: + - 'codefresh run -b=${{CF_BRANCH}}' -t + when: + condition: + all: + validateTargetBranch: '"${{CF_PULL_REQUEST_TARGET}}" == "production"' + validatePRAction: '''${{CF_PULL_REQUEST_ACTION}}'' == ''opened''' +{% endraw %} +{% endhighlight %} + +This step only calls pipeline B when a pull request is opened for the branch named `production`. + +Note that when you use the Codefresh CLI in a pipeline step, it is already configured, authenticated, and ready for use. +No additional authentication is required. + +## Related articles +[Codefresh API documentation](https://g.codefresh.io/api/){:target="\_blank"} +[Codefresh CLI documentation](https://codefresh-io.github.io/cli/){:target="\_blank"} + + diff --git a/_docs/integrations/codefresh-hosted-gitops.md b/_docs/integrations/codefresh-hosted-gitops.md new file mode 100644 index 000000000..14cbf9e2e --- /dev/null +++ b/_docs/integrations/codefresh-hosted-gitops.md @@ -0,0 +1,96 @@ +--- +title: "Hosted GitOps integration" +description: "Connect with our Hosted GitOps to leverage Managed Argo CD" +group: integrations +toc: true +--- + +Integrate Codefresh Classic with Codefresh's Hosted GitOps for deployments powered by managed Argo CD. +Use Codefresh Classic for pipelines, and Codefresh Hosted GitOps for deployments. + +Codefresh Hosted GitOps includes a dedicated report image step that both reports and enriches deployed images. Add the report image step in your Codefresh Classic pipeline and reference integrations with issue-tracking and container registry tools for Codefresh to retrieve and enrich image information. + +For a brief overview of what you get with Codefresh Hosted GitOps, read the next section. + +For information on how to connect Codefresh Classic to Codefresh Hosted GitOps, see [CI integration with Codefresh Classic](https://codefresh.io/docs/gitops-integrations/ci-integrations/codefresh-classic/){:target="\_blank"}. + +## Codefresh Hosted GitOps features + +### Hosted and hybrid runtimes +Codefresh Hosted GitOps is based on a hosted environemt, with the runtime hosted and managed by Codefresh. + +After the three-step process of provisioning your hosted runtime, Codefresh handles administration and maintenance of the hosted runtime, including version and security updates. + +{% include +image.html +lightbox="true" +file="/images/integrations/codefresh-hosted-gitops/hosted-runtime.png" +url="/images/integrations/codefresh-hosted-gitops/hosted-runtime.png" +caption="Provisioning a Hosted GitOps runtime" +alt="Provisioning a Hosted GitOps runtime" +max-width="70%" +%} + +### Dashboards for visibility and traceability + +A set of dashboards provides visibility into all aspects of deployment: + +* The Home dashboard presents enterprise-wide deployment highlights across runtimes and clusters. + Get insights into important KPIs and deployments, all in the same location. View status of runtimes and managed clusters, deployments, failed deployments with rollbacks, most active applications. Use filters to narrow the scope to focus on anything specific. + + {% include +image.html +lightbox="true" +file="/images/integrations/codefresh-hosted-gitops/hosted-home-dashboard.png" +url="/images/integrations/codefresh-hosted-gitops/hosted-home-dashboard.png" +caption="Home dashboard in Hosted GitOps" +alt="Home dashboard in Hosted GitOps" +max-width="70%" +%} + +* The Applications dashboard displays applications, also across runtimes and clusters, from which you can select individual applications for further analysis. + Individual application information is grouped by current and historical deployments, enriched with Argo, Jira, and Git details, including rollout visualizations for ongoing deployments, and an interactive tree view of application resources. + +{% include +image.html +lightbox="true" +file="/images/integrations/codefresh-hosted-gitops/hosted-app-dashboard.png" +url="/images/integrations/codefresh-hosted-gitops/hosted-app-dashboard.png" +caption="Applications dashboard in Hosted GitOps" +alt="Applications dashboard in Hosted GitOps" +max-width="70%" +%} + + +* The DORA metrics dashboard in Codefresh helps quantify DevOps performance. Apart from the metrics themselves, the DORA dashboard in Codefresh has several unique features to pinpoint just which applications or runtimes are contributing to problematic metrics. + +{% include +image.html +lightbox="true" +file="/images/integrations/codefresh-hosted-gitops/hosted-dora-metrics.png" +url="/images/integrations/codefresh-hosted-gitops/hosted-dora-metrics.png" +caption="DORA metrics in Hosted GitOps" +alt="DORA metrics in Hosted GitOps" +max-width="60%" +%} + +### Application management + +Manage the application lifecycle in the Codefresh UI, from creating, editing, and deleting applications, to quick manual sync when needed. + + +### Third-party integrations for image enrichment +Add integrations to issue-tracking tools such as Jira, and container-registries such as Docker Hub, JFrog and more, to enrich images. + +{% include +image.html +lightbox="true" +file="/images/integrations/codefresh-hosted-gitops/hosted-int-tools.png" +url="/images/integrations/codefresh-hosted-gitops/hosted-int-tools.png" +caption="Integrations in Hosted GitOps" +alt="Integrations in Hosted GitOps" +max-width="60%" +%} + +## Related articles +[CI integrations with GitOps]({{site.baseurl}}/docs/gitops-integrations/ci-integrations) diff --git a/_docs/integrations/datadog.md b/_docs/integrations/datadog.md new file mode 100644 index 000000000..609752af2 --- /dev/null +++ b/_docs/integrations/datadog.md @@ -0,0 +1,133 @@ +--- +title: "Datadog integration" +description: "Integrate Codefresh pipelines with Datadog for monitoring and analysis" +group: integrations +toc: true +--- + +Datadog is a SaaS-based monitoring and analytics platform for large-scale applications and infrastructure. Integrating Datadog with Codefresh allows you to leverage Codefresh to create your pipelines, and Datadog to monitor and analyze them. + +When a pipeline completes execution in Codefresh, Codefresh reports pipeline-execution data to Datadog for viewing in Datadog's Continuous Integration (CI) Visibility interface. + +For Datadog and Codefresh integration, you need: +* An API token from your Datadog account +* To define the settings in Codefresh + +> Note: Please reach out to Support if you’re interested in enabling Datadog for your account. + +## Get API token from Datadog account +If you already have a Datadog account, you can copy the API key if you have one, or generate a new API key. + +1. Log in to your Datadog account. +1. Go to **Organization Settings**, and select **API Keys**. + + {% include image.html +lightbox="true" +file="/images/integrations/datadog/datadog-api-key.png" +url="/images/integrations/datadog/datadog-api-key.png" +max-width="30%" +caption="Getting an API Key from your Datadog account" +alt="Getting an API Key from your Datadog account" +%} + +{:start="3"} +1. Copy the API key to use with your Codefresh integration. + +## Set up Datadog integration in Codefresh + +Configure the integration settings for Datadog within Codefresh. + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Datadog**, and then **Configure**. +1. Click **Add Integration**. + + {% include image.html +lightbox="true" +file="/images/integrations/datadog/datadog-config-settings.png" +url="/images/integrations/datadog/datadog-config-settings.png" +max-width="30%" +caption="Datadog configuration settings" +alt="Datadog configuration settings" +%} + +{:start="4"} +1. Define the following: + * **Datadog site**: Select the site with your data. If you are not sure which Datadog site to select, select the _View documentation_ link below the field, and read Datadog's official documentation. + * **Token**: The API token you copied from your Datadog account. +1. To verify the connection details, click **Test Connection**. +1. To apply the changes, click **Save**. + + +## Pipeline data from Codefresh in Datadog +See pipeline data in Datadog's CI Visibility interface. + +We have highlighted the main features in Datadog for Codefresh pipelines. For detailed descriptions and options, see [Datadog documentation on Exploring Pipelines](https://docs.datadoghq.com/continuous_integration/explore_pipelines/){:target="\_blank"}. + + +### Pipelines page in Datadog + +The Pipelines page shows aggregated data for each pipeline, for the selected time range. You can see the failure rate and average build duration against the total number of executions of a pipeline, alongside the metrics from the most recent build of the same pipeline. + +Below is an example of the Pipelines page with Codefresh pipelines, prefixed by the Codefresh logo. + + {% include image.html +lightbox="true" +file="/images/integrations/datadog/datadog-pipelines-page.png" +url="/images/integrations/datadog/datadog-pipelines-page.png" +max-width="30%" +caption="Pipelines page in Datadog with Codefresh pipelines" +alt="Pipelines page in Datadog with Codefresh pipelines" +%} + +### Pipeline Details page in Datadog + +Selecting a pipeline takes you to the Pipeline Details page in Datadog which provides in-depth data for the pipeline. +Here you can see the failure rate and average build duration for the selected pipeline, and data on its branches, and jobs (referred to as steps in Codefresh). +You also have the option of viewing executions in the dedicated Pipeline Executions page. + +Below is an example of the Pipeline Details page for the selected Codefresh pipeline. + + + {% include image.html +lightbox="true" +file="/images/integrations/datadog/datadog-pipeline-drilldown.png" +url="/images/integrations/datadog/datadog-pipeline-drilldown.png" +max-width="30%" +caption="Drilldown view for selected pipeline in Datadog Pipeline Details page" +alt="Drilldown view for selected pipeline in Datadog Pipeline Details page" +%} + +### Pipeline Executions page in Datadog + +The Pipeline Executions page shows day-by-day execution data for the selected pipeline or pipelines. + +Below is an example of the Pipeline Executions page with execution data for Codefresh pipelines. + + + {% include image.html +lightbox="true" +file="/images/integrations/datadog/datadog-pipeline-executions.png" +url="/images/integrations/datadog/datadog-pipeline-executions.png" +max-width="30%" +caption="Execution timeline view in Datadog Pipeline Executions page" +alt="Execution timeline view in Datadog Pipeline Executions page" +%} + +### Pipeline Dashboards page in Datadog + +Pipeline Dashboards is your go-to location for a quick look at performance and step metrics across pipelines. You can customize the widgets in the dashboard to display the data that is of interest to you. + +Below is an example of the Pipeline Dashboards page. + + + {% include image.html +lightbox="true" +file="/images/integrations/datadog/datadog-pipeline-dashboard.png" +url="/images/integrations/datadog/datadog-pipeline-dashboard.png" +max-width="30%" +caption="Pipelines Dashboards page in Datadog" +alt="Pipelines Dashboards in Datadog" +%} + +## Related articles +[Integration Tests]({{site.baseurl}}/docs/testing/integration-tests/) \ No newline at end of file diff --git a/_docs/integrations/docker-registries.md b/_docs/integrations/docker-registries.md new file mode 100644 index 000000000..271201060 --- /dev/null +++ b/_docs/integrations/docker-registries.md @@ -0,0 +1,141 @@ +--- +title: "Docker Registries for pipeline integrations" +description: "Connect your Docker Registry to Codefresh CI pipelines" +group: integrations +redirect_from: + - /docs/docker-registry/ + - /docs/docker-registries/external-docker-registries/ + - /docs/docker-registries/ + - /docs/codefresh-registry/ + - /docs/docker-registries/codefresh-registry/ +toc: true +--- +Codefresh enables you to integrate with several Docker container registries, including (but not limited to): + +* [Docker Hub](docker-hub) +* [Azure Container Registry](azure-docker-registry) +* [Google Container Registry](google-container-registry) +* [Google Artifact Registry](google-artifact-registry) +* [Amazon EC2 Container Registry](amazon-ec2-container-registry) +* [Bintray.io/Artifactory](bintray-io) +* [Quay.io](quay-io) +* [Github Container Registry](github-container-registry) + +For a different registry choose to configure using the [Other](other-registries) option. + +The registries can either be public or private. + +## General Configuration + + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Docker Registries** and then click **Configure**. + +{% include image.html + lightbox="true" + file="/images/integrations/codefresh-integrations.png" + url="/images/integrations/codefresh-integrations.png" + alt="Codefresh Account Integration" + max-width="80%" %} + +{:start="4"} +1. From the **Add Registry Provider** drop-down, select the regsitry type to add. + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/add-docker-registry.png" + url="/images/integrations/docker-registries/add-docker-registry.png" + alt="Add Docker Registry" + max-width="45%" %} + +{:start="5"} +1. Each configuration must be given a unique name, which you can later reference in a codefresh.yml file. + + + +## Define fallback registry + +Codefresh has a feature that allows users to designate a fallback registry for Docker integrations. If a Codefresh pipeline attempts to pull an image and that image fails for any reason (authorization issue, the registry server is down, etc.), a retry mechanism will attempt to pull it successfully. If this mechanism fails, the fallback registry feature provides the opportunity to pull the image from a different registry you have specified. + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Docker Registries** and then click **Configure**. +1. In the list of registries, select the registry to configure as the fallback registry, and click **Edit**. +1. Expand **Advanced Options**, and select the registry from the **Fallback Registry** list. + You can also specify a fallback registry when creating a new integration as long as another integration exists. + +## Using an optional repository prefix + +For each supported Registry, define a prefix string for your Docker images to be used globally. + +This is handy for registries that require a prefix (usually the name of an organization or repository) as you can set it once, instead of having each pipeline using the prefix by itself. + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/repository-prefix.png" + url="/images/integrations/docker-registries/repository-prefix.png" + alt="Setting a Registry prefix" + caption="Setting a Registry prefix" + max-width="60%" + %} + +See more details at [pushing Docker images]({{site.baseurl}}/docs/#pushing-docker-images). + +## Pushing an image + +Once your registry configuration is all set up you can start pushing your images to it. + +Within a [push step]({{site.baseurl}}/docs/pipelines/steps/push/), add your registry configuration name in the `registry` field + + `codefresh.yml` +{% highlight yaml %} +push_step: + type: push + description: Free text description + candidate: {% raw %}${{build_step}}{% endraw %} + tag: {% raw %}${{CF_BRANCH}}{% endraw %} + registry: your-registry-configuration-name +{% endhighlight %} + +For more details, see the [example for image push]({{site.baseurl}}/docs/example-catalog/ci-examples/build-and-push-an-image/). + +## Internal caching registry + +You can also select a single registry that will serve as your [caching registry]({{site.baseurl}}/docs/pipelines/pipeline-caching/#docker-registry-caching). + +> You cannot select Dockerhub as a caching registry, because it has very strict requirements for naming images, and our caching mechanism needs capabilities which are not possible with Dockerhub. + +Codefresh uses that registry efficiently to perform advanced caching logic for your builds by automatically: + +* Checking the stored metadata to decide which past image is most relevant for caching purposes +* Pulling images from this registry for caching purposes +* Using that registry for distributed Docker layer caching to make your Docker builds faster + +We give you the ability to define a separate registry for caching purposes for the following scenarios: + +1. You don't want extra traffic to be sent to your main deployment registry. Maybe you want to avoid bandwidth/storage limits in your production registry +1. You have lots of build steps in pipelines with intermediate docker images that you are certain you don't need outside of the pipeline itself. In that case you can use the `disable_push` property in those pipelines. +1. You have speed concerns regarding image pulling/pushing. For example your development team is in Europe, but your production servers are in the USA. You would probably choose a caching registry in a European region (so that developers get the best experience), where your main registry is in the USA (close to your production servers) + +Therefore, in most cases you should make your main registry your caching registry as well. For extra control, you can either define a different caching registry or disable selectively automatic pushes with the `disable_push` property. + +>Notice that the dynamic image feature of Codefresh (creating docker images on demand in the same pipeline that is using them) will always work regardless of a caching registry. + +## Default registry + +If you define more than one registry, you can select a registry as the default one. Codefresh uses the default registry in both [build]({{site.baseurl}}/docs/pipelines/steps/build/) and [push]({{site.baseurl}}/docs/pipelines/steps/push/) steps if they don't already include a `registry` property. + +> Successful build steps always push to the default Codefresh registry, unless you also define the `disable_push` property. + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Docker Registries** and then click **Configure**. +1. From the context menu of the Docker registry integration to be used as the default registry, select **Set as default**. + + + +## Related articles +[Examples of pushing Docker images]({{site.baseurl}}/docs/example-catalog/ci-examples/build-and-push-an-image/) diff --git a/_docs/integrations/docker-registries/amazon-ec2-container-registry.md b/_docs/integrations/docker-registries/amazon-ec2-container-registry.md new file mode 100644 index 000000000..8e534868a --- /dev/null +++ b/_docs/integrations/docker-registries/amazon-ec2-container-registry.md @@ -0,0 +1,179 @@ +--- +title: "Amazon EC2 Container Registry" +description: "Use the Amazon Docker Registry for pipeline integrations" +group: integrations +sub_group: docker-registries +redirect_from: + - /docs/aws/ + - /docs/docker-registries/external-docker-registries/amazon-ec2-container-registry/ +toc: true +--- + +## Set up ECR integration for IAM user + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Docker Registries** and then click **Configure**. +1. From the **Add Registry Provider** dropdown, select **Amazon ECR**. +1. Define the following: + * **Registry name**: A unique name for this configuration. + * **Region**: AWS region. + * **Access Key ID**: Your AWS accessKeyId. + * **Secret Access Key**: Your AWS accessKeyId. + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/add-amazon-ecr-registry.png" + url="/images/integrations/docker-registries/add-amazon-ecr-registry.png" + alt="Amazon EC2 Container Registry settings" + caption="Amazon EC2 Container Registry settings" + max-width="60%" %} + +{:start="5"} +1. To verify the connection details, click **Test Connection**. +1. To apply the changes, click **Save**. + +Codefresh makes sure to automatically refresh the AWS token for you. + +For more information on how to obtain the needed tokens, read the [AWS documentation](http://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys){:target="_blank"}. + +> Note: + You must have an active registry set up in AWS.
+ Amazon ECR push/pull operations are supported with two permission options: user-based and resource-based. + + + * User-based permissions: User account must apply `AmazonEC2ContainerRegistryPowerUser` policy (or custom based on that policy). + For more information and examples, click [here](http://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr_managed_policies.html){:target="_blank"}. + * Resource-based permissions: Users with resource-based permissions must be allowed to call `ecr:GetAuthorizationToken` before they can authenticate to a registry, and push or pull any images from any Amazon ECR repository, than you need provide push/pull permissions to specific registry. + For more information and examples, click [here](http://docs.aws.amazon.com/AmazonECR/latest/userguide/RepositoryPolicies.html){:target="_blank"}. + + +## Set up ECR integration for service account + +Setting up ECR integration for a service account applies to accounts with the Codefresh Runner installation. + +### Kubernetes service account setup +To use an IAM role, you must set up a Kubernetes service account, as described in the [AWS Documentation](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html){:target="\_blank"}. +You can define the service account at four different levels, based on the required priority. The levels are listed below in ascending order of priority: + +* Runtime + The runtime level has the lowest priority. Define it in the Runtime Specification under `runtimeScheduler > Cluster` (same level as `namespace`), and specify the service account. The key is `serviceAccount`. Use the default, and make sure you have the correct annotation added to the service account. Another option is to create a new service account with the proper permissions and annotations. + +```yaml +runtimeScheduler: + cluster: + namespace: codefresh + clusterProvider: + accountId: 5c1658d1736122ee1114c842 + selector: docker-desktop + serviceAccount: codefresh-engine + +``` + +* Account + The Account-level service account has higher priority than the runtime-level service account. To define the service account at the account level, turn on the setting as part of the integration as described below. + +* Pipeline + The Pipeline-level service account has higher priority than the account-level service account. Define the service account as part of the pipeline's runtime settings (Pipeline > Settings > Runtime). + +* Trigger + The Trigger-level service account has the highest priority. Define the service account as part of the trigger settings for the specific pipeline (Workflow > Triggers (modify or add) > Advanced Options). + + +### How to + +**Before you begin** +* Define a Kubernetes service account for the runtime, account, pipeline, or pipeline-trigger + +**How to** + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Docker Registries** and then click **Configure**. +1. From the **Add Registry Provider** dropdown, select **Amazon ECR**. +1. Do the following: + * **Registry name**: Enter a unique name for this configuration. + * **Region**: Select the AWS region. + * Select **Resolve credentials from servce account**. + + The Access Key ID and Secret Access Key fields are disabled. + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/add-amazon-ecr-registry.png" + url="/images/integrations/docker-registries/add-amazon-ecr-registry.png" + alt="Amazon EC2 Container Registry settings" + caption="Amazon EC2 Container Registry settings" + max-width="60%" %} + +{:start="5"} +1. To verify the connection details, click **Test Connection**. +1. To apply the changes, click **Save**. + + + +## Pushing Docker images to Amazon ECR + +There are two ways to push images: + +1. (Recommended) Using the YAML [push step]({{site.baseurl}}/docs/pipelines/steps/push/). +1. Manually promoting manually an image (described below) + +For more details on how to push a Docker image in a pipeline see the [build and push example]({{site.baseurl}}/docs/yaml-examples/examples/build-and-push-an-image/). + + + +### Manually promoting an image + + + +The **Images** view has an option to manually push images to a registry. +You need to specify the repository name as the name of your repository as set in ECR, as in the example below. + +{% include image.html +lightbox="true" +file="/images/integrations/docker-registries/ecr/ecr-manual-promote-repo-name.png" +url="/images/integrations/docker-registries/ecr/ecr-manual-promote-repo-name.png" +alt="Repository name in ECR" +caption ="Repository name in ECR" +max-width="40%" +%} + +1. In the Codefresh UI, from Artifacts in the sidebar, select [**Images**](https://g.codefresh.io/2.0/images){:target="\_blank"}. +1. Click **Promote**. + + {% include image.html +lightbox="true" +file="/images/integrations/docker-registries/ecr/ecr-manual-promote-button.png" +url="/images/integrations/docker-registries/ecr/ecr-manual-promote-button.png" +alt="Promote image icon" +caption="Promote image icon" +max-width="40%" +%} + +{:start="3"} +1. Do the following: + * Enter the **Repository Name**. + * Enter the **Tag**. Copy and paste the text after the `:` in the Repository Name. For example, `repository-name:tag`. + * From the **Registry** dropdown, select your ECR configuration. + +{% include image.html +lightbox="true" +file="/images/integrations/docker-registries/ecr/ecr-manual-promote-settings.png" +url="/images/f2b0ec5-ecr3.png" +alt="Image promotion settings" +caption="Image promotion settings" +max-width="40%" +%} + +{:start="3"} +3. Click **Promote**. + + +>It is possible to change the image name if you want, but make sure that the new name exists as a repository in ECR. + + +## Related articles +[Docker registries for pipeline integrations]({{site.baseurl}}/docs/integrations/docker-registries) +[Working with Docker Registries]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/) +[Push step]({{site.baseurl}}/docs/pipelines/steps/push/) +[Building and pushing an image]({{site.baseurl}}/docs/example-catalog/ci-examples/build-and-push-an-image/) + diff --git a/_docs/integrations/docker-registries/azure-docker-registry.md b/_docs/integrations/docker-registries/azure-docker-registry.md new file mode 100644 index 000000000..e19f41f81 --- /dev/null +++ b/_docs/integrations/docker-registries/azure-docker-registry.md @@ -0,0 +1,73 @@ +--- +title: "Azure Docker registry" +description: "Use the Azure Docker Registry for pipeline integrations" +group: integrations +sub_group: docker-registries +redirect_from: + - /docs/docker-registries/external-docker-registries/azure-docker-registry/ +toc: true +--- +Configure [Azure Docker registry](https://docs.microsoft.com/en-us/azure/container-registry/){:target=\_blank"} for pipeline integrations. + +## Configure Azure portal + +1. Log in to the Azure Portal. +1. Click **Settings** and from the sidebar, select **Access Keys**. + + {% include +image.html +lightbox="true" +file="/images/integrations/docker-registries/azure-registry-admin.png" +url="/images/integrations/docker-registries/azure-registry-admin.png" +alt="Docker credentials for the Azure registry" +caption="Docker credentials for the Azure registry" +max-width="80%" +%} + +1. For **Admin user**, click **Enable**. +1. Change the username (optional), and make sure that you note down one of the passwords shown on the screen. + +## Configure Azure Docker registry settings in Codefresh + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Docker Registries** and then click **Configure**. +1. From the **Add Registry Provider** dropdown, select **Other Registries**. +1. Define the following: + * **Registry Name**: Unique name for this configuration. + * **Username**: Your Azure Registry username. + * **Password**: Your Azure Registry password. + * **Domain**: `.azurecr.io`. + +{% include image.html + lightbox="true" +file="/images/integrations/docker-registries/add-azure-registry.png" +url="/images/integrations/docker-registries/add-azure-registry.png" +alt="Adding the Azure Docker registry" +caption="Adding the Azure Docker registry" +max-width="60%" %} + +{:start="5} +1. To verify the connection details, click **Test connection**. +1. To apply the changes, click **Save**. + +## Using the Azure Registry + +You can now use the Azure Registry in your CI pipelines, either via the UI or through the YAML [push step]({{site.baseurl}}/docs/pipelines/steps/push/) (recommended). + +It is also possible to use the registry from the command line with: + +``` +docker login .azurecr.io -u -p +``` + +You can also inspect the pushed images either using Azure portal or with [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/?view=azure-cli-latest){:target="\_blank"} + +``` +az acr repository list --name --output table +``` + + +## Related articles +[Docker registries for pipeline integrations]({{site.baseurl}}/docs/integrations/docker-registries) +[Working with Docker Registries]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/) +[Building and pushing an image]({{site.baseurl}}/docs/example-catalog/ci-examples/build-and-push-an-image/) \ No newline at end of file diff --git a/_docs/integrations/docker-registries/bintray-io.md b/_docs/integrations/docker-registries/bintray-io.md new file mode 100644 index 000000000..7b158628e --- /dev/null +++ b/_docs/integrations/docker-registries/bintray-io.md @@ -0,0 +1,91 @@ +--- +title: "Bintray.io/Artifactory" +description: "Use JFrog Bintray/Artifactory with pipeline integrations " +group: integrations +sub_group: docker-registries +redirect_from: + - /docs/bitrayio/ + - /docs/docker-registries/external-docker-registries/bintray-io/ +toc: true +--- + +Configure JFrog Bintray/Artifactory as your Docker registry provider. +You need to get the API key for your profile, and the correct registry domain. + +>Passing Codefresh metadata to Bintray is supported through Grafeas. More info is available [in this blogpost](https://codefresh.io/blog/write-this-down-grafeas/){:target="_blank"}. + +## Set up Bintray integration + +**Before you begin** +* [Get your API key](#find-your-api-key) +* [Get your regsitry domain](#find-your-registry-domain) + +**How to** + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Docker Registries** and then click **Configure**. +1. From the **Add Registry Provider** dropdown, select **JFrog Bintray**. +1. Define the following: + * **Registry name**: A unique name for this configuration. + * **Username**: Your Bintray.io/Artifactory username. + * **API key**: The Bintray.io/Artifactory API key you retrieved from your profile. + * **Domain**: Your Bintray.io registry address, for example, `docker-new-repository.bintray.io`, or Artifactory registry address, for example `my-company-docker-snapshot.jfrog.io`. + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/add-bintray-registry.png" + url="/images/integrations/docker-registries/add-bintray-registry.png" + alt="JFrog Bintray registry settings" + caption="JFrog Bintray registry settings" + max-width="70%" %} + +{:start="5"} +1. To verify the connection details, click **Test Connection**. +1. To apply the changes, click **Save**. + + +## Getting Bintray.io settings + +To obtain Bintray.io information, follow the steps. + +### Find your API key + +1. Go to your Bitray.io profile. +1. Select **API Key** from the side menu. + +{% include image.html +lightbox="true" +file="/images/integrations/docker-registries/bintray/bintray-api-key.png" +url="/images/integrations/docker-registries/bintray/bintray-api-key.png" +alt="Bintray.io API key" +caption="Bintray.io API key" +max-width="60%" %} + +### Find your registry domain + +1. Navigate to your bintray.com repository, or add a new one. +1. Click **SET ME UP!**. +{% include image.html lightbox="true" file="/images/integrations/docker-registries/bintray/bintray-set-me-up.png" url="/images/integrations/docker-registries/bintray/bintray-set-me-up.png" alt="Bintray.io SET ME UP" caption="Bintray.io SET ME UP" max-width="45%" %} + +{:start="3"} +1. Copy the registry address. +{% include image.html lightbox="true" file="/images/integrations/docker-registries/bintray/bintray-domain.png" url="/images/integrations/docker-registries/bintray/bintray-domain.png" alt="Bintray.io registry address" caption="Bintray.io registry address" max-width="45%" %} + +### Basic metadata upload + +Codefresh automatically sets some version attributes in Bintray every time you upload a Docker image. + +{% + include image.html lightbox="true" + file="/images/integrations/docker-registries/bintray/bintray-metadata.png" + url="/images/integrations/docker-registries/bintray/bintray-metadata.png" + alt="Basic Bintray metadata" + caption="Basic Bintray metadata" + max-width="50%" + %} + +## Related articles +[Docker registries for pipeline integrations]({{site.baseurl}}/docs/integrations/docker-registries) +[Working with Docker Registries]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/) +[Push step]({{site.baseurl}}/docs/pipelines/steps/push/) +[Building and pushing an image]({{site.baseurl}}/docs/yaml-examples/examples/build-and-push-an-image/) \ No newline at end of file diff --git a/_docs/integrations/docker-registries/digital-ocean-container-registry.md b/_docs/integrations/docker-registries/digital-ocean-container-registry.md new file mode 100644 index 000000000..b36283a48 --- /dev/null +++ b/_docs/integrations/docker-registries/digital-ocean-container-registry.md @@ -0,0 +1,148 @@ +--- +title: "DigitalOcean Container Registry" +description: "Push Docker images to your DigitalOcean Container Registry with pipeline integration" +group: integrations +sub_group: docker-registries +toc: true +--- + +You can configure [DigitalOcean Container Registry](https://www.digitalocean.com/products/container-registry/){:target="\_blank"} as your Docker Registry, and use it in your Codefresh pipeline. + + +The DigitalOcean Container Registry is directly integrated into your DigitalOcean Dashboard. While it is optional to use the DigitalOcean Container registry with your DigitalOcean Kubernetes cluster, it allows for easier integration between resources. + +The next sections will look at: +1. Creating the DigitalOcean Container Registry +2. Generating a DigitalOcean Access token +3. Adding the DigitalOcean Container Registry to our Docker Registry in Codefresh +4. Modifying the Build step in our Codefresh pipeline +5. Viewing the built image in the DigitalOcean Container Registry + +## Building and pushing a container image with DigitalOcean and Codefresh + +Building and pushing a container image with DigitalOcean and Codefresh, requires: +* A DigitalOcean account (your GitHub username) +* A DigitalOcean access token +* An application with a Dockerfile to build images + +### Creating the DigitalOcean Container Registry + +Once you are logged into your DigitalOcean Account, open the `Container Registry` tap and provide the name of your registry. Note that the name has to be unique. + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/digital-ocean/create-registry.png" + url="/images/integrations/docker-registries/digital-ocean/create-registry.png" + alt="Create Container Registry in DigitalOcean" + caption="Create Container Registry in DigitalOcean" + max-width="100%" +%} + +### Creating an access token + +Now that we are already in DigitalOcean, we have to create an [access token](https://www.digitalocean.com/docs/apis-clis/api/create-personal-access-token/){:target="\_blank"}. Note that it requires read and write access. +Select **API** at the bottom right of your left-side menu under **Tokens/Keys**. +Copy and paste the token somewhere secure and where you will find it again. + +## Set up DigitalOcean Container Registry integration + + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Docker Registries** and then click **Configure**. +1. From the **Add Registry Provider** dropdown, select **Other Registries**. +1. Define the following: + * **Registry name**: A unique name for this configuration. + * **Username**: The DigitalOcean access token you created. + * **Password**: The DigitalOcean access token you created. + * **Domain**: `registry.digitalocean.com`. + + Optional, you can add your registry name to the advanced settings section. For instance, if you named it in Digital Ocean "anais-codefresh", you can ensure that every time the registry is used, it is automatically referenced in the build step of your pipeline. + + + +{:start="5"} +1. To verify the connection details, click **Test Connection**. +1. To apply the changes, click **Save**. + + + +### Modify your build step + +Within your Codefresh YAML file, modify the build step to push to the DigitalOcean Container Registry. If you set the DigitalOcean Container Registry as default registry, note that you do not have to specify the Registry. + +Add the following line to your build step: +`registry: "digital-ocean"` + +This is an example of the complete build step: + +{% highlight yaml %} +{% raw %} +version: "1.0" +stages: + - "clone" + - "build" + +steps: + clone: + title: "Cloning repository" + type: "git-clone" + repo: "anais-codefresh/react-article-display" + # CF_BRANCH value is auto set when pipeline is triggered + # Learn more at codefresh.io/docs/docs/codefresh-yaml/variables/ + revision: "${{CF_BRANCH}}" + git: "github" + stage: "clone" + + build: + title: "Building Docker image" + type: "build" + image_name: "anais-codefresh/react-article-display-do-registry" + tags: + - "1.0.0" + working_directory: "${{clone}}" + dockerfile: "Dockerfile" + stage: "build" + registry: "digital-ocean" +{% endraw %} +{% endhighlight %} + +Note that Codefresh builds AND pushes images both in the same step. + +### Running the pipeline and viewing the image in the DigitalOcean Container Registry + +Once you have modified the step, save and run your pipeline. Below is an example of the pipeline in it's simplest form. + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/digital-ocean/codefresh-pipeline.png" + url="/images/integrations/docker-registries/digital-ocean/codefresh-pipeline.png" + alt="Codefresh Pipeline" + caption="Codefresh Pipeline" + max-width="100%" +%} + +You can then view the image in the DigitalOcean Container Registry: + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/digital-ocean/container-registry-do.png" + url="/images/integrations/docker-registries/digital-ocean/container-registry-do.png" + alt="DigitalOcean Container Registry" + caption="DigitalOcean Container Registry" + max-width="100%" +%} + +## Related articles +[Docker registries for pipeline integrations]({{site.baseurl}}/docs/integrations/docker-registries) +[Working with Docker Registries]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/) +[Building Docker images]({{site.baseurl}}/docs/ci-cd-guides/building-docker-images/) +[Push step]({{site.baseurl}}/docs/pipelines/steps/push/) +[Building and pushing an image]({{site.baseurl}}/docs/yaml-examples/examples/build-and-push-an-image/) + diff --git a/_docs/integrations/docker-registries/docker-hub.md b/_docs/integrations/docker-registries/docker-hub.md new file mode 100644 index 000000000..46a71832f --- /dev/null +++ b/_docs/integrations/docker-registries/docker-hub.md @@ -0,0 +1,123 @@ +--- +title: "Docker Hub" +description: "Use DockerHub for pipeline integration" +group: integrations +sub_group: docker-registries +redirect_from: + - /docs/dockerhub/ + - /docs/docker-registries/external-docker-registries/docker-hub/ +toc: true +--- +Configure Docker Hub as a Docker registry for CI pipelines to push images to it. + +1.[ Select **Docker Hub** as the registry provider]({{site.baseurl}}/docs/integrations/docker-registries/#general-configuration). +1. Define the following: + * Registry Name: A unique name for this configuration. + * Username: Docker Hub username. + * Password: Docker Hub [personal account token](https://docs.docker.com/docker-hub/access-tokens/){:target="\_blank"}, or Dockerhub account password (not recommended). + >If you have enabled [two-factor-authentication in Docker Hub](https://docs.docker.com/docker-hub/2fa/){:target="\_blank"}, then in the Password field, paste a Docker personal access token (instead of your Docker Hub master password). Otherwise, Codefresh will not be able to push your image. + If you don't have 2FA enabled in Dockerhub, then you can also use your Dockerhub account password. But in all cases we suggest you create a personal access token for Codefresh (personal access tokens are more secure as you can revoke them on demand and see when they were last used). + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/dockerhub/add-dockerhub-registry.png" + url="/images/integrations/docker-registries/dockerhub/add-dockerhub-registry.png" + alt="Add Docker Hub registry" + caption="Add Docker Hub registry" + max-width="50%" +%} + + +>Docker.io only allows you to push images that are tagged with your username. If you have a choice, create +a Docker Hub account with the same username that you have in Codefresh. If not, you need to change the Docker image +created to match your username in every [push step]({{site.baseurl}}/docs/pipelines/steps/push/#examples). + + +## Adding more Docker Hub integrations + +You can add additional Docker Hub accounts using the same process. + + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/dockerhub/two-dockerhub-integrations.png" + url="/images/integrations/docker-registries/dockerhub/two-dockerhub-integrations.png" + alt="Additional Docker Hub integrations" + caption="Additional Docker Hub integrations" + max-width="80%" +%} + + +You can specify which registry will be used as primary/default for the `docker.io` domain. +Use the appropriate `registry name` value in your pipelines in order to decide which Dockerhub account will be used. + +Here is a pipeline that pushes to two different Docker Hub accounts: + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/dockerhub/pushing-two-dockerhub-accounts.png" + url="/images/integrations/docker-registries/dockerhub/pushing-two-dockerhub-accounts.png" + alt="Pushing to multiple Dockerhub accounts" + caption="Pushing to multiple Dockerhub accounts" + max-width="90%" +%} + +This is the [definition]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) for the pipeline: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +stages: + - "clone" + - "build" + - "push" + +steps: + clone: + title: "Cloning repository" + type: "git-clone" + repo: "kostis-codefresh/trivial-go-web" + revision: "master" + stage: "clone" + build: + title: "Building Docker image" + type: "build" + image_name: "trivial-go-web" + working_directory: "${{clone}}" + tag: "latest" + dockerfile: "Dockerfile.multistage" + stage: "build" + disable_push: true + push1: + title: "Pushing 1st Docker image" + type: push + image_name: "kostiscodefresh/trivial-go-web" + tag: "latest" + stage: "push" + registry: dockerhub + candidate: ${{build}} + push2: + title: "Pushing 2nd Docker image" + type: push + image_name: "kkapelon/trivial-go-web" + tag: "latest" + stage: "push" + registry: second-dockerhub + candidate: ${{build}} +{% endraw %} +{% endhighlight %} + +The two Dockerhub accounts are `kkapelon` and `kostiscodefresh`, and Codefresh automatically uses the correct integration by looking at the `image_name` property of the push step. + + +## Related articles +[Docker registries for pipeline integrations]({{site.baseurl}}/docs/integrations/docker-registries) +[Working with Docker Registries]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/) +[Push step]({{site.baseurl}}/docs/pipelines/steps/push/) +[Building and pushing an image]({{site.baseurl}}/docs/yaml-examples/examples/build-and-push-an-image/) + + + + + diff --git a/_docs/integrations/docker-registries/github-container-registry.md b/_docs/integrations/docker-registries/github-container-registry.md new file mode 100644 index 000000000..7d2ff3dd4 --- /dev/null +++ b/_docs/integrations/docker-registries/github-container-registry.md @@ -0,0 +1,186 @@ +--- +title: "GitHub Container Registry" +description: "Push Docker images to GitHub Container Registry with pipeline integrations" +group: integrations +sub_group: docker-registries +redirect_from: + - /docs/integrations/docker-registries/github-packages/ +toc: true +--- + +Configure [GitHub Container Registry](https://docs.github.com/en/free-pro-team@latest/packages/getting-started-with-github-container-registry){:target="\_blank"} as your Docker Registry, and use it in your Codefresh pipeline. + + +The GitHub Container Registry allows you to host and manage Docker container images in your personal or organisation account on GitHub. One of the benefits is that permissions can be defined for the Docker image, independent of any repository. Thus, your repository could be private and your Docker image public. +See GitHub documentation for more [information on permissions](https://docs.github.com/en/free-pro-team@latest/packages/managing-container-images-with-github-container-registry/configuring-access-control-and-visibility-for-container-images){:target="\_blank"}. + +You can use the GitHub Container Registry manually or automate the process by connecting the registry to your Codefresh pipeline. + + +## Using the GitHub Container Registry + +You will need the following +* A GitHub account with your GitHub username +* A personal access token +* The Docker image you want to push or use in your Codefresh pipeline + +### Create a personal token + +The username to the registry is the same as your GitHub username. +For the password you need to [create a GitHub personal token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token){:target="\_blank"}. + +When creating the personal token, you need to select at least the following scopes: + +* `write:packages` +* `read:packages` +* `delete:packages` +* `repo` if your repository is private; if public, do not select + +Once you create the token, note it down. + +You can make sure that your token is valid by using it as a password on your local workstation with the Docker command: + +``` +docker login ghcr.io --username github-account +[Paste your GitHub token on this prompt] +``` + +**Important** Make sure that the URL is correct, otherwise, you will receive login errors later on. The github-account is your GitHub username. + +### Tag and push your Docker image + +After you are logged in, you can now tag and push your Docker image to the GitHub Container Registry. The first method is the manual setup with the Docker CLI, and the second one uses Codefresh to automate the process. + +Use the following command to tag your Docker image: + +``` +docker tag image-id ghcr.io/github-account/image-name:image-version +``` + +For example: + +``` +docker tag 5e369524eecb ghcr.io/anais-codefresh/react-example:1.0 +``` + +You can find your image-id by running: + +``` +docker images +``` + +Once pushed, you will see the Docker image in the packages section of your repository. +If you want, you can connect the Docker image to a repository [using the GitHub interface](https://docs.github.com/en/free-pro-team@latest/packages/managing-container-images-with-github-container-registry/connecting-a-repository-to-a-container-image) or by adding a label to your Dockerfile. + +``` +LABEL org.opencontainers.image.source https://github.com/OWNER/REPO +``` + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/github/manual-docker-push.png" + url="/images/integrations/docker-registries/github/manual-docker-push.png" + alt="Pushing a Docker image manually to GitHub packages" + caption="Pushing a Docker image manually to GitHub packages" + max-width="100%" +%} + +Now that you have verified your token, we can connect the registry to Codefresh. + +## Set up GitHub Container Registry integration + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Docker Registries** and then click **Configure**. +1. From the **Add Registry Provider** dropdown, select **Other Registries**. +1. Define the following: + * **Registry name**: A unique name for this configuration. + * **Username**: Your GitHub username. + * **Password**: Your GitHub personal token. + * **Domain**: `ghcr.io`. + * Expand **Advanced Options** and define the [**Repository Prefix**]({{site.baseurl}}/docs/integrations/docker-registries/#using-an-optional-repository-prefix) as your GitHub username. + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/github/github-registry-codefresh.png" + url="/images/integrations/docker-registries/github/github-registry-codefresh.png" + alt="GitHub Container Registry settings" + caption="GitHub Container Registry settings" + max-width="70%" +%} + +{:start="5"} +1. To verify the connection details, click **Test Connection**. +1. To apply the changes, click **Save**. + + + +## Pushing Docker image to registry + +With the registry integration in place, you can now push a Docker image in any Codefresh pipeline, simply by mentioning the registry by name (`github-container-registry` in the example). + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/github/github-registry-pipeline.png" + url="/images/integrations/docker-registries/github/github-registry-pipeline.png" + alt="Codefresh pipeline for GitHub packages" + caption="Codefresh pipeline for GitHub packages" + max-width="100%" +%} + +Here is the definition of the Codefresh pipeline. + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - "clone" + - "build" + +steps: + clone: + title: "Cloning repository" + type: "git-clone" + repo: "anais-codefresh/react-article-display" + revision: "${{CF_BRANCH}}" + git: "github" + stage: "clone" + + build: + title: "Building Docker image" + type: "build" + image_name: "react-article-display" + working_directory: "${{clone}}" + tags: + - "${{CF_BRANCH_TAG_NORMALIZED}}" + - "2.0.0" + dockerfile: "Dockerfile" + stage: "build" + registry: "github-container-registry" +{% endraw %} +{% endhighlight %} + +Notice: + +* The `registry: github-container-registry` property in the `build` step, which is the name of the registry that you set-up in the previous step +* The fact that we push multiple Docker tags in a single step; you can define all tags in the `build` step. + +After the pipeline has finished the Docker tags can also be seen in the GitHub packages section of the repository. + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/github/multiple-docker-tags.png" + url="/images/integrations/docker-registries/github/multiple-docker-tags.png" + alt="Pushing different Docker tags" + caption="Pushing different Docker tags" + max-width="100%" +%} + +You can now treat this registry like any other Codefresh registry. + + +## Related articles +[Docker registries for pipeline integrations]({{site.baseurl}}/docs/integrations/docker-registries) +[Working with Docker Registries]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/) +[Push step]({{site.baseurl}}/docs/pipelines/steps/push/) +[Building and pushing an image]({{site.baseurl}}/docs/yaml-examples/examples/build-and-push-an-image/) diff --git a/_docs/integrations/docker-registries/google-artifact-registry.md b/_docs/integrations/docker-registries/google-artifact-registry.md new file mode 100644 index 000000000..fd9e36620 --- /dev/null +++ b/_docs/integrations/docker-registries/google-artifact-registry.md @@ -0,0 +1,64 @@ +--- +title: "Google Artifact Registry (GCAR)" +description: "Use Google Artifact Registry with pipeline integrations" +group: integrations +sub_group: docker-registries +toc: true +--- + +Configure GCAR (Google Artifact Registry) as your Docker registry provider. + +## Set up GCAR integration + +**Before you begin** +* [Generate a JSON key file](#generate-a-json-key-file) + +**How to** + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Docker Registries** and then click **Configure**. +1. From the **Add Registry Provider** dropdown, select **Google Artifact Registry**. +1. Define the following: + * **Registry name**: A unique name for this configuration. + * **Location**: Select the location. + * **JSON Keyfile**: The content of the generated JSON key file. + + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/google-artifact-registry-settings.png" + url="/images/integrations/docker-registries/google-artifact-registry-settings.png" + alt="Google Artifact Registry (GCAR) settings" + caption="Google Artifact Registry (GCAR) settings" + max-width="60%" %} + +{:start="5"} +1. To verify the connection details, click **Test Connection**. +1. To apply the changes, click **Save**. + + +## Generate a JSON key file +The JSON key file holds your credentials for a given [service account](https://cloud.google.com/compute/docs/access/service-accounts){:target="\_blank"}. +To generate your key file follow these instructions: + +1. Go to your [Cloud Platform Console Credentials page](https://console.cloud.google.com/apis/credentials){:target="\_blank"}. +1. Select the project that you're creating credentials for. +1. To set up a new service account, click **Create credentials**, and then select Service account key. +1. Choose the service account to use for the key. +1. Choose to download the service account's public/private key as a JSON file. + +You can find the complete guide [here](https://support.google.com/cloud/answer/6158849#serviceaccounts){:target="\_blank"}. + +## Working with multiple projects + +If you have more than one repository/project in Google cloud, you can connect multiple GCR registries and define one as the "primary" for the `gcr.io` domain. + +This means that every time Codefresh needs to pull an image it will use that integration. If you wish to use another project for pulling images, +you can use the `registry_context` property as described in [working with multiple registries]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/#working-with-multiple-registries-with-the-same-domain). + + +## Related articles +[Docker registries for pipeline integrations]({{site.baseurl}}/docs/integrations/docker-registries) +[Working with Docker Registries]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/) +[Push step]({{site.baseurl}}/docs/pipelines/steps/push/) +[Building and pushing an image]({{site.baseurl}}/docs/yaml-examples/examples/build-and-push-an-image/) \ No newline at end of file diff --git a/_docs/integrations/docker-registries/google-container-registry.md b/_docs/integrations/docker-registries/google-container-registry.md new file mode 100644 index 000000000..e87548886 --- /dev/null +++ b/_docs/integrations/docker-registries/google-container-registry.md @@ -0,0 +1,66 @@ +--- +title: "Google Container Registry (GCR)" +description: "Use GCR with pipeline integrations" +group: integrations +sub_group: docker-registries +redirect_from: + - /docs/google-cloud-registry/ + - /docs/docker-registries/external-docker-registries/google-container-registry/ +toc: true +--- +Configure GCR (Google Container Registry) as your Docker registry provider. + +## Set up GCR integration + +**Before you begin** +* [Generate a JSON key file](#generate-a-json-key-file) + +**How to** + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Docker Registries** and then click **Configure**. +1. From the **Add Registry Provider** dropdown, select **Google Container Registry**. +1. Define the following: + * **Registry name**: A unique name for this configuration. + * **Domain**: Select the domain. + * **JSON Keyfile**: The content of the generated JSON key file. + + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/google-gcr-registry-settings.png" + url="/images/integrations/docker-registries/google-gcr-registry-settings.png" + alt="Google Container Registry (GCR) settings" + caption="Google Container Registry (GCR) settings" + max-width="60%" %}` + +{:start="5"} +1. To verify the connection details, click **Test Connection**. +1. To apply the changes, click **Save**. + + +## Generate a JSON key file +The JSON key file holds your credentials for a given [service account](https://cloud.google.com/compute/docs/access/service-accounts){:target="\_blank"}. +To generate your key file follow these instructions: + +1. Go to your [Cloud Platform Console Credentials page](https://console.cloud.google.com/apis/credentials){:target="\_blank"}. +1. Select the project that you're creating credentials for. +1. To set up a new service account, click **Create credentials**, and then select Service account key. +1. Choose the service account to use for the key. +1. Choose to download the service account's public/private key as a JSON file. + +You can find the complete guide [here](https://support.google.com/cloud/answer/6158849#serviceaccounts){:target="\_blank"}. + +## Working with multiple projects + +If you have more than one repository/project in Google cloud, you can connect multiple GCR registries and define one as the "primary" for the `gcr.io` domain. + +This means that every time Codefresh needs to pull an image it will use that integration. If you wish to use another project for pulling images, +you can use the `registry_context` property as described in [working with multiple registries]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/#working-with-multiple-registries-with-the-same-domain). + + +## Related articles +[Docker registries for pipeline integrations]({{site.baseurl}}/docs/integrations/docker-registries) +[Working with Docker Registries]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/) +[Push step]({{site.baseurl}}/docs/pipelines/steps/push/) +[Building and pushing an image]({{site.baseurl}}/docs/yaml-examples/examples/build-and-push-an-image/) \ No newline at end of file diff --git a/_docs/integrations/docker-registries/other-registries.md b/_docs/integrations/docker-registries/other-registries.md new file mode 100644 index 000000000..785cba530 --- /dev/null +++ b/_docs/integrations/docker-registries/other-registries.md @@ -0,0 +1,61 @@ +--- +title: "Other Registries" +description: "Connect any Docker registry for pipeline integration" +group: integrations +sub_group: docker-registries +redirect_from: + - /docs/other-registries/ + - /docs/docker-registries/external-docker-registries/other-registries/ +toc: true +--- +Codefresh provides an option to configure a Docker Registry not in the list of Docker registry providers. +Use this option for any cloud or hosted registry that follows the V2 Docker registry protocol. + +Some examples of self-hosted registries are: +* The [official registry](https://github.com/docker/distribution){:target="\_blank"} by Docker +* [Nexus](https://www.sonatype.com/nexus-repository-sonatype){:target="\_blank"} by Sonatype +* [Harbor](https://goharbor.io/){:target="\_blank"} by VMware +* [Portus](http://port.us.org/){:target="\_blank"} by Suse +* [Container Registry](https://www.alibabacloud.com/product/container-registry){:target="\_blank"} by Alibaba +* [Openshift registry](https://www.openshift.com/){:target="\_blank"} by Redhat +* [Kraken](https://github.com/uber/kraken){:target="\_blank"} by Uber +* [Proget](https://inedo.com/proget){:target="\_blank"} by Inedo + +## Set up Other Registry integration + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Docker Registries** and then click **Configure**. +1. From the **Add Registry Provider** dropdown, select **Other Registries**. +1. Define the following: + * **Registry name**: A unique name for this configuration. + * **Username**: Your registry username.. + * **Password**: Your registry encrypted password. + * **Domain**: Your registry address, `mydomain.com`. + +{% include + image.html + lightbox="true" + file="/images/integrations/docker-registries/add-other-docker-registry.png" + url="/images/integrations/docker-registries/add-other-docker-registry.png" + alt="Other Registry settings" + caption="Other Registry settings" + max-width="60%" %} + +{:start="5"} +1. To verify the connection details, click **Test Connection**. +1. To apply the changes, click **Save**. + + +## Heroku Registries + +To authenticate to the Heroku registry, instead of using your password, you will need to use the authorization token. You can find that by running: + +{% highlight bash %} +heroku auth:token +{% endhighlight %} + +## Related articles +[Docker registries for pipeline integrations]({{site.baseurl}}/docs/integrations/docker-registries) +[Working with Docker Registries]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/) +[Push step]({{site.baseurl}}/docs/pipelines/steps/push/) +[Building and pushing an image]({{site.baseurl}}/docs/yaml-examples/examples/build-and-push-an-image/) diff --git a/_docs/integrations/docker-registries/quay-io.md b/_docs/integrations/docker-registries/quay-io.md new file mode 100644 index 000000000..fb2912951 --- /dev/null +++ b/_docs/integrations/docker-registries/quay-io.md @@ -0,0 +1,43 @@ +--- +title: "Quay.io" +description: "Use Quay registries with pipeline integration" +group: integrations +sub_group: docker-registries +redirect_from: + - /docs/quayio/ + - /docs/docker-registries/external-docker-registries/quay-io/ +toc: true +--- + +Configure Quay as your Docker registry provider. + +## Set up Quay integration + + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Docker Registries** and then click **Configure**. +1. From the **Add Registry Provider** dropdown, select **Other Registries**. +1. Define the following: + * **Registry name**: A unique name for this configuration. + * **Username**: Your `Quay.io` username. + * **Password**: Your `Quay.io` encrypted password. + * **Domain**: `quay.io`. + +{% include image.html + lightbox="true" + file="/images/integrations/docker-registries/add-quay-registry.png" + url="/images/integrations/docker-registries/add-quay-registry.png" + alt="Quay Docker registry settings" + caption="Quay Docker registry settings" + max-width="60%" %}` + +{:start="5"} +1. To verify the connection details, click **Test Connection**. +1. To apply the changes, click **Save**. + + +## Related articles +[Docker registries for pipeline integrations]({{site.baseurl}}/docs/integrations/docker-registries) +[Working with Docker Registries]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/) +[Push step]({{site.baseurl}}/docs/pipelines/steps/push/) +[Building and pushing an image]({{site.baseurl}}/docs/yaml-examples/examples/build-and-push-an-image/) \ No newline at end of file diff --git a/_docs/integrations/gcloud-builder.md b/_docs/integrations/gcloud-builder.md new file mode 100644 index 000000000..06064a175 --- /dev/null +++ b/_docs/integrations/gcloud-builder.md @@ -0,0 +1,312 @@ +--- +title: "Google Cloud Builder" +description: "Use the Google Cloud builder to create Docker images in Codefresh pipelines" +group: integrations + +toc: true +--- + +Google Cloud builder is an online service that allows you to build Docker images using the Google infrastructure and also push them to the Google Cloud registry. + +You can also use Cloud builder in a Codefresh pipeline in place of the [normal build step]({{site.baseurl}}/docs/pipelines/steps/build/). This way you can take advantage of the Cloud builder in your Codefresh pipelines, but still push to other registries that are connected to Codefresh (and not just GCR). + + +## Prerequisites + +To use the Cloud builder service in your Codefresh pipeline you need: + +1. A free Docker Hub account and [Docker Hub connected to Codefresh]({{site.baseurl}}/docs/integrations/docker-registries/docker-hub/). +1. A Google Cloud subscription and a [service account for the Cloud builder service](https://cloud.google.com/cloud-build/docs/securing-builds/set-service-account-permissions){:target="\_blank"}. + +Save your service account as a JSON file, and make sure you select at least the [following roles](https://cloud.google.com/container-registry/docs/access-control){:target="\_blank"}: + +* Cloud storage Admin +* Storage Admin +* Storage Object Viewer +* Storage Object Creator + +You will use this JSON file either by integrating a [Google Docker registry]({{site.baseurl}}/docs/integrations/docker-registries/google-container-registry/) in Codefresh, or directly in a pipeline as we will see later. + +## How it works + +The Google Cloud builder integration/authentication can be used in the following ways: + +1. Authentication is retrieved from the GCR integration in your Codefresh account, and the resulting Docker image: + * Is also be pushed to GCR. + * Is pushed to any other [external registry connected to Codefresh]({{site.baseurl}}/docs/integrations/docker-registries/). +1. Authentication is defined in the pipeline itself, and the resulting image can be pushed to any registry connected to Codefresh + +In the first case, you will define the service account file centrally in the GCR integration screen, and then any pipeline can authenticate to Google Cloud builder without any further configuration. + +{% + include image.html + lightbox="true" + file="/images/integrations/docker-registries/add-gcr-registry.png" + url="/images/integrations/docker-registries/add-gcr-registry.png" + alt="Using the JSON service account in Codefresh" + caption="Using the JSON service account in Codefresh" + max-width="50%" +%} + + + +## Using Google Cloud builder in a Codefresh pipeline + +In the most straightforward scenario, you want to create a Docker image with Google Cloud builder and also push to GCR. + +{% include image.html +lightbox="true" +file="/images/integrations/gcloud-builder/build-push-gcr.png" +url="/images/integrations/gcloud-builder/build-push-gcr.png" +max-width="90%" +caption="Using Google cloud builder in Codefresh" +alt="Using Google cloud builder in Codefresh" +%} + +Here is the full pipeline: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'codefresh-contrib/golang-sample-app' + revision: master + git: github + MyAppDockerImage: + title: Building Docker Image + type: build + image_name: my-golang-image + working_directory: ./ + tag: slim + registry: gcr + dockerfile: Dockerfile.multistage + provider: + type: gcb + arguments: + cache: + repo: "my-kaniko-cache" + ttl: "10h" +{% endraw %} +{% endhighlight %} + + +The `build` step of the pipeline has an extra property, `provider`, that specifies we want to use Cloud builder instead of the Codefresh native build step. + +The only required argument is the repository to be used for [Kaniko caching](https://cloud.google.com/cloud-build/docs/kaniko-cache){:target="\_blank"} and speed up subsequent builds. + +>Note that the Kaniko repo should NOT be the same as the repository used for the image itself. + +{% include image.html +lightbox="true" +file="/images/integrations/gcloud-builder/image-dashboard.png" +url="/images/integrations/gcloud-builder/image-dashboard.png" +max-width="70%" +caption="Inspecting an image from Google Cloud build" +alt="Inspecting an image from Google Cloud build" +%} + +After runing the pipeline, you will see your Docker image in the [Image dashboard]({{site.baseurl}}/docs/ci-cd-guides/working-with-docker-registries/). + +The Docker image is also visible in the Google Cloud Console view of your registry. + +### Pushing to a different registry + +Even though the Cloud builder pipeline step authentication is fetched from the GCR configuration, you don't have to push to GCR. +To push the Docker image to another connected registry, simply change the `registry` property in the build step: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'codefresh-contrib/golang-sample-app' + revision: master + git: github + MyAppDockerImage: + title: Building Docker Image + type: build + image_name: my-golang-image + working_directory: ./ + tag: slim + registry: azure + dockerfile: Dockerfile.multistage + provider: + type: gcb + arguments: + cache: + repo: "my-kaniko-cache" + ttl: "10h" +{% endraw %} +{% endhighlight %} + +This pipeline pushes the Docker image created to another registry that is identified by [azure]({{site.baseurl}}/docs/integrations/docker-registries/azure-docker-registry/). + +### Authenticating to Cloud Builder in the pipeline + +If you don't want to reuse the Registry integration provided by Codefresh for easy authentication to Google Cloud builder, you can also use your service account JSON file directly in the pipeline. + +You can pass the contents of the JSON file as a variable in the pipeline and the build step will use it to authenticate. + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'codefresh-contrib/golang-sample-app' + revision: master + git: github + MyAppDockerImage: + title: Building Docker Image + type: build + image_name: my-golang-image + working_directory: ./ + tag: slim + registry: azure + dockerfile: Dockerfile.multistage + provider: + type: gcb + arguments: + google_app_creds: '${{G_CREDS_B64}}' + cache: + repo: "my-kaniko-cache" + ttl: "10h" +{% endraw %} +{% endhighlight %} + +Here the pipeline will try to authenticate to Google Cloud builder using the contents of the `google_app_creds` property. + +The value of this property can be a pipeline variable, or project variable or any other standard Codefresh method such as [shared configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/). + +You need to escape the contents of the service account before you use in the pipeline with either of these commands on your local workstation: + +* `cat _json_key_file | base64 | tr -d ‘\n’` +* `cat _json_key_file | base64 -w 0` + +### Using extra properties for Google Cloud builder + +The build step has several other properties can be used to fine-tune the Google Cloud builder behavior. + +Here is the full syntax: + + +{% highlight yaml %} +{% raw %} + +step_name: + type: build + title: Step Title + description: Free text description + working_directory: ${{clone_step_name}} + dockerfile: path/to/Dockerfile + image_name: owner/new-image-name + tag: develop + build_arguments: + - key=value + target: stage1 + no_cache: false + no_cf_cache: false + fail_fast: false + registry: my-registry + provider: + type: gcb + arguments: + google_app_creds: '${{G_CREDS_B64}}' + cache: + repo: "repositoryname/kaniko-cache" + ttl: "10h" + timeout: "600s" + machineType: 'N1_HIGHCPU_8' + logsBucket: "gs://your-project_cloudbuild/logs" + diskSizeGb: 10 + +{% endraw %} +{% endhighlight %} + +The extra fields are: + +{: .table .table-bordered .table-hover} +| Field | Description | Required/Optional/Default | +| ------------------------------------------ | ------------------------------------------------------ | ------------------------- | +| `type` | Defines which provider to use (currently `gcb` and `cf` types are available). It uses `cf` provider by default and the whole provider section can be omitted for a regular build step. | Required | +| `arguments` | Parameters for Google Cloud builder | Required | +| `google_app_creds` | base64 encoded string of the [Google app credentials JSON](https://cloud.google.com/docs/authentication/production){:target="\_blank"}. By default, taken from the existing GCR integration. | Optional | +| `cache` | The list of Kaniko cache parameters | Required | +| `repo` | Docker repository path for the Kaniko cache | Required | +| `ttl` | Kaniko cache retention. Default value is `336h` | Optional | +| `timeout` | This field is directly translated into the corresponding field of the [GCB manifest file](https://cloud.google.com/cloud-build/docs/build-config#structure_of_a_build_config_file){:target="\_blank"}. Default is `10m` | Optional | +| `machineType` | This field is directly translated into the corresponding field of the [GCB manifest file](https://cloud.google.com/cloud-build/docs/build-config#structure_of_a_build_config_file){:target="\_blank"} | Optional | +| `diskSizeGb` | This field is directly translated into the corresponding field of the [GCB manifest file](https://cloud.google.com/cloud-build/docs/build-config#structure_of_a_build_config_file) {:target="\_blank"} | Optional | +| `logsBucket` | This field is directly translated into the corresponding field of the [GCB manifest file](https://cloud.google.com/cloud-build/docs/build-config#structure_of_a_build_config_file){:target="\_blank"} | Optional | + + + + +The step also accepts all the field of the [standard build step]({{site.baseurl}}/docs/pipelines/steps/build/) but notice that the following fields are not supported in the current implementation and simply ignored by the GCB step logic: + +* `no_cache` +* All the [buildkit]({{site.baseurl}}/docs/pipelines/steps/build/#buildkit-support) related fields + +Here is an example that uses all possible fields: + + `YAML` +{% highlight yaml %} +{% raw %} +GCBuild: + type: build + image_name: '${{IMAGE_NAME}}' + working_directory: ${{CloneStep}} + tag: your-tag1 + tags: + - your-tag2 + - your-tag3 + target: 'test' + no_cf_cache: false + metadata: + set: + - qa: pending + build_arguments: + - WORD=Hello + registry: 'reg-integration-name' + dockerfile: + content: |- + FROM alpine as test + RUN apk add skopeo + ARG WORD + RUN echo $WORD + provider: + type: gcb + arguments: + google_app_creds: '${{G_CREDS_B64}}' + cache: + repo: "repositoryname/kaniko-cache" + ttl: "10h" + timeout: "600s" + machineType: 'N1_HIGHCPU_8' + logsBucket: "gs://your-project_cloudbuild/logs" + diskSizeGb: 10 +{% endraw %} +{% endhighlight %} + + + + + + +## Related articles +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[Pipeline steps]({{site.baseurl}}/docs/pipelines/steps/) +[Google Registry integration]({{site.baseurl}}/docs/integrations/docker-registries/google-container-registry/) +[Build and push an image]({{site.baseurl}}/docs/example-catalog/ci-examples/build-and-push-an-image/) + + + diff --git a/_docs/integrations/git-providers.md b/_docs/integrations/git-providers.md new file mode 100644 index 000000000..496a7d631 --- /dev/null +++ b/_docs/integrations/git-providers.md @@ -0,0 +1,400 @@ +--- +title: "Git Providers" +description: "Easily check out code in Codefresh CI pipelines" +group: integrations +redirect_from: + - /docs/git-provider/ + - /docs/integrations/ + - /docs/integrations/git-provider/ + - /docs/integrations/git-providers/integrating-codefresh-with-multiple-git-providers/ + - /docs/integrations/git-providers/configure-a-bitbucket-server-webhook/ +toc: true +--- +Creating an account with Codefresh using one of the supported Git providers (GitHub, GitLab, Bitbucket) gives you immediate access to the repositories of the linked provider. + +You can add repositories from the other Git providers regardless of the one that you used for sign-up. For example, you can use GitLab to sign up with Codefresh, but still build repositories that exist in Bitbucket. + +You can even add multiple accounts from each Git provider (if you have more than one) allowing you to use Codefresh as a central CI/CD solution that can access all your Git repositories regardless of the backing Git provider. + +Currently Codefresh supports: + +* GitHub Cloud +* GitHub On-premises +* Bitbucket +* GitLab Cloud +* GitLab On-premises +* Azure DevOps Git +* Atlassian Stash (old version of Bibucket Server) +* Bitbucket Server (new version of Stash) + +Atlassian Stash/Bitbucket server as well as the on-premises version of GitLab and GitHub are only available to Codefresh enterprise customers. + +## Adding more Git providers to your Codefresh Account + +By default, you have direct access to Git repositories that exist in the Git provider that you used while signing up for Codefresh. You can easily create Codefresh projects that checkout code from that Git provider without any extra configurations. + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Git** and then click **Configure**. +1. From the **Add Git Provider** drop-down, select the Git provider to add. +1. Define the settings as required. + + +{% include image.html lightbox="true" file="/images/integrations/codefresh-integrations.png" url="/images/integrations/codefresh-integrations.png" alt="Codefresh Account Integration" max-width="80%" %} + + +{% include image.html +lightbox="true" +file="/images/integrations/git/git-provider-menu.png" +url="/images/integrations/git/git-provider-menu.png" +max-width="60%" +caption="Add Git provider" +alt="Add Git provider" +%} + +For each Git provider you need to set up authentication, for Codefresh to get access to the public and private repositories of the respective provider. + +The easiest way to set up authentication is with OAuth2 if supported by the Git provider. You only need to name your integration +and Codefresh will automatically set it up once you accept the permissions required. If you have problems with OAuth2 +or the provider does not support it, you need to manually create credentials by yourself in your git account and then enter them into Codefresh. + +In the case of an on-premises Git provider you also need to fill in the URL where the provider is installed. + +## SSH Keys + +> Please contact support to enable this feature. + +You have the ability to specify whether you want to clone via HTTPS or SSH. + +1. Select the required Git integration, and click **Edit**. +1. Expand **Advanced Options** and toggle to **HTTPS** or **SSH**. +1. For SSH, paste your **raw**, private key into the SSH Key text box and click **Save**. + + +{% include image.html +lightbox="true" +file="/images/integrations/git/github-ssh.png" +url="/images/integrations/git/github-ssh.png" +max-width="40%" +caption="Git clone via SSH" +alt="Git clone via SSH" +%} + + +For more information on generating SSH keys and adding your public key to your VCS provider, see its official documentation: + +* [GitHub documentation](https://help.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent){:target="\_blank"} +* [GitLab documentation](https://docs.gitlab.com/ee/ssh/#generating-a-new-ssh-key-pair){:target="\_blank"} +* [Bitbucket documentation](https://confluence.atlassian.com/bitbucket/set-up-an-ssh-key-728138079.html){:target="\_blank"} +* [Azure documentation](https://docs.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops&tabs=current-page){:target="\_blank"} + +## GitHub + +For the **OAuth2 method** you only need to decide on public/private repository access, enter a name for your connection and click *Save*. Then accept the permissions dialog. This is the easiest and recommended way to integrate GitHub. Notice that if +you used GitHub when you [created your Codefresh account]({{site.baseurl}}/docs/administration/create-a-codefresh-account/), this integration is already setup for you. + +For the **Access Token** method you need + +* A friendly name for the Git context (it can be anything you want) +* An access token + +>Note that the access token for an organization should be created by somebody who has **Owner** role and not just **Member** role. + +To create an [access token](https://github.com/settings/tokens){:target="\_blank"}, go to your GitHub *settings* and select the *Developer settings* option from the left +sidebar. Then select *Personal access tokens* from the left menu. +For more information see the [GitHub Documentation page](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/){:target="_blank"}. + +The "token description" you enter in your GitHub account in order to create the token is completely arbitrary (use "Codefresh" for an example). Once you have the token, paste it in the Codefresh UI and click *Test connection*. If everything is OK, you can +now save the Git integration. + +The minimum permissions for the token are: + +* `repo.*` +* `admin:repo_hook.*` + +{% include image.html +lightbox="true" +file="/images/integrations/git/github-required-scopes.png" +url="/images/integrations/git/github-required-scopes.png" +max-width="40%" +caption="GitHub permissions" +alt="GitHub permissions" +%} + +For GitHub on-premises you also need to provide the URL of the GitHub server in your organization. If enabled in your account you can setup [Pipeline definition restrictions]({{site.baseurl}}/docs/administration/access-control/#pipeline-definition-restrictions) by expanding the *YAML Options* segment. + +### Using External Secrets for GitHub Token + +If your GitHub installation is behind your firewall, you can also +use any [external secrets that you have defined]({{site.baseurl}}/docs/integrations/secret-storage/) (such as Kubernetes secrets) as values by entering a secrets value +with the same syntax [shown in pipelines]({{site.baseurl}}/docs/pipelines/secrets-store/). + +For example if you already have a `token` on a resource call `git-credentials` you can put in the token field the expression {% raw %}`${{secrets.git-credentials.token}}`{% endraw %}. + +### Level of access + +When the admin clicks off "Allow access to all users" another toggle appears; “Allow these credentials to be shared within a pipeline for cloning a repository“ + +1. When its turned on, the user that runs a pipeline will be able to clone the repo. +2. When its turned off, the user that runs a pipeline cannot use this integration. +In both cases, the user cannot decrypt the token used in Git integration with CLI or API. + +>Important note: The credentials will be shared only to clone repos using an official git-clone step. + +## GitHub-App + +An alternative way to authenticate with Github is via the App mechanism. + +### Codefresh Github App + +> Note: The Codefresh App has READ permissions to issues, metadata, and pull requests, and READ and WRITE permissions to code, commit statuses, and repository hooks. If you need additional permission for your integration, use the Manual Creation steps. + +1. In the Codefresh UI, follow the steps to [add a new Git provider](#adding-more-git-providers-to-your-codefresh-account). +1. From the list of Git providers, select **Codefresh Github App**. +1. Select Setup GitHub App integration via [**GitHub Marketplace**](https://github.com/apps/codefresh-githubapp){:target=\_blank"}. +1. Follow the instructions on GitHub to install the application. + Once completed, the fields are automatically populated with the information. +1. To verify your integration, click **Test connection**. +1. To apply your changes, click **Save**. + +### Manual creation + +**Step 1** - Log in your GitHub account and visit [https://github.com/settings/apps](https://github.com/settings/apps){:target="\_blank"}. Click the *New GitHub App* button. + +**Step 2** - On the New app screen + +1. Give an arbitrary name to your app (e.g. codefresh-integration) +1. Fill *Homepage URL* with `http://www.codefresh.io` +1. Uncheck the *Active* checkbox under the Webhook section +1. In the *Repository permissions* section give the minimum of + * **Contents** - read + * **Issues** - read + * **Metadata** - read + * **Pull requests** - read + * **Webhooks** - read, write + * **Commit statuses** - read, write + * **Email addresses** - read +1. Click the *Create GitHub app* button. + +**Step 3** - In the next screen + +1. Note down the *App ID* number under the *About* section +1. Click the *Generate a private key* button and save the file locally + +**Step 4** - Click the *Install App* item from the left sidebar menu and then click the *Install* button next to your codefresh app + +**Step 5** - Accept the permissions and in the next screen define the repositories that you need Codefresh to access + +Also from the URL of the browser note the ending number (this is your installation id). For example if the URL is `https://github.com/settings/installations/10042353` then your installation number is 10042353 + +**Step 6** - Visit [https://g.codefresh.io/account-admin/account-conf/integration/git](https://g.codefresh.io/account-admin/account-conf/integration/git) in Codefresh, add a new Git provider and choose *Github App* from the drop-down menu + +For the required fields use: + +* **Installation id** - found in step 5 +* **App ID** - found in step 3 +* **Private key** - the contents of the file your created in step 3 (but convert it to base64 first) + +Click *Test connection* to verify your integration and apply your changes with the *Save* button. If enabled in your account you can setup [Pipeline definition restrictions]({{site.baseurl}}/docs/administration/access-control/#pipeline-definition-restrictions) by expanding the *YAML Options* segment. + +## GitLab + +For the **OAuth2 method** you only need to enable private repository access, enter a name for your connection and click *Save*. Then accept the permissions dialog. This is the easiest and recommended way to integrate GitLab. Notice that if +you used GitLab when you [created your Codefresh account]({{site.baseurl}}/docs/administration/create-a-codefresh-account/), this integration is already setup for you. + +For the **Access Key** method you need: + +* A friendly name for the Git context (it can be anything you want.) +* An access token/key + +To create an access token, go to your GitLab *settings* and select the *Access tokens* options. +For more information see the [GitLab Documentation page](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html){:target="_blank"} + +The name you enter in order to create the token in the GitLab UI is completely arbitrary (use "Codefresh" for an example) + +Once you have the token, paste it in the Codefresh UI and click *Test connection*. If everything is OK can +now save the Git integration. + +For GitLab on-premises you also need to provide the URL of the GitLab server in your organization. If enabled in your account you can setup [Pipeline definition restrictions]({{site.baseurl}}/docs/administration/access-control/#pipeline-definition-restrictions) by expanding the *YAML Options* segment. + +### Using External Secrets for GitLab Token + +If your GitLab installation is behind your firewall, you can also +use any [external secrets that you have defined]({{site.baseurl}}/docs/integrations/secret-storage/) (such as Kubernetes secrets) as values by entering a secrets value +with the same syntax [shown in pipelines]({{site.baseurl}}/docs/pipelines/secrets-store/). + +For example if you already have a `token` on a resource call `git-credentials` you can put in the token field the expression {% raw %}`${{secrets.git-credentials@token}}`{% endraw %}. + +## Bitbucket + +For the **OAuth2 method** you only need to enter a name for your connection and click *Save*. Then accept the permissions dialog. This is the easiest and recommended way to integrate Bitbucket. Notice that if +you used Bitbucket when you [created your Codefresh account]({{site.baseurl}}/docs/administration/create-a-codefresh-account/), this integration is already setup for you. + +For the **Application Password** method you need: + +* A friendly name for the Git context (It can be anything you want.) +* The name of your Bitbucket account/email address +* A Bitbucket application password + +To create an application password, go to your *Bitbucket settings* and select *App passwords* from the sidebar. +Click the button to create one. For more information see the [Bitbucket Documentation page](https://confluence.atlassian.com/bitbucket/app-passwords-828781300.html){:target="_blank"}. + +The minimum permissions needed by Codefresh are shown below. + +{% include image.html +lightbox="true" +file="/images/integrations/git/bitbucket-permissions.png" +url="/images/integrations/git/bitbucket-permissions.png" +max-width="40%" +caption="Bitbucket permissions" +alt="Bitbucket permissions" +%} + +The "label" you enter in your Bitbucket account in order to create the application password is completely arbitrary (use "Codefresh" for an example). Once you have the token, paste it in the Codefresh UI and click *Test connection*. If everything is OK you can +now save the Git integration. + + If enabled in your account you can setup [Pipeline definition restrictions]({{site.baseurl}}/docs/administration/access-control/#pipeline-definition-restrictions) by expanding the *YAML Options* segment. + +## Azure DevOps + +For Azure you need to create a [personal access token](https://docs.microsoft.com/en-us/azure/devops/integrate/get-started/authentication/pats?view=azure-devops){:target="\_blank"}. Sign in your Azure DevOps account and click on your profile icon on the top right corner. Then select *Security*: + +{% include image.html +lightbox="true" +file="/images/integrations/git/azure-devops-security.png" +url="/images/integrations/git/azure-devops-security.png" +max-width="60%" +caption="Azure DevOps Security" +alt="Azure DevOps Security" +%} + +On the screen that will appear click the *New token* Button. Enter an arbitrary name for the token and select the correct +**Organization** from the drop-down menu. Remember your organization name as you will use it later in the Codefresh side. +Select an expiration date for your token + +> At the time of writing Azure DevOps does not have the option to create a token that is valid for ever. Choose a large +time period and make sure that you have a policy in place for renewing your tokens so that Codefresh can continue to read your Git repo. + +{% include image.html +lightbox="true" +file="/images/integrations/git/azure-devops-token.png" +url="/images/integrations/git/azure-devops-token.png" +max-width="60%" +caption="Azure DevOps Token" +alt="Azure DevOps Token" +%} + +From the *Scope* section choose the option *Show all scopes* and choose the following: + +* Code - read +* Code - status +* Graph - read +* Project and Team - read +* User profile - read + +Finally click the *Create* button and copy your token (it will never be shown again). + +Then at the Codefresh configuration enter your organization name and your token. + +{% include image.html +lightbox="true" +file="/images/integrations/git/azure-devops-verify.png" +url="/images/integrations/git/azure-devops-verify.png" +max-width="40%" +caption="Codefresh integration with Azure Devops" +alt="Codefresh integration with Azure Devops" +%} + +Click on *Test connection* to verify your settings and finally click save. Now you can [create pipelines]({{site.baseurl}}/docs/pipeline /pipelines/) +that use Azure DevOps Git repos. + +{% include image.html +lightbox="true" +file="/images/integrations/git/azure-devops-connected.png" +url="/images/integrations/git/azure-devops-connected.png" +max-width="40%" +caption="Codefresh integration with Azure Devops" +alt="Codefresh integration with Azure Devops" +%} + +Your Azure DevOps repositories will be available when [creating a new project in Codefresh]({{site.baseurl}}/docs/getting-started/create-a-basic-pipeline/). + + If enabled in your account you can setup [Pipeline definition restrictions]({{site.baseurl}}/docs/administration/access-control/#pipeline-definition-restrictions) by expanding the *YAML Options* segment. + +## Atlassian Stash + +Atlassian stash is only available for an on-premises connection. Follow the same instructions as Bitbucket. +You also need to provide the URL of the Stash server in your organization. + +This option is only for Atlassian stash until version 3.10 which is the old version. It was then renamed +to Bitbucket server. + +## Bitbucket Server + +Bitbucket server is the new and current name of Atlassian Stash. Again, it is only available for an on-premises +installation. + +Codefresh supports Bitbucket server versions 5.4.0+ since those expose the API used by the integration. + +### Using External Secrets for BitBucket Token + +If your Bitbucket Server installation is behind your firewall, you can also +use any [external secrets that you have defined]({{site.baseurl}}/docs/integrations/secret-storage/) (such as Kubernetes secrets) as values by entering a secrets value +with the same syntax [shown in pipelines]({{site.baseurl}}/docs/pipelines/secrets-store/). + +For example if you already have a `token` on a resource call `git-credentials` you can put in the token field the expression {% raw %}`${{secrets.git-credentials@token}}`{% endraw %}. + +## Using your Git provider + +Once your provider is active, you can add a new project into Codefresh and then during the [repository selection screen]({{site.baseurl}}/docs/getting-started/create-a-basic-pipeline/) you will have access to the additional Git providers. + +{% include image.html +lightbox="true" +file="/images/integrations/git/select-git.png" +url="/images/integrations/git/select-git.png" +max-width="60%" +caption="Select Git provider" +alt="Select Git provider" +%} + +>Notice that for all supported Git providers Codefresh will automatically create all the webhooks needed for +triggering pipelines when a commit (or another event) happens. + +After adding the repository Codefresh will behave exactly the same, regardless of the selected Git provider. +You will be able to [create pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) for different Git providers in exactly the same manner. + +## ABAC for Git Contexts + +**Account Level:** Pro and above + +> At this time, you will need to reach out to support to enable ABAC for Git Context and [Pipeline Execution Context]({{site.baseurl}}/docs/administration/pipeline-execution-context/). + +ABAC for Git Context gives the ability to restrict using and handling of Git Contexts. We use tags on the git context to limit Teams and Execution Contexts for access control. There are four actions controlled by ABAC: Creating, Updating, Deleting, and Using Git Contexts. + +The Using means the following use cases: + +* Creating trigger +* Getting YAML from a repository +* Using the Git Context in a pipeline (git clone step etc.) via Execution Context. + +You will get an error of Permission Denied or Forbidden to a Git Context that you do not have the correct permissions for that action. + +### Tagging the Git Context + +1. Navigate to Account Settings > Integrations > Configure for Git. +1. Hovering over the integration name (Git Context), you will see "Edit Tags" just before the edit symbol. +1. Select "Edit Tags," and you can add and remove tags. +1. Click Save when done. + +### Setting the Permissions + +1. Navigate to Account Settings > Permissions > Teams or Execution Context. +1. Scroll to Git Contexts. +1. Here, you can set [permissions]({{site.baseurl}}/docs/administration/access-control/#creating-a-security-policy) similar to other ABAC rules for Teams or Execution Context to Create or Use, Update, and Delete actions. +1. Click Add Rule when done. + +## Related articles +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[Git triggers]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/) +[Git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/) +[Checking out source code]({{site.baseurl}}/docs/example-catalog/ci-examples/git-checkout/) + diff --git a/_docs/integrations/github-actions.md b/_docs/integrations/github-actions.md new file mode 100644 index 000000000..fce8587be --- /dev/null +++ b/_docs/integrations/github-actions.md @@ -0,0 +1,161 @@ +--- +title: "GitHub Actions pipeline integration" +description: "Using the GitHub action converter in Codefresh pipelines" +group: integrations + +toc: true +--- + +[GitHub Actions](https://github.com/features/actions){:target="\_blank"} are a set of reusable workflows that can be composed to create automation sequences for GitHub projects. GitHub Actions are supported natively with GitHub, but you can also use them in Codefresh pipelines by automatically converting them to [Codefresh pipeline steps]({{site.baseurl}}/docs/pipelines/steps/). + + +{% include image.html +lightbox="true" +file="/images/integrations/github-actions/github-actions-marketplace.png" +url="/images/integrations/github-actions/github-actions-marketplace.png" +max-width="60%" +caption="GitHub Actions Marketplace" +alt="GitHub Actions Marketplace" +%} + +By using GitHub Actions in your marketplace, you have the following benefits: +* Access to a vast catalog of reusable pipeline components +* Ability to use GitHub Actions with any provider, as Codefresh supports [Bitbucket, GitLab, Azure Git etc.]({{site.baseurl}}/docs/integrations/git-providers/) + + +## Prerequisites + +To use a GitHub Action in Codefresh you need to make sure that the following apply: + +1. The [GitHub action](https://github.com/marketplace?type=actions){:target="\_blank"} you have selected is Docker-based and has a self-contained and valid Dockerfile +1. You have read the documentation of the GitHub Action and know what arguments/input it requires + + +>Tip: + Since GitHub Actions are created by the community, it is your responsibility to filter and curate any GitHub Action you wish to use in Codefresh pipelines. If for example you use a GitHub Action that is then removed by its owner, the Codefresh pipeline that uses it will break as well. + We suggest you first use a GitHub Action in a GitHub workflow in order to understand its requirements, before you use it in a Codefresh pipeline. + +## How it works + +Codefresh offers a `github-action-to-codefresh` step converter. +This converter has two functions: + +1. When you create your pipeline it will analyze the GitHub Action and find what arguments it requires. + You must then define the values needed by yourself. +1. When the pipeline runs, it automatically finds the Dockerfile of the GitHub Action, builds it, and makes available the Docker image in any subsequent step in the same pipeline. + +All this process is automatic. You just need to make sure that all arguments/inputs of the GitHub Action are provided using [pipeline variables]({{site.baseurl}}/docs/pipelines/pipelines/#creating-new-pipelines), [shared configuration]({{site.baseurl}}/docs/pipeplines/pipeline/shared-configuration/), or any other standard mechanism you already use in Codefresh. + +## Inserting a GitHub Action in Codefresh pipeline + +1. [Create a Codefresh pipeline]({{site.baseurl}}/docs/pipelines/pipelines/#creating-new-pipelines) by visiting the pipeline editor. +1. In the **Steps** tab on the right-hand side, search for `actions` and select **GitHub Actions**. + +{% include image.html +lightbox="true" +file="/images/integrations/github-actions/github-action-step-browser.png" +url="/images/integrations/github-actions/github-action-step-browser.png" +max-width="50%" +caption="Step browser" +alt="Step browser" +%} + +{:start="3"} +1. Scroll down to find the GitHub Action that you want to use or enter a keyword to filter the list. + +{% include image.html +lightbox="true" +file="/images/integrations/github-actions/select-github-action.png" +url="/images/integrations/github-actions/select-github-action.png" +max-width="70%" +caption="Select GitHub action" +alt="Select GitHub action" +%} + +{:start="4"} +1. Click on the GitHub Action you want to use in your pipeline. + Codefresh displays the [YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) for the action on the right. This is the step that you need to insert in your pipeline. + +{% include image.html +lightbox="true" +file="/images/integrations/github-actions/snyk-action-arguments.png" +url="/images/integrations/github-actions/snyk-action-arguments.png" +max-width="70%" +caption="Using the Snyk GitHub action" +alt="Using the Snyk GitHub action" +%} + + The YAML snippet is a template, and you need to fill the `env` block below the `arguments` block. + The required environment variables are specific to each GitHub Action, so check the documentation of the action itself. + + + In the example above, we use the Snyk GitHub Action. By visiting the [documentation page](https://github.com/marketplace/actions/snyk-cli-action){:target="\_blank"}, we find that this action expects a `SNYK_TOKEN` as input. + + We therefore add the token as a pipeline variable: + +{% include image.html +lightbox="true" +file="/images/integrations/github-actions/environment-variables.png" +url="/images/integrations/github-actions/environment-variables.png" +max-width="60%" +caption="Pipeline variables" +alt="Pipeline variables" +%} + +Here is the final pipeline: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + snyk-cli-action: + title: snyk + description: snyk + type: github-action-executor + arguments: + url: 'https://github.com/marketplace/actions/snyk-cli-action' + envs: + - SNYK_TOKEN: '${{SNYK_TOKEN}}' + cmd: test alpine@latest +{% endraw %} +{% endhighlight %} + +The `cmd` property is specific to each GitHub action and in the case of Snyk we say we want to scan an alpine image for security issues. + + + +## Running a Codefresh pipeline with GitHub Actions + + +You can run the pipeline as any other Codefresh pipeline. + +{% include image.html +lightbox="true" +file="/images/integrations/github-actions/github-action-pipeline.png" +url="/images/integrations/github-actions/github-action-pipeline.png" +max-width="80%" +caption="Using GitHub Actions in a Codefresh pipeline" +alt="Using GitHub Actions in a Codefresh pipeline" +%} + + +Once the pipeline reaches the GitHub Action step, the converter automatically does the following: + +1. Finds the Dockerfile of the GitHub Action +1. Builds the Dockerfile +1. Takes the resulting image and inserts it as Codefresh step +1. Passes the environment variables as arguments to the GitHub Action +1. Runs the `cmd` command + +If you have issues, please contact us or open a support ticket, and let us know which GitHub Action you are trying to use and the URL of the Codefresh build that fails. + + +## Related articles +[Pipeline steps]({{site.baseurl}}/docs/pipelines/steps/) +[Plugin marketplace](https://codefresh.io/steps/){:target="\_blank"} + + + + + diff --git a/_docs/integrations/google-cloud.md b/_docs/integrations/google-cloud.md new file mode 100644 index 000000000..89b673ea9 --- /dev/null +++ b/_docs/integrations/google-cloud.md @@ -0,0 +1,122 @@ +--- +title: "Google Cloud pipeline integration" +description: "Use Google Cloud with Codefresh pipelines" +group: integrations +redirect_from: + - /docs/deploy-your-containers/kubernetes/ +toc: true +--- + +Codefresh has native support for Google Cloud in the following areas: + +- [Connecting to Google registries]({{site.baseurl}}/docs/docker-registries/google-container-registry/) +- [Deploying to GKE]({{site.baseurl}}/docs/integrations/kubernetes/#adding-gke-cluster) +- [Using Google Storage for Test reports]({{site.baseurl}}/docs/testing/test-reports/#connecting-a-google-bucket) +- [Using Google Storage for Helm charts]({{site.baseurl}}/docs/deployments/helm/add-helm-repository/#private-repository---gcs) +- [Using Cloud Build]({{site.baseurl}}/docs/integrations/gcloud-builder/) +- [Installing the Runner via the Marketplace]({{site.baseurl}}/docs/integrations/google-marketplace/) + + +## Using Google Container Registries + +Google Container registries are fully compliant with the Docker registry API that Codefresh follows. You can connect GCR like any [other Docker registry]({{site.baseurl}}/docs/docker-registries/google-container-registry/). + +{% + include image.html + lightbox="true" +file="/images/integrations/docker-registries/add-gcr-registry.png" +url="/images/integrations/docker-registries/add-gcr-registry.png" +alt="Connecting to GCR" +caption="Connecting to GCR" +max-width="70%" +%} + +Once the registry is added, you can the [standard push step]({{site.baseurl}}/docs/pipelines/steps/push/) in pipelines. See also the documentation page for [working with Docker registries]({{site.baseurl}}/docs/integrations/docker-registries/). + +## Deploying to Google Kubernetes Engine + +Codefresh has native support for connecting a GKE cluster in the [cluster configuration screen]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster). + +{% + include image.html + lightbox="true" +file="/images/integrations/google-cloud/gke-integration.png" +url="/images/integrations/google-cloud/gke-integration.png" +alt="Connecting a GKE cluster" +caption="Connecting a GKE cluster" +max-width="40%" +%} + +Once the cluster is connected, you can use any of the [available deployment options]({{site.baseurl}}/docs/deployments/kubernetes/deployment-options-to-kubernetes/) for Kubernetes clusters. +You also get access to all other Kubernetes dashboards such as the [cluster dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) or the [environment dashboard]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/). + +## Storing test reports in Google Cloud storage + +Codefresh has native support for test reports. You can store the reports on Google Cloud storage. + +{% include +image.html +lightbox="true" +file="/images/integrations/google-cloud/google-cloud-storage.png" +url="/images/integrations/google-cloud/google-cloud-storage.png" +alt="Google cloud storage" +caption="Google cloud storage" +max-width="50%" +%} + +See the full documentation for [test reports]({{site.baseurl}}/docs/testing/test-reports/). + +## Using Google Storage for storing Helm charts + +You can connect Google storage as a Helm repository by setting up a [Helm integration]({{site.baseurl}}/docs/integrations/helm/#$add-helm-repository/) in Codefresh. + +{% include +image.html +lightbox="true" +file="/images/integrations/google-cloud/google-storage-helm-repo.png" +url="/images/integrations/google-cloud/google-storage-helm-repo.png" +alt="Using Google Cloud for Helm charts" +caption="Using Google Cloud for Helm charts" +max-width="60%" +%} + +Once you connect your Helm repository you can use it any [Codefresh pipeline with the Helm step]({{site.baseurl}}/docs/new-helm/using-helm-in-codefresh-pipeline/). + +## Using Google Cloud build + +Codefresh has a [native Docker build step]({{site.baseurl}}/docs/pipelines/steps/build/) for creating Docker images. As an alternative method of building Docker images, you can also use [Google Cloud Build]({{site.baseurl}}/docs/integrations/gcloud-builder/) in a Codefresh pipeline. + +## Installing the Codefresh runner from the Google Marketplace + +The [Codefresh runner]({{site.baseurl}}/docs/installation/codefresh-runner/) is a Kubernetes native application that allows you to run pipelines on your own Kubernetes cluster (even behind the firewall). Specifically for Google Cloud, the runner is also available via the [marketplace]({{site.baseurl}}/docs/integrations/google-marketplace/){:target="\_blank"}. + + +## Traditional Google Cloud deployments + +For any other Google Cloud deployment you can use the [Google Cloud CLI from a Docker image](https://hub.docker.com/r/google/cloud-sdk/){:target="\_blank"} in a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). + +`YAML` +{% highlight yaml %} +{% raw %} + create_a_vm: + title: "Creating a Virtual machine" + type: "freestyle" + arguments: + image: "google/cloud-sdk:slim" + commands: + - echo $KEY_FILE | base64 --decode > key_file.json + - gcloud compute instances create demo-codefresh --image codefresh-simple-ubuntu-vm --zone europe-west1-b --metadata-from-file startup-script=startup.sh --tags http-server --preemptible --quiet +{% endraw %} +{% endhighlight %} + +See the example of [uploading to a Google Bucket]({{site.baseurl}}/docs/example-catalog/ci-examples/uploading-or-downloading-from-gs/) or [creating a VM]({{site.baseurl}}/docs/example-catalog/ci-examples/packer-gcloud/) for more details. + + + + + +## Related articles +[Add your cluster]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster) +[Manage your Kubernetes cluster]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) +[Cloning Git repositories]({{site.baseurl}}/docs/example-catalog/ci-examples/git-checkout/) + diff --git a/_docs/integrations/google-marketplace.md b/_docs/integrations/google-marketplace.md new file mode 100644 index 000000000..e02b36aa1 --- /dev/null +++ b/_docs/integrations/google-marketplace.md @@ -0,0 +1,249 @@ +--- +title: "Google Marketplace integration" +description: "Learn how to run Codefresh pipelines inside your GKE cluster" +group: integrations +toc: true +--- + +Codefresh has partnered with [Google Cloud](https://cloud.google.com/){:target="\_blank"} and allows you to install a Codefresh pipeline builder within your own Kubernetes cluster. +The integration is available in the Google Marketplace for Kubernetes apps at [https://console.cloud.google.com/marketplace/details/codefresh-gke/codefresh](https://console.cloud.google.com/marketplace/details/codefresh-gke/codefresh){:target="\_blank"}. + +Once you configure Codefresh to use your own Kubernetes cluster for builds, you can enjoy all benefits of a **hybrid** installation as the Codefresh UI and management dashboards will still run in a SAAS manner, while the actual builds and pipelines will execute in your own cluster. + +The major benefits are the following: + + * You define exactly what resources are used for your builds instead of relying on Codefresh infrastructure. + * The management UI still runs in the Codefresh premises and is managed by the Codefresh team allowing you to focus on your builds. + * The Codefresh builder has access to all private resources that run in your cluster so it is very easy to use resources that should not be exposed to the Internet for any reason. + * Unified billing. You pay a single bill to Google that includes the price for your Kubernetes cluster as well as the Codefresh pipelines. + + +To start the integration, you need the following: + +1. A [Google Cloud account](https://cloud.google.com/){:target="\_blank"} with billing enabled +1. A [GKE cluster](https://cloud.google.com/kubernetes-engine/docs/quickstart){:target="\_blank"} that will run all builds and pipelines +1. A [Codefresh account]({{site.baseurl}}/docs/administration/create-a-codefresh-account/) (creating an account is free, you pay only for builds) + +Then visit the Codefresh GKE page at [https://console.cloud.google.com/marketplace/details/codefresh-gke/codefresh](https://console.cloud.google.com/marketplace/details/codefresh-gke/codefresh){:target="\_blank"}. + +## Using Codefresh from the Google Marketplace + +When you configure Codefresh integration from the Google Marketplace, a special Codefresh runner [is installed](https://github.com/codefresh-io/google-marketplace-integration){:target="\_blank"} in your own cluster. + +{% include image.html +lightbox="true" +file="/images/integrations/google-marketplace/architecture.png" +url="/images/integrations/google-marketplace/architecture.png" +max-width="80%" +caption="Google Marketplace integration" +alt="Google Marketplace integration" +%} + +The Codefresh UI is still hosted by Codefresh in a SAAS manner. The builds themselves however +run inside your own cluster. + +The builder is responsible for executing all your builds and notifying the Codefresh UI of their status. You can also access internal cluster resources that are normally not accessible to the SAAS hosted Codefresh builders. + +You can still run builds in the Codefresh SAAS infrastructure if you wish, and therefore both approaches are valid at the same time. + +## Usage and billing + +To start using the service, you need to [enable billing](https://cloud.google.com/billing/docs/how-to/modify-project){:target="\_blank"} in your Google Cloud account. Once that is done, Codefresh billing is integrated into your Google invoices. + +You will pay for the cluster resources to Google, plus the Codefresh builds. Codefresh does not collect any payment from you directly. Google Cloud will invoice you for both the cluster infrastructure and the cluster usage. + +Current pricing for Codefresh builds is always shown in the [marketplace page](https://console.cloud.google.com/marketplace/details/codefresh-gke/codefresh){:target="\_blank"}. + +## Install the Google Marketplace + +### Step 1: Create a Codefresh API key + +1. Log in to your Codefresh account, and from your avatar dropdown, select [**User Settings**](https://g.codefresh.io/user/settings){:target="\_blank"}. +1. Scroll down to **API Keys**. +1. To create a new API key, click **Generate**, and do the following: + * **Key Name**: Enter the name of the key, preferable one that will help you remember its purpose. The token is tied to your Codefresh account and should be considered sensitive information. + * **Scopes**: Select _all_ the scopes. +1. Copy the token to your clipboard. +1. Click **Create**. + +{% include image.html +lightbox="true" +file="/images/integrations/google-marketplace/generate-token.png" +url="/images/integrations/google-marketplace/generate-token.png" +max-width="40%" +caption="Generating a Codefresh API token" +alt="Generating a Codefresh API token" +%} + + +With the token at hand, we can go to the Google marketplace. + +### Step 2: Install the Codefresh application in your Google Cloud cluster + +1. Navigate to [https://console.cloud.google.com/marketplace/details/codefresh-gke/codefresh](https://console.cloud.google.com/marketplace/details/codefresh-gke/codefresh){:target="\_blank"}. +1. From the dropdown menu at the top of the page, select a Google project that has billing enabled. +1. Click **Configure**. + +{% include image.html +lightbox="true" +file="/images/integrations/google-marketplace/configure-plan.png" +url="/images/integrations/google-marketplace/configure-plan.png" +max-width="50%" +caption="Installing the Codefresh application" +alt="Installing the Codefresh application" +%} + +{:start="4"} +1. Define the general settings for the installation. + These include: + * The cluster that will be used for installation + * An existing or new namespace where the Codefresh builder will reside + * A name for your installation (arbitrary choice) + * Your Codefresh API token that you created in the previous section + * The selection of the account that will be used for the cluster management. + +{% include image.html +lightbox="true" +file="/images/integrations/google-marketplace/settings.png" +url="/images/integrations/google-marketplace/settings.png" +max-width="50%" +caption="Codefresh installation settings" +alt="Codefresh installation settings" +%} + +{:start="5"} +1. Note down the namespace you used, as it becomes important later on inside the Codefresh UI. +1. After defining all the settings, to install, click **Deploy**. Wait for a few minutes for the installation complete. + +{% include image.html +lightbox="true" +file="/images/integrations/google-marketplace/deploying.png" +url="/images/integrations/google-marketplace/deploying.png" +max-width="50%" +caption="Deploying the Codefresh application" +alt="Deploying the Codefresh application" +%} + +The Codefresh application is now installed on your cluster. + +### Step 3: Set up communication with Codefresh SaaS + +To finish the installation, we need to make Codefresh SaaS aware of the new builder. + +1. On the right hand-side, copy the full command to complete the installation. + +{% include image.html +lightbox="true" +file="/images/integrations/google-marketplace/run-command.png" +url="/images/integrations/google-marketplace/run-command.png" +max-width="60%" +caption="Endpoint command" +alt="Endpoint command" +%} + +This command must be executed from a shell that has `kubectl` installed with the correct configuration for the cluster that was used for the installation. If you already have a local shell that points to your cluster, feel free to paste the command there and run it. + +The easiest way to run it in any other case is via the [Google shell](https://cloud.google.com/shell/docs/){:target="\_blank"}. Click the *Activate Google shell* icon from the top right and wait a bit until the shell appears at the bottom part of the window. + +First you need to set up `kubectl` access. Run: + +`Google shell` +{% highlight shell %} +{% raw %} +gcloud container clusters list +{% endraw %} +{% endhighlight %} + +This will show you a list of your clusters. Find the one that has the Codefresh application and run: + +`Google shell` +{% highlight shell %} +{% raw %} +gcloud container clusters get-credentials [my-cluster-name] --zone=[my-cluster-zone] +{% endraw %} +{% endhighlight %} + +This will set up `kubectl` access. You can try running some command such as `kubectl get nodes` and `kubectl cluster-info` to verify that cluster communication is setup correctly. + +Then run the full command. Here is an example: + +`Google shell` +{% highlight shell %} +{% raw %} +$ APP="codefresh-kostis" NS="kostisdemo" ENDPOINT="$(kubectl cluster-info | head -1 | cut -d' ' -f6 | sed 's/\x1b[[0-9;]*m//g' | tr -d '\n' | base64)" && kubectl -n $NS get s +ecret $APP-secret -o yaml | sed -r "s/(kubeEndpoint: ).*$/\1$ENDPOINT/" | kubectl apply -f - && kubectl -n $NS delete pod -l app.kubernetes.io/name=$APP + +secret "codefresh-kostis-secret" configured +pod "codefresh-kostis-kube-agent-86dbcc67c4-9gqqb" deleted +{% endraw %} +{% endhighlight %} + + +Once the command is run, you can visit the [Codefresh Kubernetes dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) and you will see your Google Cloud cluster already configured. + +{% include image.html +lightbox="true" +file="/images/integrations/google-marketplace/cluster-details.png" +url="/images/integrations/google-marketplace/cluster-details.png" +max-width="60%" +caption="Codefresh Kubernetes dashboard" +alt="Codefresh Kubernetes dashboard" +%} + +The full integration is now ready, and you can start running Codefresh pipelines in your own cluster. + +### Step 4: Start running pipelines + +Now whenever you set up a [Codefresh pipeline]({{site.baseurl}}/docs/pipelines/pipelines/), you can choose its execution environment and point it to your own cluster with the Codefresh builder. + +>At this point if you have a Codefresh browser window open, make sure that you log out and then log in again so that the new UI options regarding your cluster become available. + +Open any Codefresh pipeline and toggle the *Run on Environment* switch. Select the cluster and the namespace that you used for the installation in step 2. + +{% include image.html +lightbox="true" +file="/images/integrations/google-marketplace/run-in-environment.png" +url="/images/integrations/google-marketplace/run-in-environment.png" +max-width="60%" +caption="Running Pipelines in your cluster" +alt="Running Pipelines in your cluster" +%} + + +You can still use the Codefresh SAAS if you don't enable this switch. You can choose which pipelines +run in Codefresh SAAS and which use your cluster depending on your needs. + + +## Alternative installation from the command line + +Instead of installing via the Google Cloud console, you can also install the Codefresh application using command line procedures. + +For this installation mode, see the [manual installation guide](https://github.com/codefresh-io/google-marketplace-integration/blob/master/README.md){:target="\_blank"}. + +## Removing the installation + +If you want to remove the Codefresh builder from your cluster, navigate to the "Applications" page in the Google Cloud console and click the Delete button. + +{% include image.html +lightbox="true" +file="/images/integrations/google-marketplace/remove.png" +url="/images/integrations/google-marketplace/remove.png" +max-width="60%" +caption="Removing the Codefresh application" +alt="Removing the Codefresh application" +%} + +You can install the Codefresh builder again from the [marketplace](https://console.cloud.google.com/marketplace/details/codefresh-gke/codefresh){:target="\_blank"}. + +## Related articles +[Manage your cluster]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) + + + + + + + + diff --git a/_docs/integrations/hashicorp-vault.md b/_docs/integrations/hashicorp-vault.md new file mode 100644 index 000000000..24301ecd6 --- /dev/null +++ b/_docs/integrations/hashicorp-vault.md @@ -0,0 +1,77 @@ +--- +title: "HashiCorp Vault" +description: "Use secrets from Vault in Codefresh pipelines" +group: integrations +toc: true +--- + +Codefresh can use secrets from your HashiCorp Vault installation. This way you have full control over secret storage and rotation. + +>This feature is for Enterprise accounts only. + +## Prerequisites + +* Up and running Vault instance + Codefresh supports HashiCorp Cloud Platform (HCP) Vault and self-managed Vault instances that run on the cloud, as well as behind the firewall (albeit with some differences in the authentication methods). + +* [Authentication method](https://www.vaultproject.io/docs/auth){:target="\_blank"} to use. + Codefresh supports the following methods: + +{: .table .table-bordered .table-hover} +| Method | Notes | +|---|--- | +| [Username/Password](https://www.vaultproject.io/docs/auth/userpass){:target="\_blank"}|Available for SaaS and Hybrid versions | +| [Access Token](https://www.vaultproject.io/docs/auth/token){:target="\_blank"}|Available for SaaS and Hybrid versions | +| [Kubernetes](https://www.vaultproject.io/docs/auth/kubernetes){:target="\_blank"}|Only available with [Codefresh Runner installation]({{site.baseurl}}/docs/reference/behind-the-firewall/) | +| [Google Cloud Engine](https://www.vaultproject.io/docs/auth/gcp){:target="\_blank"}|Only available with [Codefresh Runner installation]({{site.baseurl}}/docs/reference/behind-the-firewall/) | +| [App Role](https://www.vaultproject.io/docs/auth/approle){:target="\_blank"}|Available for SaaS and Hybrid versions | + +## Set up HashiCorp Vault integration in the Codefresh UI + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Secret Store** and then click **Configure**. +1. From the **Add Provider** dropdown, select **Hashicorp vault**. +1. Do the following: + * **Name**: A unique name for the integration which is referenced in `codefresh.yaml`. + * If your Vault instance is behind a firewall, toggle **Vault is behind a firewall** to ON. + * To allow only Codefresh admins to change the Vault configuration, toggle **Allow access to all users** to OFF. + > The other settings are specific to your [Vault authentication](https://www.vaultproject.io/docs/auth){:target="\_blank"} method. Refer to the Vault documentation on how to get the required values. + +{% include image.html +lightbox="true" +file="/images/integrations/hashicorp-vault/hashicorp-vault.png" +url="/images/integrations/hashicorp-vault/hashicorp-vault.png" +alt="HashiCorp-vault Secret" +caption="HashiCorp-vault Secret" +max-width="80%" + %} + +{:start="5"} +1. To apply the changes, click **Save**. + + +### Set up HashiCorp Vault integration via Codefresh CLI + +You can also create Vault integrations with the [CLI](https://codefresh-io.github.io/cli/){:target="\_blank"}. + +Use the [create context command](https://codefresh-io.github.io/cli/contexts/create-context/create-secret-store-context/hashicorp-vault/){:target="\_blank"}. + +The options available are identical to the UI settings. +For example, to create an integration with user/password authentication, you would run this command: + +`codefresh create context secret-store hashicorp-vault --sharing-policy AccountAdmins -app-url --password ` + + + +### Using the HashiCorp Vault secret + +To use the Vault secrets in pipelines, see our [secrets guide]({{site.baseurl}}/docs/pipelines/secrets-store/). +Because a secret in Vault can contain multiple key-value pairs, you will need to put in the key name as well, according to the syntax `{secrets.vault-store-name.path/to/secret@key}`. + +## Related articles +[Shared Configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/) +[Git integration for pipelines]({{site.baseurl}}/docs/integrations/git-providers/) +[Kubernetes integration for pipelines]({{site.baseurl}}/docs/integrations/kubernetes/) +[Container registry integration for pipelines]({{site.baseurl}}/docs/integrations/docker-registries/) diff --git a/_docs/integrations/helm.md b/_docs/integrations/helm.md new file mode 100644 index 000000000..96700f20b --- /dev/null +++ b/_docs/integrations/helm.md @@ -0,0 +1,189 @@ +--- +title: "Helm Integration" +description: "Manage Helm releases and repositories with Codefresh pipelines" +group: integrations +toc: true +--- + +Codefresh is one of the few DevOps platforms with native support for Helm releases and deployments. +In addition to the [built-in Helm repository]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/) available to all Codefresh accounts,you can add any external Helm repository to Codefresh through integrations. + +Native support for Helm in Codefresh includes: + * A pipeline [step for deploying Helm applications]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/) + * A dashboard for your [Helm charts]({{site.baseurl}}/docs/deployments/helm/add-helm-repository/) + * A dashboard for your [Helm releases]({{site.baseurl}}/docs/deployments/helm/helm-releases-management/) + * A dashboard for [promoting Helm releases]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion/) between different environments + * A dashboard for [Helm environments]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/) + +The built-in Helm repository is production ready. You can start using Helm right away with your Codefresh account, +even if you don't have an external Helm repository. See our [quick start guide for Helm]({{site.baseurl}}/docs/quick-start/ci-quickstart/deploy-with-helm/) or the [complete Helm example]({{site.baseurl}}/docs/example-catalog/cd-examples/helm/). + +For each Helm integration, you can toggle the level of access by [non-admin users]({{site.baseurl}}/docs/administration/access-control/#users-and-administrators). + +## Set up external Helm integration + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Helm** and then click **Configure**. +1. From the **Add Helm Repository** dropdown, select the type of external Helm repository: + * [Azure Registry](#azure-registry-helm-repository-settings) + * [Azure Regsitry MI](#azure-registry-with-managed-identity-mi-helm-repository-settings) + * [Azure Registry SP](#azure-registry-with-service-principal-sp-helm-repository-settings) + * [Codefresh](#helm-repository-from-another-codefresh-account) + * [Google Cloud Storage](#google-cloud-storage-gcs-helm-repository-settings) + * [HTTP Basic Authentication](#http-basic-authentication-settings) + * [Amazon AWS S3](#amazon-aws-s3-helm-repository-settings) + + +1. To restrict access to only Codefresh admins, toggle **Allow access to all users** to OFF. + >When access is restricted, users **cannot** use the [CLI](https://codefresh-io.github.io/cli/){:target="\_blank"} or [API]({{site.baseurl}}/docs/integrations/codefresh-api/) to [programmatically access this Helm repository](https://codefresh-io.github.io/cli/contexts/){:target="\_blank"}. + Otherwise, all users from all your Codefresh teams will be able to access this Helm repository with CLI commands or API calls. + + + +### HTTP Basic Authentication settings + +You can connect to your external repository with HTTP Basic authentication. +The table below describes the settings. + +Setting|Description +---|--- +**Helm Repository Name**|The unique name of integration which is used to reference the integration in `codefresh.yaml` +**Repository URL**|The URL to the Helm repository with `http://` protocol prefix. +**Helm Repo Username**|The username to authenticate with. +**Helm Repo Password**|The password for the username provided. + +### Amazon AWS S3 Helm repository settings + +You can connect to Amazon AWS S3 Helm repository. Supply the AWS authentication credentials as you would for the AWS CLI, or the S3 plugin for Helm. For details, see [Configuring the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html){:target="\_blank"}. + +The table below describes the settings. + +Setting|Description +---|--- +**Helm Repository Name**|The unique name of integration which is used to reference the integration in `codefresh.yaml` +**Helm Repository URL**|The URL to the Helm repository in the format `s3://bucketname`. +**AWS Access Key ID**|The ID of the key with permissions to the S3 bucket. +**AWS Secret Access Key**|The Secret of the key with permissions to the S3 bucket. +**AWS Default Region**|The region where the S3 bucket is located. + + +### Google Cloud Storage (GCS) Helm repository settings + +You can connect to a Google Cloud Storage (GCS) Helm repository. Supply the GCS authentication credentials as you would for the GCloud CLI, or the GCS plugin for Helm. For details, see [Creating Service Account](https://cloud.google.com/docs/authentication/getting-started){:target="\_blank"}{:target="\_blank"}. + +The table below describes the settings. + +Setting|Description +---|--- +**Helm Repository Name**|The unique name for the Helm repository integration which is used to reference the integration in `codefresh.yaml` +**Helm Repository URL**|The URL to the Helm repository in the format `gs://bucketname`. +**Google Application Credentials JSON**|The JSON content with the credentials of the service account. + + + +### Azure Registry Helm repository settings + +**Prerequsities** +1. [Create the Helm repository](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-helm-repos){:target="\_blank"} in Azure. +1. Click **Authenticate**. +1. In the permissions dialog, to allow Codefresh to access the Azure services, click **Accept**. + +>Make sure that you are using an organizational/company Azure account, and not a personal one. We are currently working with Microsoft to improve this integration. + +**Settings** + + +{% include image.html +lightbox="true" +file="/images/integrations/helm/select-azure-helm-repository.png" +url="/images/integrations/helm/select-azure-helm-repository.png" +alt="Selecting an Azure Helm repository" +caption="Selecting an Azure Helm repository" +max-width="70%" +%} + +Setting|Description +---|--- +**Subscriptions**|Select your Azure subscription. +**Registry**|The Helm repository to connect to. + +>If you are already authenticated to Azure, and cannot find your Helm repository in the list, try revoking access, and authenticating again. + + +### Azure Registry with Service Principal (SP) Helm repository settings + +An alternative method of adding an Azure Helm repository is by using a service principal. + +**Prerequsities** +* [Create a service principal in the Azure portal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal){:target="\_blank"}. + + + +**Settings** +1. Click **Authenticate**. +1. Enter the following: + * **Client ID** + * **Tenant** + * **Client secret** + +{% include image.html +lightbox="true" +file="/images/integrations/helm/add-azure-helm-spn.png" +url="/images/integrations/helm/add-azure-helm-spn.png" +alt="Azure Service Service Principal details" +caption="Azure Service Principal details" +max-width="60%" + %} + +1.Click **Authenticate**. Assuming that the authentication is successful, you can view your available Azure registries that can be used as a Helm repository. + + + + + +### Helm repository from another Codefresh account + +You also add the private Helm repository of another Codefresh user as your integration. + +>We **don't** recommend sharing the Codefresh Helm repository between accounts. The built-in Helm repository of each account is best used as a private Helm repository of that account. See more details on [how to make your private Helm public]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/#repo-access-level). + +The table below describes the settings. + +Setting|Description +---|--- +**Helm Repository Name**|The unique name for the Helm repository integration which is used to reference the integration in `codefresh.yaml` +**Helm Repository URL**|The URL to the Helm repository in the format `cm://repository-name`. +**CF API Key**|A token [to access the other Codefresh account]({{site.baseurl}}/docs/integrations/codefresh-api/#authentication-instructions). + + +## Related articles +[Private external Helm repositories]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/) +[How to use Helm in a Codefresh pipeline]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/) +[Managing Helm releases]({{site.baseurl}}/docs/deployments/helm/helm-releases-management/) +[Helm best practices]({{site.baseurl}}/docs/deployments/helm/helm-best-practices/) + diff --git a/_docs/integrations/jenkins-integration.md b/_docs/integrations/jenkins-integration.md new file mode 100644 index 000000000..157806ac8 --- /dev/null +++ b/_docs/integrations/jenkins-integration.md @@ -0,0 +1,907 @@ +--- +title: "Jenkins integration/migration" +description: "Migration from Jenkins to Codefresh pipelines" +group: integrations +redirect_from: + - /docs/jenkins-integration/ +toc: true +--- + +Codefresh offers a superset of the capabilities offered by Jenkins, and therefore you can fully replace a Jenkins solution using only Codefresh on its own. + +During the migration period, it is very easy to make both solutions work together. This allows you to move gradually new CI/CD tasks to Codefresh and still keep the existing functionality in Jenkins jobs. + +## Calling Codefresh pipelines from Jenkins Jobs + +This is the most common scenario during the migration period. The CI part (code packaging) is still in Jenkins, while the CD part (actual deployments) happen with Codefresh. + +{% include image.html +lightbox="true" +file="/images/integrations/jenkins/calling-codefresh-from-jenkins.png" +url="/images/integrations/jenkins/calling-codefresh-from-jenkins.png" +alt="Calling a Codefresh pipeline from a Jenkins Job" +caption="Calling a Codefresh pipeline from a Jenkins Job" +max-width="100%" +%} + +1. For Jenkins to connect to your Codefresh account, create [a Codefresh API token]({{site.baseurl}}/docs/integrations/codefresh-api/#authentication-instructions). +1. Enter the token in Jenkins [as a global Credential](https://jenkins.io/doc/book/using/using-credentials/){:target="\_blank"}. + +{% include image.html +lightbox="true" +file="/images/integrations/jenkins/jenkins-credentials.png" +url="/images/integrations/jenkins/jenkins-credentials.png" +alt="Storing the Codefresh API token in Jenkins" +caption="Storing the Codefresh API token in Jenkins" +max-width="60%" +%} + +Now you can create any declarative or scripted Jenkins pipeline that uses the token and the [Codefresh CLI](https://codefresh-io.github.io/cli/){:target="\_blank"} to call Codefresh pipelines from Jenkins. + +Here is a very simple example: + + `Jenkinsfile` +{% highlight groovy %} +{% raw %} +pipeline { + agent { + docker { + image 'codefresh/cli:latest' + args '--entrypoint=""' + } + } + environment { + CODEFRESH_API_TOKEN= credentials('codefresh-token') + } + stages { + stage('Calling Codefresh pipeline') { + steps { + sh 'codefresh auth create-context --api-key $CODEFRESH_API_TOKEN' + sh 'codefresh run my-first-project/basic-build -t my-trigger -b master' + } + } + } +} +{% endraw %} +{% endhighlight %} + +Run the Jenkins job, and it also triggers a Codefresh pipeline: + +{% include image.html +lightbox="true" +file="/images/integrations/jenkins/call-a-codefresh-pipeline.png" +url="/images/integrations/jenkins/call-a-codefresh-pipeline.png" +alt="Calling a Codefresh pipeline in a Jenkins step" +caption="Calling a Codefresh pipeline in a Jenkins step" +max-width="30%" +%} + +In the logs of the Jenkins job, you will see the Codefresh logs (the Codefresh CLI automatically shows logs in standard output). + +{% include image.html +lightbox="true" +file="/images/integrations/jenkins/codefresh-logs-from-jenkins.png" +url="/images/integrations/jenkins/codefresh-logs-from-jenkins.png" +alt="Viewing Codefresh logs from Jenkins" +caption="Viewing Codefresh logs from Jenkins" +max-width="60%" +%} + +Of course, if you visit the Codefresh UI you will also see the [running pipeline]({{site.baseurl}}/docs/pipelines/monitoring-pipelines/). +With this kind of integration, it is very easy to create Jenkins Jobs that compile/package code and add a step in the jobs +that calls Codefresh for deployment. + + +## Calling Jenkins jobs from Codefresh pipelines + +This is the opposite scenario. As you move more functionality into Codefresh, it might make sense to have a Codefresh pipeline that actually calls Jenkins jobs for tasks that are not migrated yet. + +{% include image.html +lightbox="true" +file="/images/integrations/jenkins/calling-jenkins-from-codefresh.png" +url="/images/integrations/jenkins/calling-jenkins-from-codefresh.png" +alt="Calling a Jenkins Job from a Codefresh pipeline" +caption="Calling a Jenkins Job from a Codefresh pipeline" +max-width="100%" +%} + +* For Codefresh to authenticate to your Jenkins instance, from **User Settings** in the Jenkins UI, create a [Jenkins API token](https://jenkins.io/blog/2018/07/02/new-api-token-system/){:target="\_blank"}. + Give your token any name that reminds you of its purpose. The name itself is arbitrary. + +{% include image.html +lightbox="true" +file="/images/integrations/jenkins/jenkins-api-token.png" +url="/images/integrations/jenkins/jenkins-api-token.png" +alt="Jenkins API token" +caption="Jenkins API token" +max-width="60%" +%} + +Once you have the token, you can use the [Codefresh plugin for triggering Jenkins Jobs](https://github.com/codefresh-io/plugins/blob/new-pipeline/plugins/run-jenkins-job/README.md){:target="\_blank"} in any pipeline +like this: + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + RunJenkins: + title: Triggering Jenkins Job + image: codefresh/cf-run-jenkins-job + environment: + - JENKINS_URL=${{JENKINS_URL}} + - JENKINS_USER=${{JENKINS_USERNAME}} + - JENKINS_TOKEN=${{JENKINS_TOKEN}} + - JENKINS_JOB=${{JENKINS_JOB}} +{% endraw %} +{% endhighlight %} + +The value of the variables can be stored either in your [Codefresh shared configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/) or directly [in the pipeline]({{site.baseurl}}/docs/pipelines/pipelines/#creating-new-pipelines): + + +{% include image.html +lightbox="true" +file="/images/integrations/jenkins/jenkins-variables.png" +url="/images/integrations/jenkins/jenkins-variables.png" +alt="Jenkins Job variables" +caption="Jenkins Jobs variables" +max-width="40%" +%} + +Launching the Codefresh pipeline also triggers the remote Jenkins Job. + + +{% include image.html +lightbox="true" +file="/images/integrations/jenkins/trigger-remote-jenkins-job.png" +url="/images/integrations/jenkins/trigger-remote-jenkins-job.png" +alt="Trigger remote Jenkins Job" +caption="Trigger remote Jenkins Job" +max-width="60%" +%} + +It is possible to mix both scenarios at the same time (Codefresh pipelines that call Jenkins Jobs and vice-versa). + +## Migrating from Jenkins to Codefresh + +Now that you know how to mix pipelines from both platforms, it is helpful to understand how you can migrate all your Jenkins Jobs to Codefresh. In most cases, several actions that require custom scripts in Jenkins (or plugins/shared libraries) are already integrated into the core Codefresh platform. Here are some high-level differences between the two platforms: + +{: .table .table-bordered .table-hover} +| Feature | Jenkins | Codefresh | +| -------------- | ---------------------------- |-------------------------| +| Architecture | VM based | [container-based]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/)| +| Pipeline definition | Groovy | [YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) | +| Tool installation | Installed on build node | dynamically launched | +| Plugin mechanism | Java/Groovy using Jenkins API | [Docker image](https://codefresh.io/steps/){:target="\_blank"} (any programming language)| +| Plugin installation | Central (requires admin access) | Per pipeline (no admin access needed) | +| Docker agent builds | Extra plugin | Built-in | +| Kubernetes agent builds | Extra plugin | Built-in | +| Docker commands | Manually run in pipelines | Built-in pipeline steps | +| Access to Kubectl | External plugin | Built-in | +| Kubernetes deployments | External plugin | Built-in | + +It is important to understand that when you are switching to Codefresh you get a set of higher level abstraction for your builds: + +* Unlike Jenkins, Codefresh automatically has a distributed fleet of build nodes and manages all builds on its own. +* With Codefresh you don't need to install anything on the build nodes (in fact you don't even have SSH access on them). All build tools are automatically launched in pipelines as Docker images. +* In Codefresh, you can use the same tools with different versions in the same pipeline, without any special configuration (for example, use Java 5 and Java 8 in the same pipeline). +* Codefresh plugins are used per pipeline by simply defining them. There is nothing to install centrally (such as Jenkins plugins or shared libraries). Different teams can use different tools on their pipeline without affecting each other. +* Codefresh plugins are just Docker images with predefined inputs/outputs. They can be programmed in any programming language (not just Java/Groovy) and are not tied to Codefresh in any way (i.e. there is no need to know the Codefresh API for writing a Codefresh plugin). +* Jenkins pipelines can be free-style (VM based), scripted (VM/container-based) or declarative (VM/container based) meaning that there are at least 5 ways on how you can write your pipeline. In Codefresh there is only way (declarative/container-based). +* Jenkins pipelines are connected to a single git repository. Codefresh pipelines can be connected to multiple [git triggers]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/) which +themselves are connected to git repositories. Therefore a Codefresh pipeline can be reused for multiple projects. +* Specifically for building Docker images, Codefresh can automatically connect to any [external Docker registry]({{site.baseurl}}/docs/docker-registries/external-docker-registries/). +* Specifically for Kubernetes deployments, Codefresh automatically sets up `kubectl` access in pipelines [from connected clusters]({{site.baseurl}}/docs/integrations/kubernetes/#add-kubernetes-cluster/). There is no configuration needed to achieve this behavior. Codefresh also has several [built-in ways for Kubernetes deployments]({{site.baseurl}}/docs/deployments/kubernetes/deployment-options-to-kubernetes/) and a [dedicated UI dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) to see what your cluster is doing. +* Specifically for Helm deployments, Codefresh includes a private Helm repository and several [Helm dashboards]({{site.baseurl}}/docs/example-catalog/cd-examples/helm/). + + +### Migrating Jenkins freestyle jobs + +If you have freestyle Jenkins Jobs (or are still using Jenkins 1.x), it is very easy to migrate your builds to Codefresh. +In Jenkins, you are accustomed to: + +1. Installing a programming tool on the Jenkins node. +1. Calling it directly in a build step. + +In Codefresh, a similar process would be the following: + +1. Find a Docker image in Docker Hub or create one by yourself that has the tools that you need. +1. Use [a freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/), and run the exact same command in a Codefresh pipeline. + +For example this Jenkins job... + +{% include image.html +lightbox="true" +file="/images/integrations/jenkins/jenkins-freestyle-job.png" +url="/images/integrations/jenkins/jenkins-freestyle-job.png" +alt="Jenkins freestyle job" +caption="Jenkins freestyle job" +max-width="60%" +%} + + +...can be easily converted to a Codefresh pipeline like this: + +`codefresh.yml` +{% highlight yaml %} +version: '1.0' +steps: + my_jar_compilation: + title: Compile/Unit test + image: maven:3.5.2-jdk-8-alpine + commands: + - mvn package + my_node_app: + title: Running unit tests + image: node:11 + commands: + - npm run test +{% endhighlight %} + +Unlike Jenkins, Codefresh does **not** need any global installation of tools beforehand. + +{% include image.html +lightbox="true" +file="/images/integrations/jenkins/jenkins-tool-installation.png" +url="/images/integrations/jenkins/jenkins-tool-installation.png" +alt="Jenkins Tool installation - not needed with Codefresh" +caption="Jenkins Tool installation - not needed with Codefresh" +max-width="40%" +%} + +In Codefresh, you can just use one or more Docker images in your pipeline. The tool versions will be launched only while the pipeline is active. Once the pipeline is finished, all Docker images that took part in it are discarded. The Codefresh build node has only Docker installed and nothing else. +This means that you can easily mix and match tool versions. + +Here is a Codefresh pipeline that uses multiple versions of Java and Node. + +`codefresh.yml` +{% highlight yaml %} +version: '1.0' +steps: + PackageMyNode1App: + title: Packaging Node application 1 + stage: packaging + image: node:11.1 + working_directory: ./dashboard + commands: + - echo "My Node version is" + - node --version + - npm install + PackageMyNode2App: + title: Packaging Node application 2 + stage: packaging + image: node:9.3.0-slim + working_directory: ./website + commands: + - echo "My Node version is" + - node --version + - npm install + RunUnitTests: + title: Running Unit tests + image: maven:3.6.1-jdk-11 + working_directory: ./backend + commands: + - java -version + - mvn test + PackageBackend: + title: Compile/Unit test + image: maven:3.5.2-jdk-8-alpine + working_directory: ./backend + commands: + - java -version + - mvn package -Dmaven.test.skip +{% endhighlight %} + +Meanwhile, another team might have a different pipeline using different versions of Maven and/or Java. Each team can decide on the exact version needed just by changing the [Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/). + +It should be easy to see now that by migrating from Jenkins to Codefresh, several problems in Jenkins are simply eliminated: + +* You do not need to be an admin to install programming tools anymore, +* Tools are defined per pipeline instead of being preloaded centrally, +* Multiple versions of the same tool can be used on the same pipeline (or different pipelines), +* Upgrading a tool to a new version is trivial (just change the docker tag in the freestyle step), +* You don't need to label build nodes anymore with specific labels that show which tools they contain, +* All public Dockerhub images can be used *with zero changes* in a Codefresh step. + +Notice that for several popular tools, Dockerhub already contains several images. + +* [Maven](https://hub.docker.com/_/maven){:target="\_blank"} +* [Gradle](https://hub.docker.com/_/gradle/){:target="\_blank"} +* [Node](https://hub.docker.com/_/node/){:target="\_blank"} +* [Python](https://hub.docker.com/_/python/){:target="\_blank"} +* [Terraform](https://hub.docker.com/r/hashicorp/terraform/){:target="\_blank"} +* [Packer](https://hub.docker.com/r/hashicorp/packer/){:target="\_blank"} +* [Sonar](https://hub.docker.com/r/skilldlabs/sonar-scanner/){:target="\_blank"} +* [Nexus](https://hub.docker.com/r/sjeandeaux/nexus-cli/){:target="\_blank"} +* [Helm](https://hub.docker.com/r/codefresh/kube-helm/tags){:target="\_blank"} +* [Kubectl](https://hub.docker.com/r/codefresh/kubectl/){:target="\_blank"} +* [gcloud](https://hub.docker.com/r/google/cloud-sdk/){:target="\_blank"} +* [Ansible](https://hub.docker.com/u/ansible){:target="\_blank"} +* [Azure CLI](https://hub.docker.com/r/microsoft/azure-cli/){:target="\_blank"} +* [AWS CLI](https://hub.docker.com/r/mesosphere/aws-cli/){:target="\_blank"} + +Of course, you can create your own Docker image with the exact tools that you want and then use it from the [any Docker registry]({{site.baseurl}}/docs/docker-registries/external-docker-registries/) or any other registry in your pipeline. + +### Migrating Jenkins pipelines + +In the case of Jenkins pipelines, things are a bit more complicated because there is not a single way anymore on how to structure your pipelines. +First of all, the best-case scenario is when you have *declarative* Jenkins pipelines that already use Docker images for stage execution. + + + `Jenkinsfile` +{% highlight groovy %} +{% raw %} +pipeline { + agent none + stages { + stage('Example Build') { + agent { docker 'maven:3-alpine' } + steps { + echo 'Hello, Maven' + sh 'mvn --version' + } + } + stage('Example Test') { + agent { docker 'openjdk:8-jre' } + steps { + echo 'Hello, JDK' + sh 'java -version' + } + } + } +} +{% endraw %} +{% endhighlight %} + +In this case, there is a 1-1 mapping between Jenkins stages and Codefresh steps as you can simply convert each stage into a Codefresh step using the respective Docker image. + +The Jenkins pipeline above can be converted to a Codefresh pipeline with a series of freestyle steps: + +`codefresh.yml` +{% highlight yaml %} +version: '1.0' +steps: + my_first_step: + title: Example Build + image: maven:3-alpine + commands: + - echo 'Hello, Maven' + - mvn --version + my_second_step: + title: Example Test + image: openjdk:8-jre + commands: + - echo 'Hello, JDK' + - java -version +{% endhighlight %} + +The final Codefresh pipeline will even look like the original Jenkins one. + +{% include image.html +lightbox="true" +file="/images/integrations/jenkins/migrate-jenkins-pipeline.png" +url="/images/integrations/jenkins/migrate-jenkins-pipeline.png" +alt="Direct migration from Jenkins to Codefresh" +caption="Direct migration from Jenkins to Codefresh" +max-width="80%" +%} + +If you don't use Docker containers in your Jenkins pipeline, then you need to follow the same advice as the previous section (i.e. find a Docker image that has the tools you need and create your own freestyle step in Codefresh). + +### Checking out source code + +In Jenkins 1.x pipelines, code is automatically checked out by Jenkins when the pipeline starts. In Jenkins 2.x pipelines you are free to insert your own git steps inside a Job: + + + `Jenkinsfile` +{% highlight groovy %} +{% raw %} +node { + stage('First Project') { + git 'https://github.com/nodegui/react-nodegui.git' + } + stage('Second Project') { + git url: 'https://github.com/jglick/simple-maven-project-with-tests.git' + branch: 'master' + } +} +{% endraw %} +{% endhighlight %} + +Codefresh has a dedicated [git clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/) that can be used in a similar manner. + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + first_project: + type: 'git-clone' + description: 'Cloning first project...' + repo: 'nodegui/react-nodegui' + revision: '${{CF_BRANCH}}' + git: github + second_project: + type: 'git-clone' + description: 'Cloning second...' + repo: 'jglick/simple-maven-project-with-tests' + revision: 'master' + git: github +{% endraw %} +{% endhighlight %} + +You don't need to define any credentials or tokens, as they are already defined centrally in the [git configuration screen]({{site.baseurl}}/docs/integrations/git-providers/). The `CF_BRANCH` variable is one of the [built-in Codefresh variables]({{site.baseurl}}/docs/pipelines/variables/) that shows the branch that was used by the git commit as it came from the [trigger]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/) attached to the pipeline. + +You can also [manually run Git commands]({{site.baseurl}}/docs/yaml-examples/examples/git-checkout-custom/) in Codefresh pipelines. + +### Step conditions + +In several cases you want to add conditionals on steps such as the branch that is being compiled: + + + `Jenkinsfile` +{% highlight groovy %} +{% raw %} +pipeline { + agent any + stages { + stage('Example Build') { + steps { + echo 'Hello World' + } + } + stage('Example Deploy') { + when { + branch 'master' + } + steps { + echo 'Deploying' + } + } + } +} +{% endraw %} +{% endhighlight %} + +This can be also configured in Codefresh using [step conditionals]({{site.baseurl}}/docs/pipelines/conditional-execution-of-steps/): + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + my_first_step: + title: Example Build + image: alpine:latest + commands: + - echo 'Hello World' + my_second_step: + title: Example Deploy + image: alpine:latest + commands: + - echo 'Deploying' + when: + branch: + only: + - master +{% endraw %} +{% endhighlight %} + +You can define much more complex conditions using the [Codefresh expression language]({{site.baseurl}}/docs/pipelines/conditional-execution-of-steps/#condition-expression-syntax). + + +### Migrating Jenkins credentials + +Codefresh contains a central repository for user variables and secrets in the form of [shared configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/). You can also +inject variables on a specific project or a specific pipeline. + +All injected variables *are automatically available to all Codefresh freestyle steps*. You don't need a special syntax or directive to enable this behavior (unlike Jenkins where you have to use a `withCredentials` block or something similar). + +In Jenkins you have to explicitly ask for a secret: + + `Jenkinsfile` +{% highlight groovy %} +{% raw %} +pipeline { + agent any + stages { + stage('Example') { + environment { + AWS_ACCESS_KEY_ID = credentials('jenkins-aws-secret-key-id') + AWS_SECRET_ACCESS_KEY = credentials('jenkins-aws-secret-access-key') + } + steps { + sh 'printenv' + } + } + } +} +{% endraw %} +{% endhighlight %} + +In Codefresh if you setup your variables in the pipeline settings, then the pipeline itself needs nothing special. + +{% include image.html +lightbox="true" +file="/images/integrations/jenkins/pipeline-variables.png" +url="/images/integrations/jenkins/pipeline-variables.png" +alt="Pipeline variables" +caption="Pipeline variables" +max-width="80%" +%} + +You can simply run the pipeline on its own: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + my_first_step: + title: Example1 + image: alpine:latest + commands: + - printenv # Injected variables are always available + my_second_step: + title: Example2 + image: alpine:latest + commands: + - echo $AWS_ACCESS_KEY_ID + - echo $AWS_SECRET_ACCESS_KEY +{% endraw %} +{% endhighlight %} + +If you want to use a different name for each variable/secret then you can simply assign them to your desired names: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + my_first_step: + title: Example + image: alpine:latest + environment: + - MY_KEY=${{AWS_ACCESS_KEY_ID}} + - MY_ACCESS_KEY=${{AWS_SECRET_ACCESS_KEY}} + commands: + - echo $MY_KEY + - echo $MY_ACCESS_KEY +{% endraw %} +{% endhighlight %} + +In the example above, even though the secrets are already available as environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, we instead pass them to the pipeline step as `MY_KEY` and `MY_ACCESS_KEY`. + + +### Migrating Jenkins shared libraries + +Creating a Jenkins shared library has a lot of challenges: + +* The library must be written in Groovy/Java, +* The library must use the Jenkins API, +* It is very hard to write unit tests for it, +* Installing it requires admin access in the Jenkins master. + +Codefresh plugins, on the other hand, are just Docker images written in any programming language. + +First of all, look at [Docker Hub](https://hub.docker.com/){:target="\_blank"} and see if there is already a utility or CLI that has the same functionality with your +shared library. Codefresh also has a [free marketplace](https://codefresh.io/steps/){:target="\_blank"} for pipeline steps (which are Docker images essentially). + +As a last resort, you need to rewrite your shared library and convert it to a Docker image. The process is the following: + +1. Start from a base image that contains Groovy and any other tool you need. +1. Convert your shared library to a single Groovy executable that reads input from environment variables and/or files and writes output data to files. +1. Remove all Jenkins specific APIs. +1. Package the image with a simple Dockerfile that just compiles your executable. + +Note that there is nothing Codefresh specific to the end-result. You should end up with a standard Docker image that would be usable in any environment that has Docker installed. + +Once you have that image you can use it like any other Codefresh freestyle step as described in the previous section. + +### Migration of Jenkins pipelines that create Docker images + +Codefresh has native support for: + +1. Building Docker images +1. Running commands inside Docker images +1. Pushing Docker images to different registries + +If you are using Docker commands directly in your Jenkins file, or prefer to take advantage of the scripted +variant of Docker image management then you can easily convert both approaches to Codefresh YAML like below: + +#### Building Docker images + +The most basic Docker operation is building an image. You will need a Dockerfile and a directory to use as build context (usually the same folder that contains the Dockerfile). + +`docker command` +``` +docker build . -t my-app-image:1.0.1 +``` + +Or if you use Jenkins scripted pipelines... + + `Jenkinsfile` +{% highlight groovy %} +{% raw %} +node { + docker.build("my-app-image:1.0.1") + docker.build("test-image", "./tests") +} +{% endraw %} +{% endhighlight %} + +...they will become in Codefresh the following [build steps]({{site.baseurl}}/docs/pipelines/steps/build/): + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + my_first_step: + title: Building My App image + type: build + image: my-app-image + tag: 1.0.1 + my_second_step: + title: Building My Test image + type: build + image: test-image + working_directory: "./tests" +{% endraw %} +{% endhighlight %} + +#### Running commands in created containers + +Sometimes you want to run commands inside a Docker image. Either to have access to specialized tools or to run tests. + +`docker command` +``` +docker build . -t my-app-image:1.0.1 +docker run my-app-image:1.0.1 npm install +``` + +Or, if you use Jenkins scripted pipelines... + + `Jenkinsfile` +{% highlight groovy %} +{% raw %} +node { + def customImage = docker.build("my-app-image:1.0.1") + + customImage.inside { + sh 'npm install' + } +} +{% endraw %} +{% endhighlight %} + +... they will become in Codefresh the following [freestyle steps]({{site.baseurl}}/docs/pipelines/steps/freestyle/). + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + my_first_step: + title: Building My App image + type: build + image: my-app-image + tag: 1.0.1 + my_second_step: + title: Install Node dependencies + image: ${{my_first_step}} + commands: + - npm install +{% endraw %} +{% endhighlight %} + +Notice that the second pipeline step actually mentions the first one by name as a running context. + +#### Pushing Docker images + +Notice that in Codefresh [all connected registries]({{site.baseurl}}/docs/docker-registries/external-docker-registries/) are automatically available to all pipelines. +You don't need special directives such as `withRegistry`. All registries can be mentioned +by their name in any push step (and will be used automatically for pulls when an image uses their domain). + +`docker command` +``` +docker build . -t my-app-image:1.0.1 +docker tag my-app-image:1.0.1 registry.example.com/my-app-image:1.0.1 +docker push registry.example.com/my-app-image:1.0.1 +``` + +Or, if you use Jenkins scripted pipelines... + + `Jenkinsfile` +{% highlight groovy %} +{% raw %} +node { + def customImage = docker.build("my-app-image:1.0.1") + + docker.withRegistry('https://registry.example.com', 'example-registry-credentials') { + customImage.push("1.0.1") + customImage.push("latest") + } +} +{% endraw %} +{% endhighlight %} + +...they will become in Codefresh the following [push steps]({{site.baseurl}}/docs/pipelines/steps/push/). + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + my_first_step: + title: Building My App image + type: build + image: my-app-image + tag: 1.0.1 + my_second_step: + title: Pushing image + type: push + candidate: ${{my_first_step}} + tags: + - 1.0.1 + - latest + registry: example-registry +{% endraw %} +{% endhighlight %} + +Notice again that the second step pushes the image created by the first one. + +##### Complete Docker pipeline + +Here is a full example with a pipeline that builds an image, runs tests, and pushes it to Dockerhub. + + `Jenkinsfile` +{% highlight groovy %} +{% raw %} +node { + def app + + stage('Clone repository') { + checkout scm + } + + stage('Build image') { + app = docker.build("my-app-image:1.0.1") + } + + stage('Test image') { + app.inside { + sh 'npm run test' + } + } + + stage('Push image') { + docker.withRegistry('https://registry.hub.docker.com', 'docker-hub-credentials') { + app.push("1.0.1") + app.push("latest") + } + } +} +{% endraw %} +{% endhighlight %} + +Here is the same pipeline in Codefresh: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + main_clone: + type: "git-clone" + description: "Clone repository" + repo: "my-account/my-git-repo" + revision: "${{CF_BRANCH}}" + git: github + my_first_step: + title: Build image + type: build + image: my-app-image + tag: 1.0.1 + my_second_step: + title: Test image + image: ${{my_first_step}} + commands: + - npm run test + my_third_step: + title: Push image + type: push + candidate: ${{my_first_step}} + tags: + - 1.0.1 + - latest + registry: dockerhub +{% endraw %} +{% endhighlight %} + +Notice that Codefresh has much more context regarding Docker registries and credentials. The same approach +is followed with Kubernetes deployments as we will see in the next section. + + +### Migration of Jenkins pipelines that deploy to Kubernetes + +Codefresh has first-class support for Kubernetes deployments. Codefresh can deploy on its own [using different options]({{site.baseurl}}/docs/deploy-to-kubernetes/deployment-options-to-kubernetes/) and no external tools (i.e. Ansible or `kubectl`) are needed. + +Specifically for [Kubernetes]({{site.baseurl}}/docs/deploy-to-kubernetes/deployment-options-to-kubernetes/) and [Helm]({{site.baseurl}}/docs/new-helm/using-helm-in-codefresh-pipeline/), Codefresh has declarative pipeline steps: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + RunningDeploy: + title: Kubernetes Deployment + type: deploy + kind: kubernetes + cluster: myDemoGKEcluster + namespace: production + service: my-service + candidate: + image: 'my-app-image:1.0.1' + registry: 'dockerhub' + DeployMyChart: + image: 'codefresh/cfstep-helm:2.9.1' + environment: + - CHART_REF=charts/python + - RELEASE_NAME=mypython-chart-prod + - KUBE_CONTEXT=myDemoAKSCluster +{% endraw %} +{% endhighlight %} + +As with Docker registries (described in the previous section), Codefresh makes all [added Kubernetes clusters]({{site.baseurl}}/docs/integrations/kubernetes/#connect-a-kubernetes-cluster) available to all pipelines. You don't need any special plugin or directive (such as `withKubeConfig`) to work with Kubernetes clusters in Codefresh. You can see that the Codefresh pipeline simply mentions Kubernetes clusters and registries without any credential information. + +Of course, it is also very easy to convert any existing Jenkins pipeline by just using any image that contains the `kubectl` executable. + + `Jenkinsfile` +{% highlight groovy %} +{% raw %} +node { + stage('Apply Kubernetes files') { + withKubeConfig([credentialsId: 'user1', serverUrl: 'https://api.k8s.my-company.com']) { + sh 'kubectl apply -f my-kubernetes-directory' + } + } +} +{% endraw %} +{% endhighlight %} + +Codefresh will automatically setup Kube config access to the pipeline behind the scenes. Zero configuration is needed for this behavior. + + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + MyCustomKubectlCommands: + title: Running Kubectl + image: codefresh/kubectl + commands: + - kubectl config use-context "my-k8s-my-company" + - kubectl apply -f my-kubernetes-directory +{% endraw %} +{% endhighlight %} + +Once you use Codefresh for your deployments you also get access to: + +* The [Kubernetes Dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) +* A free [built-in Helm repository]({{site.baseurl}}/docs/deployments/helm/managed-helm-repository/) with each Codefresh account +* The [Helm chart dashboard]({{site.baseurl}}/docs/deployments/helm/add-helm-repository/) +* The [Helm Release dashboard]({{site.baseurl}}/docs/deployments/helm/helm-releases-management/) +* The [Helm environment dashboard]({{site.baseurl}}/docs/deployments/helm/helm-environment-promotion/) + +For some easy templating see also the [cf-deploy]({{site.baseurl}}/docs/deployments/ci-cd-guides/kubernetes-templating/) plugin. + +## Related articles +[Codefresh API]({{site.baseurl}}/docs/integrations/codefresh-api/) +[Codefresh CLI](https://codefresh-io.github.io/cli/){:target="\_blank"} +[Creating Codefresh pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) + + + + diff --git a/_docs/integrations/jira.md b/_docs/integrations/jira.md new file mode 100644 index 000000000..34bb27b66 --- /dev/null +++ b/_docs/integrations/jira.md @@ -0,0 +1,133 @@ +--- +title: "Jira pipeline integration" +description: "" +group: integrations +redirect_from: + - /docs/jira-integration-1/ + - /docs/integrations/jira-integration-1/ +toc: true +--- +Codefresh integrates with Jira in several ways. This article describes how to integrate with Jira in Codefresh for the highest visibility into your GitOps deployments. +Alternatively, you can connect to Jira: +* Through a [custom step]({{site.baseurl}}/docs/integrations/notifications/jira-integration/#use-jira-within-your-codefresh-pipeline) from our step marketplace so that you can connect your pipelines with Jira +* By using your own [jira-cli]({{site.baseurl}}/docs/integrations/notifications/jira-integration/#using-your-own-jira-cli) + +## Prerequisites +* [Codefresh Account]({{site.baseurl}}/docs/administration/create-a-codefresh-account/) +* [Jira Account](https://www.atlassian.com/software/jira){:target="\_blank"} + +## Set up Jira integration in Codefresh + + +When you add a new Jira integration in Codefresh, you can authenticate either using the: +* [Codefresh Marketplace App]({{site.baseurl}}/docs/integrations/notifications/jira-integration/#authenticate-with-the-jira-client-key) + We recommended setting up your Jira integration through our Marketplace App. + > Note that Codefresh currently has to provide you with access to use the Jira Marketplace App. Please get in touch for more information. + +* [Jira Account Details]({{site.baseurl}}/docs/integrations/notifications/jira-integration/#provide-account-details) + +
+ +**Before you begin** + +* To authenticate through the Marketplace App: + * Get the [Organization URL and the Client Key for the Codfresh App from Jira](#jira-integration-with-marketplace-app) + +**How to** + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Atlassian Jira** and then click **Configure**. +1. Click **Add Jira**. +1. In the **Integration Name** field, enter a name for the integration which is used to reference it in `codefresh.yaml`. +1. To restrict access to only Codefresh admins, toggle **Allow access to all users** to OFF. + +1. To integrate with the Codefresh Marketplace App, click **Jira Marketplace App**: + * Paste the **Organization URL** and the **Client Key** you generated for the Codefresh App in Jira. +1. To integrate with your Jira account details, select **User/Pass**, and define the following: + * **Jira URL**: The URL of your organization, for example, `https://company-name.atlassian.net’. + * **Username**: Your Jira username, usually the e-mail with which you are logged in to Jira. + * **Password**: Your Jira password, or alternatively, the Jira Client Key. + +{% include image.html +lightbox="true" +file="/images/integrations/jira/add-jira-password.png" +url="/images/integrations/jira/add-jira-password.png" +alt="Account Information" +max-width="90%" +%} + +{:start="7"} +1. For integration with Jira Marketplace App, to verify the connection details, click **Test Connection**. +1. To apply the changes, click **Save**. + + + + +## Jira integration with Marketplace App +You need to generate the Organization URL and Client Key for the Codefresh application in Atlassian Jira. + + +1. In the Atlassian Marketplace, go to the [Codefresh Application](https://marketplace.atlassian.com/apps/1224560/codefresh){:target="\_blank"}. + {% include image.html + lightbox="true" + file="/images/integrations/jira/add-app.png" + url="/images/integrations/jira/add-app.png" + alt="Add Codefresh from Jira App Marketplace" + caption="Add Codefresh from Jira App Marketplace" + max-width="90%" + %} + +{:start="2"} +1. To install the application, click **Get it now**. When prompted, confirm the installation. + {% include image.html + lightbox="true" + file="/images/integrations/jira/confirm.png" + url="/images/integrations/jira/confirm.png" + alt="Confirm installation" + caption="Confirm installation" + max-width="90%" + %} + +{:start="3"} +1. When the installation has completed, in your Jira account, go to the **Apps** menu. +1. Click **Manage your apps**. + + {% include image.html + lightbox="true" + file="/images/integrations/jira/manage-apps.png" + url="/images/integrations/jira/manage-apps.png" + alt="Select Manage Apps within Your Jira Account" + caption="Select Manage Apps within Your Jira Account" + max-width="90%" + %} + +{:start="5"} +1. In **User-installed apps**, locate the Codefresh CI/CD platform integration. +1. Click **Configure**. + This will provide you with your Organization URL and the Client Key. + + {% include image.html + lightbox="true" + file="/images/integrations/jira/configure.png" + url="/images/integrations/jira/configure.png" + alt="Account information" + caption="Account information" + max-width="90%" + %} + + +{:start="7"} +1. Copy **Organization URL** and the **Client Key**. You will need these to set up Jira integration with the Codefresh Marketplace App. + + + + +## Using the Jira Integration + +Once Jira is connected to your Codefresh account, you can use both platforms in combination and integrate Jira into your [GitOps workflow]({{site.baseurl}}/docs/ci-cd-guides/gitops-deployments/). + +## Related articles +[Example for sending notifications to Jira]({{site.baseurl}}/docs/example-catalog/ci-examples/sending-the-notification-to-jira/) +[Examples for Codefresh pipelines]({{site.baseurl}}/docs/example-catalog/examples/) +[Create a pipeline]({{site.baseurl}}/docs/pipelines/pipelines/) \ No newline at end of file diff --git a/_docs/integrations/kubernetes.md b/_docs/integrations/kubernetes.md new file mode 100644 index 000000000..364c17511 --- /dev/null +++ b/_docs/integrations/kubernetes.md @@ -0,0 +1,741 @@ +--- +title: "Kubernetes pipeline integration" +description: "How Codefresh supports Kubernetes clusters" +group: integrations +toc: true +--- + +Codefresh is one of the few CI/CD solutions that has native support for Kubernetes clusters, not only for deploying applications to Kubernetes, but also for running pipelines on Kubernetes. + +Codefresh has native support for Kubernetes in the following areas: + +- [Connecting a cluster globally](#connect-a-kubernetes-cluster) +- [Viewing the cluster status]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) +- [Viewing the environment dashboard]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/) +- [Deploying to a cluster with the GUI]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/#deploying-a-new-service) +- [Deploying to a cluster with a pipeline]({{site.baseurl}}/docs/deployments/kubernetes/deployment-options-to-kubernetes/) +- [Running pipelines on a cluster]({{site.baseurl}}/docs/installation/codefresh-runner/) + + + + +Codefresh offers its own Kubernetes dashboard that allows you to inspect the services and namespaces +in your cluster. To activate this dashboard, you need to connect your cluster to your Codefresh account first. + +## Connect a Kubernetes cluster + +### Prerequisites + +Codefresh SaaS needs network connectivity to connect to your cluster. +If your cluster is behind a firewall, make sure that you allow access to the [required IPs]({{site.baseurl}}/docs/administration/platform-ip-addresses/). + + +For On-premises and [Hybrid installations]({{site.baseurl}}/docs/administration/behind-the-firewall/), there is no need to tamper with your firewall. + +### Set up Kubernetes integration + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Kubernetes** and then click **Configure**. +1. From the **Add Provider** list, select the Kubernetes provider. + +{% include image.html + lightbox="true" + file="/images/integrations/codefresh-integrations.png" + url="/images/integrations/codefresh-integrations.png" + alt="Codefresh integrations" + caption="Codefresh integrations" + max-width="70%" + %} + + + +#### Adding a GKE Cluster +Adding a cluster in GKE can be done by clicking the **Add cluster** button under **Google Cloud Provider** and selecting the desired project and cluster. + +If this is your first time, you'll be prompted to authenticate using your Google credentials, make sure you're doing so with a user that have access to your GKE projects. + +For GKE cluster versions >=1.19 basic authentication is deprecated. You can add the GKE cluster manually by [using the custom Kubernetes integration option]({{site.baseurl}}/docs/deploy-to-kubernetes/add-kubernetes-cluster/#adding-any-other-cluster-type-not-dependent-on-any-provider) instead. + +{{site.data.callout.callout_info}} + +If you are a new customer of Google Cloud, you are also eligible to receive a Codefresh offer to get up to $500 in Google credits. As soon at the GKE integration is complete within Codefresh, you will get an email with extra details on how to claim your credits. + +Follow the link in the email to fill in an application for the free credits. Once Google approves the application (usually within 1-2 days) your credits will be available to your account. Make sure to check your spam folder for that email. + +{{site.data.callout.end}} + +{:.text-secondary} + +#### Adding an AKS cluster + +To add an Azure cluster, select *Azure AKS* from the drop-down menu instead of *Azure AKS SP*. Click the *Authenticate button* and enter your Azure credentials. You will see a description of all permissions that Codefresh needs +in order to access your cluster. Accept them and Codefresh will connect to Azure to get the cluster information. + +>If you experience difficulties at this point try logging into Azure first in your browser *before* clicking +the authenticate button. Also make sure that you are using an organizational/company Azure account and not a personal one. We are currently working with Microsoft to improve this integration. + +If everything is ready you will see a dialog that allows you to select your Azure subscription and the +cluster name that you wish to use. + +{% include image.html +lightbox="true" +file="/images/integrations/kubernetes/add-cluster/select-aks-cluster.png" +url="/images//integrations/kubernetes/add-cluster/select-aks-cluster.png" +alt="Selecting the Azure cluster" +caption="Selecting the Azure cluster" +max-width="60%" + %} + +Codefresh will query the cluster and show its nodes. You are now ready to [deploy to Azure kubernetes]({{site.baseurl}}/docs/getting-started/deployment-to-kubernetes-quick-start-guide/). + +>If you wish for any reason to revoke the granted access from the Azure side, visit [https://account.activedirectory.windowsazure.com/r#/applications](https://account.activedirectory.windowsazure.com/r#/applications) and remove "Codefresh" from the list. + +#### Adding an AKS cluster with a service principal + +An alternative method of adding an Azure cluster is by using a service principal (*Azure AKS SP*). + + +**Before you begin** +* Follow the [instructions for creating a service principal in the Azure portal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal){:target="\_blank"}. + +**How to** + +1. From the **Add Provider** list, select the **Azure AKS SP**. +1. Click the arrow on the right, and then click **Add Cluster**. +1. Enter the following: + * `Client ID` + * `Tenant` + * `Client secret` + +1. Click **Authenticate**. + +{% include image.html +lightbox="true" +file="/images/kubernetes/integrations/add-cluster/connect-azure-spn.png" +url="/images/kubernetes/integrations/add-cluster/connect-azure-spn.png" +alt="Azure Service principal details" +caption="Azure Service principal details" +max-width="60%" + %} + + + +Codefresh will query the cluster and show its nodes. You are now ready to [deploy to Azure kubernetes]({{site.baseurl}}/docs/getting-started/deployment-to-kubernetes-quick-start-guide/). + + +#### Adding EKS Cluster + +To add an Amazon EKS cluster, you must first obtain `kubectl` access to it. Follow the instructions for using the +[AWS CLI](https://aws.amazon.com/premiumsupport/knowledge-center/eks-cluster-connection/){:target="\_blank"} in order to obtain your kubeconfig locally. + +``` +aws eks --region region update-kubeconfig --name cluster_name +``` + +Once you have access via `kubectl` then follow the [instructions](#get-cluster-configuration-manually) to obtain all the cluster details. +To add the Amazon cluster, select *Amazon AWS* from the *ADD PROVIDER* drop-down menu and enter all details in the respective field in the Codefresh UI. + +#### Adding a DigitalOcean cluster + +DigitalOcean also offers a hosted solution for Kubernetes. + +To add a DO cluster select *DigitalOcean* from the *Add provider* menu in your [integration settings](https://g.codefresh.io/account-admin/account-conf/integration/kubernetes). Click the authenticate button and enter your DO account credentials: + +{% include image.html +lightbox="true" +file="/images/kubernetes/integrations/add-cluster/authorize-do.png" +url="/images/kubernetes/integrations/add-cluster/authorize-do.png" +alt="Authorizing DigitalOcean Integration" +caption="Authorizing DigitalOcean Integration" +max-width="35%" + %} + +Click on the checkbox next to your account name and select the *Authorize application* button. Codefresh has now access to your DigitalOcean cluster. You need to authenticate only once. + +{% include image.html +lightbox="true" +file="/images/kubernetes/integrations/add-cluster/do-authorized.png" +url="/images/kubernetes/integrations/add-cluster/do-authorized.png" +alt="DigitalOcean is now authorized" +caption="DigitalOcean is now authorized" +max-width="70%" + %} + +Next, expand the DigitalOcean row from the triangle icon on the right and click on the *Add cluster* button. The drop-down menu should contain all your DigitalOcean Kubernetes clusters. Select the one that you want to connect into Codefresh and click the *Add* button. + +{% include image.html +lightbox="true" +file="/images/kubernetes/add-cluster/add-do-cluster.png" +url="/images/kubernetes/add-cluster/add-do-cluster.png" +alt="Selecing the DigitalOcean cluster" +caption="Selecing the DigitalOcean cluster" +max-width="40%" + %} + +Your cluster is now connected. You should be able to see it your [Kubernetes dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) and start [deploying]({{site.baseurl}}/docs/getting-started/deployment-to-kubernetes-quick-start-guide/) on it. + +Note that you can als add a DigitalOcean cluster as a generic cluster as well (explained below). + + +#### Adding any other cluster type (not dependent on any provider) + + + +1. To add any other type of cluster, outside of GKE, from the **Add Provider** list, select the **Custom Providers**. + + + +{% include image.html +lightbox="true" +file="/images/integrations/kubernetes/add-cluster/add-cluster-button.png" +url="/images/integrations/kubernetes/add-cluster/add-cluster-button.png" +alt="Adding a custom K8s cluster in Codefresh" +caption="Adding a custom K8s cluster in Codefresh" +max-width="60%" + %} + +The integration between Codefresh and your Kubernetes cluster is API based, and relies on a Kubernetes service account of your choosing that will be used to manage the integration. + +The configurations you'll be required to add are: + + +1. Name: Any name of your choosing, that will represent your cluster context in Codefresh. Do not use spaces, dots or other strange characters in the name. +1. Host: The full URL of the Kubernetes API endpoints including protocol and port. +1. Certificate: The Kubernetes service account certificate used for the integration with Codefresh (base64 encoded). +1. Token: The Kubernetes service account token used for the integration with Codefresh (base64 encoded) +1. (Optional) Namespace: Restrict Codefresh [access to a specific namespace](#restrict-codefresh-access-to-a-specific-namespace) + + +{% include image.html + lightbox="true" + file="/images/integrations/kubernetes/add-cluster/add-cluster-fields.png" + url="/images/integrations/kubernetes/add-cluster/add-cluster-fields.png" + alt="Adding a custom cluster in Codefresh" + caption="Adding a custom cluster in Codefresh" + max-width="80%" + %} + +There is also a toggle for [private clusters behind a firewall]({{site.baseurl}}/docs/reference/behind-the-firewall/). + + In the section below, we'll provide you with easy instructions how to get all your cluster configurations in order to add it to Codefresh. + +### Get cluster configuration manually + +Codefresh accesses any custom cluster using a [service account](https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/){:target="\_blank"}. You can define the privileges Codefresh has on your cluster +using the standard authorization methods (i.e. [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/){:target="\_blank"}) supported by your Kubernetes infrastructure. + +You need a terminal with `kubectl` access on your cluster. You can even use the "cloud shell" of your +cloud provider for this purpose. + +#### The easy and insecure way + +If you are evaluating Codefresh and want to connect your cluster as fast as possible with no issues +follow these steps: + +>Note that this method is only suggested for non-production clusters, and quick demos. See the next section for the proper way to use Codefresh in production environments. + +First make sure that you are giving commands to the appropriate cluster if you have more than one: + +`Choose cluster` +{% highlight shell %} +{% raw %} +kubectl config use-context+{% endraw %} +{% endhighlight %} + +Then give full admin privileges to the default account. + +`Make default account cluster administrator` +{% highlight shell %} +{% raw %} +kubectl create clusterrolebinding default-admin --clusterrole cluster-admin --serviceaccount=default:default -n default +{% endraw %} +{% endhighlight %} + +Finally run the following commands and copy-paste the result to each Codefresh field in the UI: + +`Host IP` +{% highlight shell %} +{% raw %} +export CURRENT_CONTEXT=$(kubectl config current-context) && export CURRENT_CLUSTER=$(kubectl config view -o go-template="{{\$curr_context := \"$CURRENT_CONTEXT\" }}{{range .contexts}}{{if eq .name \$curr_context}}{{.context.cluster}}{{end}}{{end}}") && echo $(kubectl config view -o go-template="{{\$cluster_context := \"$CURRENT_CLUSTER\"}}{{range .clusters}}{{if eq .name \$cluster_context}}{{.cluster.server}}{{end}}{{end}}") +{% endraw %} +{% endhighlight %} + +`Certificate` +{% highlight shell %} +{% raw %} +echo $(kubectl get secret -o go-template='{{index .data "ca.crt" }}' $(kubectl get sa default -o go-template="{{range .secrets}}{{.name}}{{end}}")) +{% endraw %} +{% endhighlight %} + +`Token` +{% highlight shell %} +{% raw %} +echo $(kubectl get secret -o go-template='{{index .data "token" }}' $(kubectl get sa default -o go-template="{{range .secrets}}{{.name}}{{end}}")) +{% endraw %} +{% endhighlight %} + +Once the cluster been added successfully you can go to the `Kubernetes` tab to start working with the services of your cluster. + +#### The proper/secure way + +For production environments you should create a service account and/or role for Codefresh access. +The minimum permissions Codefresh needs to work with the cluster are the following: + +`codefresh-role.yml` +{% highlight yaml %} +{% raw %} +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: codefresh-role +rules: + - apiGroups: [""] + resources: ["*"] + verbs: ["list", "watch", "get"] +{% endraw %} +{% endhighlight %} + +Note that these permissions will only allow Codefresh to read the cluster resources and populate the respective dashboards. You need to give more privileges for actual deployments. For more information see the [Kubernetes RBAC documentation page](https://kubernetes.io/docs/reference/access-authn-authz/rbac/){:target="\_blank"}. + +Here is an example with role + service account + binding. + +`codefresh-role-sa-bind.yml` +{% highlight yaml %} +{% raw %} +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: codefresh-role +rules: + - apiGroups: [ "*"] + resources: ["*"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: codefresh-user + namespace: kube-system +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: codefresh-user +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: codefresh-role +subjects: +- kind: ServiceAccount + name: codefresh-user + namespace: kube-system +{% endraw %} +{% endhighlight %} + +Select the appropriate cluster if you have more than one: + +`Choose cluster` +{% highlight shell %} +{% raw %} +kubectl config use-context +{% endraw %} +{% endhighlight %} + +Create the Codefresh user/role: + +`Apply Codefresh access rules` +{% highlight shell %} +{% raw %} +kubectl apply -f codefresh-role-sa-bind.yml +{% endraw %} +{% endhighlight %} + +Finally run the following commands and copy-paste the result to each Codefresh field in the UI: + +`Host IP` +{% highlight shell %} +{% raw %} +export CURRENT_CONTEXT=$(kubectl config current-context) && export CURRENT_CLUSTER=$(kubectl config view -o go-template="{{\$curr_context := \"$CURRENT_CONTEXT\" }}{{range .contexts}}{{if eq .name \$curr_context}}{{.context.cluster}}{{end}}{{end}}") && echo $(kubectl config view -o go-template="{{\$cluster_context := \"$CURRENT_CLUSTER\"}}{{range .clusters}}{{if eq .name \$cluster_context}}{{.cluster.server}}{{end}}{{end}}") +{% endraw %} +{% endhighlight %} + +`Certificate` +{% highlight shell %} +{% raw %} +echo $(kubectl get secret -n kube-system -o go-template='{{index .data "ca.crt" }}' $(kubectl get sa codefresh-user -n kube-system -o go-template="{{range .secrets}}{{.name}}{{end}}")) +{% endraw %} +{% endhighlight %} + +`Token` +{% highlight shell %} +{% raw %} +echo $(kubectl get secret -n kube-system -o go-template='{{index .data "token" }}' $(kubectl get sa codefresh-user -n kube-system -o go-template="{{range .secrets}}{{.name}}{{end}}")) +{% endraw %} +{% endhighlight %} + +#### The proper/secure way for Kubernetes Cluster 1.24+ + +For production environments, create a service account and/or role for Codefresh access. + +Codefresh needs these minimum permissions to work with the cluster: + +`codefresh-role.yml` +{% highlight yaml %} +{% raw %} +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: codefresh-role +rules: + - apiGroups: [“”] + resources: [“*”] + verbs: [“list”, “watch”, “get”] +{% endraw %} +{% endhighlight %} + +Note that these permissions will only allow Codefresh to read the cluster resources and populate the respective dashboards. You need to give more privileges for actual deployments. For more information see the [Kubernetes RBAC documentation page](https://kubernetes.io/docs/reference/access-authn-authz/rbac/){:target="\_blank"}. + +Here is an example with role + service account + binding. + +`codefresh-role-sa-bind.yml` +{% highlight yaml %} +{% raw %} +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: codefresh-role +rules: + - apiGroups: [ “*”] + resources: [“*”] + verbs: [“get”, “list”, “watch”, “create”, “update”, “patch”, “delete”] +— +apiVersion: v1 +kind: ServiceAccount +metadata: + name: codefresh-user + namespace: kube-system +— +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: codefresh-user +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: codefresh-role +subjects: +- kind: ServiceAccount + name: codefresh-user + namespace: kube-system +— +apiVersion: v1 +kind: Secret +type: kubernetes.io/service-account-token +metadata: + name: codefresh-user-token + namespace: kube-system + annotations: + kubernetes.io/service-account.name: “codefresh-user” + +{% endraw %} +{% endhighlight %} + +
+ +1. Select the appropriate cluster if you have more than one: +`Choose cluster` +{% highlight shell %} +{% raw %} +kubectl config use-context+{% endraw %} +{% endhighlight %} + +1. Create the Codefresh user/role: + +`Apply Codefresh access rules` +{% highlight shell %} +{% raw %} +kubectl apply -f codefresh-role-sa-bind.yml +{% endraw %} +{% endhighlight %} + +1. Finally run the following commands, and copy-paste the results to the respective Codefresh field in the UI: + +`Host IP` +{% highlight shell %} +{% raw %} +export CURRENT_CONTEXT=$(kubectl config current-context) && export CURRENT_CLUSTER=$(kubectl config view -o go-template=“{{\$curr_context := \”$CURRENT_CONTEXT\” }}{{range .contexts}}{{if eq .name \$curr_context}}{{.context.cluster}}{{end}}{{end}}”) && echo $(kubectl config view -o go-template=“{{\$cluster_context := \”$CURRENT_CLUSTER\”}}{{range .clusters}}{{if eq .name \$cluster_context}}{{.cluster.server}}{{end}}{{end}}”) +{% endraw %} +{% endhighlight %} + +`Certificate` +{% highlight shell %} +{% raw %} +echo $(kubectl get secret -n kube-system -o go-template=‘{{index .data “ca.crt” }}’ codefresh-user-token) +{% endraw %} +{% endhighlight %} + +`Token` +{% highlight shell %} +{% raw %} +echo $(kubectl get secret -n kube-system -o go-template=‘{{index .data “token” }}’ codefresh-user-token) +{% endraw %} +{% endhighlight %} + +#### Restrict Codefresh access to a specific namespace + +In most cases, you want to allow Codefresh to access all namespaces inside the cluster. This is the most convenient option as it will make +the [services dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) (and other GUI dashboards) the central way to manage your clusters. + +You can also restrict Codefresh only to an specific namespace of your choosing. To achieve this, use the details of service account in the previous section that has access only to that specific namespace, and also fill the *namespace* field in the cluster details form. + +{% include image.html + lightbox="true" + file="/images/integrations/kubernetes/add-cluster/restrict-namespace.png" + url="/images/integrations/kubernetes/add-cluster/restrict-namespace.png" + alt="Allows Codefresh access to a single namespace only" + caption="Allows Codefresh access to a single namespace only" + max-width="80%" + %} + +Notice that if you follow this approach several built-in Codefresh capabilities will be disabled (e.g. creating new namespaces from the GUI). + + + +### Adding a Rancher cluster + +Rancher clusters are currently supported as generic clusters. Rancher clusters have a specific authentication configuration (the details are here: [https://rancher.com/kubernetes-authentication-in-rancher-and-rbac](https://rancher.com/kubernetes-authentication-in-rancher-and-rbac){:target="\_blank"} for Rancher 1.x and at [https://rancher.com/blog/2018/2018-05-04-authentication-authorization-rancher2/](https://rancher.com/blog/2018/2018-05-04-authentication-authorization-rancher2/){:target="\_blank"} for Rancher 2.x). + +Authentication using a token of a Kubernetes Service Account, which is usually used by Codefresh, doesn't work with Rancher clusters. Also, Rancher doesn't do proper TLS termination out-of-the-box for Kubernetes clusters hosted on it, so one needs to configure a load balancer for that purpose. + +In summary, the following conditions should be met in order to add the cluster, hosted on Rancher to Codefresh: + +#### For Rancher version 1.x + +1. The token should be taken from the kubeconfig provided by Rancher and it has to be encoded with base64 before putting it into Codefresh. Be careful with the '\n' characters when encoding. The command for Linux is: `echo | tr -d '\n' | base64 | tr -d '\n'`. +1. The CA certificate should be the CA of the Load Balancer standing in front of Rancher. +1. The hostname and port should be corresponding to your Load Balancer. + +{% include image.html + lightbox="true" + file="/images/integrations/kubernetes/add-cluster/rancher-token.png" + url="/images/integrations/kubernetes/add-cluster/rancher-token.png" + alt="Getting the Rancher token" + caption="Getting the Rancher token" + max-width="40%" + %} + +#### For Rancher version 2.x + +1. Kubernetes HOST is in the kubeconfig provided by Rancher for the Kubernetes cluster based on the domain name of Rancher + the Kubernetes cluster endpoint exposed through Rancher in cluster -> server. Example: `https://rancher.localhost/k8s/clusters/c-npft4`. +1. The token should be taken from the kubeconfig provided by Rancher under user -> token section of YAML and it has to be encoded with base64 before putting it into Codefresh. Be careful with the '\n' characters when encoding, do not wrap token in quotes when running echo command. The command for Linux is: `echo | tr -d '\n' | base64 | tr -d '\n'` Example: `kubeconfig-user-xtnt4:cppxv6db…`. +1. The CA certificate should be the CA of the Load Balancer standing in front of Rancher base64 encoded `openssl base64 -in cert -out b64`. And then run this command on the file to remove any white space. `cat b64 | tr -d '\040\011\012\015' > b64_cert` then copy and paste this base64 encoded value into Codefresh UI Cert field. +1. The hostname and port should be corresponding to your Load Balancer. + +{% include image.html + lightbox="true" + file="/images/integrations/kubernetes/add-cluster/rancher-2.png" + url="/images/integrations/kubernetes/add-cluster/rancher-2.png" + alt="Rancher 2.x cluster details" + caption="Rancher 2.x cluster details" + max-width="40%" + %} + + +Once you have all the information follow the instructions for adding a generic Kubernetes cluster in Codefresh as described in the previous section. + + +### Troubleshooting cluster addition + +After adding your cluster configurations and in case the test fails, click "Save" to get the error message back. + +{% include image.html + lightbox="true" + file="/images/integrations/kubernetes/add-cluster/click-save-error-message.png" + url="/images/integrations/kubernetes/add-cluster/click-save-error-message.png" + alt="Get error message for troubleshooting" + caption="Get error message for troubleshooting" + max-width="40%" + %} + +{:.text-secondary} +#### Error: Cannot list namespaces + + `Add Cluster Error` +{% highlight shell %} +{% raw %} +Failed to add cluster: namespaces is forbidden: User "system:serviceaccount:default:default" cannot list namespaces at the cluster scope +{% endraw %} +{% endhighlight %} + +The service account used for the integration doesn't have the minimal permissions required. To fix this add a service account that have the required permissions. + +The easiest way to do this is to create a cluster binding role between the default service account and cluster-admin role: + + `Create cluster binding with admin permissions` +{% highlight shell %} +{% raw %} +kubectl create clusterrolebinding default-admin --clusterrole cluster-admin --serviceaccount=default:default +{% endraw %} +{% endhighlight %} + +### Kubernetes cluster - using an external reverse proxy (edge case) + +In case you're using an external reverse proxy to manage inbound traffic to your Kubernetes API, please read [this article]({{site.baseurl}}/docs/deploy-to-kubernetes/verify-cluster-tls-ssl-configuration/) to make sure your certificate setup are managed correctly in order to add your cluster successfully to Codefresh. + +### Multiple CAs in certificate chain + +Ideally your Kubernetes cluster will have a single certificate which is used directly on the API endpoint. Some organizations +place clusters behind a load balancer or other proxy mechanism that uses a chain or certificates. + +When that happens and you more than one [CA](https://en.wikipedia.org/wiki/Certificate_authority) in your certification chain, you need to provide Codefresh with a [Certificate bundle](https://en.wikipedia.org/wiki/Chain_of_trust) (a file that containers the intermediate CAs as well). + +You will know when this is the case as this error will appear when you try to connect your cluster: + +``` +{"status":400,"code":"1004","name":"BAD_REQUEST_ERROR","message":"Failed to add cluster: unable to get local issuer certificate","context":{}} +``` + +To get the whole certificate open the URL of your Kubernetes in Chrome or Firefox and export all individual certificates as files + +{% include image.html + lightbox="true" + file="/images/kubernetes/add-cluster/cert-hierarchy.png" + url="/images/kubernetes/add-cluster/cert-hierarchy.png" + alt="A Certificate chain" + caption="A Certificate chain" + max-width="60%" + %} + +The steps needed are: + +1. Connect all certificates (apart from the API/endpoint one) to a bundle: +`cat rootCA.crt intermediateCA.crt > ca_bundle_cert`. +1. Run the following to check the validity of the certificate: +`openssl verify -verbose -CAfile ca_bundle_cert k8_server_cert`. +1. If the check above passes fine, go on and run the following on your CA bundle file: +`base64 ca_bundle_cert | tr -d '\n'`. +1. Copy the output string (be careful when copying) and check whether you have copied it correctly: +`openssl x509 -text -in <(echo | base64 -d)` - you should see the contents of your CA bundle file. +1. Put the copied string into the Codefresh Kubernetes integration form and test the connection. + +Please make sure the certs are in order Root -> Intermediate -> Server. + + + + + + + + + +Once you connect a cluster it gets a unique name inside your account that is important when it comes to using this cluster inside a pipeline. From the same screen you can also connect [internal clusters that are behind your firewall]({{site.baseurl}}/docs/reference/behind-the-firewall/#deploying-to-an-internal-kubernetes-cluster/). +. + +## Viewing the Codefresh cluster dashboard + +After you connect a cluster, several graphical dashboards are automatically populated. The first one is the [Codefresh Kubernetes dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/). + +{% + include image.html + lightbox="true" +file="/images/integrations/kubernetes/kubernetes-dashboard.png" +url="/images/integrations/kubernetes/kubernetes-dashboard.png" +alt="Integrated Kubernetes Dashboard" +caption="Integrated Kubernetes Dashboard" +max-width="100%" +%} + +You can use this Dashboard to get basic information for your cluster such as services, pods, deployments etc. + +{% + include image.html + lightbox="true" +file="/images/integrations/kubernetes/change-kubernetes-manifest.png" +url="/images/integrations/kubernetes/change-kubernetes-manifest.png" +alt="Changing a Kubernetes Manifest" +caption="Changing a Kubernetes Manifest" +max-width="100%" +%} + +From the same dashboard you can also add/change configmaps and even edit directly the manifest of a resource. + + + +## Viewing the environment dashboard + +The second dashboard that is enabled after you connect a cluster (but not automatically populated), is the [environment dashboard]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/). + +{% include +image.html +lightbox="true" +file="/images/integrations/kubernetes/environments.png" +url="/images/integrations/kubernetes/environments.png" +alt="Codefresh Environment Dashboard" +caption="Codefresh Environment Dashboard" +max-width="100%" +%} + +This dashboard shows a live view of a Kubernetes application along with the status of the latest builds that affected this environment. You can define such environments either directly from the GUI or [programmatically in a pipeline]({{site.baseurl}}/docs/pipelines/deployment-environments/). + +## Ad-hoc deployments with the Codefresh UI + +One of the [easiest ways to deploy to Kubernetes]({{site.baseurl}}/docs/deployments/kubernetes/deployment-options-to-kubernetes/) is to use the Codefresh UI and [manually deploy a docker image]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/#deploying-a-new-service): + +{% include image.html +lightbox="true" +file="/images/integrations/kubernetes/deploy-with-ui.png" +url="/images/integrations/kubernetes/deploy-with-ui.png" +alt="Deploying with the quick UI dialog" +caption="Deploying with the quick UI dialog" +max-width="80%" +%} + +You can also [create a pull Secret]({{site.baseurl}}/docs/deployments/kubernetes/access-docker-registry-from-kubernetes/) from the GUI. + + +## Automated deployments with Codefresh pipelines + +You can also deploy to a cluster in a pipeline. Codefresh offers [several ways for Kubernetes deployments]({{site.baseurl}}/docs/deployments/kubernetes/deployment-options-to-kubernetes/). The important point here is that all connected clusters are automatically available to all pipelines with their unique name as a `kubectl` context. + + {% include +image.html +lightbox="true" +file="/images/getting-started/quick-start-k8s/deployment-build.png" +url="/images/getting-started/quick-start-k8s/deployment-build.png" +alt="Kubernetes deployment in a pipeline" +caption="Kubernetes deployment in a pipeline" +max-width="100%" +%} + + +You can use the [integrated Codefresh deployment methods]({{site.baseurl}}/docs/pipelines/steps/deploy/) or even run [custom kubectl commands directly on your cluster]({{site.baseurl}}/docs/deployments/kubernetes/custom-kubectl-commands/). + +Codefresh also offers a simple solution for [templating]({{site.baseurl}}/docs/deployments/kubernetes/kubernetes-templating/) but you can use another templating methods such as [kustomize]({{site.baseurl}}/docs/example-catalog/cd-examples/deploy-with-kustomize/). + + + +## Using a cluster for running CI/CD pipelines + +Finally you can also use the [Codefresh Runner]({{site.baseurl}}/docs/installation/codefresh-runner/) to run pipelines in your cluster. + + + +## Related articles +[Manage your Kubernetes cluster]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) +[Cloning Git repositories]({{site.baseurl}}/docs/example-catalog/ci-examples/git-checkout/) + diff --git a/_docs/integrations/microsoft-azure.md b/_docs/integrations/microsoft-azure.md new file mode 100644 index 000000000..25c6aa4dc --- /dev/null +++ b/_docs/integrations/microsoft-azure.md @@ -0,0 +1,135 @@ +--- +title: "Microsoft Azure CI integration" +description: "How to use Codefresh with Azure" +group: integrations +redirect_from: + - /docs/microsoft-azure/ + - /docs/deploy-your-containers/microsoft-azure/ +toc: true +--- + +Codefresh has native support for Azure in the following areas: + +- [Integration with Azure Git]({{site.baseurl}}/docs/integrations/git-providers/#azure-devops) +- [Connecting to Azure registries]({{site.baseurl}}/docs/docker-registries/azure-docker-registry/) +- [Deploying to AKS]({{site.baseurl}}/docs/deployments/kubernetes/#adding-aks-cluster) +- [Using Azure Storage for Test reports]({{site.baseurl}}/docs/testing/test-reports/#connecting-azure-storage) +- [Using Azure Storage for Helm charts]({{site.baseurl}}/docs/deployments/helm/add-helm-repository/#private-repository---azure) +- [Azure SSO]({{site.baseurl}}/docs/administration/single-sign-on/sso-azure/) + +## Using Azure Git repositories + +Codefresh can easily check out code from Azure Git repositories: + +{% include +image.html +lightbox="true" +file="/images/integrations/azure/azure-git-integration.png" +url="/images/integrations/azure/azure-git-integration.png" +alt="Azure Git integration" +caption="Azure Git integration" +max-width="70%" +%} + +For more details see the [documentation page]({{site.baseurl}}/docs/integrations/git-providers/#azure-devops). Once your repository is connected, you can use the [native clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/) as well as [Git triggers]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/) like all other git providers. + +## Using Azure Docker registries + +Azure Docker registries are fully compliant with the Docker registry API that Codefresh follows. You can connect an Azure Registry like any [other Docker registry]({{site.baseurl}}/docs/docker-registries/). + +{% + include image.html + lightbox="true" +file="/images/integrations/docker-registries/add-azure-registry.png" +url="/images/integrations/docker-registries/add-azure-registry.png" +alt="Adding the Azure Docker registry" +caption="Adding the Azure Docker registry" +max-width="70%" +%} + +Once the registry is added you can the [standard push step]({{site.baseurl}}/docs/pipelines/steps/push/) step in pipelines. + +## Deploying to Azure Kubernetes + +Codefresh has native support for connecting an Azure cluster in the [cluster configuration screen]({{site.baseurl}}/docs/deployments/kubernetes/#connect-a-kubernetes-cluster). + +{% + include image.html + lightbox="true" +file="/images/integrations/azure/aks-integration.png" +url="/images/integrations/azure/aks-integration.png" +alt="Connecting an Azure cluster" +caption="Connecting an Azure cluster" +max-width="40%" +%} + +Once the cluster is connected you can use any of the [available deployment options]({{site.baseurl}}/docs/deployments/kubernetes/deployment-options-to-kubernetes/) for Kubernetes clusters. You also get access to all other Kubernetes dashboards such as the [cluster dashboard]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) or the [environment dashboard]({{site.baseurl}}/docs/deployments/kubernetes/environment-dashboard/) . + +## Storing test reports in Azure storage + +Codefres has native support for test reports. You can store the reports on Azure storage. + +{% include +image.html +lightbox="true" +file="/images/integrations/azure/azure-storage.png" +url="/images/integrations/azure/azure-storage.png" +alt="Azure cloud storage" +caption="Azure cloud storage" +max-width="50%" +%} + +See the full documentation for [test reports]({{site.baseurl}}/docs/testing/test-reports/). + +## Using Azure storage for storing Helm charts + +You can connect Azure Storage as a Helm repository in the [integrations screen]({{site.baseurl}}/docs/deployments/helm/add-helm-repository/). + +{% include +image.html +lightbox="true" +file="/images/integrations/azure/azure-helm-repo.png" +url="/images/integrations/azure/azure-helm-repo.png" +alt="Using Azure for Helm charts" +caption="Using Azure for Helm charts" +max-width="80%" +%} + +Once you connect your Helm repository you can use it any [Codefresh pipeline with the Helm step]({{site.baseurl}}/docs/deployments/helm/using-helm-in-codefresh-pipeline/). + +## Azure Single Sign-on + +You can use Azure Active Directory as an [SSO mechanism]({{site.baseurl}}/docs/administration/single-sign-on/) in Codefresh. + +{% include +image.html +lightbox="true" +file="/images/integrations/azure/azure-sso-integration.png" +url="/images/integrations/azure/azure-sso-integration.png" +alt="Azure SSO integration" +caption="Azure SSO integration" +max-width="70%" +%} + +Once configuration is complete all Codefresh users can login using their Azure credentials instead of personal accounts. + +## Traditional Azure deployments + +For any other Azure deployment you can use the [Azure CLI from a Docker image](https://hub.docker.com/_/microsoft-azure-cli){:target="\_blank"} in a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/). + +`YAML` +{% highlight yaml %} +{% raw %} + create_my_vm: + title: Creating a VM + image: mcr.microsoft.com/azure-cli + commands: + - az vm create --resource-group TutorialResources --name TutorialVM1 --image UbuntuLTS --generate-ssh-keys +{% endraw %} +{% endhighlight %} + +For authentication see the [Microsoft documentation page](https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli?view=azure-cli-latest){:target="\_blank"}. + +## Related articles +[Manage your Kubernetes cluster]({{site.baseurl}}/docs/deployments/kubernetes/manage-kubernetes/) +[Cloning Git repositories]({{site.baseurl}}/docs/example-catalog/ci-examples/git-checkout/) diff --git a/_docs/integrations/notifications.md b/_docs/integrations/notifications.md new file mode 100644 index 000000000..21b12ae77 --- /dev/null +++ b/_docs/integrations/notifications.md @@ -0,0 +1,15 @@ +--- +title: "Notifications" +description: "" +group: integrations +redirect_from: + - /docs/notifications/ +toc: true +--- +Codefresh enables you to send notifications about events. + +Codefresh supports the following notification channels: +- Email +- [Slack]({{ site.baseurl }}/docs/integrations/notifications/slack-integration/) +- [Jira]({{ site.baseurl }}/docs/integrations/notifications/jira-integration/) + diff --git a/_docs/integrations/notifications/jira-integration.md b/_docs/integrations/notifications/jira-integration.md new file mode 100644 index 000000000..ee47311ce --- /dev/null +++ b/_docs/integrations/notifications/jira-integration.md @@ -0,0 +1,47 @@ +--- +title: "Jira notification integrations for piplines" +description: "" +group: integrations +redirect_from: + - /docs/jira-integration-1/ + - /docs/integrations/jira-integration-1/ +toc: true +--- +Codefresh integrates with Jira in several ways: +* Through the [Jira integration]({{site.baseurl}}/docs/integrations/jira/) for the highest visibility into your GitOps deployments +* Through a [custom step]({{site.baseurl}}/docs/integrations/notifications/jira-integration/#use-jira-within-your-codefresh-pipeline) from our step marketplace so that you can connect your pipelines with Jira +* Alternatively, through using your own [jira-cli]({{site.baseurl}}/docs/integrations/notifications/jira-integration/#using-your-own-jira-cli) + + +## Prerequisites +* [Codefresh Account]({{site.baseurl}}/docs/administration/create-a-codefresh-account/) +* [Jira Account](https://www.atlassian.com/software/jira){:target="\_blank"} + +## Use Jira in your Codefresh pipeline + +The step marketplace offers several freestyle steps that can be used in your Codefresh pipeline through steps. + +One of those steps is the [Jira Issue Manager](https://codefresh.io/steps/step/jira-issue-manager){:target="\_blank"}. +It can be used to: +* Create a Jira issue +* Comment on existing Jira issues +* Change the status of an issue, for example, once the build is successful +* Add a description to your issue +* And more + +More information is provided [directly in the example]({{site.baseurl}}/docs/example-catalog/ci-examples/sending-the-notification-to-jira/). + +## Using your own jira-cli + +Alternatively, you can use your own jira-cli by adding the following steps to your Dockerfile: + +{% highlight yaml %} +FROM python:2-alpine +RUN apk add -U gcc musl-dev linux-headers openssl-dev libffi-dev && pip install jira-cli +{% endhighlight %} + +And then running the Dockerfile. + +## Related articles +[Codefresh pipeline examples]({{site.baseurl}}/docs/example-catalog/examples/) +[Create a pipeline]({{site.baseurl}}/docs/pipelines/pipelines/) \ No newline at end of file diff --git a/_docs/integrations/notifications/slack.md b/_docs/integrations/notifications/slack.md new file mode 100644 index 000000000..8300c42c7 --- /dev/null +++ b/_docs/integrations/notifications/slack.md @@ -0,0 +1,66 @@ +--- +title: "Slack" +description: "Get Slack notifications with pipeline integrations" +group: integrations +sub_group: notifications +permalink: /:collection/integrations/notifications/slack-integration/ +toc: true +excerpt: "Integrate Codefresh with Slack to get updates on development and testing progress and feedback." +--- + +You can integrate Slack globally, or for specific pipelines and builds. + +## Set up global Slack integration in Codefresh + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Slack** and then click **Configure**. +1. Select **Quick Setup**. +1. Select the types of notifications you want to receive on Slack. +1. Click **Add to Slack**.
You are redirected to the Slack sign-in page. +1. Log in with your Slack credentials. +1. Select the check box to enable notifications for Slack. +1. Click **Save**. + +{% include image.html +lightbox="true" +file="/images/integrations/slack/add-slack-integration.png" +url="/images/integrations/slack/add-slack-integration.png" +alt="Add Slack integration" +max-width="50%" +%} + + +Codefresh can now post notifications to Slack, for example, notifications of successful and failed builds, and direct messages received within the Codefresh app. + +## How Slack notifications work + +When you have Slack integration enabled: + +1. All pipelines that are launched automatically by [triggers]({{site.baseurl}}/docs/pipelines/triggers/) send Slack notifications +1. All pipelines that are executed manually do **NOT** send Slack notifications. + +You can override this behavior by toggling the checkbox **Report notification on pipeline execution** under **Advanced Settings** +either in a Git trigger dialog or the Run settings of a pipeline. + +{% include image.html +lightbox="true" +file="/images/integrations/slack/report-notifications.png" +url="/images/integrations/slack/report-notifications.png" +alt="Manual Slack override" +caption="Manual Slack override" +max-width="40%" +%} + +## Individual pipeline Slack integration + +If you wish for more fine-grained control over Slack notifications, then take a look at any of the available slack plugins + +* [https://codefresh.io/steps/step/slack-message-sender](https://codefresh.io/steps/step/slack-message-sender){:target="\_blank"} +* [https://codefresh.io/steps/step/slack-notifier](https://codefresh.io/steps/step/slack-notifier){:target="\_blank"} +* [https://github.com/cloudposse/slack-notifier](https://github.com/cloudposse/slack-notifier){:target="\_blank"} + + +## Related articles +[Git triggers in pipelines]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[Monitoring pipelines]({{site.baseurl}}/docs/pipelines/monitoring-pipelines/) diff --git a/_docs/integrations/secret-storage.md b/_docs/integrations/secret-storage.md new file mode 100644 index 000000000..a7f268fa2 --- /dev/null +++ b/_docs/integrations/secret-storage.md @@ -0,0 +1,158 @@ +--- +title: "Secret Storage" +description: "Manage Kubernetes secrets with Codefresh" +group: integrations +toc: true +--- + +Codefresh can resolve variables storing secrets from remote sources. This allows you to keep sensitive data on your cluster, and for Codefresh to request it during pipeline execution on-demand. + +Secret-Store is an additional context in Codefresh. Codefresh supports two types of secret storage: +* Kubernetes secrets for SaaS versions +* Runtime-Kubernetes for hybrid deployments with Codefresh Runner + +You can set up both types either in the Codefresh UI or via the CLI (`codefresh create context secret-store --help`). + +> This feature is for Enterprise accounts only. + +## Kubernetes secret store setup +Kubernetes the native secrets supported by a cluster. + +### Prerequisites + +* For the Kubernetes secret store, [connect your Kubernetes cluster to Codefresh]({{site.baseurl}}/docs/integrations/kubernetes/connect-a-kubernetes-cluster/). +* Create a Kubernetes secret: + + +### Create a Kubernetes Secret + +Create your secret in Kubernetes: + +``` +kubectl create secret generic my-secret --from-literal=key1=supersecret +``` + +``` +kubectl create configmap my-config-map --from-literal=key1=config1 +``` + + +### Set up Kubernetes secret integration in Codefresh UI + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Secret Store** and then click **Configure**. +1. From the **Add Provider** dropdown, select **Kubernetes**. +1. Do the following: + * **Name**: A unique name given to your context, which will be referenced in `codefresh.yaml`. + * **Resource Type**: Select **Secret** and the data is base64 decoded during resolution. + * **Cluster**: The name of the cluster as it is configured in Codefresh. + * **Namespace**: The namespace where the secret is stored. + * **Resource Name**: Optional. The name of the secret. + * To allow all users in the account access to the secret, enable **Allow access to all users**. + + + +{% include +image.html +lightbox="true" +file="/images/integrations/secret-storage/secrets-ui-view.png" +url="/images/integrations/secret-storage/secrets-ui-view.png" +alt="Kubernetes Secret Store" +caption="Kubernetes Secret Store" +max-width="80%" +%} + +{:start="5"} +1. To apply the changes, click **Save**. + + +### Set up Kubernetes secret integration via Codefresh CLI + +* To create a secret store context for **Kubernetes**, run: +``` +codefresh create context secret-store kubernetes "$NAME_IN_CODEFRESH" --cluster "$CLUSTER" --namespace "$NAMESPACE" --resource-type "$TYPE" --resource-name ”$NAME” +``` +OR, for our example: + +``` +codefresh create context secret-store kubernetes "test" --cluster "anna-demo@FirstKubernetes" --namespace "default" --resource-type secret --resource-name "my-secret" +``` + +where: + +- `$NAME_IN_CODEFRESH` is a unique name given to your context, which will be referenced in `codefresh.yaml` later. +- `$CLUSTER` is the name of the cluster as it is configured in Codefresh. +- `$NAMESPACE` is the Kubernetes namespace where the secret is stored. +- `$TYPE` is of either `secret` or `configmap` + - if `secret`, data will be base64 decoded during resolution + - if `configmap`, data will be replaced as is +- `$RESOURCE_NAME` is optional and is the name of the secret + +## Runtime secret store setup for Codefresh Runner installation + + +For [Codefresh Runner]({{site.baseurl}}/docs/installation/codefresh-runner/) installations, you can also store secrets in your own runtime. + +### Set up runtime secret store in Codefresh UI + +1. In the Codefresh UI, on the toolbar, click the **Settings** icon, and then from the sidebar, select [**Pipeline Integrations**](https://g.codefresh.io/account-admin/account-conf/integration){:target="\_blank"}. +1. Select **Secret Store** and then click **Configure**. +1. From the **Add Provider** dropdown, select **Runtime secret**. +1. Do the following: + * **Name**: A unique name given to your context, which will be referenced in `codefresh.yaml`. + * **Resource Name**: The name of the secret. + * **Resource Type**: Select the type of secret . - if `secret`, data will be base64 decoded during resolution + - if `configmap`, data will be replaced as is + * **Runtime Environment**: Select the runtime environment with the secret. + * To allow all users in the account access to the secret, enable **Allow access to all users**. + +{% include +image.html +lightbox="true" +file="/images/integrations/secret-storage/secrets-ui-view2.png" +url="/images/integrations/secret-storage/secrets-ui-view2.png" +alt="Runtime Secret Store" +caption="Runtime Secret Store" +max-width="80%" +%} + +{:start="5"} + +1. To apply the changes, click **Save**. + + + +### Set up runtime secret store with Codefresh CLI + +To create a secret store context for **Runtime-Kubernetes** environments ([behind the firewall]({{site.baseurl}}/docs/reference/codefresh-runner/)), run: + +``` +codefresh create context secret-store kubernetes-runtime "$NAME_IN_CODEFRESH" --runtime "$RUNTIME_NAME" --resource-type "$TYPE" --resource-name ”$NAME” +``` + +or, for our example: + +``` +codefresh create context secret-store kubernetes-runtime "test" --runtime "gke_firstkubernetes-176201_us-central1-a_anna-demo" --resource-type secret --resource-name "my-secret" +``` + +where: + +- `$NAME_IN_CODEFRESH` is a unique name given to your context, which will be referenced in `codefresh.yaml` later. +- `$CLUSTER` is the name of the cluster as it is configured in Codefresh +- `$NAMESPACE` is the Kubernetes namespace +- `$TYPE` is of either `secret` or `configmap` + - if `secret`, data will be base64 decoded during resolution + - if `configmap`, data will be replaced as is +- `$RESOURCE_NAME` is the name of the secret (optional) +- `$RUNTIME_NAME` is the name of the run-time environment to be configured as secret store. If not set, *any* runtime-environment will be considered. + +## Using the secrets + +Once Codefresh is linked to your secrets, you can use them either in pipelines or any relevant section in the Codefresh UI. For details, see [Using secrets]({{site.baseurl}}/docs/pipelines/secrets-store/) for the details. + +## Related articles +[Shared Configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/) +[Git integration for pipelines]({{site.baseurl}}/docs/integrations/git-providers/) +[Kubernetes integration for pipelines]({{site.baseurl}}/docs/integrations/kubernetes/) +[Container registry integration for pipelines]({{site.baseurl}}/docs/integrations/docker-registries/) diff --git a/_docs/migration/gitops-dashboard.md b/_docs/migration/gitops-dashboard.md deleted file mode 100644 index 17429bace..000000000 --- a/_docs/migration/gitops-dashboard.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -title: "GitOps dashboard" -description: "" -group: migration -toc: true ---- - -Coming soon diff --git a/_docs/migration/pipelines.md b/_docs/migration/pipelines.md deleted file mode 100644 index 176f096c8..000000000 --- a/_docs/migration/pipelines.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -title: "Pipelines" -description: "" -group: migration -toc: true ---- - -Coming soon diff --git a/_docs/pipelines/advanced-workflows.md b/_docs/pipelines/advanced-workflows.md new file mode 100644 index 000000000..8bb797dcd --- /dev/null +++ b/_docs/pipelines/advanced-workflows.md @@ -0,0 +1,972 @@ +--- +title: "Advanced workflows with parallel steps" +description: "Create complex workflows in Codefresh with step dependencies" +group: pipelines +toc: true +--- + +Codefresh is very flexible when it comes to pipeline complexity and depth. + +You can easily create: + * Sequential pipelines where step order is the same as the listing order in YAML (simple) + * Sequential pipelines that have some parallel parts (intermediate) + * Parallel pipelines where step order is explicitly defined (advanced) + +With the parallel execution mode, you can define complex pipelines with fan-in/out configurations capable of matching even the most complicated workflows within an organization. + +>In Codefresh, parallel execution is unrelated to [stages]({{site.baseurl}}/docs/codefresh-yaml/stages/). Stages are only a way to visually organize your pipeline steps. The actual execution is independent from the visual layout in the logs view. + +Before going any further make sure that you are familiar with the [basics of Codefresh pipelines]({{site.baseurl}}/docs/configure-ci-cd-pipeline/introduction-to-codefresh-pipelines/). + +Codefresh offers two modes of execution: + +1. Sequential mode (which is the default) +1. Parallel mode + +## Sequential execution mode + +The sequential mode is very easy to understand and visualize. + +In sequential mode, the Codefresh execution engine starts from the first step defined at the top of the `codefresh.yml` file, and executes all steps one by one going down to the end of the file. A step is either executed or skipped according to its conditions. + +>The condition for each step is only examined **once**. + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +mode: sequential +steps: + MyAppDockerImage: + title: Building Docker Image + type: build + image_name: sample-python-image + working_directory: ./ + tag: ${{CF_BRANCH_TAG_NORMALIZED}} + dockerfile: Dockerfile + MyUnitTests: + title: Running Unit tests + image: ${{MyAppDockerImage}} + commands: + - python setup.py test +{% endraw %} +{% endhighlight %} + +Here we have two steps, one that creates a Docker image and a second one that runs [unit tests]({{site.baseurl}}/docs/testing/unit-tests/) inside it. The order of execution is identical to the order of the steps in the YAML file. This means that unit tests will always run after the Docker image creation. + +Notice that the line `mode: sequential` is shown only for illustration purposes. Sequential mode is the default, and therefore this line can be omitted. + + +## Inserting parallel steps in a sequential pipeline + +You don't have to activate parallel execution mode for the whole pipeline if only a part of it needs to run in parallel. Codefresh allows you insert a parallel phase inside a sequential pipeline with the following syntax: + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + my_task1: + title: My Task 1 + [...] + my_parallel_tasks: + type: parallel + steps: + my_task2a: + title: My Task 2A + [...] + my_task2b: + title: My Task 2B + [...] + my_task3: + title: My Task3 + [...] +{% endraw %} +{% endhighlight %} + + +In this case tasks 2A and 2B will run in parallel. +The step name that defines the parallel phase (`my_parallel_tasks` in the example above), is completely arbitrary. + +The final order of execution will be + +1. Task 1 +1. Task 2A and Task2B at the same time +1. Task 3 + +This is the recommended way to start using parallelism in your Codefresh pipelines. It is sufficient for most scenarios that require parallelism. + +>The step names must be unique within the same pipeline. The parent and child steps should NOT share the same name. + +### Example: pushing multiple Docker images in parallel + +Let's see an example where a Docker image is created and then we push it to more than one registry. This is a perfect candidate for parallelization. Here is the `codefresh.yml`: + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: +- build +- push +steps: + MyAppDockerImage: + title: Building Docker Image + stage: 'build' + type: build + image_name: trivialgoweb + working_directory: ./ + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + dockerfile: Dockerfile + PushingToRegistries: + type: parallel + stage: 'push' + steps: + jfrog_PushingTo_jfrog_BintrayRegistry: + type: push + title: jfrog_Pushing To Bintray Registry + candidate: ${{MyAppDockerImage}} + tag: '${{CF_SHORT_REVISION}}' + registry: bintray + PushingToGoogleRegistry: + type: push + title: Pushing To Google Registry + candidate: ${{MyAppDockerImage}} + tag: '${{CF_SHORT_REVISION}}' + registry: gcr + PushingToDockerRegistry: + type: push + title: Pushing To Dockerhub Registry + candidate: ${{MyAppDockerImage}} + tag: '${{CF_SHORT_REVISION}}' + image_name: kkapelon/trivialgoweb + registry: dockerhub +{% endraw %} +{% endhighlight %} + +The order of execution is the following: + +1. MyAppDockerImage ([build step]({{site.baseurl}}/docs/pipelines/steps/build/)) +1. jfrog_PushingTo_jfrog_BintrayRegistry, PushingToGoogleRegistry, PushingToDockerRegistry ([push steps]({{site.baseurl}}/docs/pipelines/steps/push/)) + +The pipeline view for this yaml file is the following. + +{% include +image.html +lightbox="true" +file="/images/codefresh-yaml/parallel-push.png" +url="/images/codefresh-yaml/parallel-push.png" +alt="Parallel Docker push" +caption="Parallel Docker push" +max-width="80%" +%} + +As you can see we have also marked the steps with [stages]({{site.baseurl}}/docs/pipelines/stages/) so that we get a visualization that matches the execution. + + +### Example: Running multiple test suites in parallel + +All types of steps can by placed inside a parallel phase. Another common use case would be the parallel execution of [freestyle steps]({{site.baseurl}}/docs/codefresh-yaml/steps/freestyle/) for unit/integration tests. + +Let's say that you have a Docker image with a Python back-end and a JavaScript front-end. You could run both types of tests in parallel with the following yaml syntax: + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + MyAppDockerImage: + title: Building Docker Image + type: build + image_name: my-full-stack-app + working_directory: ./ + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + dockerfile: Dockerfile + MyTestingPhases: + type: parallel + steps: + my_back_end_tests: + title: Running Back end tests + image: ${{MyAppDockerImage}} + commands: + - python setup.py test + my_front_end_tests: + title: Running Front End tests + image: ${{MyAppDockerImage}} + commands: + - npm run test +{% endraw %} +{% endhighlight %} + +Running different types of tests (unit/integration/load/acceptance) in parallel is a very common use case for parallelism inside an otherwise sequential pipeline. + +### Defining success criteria for a parallel step + +By default, any failed step in a Codefresh pipeline will fail the whole pipeline. There are ways to change this behavior (the `fail_fast` property is explained later in this page), but specifically for parallel steps you can define exactly when the whole step succeeds or fails. + +You can define steps that will be used to decide if a parallel step succeeds with this syntax: + +{% highlight yaml %} +second_step: + title: Second step + success_criteria: + steps: + only: + - my_unit_tests + type: parallel + steps: + my_unit_tests: + title: Running Back end tests + image: node + commands: + - npm run test + my_integration_tests: + title: Running Integration tests + image: node + commands: + - npm run int-test + my_acceptance_tests: + title: Running Acceptance tests + image: node + commands: + - npm run acceptance-test +{% endhighlight %} + +In the example above, if integration and/or acceptance tests fail, the whole pipeline will continue, because we have defined that only the results of unit test matter for the whole parallel step. + +The reverse relationship (i.e., defining steps to be ignored) can be defined with the following syntax + +{% highlight yaml %} +second_step: + title: Second step + success_criteria: + steps: + ignore: + - my_integration_tests + - my_acceptance_tests + type: parallel + steps: + my_unit_tests: + title: Running Back end tests + image: node + commands: + - npm run test + my_integration_tests: + title: Running Integration tests + image: node + commands: + - npm run int-test + my_acceptance_tests: + title: Running Acceptance tests + image: node + commands: + - npm run acceptance-test +{% endhighlight %} + +In the example above we have explicitly defined that even if the integration or acceptance tests fail the whole pipeline will continue. + +### Shared Codefresh volume and race conditions + +In any pipeline step, Codefresh automatically attaches a [shared volume]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/#sharing-the-workspace-between-build-steps) that is used to transfer artifacts between steps. The same volume is also shared between steps that run in parallel. + + +Here is an example where two parallel steps are writing two files. After they finish execution, we list the contents of the project folder. + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + WritingInParallel: + type: parallel + steps: + writing_file_1: + title: Step1A + image: alpine + commands: + - echo "Step1A" > first.txt + writing_file_2: + title: Step1B + image: alpine + commands: + - echo "Step1B" > second.txt + MyListing: + title: Listing of files + image: alpine + commands: + - ls +{% endraw %} +{% endhighlight %} + +The results from the `MyListing` step is the following: + +``` +first.txt second.txt +``` + +This illustrates the side effects for both parallel steps that were executed on the same volume. + +>It is therefore your responsibility to make sure that steps that run in parallel play nice with each other. Currently, Codefresh performs no conflict detection at all. If there are race conditions between your parallel steps, (e.g. multiple steps writing at the same files), the final behavior is undefined. It is best to start with a fully sequential pipeline, and use parallelism in a gradual manner if you are unsure about the side effects of your steps + +## Implicit parallel steps +> If you use implicit parallel steps, you _cannot_ use _parallel pipeline mode_. + +In all the previous examples, all parallel steps have been defined explicitly in a pipeline. This works well for a small number of steps, but in some cases it can be cumbersome to write such a pipeline, especially when the parallel steps are similar. + +Codefresh offers two handy ways to lessen the amount of YAML you have to write and get automatic parallelization with minimum effort. + +* The `scale` syntax allows you to quickly create parallel steps that are mostly similar (but still differ) +* The `matrix` syntax allows you to quickly create parallel steps for multiple combinations of properties + +### Scale parallel steps (one dimension) + +If you look back at the parallel docker push example you will see that all push steps are the same. The only thing that changes is the registry that they push to. + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: +- build +- push +steps: + MyAppDockerImage: + title: Building Docker Image + stage: 'build' + type: build + image_name: trivialgoweb + working_directory: ./ + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + dockerfile: Dockerfile + PushingToRegistries: + type: parallel + stage: 'push' + steps: + jfrog_PushingTo_jfrog_BintrayRegistry: + type: push + title: jfrog_Pushing To Bintray Registry + candidate: ${{MyAppDockerImage}} + tag: '${{CF_SHORT_REVISION}}' + registry: bintray + PushingToGoogleRegistry: + type: push + title: Pushing To Google Registry + candidate: ${{MyAppDockerImage}} + tag: '${{CF_SHORT_REVISION}}' + registry: gcr + PushingToDockerRegistry: + type: push + title: Pushing To Dockerhub Registry + candidate: ${{MyAppDockerImage}} + tag: '${{CF_SHORT_REVISION}}' + image_name: kkapelon/trivialgoweb + registry: dockerhub +{% endraw %} +{% endhighlight %} + + +This pipeline can be simplified by using the special `scale` syntax to create a common parent step with all similarities: + + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: +- build +- push +steps: + MyAppDockerImage: + title: Building Docker Image + stage: 'build' + type: build + image_name: trivialgoweb + working_directory: ./ + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + dockerfile: Dockerfile + PushingToRegistries: + stage: 'push' + type: push + tag: '${{CF_SHORT_REVISION}}' + candidate: ${{MyAppDockerImage}} + scale: + jfrog_PushingTo_jfrog_BintrayRegistry: + registry: bintray + PushingToGoogleRegistry: + registry: gcr + PushingToDockerRegistry: + image_name: kkapelon/trivialgoweb + registry: dockerhub +{% endraw %} +{% endhighlight %} + +You can see now that all common properties are defined once in the parent step (`PushingToRegistries`) while each push step only contains what differs. Codefresh will automatically create parallel steps when it encounters the `scale` syntax. + +The resulting pipeline is more concise but runs in the same manner as the original YAML. For a big number of parallel steps, the `scale` syntax is very helpful for making the pipeline definition more clear. + +You can use the `scale` syntax with all kinds of steps in Codefresh and not just push steps. Another classic example would be running tests in parallel with different environment variables. + + +`YAML` +{% highlight yaml %} +{% raw %} + run_tests_in_parallel: + stage: 'Microservice A' + working_directory: './my-front-end-code' + image: node:latest + commands: + - npm run test + scale: + first: + environment: + - TEST_NODE=0 + second: + environment: + - TEST_NODE=1 + third: + environment: + - TEST_NODE=2 + fourth: + environment: + - TEST_NODE=3 +{% endraw %} +{% endhighlight %} + +This pipeline will automatically create 4 parallel freestyle steps. All of them will use the same Docker image and executed the same command (`npm run test`) but each one will receive a different value for the environment variable called `TEST_NODE`. + +Notice that if you define environment variables on the parent step (`run_tests_in_parallel` in the example above), they will also be available on the children parallel steps. And if those define, environment variables as well, all environment variables will be available. + + +### Matrix parallel steps (multiple dimensions) + +The `scale` syntax allows you to easily create multiple parallel steps that differ only in a single dimension. If you have multiple dimensions of properties that differ and you want to run all possible combinations (Cartesian product) then the `matrix` syntax will do that for you automatically. + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - test +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'codefreshdemo/cf-example-unit-test' + revision: 'master' + git: github + stage: prepare + run_my_tests_before_build: + stage: test + working_directory: './golang-app-A' + commands: + - go test -v + matrix: + image: + - golang:1.11 + - golang:1.12 + - golang:1.13 + environment: + - [CGO_ENABLED=1] + - [CGO_ENABLED=0] +{% endraw %} +{% endhighlight %} + +Here we want run unit tests with 3 different versions of GO and also try with CGO enabled or not. Instead of manually writing 6 parallel steps in your pipeline with all possible combinations, we can simply use the `matrix` syntax to create the following parallel steps: + +* Go 1.11 with CGO enabled +* Go 1.11 with CGO disabled +* Go 1.12 with CGO enabled +* Go 1.12 with CGO disabled +* Go 1.13 with CGO enabled +* Go 1.13 with CGO disabled + +The resulting Codefresh YAML is much more compact. Notice that because the `environment` property in Codefresh is already an array on its own, when we use it with the `matrix` syntax we need to enclose its value with `[]` (array of arrays). + +You can add more dimensions to a matrix build (and not just two as shown in the example). Here is another example with 3 dimensions: + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - test +steps: + main_clone: + title: Cloning main repository... + stage: prepare + type: git-clone + repo: 'codefresh-contrib/spring-boot-2-sample-app' + revision: master + git: github + MyUnitTests: + stage: test + matrix: + image: + - 'maven:3.5.2-jdk-8-alpine' + - 'maven:3.6.2-jdk-11-slim' + - 'maven:3-jdk-8' + commands: + - ["mvn --version", "mvn -Dmaven.repo.local=/codefresh/volume/m2_repository test"] + - ["mvn --version", "mvn -Dmaven.test.skip -Dmaven.repo.local=/codefresh/volume/m2_repository package"] + environment: + - [MAVEN_OPTS=-Xms1024m] + - [MAVEN_OPTS=-Xms512m] +{% endraw %} +{% endhighlight %} + +This pipeline creates 3 x 2 x 2 = 12 parallel steps with all the possible combinations of: + +* Maven version +* Running or disabling tests +* Using 1GB or 512MBs of memory. + +Remember that all parallel steps run within the same pipeline executor so make sure that you have enough resources as the number +of matrix variations can quickly grow if you add too many dimensions. + +Notice that, as with the `scale` syntax, the defined values/properties are merged between parent step (`MyUnitTests` in the example above) and children steps. For example, if you set an environment variable on the parent and also on child matrix steps , the result will a merged environment where all values are available. + +## Parallel pipeline execution +> If you use parallel execution mode for pipelines, you _cannot_ use _implicit parallel steps_. + +To activate advanced parallel mode for the whole pipeline, you need to declare it explicitly at the root of the `codefresh.yml` file: + +``` +version: '1.0' +mode: parallel +steps: +[...] +``` + +In full parallel mode, the order of steps inside the `codefresh.yml` **does not** affect the order of execution at all. The Codefresh pipeline engine instead: + +1. Evaluates all step-conditions *at the same* time +2. Executes those that have their requirements met +3. Starts over with the remaining steps +4. Stops when there are no more steps to evaluate + +This means that in parallel mode the conditions of a step are evaluated **multiple times** as the Codefresh execution engine tries to find which steps it should run next. This implication is very important when you try to understand the order of step execution. + +Notice also that in parallel mode, if you don't define any step conditions, Codefresh will try to run **all** steps at once, which is probably not what you want in most cases. + +With parallel mode you are expected to define the order of steps in the yaml file, and the Codefresh engine will create a *graph* of execution that satisfies your instructions. This means that writing the `codefresh.yml` file requires more effort on your part, but on the other hand allows you to define the step order in ways not possible with the sequential mode. You also need to define which steps should depend on the automatic cloning of the pipeline (which is special step named `main_clone`). + +In the next sections we describe how you can define the steps dependencies in a parallel pipeline. + +### Single step dependencies + +At the most basic level, you can define that a step *depends on* the execution of another step. This dependency is very flexible as Codefresh allows you run a second step once: + +1. The first step is finished with success +1. The first step is finished with failure +1. The first completes (regardless of exit) status + +The syntax for this is the following post-condition: + +{% highlight yaml %} +second_step: + title: Second step + when: + steps: + - name: first_step + on: + - success +{% endhighlight %} + +If you want to run the second step only if the first one fails the syntax is: + +{% highlight yaml %} +second_step: + title: Second step + when: + steps: + - name: first_step + on: + - failure +{% endhighlight %} + +Finally, if you don't care about the completion status the syntax is: + +{% highlight yaml %} +second_step: + title: Second step + when: + steps: + - name: first_step + on: + - finished +{% endhighlight %} + +Notice that `success` is the default behavior so if you omit the last two lines (i.e., the `on:` part) the second step +will wait for the next step to run successfully. + +>Also notice that the name `main_clone` is reserved for the automatic clone that takes place in the beginning of pipelines that are linked to a git repository. You need to define which steps depend on it (probably the start of your graph) so that `git checkout` happens before the other steps. + +As an example, let's assume that you have the following steps in a pipeline: + +1. A build step that creates a Docker image +1. A freestyle step that runs [unit tests]({{site.baseurl}}/docs/testing/unit-tests/) inside the Docker image +1. A freestyle step that runs [integrations tests]({{site.baseurl}}/docs/testing/integration-tests/) *After* the unit tests, even if they fail +1. A cleanup step that runs after unit tests if they fail + +Here is the full pipeline. Notice the explicit dependency to the `main_clone` step that checks out the code. + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +mode: parallel +steps: + MyAppDockerImage: + title: Building Docker Image + type: build + image_name: my-node-js-app + working_directory: ./ + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + dockerfile: Dockerfile + when: + steps: + - name: main_clone + on: + - success + MyUnitTests: + title: Running unit tests + image: ${{MyAppDockerImage}} + fail_fast: false + commands: + - npm run test + when: + steps: + - name: MyAppDockerImage + on: + - success + MyIntegrationTests: + title: Running integration tests + image: ${{MyAppDockerImage}} + commands: + - npm run integration-test + when: + steps: + - name: MyUnitTests + on: + - finished + MyCleanupPhase: + title: Cleanup unit test results + image: alpine + commands: + - ./cleanup.sh + when: + steps: + - name: MyUnitTests + on: + - failure +{% endraw %} +{% endhighlight %} + +If you run the pipeline you will see that Codefresh automatically understands that `MyIntegrationTests` and `MyCleanupPhase` can run in parallel right after the unit tests finish. + +Also notice the `fail_fast: false` line in the unit tests. By default, if *any* steps fails in a pipeline the whole pipeline is marked as a failure. With the `fail_fast` directive we can allow the pipeline to continue so that other steps that depend on the failed step can still run even. + + +### Multipl step dependencies + +A pipeline step can also depend on multiple other steps. + +The syntax is: + +{% highlight yaml %} +third_step: + title: Third step + when: + steps: + all: + - name: first_step + on: + - success + - name: second_step + on: + - finished +{% endhighlight %} + +In this case, the third step will run only when BOTH first and second are finished (and first is actually a success) + +*ALL* is the default behavior so it can be omitted if this is what you need. The example above +is example the same as below: + +{% highlight yaml %} +third_step: + title: Third step + when: + steps: + - name: first_step + on: + - success + - name: second_step + on: + - finished +{% endhighlight %} + +Codefresh also allows you to define *ANY* behavior in an explicit manner: + +{% highlight yaml %} +third_step: + title: Third step + when: + steps: + any: + - name: first_step + on: + - success + - name: second_step + on: + - finished +{% endhighlight %} + +Here the third step will run when either the first one *OR* the second one have finished. + +As an example let's assume this time that we have: + +1. A build step that creates a docker image +1. Unit tests that will run when the docker image is ready +1. Integration tests that run either after unit tests or if the docker image is ready (contrived example) +1. A cleanup step that runs when both kinds of tests are finished + +Here is the full pipeline + +`YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +mode: parallel +steps: + MyAppDockerImage: + title: Building Docker Image + type: build + image_name: my-node-js-app + working_directory: ./ + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + dockerfile: Dockerfile + MyUnitTests: + title: Running unit tests + image: ${{MyAppDockerImage}} + fail_fast: false + commands: + - npm run test + when: + steps: + - name: MyAppDockerImage + on: + - success + MyIntegrationTests: + title: Running integration tests + image: ${{MyAppDockerImage}} + commands: + - npm run integration-test + when: + steps: + any: + - name: MyUnitTests + on: + - finished + - name: MyAppDockerImage + on: + - success + MyCleanupPhase: + title: Cleanup unit test results + image: alpine + commands: + - ./cleanup.sh + when: + steps: + all: + - name: MyUnitTests + on: + - finished + - name: MyIntegrationTests + on: + - finished +{% endraw %} +{% endhighlight %} + +In this case Codefresh will make sure that cleanup happens only when both unit and integration tests are finished. + + +### Custom step dependencies + +For maximum flexibility you can define a custom condition for a step. + +It is hard to describe all possible cases, because Codefresh supports a [mini DSL]({{site.baseurl}}/docs/pipelines/conditional-execution-of-steps/#condition-expression-syntax) for conditions. All examples mentioned in conditional execution are still valid in parallel pipelines. + +For example, run this step only if a PR is opened against the production branch: + +{% highlight yaml %} +{% raw %} +my_step: + title: My step + when: + condition: + all: + validateTargetBranch: '"${{CF_PULL_REQUEST_TARGET}}" == "production"' + validatePRAction: '''${{CF_PULL_REQUEST_ACTION}}'' == ''opened''' +{% endraw %} +{% endhighlight %} + +Run this step only for the master branch and when the commit message does not include "skip ci": + +{% highlight yaml %} +{% raw %} +my_step: + title: My step + when: + condition: + all: + noSkipCiInCommitMessage: 'includes(lower("${{CF_COMMIT_MESSAGE}}"), "skip ci") == false' + masterBranch: '"${{CF_BRANCH}}" == "master"' +{% endraw %} +{% endhighlight %} + +You can now add extra conditions regarding the completion state of specific steps. A global object called `steps` contains all steps by name along with a `result` property with the following possible completion states: + +* Success +* Failure +* Skipped (only valid in sequential mode) +* Finished (regardless of status) +* Pending +* Running + +Finished is a shorthand for `success` or `failure` or `skipped`. It is only valid when used in [step dependencies]({{site.baseurl}}/docs/codefresh-yaml/advanced-workflows/#single-step-dependencies), and cannot be used in custom conditions. + +You can mix and match completion states from any other step in your pipeline. Here are some examples: + +{% highlight yaml %} +my_step: + title: My step + when: + condition: + all: + myCondition: steps.MyUnitTests.result == 'failure' || steps.MyIntegrationTests.result == 'failure' +{% endhighlight %} + +{% highlight yaml %} +my_step: + title: My step + when: + condition: + any: + myCondition: steps.MyLoadTesting.result == 'success' + myOtherCondition: steps.MyCleanupStep.result == 'success' +{% endhighlight %} + +You can also use conditions in the success criteria for a parallel step. Here is an example + +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: +- start +- tests +- cleanup +steps: + MyAppDockerImage: + stage: 'start' + title: Building Docker Image + type: build + image_name: my-full-stack-app + working_directory: ./01_sequential/ + tag: '${{CF_BRANCH_TAG_NORMALIZED}}' + dockerfile: Dockerfile + MyTestingPhases: + type: parallel + stage: 'tests' + success_criteria: + condition: + all: + myCondition: ${{steps.my_back_end_tests.result}} === 'success' && ${{steps.my_front_end_tests.result}} === 'success' + steps: + my_back_end_tests: + title: Running Back end tests + image: ${{MyAppDockerImage}} + commands: + - exit 1 + my_front_end_tests: + title: Running Front End tests + image: ${{MyAppDockerImage}} + commands: + - echo "Second" + MyCleanupPhase: + stage: 'cleanup' + title: Cleanup unit test results + image: alpine + commands: + - echo "Finished" +{% endraw %} +{% endhighlight %} + + +## Handling error conditions in a pipeline + +It is important to understand the capabilities offered by Codefresh when it comes to error handling. You have several options in different levels of granularity to select what constitutes a failure and what not. + +By default, *any* failed step in a pipeline will abort the whole pipeline and mark it as failure. + +You can use the directive `fail_fast: false`: +* In a specific step to mark it as ignored if it fails +* At the root level of the pipeline if you want to apply it to all steps + +Therefore, if you want your pipeline to keep running to completion regardless of errors the following syntax is possible: + +``` +version: '1.0' +fail_fast: false +steps: +[...] +``` + +You also have the capability to define special steps that will run when the whole pipeline has a special completion status. Codefresh offers a special object called `workflow` that represents the whole pipeline and allows you to evaluate its status in a step. + +For example, you can have a cleanup step that will run only if the workflow fails (regardless of the actual step that created the error) with the following syntax: + +{% highlight yaml %} +my_cleanup_step: + title: My Pipeline Cleanup + when: + condition: + all: + myCondition: workflow.result == 'failure' +{% endhighlight %} + +As another example we have a special step that will send an email if the pipeline succeeds or if load-tests fail: + +{% highlight yaml %} +my_email_step: + title: My Email step + when: + condition: + any: + myCondition: workflow.result == 'success' + myTestCondition: steps.MyLoadTesting.result == 'failure' +{% endhighlight %} + +Notice that both examples assume that `fail_fast: false` is at the root of the `codefresh.yaml` file. + +The possible values for `workflow.result` are: + +* `running` +* `terminated` +* `failure` +* `pending-approval` +* `success` + + +## Related articles +[Variables in pipelines]({{site.baseurl}}/docs/pipelines/variables/) +[Hooks in pipelines]({{site.baseurl}}/docs/pipelines/hooks/) + + + + + + + + diff --git a/_docs/pipelines/annotations.md b/_docs/pipelines/annotations.md new file mode 100644 index 000000000..036bf41e5 --- /dev/null +++ b/_docs/pipelines/annotations.md @@ -0,0 +1,301 @@ +--- +title: "Annotations in CI pipelines" +description: "Mark your builds and projects with extra annotations" +group: pipelines +toc: true +--- + +Codefresh supports the annotations of several entities with custom annotations. You can use these annotations to store any optional information that you wish to keep associated with each entity. Examples would be storing the test coverage for a particular build, or a special settings file for a pipeline. + +Currently Codefresh supports extra annotations for: + +* Projects +* Pipelines +* Builds +* Docker images + +You can view/edit annotations using the [Codefresh CLI](https://codefresh-io.github.io/cli/annotations/) or directly in the Codefresh Web UI. + +>Notice that the syntax shown in this page is deprecated but still supported. For the new syntax +see [hooks]({{site.baseurl}}/docs/pipelines/hooks/). + + +## Adding annotations + +In the most basic scenario you can use the [post operations]({{site.baseurl}}/docs/pipelines/post-step-operations/) of any Codefresh [step]({{site.baseurl}}/docs/pipelines/steps/) to add annotations: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + my_custom_step: + title: Adding annotations to a project + image: alpine:3.9 + commands: + - echo "Hello" + on_success: + annotations: + set: + - entity_id: annotate-examples + entity_type: project + annotations: + - my_annotation_example1: 10.45 + - my_empty_annotation + - my_string_annotation: Hello World +{% endraw %} +{% endhighlight %} + + +This pipeline adds three annotations to a project called `annotate-examples`. The name of each annotation can only contain letters (upper and lowercase), numbers and the underscore character. The name of each annotation must start with a letter. + + +For the `entity_id` value you can also use an actual ID instead of a name. The `entity_id` and `entity_type` are define which entity will hold the annotations. The possible entity types are: + +* `project` (for a project, even a different one) +* `pipeline` (for a pipeline, even a different one) +* `build` (for a build, even a different one) +* `image` (for a docker image) + +If you don't define them, then by default the current build will be used with these values: +* `entity_id` is `{% raw %}${{CF_BUILD_ID}}{% endraw %}` (i.e. the current build) +* `entity_type` is `build` + +Here is another example where we add annotations to another pipeline as well as another build (instead of the current one) + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + my_custom_step: + title: Adding annotations to multiple entities + image: alpine:3.9 + commands: + - echo "Hello" + on_success: + annotations: + set: + - entity_id: my-project/my-basic-pipeline + entity_type: pipeline + annotations: + - my_annotation_example1: 10.45 + - my_empty_annotation + - my_string_annotation: Hello World + - entity_id: 5ce2a0e869e2ed0a60c1e203 + entity_type: build + annotations: + - my_coverage: 70 + - my_url_example: http://www.example.com +{% endraw %} +{% endhighlight %} + +It is therefore possible to store annotations on any Codefresh entity (and not just the ones that are connected to the build that is adding annotations). + +## Viewing/editing annotations + +You can view the annotations using the Codefresh CLI + +```shell +codefresh get annotation project annotate-examples +``` + +You can also view annotations within the Codefresh UI. + +For build annotations click the *Annotations* on the build details: + +{% include +image.html +lightbox="true" +file="/images/pipeline/codefresh-yaml/annotations/view-build-annotations.png" +url="/images/pipeline/codefresh-yaml/annotations/view-build-annotations.png" +alt="Viewing Build annotations" +caption="Viewing Build annotations" +max-width="80%" +%} + +For pipeline annotations click the *Annotations* button in the pipeline list view: + +{% include +image.html +lightbox="true" +file="/images/pipeline/codefresh-yaml/annotations/view-pipeline-annotations.png" +url="/images/pipeline/codefresh-yaml/annotations/view-pipeline-annotations.png" +alt="Viewing Pipeline annotations" +caption="Viewing Pipeline annotations" +max-width="80%" +%} + +For project annotations click the *Annotations* button in the project list view: + +{% include +image.html +lightbox="true" +file="/images/pipeline/codefresh-yaml/annotations/view-project-annotations.png" +url="/images/pipeline/codefresh-yaml/annotations/view-build-annotations.png" +alt="Viewing project annotations" +caption="Viewing project annotations" +max-width="80%" +%} + +In all cases you will see a dialog with all existing annotations. + + +{% include +image.html +lightbox="true" +file="/images/pipeline/codefresh-yaml/annotations/edit-project-annotations.png" +url="/images/pipeline/codefresh-yaml/annotations/edit-project-annotations.png" +alt="Editing annotations" +caption="Editing annotations" +max-width="50%" +%} + +You can add additional annotations manually by clicking the *Add annotation* button and entering: + +* The name of the annotation +* The type of the annotation (text, number, percentage, link, boolean) +* The desired value + +Click *Save* to apply your changes. + +## Complex annotation values + +Apart from scalar values, you can also store more complex expressions in annotations. You have access to all [Codefresh variables]({{site.baseurl}}/docs/pipelines/variables/), text files from the build and even evaluations from the [expression syntax]({{site.baseurl}}/docs/pipelines/condition-expression-syntax/). + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'kostis-codefresh/nestjs-example' + revision: '${{CF_REVISION}}' + my_custom_step: + title: Complex annotations + image: alpine:3.9 + commands: + - echo "Hello" + - echo "Sample content" > /tmp/my-file.txt + on_finish: + annotations: + set: + - entity_id: annotate-examples/simple + entity_type: pipeline + annotations: + - qa: pending + - commit_message: ${{CF_COMMIT_MESSAGE}} + - is_main_branch: + evaluate: "'${{CF_BRANCH}}' == 'main'" + - my_json_file: "file:/tmp/my-file.txt" + - my_docker_file: "file:Dockerfile" +{% endraw %} +{% endhighlight %} + +>Notice that this pipeline is using dynamic git repository variables, so it must be linked to a least one [git trigger]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/) in order to work. + +The last two annotations add the text of a file as a value. You can define an absolute or relative path. No processing is done on the file before being stored. If a file is not found, the annotation will still be added verbatim. +We suggest you only store small text files in this manner as annotations values. + +## Removing annotations + +You can also remove annotations by mentioning their name: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + my_custom_step: + title: Adding annotations to a pipeline + image: alpine:3.9 + commands: + - echo "Hello" + on_success: + annotations: + set: + - entity_id: my-project/my-basic-pipeline + entity_type: pipeline + annotations: + - my_annotation_example1: 10.45 + - my_empty_annotation + - my_string_annotation: Hello World + - my_second_annotation: This one will stay + my_unit_tests: + title: Removing annotations + image: alpine:3.9 + commands: + - echo "Tests failed" + - exit 1 + on_fail: + annotations: + unset: + - entity_id: my-project/my-basic-pipeline + entity_type: pipeline + annotations: + - my_annotation_example1 + - my_empty_annotation + - my_string_annotation +{% endraw %} +{% endhighlight %} + +You can also use both `unset` and `set` block in a single `annotations` block. And of course, you can remove annotations from multiple entities. + +The `unset` annotation can be used with all post-step operations (`on_success`, `on_fail`, `on_finish`). + + +## Adding annotations to the current build/image + +As a convenience feature: + +1. If your pipeline has a build step +1. If you want to add annotations to the present build or image + +you can also define annotations in the root level of the build step and not mention the entity id and type. Annotations will then be added in the present build. + + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'kostis-codefresh/nestjs-example' + revision: 'master' + MyAppDockerImage: + title: Building Docker Image + type: build + image_name: my-app-image + working_directory: ./ + tag: 'sample' + dockerfile: Dockerfile + annotations: + set: + - annotations: + - my_number_annotation: 9999 + - my_empty_annotation + - my_docker_file: "file:Dockerfile" + - my_text_annotation: simple_text +{% endraw %} +{% endhighlight %} + +After running this pipeline at least once, you can retrieve the annotations from any previous build by using the respective id: + +```shell +codefresh get annotation build 5ce26f5ff2ed0edd561fa2fc +``` + +You can also define `entity_type` as `image` and don't enter any `entity_id`. In this case the image created from the build step will be annotated. + + +Note that this syntax is optional. You can still define annotations for a build/image or any other entity using the post operations of any step by mentioning explicitly the target id and type. + +## Related articles +[Image annotations]({{site.baseurl}}/docs/docker-registries/metadata-annotations/) +[Post-step operations]({{site.baseurl}}/docs/pipelines/post-step-operations/) +[Creating CI pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[Hooks in CI pipelines]({{site.baseurl}}/docs/pipelines/hooks/) diff --git a/_docs/pipelines/condition-expression-syntax.md b/_docs/pipelines/condition-expression-syntax.md new file mode 100644 index 000000000..322549f3c --- /dev/null +++ b/_docs/pipelines/condition-expression-syntax.md @@ -0,0 +1,107 @@ +--- +title: "Condition Expression Syntax" +description: "Condition expressions can be included in each step in your codefresh.yml, and must be satisfied for the step to execute." +group: pipelines +redirect_from: + - /docs/condition-expression-syntax/ + - /docs/codefresh-yaml/expression-condition-syntax/ +toc: true +--- +Each step in `codefresh.yml` file can contain conditions expressions that must be satisfied for the step to execute. + +This is a small example of where a condition expression can be used: + `YAML` +{% highlight yaml %} +step-name: + description: Step description + image: image/id + commands: + - bash-command1 + - bash-command2 + when: + condition: + all: + executeForMasterBranch: "{% raw %}'${{CF_BRANCH}}{% endraw %}' == 'master'" +{% endhighlight %} + +A condition expression is a basic expression that is evaluated to true/false (to decide whether to execute or not to execute), and can have the following syntax: + +### Types + +{: .table .table-bordered .table-hover} +| Type | True/False Examples | True/False | +| ------- | ----------------------------------------- | --------------| +| String | True: "hello"
False: "" | {::nomarkdown}- String with content = true
- Empty string = false
- String with content = true
True: 3.4
True: 1.79E+308 | {::nomarkdown}- Any number other than 0 = true.
- 0 = false
False: false | {::nomarkdown}- True = true
- False = false
`Number('hello')` is invalid | +| Boolean | 0: number or string | Boolean of input value. | `Boolean('123') == true`
`Boolean('') == false`
`Boolean(583) == true`
`Boolean(0) == false` | +| round | 0: number | Rounded number. | `round(1.3) == 1`
`round(1.95) == 2` | +| floor | 0: number | Number rounded to floor. | `floor(1.3) == 1`
`floor(1.95) == 1` | +| upper | 0: string | String in upper case. | `upper('hello') == 'HELLO'` | +| lower | 0: string | String in lower case. | `lower('BYE BYE') == 'bye bye'` | +| trim | 0: string | Trimmed string. | `trim(" abc ") == "abc"` | +| trimLeft | 0: string | Left-trimmed string. | `trimLeft(" abc ") == "abc "`| +| trimRight | 0: string | Right-trimmed string. | `trimRight(" abc ") == " abc"` | +| replace | 0: string - main string
1: string - substring to find
2: string - substring to replace | Replace all instances of the sub-string (1) in the main string (0) with the sub-string (2). | `replace('hello there', 'e', 'a') == 'hallo thara'`| +| substring | 0: string - main string
1: string - index to start
2: string - index to end | Returns a sub-string of a string. | `substring("hello world", 6, 11) == "world"` | +| length | string | Length of a string. | `length("gump") == 4` | +| includes | 0: string - main string
1: string - string to search for | Whether a search string is located within the main string. | `includes("codefresh", "odef") == true` | +| indexOf | 0: string - main string
1: string - string to search for | Index of a search string if it is found inside the main string | `indexOf("codefresh", "odef") == 1` | +| match | 0: string - main string
1: string - regular expression string, [JS style](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions) (Note: in JS strings, the backslash `\` is an escape character so in order to use a literal backslash, you need to escape it. For example: `"^\\d+$"` instead of `"^\d+$"`)
2: boolean - ignore case | Search for a regular expression inside a string, ignoring or not ignoring case | `match("hello there you", "..ll.", false) == true`
`match("hello there you", "..LL.", false) == false`
`match("hello there you", "hell$", true) == false`
`match("hello there you", "^hell", true) == true`
`match("hello there you", "bye", false) == false` | +| Variable | string | Search for the value of a variable | `Variable('some-clone')` | +| Member | 0: string - variable name
1: string - member name | Search for the value of a variable member | `Member('some-clone', 'working-directory')` | + +## What to read next + +* [Conditional Execution of Steps]({{site.baseurl}}/docs/codefresh-yaml/conditional-execution-of-steps/) +* [Condition Expression Syntax]({{site.baseurl}}/docs/codefresh-yaml/condition-expression-syntax/) +* [Working Directories]({{site.baseurl}}/docs/codefresh-yaml/working-directories/) +* [Annotations]({{site.baseurl}}/docs/codefresh-yaml/annotations/) +* [Pipeline/Step hooks]({{site.baseurl}}/docs/codefresh-yaml/hooks/) diff --git a/_docs/pipelines/conditional-execution-of-steps.md b/_docs/pipelines/conditional-execution-of-steps.md new file mode 100644 index 000000000..ef0928317 --- /dev/null +++ b/_docs/pipelines/conditional-execution-of-steps.md @@ -0,0 +1,248 @@ +--- +title: "Conditional execution of steps" +description: "Skip specific pipeline steps according to one or more conditions" +group: pipelines +sub_group: steps +redirect_from: + - /docs/conditional-execution-of-steps/ +toc: true +--- +For each step in a `codefresh.yml` file, you can define a set of conditions which need to be satisfied in order to execute the step. (An introduction to the `codefresh.yml` file can be found [here]({{site.baseurl}}/docs/codefresh-yaml/what-is-the-codefresh-yaml/).) + +There are currently two main methods to define conditions: +* Branch conditions +* Expression conditions + +## Branch Conditions + +Usually, you'll want to define a branch condition, be it of the type ```ignore``` for blacklisting a set of branches or of the type ```only``` for allowlisting a set of branches. Each branch specification can either be an exact branch name, e.g. ```master```, or a regular expression, e.g. ```/hotfix$/```. Case insensitive regexps (```/^FB-/i```) are also supported. + +Here are some examples: + +Only execute for the ```master``` branch: + + `only-master-branch.yml` +{% highlight yaml %} +build-step: + description: Building the image. + type: build + dockerfile: Dockerfile + image-name: someRepo/someUser + when: + branch: + only: + - master +{% endhighlight %} + +Only execute for branches whose name begins with ```FB-``` prefix (feature branches): + + `only-feature-branches.yml` +{% highlight yaml %} +build-step: + description: Building the image. + type: build + dockerfile: Dockerfile + image-name: someRepo/someUser + when: + branch: + only: + - /^FB-.*/i +{% endhighlight %} + +Ignore the develop branch and master branch: + + `ignore-master-and-develop-branch.yml` +{% highlight yaml %} +build-step: + description: Building the image. + type: build + dockerfile: Dockerfile + image-name: someRepo/someUser + when: + branch: + ignore: + - master + - develop +{% endhighlight %} + + +>We use [JavaScript regular expressions](https://developer.mozilla.org/en/docs/Web/JavaScript/Guide/Regular_Expressions) for the syntax in branch conditions. + + +## Condition expressions + +Alternatively, you can use more advanced condition expressions. + +This follows the standard [condition expression syntax](#condition-expression-syntax). In this case, you can choose to execute if ```all``` expression conditions evaluate to ```true```, or to execute if ```any``` expression conditions evaluate to ```true```. + +> Note: Use "" around variables with text to avoid errors in processing the conditions. Example: "${{CF_COMMIT_MESSAGE}}" + +Here are some examples. Execute if the string ```[skip ci]``` is not part of the main repository commit message AND if the branch is ```master``` + + `all-conditions.yml` +{% highlight yaml %} +build-step: + description: Building the image. + type: build + dockerfile: Dockerfile + image-name: someRepo/someUser + when: + condition: + all: + noSkipCiInCommitMessage: 'includes(lower({% raw %}"${{CF_COMMIT_MESSAGE}}"{% endraw %}), "skip ci") == false' + masterBranch: '{% raw %}"${{CF_BRANCH}}{% endraw %}" == "master"' +{% endhighlight %} + +Execute if the string ```[skip ci]``` is not part of the main repository commit message, OR if the branch is not a feature branch (i.e. name starts with FB-) + + `any-condition.yml` +{% highlight yaml %} +build-step: + description: Building the image. + type: build + dockerfile: Dockerfile + image-name: someRepo/someUser + when: + condition: + any: + noSkipCiInCommitMessage: 'includes(lower({% raw %}"${{CF_COMMIT_MESSAGE}}"{% endraw %}), "skip ci") == false' + notFeatureBranch: 'match({% raw %}"${{CF_BRANCH}}"{% endraw %}, "^FB-", true) == false' +{% endhighlight %} + +Each step in `codefresh.yml` file can contain conditions expressions that must be satisfied for the step to execute. + +This is a small example of where a condition expression can be used: + `YAML` +{% highlight yaml %} +step-name: + description: Step description + image: image/id + commands: + - bash-command1 + - bash-command2 + when: + condition: + all: + executeForMasterBranch: "{% raw %}'${{CF_BRANCH}}{% endraw %}' == 'master'" +{% endhighlight %} + +### Condition expression syntax +A condition expression is a basic expression that is evaluated to true/false (to decide whether to execute or not to execute), and can have the following syntax: + +#### Types + +{: .table .table-bordered .table-hover} +| Type | True/False Examples | True/False | +| ------- | ----------------------------------------- | --------------| +| String | True: "hello"
False: "" | {::nomarkdown}- String with content = true
- Empty string = false
- String with content = true
True: 3.4
True: 1.79E+308 | {::nomarkdown}- Any number other than 0 = true.
- 0 = false
False: false | {::nomarkdown}- True = true
- False = false
`Number('hello')` is invalid | +| Boolean | 0: number or string | Boolean of input value. | `Boolean('123') == true`
`Boolean('') == false`
`Boolean(583) == true`
`Boolean(0) == false` | +| round | 0: number | Rounded number. | `round(1.3) == 1`
`round(1.95) == 2` | +| floor | 0: number | Number rounded to floor. | `floor(1.3) == 1`
`floor(1.95) == 1` | +| upper | 0: string | String in upper case. | `upper('hello') == 'HELLO'` | +| lower | 0: string | String in lower case. | `lower('BYE BYE') == 'bye bye'` | +| trim | 0: string | Trimmed string. | `trim(" abc ") == "abc"` | +| trimLeft | 0: string | Left-trimmed string. | `trimLeft(" abc ") == "abc "`| +| trimRight | 0: string | Right-trimmed string. | `trimRight(" abc ") == " abc"` | +| replace | 0: string - main string
1: string - substring to find
2: string - substring to replace | Replace all instances of the sub-string (1) in the main string (0) with the sub-string (2). | `replace('hello there', 'e', 'a') == 'hallo thara'`| +| substring | 0: string - main string
1: string - index to start
2: string - index to end | Returns a sub-string of a string. | `substring("hello world", 6, 11) == "world"` | +| length | string | Length of a string. | `length("gump") == 4` | +| includes | 0: string - main string
1: string - string to search for | Whether a search string is located within the main string. | `includes("codefresh", "odef") == true` | +| indexOf | 0: string - main string
1: string - string to search for | Index of a search string if it is found inside the main string | `indexOf("codefresh", "odef") == 1` | +| match | 0: string - main string
1: string - regular expression string, [JS style](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions) (Note: in JS strings, the backslash `\` is an escape character so in order to use a literal backslash, you need to escape it. For example: `"^\\d+$"` instead of `"^\d+$"`)
2: boolean - ignore case | Search for a regular expression inside a string, ignoring or not ignoring case | `match("hello there you", "..ll.", false) == true`
`match("hello there you", "..LL.", false) == false`
`match("hello there you", "hell$", true) == false`
`match("hello there you", "^hell", true) == true`
`match("hello there you", "bye", false) == false` | +| Variable | string | Search for the value of a variable | `Variable('some-clone')` | +| Member | 0: string - variable name
1: string - member name | Search for the value of a variable member | `Member('some-clone', 'working-directory')` | + +## Execute steps according to the presence of a variable + +If a variable does not exist in a Codefresh pipeline, then it will simply stay as a string inside the definition. When the `{% raw %}${{MY_VAR}}{% endraw %}` variable is not available, the engine will literally print `{% raw %}${{MY_VAR}}{% endraw %}`, because that variable doesn't exist. + +You can use this mechanism to decide which steps will be executed if a [variable]({{site.baseurl}}/docs/pipelines/variables/) exists or not. + + + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +steps: + step1: + title: "Running if variable exists" + type: "freestyle" + image: "alpine:3.9" + commands: + - echo "Step 1 is running" + when: + condition: + all: + whenVarExists: 'includes("${{MY_VAR}}", "{{MY_VAR}}") == false' + step2: + title: "Running if variable does not exist" + type: "freestyle" + image: "alpine:3.9" + commands: + - echo "Step 2 is running" + when: + condition: + all: + whenVarIsMissing: 'includes("${{MY_VAR}}", "{{MY_VAR}}") == true' +{% endraw %} +{% endhighlight %} + +Try running the pipeline above and see how it behaves when a variable called `MY_VAR` exists (or doesn't exist). + +>Notice that if you use this pattern a lot it means that you are trying to create a complex pipeline that is very smart. We suggest you create instead multiple [simple pipelines for the same project]({{site.baseurl}}/docs/ci-cd-guides/pull-request-branches/#trunk-based-development). + +## Related articles +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Variables]({{site.baseurl}}/docs/pipelines/variables/) +[Pull Requests and Branches]({{site.baseurl}}/docs/ci-cd-guides/pull-request-branches/) +[Pipeline/Step hooks]({{site.baseurl}}/docs/pipelines/hooks/) diff --git a/_docs/pipelines/configuration/build-status.md b/_docs/pipelines/configuration/build-status.md new file mode 100644 index 000000000..f30776a68 --- /dev/null +++ b/_docs/pipelines/configuration/build-status.md @@ -0,0 +1,151 @@ +--- +title: "Public logs and status badges" +description: "Embedding Status Images and viewing public logs" +group: pipelines +sub_group: configuration +toc: true +redirect_from: + - /docs/build-status + - /docs/build-status/ + - /docs/build-badges-1 + - /docs/build-badges-1/ +--- + + +Badges are simple images that show you the last build status. They support both the pipeline and branch service status. +The badges can be embedded into your repository’s `readme.md` file or any other website. + +Here is an example: + +{% include +image.html +lightbox="true" +file="/images/pipeline/badges/badge.png" +url="/images/pipeline/badges/badge.png" +alt="Build badge example" +caption="Build badge example" +max-width="80%" +%} + +Clicking the badge takes you into the build view of the pipeline. + +## Finding the build badge of your project + +In the pipeline view of a project, select the *Settings* tab and then click *General*. Next to the *badges* section you will find a link to the build badge. + +{% include +image.html +lightbox="true" +file="/images/pipeline/badges/get-build-badge.png" +url="/images/pipeline/badges/get-build-badge.png" +alt="Build badge setup" +caption="Build badge setup" +max-width="80%" +%} + +Click on it and you will get a new dialog where you can select + + * The graphical style of the badge (two styles are offered) + * The syntax for the badge + +{% include + image.html + lightbox="true" + file="/images/a0c4aed-codefresh_badges_2.png" + url="/images/a0c4aed-codefresh_badges_2.png" + alt="Codefresh badges syntax" + caption="Codefresh badges syntax" + max-width="70%" + %} + + The following embedding options are available: + + * Markdown for usage in text files (e.g. `README.MD`) + * Plain HTML for normal websites + * AsciiDoc for documentation pages + * Image for any other document type + + +Copy the snippet in your clipboard. + +## Using the build badge + +Paste the snippet in the file/document where you want the badge to be visible (e.g. in a Readme file in GitHub). + +For example, the markdown syntax is + +``` +[]( URL_TO_PIPELINE ) +``` + +You can also manually change the parameters of the link by using +`https://g.codefresh.io/api/badges/build?*param1*=xxx&*param2*=yyy`\\ +when *param1*, *param2*, etc... are the parameters from the table below. + +{: .table .table-bordered .table-hover} +| Query parameter | Description | +| -----------------------|--------------------------------------------------------- | +| **branch** - optional | Name of the branch
If not supplied, default is master | +| **repoName** | Name of the repository | +| **pipelineName** | Name of the pipeline | +| **accountName** | Name of the account | +| **repoOwner** | The name of the repository owner | +| **key** - optional | Token related to the account | +| **type** - optional | Badge types
cf-1:  - also the default badge.
cf-2:  | + +Everybody who looks at your readme file will also see the current build status of the associated Codefresh pipeline. + +## Public build logs + +By default, even though the badge shows the build status for everybody, clicking the badge allows only Codefresh registered users that also have access to the pipeline to view the actual builds. + +If you are working on an open-source project and wish for greater visibility, you can enable public logs (and associated badge) for your project so that any user can see the pipeline results (even if they are not logged into Codefresh). + +Public logs are disabled by default and you need to explicitly enable them. + +>This happens for security reasons. Make sure that the logs you are exposing to the Internet do not have any sensitive information. If you are unsure, you can still use the private badge that shows project status only as explained in the previous section. + +To enable the public logs, toggle the respective switch in the pipeline settings: + +{% include +image.html +lightbox="true" +file="/images/pipeline/badges/toggle-public-logs.png" +url="/images/pipeline/badges/toggle-public-logs.png" +alt="Enabling public logs" +caption="Enabling public logs" +max-width="80%" +%} + +Then click the *Save* button to apply changes for your pipeline. Once that is done you will also get a second badge (public) as well as the public URL to your project. + +{% include +image.html +lightbox="true" +file="/images/pipeline/badges/get-public-url.png" +url="/images/pipeline/badges/get-public-url.png" +alt="Getting the public URL log view" +caption="Getting the public URL log view" +max-width="70%" +%} + +Now you can use this badge and/or public URL anywhere and all users can view your logs without being logged into Codefresh at all (or having access to your pipeline). + +{% include +image.html +lightbox="true" +file="/images/pipeline/badges/view-public-logs.png" +url="/images/pipeline/badges/view-public-logs.png" +alt="Public logs" +caption="Public logs" +max-width="90%" +%} + +Your visitors can also click on each individual pipeline step and see the logs for that step only. + +If you are using Codefresh to manage a public project, you should also use the capability to [trigger builds from external forks]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/#support-for-building-pull-requests-from-forks). + +## Related articles +[Introduction to Codefresh pipelines]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines) +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[Monitoring pipelines]({{site.baseurl}}/docs/pipelines/monitoring-pipelines/) diff --git a/_docs/pipelines/configuration/pipeline-settings.md b/_docs/pipelines/configuration/pipeline-settings.md new file mode 100644 index 000000000..b9c78e298 --- /dev/null +++ b/_docs/pipelines/configuration/pipeline-settings.md @@ -0,0 +1,87 @@ +--- +title: "Global settings for pipelines" +description: "Define global options for pipeline templates, yaml sources and approval behavior" +group: pipelines +sub_group: configuration +toc: true +--- + +To access your global pipeline settings navigate to [https://g.codefresh.io/account-admin/account-conf/pipeline-settings](https://g.codefresh.io/account-admin/account-conf/pipeline-settings) or click on *Account settings* on the left sidebar and then choose *Pipeline settings* item on the next screen. + +On this page, you can define global parameters for the whole Codefresh account regarding pipeline options. Users can still override some of these options for individual pipelines. + +{% include image.html +lightbox="true" +file="/images/pipeline/pipeline-settings/pipeline-settings-ui.png" +url="/images/pipeline/pipeline-settings/pipeline-settings-ui.png" +alt="Pipeline settings" +caption="Pipeline settings" +max-width="80%" +%} + + +## Pause pipeline executions + +Pause builds for pipelines at the account level, for example, during maintenance. + +* **Pause build execution** is disabled by default. +* When enabled: + * New pipelines in the account are paused immediately. + * Existing pipelines with running builds are paused only after the builds have completed execution. +* Paused pipelines are set to status Pending, and remain in this status until **Pause build execution** is manually disabled for the account. + +{% include image.html +lightbox="true" +file="/images/pipeline/pipeline-settings/pause-pipeline-enabled.png" +url="/images/pipeline/pipeline-settings/pause-pipeline-enabled.png" +alt="Pause Build Execution pipeline setting enabled" +caption="Pause Build Execution pipeline setting enabled" +max-width="80%" +%} + +## Template section + +Here you can define global template behavior. The options are: + +* Enable [pipeline templates]({{site.baseurl}}/docs/docs/pipelines/pipelines/#using-pipeline-templates) for users. If this is enabled some pipelines can be marked as templates and users can still select them when creating a new pipeline. +* Decide if users can clone an existing pipeline (along with its triggers and associated parameters) when [creating a new pipeline]({{site.baseurl}}/docs/docs/pipelines/pipelines/#creating-new-pipelines). + +Note that templates are simply normal pipelines “marked” as a template. There is no technical difference between templates and actual pipelines. + +## Pipeline YAML section + +Here you can restrict the sources of pipeline YAML that users can select. The options are: + +* Enable/Disable the [inline editor]({{site.baseurl}}/docs/docs/pipelines/pipelines/#using-the-inline-pipeline-editor) where YAML is stored in Codefresh SaaS +* Enable/disable pipeline YAML from connected Git repositories +* Enable/disable pipeline YAML from [external URLs]({{site.baseurl}}/docs/docs/pipelines/pipelines/#loading-codefreshyml-from-version-control) + +You need to allow at least one of these options so that users can create new pipelines. We suggest leaving the first option enabled when users are still learning about Codefresh and want to experiment. + +## Advanced pipeline options + +Here you can set the defaults for advanced pipeline behavior. The options are: + +* [Keep or discard]({{site.baseurl}}/docs/pipelines/steps/approval/#keeping-the-shared-volume-after-an-approval) the volume when a pipeline is entering approval state +* Whether pipelines in approval state [count or not against concurrency]({{site.baseurl}}/docs/pipelines/steps/approval/#define-concurrency-limits) +* Define the [Service Account]({{site.baseurl}}/docs/integrations/docker-registries/amazon-ec2-container-registry/#setting-up-ecr-integration---service-account) for Amazon ECR integration. +* Set the default registry where all Public Marketplace Step images are pulled from. Registries listed are from the [Docker Registry]({{site.baseurl}}/docs/integrations/docker-registries/) integration page. + * Example: Public Marketplace Step image is defined to use Docker Hub. If you select a quay.io integration, all Public Marketplace Step images will be pulled from quay.io instead of Docker Hub. + * Note: This does not affect Freestyle Steps. + +Note that the first option affects pipeline resources and/or billing in the case of SaaS pricing. It will also affect users of existing pipelines that depend on this behavior. It is best to enable/disable this option only once at the beginning. + +## Default Behavior for Build Step + +Here you can decide if the build step will push images or not according to your organization’s needs. The options are: + +1. Users need to decide if an image will be pushed or not after it is built +2. All built images are automatically pushed to the default registry +3. All built images are NOT pushed anywhere by default + +Note that this behavior is simply a convenience feature for legacy pipelines. Users can still use a [push step]({{site.baseurl}}/docs/pipelines/steps/push/) in a pipeline and always push an image to a registry regardless of what was chosen in the build step. + +## Related articles +[Creating Pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Git Integration]({{site.baseurl}}/docs/integrations/git-providers/) diff --git a/_docs/pipelines/configuration/secrets-store.md b/_docs/pipelines/configuration/secrets-store.md new file mode 100644 index 000000000..cccd4a47b --- /dev/null +++ b/_docs/pipelines/configuration/secrets-store.md @@ -0,0 +1,97 @@ +--- +title: "Secrets in pipelines" +description: "Use Kubernetes secrets in Codefresh" +group: pipelines +sub_group: configuration +toc: true +--- + +Once you have [connected Codefresh to your secrets storage]({{site.baseurl}}/docs/integrations/secret-storage/), you can use them in any pipeline or UI screen. + +> Note: This feature is for Enterprise accounts only. + +## Using secrets in pipelines + +The syntax for using the secret is {% raw %}`${{secrets.NAME_IN_CODEFRESH.KEY}}`{% endraw %}. + +> If you did not include the resource-name as a part of your secret store context creation, the syntax for using your secret differs slightly: + {% raw %}${{secrets.NAME_IN_CODEFRESH.RESOURCE-NAME@KEY}}{% endraw %} + The previous KEY portion is now made of two parts separated using @, where the left side is the name of the resource in the namespace, and the right side the key in that resource. + +To use the secret in your pipeline, you have two options: + +* Define it as a pipeline variable: + +{% include +image.html +lightbox="true" +file="/images/pipeline/secrets/secrets-pipeline-var.png" +url="/images/pipeline/secrets/secrets-pipeline-var.png" +alt="Secrets Pipeline Variable" +caption="Secrets stored in Pipeline Variable" +max-width="80%" +%} + +`codefresh.yaml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + step: + type: freestyle + arguments: + image: alpine + commands: + - echo $SECRET +{% endraw %} +{% endhighlight %} + +* Use the secret directly in your YAML + +`codefresh.yaml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + step: + type: freestyle + arguments: + image: alpine + environment: + - SECRET=${{secrets.test.key1}} + commands: + - echo $SECRET +{% endraw %} +{% endhighlight %} + + +## Using secrets in the Codefresh UI + +You can also use secrets in the GUI screens that support them. Currently you can use secrets in: + +* Values in [shared configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/) +* Integration with [cloud storage]({{site.baseurl}}/docs/testing/test-reports/#connecting-your-storage-account) + +Where secret integration is supported, click on the lock icon and enable the toggle button. You will get a list of your connected secrets: + + +{% include +image.html +lightbox="true" +file="/images/pipeline/shared-configuration/shared-conf-secret-integration.png" +url="/images/pipeline/shared-configuration/shared-conf-secret-integration.png" +alt="Using external secrets in shared configuration values" +caption="Using external secrets in shared configuration values" +max-width="50%" +%} + +If you have already specified the resource field during secret definition the just enter on the text field the name of the secret directly, i.e. `my-secret-key`. +If you didn't include a resource name during secret creation then enter the full name in the field like `my-secret-resource@my-secret-key`. + + +## Related articles +[Shared Configuration]({{site.baseurl}}/docs/pipelines/shared-configuration/) +[Git triggers]({{site.baseurl}}/docs/pipelines/triggers/git-triggers/) +[Running pipelines locally]({{site.baseurl}}/docs/pipelines/running-pipelines-locally/) +[Debugging Pipelines]({{site.baseurl}}/docs//yaml-examples/examples/trigger-a-k8s-deployment-from-docker-registry/) + diff --git a/_docs/pipelines/configuration/shared-configuration.md b/_docs/pipelines/configuration/shared-configuration.md new file mode 100644 index 000000000..1485e832d --- /dev/null +++ b/_docs/pipelines/configuration/shared-configuration.md @@ -0,0 +1,265 @@ +--- +title: "Shared configuration for pipelines" +description: "How to keep your pipelines DRY" +group: pipelines +sub_group: configuration +toc: true +--- + +After creating several pipelines in Codefresh, you will start to notice several common values between them. Common examples are access tokens, environment URLs, configuration properties etc. + +Codefresh allows you to create those shared values in a central place and then reuse them in your pipelines +avoiding the use of copy-paste. + +You can share: + +* Environment parameters (easy) +* Helm values (easy) +* Any kind of YAML data (advanced) + + +## Creating shared configuration + +From the left sidebar click *Account settings* to enter your global settings. Then choose *Shared Configuration* from the left menu. + +{% include +image.html +lightbox="true" +file="/images/pipeline/shared-configuration/shared-configuration.png" +url="/images/pipeline/shared-configuration/shared-configuration.png" +alt="Creating shared configuration snippets" +caption="Creating shared configuration snippets" +max-width="50%" +%} + +You can create four types of shared configuration: + +* **Shared Configuration**: for environment variables +* **Shared Secret**: for encrypted environment variables of sensitive data (access tokens, etc.) +* **YAML**: for Helm values or any other generic information +* **Secret YAML**: for above, but encrypts the contents + +>RBAC is supported for all types of shared configurations. + +You can create as many shared snippets as you want (with unique names). + +### Using external secrets as values + +Note that the default "shared secrets" and "secret yaml" entities use the built-in secret storage of Codefresh. You can also +use any [external secrets that you have defined]({{site.baseurl}}/docs/integrations/secret-storage/) (such as Kubernetes secrets), by using the normal entities and then clicking on the lock icon that appears. + +{% include +image.html +lightbox="true" +file="/images/pipeline/shared-configuration/shared-conf-secret-integration.png" +url="/images/pipeline/shared-configuration/shared-conf-secret-integration.png" +alt="Using external secrets in shared configuration values" +caption="Using external secrets in shared configuration values" +max-width="50%" +%} + +If you have already specified the resource field during secret definition the just enter on the text field the name of the secret directly, i.e. `my-secret-key`. +If you didn't include a resource name during secret creation then enter the full name in the field like `my-secret-resource@my-secret-key`. + +### Level of access + +For each set of values you can toggle the level of access by [non-admin users]({{site.baseurl}}/docs/administration/access-control/#users-and-administrators). If it is off, users will **not** be able to use the [CLI](https://codefresh-io.github.io/cli/) or [API]({{site.baseurl}}/docs/integrations/codefresh-api/) +to access these [values](https://codefresh-io.github.io/cli/contexts/). If it is on, all users from all your Codefresh teams will be able to access this set of values +with CLI commands or API calls. + +{% include +image.html +lightbox="true" +file="/images/pipeline/shared-configuration/shared-config-access.png" +url="/images/pipeline/shared-configuration/shared-config-access.png" +alt="Allow access to non-admin users" +caption="Allow access to non-admin users" +max-width="60%" +%} + +We recommend that you disable access for all values of type *shared secret* and *secret YAML* unless your organization has different needs. + + +## Using shared environment variables + +Each pipeline has a set of environment variables that can be defined in the *Workflow* screen. +To import a shared configuration open the pipeline editor, and from the tabs on the right side select *VARIABLES*. Then click the gear icon to *Open Advanced Options*: + +{% include +image.html +lightbox="true" +file="/images/pipeline/shared-configuration/environment-variables.png" +url="/images/pipeline/shared-configuration/environment-variables.png" +alt="Pipeline environment variables" +caption="Pipeline environment variables" +max-width="50%" +%} + +To use your shared configuration, click the *Import from shared configuration* button and select the snippet from the list: + +{% include +image.html +lightbox="true" +file="/images/pipeline/shared-configuration/import-variables.png" +url="/images/pipeline/shared-configuration/import-variables.png" +alt="Importing shared configuration" +caption="Importing shared configuration" +max-width="50%" +%} + +Once you click *Add* the values from the shared configuration will be appended to the ones +you have in your pipelines. In case of similar values the shared configuration will follow the [precedence rules]({{site.baseurl}}/docs/pipelines/variables/#user-provided-variables). + + +## Using shared Helm values + +To use a shared YAML snippet for Helm values you can install a new Helm chart either from: + +* The [Helm chart list]({{site.baseurl}}/docs/new-helm/add-helm-repository/#install-chart-from-your-helm-repository) +* The [Helm environment board]({{site.baseurl}}/docs/new-helm/helm-environment-promotion/#moving-releases-between-environments). + +In both cases, when you see the Helm installation dialog you can import any of your YAML snippets +to override the default chart values. + +{% include +image.html +lightbox="true" +file="/images/pipeline/shared-configuration/helm-import.png" +url="/images/pipeline/shared-configuration/helm-import.png" +alt="Importing Helm values" +caption="Importing Helm values" +max-width="50%" +%} + +From the same dialog you can also create a brand-new shared configuration snippet of type YAML. +Not only it will be used for this Helm chart, but it will be added in your global shared configuration as well. + +## Using values from the Shared Configuration in your Helm step + +Additionally, you can define shared variables in your account settings and reuse those across your Helm steps, and specifically, in your [custom Helm values]({{site.baseurl}}/docs/docs/new-helm/using-helm-in-codefresh-pipeline/#helm-values). + +Under *Account Setting* > *Shared Configuration*, add the variable to your shared configuration. + +{% include +image.html +lightbox="true" +file="/images/pipeline/shared-configuration/helm-shared-variables.png" +url="/images/pipeline/shared-configuration/helm-version-shared.png" +alt="Adding shared configuration variables" +caption="Adding shared configuration variables" +max-width="50%" +%} + +Go to the workflow of the Codefresh pipeline to which you want to add the variable. Then select *variables* from the right sidebar. *Open advanced configuration* and select *Import from shared configuration*. + +{% include +image.html +lightbox="true" +file="/images/pipeline/shared-configuration/environment-variables.png" +url="/images/pipeline/shared-configuration/environment-variables.png" +alt="Pipeline environment variables" +caption="Pipeline environment variables" +max-width="50%" +%} + +This will allow you to add shared variables. + +{% include +image.html +lightbox="true" +file="/images/pipeline/shared-configuration/shared-helm-variables.png" +url="/images/pipeline/shared-configuration/shared-helm-variables.png" +alt="Shared helm variable" +caption="Shared helm variable" +max-width="50%" +%} + +Add the shared variables to your Helm step: + +{% highlight shell %} +{% raw %} +deploy: + type: "helm" + working_directory: "./react-article-display" + stage: "deploy" + arguments: + action: "install" + chart_name: "charts/example-chart" + release_name: "test-chart" + helm_version: "${{HELM_VERSION}}" + kube_context: "anais-cluster@codefresh-sa" + custom_values: + - 'pullPolicy=${{PULL_POLICY}}' +{% endraw %} +{% endhighlight %} + +The shared variables can now be used across your pipelines. + +## Sharing any kind of YAML data in pipelines + +All the snippets from shared configuration are also available as context in the [Codefresh CLI](https://codefresh-io.github.io/cli/contexts/) + +This means that you can manipulate them programmatically and read their values in the pipeline in any way you see fit. + +If for example you have a shared configuration named `my-global-config` you can easily read its contents programmatically using the CLI: + +{% highlight shell %} +$codefresh get context my-global-config --output=yaml + +apiVersion: v1 +kind: context +metadata: + default: false + system: false + name: my-global-config +type: config +spec: + type: config + data: + foo: bar +{% endhighlight %} + +### Example - custom value manipulation + +Let's say that you have a YAML segment with the following contents: + +{% highlight yaml %} +favorite: + drink: coffee + food: pizza +{% endhighlight %} + +Here is a pipeline step that is reading the yaml snippet and extracts a value + + `YAML` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + MyFavoriteFoodStep: + title: Favorite food + image: codefresh/cli + commands: + - echo I love eating $(codefresh get context my-food-values --output=json | jq -r '.spec.data.favorite.food') +{% endraw %} +{% endhighlight %} + +Once the pipeline runs, you will see in the logs: + +``` +I love eating pizza +``` + +## Manipulating shared configuration programmatically + +You can also create/update/delete shared configuration via the [Codefresh CLI](https://codefresh-io.github.io/cli/) or [API]({{site.baseurl}}/docs/integrations/codefresh-api/). + +See the [context section](https://codefresh-io.github.io/cli/contexts/create-context/) in the CLI documentation. + + + +## Related articles +[Variables]({{site.baseurl}}/docs/pipelines/variables/) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Pipeline steps]({{site.baseurl}}/docs/pipelines/steps/) + diff --git a/_docs/pipelines/debugging-pipelines.md b/_docs/pipelines/debugging-pipelines.md new file mode 100644 index 000000000..7e88d0e36 --- /dev/null +++ b/_docs/pipelines/debugging-pipelines.md @@ -0,0 +1,250 @@ +--- +title: "Debugging pipelines" +description: "Pause and inspect pipelines" +group: pipelines +toc: true +--- + +In addition to [running pipelines locally]({{site.baseurl}}/docs/pipelines/running-pipelines-locally/), Codefresh also allows you to debug pipelines by stopping their execution and inspecting manually their state (files, environment variables, tools etc.) + + +The Codefresh pipeline debugger works similar to your IDE debugger. You can place breakpoints on one or more pipeline steps and once the pipeline hits one of them, it will stop. You will then get a terminal like interface inside your pipeline step where you can run any commands that you wish in order to understand the state of the container. + + +{% + include image.html + lightbox="true" + file="/images/pipeline/debug/debug-session.png" + url="/images/pipeline/debug/debug-session.png" + alt="A debugging session" + caption="A debugging session" + max-width="70%" +%} + +There are several options for defining exactly when a step will stop. + +## Entering the debugger mode + +There are threes ways to enter the debugging mode in a pipeline. You can activate the debugging button when your run the pipeline: + +{% + include image.html + lightbox="true" + file="/images/pipeline/debug/run-pipeline-debug.png" + url="/images/pipeline/debug/run-pipeline-debug.png" + alt="Running a pipeline in debug mode" + caption="Running a pipeline in debug mode" + max-width="30%" +%} + +Alternatively if a pipeline is already running normally, you can enter debugging mode by clicking on the bug icon on the top right. + +{% + include image.html + lightbox="true" + file="/images/pipeline/debug/enter-debug-mode.png" + url="/images/pipeline/debug/enter-debug-mode.png" + alt="Switching to debug mode" + caption="Switching to debug mode" + max-width="60%" +%} + +You can restart a pipeline that has already finished in debug mode: + +{% + include image.html + lightbox="true" + file="/images/pipeline/debug/restart-in-debug.png" + url="/images/pipeline/debug/restart-in-debug.png" + alt="Restart in debug mode" + caption="Restart in debug mode" + max-width="70%" +%} + +Now you are ready to place breakpoints in steps. + + +## Placing breakpoints + +Once the debugging mode is active, all pipeline steps will get an extra breakpoint icon on the far right of their box. + +{% + include image.html + lightbox="true" + file="/images/pipeline/debug/breakpoint.png" + url="/images/pipeline/debug/breakpoint.png" + alt="A step breakpoint" + caption="A step breakpoint" + max-width="70%" +%} + + +You can click on this icon and define a breakpoint for this particular step. You have the following options + +* *Before* - place a breakpoint before the step is initialized +* *Override* - place a breakpoint after the step has initialized but before its execution ([freestyle steps]({{site.baseurl}}/docs/pipelines/steps/freestyle/)) +* *After* - place a breaking point after the step has finished execution. + +You can choose multiple debugging phases. In most cases the `Override` option is the most useful one. The `before` phase allows you to inspect +a pipeline step even before [service containers]({{site.baseurl}}/docs/pipelines/service-containers/) are up. + +The `after` phase is useful if you want to verify files or variables after a step has finished its execution but before the next step starts. + + +## Using the debugger terminal + +Once the pipeline reaches a step that has a breakpoint, execution will pause and a new debugger terminal will become available: + +{% + include image.html + lightbox="true" + file="/images/pipeline/debug/debug-window.png" + url="/images/pipeline/debug/debug-window.png" + alt="The debugging terminal" + caption="The debugging terminal" + max-width="60%" +%} + +You can now manually type commands to inspect your container. If your Codefresh plan has the basic debugging capabilities you can run the following commands: + +* `cd, ls` to see files +* `printenv` to see environment variables +* `cat` to read files +* `top` to see what is running +* `export` and [cf_export]({{site.baseurl}}/docs/pipelines/variables/#using-cf_export-command) to create environment variables +* `exit` to finish the debugging session + +If you have placed a breakpoint in the `override` phase of a freestyle step then the container image is the same as the one defined in the step. Therefore you can execute all tools that you have placed in the image (e.g. compilers, linters, test frameworks etc.) + +In all cases the [shared Codefresh volume]({{site.baseurl}}/docs/pipelines/introduction-to-codefresh-pipelines/#sharing-the-workspace-between-build-steps) is automounted so you can examine your source code or any other intermediate artifacts placed in your project folder or the pipeline cache. + +If the breakpoint is on a `before` or `after` phase, the command line terminal is powered by an [alpine](https://alpinelinux.org/) image. The image has already useful tools such as `wget`, `nc` and `vi`. If you have the advanced debugging capabilities in your Codefresh plan you can then install additional tools on your own directly in the terminal with [apk](https://wiki.alpinelinux.org/wiki/Alpine_Linux_package_management). Examples: + +* `apk add curl` +* `apk add nano` +* `apk add go` +* `apk add python` + +Use the command `apk search foo` to search for a package named foo. + + +## Resuming execution + +Once you are happy with your debugging session, click the continue button to resume. + +{% + include image.html + lightbox="true" + file="/images/pipeline/debug/resume-button.png" + url="/images/pipeline/debug/resume-button.png" + alt="Continue execution button" + caption="Continue execution button" + max-width="60%" +%} + +The pipeline will continue and then stop for the next breakpoint (if any). You can still revisit the debugger window for previous steps to see what debugging commands you had executed. + +>Notice that to conserve resources, there is a 15 minute limit on each open debug session. If you don't resume the pipeline within 15 minutes after hitting a breakpoint the whole pipeline will stop with a timeout error. + +It is important to understand that if you have chosen the `override` phase in a freestyle step, then the commands mentioned in the pipeline definition are completely ignored. + +## Using the alternative debug window + +If you enable the debugger on a freestyle step with the "override" option, Codefresh will install some extra tooling on the Docker image that is needed for the debugger itself. + +By default, the internal debugger tooling is using node.js, so if your image is already based on Node.js, you might get version conflicts in your application. + +You can enable an alternative debugger by passing the variable `DEBUGGER_RUNNER = 2` on the whole pipeline: + +{% + include image.html + lightbox="true" + file="/images/pipeline/debug/alternative-debugger.png" + url="/images/pipeline/debug/alternative-debugger.png" + alt="Enabling the Python based debugger" + caption="Enabling the Python based debugger" + max-width="60%" +%} + +This debugger is based on Python instead of Node.js and it can work with both Python 2 and 3 Docker images. +This way the debugger tools will not affect your application. You can also use the same method in a specific freestyle step like this: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +steps: + hello_world_step: + title: freestyle step + image: node:11.1 + environment: + - 'DEBUGGER_RUNNER=2' +{% endraw %} +{% endhighlight %} + + + + + +## Inserting breakpoints in the pipeline definition + +It is also possible to mention breakpoints in the Codefresh YAML instead of using the UI. Breakpoints mentioned in the `codefresh.yml` file have no effect when the pipeline is not running in Debug mode. You need to run the pipeline in debug mode in order for them to stop the pipeline. + +Here is the syntax: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: '1.0' +stages: + - prepare + - build + - test +steps: + main_clone: + title: Cloning main repository... + type: git-clone + repo: 'codefresh-contrib/python-flask-sample-app' + revision: 'master' + git: github + stage: prepare + MyAppDockerImage: + title: Building Docker Image + type: build + stage: build + image_name: my-app-image + working_directory: ./ + tag: 'master' + dockerfile: Dockerfile + debug: + phases: + before: true + after: false + MyUnitTests: + title: Running Unit tests + stage: test + image: '${{MyAppDockerImage}}' + debug: + phases: + before: false + override: true + after: false + commands: + - python setup.py test +{% endraw %} +{% endhighlight %} + +Once you run this pipeline in debug mode, it will automatically have breakpoints in the respective steps (but you can still override/change them using the GUI). + + +## Troubleshooting + +The debugger windows needs some extra tools in a docker image in order to work (such as the `bash` shell). Codefresh automatically installs these tools on your image without any configuration. + +If you get the message *your linux distribution is not supported* please contact us so that we can examine your docker image and make sure it is compatible with the Codefresh debugger. + + +## Related articles +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Steps in CI pipelines]({{site.baseurl}}/docs/pipelines/steps/) +[Running pipelines locally]({{site.baseurl}}/docs/pipelines/running-pipelines-locally/) diff --git a/_docs/pipelines/docker-image-metadata.md b/_docs/pipelines/docker-image-metadata.md new file mode 100644 index 000000000..e2afc96c9 --- /dev/null +++ b/_docs/pipelines/docker-image-metadata.md @@ -0,0 +1,226 @@ +--- +title: "Docker image metadata" +description: "How to use custom metadata in your Docker images" +group: pipelines +redirect_from: + - /docs/metadata-annotations/ + - /docs/docker-registries/metadata-annotations/ +toc: true +--- +Images built by Codefresh can be annotated with customized metadata. +This article explains how to create advanced view of your images and enrich them with custom metadata which perfectly fits your flow and image management process. + +{% + include image.html + lightbox="true" + file="/images/pipeline/codefresh-yaml/docker-image-metadata/metadata.png" + url="/images/pipeline/codefresh-yaml/docker-image-metadata/metadata.png" + alt="Codefresh Docker registry metadata" + max-width="65%" +%} + +>We have since expanded this feature and now you are able to add custom annotations to [pipelines and builds as well]({{site.baseurl}}/docs/pipelines/annotations/). Notice also that the syntax shown in this page is deprecated but still supported. For the new syntax +see [Hooks in pipelines]({{site.baseurl}}/docs/pipelines/hooks/). + +## Metadata types + +Images built by Codefresh can be annotated with an array of key-value metadata. +Metadata values may be of the following types: + +{: .table .table-bordered .table-hover} +| Annotation type | Guidelines | Example | +| --------------- | ------------------------------------------------ | -------------------------------------------------------- | +| String | Use string | 'Example note' | +| Number | use numeric value to set this kind of annotation | 9999 | +| Boolean | Use true / false value | true | +| Percentage bar | use 0-100 value ending with % | 85% | +| Link | use url | {% raw %}`${{CF_COMMIT_URL}}`{% endraw %} | + +You can also use [Expression evaluations]({{site.baseurl}}/docs/pipelines/conditional-execution-of-steps/#condition-expression-syntax) to set metadata. + +## Annotate your images using Codefresh YAML +You can annotate an image as part of its build process and also on post build steps. + +{:.text-secondary} +### Build step Image Metadata Annotation +You can annotate an image as part of its build process by declaring the metadata value on the [Build step]({{site.baseurl}}/docs/pipelines/steps/build/): +1. The `metadata` attribute +2. The `set` operation +3. An array of key-value metadata + + `build-metadata-annotation` +{% highlight yaml %} +build_step: + type: build + ... + metadata: # Declare the metadata attribute + set: # Specify the set operation + - qa: pending + - commit_message: {% raw %}${{CF_COMMIT_MESSAGE}}{% endraw %} + - exit_code: 0 + - is_main: + evaluate: "{% raw %}'${{CF_BRANCH}}{% endraw %}' == 'main'" +{% endhighlight %} + +{:.text-secondary} +### Adding annotations to Built images on post-build steps +Any step in the YAML workflow can annotate built images by using [Post-Step Operations]({{site.baseurl}}/docs/pipelines/post-step-operations/). +To annotate a built image, configure any step with: +1. The post-step operation +2. The `metadata` attribute +3. The `set` operation +4. A list of target images with the variable syntax of {% raw %}`${{build_step_name.imageId}}`{% endraw %} +5. An array of key-value metadata + + `annotating-step` +{% highlight yaml %} +build_step: + type: build + ... + +any_step: + ... + on_success: # Execute only once the step succeeded + metadata: # Declare the metadata attribute + set: # Specify the set operation + - {% raw %}${{build_step.imageId}}{% endraw %}: # Select any number of target images + - qa: pending + + on_fail: # Execute only once the step failed + metadata: # Declare the metadata attribute + set: # Specify the set operation + - {% raw %}${{build_step.imageId}}{% endraw %}: # Select any number of target images + - exit_code: 1 + + on_finish: # Execute in any case + metadata: # Declare the metadata attribute + set: # Specify the set operation + - {% raw %}${{build_step.imageId}}{% endraw %}: # Select any number of target images + - is_main: + evaluate: "{% raw %}'${{CF_BRANCH}}'{% endraw %} == 'main'" +{% endhighlight %} + +### Example - Quality Image Metadata Annotation +You can set a quality indicator to images to show if they passed or failed tests. An image with the boolean annotation `CF_QUALITY` set to true will have a quality indicator in the 'Images' view. + + `YAML` +{% highlight yaml %} +version: '1.0' +steps: + build_step: + type: build + image_name: myrepo/imagename + working_directory: ./ + dockerfile: Dockerfile + + unit_test: + image: {% raw %}'${{build_step}}'{% endraw %} + working_directory: IMAGE_WORK_DIR + commands: + - echo test + on_success: + metadata: + set: + - {% raw %}'${{build_step.imageId}}'{% endraw %}: + - CF_QUALITY: true + on_fail: + metadata: + set: + - {% raw %}'${{build_step.imageId}}'{% endraw %}: + - CF_QUALITY: false +{% endhighlight %} + +Image quality has 3 indicators: +* True - this image is considered a quality image (ex. passed tests), +* False - this image is not considered a quality image (ex. when tests failed but the image was already built). +* No value (nobody set the annotation) - this image has no quality indicator. + +{% include image.html lightbox="true" file="/images/pipeline/docker-image/quality-image-annotation.png" url="/images/pipeline/docker-image/quality-image-annotation.png" caption="Quality image annotation" max-width="40%" %} + + +## Viewing Image Metadata Annotations +You can view an image's metadata annotation by: +1. Navigating to the `Images` view +2. Selecting the target image +3. Selecting the `Annotations` tab + +{% + include image.html + lightbox="true" + file="/images/pipeline/codefresh-yaml/docker-image-metadata/annotations.png" + url="/images/pipeline/codefresh-yaml/docker-image-metadata/annotations.png" + alt="Image annotations" + caption="Image annotations" + max-width="65%" +%} + +In addition, you can add selected annotations to the images table on images page. To display an annotation in the image table, click on the gear icon at the top right corner of image page and then select all annotations you want to display. + +{% + include image.html + lightbox="true" + file="/images/pipeline/codefresh-yaml/docker-image-metadata/annotations-image-table.png" + url="/images/pipeline/codefresh-yaml/docker-image-metadata/annotations-image-table.png" + alt="Annotations in image table" + caption="Annotations in image table" + max-width="65%" +%} + + +## Annotating images programmatically + +It is also possible to annotate images with the [Codefresh CLI](https://codefresh-io.github.io/cli/). + +First find the id of an image that you wish to annotate with the command + +``` +codefresh get images +``` + +You can also search for a specific image by name: + +``` +$ codefresh get images --image-name custom +ID NAME TAG CREATED SIZE PULL +b5f103a87856 my-custom-docker-image bla Fri Feb 01 2019 91.01 MB r.cfcr.io/kostis-codefresh/my-custom-docker-image:bla +``` +Then once you have the ID of the image you can use the [annotate command](https://codefresh-io.github.io/cli/images/annotate-image/) to add extra metadata: + +``` +codefresh annotate image b5f103a87856 -l coverage=75 +``` + +## Using custom metadata in Codefresh pipelines + +You can also use the Codefresh CLI to fetch existing metadata from images. It is then very easy to extract and process specific fields with [yq](https://github.com/kislyuk/yq) + +Here is an example +``` +$ codefresh get image b5f103a87856 --output=yaml | yq -r .annotations.coverage +75 +``` + +You can then easily process the metadata (e.g. with scripts) and take decisions according to them. Here is an example +step that will fail the build if test coverage on an image is less than 80% + + `YAML` +{% highlight yaml %} +version: '1.0' +steps: + findLabel: + title: Get image label for coverage + image: codefresh/cli + commands: + - export MY_COVERAGE=$(codefresh get image b5f103a87856 --output=yaml | yq -r .annotations.coverage) + - echo "Coverage is $MY_COVERAGE" + - if [[ $MY_COVERAGE -lt "80" ]]; then exit 1 ; fi + +{% endhighlight %} + +The possibilities are endless as you can take any combination of image metadata and use any complex conditional +in order to process them in a Codefresh pipeline. + + +## Related articles +[External Docker Registries]({{site.baseurl}}/docs/docker-registries/external-docker-registries/) +[Accessing a Docker registry from your Kubernetes cluster]({{site.baseurl}}/docs/deploy-to-kubernetes/access-docker-registry-from-kubernetes/) diff --git a/_docs/pipelines/docker-operations.md b/_docs/pipelines/docker-operations.md deleted file mode 100644 index 4678a46d4..000000000 --- a/_docs/pipelines/docker-operations.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -title: "Using Docker" -description: "" -group: pipelines -toc: true ---- - -Coming soon diff --git a/_docs/pipelines/hooks.md b/_docs/pipelines/hooks.md new file mode 100644 index 000000000..9460bc536 --- /dev/null +++ b/_docs/pipelines/hooks.md @@ -0,0 +1,634 @@ +--- +title: "Hooks in pipelines" +description: "Execute commands before/after each pipeline or step" +group: pipelines +toc: true +--- + +Hooks in pipelines allow you to run specific actions at the end and the beginning of the pipeline, as well as before/after a step. + +Hooks can be a [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/), as you need to define: + +1. A Docker image that will be used to run specific commands. +1. One or more commands to run within the context of that Docker image. + +For simple commands we suggest you use a small image such as `alpine`, but any Docker image can be used in hooks. + +## Pipeline hooks + +Codefresh allows you to run a specific step before each pipeline as well as after it has finished. + +### Running a step at the end of the pipeline + +You can easily run a step at the end of pipeline, that will execute even if one of the steps have failed (and thus the pipeline is stopped in middle): + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +hooks: + on_finish: + exec: + image: alpine:3.9 + commands: + - echo "cleanup after end of pipeline" + +steps: + step1: + title: "Step 1" + type: "freestyle" + image: node:10-buster + commands: + - echo "Hello world" + step2: + title: "Step 2" + type: "freestyle" + image: node:10-buster + commands: + - echo "There was an error" + - exit 1 +{% endraw %} +{% endhighlight %} + +In the example above we define a hook for the whole pipeline that will run a step (the `exec` keyword) inside `alpine:3.9` and will simply execute an `echo` command. Because we have used the `on_finish` keyword, this step will execute even if the whole pipeline fails. + +This scenario is very common if you have a cleanup step or a notification step that you always want to run at the end of the pipeline. You will see the cleanup logs in the top pipeline step. + + {% include +image.html +lightbox="true" +file="/images/pipeline/codefresh-yaml/hooks/cleanup-step.png" +url="/images/pipeline/codefresh-yaml/hooks/cleanup-step.png" +alt="Running a cleanup step" +caption="Running a cleanup step" +max-width="80%" +%} + +Apart from the `on_finish` keyword you can also use `on_success` and `on_fail` if you want the step to only execute according to a specific result of the pipeline. It is also possible to use multiple hooks at the same time: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +hooks: + on_finish: + exec: + image: alpine:3.9 + commands: + - echo "cleanup after end of pipeline" + on_success: + exec: + image: alpine:3.9 + commands: + - echo "Send a notification only if pipeline was successful" + on_fail: + exec: + image: alpine:3.9 + commands: + - echo "Send a notification only if pipeline has failed" +steps: + step1: + title: "Step 1" + type: "freestyle" + image: node:10-buster + commands: + - echo "Hello world" + step2: + title: "Step 2" + type: "freestyle" + image: node:10-buster + commands: + - echo "There was an error" + - exit 1 #Comment this line out to see how hooks change + +{% endraw %} +{% endhighlight %} + +Note that if you have multiple hooks like the example above, the `on_finish` segment will always execute after any `on_success`/`on_fail` segments (if they are applicable). + + +### Running a step at the start of the pipeline + +Similar to the end of the pipeline, you can also execute a step at the beginning of the pipeline with the `on_elected` keyword: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +hooks: + on_elected: + exec: + image: alpine:3.9 + commands: + - echo "Creating an adhoc test environment" + on_finish: + exec: + image: alpine:3.9 + commands: + - echo "Destroying test environment" +steps: + step1: + title: "Step 1" + type: "freestyle" + image: node:10-buster + commands: + - echo "Running Integration tests on test environment" + step2: + title: "Step 2" + type: "freestyle" + image: node:10-buster + commands: + - echo "Running acceptance tests on test environment" + +{% endraw %} +{% endhighlight %} + +All pipeline hooks will be shown in the "initializing process" logs: + + {% include +image.html +lightbox="true" +file="/images/pipeline/codefresh-yaml/hooks/before-pipeline.png" +url="/images/pipeline/codefresh-yaml/hooks/before-pipeline.png" +alt="Hooks before a pipeline" +caption="Hooks before a pipeline" +max-width="80%" +%} + +It is possible to define all possible hooks (`on_elected`, `on_finish`, `on_success`, `on_fail`) in a single pipeline, if this is required by your development process. + +## Step hooks + +Hooks can also be defined for individual steps inside a pipeline. This capability allows for more granular control on defining prepare/cleanup phases for specific steps. + +The syntax for step hooks is the same as pipeline hooks (`on_elected`, `on_finish`, `on_success`, `on_fail`), you just need to put the respective segment under a step instead of the root of the pipeline. + +For example, this pipeline will always run a cleanup step after integration tests (even if the tests themselves fail). + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +steps: + step1: + title: "Compile application" + type: "freestyle" + image: node:10-buster + commands: + - echo "Building application" + step2: + title: "Unit testing" + type: "freestyle" + image: node:10-buster + commands: + - echo "Running unit tests" + hooks: + on_finish: + exec: + image: alpine:3.9 + commands: + - echo "Create test report" + step3: + title: "Uploading artifact" + type: "freestyle" + image: node:10-buster + commands: + - echo "Upload to artifactory" +{% endraw %} +{% endhighlight %} + + +Logs for steps hooks are shown in the log window of the step itself. + + {% include +image.html +lightbox="true" +file="/images/pipeline/codefresh-yaml/hooks/step-after.png" +url="/images/pipeline/codefresh-yaml/hooks/step-after.png" +alt="Hooks before a pipeline" +caption="Hooks before a pipeline" +max-width="80%" +%} + +As with pipeline hooks, it is possible to define multiple hook conditions for each step. + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +steps: + step1: + title: "Compile application" + type: "freestyle" + image: node:10-buster + commands: + - echo "Building application" + step2: + title: "Security scanning" + type: "freestyle" + image: node:10-buster + commands: + - echo "Running Security scan" + hooks: + on_elected: + exec: + image: alpine:3.9 + commands: + - echo "Authenticating to security scanning service" + on_finish: + exec: + image: alpine:3.9 + commands: + - echo "Uploading security scan report" + on_fail: + exec: + image: alpine:3.9 + commands: + - echo "Sending slack notification" + +{% endraw %} +{% endhighlight %} + +The order of events in the example above is the following: + +1. The `on_elected` segment executes first (authentication) +1. The step itself executes (the security scan) +1. The `on_fail` segment executes (only if the step throws an error code) +1. The `on_finish` segment always executes at the end + + +## Running steps/plugins in hooks + +Hooks can use [steps/plugins](https://steps.codefresh.io). With plugins you have to specify: + +- The type field for the step/plugin. +- The arguments needed for the step/plugin. + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" + +hooks: #run slack-notifier hook on build completion + on_finish: + steps: + exec: + type: slack-notifier + arguments: + SLACK_HOOK_URL: '${{SLACK_WEBHOOK_URL}}' + SLACK_TEXT: '${{SLACK_TEXT}}' + +steps: + step1: + title: "Freestyle step" + type: "freestyle" + image: alpine + commands: + - echo "Codefresh" + hooks: #run slack-notifier hook on step completion + on_finish: + steps: + exec: + type: slack-notifier + arguments: + SLACK_HOOK_URL: '${{SLACK_WEBHOOK_URL}}' + SLACK_TEXT: '${{SLACK_TEXT}}' +{% endraw %} +{% endhighlight %} + +## Controlling errors inside pipeline/step hooks + +By default if a step fails within a pipeline, the whole pipeline will stop and be marked as failed. +This is also true for `on_elected` segments as well. If they fail, then the whole pipeline will fail (regardless of the position of the segment in a pipeline or step). However, this only applies to `on_elected` segments. +`on_success`, `on_fail` and `on_finish` segments do not affect the pipeline outcome at all, and a pipeline will continue even if one of these segments fails. + +For example the following pipeline will fail right away, because the pipeline hook fails at the beginning. + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +hooks: + on_elected: + exec: + image: alpine:3.9 + commands: + - echo "failing on purpose" + - exit 1 +steps: + step1: + title: "Step 1" + type: "freestyle" + image: node:10-buster + commands: + - echo "Running Integration tests on test environment" +{% endraw %} +{% endhighlight %} + +You can change this behavior by using the existing [fail_fast property]({{site.baseurl}}/docs/codefresh-yaml/what-is-the-codefresh-yaml/#execution-flow) inside an `on_elected` hook. + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +hooks: + on_elected: + exec: + image: alpine:3.9 + fail_fast: false + commands: + - echo "failing on purpose" + - exit 1 +steps: + step1: + title: "Step 1" + type: "freestyle" + image: node:10-buster + commands: + - echo "Running Integration tests on test environment" +{% endraw %} +{% endhighlight %} + +This pipeline will now execute successfully and `step1` will still run as normal, because we have used the `fail_fast` property. You can also use the `fail_fast` property on step hooks as well: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +steps: + step1: + title: "Step 1" + type: "freestyle" + image: node:10-buster + commands: + - echo "Running Integration tests on test environment" + hooks: + on_elected: + exec: + image: alpine:3.9 + fail_fast: false + commands: + - echo "failing on purpose" + - exit 1 +{% endraw %} +{% endhighlight %} + + +>Notice that the `fail_fast` property is only available for `on_elected` hooks. The other types of hooks (`on_finish`, `on_success`, `on_fail`) do not affect the outcome of the pipeline in any way. Even if they fail, the pipeline will continue running to completion. This behavior is not configurable. + + +## Using multiple steps for hooks + +In all the previous examples, each hook was a single step running on a single Docker image. You can also define multiple steps for each hook. This is possible by inserting an extra `steps` keyword inside the hook and listing multiple Docker images under it: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +hooks: + on_finish: + steps: + mycleanup: + image: alpine:3.9 + commands: + - echo "echo cleanup step" + mynotification: + image: cloudposse/slack-notifier + commands: + - echo "Notify slack" +steps: + step1: + title: "Step 1" + type: "freestyle" + image: node:10-buster + commands: + - echo "Running Integration tests on test environment" +{% endraw %} +{% endhighlight %} + +By default all steps in a single hook segment are executed one after the other. But you can also run them in [parallel]({{site.baseurl}}/docs/pipelines/advanced-workflows/#inserting-parallel-steps-in-a-sequential-pipeline): + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +steps: + step1: + title: "Compile application" + type: "freestyle" + image: node:10-buster + commands: + - echo "Building application" + step2: + title: "Unit testing" + type: "freestyle" + image: node:10-buster + commands: + - echo "Running Integration tests" + - exit 1 + hooks: + on_fail: + mode: parallel + steps: + upload-my-artifact: + image: maven:3.5.2-jdk-8-alpine + commands: + - echo "uploading artifact" + my-report: + image: alpine:3.9 + commands: + - echo "creating test report" +{% endraw %} +{% endhighlight %} + +You can use multiple steps in a hook in both the pipeline and the step level. + + +## Using annotations and labels in hooks + +The hook syntax can also be used as a unified interface for encompassing the existing syntax of [build annotations]({{site.baseurl}}/docs/pipelines/annotations/) and [metadata]({{site.baseurl}}/docs/pipelines/docker-image-metadata/). + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +hooks: + on_elected: + exec: + image: alpine:3.9 + commands: + - echo "Creating an adhoc test environment" + annotations: + set: + - entity_type: build + annotations: + - my_annotation_example1: 10.45 + - my_string_annotation: Hello World +steps: + clone: + title: Cloning source code + type: git-clone + arguments: + repo: 'codefresh-contrib/golang-sample-app' + revision: master + build-image: + type: build + image_name: my-golang-image + working_directory: '${{clone}}' + tag: master + hooks: + on_success: + exec: + image: alpine:3.9 + commands: + - echo "Scanning docker image" + metadata: # setting metadata + set: + - '${{build-image.imageId}}': + - status: 'Success' +{% endraw %} +{% endhighlight %} + +Note however, that if you decide to use annotations and metadata inside hooks, you cannot mix and max the old syntax with the new syntax. + +The following pipeline is **NOT** valid: + +`invalid-codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +steps: + test: + image: alpine + on_success: # you cannot use old style together with hooks + annotations: + set: + - entity_type: build + annotations: + - status: 'success' + commands: + - echo block + hooks: + on_success: + annotations: + set: + - entity_type: build + annotations: + - status: 'success' +{% endraw %} +{% endhighlight %} + +The pipeline is not correct, because the first segment of annotations is directly under `on_success` (the old syntax), while the second segment is under `hooks/on_success` (the new syntax). + + +## Syntactic sugar syntax + +To simplify the syntax for hooks, the following simplifications are also offered: + +If you do not want to use metadata or annotations in your hook the keyword `exec` can be omitted: + +`codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +hooks: + on_finish: # no exec keyword + image: notifications:master + commands: + - ./send_workflow_finished.js +steps: + build: + type: build + image_name: my_image + tag: master + hooks: + on_fail: # no exec keyword + image: notifications:master + commands: + - ./send_build_failed.js +{% endraw %} +{% endhighlight %} + + +If you do not want to specify the Docker image you can simply omit it. Codefresh will use the `alpine` image in that case to run the hook: + + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +hooks: + on_elected: + exec: # no image keyword - alpine image will be used + - echo "Pipeline starting" +steps: + build: + type: build + image_name: my_image + tag: master + hooks: + on_success: # no image keyword - alpine image will be used + exec: + - echo "Docker image was built successfully" + annotations: + set: + - entity_type: build + annotations: + - status: 'Success' +{% endraw %} +{% endhighlight %} + + + If you don't use metadata or annotations, you can also completely remove the `exec` keyword and just mention the commands you want to run (`alpine` image will be used by default): + + `codefresh.yml` +{% highlight yaml %} +{% raw %} +version: "1.0" +hooks: + on_elected: # no exec/image keyword - alpine image will be used + - echo "Pipeline starting" +steps: + build: + type: build + image_name: my_image + tag: master + hooks: + on_success: # no exec/image keyword - alpine image will be used + - echo "Docker image was built successfully" +{% endraw %} +{% endhighlight %} + +## Using Type Steps / Plugins in hooks + +You can use a type step / plugins in hooks. With this you will need to change `exec` into `steps` with the information needed for the step. + +Below is an example pipeline hook using the `slack-notifier` step/plugin for when the pipeline starts. + +```yaml +hooks: + on_elected: + steps: + exec: + slack_pending: + type: slack-notifier + arguments: + SLACK_HOOK_URL: {% raw %}'${{SLACK_WEBHOOK_URL}}'{% endraw %} + SLACK_TEXT: '*Build Started* :crossed_fingers:' +``` + +## Limitations of pipeline/step hooks + +With the current implementation of hooks, the following limitations are present: + +* The [debugger]({{site.baseurl}}/docs/pipelines/debugging-pipelines/) cannot inspect commands inside hook segments +* Hooks are not supported for [parallel steps]({{site.baseurl}}/docs/pipelines/advanced-workflows/) +* Storage integrations don't resolve in hooks (for example, [test reports]({{site.baseurl}}/docs/testing/test-reports/#producing-allure-test-reports-from-codefresh-pipelines)) +* Step hook does not support the working_directory field aka `working_directory: ${{clone}}` + +## Related articles +[Conditional execution of steps]({{site.baseurl}}/docs/pipelines/conditional-execution-of-steps/) +[Working Directories]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Annotations in CI pipelines]({{site.baseurl}}/docs/pipelines/annotations/) + + diff --git a/_docs/pipelines/introduction-to-codefresh-pipelines.md b/_docs/pipelines/introduction-to-codefresh-pipelines.md new file mode 100644 index 000000000..6369bb74a --- /dev/null +++ b/_docs/pipelines/introduction-to-codefresh-pipelines.md @@ -0,0 +1,336 @@ +--- +title: "Introduction to Codefresh pipelines" +description: "Understand how Codefresh pipelines work" +group: pipelines +redirect_from: + - /docs/introduction-to-codefresh-pipelines/ + - /docs/configure-ci-cd-pipeline/ +toc: true +--- + + +The central component of the Codefresh platform for continuous integration (CI) are pipelines. Pipelines are workflows that contain individual steps, with each step responsible for a specific action in the CI process. + +Use CI pipelines to: + +* Compile and package code +* Build Docker images +* Push Docker images to any [Docker Registry]({{site.baseurl}}/docs/docker-registries/external-docker-registries/) +* Deploy applications/artifacts to VMs, Kubernetes clusters, FTP sites, S3 buckets etc. +* Run [unit tests]({{site.baseurl}}/docs/testing/unit-tests/), [integration tests]({{site.baseurl}}/docs/testing/integration-tests/), acceptance tests etc. +* Any custom action that you define + +{% include +image.html +lightbox="true" +file="/images/pipeline/codefresh-yaml/stages/complex-pipeline.png" +url="/images/pipeline/codefresh-yaml/stages/complex-pipeline.png" +alt="Codefresh CI pipelines" +caption="Codefresh CI pipelines" +max-width="90%" +%} + +## Why are Codefresh CI pipelines different? + +Codefresh offers unique characteristics in CI pipelines that serve as the cornerstone of the build/deploy process: + +1. All [steps]({{site.baseurl}}/docs/pipelines/steps/) in Codefresh pipelines are executed inside a Docker container of your choosing. +1. All steps in Codefresh share the same "workspace" in the form of a shared Docker volume. +1. The shared Docker volume is automatically cached between pipeline executions. +1. Every successful pipeline automatically pushes its Docker image to the default Docker registry defined in your account. +1. Codefresh has a distributed Docker cache for all build nodes and caches layers similar to the docker daemon on your workstation. This is fully automated, and does not need to be configured to activate it. + +### Using Docker containers as build tooling + +Unlike traditional solutions, Codefresh was built from the ground up to have full Docker support. All Codefresh pipelines +deal with Docker images, either using them as runtime tools or creating them as deployment artifacts. +Everything that happens in Codefresh uses a Docker image behind the scenes. + +It is important that you understand how to take advantage of Docker-based pipelines as they are much more powerful than +traditional VM solutions. The capability to define your own tooling cannot be understated. It is the fastest way to take +full control of your build tools and to upgrade them easily. + +With traditional VM-based build solutions, you are constrained on the build and deployment tools provided by the vendor. +If for example you need a new version of Node/Java/Python other than the one that is provided on the build slave, you have to wait for your vendor to add it. If you need to use a special tool (e.g terraform, gcloud) and the vendor does +not support it you are out of luck. + +With Codefresh you don't have to care about what is installed in the Runners that execute your builds. They can run *any* Docker image of your choosing. You are free to update the version of the image used at any given time. + +Here is an example: + +{% include +image.html +lightbox="true" +file="/images/pipeline/introduction/steps-example1.png" +url="/images/pipeline/introduction/steps-example1.png" +alt="Pipeline with three steps" +caption="Pipeline with three steps" +max-width="70%" +%} + + +1. The first step runs under the context of a Node image that prepares the application and runs [unit tests]({{site.baseurl}}/docs/testing/unit-tests/). +1. The second step uses an image with s3 command line tools and uploads the test results to a bucket that holds test reports. +1. The helm step creates a Helm chart and pushes it to a Helm repository. + +You don't need to contact Codefresh and ask them to add the S3 executable on the build runners. You just use a prebuilt Docker image that contains it. The version used for Node is defined by you and if you wish to upgrade to another version +you simply change the definition of the pipeline. + + +Here is another example: + +{% include +image.html +lightbox="true" +file="/images/pipeline/introduction/steps-example2.png" +url="/images/pipeline/introduction/steps-example2.png" +alt="Codefresh steps example 2" +caption="Pipeline with 4 steps" +max-width="70%" +%} + +1. The first step runs under the context of a Maven image that compiles the code and creates an executable. +1. The second step uses a Docker image that contains terraform and creates a single ECS instance in AWS. +1. The third step uses a custom Docker image that deploys to the ECS container that was just created. +1. The last step uploads the Maven reports that were created in step 1 to an FTP site. + +You should now start seeing the endless possibilities. You can mix and match any Docker image (either a public one +or your own) to use a build context in your step. This makes Codefresh a future-proof solution for all build tools +that exist now and all of them that will appear in the future. As long as there is a Docker image for a tool, Codefresh +can use it in a pipeline without any extra configuration. + +Codefresh also offers a [marketplace](https://codefresh.io/steps/){:target="\_blank"} with several existing plugins. + +{% include +image.html +lightbox="true" +file="/images/pipeline/plugin-directory.png" +url="/images/pipeline/plugin-directory.png" +alt="Codefresh steps directory" +caption="Codefresh steps directory" +max-width="80%" +%} + + +All plugins in the marketplace are open-source, and we accept external contributions so you can easily add your own. + + +### Sharing the workspace between build steps + +We have seen in the previous section that Codefresh can use Docker images as the context of a build step. The second +important point to understand regarding Codefresh CI pipelines is that the default workspace of each step is shared between all steps in a pipeline. + +This happens via a Docker volume which is attached to all Docker containers that represent each step. This volume is +always available at `/codefresh/volume`, and is used as the parent folder where the project is cloned. + +{% include +image.html +lightbox="true" +file="/images/pipeline/introduction/codefresh-volume.png" +url="/images/pipeline/introduction/codefresh-volume.png" +alt="Codefresh volume" +caption="All steps share the same volume" +max-width="90%" +%} + +Anything in this volume is available to all steps of the pipeline (as well as to subsequent executions of the same pipeline as we will see later). + +Again, this places Codefresh ahead of traditional solutions that execute build steps in a completely isolated manner. +In traditional VM-based builds, using artifacts produced from one step in another step, is a complicated process as one +must declare which artifact folders should be re-used. Artifact re-use sometimes happens with compression/decompression +of the respective folder resulting in really slow builds if a project is very big. + +Codefresh does not need to bother the user with artifact reuse across steps. *Anything* that is placed in the shared Codefresh volume will automatically be available to the next steps in the pipeline without any extra configuration. + +Example 1 + +{% include +image.html +lightbox="true" +file="/images/pipeline/introduction/codefresh-volume-example1.png" +url="/images/pipeline/introduction/codefresh-volume-example1.png" +alt="Codefresh volume example 1" +caption="Re-using Node Modules" +max-width="90%" +%} + +1. The first step runs `npm install` and downloads all libraries in `node_modules` into the shared Codefresh volume. +1. The second step runs `npm test`. The folder `node_modules` is still present from the previous step. + +Example 2 + +{% include +image.html +lightbox="true" +file="/images/pipeline/introduction/codefresh-volume-example2.png" +url="/images/pipeline/introduction/codefresh-volume-example2.png" +alt="Codefresh volume example 2" +caption="Re-using Test reports" +max-width="90%" +%} + +1. The first step runs `mvn test` and produces some test reports in `target/surefire-reports` into the shared Codefresh volume. +1. The next step uploads these reports using FTP to an external site. + + +The common volume shared among build steps makes it very easy to create pipelines that work in a gradual manner +where any step in the pipeline is using artifacts produced by a previous one. + +>The shared volume is **NOT available** in [build steps]({{site.baseurl}}/docs/pipelines/steps/build/). This is not a Codefresh limitation. Docker itself [does not allow volumes during builds](https://github.com/moby/moby/issues/14080){:target="\_blank"}. There is no folder `/codefresh/volume` inside a Dockerfile for you to access. + +You can also use [environment variables]({{site.baseurl}}/docs/pipelines/variables/) to share information between steps. All predefined environment variables are available to all steps, and each individual step can use `cf_export` to dynamically inject extra environment variables during the build process. + + +## Working with Codefresh pipelines + +Now that we know the basics, we can see how you can take advantage of Docker-based pipelines in order to build and deploy your projects. + + +### Cloning the source code + +You can clone source code using the built-in [Git-clone step]({{site.baseurl}}/docs/pipelines/steps/git-clone/) as the first step in a CI pipeline, or manually run your own Git clone commands in a freestyle step. Codefresh has built-in [Git integration]({{site.baseurl}}/docs/integrations/git-providers/) with all popular git providers (both cloud and on-premises installations). + +Codefresh uses the shared volume as the parent folder of the project. So if your pipeline is connected to a Git repo that contains `my-project` the following will happen: + +* `/codefresh/volume` is the shared directory for all steps +* `/codefresh/volume/my-project` is where the source code exists. This is also the current working directory +* Any other directory (e.g. `/bin`, `/var`, `/opt`) depends on the current container image that is used as build context + +{% include +image.html +lightbox="true" +file="/images/pipeline/introduction/checkout.png" +url="/images/pipeline/introduction/checkout.png" +alt="Codefresh checkout folder" +caption="Codefresh checkout folder" +max-width="80%" +%} + +There are three important points to consider regarding these folders: + +1. The [working directory]({{ site.baseurl }}/docs/pipelines/what-is-the-codefresh-yaml/#working-directories) of each step is by default the project folder (e.g. `/codefresh/volume/my-project`). Therefore +your build step can run commands exactly as you would run them locally (e.g. `npm install, pip install, mvn package, bundle install`). + +1. Notice that the project folder is placed on the Codefresh volume, so by default it is also available to all other steps. The code that you check out in the beginning, as well as all other files that are created on it, are available to all steps. Once you create `node_modules`, or any other folder that exists inside the project folder, it will automatically persist for all other steps. + +1. Finally, `/codefresh/volume` is an internal folder name, and you should use `{% raw %}${{CF_VOLUME_PATH}}{% endraw %}` in your codefresh.yml file +if you really want to reference this folder. You can also reference your project folder as `{% raw %}${{CF_VOLUME_PATH}}/${{CF_REPO_NAME}}{% endraw %}` if you need it. + +See the [System Provided Variables]({{site.baseurl}}/docs/pipelines/variables/#system-provided-variables) section for more information. + +### Working with Docker inside a Codefresh pipeline + +We have already seen that Codefresh pipelines are based on Docker images and that each step runs inside the context of a Docker container. You might be wondering how you can run Docker commands directly inside a Codefresh pipeline. + +The answer is that you don't. Even though in the future Codefresh might allow for Docker-in-Docker capabilities, at the moment this is not supported for security reasons (only enterprise customers have access to the underlying Docker daemon). Any scripts that you already have that run Docker commands on their own will need to be adapted to Codefresh pipelines. + +Usually you want to run a docker command for four reasons: + +1. To build a Docker image +1. To push a Docker image +1. To run a docker-compose setup +1. To run a Docker container + +For all these situations Codefresh gives you special pipeline steps that perform the respective action. These are: + +1. The [build step]({{site.baseurl}}/docs/pipelines/steps/build/) +1. The [push step]({{site.baseurl}}/docs/pipelines/steps/push/) +1. The [compositions step]({{site.baseurl}}/docs/pipelines/steps/composition/) +1. The [freestyle step]({{site.baseurl}}/docs/pipelines/steps/freestyle/) + +The commands you define in a freestyle step run automatically in a Docker container that is attached to that step once the pipeline executes. + +Therefore, this command on your local workstation: + +``` +docker run python:3.6.4-alpine3.6 pip install . +``` + +will become in Codefresh + +``` +CollectAllMyDeps: + title: Install dependencies + image: python:3.6.4-alpine3.6 + commands: + - pip install . +``` +For the plugins in the [Step Marketplace](https://codefresh.io/steps/) we already give an example of the YAML part that must be included in your pipeline: + +{% include +image.html +lightbox="true" +file="/images/pipeline/plugin-example.png" +url="/images/pipeline/plugin-example.png" +alt="Codefresh steps directory" +caption="Codefresh steps directory" +max-width="50%" +%} + +Each plugin also defines its input/output in the form of environment variables and files. + +### Creating Docker images dynamically as build tools + + +Now we reach one of the most powerful features of Codefresh pipelines. We have already seen that [freestyle pipeline steps]({{site.baseurl}}/docs/pipelines/steps/freestyle/) are just a series of commands that run inside the context of a Docker container. In most cases the images used +for the freestyle steps are known in advance and come from public (e.g. Dockerhub) or [private Docker registries]({{site.baseurl}}/docs/docker-registries/external-docker-registries/). + +Codefresh is one the few CI/CD solutions that not only offers easy Docker registry integration + accessible to all pipelines +but also allows you to **build docker images on demand in the same pipeline where they are required**. + +This means that you can create a special Docker image in an early step inside a Codefresh pipeline and then reference it in a later step in the same pipeline. + +{% include +image.html +lightbox="true" +file="/images/pipeline/introduction/dynamic-docker-builds.png" +url="/images/pipeline/introduction/dynamic-docker-builds.png" +alt="Codefresh dynamic docker builds" +caption="Creating dynamically Docker images as build steps" +max-width="90%" +%} + +Let's say for example that you are moving a legacy application to Codefresh which is deployed using a special Python script. Your main application is a Ruby-On-Rails app. Both applications exist in the same git repository (we mention this for simplicity reasons, Codefresh also supports checking out code from multiple repositories). + +You can create a single pipeline with Codefresh that does the following: + +1. Checks out the code +1. Creates a Docker image based on Python for the deployment tool +1. Uploads the Python tool Docker image to the internal registry +1. Builds the Ruby application using a freestyle step with the R-O-R image from Dockerhub +1. Deploys the Ruby application by running the Python based deployment tool image (after pulling it first) + +This concept is ground-breaking as it allows you to automatically update your build tools that are used in any pipeline. +Even though you could manually create the Docker images yourself before-hand, it is better to completely automate them +inside the pipeline they are actually needed. This ensures that both the application and its tooling are always at the latest version. + +### How caching works in Codefresh + +Codefresh employs several caching mechanisms for both Dockerized and non-dockerized applications. The shared volume is also cached behind the scenes automatically. See our [caching guide]({{site.baseurl}}/docs/configure-ci-cd-pipeline/pipeline-caching/) for more details. + +### Calling other pipelines + +It is also possible to chain multiple pipelines together in Codefresh. To accomplish this, Codefresh offers +a special Docker image that contains the [Codefresh CLI](https://codefresh-io.github.io/cli/){:target="\_blank"} and allows you to trigger another pipeline using its name. + +{% include +image.html +lightbox="true" +file="/images/pipeline/introduction/call-pipeline.png" +url="/images/pipeline/introduction/call-pipeline.png" +alt="Codefresh call pipeline" +caption="Calling another pipeline" +max-width="80%" +%} + +Notice that each pipeline in Codefresh is completely isolated from the other. They use a different Docker volume so the build context of each one cannot access files from the other. This may change in the future, but for the time being +you should know that only steps within the same pipeline can share artifacts. + +## Related articles +[Creating pipelines]({{site.baseurl}}/docs/pipelines/pipelines/) +[Codefresh YAML]({{site.baseurl}}/docs/pipelines/what-is-the-codefresh-yaml/) +[Build and Docker caching]({{site.baseurl}}/docs/pipelines/pipeline-caching/) + + + diff --git a/_docs/pipelines/marketplace.md b/_docs/pipelines/marketplace.md deleted file mode 100644 index 4295f74ee..000000000 --- a/_docs/pipelines/marketplace.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -title: "Codefresh marketplace" -description: "" -group: pipelines -toc: true ---- - -The Codefresh Hub for Argo documentation can be found in its [official repository](https://github.com/codefresh-io/argo-hub). - -Codefresh is fully backing this project and will help all developers that want to contribute to succeed. - -You can find documentation about how to contribute to the argo hub in the [official repository contribute section](https://github.com/codefresh-io/argo-hub#How-to-Contribute) - diff --git a/_docs/pipelines/monitoring-pipelines.md b/_docs/pipelines/monitoring-pipelines.md new file mode 100644 index 000000000..3f9f4426e --- /dev/null +++ b/_docs/pipelines/monitoring-pipelines.md @@ -0,0 +1,472 @@ +--- +title: "Monitoring pipelines" +description: "Viewing your builds and logs" +group: pipelines +toc: true +--- + + +All pipeline activity in Codefresh can be viewed in the *Builds* tab. +* The global build view shows builds for all projects across your organization +* The project-based view from the settings inside an individual project shows the builds for the selected project + +Both views have the same controls and filters. + +## Viewing pipeline status + +Each screen contains all builds sorted from the most recent to the oldest. The first time you visit +the screen there are no filters defined. + +{% include +image.html +lightbox="true" +file="/images/pipeline/monitoring/builds-dashboard.png" +url="/images/pipeline/monitoring/builds-dashboard.png" +alt="Pipeline Activity in Codefresh" +caption="Pipeline activity" +max-width="80%" +%} + +By default, it shows all builds that is happening in Codefresh. To narrow the list you can use the filters on the top +of the screen. + +### Applying filters on the build view + +Directly above the list you can find several filters. + +At the most basic level you can choose between + + * *Running* builds that are currently executing + * *Pending* builds which are queued and waiting to start + * *Delayed* builds which cannot run yet, because there are no free pipeline builders. + A build can be delayed for a maximum of seven days, and each account can have up to 1000 delayed builds at any time. + * Builds that are delayed for more than seven days are terminated with a _Delay time limit exceeded_ reason. + * If the total number of delayed builds exceed 1000, older builds are terminated with a _Maximum delayed workflows exceeded_ reason. + + * *All* builds regardless of running stage (this is the default) + +You can further filter the builds by choosing the various filter types that specify the build job. + +{% include +image.html +lightbox="true" +file="/images/pipeline/monitoring/build-filtering.png" +url="/images/pipeline/monitoring/build-filtering.png" +alt="Pipeline filters in Codefresh" +caption="Available filters" +max-width="50%" +%} + +The available filters are: + +* *Pipeline* - any of the pipelines available. +* *Provider* - type of [Git provider]({{site.baseurl}}/docs/integrations/git-providers/). +* *Repository* - Git repository from the attached [trigger]({{site.baseurl}}/docs/configure-ci-cd-pipeline/triggers/). +* *Type* - build, [launch a test environment]({{site.baseurl}}/docs/getting-started/on-demand-environments/#launching-a-docker-image-using-codefresh). +* *Branch* - any of the available branches from the attached Git trigger. +* *Committer* - person that made the commit that triggered the build. +* *Environment* - which [environment]({{site.baseurl}}/docs/deploy-to-kubernetes/environment-dashboard/) was affected. +* *Status* - success, error, in-progress, pending, terminated etc. A Pending status can also indicate that [pipeline build execution has been paused]({{site.baseurl}}/docs/administration/pipeline-settings/#pause-pipeline-executions) for the account. +* *Trigger type* - what type of trigger was responsible for this build +* *Git event* - in the case of [git triggers]({{site.baseurl}}/docs/configure-ci-cd-pipeline/triggers/git-triggers/) the exact event + +Notice that all filters are multiple-choice so you can select multiple values for each filter category. +At any given point you can see all the active filters on top of the screen. + +{% include +image.html +lightbox="true" +file="/images/pipeline/monitoring/possible-filters.png" +url="/images/pipeline/monitoring/possible-filters.png" +alt="Pipeline filters in Codefresh" +caption="Active filters" +max-width="50%" +%} + +You can easily remove active filters, by clicking on them and adding/removing values. + +On the right hand side you can also find a filtering toolbar with time options: + +{% include +image.html +lightbox="true" +file="/images/pipeline/monitoring/build-filter-date.png" +url="/images/pipeline/monitoring/build-filter-date.png" +alt="Filtering options for time" +caption="Filtering options for time" +max-width="60%" +%} + +You can combine all previously mentioned filters with the time based filters. + +### Creating build views + +Once you have a set of filters that you use regularly, you can save them as a custom *Build View* by clicking the *Save as View* button +and providing a name. + +Now you can select at the top of the window any of the available build views to automatically filter results according to the respective sets of filters. + +{% include +image.html +lightbox="true" +file="/images/pipeline/monitoring/build-view-selection.png" +url="/images/pipeline/monitoring/build-view-selection.png" +alt="Build View selection" +caption="Build View selection (click to enlarge)" +max-width="50%" +%} + +You can delete existing build-views by clicking on the *manage views* button. +You can change the filters of an existing build view by making a new filter selection and then saving the view with an existing name (effectively overwriting it). + + +### Build details + + +For each individual build you can see several details such as the git hash, the person who made the commit, the pipeline that was triggered as well as how much time it took. For each event type you will also see additional context related information. + +{% include +image.html +lightbox="true" +file="/images/pipeline/monitoring/build-details-entry.png" +url="/images/pipeline/monitoring/build-details-entry.png" +alt="build details in Codefresh" +caption="Build details" +max-width="100%" +%} + +Child builds triggered by other builds are identified in the Event column by the icon {::nomarkdown}
{:/}.
+The Parent Build column shows the link to the parent build. Mouse over to see the tooltip with information on the parent build. The tooltip includes links to the parent build, repo, branch, commit message, and the ability to filter by repo and branch.
+
+{% include
+image.html
+lightbox="true"
+file="/images/pipeline/monitoring/child-parent-build-info.png"
+url="/images/pipeline/monitoring/child-parent-build-info.png"
+alt="Child build in Builds list"
+caption="Child build in Builds list"
+max-width="70%"
+%}
+
+There are also extra options if you click the small "3-dot" menu button on the right. For a particular build, you can:
+
+- View the logs
+- View the YAML
+- View or add [annotations]({{site.baseurl}}/docs/pipelines/annotations/)
+- View the images produced (and consequently launch an on-demand [test environment]({{site.baseurl}}/docs/getting-started/on-demand-environments/#launching-a-docker-image-using-codefresh))
+
+Notice that if you restart a pipeline it will trigger with the exact settings it *originally* had. So
+if this was a manual trigger where you [disabled caching]({{site.baseurl}}/docs/troubleshooting/common-issues/disabling-codefresh-caching-mechanisms/) or changed the [notification options](#monitoring-pipelines-that-check-pull-requests), the new
+execution will still honor those settings (even if you have changed them for later builds).
+
+An extra button for test reports will be visible if you are using the [test report feature]({{site.baseurl}}/docs/testing/test-reports/) of Codefresh.
+
+
+## Viewing details for an individual pipeline build
+
+If you click on any individual pipeline build, you will enter the pipeline build information screen.
+From here you can see more details for a build such as the logs, running time and resource metrics.
+
+{% include
+image.html
+lightbox="true"
+file="/images/pipeline/monitoring/pipeline-view.png"
+url="/images/pipeline/monitoring/pipeline-view.png"
+alt="Pipeline view"
+caption="Pipeline view"
+max-width="80%"
+%}
+
+Each section in this screen corresponds to each pipeline step. There are two special steps:
+
+* *Initializing Process*
+* *Cloning Main Repository*
+
+These are Codefresh built-in steps and will appear for most builds (you can also create a pipeline that doesn't clone a git repository by default). The rest of the step names depend on your `codefresh.yml` (or the default step names provided by Codefresh). The different columns take the names from the defined [pipeline stages]({{site.baseurl}}/docs/pipelines/stages/).
+
+### Viewing status for pipeline steps
+
+Monitor the status of the steps in the pipeline as they are executed.
+
+{: .table .table-bordered .table-hover}
+| Step Status Icon | Description |
+| ------------------------| ---------------- |
+|{::nomarkdown}
{:/}| Pipeline step completed successfully. |
+|{::nomarkdown}
{:/}| Pipeline step pending approval has been approved, either manually or automatically. |
+|{::nomarkdown}
{:/}| Pipeline step pending approval has been denied approval. |
+|{::nomarkdown}
{:/}| Pipeline step currently running. |
+|{::nomarkdown}
{:/}| Pipeline step running in debug mode. See [Debugging pipelines]({{site.baseurl}}/docs/configure-ci-cd-pipeline/debugging-pipelines/) for more information. |
+|{::nomarkdown}
{:/}| Pipeline step gracefully terminating execution. |
+|{::nomarkdown}
{:/}| Pipeline step execution has been manually or automatically terminated. |
+|{::nomarkdown}
{:/}| Pipeline step execution has been terminated because of error. |
+
+
+
+### Viewing/downloading logs for builds and build steps
+
+View logs for running and completed builds and download them in HTML or text formats.
+You can view logs online, for the entire build or for single or specific steps in the build. Similarly, you can download the logs for the entire build, or for single or specific steps.
+The Filter Logs option is useful to view and manage logs, especially for large builds as there is a max size limit for logs. You can also search logs.
+
+>Note:
+ The max log size for the entire build is 100MB, and 20MB per step. The system stops generating logs once the build size is exceeded.
+ For large builds, it is easier to filter the logs by single or multiple steps, and then view/download them.
+
+1. In the **Builds** page, select a build.
+1. To view logs online for the selected build, click **Output** in the lower part of the Build page.
+1. Optional. Select **Filter Logs** and then select the step or steps for which view/download logs.
+ Logs are displayed for the selected steps.
+1. From either the context menu on the top-right of the toolbar or from the Output pane, select **Download as HTML** or **Download as text**.
+ The log file is downloaded with the build ID as the filename, including also the step name if the log is for a single step, in the format `/ + isPublic: false + description: >- + The plugin exports KV pairs from Hashicorp Vault to Codefresh pipeline ENV + variables + sources: + - 'https://github.com/codefresh-io/steps/tree/master/incubating/vault' + stage: incubating + maintainers: + - name: Alexander Aladov + categories: + - featured + official: false + tags: [] + icon: + type: svg + url: 'https://cdn.jsdelivr.net/gh/codefresh-io/steps/incubating/vault/icon.svg' + background: '#f4f4f4' + examples: + - description: example-1 + workflow: + version: '1.0' + steps: + Vault_to_Env: + title: Importing vault values + type: vault + arguments: + VAULT_ADDR: '${{VAULT_ADDR}}' + VAULT_PATH: '${{VAULT_PATH}}' + VAULT_AUTH_TOKEN: '${{VAULT_AUTH_TOKEN}}' + VAULT_CLIENT_CERT_BASE64: '${{VAULT_CLIENT_CERT_BASE64}}' + VAULT_CLIENT_KEY_BASE64: '${{VAULT_CLIENT_KEY_BASE64}}' + created_at: '2019-07-03T14:57:02.057Z' + updated_at: '2019-09-18T08:15:28.476Z' + latest: true + version: 0.0.1 + id: 5d1cc23ea7e22e40227ea75d +spec: + arguments: |- + { + "definitions": {}, + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "additionalProperties": false, + "patterns": [], + "required": [ + "VAULT_ADDR", + "VAULT_PATH", + "VAULT_AUTH_TOKEN" + ], + "properties": { + "VAULT_ADDR": { + "type": "string", + "description": "Vault server URI. Example: https://vault.testdomain.io:8200 (required)" + }, + "VAULT_PATH": { + "type": "string", + "description": "Path to secrets in vault. Example: secret/codefreshsecret (required)" + }, + "VAULT_AUTH_TOKEN": { + "type": "string", + "description": "Vault authentication token (required)" + }, + "VAULT_CLIENT_CERT_BASE64": { + "type": "string", + "description": "Base64 encoded client cerificate" + }, + "VAULT_CLIENT_KEY_BASE64": { + "type": "string", + "description": "Base64 encoded client key" + } + } + } + steps: + main: + name: vault + image: codefreshplugins/vault + environment: + - 'VAULT_ADDR=${{VAULT_ADDR}}' + - 'VAULT_PATH=${{VAULT_PATH}}' + - 'VAULT_AUTH_TOKEN=${{VAULT_AUTH_TOKEN}}' + - 'VAULT_CLIENT_CERT_BASE64=${{VAULT_CLIENT_CERT_BASE64}}' + - 'VAULT_CLIENT_KEY_BASE64=${{VAULT_CLIENT_KEY_BASE64}}' +{% endraw %} +{% endhighlight %} + +For each step you define the following sections: + +* Metadata to describe the characteristics of the step +* The description of its arguments +* The implementation (i.e. what yaml gets inserted in the pipeline) + +For the metadata section note the following: + +* `isPublic` decides if this step is visible only to your and your team, or visible to all (in the marketplace) +* The `name` of the step **must** be prefixed with your Codefresh account name. Steps created by the Codefresh team are on the root level of the hierarchy (without prefix). This is the same pattern that Dockerhub is using for images. +* `stage` shown if this step is ready for production or still incubating. This is just an indication to users. It doesn't affect the implementation of the step in any way +* `icon`. Ideally you provide a transparent svg so that the icon is scalable. The icon for a step is used both in the marketplace as well as the pipeline view. You can also select a default background to be used. Alternatively, you can define jpg/png icons for large/medium/small sizes. We suggest the svg approach +* The `version` property allows you to update your plugin and keep multiple variants of it in the marketplace +* The `examples` section will be shown in the marketplace as documentation for your step + +For the argument section we follow the [JSON Schema](http://json-schema.org/learn/miscellaneous-examples.html). You can use the [Schema generator](https://jsonschema.net/) to easily create a schema. JSON schema is used for arguments (i.e. input parameters) as well as output parameters as we will see later on. + +The property `additionalProperties` defines how strict the plugin will be with its arguments. If you set it to `false` (which is usually what you want) the pipeline will fail if the plugin is given more arguments that it is expecting. If you set it to `true`, then the plugin will only use the arguments it understands and will ignore the rest. + +The final part is the step implementation. Here you can define exactly the yaml that this step will insert in the pipeline. You can use any of the built-in steps in Codefresh and even add multiple steps. + +>Note that currently you cannot nest custom pipeline steps. We are aware of this limitation and are actively working on it, but at the time or writing you cannot use a typed step inside another typed step. + +Once you are done with your step, use the Codefresh CLI to upload it to the marketplace. If you want the step to be available only to you and your team make sure that the property `isPublic` is false (and then it will not be shown in the marketplace). + +{% highlight bash %} +codefresh create step-type -f my-custom-step.yml +{% endhighlight %} + +If you make further changes to your step you can update it: + +{% highlight bash %} +codefresh replace step-type -f my-custom-step.yml +{% endhighlight %} + +If you want to remove your step from the marketplace, you can delete it completely: + +{% highlight bash %} +codefresh delete step-type kostis-codefresh/sample +{% endhighlight %} + +### Versioning of typed steps + +The `version` property under `metadata` in the plugin manifest allows you to publish multiple releases of the same plugin in the marketplace. Codefresh will keep all previous plugins and users are free to choose which version they want. + +To create a new version of your plugin: + +1. Update the `version` property under `metadata` in your custom YAML. +2. Run: + +{% highlight bash %} +codefresh create step-type -f custom-plugin.yaml +{% endhighlight %} + +You will now be able to see the new versions of your plugin in the step marketplace drop-down: + +{% include +image.html +lightbox="true" +file="/images/pipeline/codefresh-yaml/steps/step-versions.png" +url="/images/pipeline/codefresh-yaml/steps/step-versions.png" +alt="Different step versions" +caption="Different step versions" +max-width="60%" +%} + +You can also use the Codefresh CLI to list all version: + +{% highlight bash %} +codefresh get step-types kostis-codefresh/sample --versions +{% endhighlight %} + +To delete a specific version, use: + +{% highlight bash %} +codefresh delete step-type 'account/plugin:
] [ --atomic ] [ --debug ] [ helm upgrade parameters ] +``` +### Step 6: Install the Codefresh Kubernetes Agent + +Install the `cf-k8s-agent` on a cluster separate from the installer, or in a different namespace on the same cluster. +The `cf-k8s-agent` accesses Kubernetes resources (pods, deployments, services, etc.) behind the firewall to display them in the Codefresh UI. The agent streams updates from cluster resources and then sends information updates to the `k8s-monitor` service. + +1. Create a staging directory for the agent: + +``` +kcfi init k8s-agent +``` + A staging directory is created, named k8s-agent with a `config.yaml`. +1. Edit k8s-agent/config.yaml ?? for what?? + +1. Run: + +``` +kcfi deploy [ -c config.yaml ] [-n namespace] +``` + where: + [namespace] is the namespace if you are installing the agent in the same cluster. + + + + +## High-Availability (HA) with active-passive clusters +Enable high-availability in the Codefresh platform for disaster recovery with an active-passive cluster configuration. +Review the prerequisites, and then do the following to configure high-availability: +* For new installations, install Codefresh on the active cluster +* Install Codefresh on the passive cluster +* When needed, switch between clusters for disaster recovery + +### Prerequisites + +* **K8s clusters** + Two K8s clusters, one designated as the active cluster, and the other designated as the passive cluster for disaster recovery. + +* **External databases and services** + Databases and services external to the clusters. + + * Postgres database (see [Configuring an external Postgres database](#configuring-an-external-postgres-database)) + * MongoDB (see [Configuring an external MongoDB](#configuring-an-external-mongodb)) + * Redis service (see [Configuring an external Redis service](#configure-an-external-redis-service)) + * RabbitMQ service (see [Configuring an external RabbitMQ service](#configure-an-external-redis-service)) + * Consul service (see [Configuring an external Consul service](#configuring-an-external-consul-service)) + +* **DNS record** + To switch between clusters for disaster recovery + +### Install Codefresh on active cluster + +If you are installing Codefresh for the first time, install Codefresh on the cluster designated as the _active_ cluster. +See [Installing the Codefresh platform]({{site.baseurl}}/docs/administration/codefresh-on-prem/#install-the-codefresh-platform). + +### Install Codefresh on passive cluster + +First get the `values.yaml` file from the current Codefresh installation on the active cluster. Then install Codefresh on the passive cluster using Helm. + +**1. Get values.yaml** +1. Switch your kube context to the active cluster. +1. Get `values.yaml` from the active cluster: + `helm get values ${release_name} -n ${namespace} > cf-passive-values.yaml` + where: + `{release-version}` is the name of the Codefresh release, and is by default `cf`. + `${namespace}` is the namespace with the Codefresh release, and is by default `codefresh`. + +{:start="3"} +1. Update the required variables in `cf-passive-values.yaml`. + > If the variables do not exist, add them to the file. + + * In the `global` section, disable `seedJobs` by setting it to `false`: + + ```yaml + global: + seedJobs: false + ``` + + * Add variable `FREEZE_WORKFLOWS_EXECUTION` to `cfapi`, and set it to `true`. + + ```yaml + cfapi: + env: + FREEZE_WORKFLOWS_EXECUTION: true + ``` + +**2. Install Codefresh on passive cluster** + +1. Download the Helm chart: + `helm repo add codefresh-onprem https://chartmuseum.codefresh.io/codefresh` + `helm fetch codefresh-onprem/codefresh --version ${release-version}` + where: + `{release-version}` is the version of Codefresh you are downloading. + +1. Unzip the Helm chart: + `tar -xzf codefresh-${release-version}.tgz` +1. Go to the folder where you unzipped the Helm chart. +1. Install Codefresh with the Helm command using `cf-passive-values.yaml`: + `helm install cf . -f ${path}/cf-passive-values.yaml -n codefresh` + + +### Switch between clusters for disaster recovery + +For disaster recovery, switch between the active and passive clusters. + +1. In the `cfapi` deployment on the _active_ cluster, change the value of `FREEZE_WORKFLOWS_EXECUTION` from `false` to `true`. + If the variable does not exist, add it, and make sure the value is set to `true`. +1. In the `cfapi` deployment on the _passive_ cluster, change the value of `FREEZE_WORKFLOWS_EXECUTION` from `true` to `false`. +1. Switch DNS from the currently active cluster to the passive cluster. + +### Services without HA + +The following services cannot run in HA, but are not critical in case of downtime or during the process of switchover from active to passive. +These services are not considered critical as they are part of build-handling. In case of failure, a build retry occurs, ensuring that the build is always handled. +* `cronus` +* `cf-sign` + + +## Additional configuration + +After you install Codefresh, these are post-installation operations that you should follow. + +### Selectively enable SSO provider for account +As a Codefresh administrator, you can select the providers you want to enable for SSO in your organization, for both new and existing accounts. +You can always renable a provider when needed. + + +1. Sign in as Codefresh admin. +1. From the left pane, select **Providers**. +1. Disable the providers not relevant for the accounts. +These providers are not displayed as options during sign-up/sign-in. + + +### (Optional) Set up Git integration + +Codefresh supports out-of-the-box Git logins using your local username and password, or logins using your Git provider, as described below.You can also configure login to supported SSO providers after installation, as described in [Setting up OpenID Connect (OIDC) Federated Single Sign-On (SSO)]({{site.baseurl}}/docs/administration/single-sign-on/oidc). + +If you’d like to set up a login to Codefresh using your Git provider, first login using the default credentials (username: `AdminCF`, password: `AdminCF` and add your Git provider OAuth integration details in our admin console: + +**Admin Management** > **IDPs** tab + +To get the Client ID and Client Secret for each of the supported Git providers, follow the instructions according to your VCS provider. + +#### GitHub Enterprise + +Navigate to your GitHub organization settings: https://github.com/organizations/your_org_name/settings. + +On the left-hand side, under **Developer settings**, select **OAuth Apps**, and click **Register an Application**. + +Complete the OAuth application registration as follows: + +- **Application name:** codefresh-on-prem (or a significant name) +- **Homepage URL:** https://your-codefresh-onprem-domain +- **Authorization callback URL:** https://your-codefresh-onprem-domain/api/auth/github/callback + +After registration, note down the created Client ID and Client Secret. They will be required for the settings in **Codefresh Admin**->**IDPs** + +#### GitLab + +Navigate to your Applications menu in GitLab User Settings: https://gitlab.com/profile/applications + +Complete the application creation form as follows: + +- **Name:** codefresh-onprem (or a significant name) +- **Redirect URI:** https://your-codefresh-onprem-domain/api/auth/gitlab/callback +- **Scopes (permissions):** + - API + - read_user + - read_registry + +Click **Save application**. + +After app creation, note down the created Application ID and Client Secret. They will be required for the settings in **Codefresh Admin**->**IDPs**. + +{% include image.html + lightbox="true" + file="/images/installation/git-idp.png" + url="/images/installation/git-idp.png" + %} + +>Note: When configuring the default IDP (for GitHub, GitLab, etc), do not modify the Client Name field. Please keep them as GitHub, GitLab, BitBucket, etc. Otherwise, the signup and login views won’t work. + +### Proxy Configuration + +If your environment resides behind HTTP proxies, you need to uncomment the following section in `config.yaml`: + +```yaml +global: + env: + HTTP_PROXY: "http://myproxy.domain.com:8080" + http_proxy: "http://myproxy.domain.com:8080" + HTTPS_PROXY: "http://myproxy.domain.com:8080" + https_proxy: "http://myproxy.domain.com:8080" + NO_PROXY: "127.0.0.1,localhost,kubernetes.default.svc,.codefresh.svc,100.64.0.1,169.254.169.254,cf-builder,cf-cfapi,cf-cfui,cf-chartmuseum,cf-charts-manager,cf-cluster-providers,cf-consul,cf-consul-ui,cf-context-manager,cf-cronus,cf-helm-repo-manager,cf-hermes,cf-ingress-nginx-controller,cf-kube-integration,cf-mongodb,cf-nats,cf-nomios,cf-pipeline-manager,cf-postgresql,cf-rabbitmq,cf-redis-master,cf-registry,cf-runner,cf-runtime-environment-manager,cf-store" + no_proxy: "127.0.0.1,localhost,kubernetes.default.svc,.codefresh.svc,100.64.0.1,169.254.169.254,cf-builder,cf-cfapi,cf-cfui,cf-chartmuseum,cf-charts-manager,cf-cluster-providers,cf-consul,cf-consul-ui,cf-context-manager,cf-cronus,cf-helm-repo-manager,cf-hermes,cf-ingress-nginx-controller,cf-kube-integration,cf-mongodb,cf-nats,cf-nomios,cf-pipeline-manager,cf-postgresql,cf-rabbitmq,cf-redis-master,cf-registry,cf-runner,cf-runtime-environment-manager,cf-store" +``` +In addition to this, you should also add your Kubernetes API IP address (`kubectl get svc kubernetes`) to both: `NO_PROXY` and `no_proxy`. + +### Storage + +Codefresh is using both cluster storage (volumes) as well as external storage. + +#### Databases + +The following table displays the list of databases created as part of the installation: + +| Database | Purpose | Latest supported version | +|----------|---------| ---------------| +| mongoDB | storing all account data (account settings, users, projects, pipelines, builds etc.) | 4.2.x | +| postgresql | storing data about events that happened on the account (pipeline updates, deletes, etc.). The audit log uses the data from this database. | 13.x | +| redis | mainly used for caching, but also used as a key-value store for our trigger manager. | 6.0.x | + +#### Volumes + +These are the volumes required for Codefresh on-premises: + + +{: .table .table-bordered .table-hover} +| Name | Purpose | Minimum Capacity | Can run on netfs (nfs, cifs) | +|----------------|------------------------|------------------|------------------------------| +| cf-mongodb* | Main database - Mongo | 8GB | Yes** | +| cf-postgresql* | Events databases - Postgres | 8GB | Yes** | +| cf-rabbitmq* | Message broker | 8GB | No** | +| cf-redis* | Cache | 8GB | No** | +| cf-store | Trigger Redis data | 8GB | No** | +| cf-cronus | Trigger crontab data | 1GB | Yes | +| datadir-cf-consul-0 | Consul datadir | 1GB | Yes | +| cf-chartmuseum | chartmuseum | 10GB | Yes | +| cf-builder-0 | /var/lib/docker for builder | 100GB | No*** | +| cf-runner-0 | /var/lib/docker for composition runner | 100GB | No*** | + +{% raw %} + + (*) Possibility to use external service + + (**) Running on netfs (nfs, cifs) is not recommended by product admin guide + + (***) Docker daemon can be run on block device only + +{% endraw %} + +StatefulSets (`cf-builder` and `cf-runner`) process their data on separate physical volumes (PVs) and can be claimed using Persistent Volume Claims (PVCs) with default initial sizes of 100Gi. Also, those StatefulSets have the ability to connect to existing pre-defined PVCs. + +The default initial volume size (100 Gi) can be overridden in the custom `config.yaml` file. Values descriptions are in the `config.yaml` file. +The registry’s initial volume size is 100Gi. It also can be overridden in a custom `config.yaml` file. There is a possibility to use a customer-defined registry configuration file (`config.yaml`) that allows using different registry storage back-ends (S3, Azure Blob, GCS, etc.) and other parameters. More details can be found in the [Docker documentation](https://docs.docker.com/registry/configuration/). + +Depending on the customer’s Kubernetes version we can assist with PV resizing. Details are can be found in this [Kubernetes blog post](https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/). + +#### Automatic Volume Provisioning + +Codefresh installation supports automatic storage provisioning based on the standard Kubernetes dynamic provisioner Storage Classes and Persistent Volume Claims. All required installation volumes will be provisioned automatically using the default Storage Class or custom Storage Class that can be specified as a parameter in `config.yaml` under `storageClass: my-storage-class`. + + + +### Retention policy for Codefresh builds +Define a retention policy to manage Codefresh builds. The retention settings are controlled through `cf-api` deployment environment variables, all of which have default settings which you can retain or customize. By default, Codefresh deletes builds older than six months, including offline logs. + +The retention mechanism, implemented as a Cron Job, removes data from collections such as: +* workflowproccesses +* workflowrequests +* workflowrevisions + +{: .table .table-bordered .table-hover} +| Env Variable | Description | Default | +|---------------|--------------------------- |---------------------- | +|`RETENTION_POLICY_IS_ENABLED` | Determines if automatic build deletion through the Cron job is enabled. | `true` | +|`RETENTION_POLICY_BUILDS_TO_DELETE`| The maximum number of builds to delete by a single Cron job. To avoid database issues, especially when there are large numbers of old builds, we recommend deleting them in small chunks. You can gradually increase the number after verifying that performance is not affected. | `50` | +|`RETENTION_POLICY_DAYS` | The number of days for which to retain builds. Builds older than the defined retention period are deleted. | `180` | +|`RUNTIME_MONGO_URI` | Optional. The URI of the Mongo database from which to remove MongoDB logs (in addition to the builds). | | + + +### Managing Codefresh backups + +Codefresh on-premises backups can be automated by installing a specific service as an addon to your Codefresh on-premises installation. It is based on the [mgob](https://github.com/stefanprodan/mgob){:target="\_blank"} open source project, and can run scheduled backups with retention, S3 & SFTP upload, notifications, instrumentation with Prometheus and more. + +#### Configure and deploy the Backup Manager + +Backup Manager is installed as an addon and therefore it needs an existing Codefresh on-premises installation. +Before installing it, please make sure you have selected a proper kube config pointing to the cluster, where you have Codefresh installed on. + +1. Go to the staging directory of your Codefresh installation, and open the config file: `your-CF-stage-dir/addons/backup-manager/config.yaml`. +1. Retain or customize the values of these configuration parameters: + * `metadada`: Various CF-installer-specific parameters, which should not be changed in this case + * `kubernetes`: Specify a kube context, kube config file, and a namespace for the backup manager + * `storage`: Storage class, storage size and read modes for persistent volumes to store backups locally within your cluster + * Backup plan configuration parameters under `jobConfigs.cfBackupPlan`: + * `target.uri` - target mongo URI. It is recommended to leave the mongo uri value blank - it will be taken automatically from the Codefresh release installed in your cluster + * `scheduler` - here you can specify cron expression for your backups schedule, backups retention and timeout values + +For more advanced backup plan settings, such as specifying various remote cloud-based storage providers for your backups, configuring notifications and other, please refer to [this](https://github.com/stefanprodan/mgob#configure) page + +To **deploy the backup manager** service, please select a correct kube context, where you have Codefresh on-premises installed and deploy backup-manager with the following command: + +``` +kcfi deploy -c `your-CF-stage-dir/addons/backup-manager/config.yaml` +``` + +#### On-demand/ad-hoc backup +``` +kubectl port-forward cf-backup-manager-0 8090 +curl -X POST http://localhost:8090/backup/cfBackupPlan +``` + +#### Restore from backup +``` +kubectl exec -it cf-backup-manager-0 bash +mongorestore --gzip --archive=/storage/cfBackupPlan/backup-archive-name.gz --uri mongodb://root:password@mongodb:27017 --drop +``` + +### Configuring AWS Load Balancers + +By default Codefresh deploys the [ingress-nginx](https://github.com/kubernetes/ingress-nginx/) controller and [Classic Load Balancer](https://docs.aws.amazon.com/eks/latest/userguide/load-balancing.html) as a controller service. + +#### NLB + +To use a **Network Load Balancer** - deploy a regular Codefresh installation with the following ingress config for the the `cf-ingress-controller` controller service. + +`config.yaml` +```yaml +ingress-nginx: + controller: + service: + annotations: + service.beta.kubernetes.io/aws-load-balancer-type: nlb + service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp + service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '60' + service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true' + +tls: + selfSigned: false + cert: certs/certificate.crt + key: certs/private.key +``` +This annotation will create a new Load Balancer - Network Load Balancer, which you should use in the Codefresh UI DNS record. +Update the DNS record according to the new service. + +#### L7 ELB with SSL Termination + +When a **Classic Load Balancer** is used, some Codefresh features that (for example `OfflineLogging`), will use a websocket to connect with Codefresh API and they will require secure TCP (SSL) protocol enabled on the Load Balancer listener instead of HTTPS. + +To use either a certificate from a third party issuer that was uploaded to IAM or a certificate [requested](https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-public.html) within AWS Certificate Manager see the followning config example: + + +`config.yaml` +```yaml +ingress-nginx: + controller: + service: + annotations: + service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp" + service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443" + service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '3600' + service.beta.kubernetes.io/aws-load-balancer-ssl-cert: < CERTIFICATE ARN > + targetPorts: + http: http + https: http + +tls: + selfSigned: true +``` + +- both http and https target port should be set to **80**. +- update your AWS Load Balancer listener for port 443 from HTTPS protocol to SSL. + +#### ALB + +To use the **Application Load Balancer** the [ALB Ingress Controller](https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html) should be deployed to the cluster. + +To support ALB: + +- First disable Nginx controller in the Codefresh init config file - __config.yaml__: + +```yaml +ingress-nginx: #disables creation of Nginx controller deployment + enabled: false + +ingress: #disables creation of Ingress object + enabled: false +``` + +- [deploy](https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html) the ALB controller; +- create a new **ingress** resource: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + annotations: + alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]' + alb.ingress.kubernetes.io/scheme: internet-facing + alb.ingress.kubernetes.io/target-type: ip + kubernetes.io/ingress.class: alb + meta.helm.sh/release-name: cf + meta.helm.sh/release-namespace: codefresh + labels: + app: cf-codefresh + release: cf + name: cf-codefresh-ingress + namespace: codefresh +spec: + defaultBackend: + service: + name: cf-cfui + port: + number: 80 + rules: + - host: myonprem.domain.com + http: + paths: + - backend: + service: + name: cf-cfapi + port: + number: 80 + path: /api/* + pathType: ImplementationSpecific + - backend: + service: + name: cf-cfapi + port: + number: 80 + path: /ws/* + pathType: ImplementationSpecific + - backend: + service: + name: cf-cfui + port: + number: 80 + path: / + pathType: ImplementationSpecific +``` + +### Configure CSP (Content Security Policy) +Add CSP environment variables to `config.yaml`, and define the values to be returned in the CSP HTTP headers. +```yaml +cfui: + env: + CONTENT_SECURITY_POLICY: " " + CONTENT_SECURITY_POLICY_REPORT_ONLY: "default-src 'self'; font-src 'self' + https://fonts.gstatic.com; script-src 'self' https://unpkg.com https://js.stripe.com; + style-src 'self' https://fonts.googleapis.com; 'unsafe-eval' 'unsafe-inline'" + CONTENT_SECURITY_POLICY_REPORT_TO: " - "
+```
+`CONTENT_SECURITY_POLICY` is the string describing content policies. Use semi-colons to separate between policies.
+`CONTENT_SECURITY_POLICY_REPORT_TO` is a comma-separated list of JSON objects. Each object must have a name and an array of endpoints that receive the incoming CSP reports.
+
+For detailed information, see the [Content Security Policy article on MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP).
+
+### Enable x-hub-signature-256 signature for GitHub AE
+Add the `USE_SHA256_GITHUB_SIGNATURE` environment variable to **cfapi** deployment in `config.yaml`.
+```yaml
+cfapi:
+ env:
+ USE_SHA256_GITHUB_SIGNATURE: "true"
+```
+
+For detailed information, see the [Securing your webhooks](https://docs.github.com/en/developers/webhooks-and-events/webhooks/securing-your-webhooks) and [Webhooks](https://docs.github.com/en/github-ae@latest/rest/webhooks).
+
+
+## Using existing external services for data storage/messaging
+
+Normally the Codefresh installer, is taking care of all needed dependencies internally by deploying the respective services (mongo, redis etc) on its own.
+
+You might want however to use your own existing options if you already have those services up and running externally.
+
+### Configuring an external Postgres database
+
+It is possible to configure Codefresh to work with your existing Postgres database service, if you don't want to use the default one as provided by the Codefresh installer.
+
+#### Configuration steps
+
+All the configuration comes down to putting a set of correct values into your Codefresh configuration file `config.yaml`, which is present in `your/stage-dir/codefresh` directory. During the installation, Codefresh will run a seed job, using the values described in the following steps:
+
+1. Specify a user name `global.postgresSeedJob.user` and password `global.postgresSeedJob.password` for a seed job. This must be a privileged user allowed to create databases and roles. It will be used only by the seed job to create the needed database and a user.
+2. Specify a user name `global.postgresUser` and password `global.postgresPassword` to be used by Codefresh installation. A user with the name and password will be created by the seed job and granted with required privileges to access the created database.
+3. Specify a database name `global.postgresDatabase` to be created by the seed job and used by Codefresh installation.
+4. Specify `global.postgresHostname` and optionally `global.postgresPort` (`5432` is a default value).
+5. Disable the postgres subchart installation with the `postgresql.enabled: false` value, because it is not needed in this case.
+
+
+Below is an example of the relevant piece of `config.yaml`:
+
+```yaml
+global:
+ postgresSeedJob:
+ user: postgres
+ password: zDyGp79XyZEqLq7V
+ postgresUser: cf_user
+ postgresPassword: fJTFJMGV7sg5E4Bj
+ postgresDatabase: codefresh
+ postgresHostname: my-postgres.ccjog7pqzunf.us-west-2.rds.amazonaws.com
+ postgresPort: 5432
+
+postgresql:
+ enabled: false #disable default postgresql subchart installation
+```
+#### Running the seed job manually
+
+If you prefer running the seed job manually, you can do it by using a script present in `your/stage-dir/codefresh/addons/seed-scripts` directory named `postgres-seed.sh`. The script takes the following set of variables that you need to have set before running it:
+
+```shell
+export POSTGRES_SEED_USER="postgres"
+export POSTGRES_SEED_PASSWORD="zDyGp79XyZEqLq7V"
+export POSTGRES_USER="cf_user"
+export POSTGRES_PASSWORD="fJTFJMGV7sg5E4Bj"
+export POSTGRES_DATABASE="codefresh"
+export POSTGRES_HOST="my-postgres.ccjog7pqzunf.us-west-2.rds.amazonaws.com"
+export POSTGRES_PORT="5432"
+```
+The variables have the same meaning as the configuration values described in the previous section about Postgres.
+
+However you **still need to specify a set of values** in the Codefresh config file as described in the section above, but with the whole **`postgresSeedJob` section omitted**, like this:
+
+```yaml
+global:
+ postgresUser: cf_user
+ postgresPassword: fJTFJMGV7sg5E4Bj
+ postgresDatabase: codefresh
+ postgresHostname: my-postgresql.prod.svc.cluster.local
+ postgresPort: 5432
+
+postgresql:
+ enabled: false #disable default postgresql subchart installation
+```
+
+### Configuring an external MongoDB
+
+Codefresh recommends to use the Bitnami MongoDB [chart](https://github.com/bitnami/charts/tree/master/bitnami/mongodb) as a Mongo database. The supported version of Mongo is 4.2.x
+
+To configure Codefresh on-premises to use an external Mongo service one needs to provide the following values in `config.yaml`:
+
+- **mongo connection string** - `mongoURI`. This string will be used by all of the services to communicate with mongo. Codefresh will automatically create and add a user with "ReadWrite" permissions to all of the created databases with the username and password from the URI. Optionally, automatic user addition can be disabled - `mongoSkipUserCreation`, in order to use already existing user. In such a case the existing user must have **ReadWrite** permissions to all of newly created databases
+Codefresh does not support [DNS Seedlist Connection Format](https://docs.mongodb.com/manual/reference/connection-string/#connections-dns-seedlist) at the moment, use the [Standard Connection Format](https://docs.mongodb.com/manual/reference/connection-string/#connections-standard-connection-string-format) instead.
+- mongo **root user** name and **password** - `mongodbRootUser`, `mongodbRootPassword`. The privileged user will be used by Codefresh only during installation for seed jobs and for automatic user addition. After installation, credentials from the provided mongo URI will be used. Mongo root user must have permissions to create users.
+
+See the [Mongo required Access](https://docs.mongodb.com/manual/reference/method/db.createUser/#required-access) for more details.
+
+Here is an example of all the related values:
+
+```yaml
+global:
+ mongodbRootUser:
+ mongodbRootPassword: + mongoURI: + mongoSkipUserCreation: true + mongoDeploy: false # disables deployment of internal mongo service + +mongo: + enabled: false + ``` + +#### MongoDB with Mutual TLS + +>The option available in kcfi **v0.5.10** + +Codefresh supports enabling SSL/TLS between cf microservices and MongoDB. To enable this option specify in `config.yaml` the following parameters: + + `global.mongoTLS: true`
+ `global.mongoCaCert` - CA certificate file path (in kcfi init directory)
+ `global.mongoCaKey` - CA certificate private key file path (in kcfi init directory) + +`config.yaml` example: +```yaml +global: + mongodbRootUser: root + mongodbRootPassword: WOIqcSwr0y + mongoURI: mongodb://my-mongodb.prod.svc.cluster.local/?ssl=true&authMechanism=MONGODB-X509&authSource=$external + mongoSkipUserCreation: true + mongoDeploy: false # disables deployment of internal mongo service + + mongoTLS: true #enable MongoDB TLS support + mongoCaCert: mongodb-ca/ca-cert.pem + mongoCaKey: mongodb-ca/ca-key.pem + + ### for OfflineLogging feature + runtimeMongoURI: mongodb://my-mongodb.prod.svc.cluster.local/?ssl=true&authMechanism=MONGODB-X509&authSource=$external + +### for OfflineLogging feature +cfapi: + env: + RUNTIME_MONGO_TLS: "true" + RUNTIME_MONGO_TLS_VALIDATE: "true" # 'false' if self-signed certificate to avoid x509 errors + +## set MONGO_MTLS_VALIDATE to `false` if self-signed certificate to avoid x509 errors +cluster-providers: + env: + MONGO_MTLS_VALIDATE: "false" + +k8s-monitor: + env: + MONGO_MTLS_VALIDATE: "false" + +mongo: + enabled: false #disable default mongodb subchart installation + ``` + + >Perform an upgarde:
+ >`kcfi deploy -c config.yaml --debug` + +### Configure an external Redis service +Codefresh recommends to use the Bitnami Redis [chart](https://github.com/bitnami/charts/tree/master/bitnami/redis) as a Redis store. + +**Limitations** + +Codefresh does not support secure connection to Redis (TLS) and AUTH username extension. + +**Configuration** + +To configure Codefresh to use an external Redis service, add the following parameters to your `config.yaml`: + +`config.yaml` example: +```yaml +global: + redisUrl: my-redis.prod.svc.cluster.local + redisPort: 6379 + redisPassword: 6oOhHI8fI5 + + runtimeRedisHost: my-redis.prod.svc.cluster.local + runtimeRedisPassword: 6oOhHI8fI5 + runtimeRedisPort: 6379 + runtimeRedisDb: 2 + +redis: + enabled: false #disable default redis subchart installation +``` + +Where `redis*` - are for the main Redis storage, and `runtimeRedis*` - for storage is used to store pipeline logs in case of `OfflineLogging` feature is turned on. In most cases the host value is the same for these two values. + + +### Configuring an external RabbitMQ service + +Codefresh recommends to use the Bitnami RabbitMQ [chart](https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq) as a RabbitMQ service. + +To use an external RabbitMQ service instead of the local helm chart, add the following values to the __config.yaml__: + +```yaml +rabbitmq: + enabled: false + +global: + rabbitmqUsername:+ rabbitmqPassword: + rabbitmqHostname: +``` + +### Configuring an external Consul service + + +Notice that at the moment Codefresh supports only the deprecated Consul API (image __consul:1.0.0__), and does not support connection via HTTPS and any authentication. +The Consul host must expose port `8500`. + +>In general, we don't recommend to take the Consul service outside the cluster. + + +To configure Codefresh to use your external Consul service, add the following values to the __config.yaml__: + +```yaml +global: + consulHost: + +consul: + enabled: false +``` + +## App Cluster Autoscaling + +Autoscaling in Kubernetes is implemented as an interaction between Cluster Autoscaler and Horizontal Pod Autoscaler + +{: .table .table-bordered .table-hover} +| | Scaling Target| Trigger | Controller | How it Works | +| ----------- | ------------- | ------- | --------- | --------- | +| [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler)| Nodes | **Up:** Pending pod
**Down:** Node resource allocations is low | On GKE we can turn on/off autoscaler and configure min/max per node group can be also installed separately | Listens on pending pods for scale up and node allocations for scaledown. Should have permissions to call cloud api. Considers pod affinity, pdb, storage, special annotations | +| [Horizontal Pod Autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) | replicas on deployments or StatefulSets | metrics value thresholds defined in HPA object | part of Kubernetes controller | Controller gets metrics from "metrics.k8s.io/v1beta1" , "custom.metrics.k8s.io/v1beta1", "external.metrics.k8s.io/v1beta1" requires [metrics-server](https://github.com/kubernetes-sigs/metrics-server) and custom metrics adapters ([prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter), [stackdriver-adapter](https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/custom-metrics-stackdriver-adapter)) to listen on this API (see note (1) below) and adjusts deployment or sts replicas according to definitions in HorizontalPodAutocaler
There are v1 and beta api versions for HorizontalPodAutocaler:
[v1](https://github.com/kubernetes/api/blob/master/autoscaling/v1/types.go) - supports for resource metrics (cpu, memory) - `kubect get hpa`
[v2beta2](https://github.com/kubernetes/api/blob/master/autoscaling/v2beta2/types.go) and [v2beta1](https://github.com/kubernetes/api/blob/master/autoscaling/v2beta1/types.go) - supports for both resource and custom metrics - `kubectl get hpa.v2beta2.autoscaling` **The metric value should decrease on adding new pods.**
*Wrong metrics Example:* request rate
*Right metrics Example:* average request rate per pod | + +Note (1) +``` +kubectl get apiservices | awk 'NR==1 || $1 ~ "metrics"' +NAME SERVICE AVAILABLE AGE +v1beta1.custom.metrics.k8s.io monitoring/prom-adapter-prometheus-adapter True 60d +v1beta1.metrics.k8s.io kube-system/metrics-server True 84d +``` + + +**Implementation in Codefresh** + +* Default “Enable Autoscaling” settings for GKE +* Using [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter) with custom metrics + +We define HPA for cfapi and pipeline-manager services + +**CFapi HPA object** + +It's based on three metrics (HPA controller scales of only one of the targetValue reached): + +``` +kubectl get hpa.v2beta1.autoscaling cf-cfapi -oyaml +``` + +{% highlight yaml %} +{% raw %} +apiVersion: autoscaling/v2beta1 +kind: HorizontalPodAutoscaler +metadata: + annotations: + meta.helm.sh/release-name: cf + meta.helm.sh/release-namespace: default + labels: + app.kubernetes.io/managed-by: Helm + name: cf-cfapi + namespace: default +spec: + maxReplicas: 16 + metrics: + - object: + metricName: requests_per_pod + target: + apiVersion: v1 + kind: Service + name: cf-cfapi + targetValue: "10" + type: Object + - object: + metricName: cpu_usage_avg + target: + apiVersion: apps/v1 + kind: Deployment + name: cf-cfapi-base + targetValue: "1" + type: Object + - object: + metricName: memory_working_set_bytes_avg + target: + apiVersion: apps/v1 + kind: Deployment + name: cf-cfapi-base + targetValue: 3G + type: Object + minReplicas: 2 + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: cf-cfapi-base +{% endraw%} +{% endhighlight %} + +* `requests_per_pod` is based on `rate(nginx_ingress_controller_requests)` metric ingested from nginx-ingress-controller +* `cpu_usage_avg` based on cadvisor (from kubelet) rate `(rate(container_cpu_user_seconds_total)` +* `memory_working_set_bytes_avg` based on cadvisor `container_memory_working_set_bytes` + +**pipeline-manager HPA** + +based on `cpu_usage_avg` + +{% highlight yaml %} +{% raw %} +apiVersion: autoscaling/v2beta1 +kind: HorizontalPodAutoscaler +metadata: + annotations: + meta.helm.sh/release-name: cf + meta.helm.sh/release-namespace: default + labels: + app.kubernetes.io/managed-by: Helm + name: cf-pipeline-manager +spec: + maxReplicas: 8 + metrics: + - object: + metricName: cpu_usage_avg + target: + apiVersion: apps/v1 + kind: Deployment + name: cf-pipeline-manager-base + targetValue: 400m + type: Object + minReplicas: 2 + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: cf-pipeline-manager-base +{% endraw%} +{% endhighlight %} + +**prometheus-adapter configuration** + +Reference: [https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/config.md](https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/config.md +) + +{% highlight yaml %} +{% raw %} +Rules: + - metricsQuery: | + kube_service_info{<<.LabelMatchers>>} * on() group_right(service) + (sum(rate(nginx_ingress_controller_requests{<<.LabelMatchers>>}[2m])) + / on() kube_deployment_spec_replicas{deployment='<>-base',namespace='< >'}) + name: + as: requests_per_pod + matches: ^(.*)$ + resources: + overrides: + namespace: + resource: namespace + service: + resource: service + seriesQuery: kube_service_info{service=~".*cfapi.*"} + - metricsQuery: | + kube_deployment_labels{<<.LabelMatchers>>} * on(label_app) group_right(deployment) + (label_replace( + avg by (container) (rate(container_cpu_user_seconds_total{container=~"cf-(tasker-kubernetes|cfapi.*|pipeline-manager.*)", job="kubelet", namespace='< >'}[15m])) + , "label_app", "$1", "container", "(.*)")) + name: + as: cpu_usage_avg + matches: ^(.*)$ + resources: + overrides: + deployment: + group: apps + resource: deployment + namespace: + resource: namespace + seriesQuery: kube_deployment_labels{label_app=~"cf-(tasker-kubernetes|cfapi.*|pipeline-manager.*)"} + - metricsQuery: "kube_deployment_labels{<<.LabelMatchers>>} * on(label_app) group_right(deployment)\n + \ (label_replace(\n avg by (container) (avg_over_time (container_memory_working_set_bytes{container=~\"cf-.*\", + job=\"kubelet\", namespace='< >'}[15m]))\n + \ , \"label_app\", \"$1\", \"container\", \"(.*)\"))\n \n" + name: + as: memory_working_set_bytes_avg + matches: ^(.*)$ + resources: + overrides: + deployment: + group: apps + resource: deployment + namespace: + resource: namespace + seriesQuery: kube_deployment_labels{label_app=~"cf-.*"} + - metricsQuery: | + kube_deployment_labels{<<.LabelMatchers>>} * on(label_app) group_right(deployment) + label_replace(label_replace(avg_over_time(newrelic_apdex_score[15m]), "label_app", "cf-$1", "exported_app", '(cf-api.*|pipeline-manager|tasker-kuberentes)\\[kubernetes\\]'), "label_app", "$1cfapi$3", "label_app", '(cf-)(cf-api)(.*)') + name: + as: newrelic_apdex + matches: ^(.*)$ + resources: + overrides: + deployment: + group: apps + resource: deployment + namespace: + resource: namespace + seriesQuery: kube_deployment_labels{label_app=~"cf-(tasker-kubernetes|cfapi.*|pipeline-manager)"} +{% endraw%} +{% endhighlight %} + +**How to define HPA in Codefresh installer (kcfi) config** + +Most of Codefresh's Microservices subcharts contain `templates/hpa.yaml`: + +{% highlight yaml %} +{% raw %} +{{- if .Values.HorizontalPodAutoscaler }} +apiVersion: autoscaling/v2beta1 +kind: HorizontalPodAutoscaler +metadata: + name: {{ template "cfapi.fullname" . }} +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: {{ template "cfapi.fullname" . }}-{{ .version | default "base" }} + minReplicas: {{ coalesce .Values.HorizontalPodAutoscaler.minReplicas .Values.replicaCount 1 }} + maxReplicas: {{ coalesce .Values.HorizontalPodAutoscaler.maxReplicas .Values.replicaCount 2 }} + metrics: +{{- if .Values.HorizontalPodAutoscaler.metrics }} +{{ toYaml .Values.HorizontalPodAutoscaler.metrics | indent 4 }} +{{- else }} + - type: Resource + resource: + name: cpu + targetAverageUtilization: 60 +{{- end }} +{{- end }} +{% endraw%} +{% endhighlight %} + +To configure HPA for CFapi add `HorizontalPodAutoscaler` values to config.yaml, for example: + +(assuming that we already have prometheus adapter configured for metrics `requests_per_pod`, `cpu_usage_avg`, `memory_working_set_bytes_avg`) + +{% highlight yaml %} +{% raw %} +cfapi: + replicaCount: 4 + resources: + requests: + memory: "4096Mi" + cpu: "1100m" + limits: + memory: "4096Mi" + cpu: "2200m" + HorizontalPodAutoscaler: + minReplicas: 2 + maxReplicas: 16 + metrics: + - type: Object + object: + metricName: requests_per_pod + target: + apiVersion: "v1" + kind: Service + name: cf-cfapi + targetValue: 10 + - type: Object + object: + metricName: cpu_usage_avg + target: + apiVersion: "apps/v1" + kind: Deployment + name: cf-cfapi-base + targetValue: 1 + - type: Object + object: + metricName: memory_working_set_bytes_avg + target: + apiVersion: "apps/v1" + kind: Deployment + name: cf-cfapi-base + targetValue: 3G +{% endraw%} +{% endhighlight %} + +**Querying metrics (for debugging)** + +CPU Metric API Call + +``` +kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/codefresh/pods/cf-cfapi-base-****-/ | jq +``` + +Custom Metrics Call + +``` +kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/codefresh/services/cf-cfapi/requests_per_pod | jq +``` + + +## Common Problems, Solutions, and Dependencies + +### Dependencies + +#### Mongo + +All services using the MongoDB are dependent on the `mongo` pod being up and running. If the `mongo` pod is down, the following dependencies will not work: + +- `runtime-environment-manager` +- `pipeline-manager` +- `cf-api` +- `cf-broadcaster` +- `context-manager` +- `nomios` +- `cronius` +- `cluster-promoters` +- `k8s-monitor` +- `charts-manager` +- `tasker-kubernetes` + +#### Logs + +There is a dependency between the `cf-broadcaster` pod and the `cf-api` pod. If your pipeline runs, but does not show any logs, try restarting the broadcaster pod. + +### Problems and Solutions + +**Problem:** installer fails because `codefresh` database does not exist. + +**Solution:** If you are using an external PostgresSQL database (instead of the internal one that the installer provides), you will first need to manually create a new database named `codefresh` inside your PostgresSQL database before running the installer. + + diff --git a/_docs/installation/codefresh-runner.md b/_docs/installation/codefresh-runner.md new file mode 100644 index 000000000..a5fde7970 --- /dev/null +++ b/_docs/installation/codefresh-runner.md @@ -0,0 +1,2072 @@ +--- +title: "Codefresh Runner installation" +description: "Run Codefresh pipelines on your private Kubernetes cluster" +group: installation +redirect_from: + - /docs/enterprise/codefresh-runner/ +toc: true +--- + +Install the Codefresh Runner on your Kubernetes cluster to run pipelines and access secure internal services without compromising on-premises security requirements. These pipelines run on your infrastructure, even behind the firewall, and keep code on your Kubernetes cluster secure. + +[Skip to quick installation →](#installation-with-the-quick-start-wizard) + +>Important: + You must install the Codefresh Runner on _each cluster running Codefresh pipelines_. + The Runner is **not** needed in clusters used for _deployment_. You can deploy applications on clusters other than the ones the runner is deployed on. + +The installation process takes care of all Runner components and other required resources (config-maps, secrets, volumes). + +## Prerequisites + +To use the Codefresh runner the following is required: + +1. A Kubernetes cluster with outgoing internet access (versions 1.10 to 1.23). Each node should have 50GB disk size. +2. A container runtime, such as [docker](https://kubernetes.io/blog/2020/12/02/dockershim-faq/), [containerd](https://containerd.io/) or [cri-o](https://cri-o.io/). Note that the runner is **not** dependent on any special dockershim features, so any compliant container runtime is acceptable. The docker socket/daemon used by Codefresh pipelines is **NOT** the one on the host node (as it might not exist at all in the case of containerd or cri-o), but instead an internal docker daemon created/managed by the pipeline itself. +3. A [Codefresh account]({{site.baseurl}}/docs/getting-started/create-a-codefresh-account/) with the Hybrid feature enabled. +4. A [Codefresh CLI token]({{site.baseurl}}/docs/integrations/codefresh-api/#authentication-instructions) that will be used to authenticate your Codefresh account. + +The runner can be installed from any workstation or laptop with access (i.e. via `kubectl`) to the Kubernetes cluster running Codefresh builds. The Codefresh runner will authenticate to your Codefresh account by using the Codefresh CLI token. + +## System Requirements + +Once installed the runner uses the following pods: + +* `runner` - responsible for picking tasks (builds) from the Codefresh API +* `engine` - responsible for running pipelines +* `dind` - responsible for building and using Docker images +* `dind-volume-provisioner` - responsible for provisioning volumes (PV) for dind +* `dind-lv-monitor` - responsible for cleaning **local** volumes + +**CPU/Memory** + +The following table shows **MINIMUM** resources for each component: + +{: .table .table-bordered .table-hover} +| Component | CPU requests| RAM requests | Storage | Type | Always on | +| -------------- | --------------|------------- |-------------------------|-------|-------| +| `runner` | 100m | 100Mi | Doesn't need PV | Deployment | Yes | +| `engine` | 100m | 500Mi | Doesn't need PV | Pod | No | +| `dind` | 400m | 800Mi | 16GB PV | Pod | No | +| `dind-volume-provisioner` | 300m | 400Mi | Doesn't need PV | Deployment | Yes | +| `dind-lv-monitor` | 300m | 400Mi | Doesn't need PV | DaemonSet | Yes | + +Components that are always on consume resources all the time. Components that are not always on only consume resources when pipelines are running (they are created and destroyed automatically for each pipeline). + +Node size and count will depend entirely on how many pipelines you want to be “ready” for and how many will use “burst” capacity. + +* Ready (nodes): Lower initialization time and faster build times. +* Burst (nodes): High initialization time and slower build times. (Not recommended) + +The size of your nodes directly relates to the size required for your pipelines and thus it is dynamic. If you find that only a few larger pipelines require larger nodes you may want to have two Codefresh Runners associated to different node pools. + + +**Storage** + +For the storage options needed by the `dind` pod we suggest: + +* [Local Volumes](https://kubernetes.io/docs/concepts/storage/volumes/#local) `/var/lib/codefresh/dind-volumes` on the K8S nodes filesystem (**default**) +* [EBS](https://aws.amazon.com/ebs/) in the case of AWS. See also the [notes](#installing-on-aws) about getting caching working. +* [Local SSD](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd) or [GCE Disks](https://cloud.google.com/compute/docs/disks#pdspecs) in the case of GCP. See [notes](#installing-on-google-kubernetes-engine) about configuration. + + +**Networking Requirements** + +* `dind` - this pod will create an internal network in the cluster to run all the pipeline steps; needs outgoing/egress access to Dockerhub and `quay.io` +* `runner` - this pod needs outgoing/egress access to `g.codefresh.io`; needs network access to [app-proxy]({{site.baseurl}}/docs/administration/codefresh-runner/#optional-installation-of-the-app-proxy) (if app-proxy is used) +* `engine` - this pod needs outgoing/egress access to `g.codefresh.io`, `*.firebaseio.com` and `quay.io`; needs network access to `dind` pod + +All CNI providers/plugins are compatible with the runner components. + +## Installation with the Quick-start Wizard + +Install the Codefresh CLI + +```shell +npm install -g codefresh +``` + +[Alternative install methods](https://codefresh-io.github.io/cli/installation/) + +Authenticate the CLI + +```shell +codefresh auth create-context --api-key {API_KEY} +``` + +You can obtain an API Key from your [user settings page](https://g.codefresh.io/user/settings). +>**Note:** Make sure when you generate the token used to authenticate with the CLI, you generate it with *all scopes*. + +>**Note:** access to the Codefresh CLI is only needed once during the Runner installation. After that, the Runner will authenticate on it own using the details provided. You do NOT need to install the Codefresh CLI on the cluster that is running Codefresh pipelines. + +Then run the wizard with the following command: + +```shell +codefresh runner init +``` + +or + +```shell +codefresh runner init --token +``` + +Brefore proceeding with installation, the wizard asks you some basic questions. + +{% include image.html + lightbox="true" + file="/images/administration/runner/installation-wizard.png" + url="/images/administration/runner/installation-wizard.png" + alt="Codefresh Runner wizard" + caption="Codefresh Runner wizard" + max-width="100%" + %} + +The wizard also creates and runs a sample pipeline that you can see in your Codefresh UI. + +{% include image.html + lightbox="true" + file="/images/administration/runner/sample-pipeline.png" + url="/images/administration/runner/sample-pipeline.png" + alt="Codefresh Runner example pipeline" + caption="Codefresh Runner example pipeline" + max-width="90%" + %} + +That's it! You can now start using the Runner. + +You can also verify your installation with: + +```shell +codefresh runner info +``` + +During installation you can see which API token will be used by the runner (if you don't provide one). The printed token is used by the runner to talk to the Codefresh platform carrying permissions that allow the runner to run pipelines. If you save the token, it can later be used to restore the runner's permissions without creating a new runner installation, if the deployment is deleted. + +**Customizing the Wizard Installation** + +You can customize the wizard installation by passing your own values in the `init` command. +To inspect all available options run `init` with the `--help` flag: + +```shell +codefresh runner init --help +``` + +**Inspecting the Manifests Before they are Installed** + +If you want to see what manifests are used by the installation wizard you can supply the `--dry-run` parameter in the installation process. + +```shell +codefresh runner init --dry-run +``` + +This will execute the wizard in a special mode that will not actually install anything in your cluster. After all configuration questions are asked, all Kubernetes manifests used by the installer will be instead saved locally in a folder `./codefresh_manifests`. + +## Install Codefresh Runner with values file + +To install the Codefresh Runner with pre-defined values file use `--values` flag: + +```shell +codefresh runner init --values values.yaml +``` + +Use [this example](https://github.com/codefresh-io/venona/blob/release-1.0/venonactl/example/values-example.yaml) as a starting point for your values file. + +## Install Codefresh Runner with Helm + +To install the Codefresh Runner using Helm, follow these steps: + +1. Download the Codefresh CLI and authenticate it with your Codefresh account. Click [here](https://codefresh-io.github.io/cli/getting-started/) for more detailed instructions. +2. Run the following command to create all of the necessary entities in Codefresh: + + ```shell + codefresh runner init --generate-helm-values-file + ``` + + * This will not install anything on your cluster, except for running cluster acceptance tests, (which may be skipped using the `--skip-cluster-test` option). Please note, that the Runner Agent and the Runtime Environment are still created in your Codefresh account. + * This command will also generate a `generated_values.yaml` file in your current directory, which you will need to provide to the `helm install` command later. If you want to install several Codefresh Runners, you will need a separate `generated_values.yaml` file for each Runner. + +3. Now run the following to complete the installation: + + ```shell + helm repo add cf-runtime https://chartmuseum.codefresh.io/cf-runtime + + helm install cf-runtime cf-runtime/cf-runtime -f ./generated_values.yaml --create-namespace --namespace codefresh + ``` + * Here is the link to a repository with the chart for reference: [https://github.com/codefresh-io/venona/tree/release-1.0/.deploy/cf-runtime](https://github.com/codefresh-io/venona/tree/release-1.0/.deploy/cf-runtime) + +4. At this point you should have a working Codefresh Runner. You can verify the installation by running: + + ```shell + codefresh runner execute-test-pipeline --runtime-name + ``` +>**Note!**
+Runtime components' (engine and dind) configuration is determined by the `runner init` command.
+The `helm install` command can only control the configuration of `runner`, `dind-volume-provisioner` and `lv-monitor` components. + +## Using the Codefresh Runner + +Once installed, the Runner is fully automated. It polls the Codefresh SAAS (by default every 3 seconds) on its own and automatically creates all resources needed for running pipelines. + +Once installation is complete, you should see the cluster of the runner as a new [Runtime environment](https://g.codefresh.io/account-admin/account-conf/runtime-environments) in Codefresh in your *Account Settings*, in the respective tab. + +{% include image.html + lightbox="true" + file="/images/administration/runner/runtime-environments.png" + url="/images/administration/runner/runtime-environments.png" + alt="Available runtime environments" + caption="Available runtime environments" + max-width="60%" + %} + +If you have multiple environments available, you can change the default (shown with a thin blue border) by clicking on the 3 dot menu on the right of each environment. The Codefresh runner installer comes with a `set-default` option that is automatically set by default in the new runtime environment. + +You can even override the runtime environment for a specific pipeline by specifying in the respective section in the [pipeline settings]({{site.baseurl}}/docs/configure-ci-cd-pipeline/pipelines/). + +{% include image.html + lightbox="true" + file="/images/administration/runner/environment-per-pipeline.png" + url="/images/administration/runner/environment-per-pipeline.png" + alt="Running a pipeline on a specific environment" + caption="Running a pipeline on a specific environment" + max-width="60%" + %} + +## Checking the Runner + +Once installed, the runner is a normal Kubernetes application like all other applications. You can use your existing tools to monitor it. + +Only the runner pod is long living inside your cluster. All other components (such as the engine) are short lived and exist only during pipeline builds. +You can always see what the Runner is doing by listing the resources inside the namespace you chose during installation: + +```shell +$ kubectl get pods -n codefresh-runtime +NAME READY STATUS RESTARTS AGE +dind-5ee7577017ef40908b784388 1/1 Running 0 22s +dind-lv-monitor-runner-hn64g 1/1 Running 0 3d +dind-lv-monitor-runner-pj84r 1/1 Running 0 3d +dind-lv-monitor-runner-v2lhc 1/1 Running 0 3d +dind-volume-provisioner-runner-64994bbb84-lgg7v 1/1 Running 0 3d +engine-5ee7577017ef40908b784388 1/1 Running 0 22s +monitor-648b4778bd-tvzcr 1/1 Running 0 3d +runner-5d549f8bc5-7h5rc 1/1 Running 0 3d +``` + +In the same manner you can list secrets, config-maps, logs, volumes etc. for the Codefresh builds. + +## Uninstall the Codefresh Runner + +You can uninstall the Codefresh runner from your cluster by running: + +```shell +codefresh runner delete +``` + +A wizard, similar to the installation wizard, will ask you questions regarding your cluster before finishing with the removal. + +Like the installation wizard, you can pass the additional options in advance as command line parameters (see `--help` output): +```shell +codefresh runner delete --help +``` + + + +## Runner architecture overview + +{% include image.html + lightbox="true" + file="/images/administration/runner/codefresh_runner.png" + url="/images/administration/runner/codefresh_runner.png" + alt="Codefresh Runner architecture overview" + caption="Codefresh Runner architecture overview" + max-width="100%" + %} + + +1. [Runtime-Environment specification]({{site.baseurl}}/docs/administration/codefresh-runner/) defines engine and dind pods spec and PVC parameters. +2. Runner pod (Agent) pulls tasks (Builds) from Codefresh API every 3 seconds. +3. Once the agent receives build task (either Manual run build or Webhook triggered build) it calls k8s API to create engine/dind pods and PVC object. +4. Volume Provisioner listens for PVC events (create) and based on StorageClass definition it creates PV object with the corresponding underlying volume backend (ebs/gcedisk/local). +5. During the build, each step (clone/build/push/freestyle/composition) is represented as docker container inside dind (docker-in-docker) pod. Shared Volume (`/codefresh/volume`) is represented as docker volume and mounted to every step (docker containers). PV mount point inside dind pod is `/var/lib/docker`. +6. Engine pod controls dind pod. It deserializes pipeline yaml to docker API calls, terminates dind after build has been finished or per user request (sigterm). +7. `dind-lv-monitor` DaemonSet OR `dind-volume-cleanup` CronJob are part of [Runtime Cleaner]({{site.baseurl}}/docs/administration/codefresh-runner/#runtime-cleaners), `app-proxy` Deployment and Ingress are described in the [next section]({{site.baseurl}}/docs/administration/codefresh-runner/#app-proxy-installation), `monitor` Deployment is for [Kubernetes Dashboard]({{site.baseurl}}/docs/deploy-to-kubernetes/manage-kubernetes/). + +## App Proxy installation + +The App Proxy is an **optional** component of the runner that is mainly used when the git provider server is installed on-premises behind the firewall. The App Proxy provides the following features once installed: + +* Enables you to automatically create webhooks for Git in the Codefresh UI (same as the SAAS experience) +* Sends commit status information back to your Git provider (same as the SAAS experience) +* Makes all Git Operations in the GUI work exactly like the SAAS installation of Codefresh + +The requirements for the App proxy is a Kubernetes cluster that: + +1. has already the Codefresh runner installed +1. has an active [ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress/) +1. allows incoming connections from the VPC/VPN where users are browsing the Codefresh UI. The ingress connection **must** have a hostname assigned for this route and **must** be configured to perform SSL termination + +>Currently the App-proxy works only for Github (SAAS and on-prem versions), Gitlab (SAAS and on-prem versions) and Bitbucket server. + +Here is the architecture of the app-proxy: + +{% include image.html + lightbox="true" + file="/images/administration/runner/app-proxy-architecture.png" + url="/images/administration/runner/app-proxy-architecture.png" + alt="How App Proxy and the Codefresh runner work together" + caption="How App Proxy and the Codefresh runner work together" + max-width="80%" + %} + +Basically when a Git GET operation takes place, the Codefresh UI will contact the app-proxy (if it is present) and it will route the request to the backing Git provider. The confidential Git information never leaves the firewall premises and the connection between the browser and the ingress is SSL/HTTPS. + +The app-proxy has to work over HTTPS and by default it will use the ingress controller to do its SSL termination. Therefore, the ingress controller will need to be configured to perform SSL termination. Check the documentation of your ingress controller (for example [nginx ingress](https://kubernetes.github.io/ingress-nginx/examples/tls-termination/)). This means that the app-proxy does not compromise security in any way. + +To install the app-proxy on a Kubernetes cluster that already has a Codefresh runner use the following command: + +```shell +codefresh install app-proxy --host=+``` + +If you want to install the Codefresh runner and app-proxy in a single command use the following: + +```shell +codefresh runner init --app-proxy --app-proxy-host= +``` + +If you have multiple ingress controllers in the Kubernetes cluster you can use the `--app-proxy-ingress-class` parameter to define which ingress will be used. For additional security you can also define an allowlist for IPs/ranges that are allowed to use the ingress (to further limit the web browsers that can access the Ingress). Check the documentation of your ingress controller for the exact details. + +By default the app-proxy ingress will use the path `hostname/app-proxy`. You can change that default by using the values file in the installation with the flag `--values values.yaml`. + +See the `AppProxy` section in the example [values.yaml](https://github.com/codefresh-io/venona/blob/release-1.0/venonactl/example/values-example.yaml#L231-L253). + +```shell +codefresh install app-proxy --values values.yaml +``` + +## Manual Installation of Runner Components + +If you don't want to use the wizard, you can also install the components of the runner yourself. + +The Codefresh runner consists of the following: + +* Runner - responsible for getting tasks from the platform and executing them. One per account. Can handle multiple runtimes +* Runtime - the components that are responsible on runtime for the workflow execution : + * Volume provisioner - (pod’s name prefix dind-volume-provisioner-runner) - responsible for volume provisioning for dind pod + * lv-monitor - (pod’s name prefix dind-lv-monitor-runner) - daemonset - responsible for cleaning volumes + +To install the runner on a single cluster with both the runtime and the agent, execute the following: + +```shell +kubectl create namespace codefresh +codefresh install agent --agent-kube-namespace codefresh --install-runtime +``` + +You can then follow the instructions for [using the runner](#using-the-codefresh-runner). + +### Installing Multiple runtimes with a Single Agent + +It is also possible, for advanced users to install a single agent that can manage multiple runtime environments. + +>NOTE: Please make sure that the cluster where the agent is installed has network access to the other clusters of the runtimes + +```shell +# 1. Create namespace for the agent: +kubectl create namespace codefresh-agent + +# 2. Install the agent on the namespace ( give your agent a unique name as $NAME): +# Note down the token and use it in the second command. +codefresh create agent $NAME +codefresh install agent --token $TOKEN --kube-namespace codefresh-agent +codefresh get agents + +# 3. Create namespace for the first runtime: +kubectl create namespace codefresh-runtime-1 + +# 4. Install the first runtime on the namespace +# 5. the runtime name is printed +codefresh install runtime --runtime-kube-namespace codefresh-runtime-1 + +# 6. Attach the first runtime to agent: +codefresh attach runtime --agent-name $AGENT_NAME --agent-kube-namespace codefresh-agent --runtime-name $RUNTIME_NAME --runtime-kube-namespace codefresh-runtime-1 + +# 7. Restart the runner pod in namespace `codefresh-agent` +kubectl delete pods $RUNNER_POD + +# 8. Create namespace for the second runtime +kubectl create namespace codefresh-runtime-2 + +# 9. Install the second runtime on the namespace +codefresh install runtime --runtime-kube-namespace codefresh-runtime-2 + +# 10. Attach the second runtime to agent and restart the Venona pod automatically +codefresh attach runtime --agent-name $AGENT_NAME --agent-kube-namespace codefresh-agent --runtime-name $RUNTIME_NAME --runtime-kube-namespace codefresh-runtime-2 --restart-agent +``` + +## Configuration Options + +You can fine tune the installation of the runner to better match your environment and cloud provider. + +### Installing on AWS + +If you've installed the Codefresh runner on [EKS](https://aws.amazon.com/eks/) or any other custom cluster (e.g. with kops) in Amazon you need to configure it properly to work with EBS volumes in order to gain [caching]({{site.baseurl}}/docs/configure-ci-cd-pipeline/pipeline-caching/). + +> This section assumes you already installed the Runner with default options: `codefresh runner init` + +**Prerequisites** + +`dind-volume-provisioner` deployment should have permissions to create/attach/detach/delete/get ebs volumes. + +There are 3 options: +* running `dind-volume-provisioner` pod on the node (node-group) with iam role +* k8s secret with [aws credentials format](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) mounted to ~/.aws/credentials (or `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` env vars passed) to the `dind-volume-provisioner` pod +* using [Aws Identity for Service Account](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) iam role assigned to `volume-provisioner-runner` service account + +Minimal policy for `dind-volume-provisioner`: +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ec2:AttachVolume", + "ec2:CreateSnapshot", + "ec2:CreateTags", + "ec2:CreateVolume", + "ec2:DeleteSnapshot", + "ec2:DeleteTags", + "ec2:DeleteVolume", + "ec2:DescribeInstances", + "ec2:DescribeSnapshots", + "ec2:DescribeTags", + "ec2:DescribeVolumes", + "ec2:DetachVolume" + ], + "Resource": "*" + } + ] +} +``` + +Create Storage Class for EBS volumes: +>Choose **one** of the Availability Zones you want to be used for your pipeline builds. Multi AZ configuration is not supported. + +**Storage Class (gp2)** + +```yaml +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: dind-ebs +### Specify name of provisioner +provisioner: codefresh.io/dind-volume-provisioner-runner-<-NAMESPACE-> # <---- rename <-NAMESPACE-> with the runner namespace +volumeBindingMode: Immediate +parameters: + # ebs or ebs-csi + volumeBackend: ebs + # Valid zone + AvailabilityZone: us-central1-a # <---- change it to your AZ + # gp2, gp3 or io1 + VolumeType: gp2 + # in case of io1 you can set iops + # iops: 1000 + # ext4 or xfs (default to xfs, ensure that there is xfstools ) + fsType: xfs +``` +**Storage Class (gp3)** + +```yaml +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: dind-ebs +### Specify name of provisioner +provisioner: codefresh.io/dind-volume-provisioner-runner-<-NAMESPACE-> # <---- rename <-NAMESPACE-> with the runner namespace +volumeBindingMode: Immediate +parameters: + # ebs or ebs-csi + volumeBackend: ebs + # Valid zone + AvailabilityZone: us-central1-a # <---- change it to your AZ + # gp2, gp3 or io1 + VolumeType: gp3 + # ext4 or xfs (default to xfs, ensure that there is xfstools ) + fsType: xfs + # I/O operations per second. Only effetive when gp3 volume type is specified. + # Default value - 3000. + # Max - 16,000 + iops: "5000" + # Throughput in MiB/s. Only effective when gp3 volume type is specified. + # Default value - 125. + # Max - 1000. + throughput: "500" +``` + +Apply storage class manifest: +```shell +kubectl apply -f dind-ebs.yaml +``` + +Change your [runtime environment]({{site.baseurl}}/docs/administration/codefresh-runner/#full-runtime-environment-specification) configuration: + +The same AZ you selected before should be used in nodeSelector inside Runtime Configuration: + +To get a list of all available runtimes execute: + +```shell +codefresh get runtime-environments +``` + +Choose the runtime you have just added and get its yaml representation: + +```shell +codefresh get runtime-environments my-eks-cluster/codefresh -o yaml > runtime.yaml +``` + + Under `dockerDaemonScheduler.cluster` block add the nodeSelector `topology.kubernetes.io/zone: `. It should be at the same level as `clusterProvider` and `namespace`. Also, the `pvcs.dind` block should be modified to use the Storage Class you created above (`dind-ebs`). + +`runtime.yaml` example: + +```yaml +version: 1 +metadata: + ... +runtimeScheduler: + cluster: + clusterProvider: + accountId: 5f048d85eb107d52b16c53ea + selector: my-eks-cluster + namespace: codefresh + serviceAccount: codefresh-engine + annotations: {} +dockerDaemonScheduler: + cluster: + clusterProvider: + accountId: 5f048d85eb107d52b16c53ea + selector: my-eks-cluster + namespace: codefresh + nodeSelector: + topology.kubernetes.io/zone: us-central1-a + serviceAccount: codefresh-engine + annotations: {} + userAccess: true + defaultDindResources: + requests: '' + pvcs: + dind: + volumeSize: 30Gi + storageClassName: dind-ebs + reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName' +extends: + - system/default/hybrid/k8s_low_limits +description: '...' +accountId: 5f048d85eb107d52b16c53ea +``` + +Update your runtime environment with the [patch command](https://codefresh-io.github.io/cli/operate-on-resources/patch/): + +```shell +codefresh patch runtime-environment my-eks-cluster/codefresh -f runtime.yaml +``` + +If necessary, delete all existing PV and PVC objects left from default local provisioner: +``` +kubectl delete pvc -l codefresh-app=dind -n +kubectl delete pv -l codefresh-app=dind -n +``` + +>You can define all these options above for clean Runner installation with [values.yaml](https://github.com/codefresh-io/venona/blob/release-1.0/venonactl/example/values-example.yaml) file: + +`values-ebs.yaml` example: + +```yaml +### Storage parameter example for aws ebs disks +Storage: + Backend: ebs + AvailabilityZone: us-east-1d + VolumeType: gp3 + #AwsAccessKeyId: ABCDF + #AwsSecretAccessKey: ZYXWV + Encrypted: # encrypt volume, default is false + VolumeProvisioner: + ServiceAccount: + Annotations: + eks.amazonaws.com/role-arn: arn:aws:iam:: :role/ +NodeSelector: topology.kubernetes.io/zone=us-east-1d +... + Runtime: + NodeSelector: # dind and engine pods node-selector (--build-node-selector) + topology.kubernetes.io/zone: us-east-1d +``` + +```shell +codefresh runner init --values values-ebs.yaml --exec-demo-pipeline false --skip-cluster-integration true +``` + +### Installing to EKS with Autoscaling + +#### Step 1- EKS Cluster Creation + +See below is a content of cluster.yaml file. We define separate node pools for dind, engine and other services(like runner, cluster-autoscaler etc). + +Before creating the cluster we have created two separate IAM policies: + +* one for our volume-provisioner controller(policy/runner-ebs) that should create and delete volumes +* one for dind pods(policy/dind-ebs) that should be able to attach/detach those volumes to the appropriate nodes using [iam attachPolicyARNs options](https://eksctl.io/usage/iam-policies/). + +`policy/dind-ebs:` + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ec2:DescribeVolumes" + ], + "Resource": [ + "*" + ] + }, + { + "Effect": "Allow", + "Action": [ + "ec2:DetachVolume", + "ec2:AttachVolume" + ], + "Resource": [ + "*" + ] + } + ] +} +``` + +`policy/runner-ebs:` + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ec2:AttachVolume", + "ec2:CreateSnapshot", + "ec2:CreateTags", + "ec2:CreateVolume", + "ec2:DeleteSnapshot", + "ec2:DeleteTags", + "ec2:DeleteVolume", + "ec2:DescribeInstances", + "ec2:DescribeSnapshots", + "ec2:DescribeTags", + "ec2:DescribeVolumes", + "ec2:DetachVolume" + ], + "Resource": "*" + } + ] +} +``` + +`my-eks-cluster.yaml` + +```yaml +apiVersion: eksctl.io/v1alpha5 +kind: ClusterConfig +metadata: + name: my-eks + region: us-west-2 + version: "1.15" + +nodeGroups: + - name: dind + instanceType: m5.2xlarge + desiredCapacity: 1 + iam: + attachPolicyARNs: + - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy + - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy + - arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess + - arn:aws:iam::XXXXXXXXXXXX:policy/dind-ebs + withAddonPolicies: + autoScaler: true + ssh: # import public key from file + publicKeyPath: ~/.ssh/id_rsa.pub + minSize: 1 + maxSize: 50 + volumeSize: 50 + volumeType: gp2 + ebsOptimized: true + availabilityZones: ["us-west-2a"] + kubeletExtraConfig: + enableControllerAttachDetach: false + labels: + node-type: dind + taints: + codefresh.io: "dinds:NoSchedule" + + - name: engine + instanceType: m5.large + desiredCapacity: 1 + iam: + withAddonPolicies: + autoScaler: true + minSize: 1 + maxSize: 10 + volumeSize: 50 + volumeType: gp2 + availabilityZones: ["us-west-2a"] + labels: + node-type: engine + taints: + codefresh.io: "engine:NoSchedule" + + - name: addons + instanceType: m5.2xlarge + desiredCapacity: 1 + ssh: # import public key from file + publicKeyPath: ~/.ssh/id_rsa.pub + minSize: 1 + maxSize: 10 + volumeSize: 50 + volumeType: gp2 + ebsOptimized: true + availabilityZones: ["us-west-2a"] + labels: + node-type: addons + iam: + attachPolicyARNs: + - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy + - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy + - arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess + - arn:aws:iam::XXXXXXXXXXXX:policy/runner-ebs + withAddonPolicies: + autoScaler: true +availabilityZones: ["us-west-2a", "us-west-2b", "us-west-2c"] +``` + +Execute: + +```shell +eksctl create cluster -f my-eks-cluster.yaml +``` + +The config above will leverage [Amazon Linux 2](https://aws.amazon.com/amazon-linux-2/) as the default operating system for the nodes in the nodegroup. To leverage [Bottlerocket-based nodes](https://aws.amazon.com/bottlerocket/), specify the AMI Family using `amiFamily: Bottlerocket` and add the following additional IAM Policies: `arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly` and `arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore`. + +>Bottlerocket is an open source Linux based Operating System specifically built to run containers. It focuses on security, simplicity and easy updates via transactions. Find more information in the [official repository](https://github.com/bottlerocket-os/bottlerocket). + +#### Step 2 - Autoscaler + +Once the cluster is up and running we need to install the [cluster autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html): + +We used iam AddonPolicies `"autoScaler: true"` in the cluster.yaml file so there is no need to create a separate IAM policy or add Auto Scaling group tags, everything is done automatically. + +Deploy the Cluster Autoscaler: + +```shell +kubectl apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml +``` + +Add the `cluster-autoscaler.kubernetes.io/safe-to-evict` annotation + +```shell +kubectl -n kube-system annotate deployment.apps/cluster-autoscaler cluster-autoscaler.kubernetes.io/safe-to-evict="false" +``` + +Edit the cluster-autoscaler container command to replace ` ` with *my-eks*(name of the cluster from cluster.yaml file), and add the following options: + `--balance-similar-node-groups` and `--skip-nodes-with-system-pods=false` + +```shell +kubectl -n kube-system edit deployment.apps/cluster-autoscaler +``` + +```yaml +spec: + containers: + - command: + - ./cluster-autoscaler + - --v=4 + - --stderrthreshold=info + - --cloud-provider=aws + - --skip-nodes-with-local-storage=false + - --expander=least-waste + - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-eks + - --balance-similar-node-groups + - --skip-nodes-with-system-pods=false +``` + +We created our EKS cluster with 1.15 version so the appropriate cluster autoscaler version from [https://github.com/kubernetes/autoscaler/releases](https://github.com/kubernetes/autoscaler/releases) is 1.15.6 + +```shell +kubectl -n kube-system set image deployment.apps/cluster-autoscaler cluster-autoscaler=us.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v1.15.6 +``` + +Check your own version to make sure that the autoscaler version is appropriate. + +#### Step 3 - Optional: We also advise to configure overprovisioning with Cluster Autoscaler + +See details at the [FAQ]( +https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-configure-overprovisioning-with-cluster-autoscaler). + +#### Step 4 - Adding an EKS cluster as a runner to the Codefresh platform with EBS support + +Make sure that you are targeting the correct cluster + +```shell +$ kubectl config current-context +my-aws-runner +``` + +Install the runner passing additional options: + +```shell +codefresh runner init \ +--name my-aws-runner \ +--kube-node-selector=topology.kubernetes.io/zone=us-west-2a \ +--build-node-selector=topology.kubernetes.io/zone=us-west-2a \ +--kube-namespace cf --kube-context-name my-aws-runner \ +--set-value Storage.VolumeProvisioner.NodeSelector=node-type=addons \ +--set-value=Storage.Backend=ebs \ +--set-value=Storage.AvailabilityZone=us-west-2a +``` + +* You should specify the zone in which you want your volumes to be created, example: `--set-value=Storage.AvailabilityZone=us-west-2a` +* (Optional) - if you want to assign the volume-provisioner to a specific node, for example a specific node group what has an IAM role which allows to create EBS volumes, example: `--set-value Storage.VolumeProvisioner.NodeSelector=node-type=addons` + +If you want to use [encrypted EBS volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html#EBSEncryption_key_mgmt) (they are unencrypted by default) - add the custom value `--set-value=Storage.Encrypted=true` +If you already have a key - add its ARN via `--set-value=Storage.KmsKeyId= value`, otherwise a key is generated by AWS. Here is the full command: + +```shell +codefresh runner init \ +--name my-aws-runner \ +--kube-node-selector=topology.kubernetes.io/zone=us-west-2a \ +--build-node-selector=topology.kubernetes.io/zone=us-west-2a \ +--kube-namespace cf --kube-context-name my-aws-runner \ +--set-value Storage.VolumeProvisioner.NodeSelector=node-type=addons \ +--set-value=Storage.Backend=ebs \ +--set-value=Storage.AvailabilityZone=us-west-2a\ +--set-value=Storage.Encrypted=[false|true] \ +--set-value=Storage.KmsKeyId= +``` + +For an explanation of all other options run `codefresh runner init --help` ([global parameter table](#customizing-the-wizard-installation)). + +At this point the quick start wizard will start the installation. + +Once that is done we need to modify the runtime environment of `my-aws-runner` to specify the necessary toleration, nodeSelector and disk size: + +```shell +codefresh get re --limit=100 my-aws-runner/cf -o yaml > my-runtime.yml +``` + +Modify the file my-runtime.yml as shown below: + +```yaml +version: null +metadata: + agent: true + trial: + endingAt: 1593596844167 + reason: Codefresh hybrid runtime + started: 1592387244207 + name: my-aws-runner/cf + changedBy: ivan-codefresh + creationTime: '2020/06/17 09:47:24' +runtimeScheduler: + cluster: + clusterProvider: + accountId: 5cb563d0506083262ba1f327 + selector: my-aws-runner + namespace: cf + nodeSelector: + node-type: engine + tolerations: + - effect: NoSchedule + key: codefresh.io + operator: Equal + value: engine + annotations: {} +dockerDaemonScheduler: + cluster: + clusterProvider: + accountId: 5cb563d0506083262ba1f327 + selector: my-aws-runner + namespace: cf + nodeSelector: + node-type: dind + annotations: {} + defaultDindResources: + requests: '' + tolerations: + - effect: NoSchedule + key: codefresh.io + operator: Equal + value: dinds + pvcs: + dind: + volumeSize: 30Gi + reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName' + storageClassName: dind-local-volumes-runner-cf + userAccess: true +extends: + - system/default/hybrid/k8s_low_limits +description: 'Runtime environment configure to cluster: my-aws-runner and namespace: cf' +accountId: 5cb563d0506083262ba1f327 +``` + +Apply changes. + +```shell +codefresh patch re my-aws-runner/cf -f my-runtime.yml +``` + +That's all. Now you can go to UI and try to run a pipeline on RE my-aws-runner/cf + +### Injecting AWS arn roles into the cluster + +**Step 1** - Make sure the OIDC provider is connected to the cluster + +See: + +* [https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) +* [https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/) + +**Step 2** - Create IAM role and policy as explained in [https://docs.aws.amazon.com/eks/latest/userguide/create-service-account-iam-policy-and-role.html](https://docs.aws.amazon.com/eks/latest/userguide/create-service-account-iam-policy-and-role.html) + +Here, in addition to the policy explained, you need a Trust Relationship established between this role and the OIDC entity. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { + "${OIDC_PROVIDER}:sub": "system:serviceaccount:${CODEFRESH_NAMESPACE}:codefresh-engine" + } + } + } + ] +} +``` + +**Step 3** - Annotate the `codefresh-engine` Kubernetes Service Account in the namespace where the Codefresh Runner is installed with the proper IAM role. + +```shell +kubectl annotate -n ${CODEFRESH_NAMESPACE} sa codefresh-engine eks.amazonaws.com/role-arn=${ROLE_ARN} +``` + +Once the annotation is added, you should see it when you describe the Service Account. + +```shell +kubectl describe -n ${CODEFRESH_NAMESPACE} sa codefresh-engine + +Name: codefresh-engine +Namespace: codefresh +Labels: app=app-proxy + version=1.6.8 +Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/Codefresh +Image pull secrets: +Mountable secrets: codefresh-engine-token-msj8d +Tokens: codefresh-engine-token-msj8d +Events: +``` + +**Step 4** - Using the AWS assumed role identity + +After annotating the Service Account, run a pipeline to test the AWS resource access: + +```yaml +RunAwsCli: + title : Communication with AWS + image : mesosphere/aws-cli + stage: "build" + commands : + - apk update + - apk add jq + - env + - cat /codefresh/volume/sensitive/.kube/web_id_token + - aws sts assume-role-with-web-identity --role-arn $AWS_ROLE_ARN --role-session-name mh9test --web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE --duration-seconds 1000 > /tmp/irp-cred.txt + - export AWS_ACCESS_KEY_ID="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.AccessKeyId")" + - export AWS_SECRET_ACCESS_KEY="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SecretAccessKey")" + - export AWS_SESSION_TOKEN="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SessionToken")" + - rm /tmp/irp-cred.txt + - aws s3api get-object --bucket jags-cf-eks-pod-secrets-bucket --key eks-pod2019-12-10-21-18-32-560931EEF8561BC4 getObjectNotWorks.txt +``` + +### Installing behind a proxy + +If you want to deploy the Codefresh runner on a Kubernetes cluster that doesn’t have direct access to `g.codefresh.io`, and has to go trough a proxy server to access `g.codefresh.io`, you will need to follow these additional steps: + +**Step 1** - Follow the installation instructions of the previous section + +**Step 2** - Run `kubectl edit deployment runner -n codefresh-runtime` and add the proxy variables like this + +```yaml +spec: + containers: + - env: + - name: HTTP_PROXY + value: http:// :port + - name: HTTPS_PROXY + value: http:// :port + - name: http_proxy + value: http:// :port + - name: https_proxy + value: http:// :port + - name: no_proxy + value: localhost,127.0.0.1, + - name: NO_PROXY + value: localhost,127.0.0.1, +``` + +**Step 3** - Add the following variables to your runtime.yaml, both under the `runtimeScheduler:` and under `dockerDaemonScheduler:` blocks inside the `envVars:` section + +```yaml +HTTP_PROXY: http:// :port +http_proxy: http:// :port +HTTPS_PROXY: http:// :port +https_proxy: http:// :port +No_proxy: localhost, 127.0.0.1, +NO_PROXY: localhost, 127.0.0.1, +``` + +**Step 4** - Add `.firebaseio.com` to the allowed-sites of the proxy server + +**Step 5** - Exec into the `dind` pod and run `ifconfig` + +If the MTU value for `docker0` is higher than the MTU value of `eth0` (sometimes the `docker0` MTU is 1500, while `eth0` MTU is 1440) - you need to change this, the `docker0` MTU should be lower than `eth0` MTU + +To fix this, edit the configmap in the codefresh-runtime namespace: + +```shell +kubectl edit cm codefresh-dind-config -n codefresh-runtime +``` + +And add this after one of the commas: +`\"mtu\":1440,` + +### Installing on Rancher RKE 2.X + +#### Step 1 - Configure the kubelet to work with the runner's StorageClass + +The runner's default StorageClass creates the persistent cache volume from local storage on each node. We need to edit the cluster config to allow this. + +In the Rancher UI (v2.5.9 and earlier), drill into the target cluster and then click the Edit Cluster button at the top-right. +{% include image.html + lightbox="true" + file="/images/administration/runner/rancher-cluster.png" + url="/images/administration/runner/rancher-cluster.png" + alt="Drill into your cluster and click Edit Cluster on the right" + caption="Drill into your cluster and click Edit Cluster on the right" + max-width="100%" + %} + +In Rancher v2.6+ with the updated UI, open the Cluster Management in the left panel, then click the three-dot menu near the corresponding cluster and select 'Edit Config'. +{% include image.html + lightbox="true" + file="/images/administration/runner/rancher-cluster-2.png" + url="/images/administration/runner/rancher-cluster-2.png" + alt="Click Edit Cluster on the right in your cluster list" + caption="Click Edit Cluster on the right in your cluster list" + max-width="100%" + %} + +On the edit cluster page, scroll down to the Cluster Options section and click its **Edit as YAML** button +{% include image.html + lightbox="true" + file="/images/administration/runner/rancher-edit-as-yaml.png" + url="/images/administration/runner/rancher-edit-as-yaml.png" + alt="Cluster Options -> Edit as YAML" + caption="Cluster Options -> Edit as YAML" + max-width="100%" + %} +Edit the YAML to include an extra mount in the kubelet service: + +```yaml +rancher_kubernetes_engine_config: + ... + services: + ... + kubelet: + extra_binds: + - '/var/lib/codefresh:/var/lib/codefresh:rshared' +``` + +{% include image.html + lightbox="true" + file="/images/administration/runner/rancher-kublet.png" + url="/images/administration/runner/rancher-kublet.png" + alt="Add volume to rancher_kubernetes_engine_config.services.kublet.extra_binds" + caption="Add volume to rancher_kubernetes_engine_config.services.kublet.extra_binds" + max-width="100%" + %} + +#### Step 2 - Make sure your kubeconfig user is a ClusterAdmin + +The user in your kubeconfig must be a cluster admin in order to install the runner. If you plan to have your pipelines connect to this cluster as a cluster admin, then you can go ahead and create a Codefresh user for this purpose in the Rancher UI with a **non-expiring** kubeconfig token. This is the easiest way to do the installation. + +However, if you want your pipelines to connect to this cluster with less privileges, then you can use your personal user account with Cluster Admin privileges for the installation, and then we'll create a Codefresh account with lesser privileges later (in Step 5). In that case, you can now move on to Step 3. + +Follow these steps to create a Codefresh user with Cluster Admin rights, from the Rancher UI: + +* Click Security at the top, and then choose Users + {% include image.html lightbox="true" file="/images/administration/runner/rancher-security.png" url="/images/administration/runner/rancher-security.png" alt="Create a cluster admin user for Codefresh" caption="Create a cluster admin ser for Codefresh" max-width="100%" %} +* Click the Add User button, and under Global Permissions check the box for **Restricted Administrstor** +* Log out of the Rancher UI, and then log back in as the new user +* Click your user icon at the top-right, and then choose **API & Keys** +* Click the **Add Key** button and create a kubeconfig token with Expires set to Never +* Copy the Bearer Token field (combines Access Key and Secret Key) +* Edit your kubeconfig and put the Bearer Token you copied in the `token` field of your user + +#### Step 3 - Install the Runner + +If you've created your kubeconfig from the Rancher UI, then it will contain an API endpoint that is not reachable internally, from within the cluster. To work around this, we need to tell the runner to instead use Kubernetes' generic internal API endpoint. Also, if you didn't create a Codefresh user in step 2 and your kubeconfig contains your personal user account, then you should also add the `--skip-cluster-integration` option. + +Install the runner with a Codefresh user (ClusterAdmin, non-expiring token): + +```shell +codefresh runner init \ + --set-value KubernetesHost=https://kubernetes.default.svc.cluster.local +``` + +Or install the runner with your personal user account: + +```shell +codefresh runner init \ + --set-value KubernetesHost=https://kubernetes.default.svc.cluster.local \ + --skip-cluster-integration +``` + +The wizard will then ask you some basic questions. + +#### Step 4 - Update the runner's Docker MTU + +By default, RKE nodes use the [Canal CNI](https://rancher.com/docs/rancher/v2.x/en/faq/networking/cni-providers/#canal), which combines elements of Flannel and Calico, and uses VXLAN encapsulation. This VXLAN encapsulation has a 50-byte overhead, thus reducing the MTU of its virtual interfaces from the standard 1500 to 1450. For example, when running `ifconfig` on an RKE 2.5.5 node, you might see several interfaces like this. Note the `MTU:1450`. + +```shell +cali0f8ac592086 Link encap:Ethernet HWaddr ee:ee:ee:ee:ee:ee + inet6 addr: fe80::ecee:eeff:feee:eeee/64 Scope:Link + UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 + RX packets:11106 errors:0 dropped:0 overruns:0 frame:0 + TX packets:10908 errors:0 dropped:0 overruns:0 carrier:0 + collisions:0 txqueuelen:0 + RX bytes:922373 (922.3 KB) TX bytes:9825590 (9.8 MB) +``` + +We must reduce the Docker MTU used by the runner's Docker in Docker (dind) pods to fit within this lower MTU. This is stored in a configmap in the namespace where the runner is installed. Assuming that you installed the runner into the `codefresh` namespace, you would edit the configmap like this: + +```shell +kubectl edit cm codefresh-dind-config -n codefresh +``` + +In the editor, update the **daemon.json** field - add `,\"mtu\":1440` just before the last curley brace. + {% include image.html + lightbox="true" + file="/images/administration/runner/rancher-mtu.png" + url="/images/administration/runner/rancher-mtu.png" + alt="Update the runner's Docker MTU" + caption="Update the runner's Docker MTU" + max-width="100%" + %} + +#### Step 5 - Create the Cluster Integration + +If you created a user in Step 2 and used it to install the runner in Step 3, then you can skip this step - your installation is complete! + +However, if you installed the runner with the `--skip-cluster-integration` option then you should follow the documentaion to [Add a Rancher Cluster]({{site.baseurl}}/docs/deploy-to-kubernetes/add-kubernetes-cluster/#adding-a-rancher-cluster) to your Kubernetes Integrations. + +Once complete, you can go to the Codefresh UI and run a pipeline on the new runtime, including steps that deploy to the Kubernetes Integration. + +#### Troubleshooting TLS Errors + +Depending on your Rancher configuration, you may need to allow insecure HTTPS/TLS connections. You can do this by adding an environment variable to the runner deployment. + +Assuming that you installed the runner into the `codefresh` namespace, you would edit the runner deployment like this: + +```shell +kubectl edit deploy runner -n codefresh +``` + +In the editor, add this environment variable under spec.containers.env[]: + +```yaml +- name: NODE_TLS_REJECT_UNAUTHORIZED + value: "0" +``` + +### Installing on Google Kubernetes Engine + +If you are installing Codefresh runner on the Kubernetes cluster on [GKE](https://cloud.google.com/kubernetes-engine/) + +* make sure your user has `Kubernetes Engine Cluster Admin` role in google console and +* bind your user with `cluster-admin` Kubernetes cluster role. + +```shell +kubectl create clusterrolebinding cluster-admin-binding \ + --clusterrole cluster-admin \ + --user $(gcloud config get-value account) +``` + + +#### Storage options on GKE + +**Local SSD** + +If you want to use *LocalSSD* in GKE: + +*Prerequisites:* [GKE cluster with local SSD](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd) + +Install Runner with the Wizard: + +```shell +codefresh runner init [options] --set-value=Storage.LocalVolumeParentDir=/mnt/disks/ssd0/codefresh-volumes \ + --build-node-selector=cloud.google.com/gke-local-ssd=true +``` + +Or with `values-example.yaml` values file: + +```yaml +... +### Storage parameters example for gke-local-ssd + Storage: + Backend: local + LocalVolumeParentDir: /mnt/disks/ssd0/codefresh-volumes + NodeSelector: cloud.google.com/gke-local-ssd=true +... + Runtime: + NodeSelector: # dind and engine pods node-selector (--build-node-selector) + cloud.google.com/gke-local-ssd: "true" +... +``` +```shell +codefresh runner init [options] --values values-example.yaml +``` + +To configure existing Runner with Local SSDs follow this article: + +[How-to: Configuring an existing Runtime Environment with Local SSDs (GKE only)](https://support.codefresh.io/hc/en-us/articles/360016652920-How-to-Configuring-an-existing-Runtime-Environment-with-Local-SSDs-GKE-only-) + + +**GCE Disks** + +If you want to use *GCE Disks*: + +*Prerequisites:* volume provisioner (dind-volume-provisioner) should have permissions to create/delete/get GCE disks + +There are 3 options to provide cloud credentials: + +* run `dind-volume-provisioner-runner` pod on a node with IAM role which is allowed to create/delete/get GCE disks +* create Google Service Account with `ComputeEngine.StorageAdmin` role, download its key in JSON format and pass it to `codefresh runner init` with `--set-file=Storage.GooogleServiceAccount=/path/to/google-service-account.json` +* use [Google Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) to assign IAM role to `volume-provisioner-runner` service account + +Notice that builds will be running in a single availability zone, so you must specify AvailabilityZone parameters. + + +##### Runner installation with GCE Disks (Google SA JSON key) + +Using the Wizard: + +```shell +codefresh runner init [options] \ + --set-value=Storage.Backend=gcedisk \ + --set-value=Storage.AvailabilityZone=us-central1-c \ + --kube-node-selector=topology.kubernetes.io/zone=us-central1-c \ + --build-node-selector=topology.kubernetes.io/zone=us-central1-c \ + --set-file=Storage.GoogleServiceAccount=/path/to/google-service-account.json +``` + +Using the values `values-example.yaml` file: +```yaml +... +### Storage parameter example for GCE disks + Storage: + Backend: gcedisk + AvailabilityZone: us-central1-c + GoogleServiceAccount: > #serviceAccount.json content + { + "type": "service_account", + "project_id": "...", + "private_key_id": "...", + "private_key": "...", + "client_email": "...", + "client_id": "...", + "auth_uri": "...", + "token_uri": "...", + "auth_provider_x509_cert_url": "...", + "client_x509_cert_url": "..." + } + NodeSelector: topology.kubernetes.io/zone=us-central1-c +... + Runtime: + NodeSelector: # dind and engine pods node-selector (--build-node-selector) + topology.kubernetes.io/zone: us-central1-c +... +``` +```shell +codefresh runner init [options] --values values-example.yaml +``` + + +##### Runner installation with GCE Disks (Workload Identity with IAM role) + +Using the values `values-example.yaml` file: + +```yaml +... +### Storage parameter example for GCE disks + Storage: + Backend: gcedisk + AvailabilityZone: us-central1-c + VolumeProvisioner: + ServiceAccount: + Annotations: #annotation to the volume-provisioner service account, using the email address of the Google service account + iam.gke.io/gcp-service-account: @ .iam.gserviceaccount.com + NodeSelector: topology.kubernetes.io/zone=us-central1-c +... + Runtime: + NodeSelector: # dind and engine pods node-selector (--build-node-selector) + topology.kubernetes.io/zone: us-central1-c +... +``` +```shell +codefresh runner init [options] --values values-example.yaml +``` + +Create the binding between Kubernetes service account and Google service account: + +```shell +export K8S_NAMESPACE=codefresh +export KSA_NAME=volume-provisioner-runner +export GSA_NAME= +export PROJECT_ID= + +gcloud iam service-accounts add-iam-policy-binding \ + --role roles/iam.workloadIdentityUser \ + --member "serviceAccount:${PROJECT_ID}.svc.id.goog[${K8S_NAMESPACE}/${KSA_NAME}]" \ + ${GSA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com +``` + +To configure existing Runner with GCE Disks follow this article: + +[How-to: Configuring an existing Runtime Environment with GCE disks](https://support.codefresh.io/hc/en-us/articles/360016652900-How-to-Configuring-an-existing-Runtime-Environment-with-GCE-disks) + + +##### Using multiple Availability Zones + +Currently, to support effective caching with GCE disks, the builds/pods need to be scheduled in a single AZ (this is more related to a GCP limitation than a Codefresh runner issue). + +If you have Kubernetes nodes running in multiple Availability Zones and wish to use the Codefresh runner we suggest the following: + +**Option A** - Provision a new Kubernetes cluster: a cluster that runs in a single AZ only. - The cluster should be dedicated for usage with the Codefresh runner. This is the preferred solution and avoids extra complexity. + +**Option B** - Install Codefresh runner in your multi-zone cluster, and let it run in the default Node Pool: - in this case, you must specify `--build-node-selector= ` (e.g.: `--build-node-selector=topology.kubernetes.io/zone=us-central1-c`) or simply modify the Runtime environment as below: + +```shell +codefresh get re $RUNTIME_NAME -o yaml > re.yaml +``` + +Edit the yaml: + +```yaml +version: 2 +metadata: + ... +runtimeScheduler: + cluster: + nodeSelector: #schedule engine pod onto a node whose labels match the nodeSelector + topology.kubernetes.io/zone: us-central1-c + ... +dockerDaemonScheduler: + cluster: + nodeSelector: #schedule dind pod onto a node whose labels match the nodeSelector + topology.kubernetes.io/zone: us-central1-c + ... + pvcs: + dind: + ... +``` + +Apply changes with: + +```shell +codefresh patch re -f re.yaml +``` + +**Option C** - Like option B, but with a dedicated Node Pool + +**Option D** - Have 2 separate Codefresh runner Runtimes, one for zone A, and the other for zone B, and so on: this technically works, but it will require you to manually set the RE to use for the pipelines that won't use the default Codefresh runner RE. To distribute the pipeline's builds across the Codefresh runner REs. + +For example, let's say Venona-zoneA is the default RE, then, that means that for the pipelines that you want to run in Venona-zoneB, then you'll need to modify their RE settings, and explicitly set Venona-zoneB as the one to use. + +Regarding [Regional Persistent Disks](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/regional-pd), their support is not currently implemented in the Codefresh runner. + + +### Installing on AKS + +**Azure Disks** + +*Prerequisite:* volume provisioner (`dind-volume-provisioner`) should have permissions to create/delete/get Azure Disks + +Minimal IAM Role for dind-volume-provisioner:
+`dind-volume-provisioner-role.json` +```json +{ + "Name": "CodefreshDindVolumeProvisioner", + "Description": "Perform create/delete/get disks", + "IsCustom": true, + "Actions": [ + "Microsoft.Compute/disks/read", + "Microsoft.Compute/disks/write", + "Microsoft.Compute/disks/delete" + + ], + "AssignableScopes": ["/subscriptions/"] +} +``` + +If you use AKS with managed [identities for node group](https://docs.microsoft.com/en-us/azure/aks/use-managed-identity), you can run the script below to assign `CodefreshDindVolumeProvisioner` role to aks node identity: + +```shell +export ROLE_DEFINITIN_FILE=dind-volume-provisioner-role.json +export SUBSCRIPTION_ID=$(az account show --query "id" | xargs echo ) +export RESOURCE_GROUP=codefresh-rt1 +export AKS_NAME=codefresh-rt1 +export LOCATION=$(az aks show -g $RESOURCE_GROUP -n $AKS_NAME --query location | xargs echo) +export NODES_RESOURCE_GROUP=MC_${RESOURCE_GROUP}_${AKS_NAME}_${LOCATION} +export NODE_SERVICE_PRINCIPAL=$(az aks show -g $RESOURCE_GROUP -n $AKS_NAME --query identityProfile.kubeletidentity.objectId | xargs echo) + +az role definition create --role-definition @${ROLE_DEFINITIN_FILE} +az role assignment create --assignee $NODE_SERVICE_PRINCIPAL --scope /subscriptions/$SUBSCRIPTION_ID/resourceGroups/$NODES_RESOURCE_GROUP --role CodefreshDindVolumeProvisioner +``` + +Now install Codefresh Runner with cli wizard: +```shell +codefresh runner init --set-value Storage.Backend=azuredisk --set Storage.VolumeProvisioner.MountAzureJson=true +``` +Or using [values-example.yaml](https://github.com/codefresh-io/venona/blob/release-1.0/venonactl/example/values-example.yaml): +```yaml +Storage: + Backend: azuredisk + VolumeProvisioner: + MountAzureJson: true +``` +```shell +codefresh runner init --values values-example.yaml +``` +Or with helm chart [values.yaml](https://github.com/codefresh-io/venona/blob/release-1.0/charts/cf-runtime/values.yaml): +```yaml +storage: + backend: azuredisk + azuredisk: + skuName: Premium_LRS + +volumeProvisioner: + mountAzureJson: true +``` +```shell +helm install cf-runtime cf-runtime/cf-runtime -f ./generated_values.yaml -f values.yaml --create-namespace --namespace codefresh +``` + + +### Internal Registry Mirror + +You can configure your Codefresh Runner to use an internal registry as a mirror for any container images that are mentioned in your pipelines. + +First setup an internal registry as described in [https://docs.docker.com/registry/recipes/mirror/](https://docs.docker.com/registry/recipes/mirror/). + +Then locate the `codefresh-dind-config` config map in the namespace that houses the runner and edit it. + +```shell +kubectl -n codefresh edit configmap codefresh-dind-config +``` + +Change the `data` field from: + +```yaml +data: + daemon.json: "{\n \"hosts\": [ \"unix:///var/run/docker.sock\",\n \"tcp://0.0.0.0:1300\"],\n + \ \"storage-driver\": \"overlay2\",\n \"tlsverify\": true, \n \"tls\": true,\n + \ \"tlscacert\": \"/etc/ssl/cf-client/ca.pem\",\n \"tlscert\": \"/etc/ssl/cf/server-cert.pem\",\n + \ \"tlskey\": \"/etc/ssl/cf/server-key.pem\",\n \"insecure-registries\" : [\"192.168.99.100:5000\"],\n + \ \"metrics-addr\" : \"0.0.0.0:9323\",\n \"experimental\" : true\n}\n" +``` + +to + +```yaml +data: + daemon.json: "{\n \"hosts\": [ \"unix:///var/run/docker.sock\",\n \"tcp://0.0.0.0:1300\"],\n + \ \"storage-driver\": \"overlay2\",\n \"tlsverify\": true, \n \"tls\": true,\n + \ \"tlscacert\": \"/etc/ssl/cf-client/ca.pem\",\n \"tlscert\": \"/etc/ssl/cf/server-cert.pem\",\n + \ \"tlskey\": \"/etc/ssl/cf/server-key.pem\",\n \"insecure-registries\" : [\"192.168.99.100:5000\"],\n + \ \"registry-mirrors\": [ \"https:// \" ], \n + \ \"metrics-addr\" : \"0.0.0.0:9323\",\n \"experimental\" : true\n}\n" +``` + +This adds the line `\ \"registry-mirrors\": [ \"https:// \" ], \n` which contains a single registry to use as a mirror. Quit and Save by typing `:wq`. + +Now any container image that is used in your pipeline and isn't fully qualified, will be pulled through the Docker registry that is configured as a mirror. + + +### Installing the monitoring component + +If your cluster is located [behind the firewall](https://codefresh.io/docs/docs/administration/behind-the-firewall/) you might want to use the runner monitoring component to get valuable information about the cluster resources to Codefresh, for example, to [Kubernetes](https://g.codefresh.io/kubernetes/services/) and [Helm Releases](https://g.codefresh.io/helm/releases/releasesNew/) dashboards. + +To install the monitoring component you can use `--install-monitor` flag in the `runner init` command: + +```shell +codefresh runner init --install-monitor +``` + +Please note, that the monitoring component will not be installed if you use `--install-monitor` with `--skip-cluster-integration` flag. In case you want to skip adding the cluster integration during the runner installation, but still want to get the cluster resources to Codefresh dashboards, you can install the monitoring component separately: + +```shell +codefresh install monitor --kube-context-name --kube-namespace --cluster-id --token +``` + + + +## Full runtime environment specification + +The following section contains an explanation of runtime environment specification and possible options to modify it. Notice that there are additional and hidden fields that are autogenerated by Codefresh that complete a full runtime spec. You can't directly see or edit them (unless you run your own [Codefresh On-Premises Installation]({{site.baseurl}}/docs/administration/codefresh-on-prem/) ) + + +To get a list of all available runtimes execute: +```shell +codefresh get runtime-environments +#or +codefresh get re +``` + +Choose the runtime that you want to inspect or modify and get its yaml/json representation: +```shell +codefresh get re my-eks-cluster/codefresh -o yaml > runtime.yaml +#or +codefresh get re my-eks-cluster/codefresh -o json > runtime.json +``` + +Update your runtime environment with the [patch command](https://codefresh-io.github.io/cli/operate-on-resources/patch/): +```shell +codefresh patch re my-eks-cluster/codefresh -f runtime.yaml +``` + +Below is the example for the default and basic runtime spec after you've installed the Runner: + +{% highlight yaml %} +{% raw %} +version: 1 +metadata: + ... +runtimeScheduler: + cluster: + clusterProvider: + accountId: 5f048d85eb107d52b16c53ea + selector: my-eks-cluster + namespace: codefresh + serviceAccount: codefresh-engine + annotations: {} +dockerDaemonScheduler: + cluster: + clusterProvider: + accountId: 5f048d85eb107d52b16c53ea + selector: my-eks-cluster + namespace: codefresh + serviceAccount: codefresh-engine + annotations: {} + userAccess: true + defaultDindResources: + requests: '' + pvcs: + dind: + storageClassName: dind-local-volumes-runner-codefresh +extends: + - system/default/hybrid/k8s_low_limits +description: '...' +accountId: 5f048d85eb107d52b16c53ea +{% endraw %} +{% endhighlight %} + +### Top level fields + +{: .table .table-bordered .table-hover} +| Field name | Type | Value | +| -------------- |-------------------------| -------------------------| +| `version` | string | Runtime environment version | +| `metadata` | object | Meta-information | +| `runtimeScheduler` | object | Engine pod definition | +| `dockerDaemonScheduler` | object | Dind pod definition | +| `extends` | array | System field (links to full runtime spec from Codefresh API) | +| `description` | string | Runtime environment description (k8s context name and namespace) | +| `accountId` | string | Account to which this runtime belongs | +| `appProxy` | object | Optional filed for [app-proxy]({{site.baseurl}}/docs/administration/codefresh-runner/#optional-installation-of-the-app-proxy) | + +### runtimeScheduler fields (engine) + +{: .table .table-bordered .table-hover} +| Field name | Type | Value | +| -------------- |-------------------------| -------------------------| +| `image` | string | Override default engine image | +| `imagePullPolicy` | string | Override image pull policy (default `IfNotPresent`) | +| `type` | string | `KubernetesPod` | +| `envVars` | object | Override or add environment variables passed into the engine pod | +| `userEnvVars` | object | Add external env var(s) to the pipeline. See [Custom Global Environment Variables]({{site.baseurl}}/docs/administration/codefresh-runner/#custom-global-environment-variables) | +| `cluster` | object | k8s related information (`namespace`, `serviceAccount`, `nodeSelector`) | +| `resources` | object | Specify non-default `requests` and `limits` for engine pod | +| `tolerations` | array | Add tolerations to engine pod | +| `annotations` | object | Add custom annotations to engine pod (empty by default `{}`) | +| `labels` | object | Add custom labels to engine pod (empty by default `{}`) | +| `dnsPolicy` | string | Engine pod's [DNS policy](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy) | +| `dnsConfig` | object | Engine pod's [DNS config](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config) | + +`runtimeScheduler` example: +{% highlight yaml %} +{% raw %} +runtimeScheduler: + imagePullPolicy: Always + cluster: + clusterProvider: + accountId: 5f048d85eb107d52b16c53ea + selector: my-eks-cluster + nodeSelector: #schedule engine pod onto a node whose labels match the nodeSelector + node-type: engine + namespace: codefresh + serviceAccount: codefresh-engine + annotations: {} + labels: + spotinst.io/restrict-scale-down: "true" #optional label to prevent node scaling down when the runner is deployed on spot instances using spot.io + envVars: + NODE_TLS_REJECT_UNAUTHORIZED: '0' #disable certificate validation for TLS connections (e.g. to g.codefresh.io) + METRICS_PROMETHEUS_ENABLED: 'true' #enable /metrics on engine pod + DEBUGGER_TIMEOUT: '30' #debug mode timeout duration (in minutes) + userEnvVars: + - name: GITHUB_TOKEN + valueFrom: + secretKeyRef: + name: github-token + key: token + resources: + requests: + cpu: 60m + memory: 500Mi + limits: + cpu: 1000m + memory: 2048Mi + tolerations: + - effect: NoSchedule + key: codefresh.io + operator: Equal + value: engine +{% endraw %} +{% endhighlight %} + +### dockerDaemonScheduler fields (dind) + +| Field name | Type | Value | +| -------------- |-------------------------| -------------------------| +| `dindImage` | string | Override default dind image | +| `type` | string | `DindPodPvc` | +| `envVars` | object | Override or add environment variables passed into the dind pod. See [IN-DIND cleaner]({{site.baseurl}}/docs/administration/codefresh-runner/#cleaners) | +| `userVolumeMounts` with `userVolumes` | object | Add volume mounts to the pipeline See [Custom Volume Mounts]({{site.baseurl}}/docs/administration/codefresh-runner/#custom-volume-mounts) | +| `cluster` | object | k8s related information (`namespace`, `serviceAccount`, `nodeSelector`) | +| `defaultDindResources` | object | Override `requests` and `limits` for dind pod (defaults are `cpu: 400m` and `memory:800Mi` ) | +| `tolerations` | array | Add tolerations to dind pod | +| `annotations` | object | Add custom annotations to dind pod (empty by default `{}`) | +| `labels` | object | Add custom labels to dind pod (empty by default `{}`) | +| `pvc` | object | Override default storage configuration for PersistentVolumeClaim (PVC) with `storageClassName`, `volumeSize`, `reuseVolumeSelector`. See [Volume Reusage Policy]({{site.baseurl}}/docs/administration/codefresh-runner/#volume-reusage-policy) | +| `dnsPolicy` | string | Dind pod's [DNS policy](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy) | +| `dnsConfig` | object | Dind pod's [DNS config](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config) | + +`dockerDaemonScheduler` example: +{% highlight yaml %} +{% raw %} +dockerDaemonScheduler: + cluster: + clusterProvider: + accountId: 5f048d85eb107d52b16c53ea + selector: my-eks-cluster + nodeSelector: #schedule dind pod onto a node whose labels match the nodeSelector + node-type: dind + namespace: codefresh + serviceAccount: codefresh-engine + annotations: {} + labels: + spotinst.io/restrict-scale-down: "true" #optional label to prevent node scaling down when the runner is deployed on spot instances using spot.io + userAccess: true + defaultDindResources: + requests: '' + limits: + cpu: 1000m + memory: 2048Mi + userVolumeMounts: + my-cert: + name: cert + mountPath: /etc/ssl/cert + readOnly: true + userVolumes: + my-cert: + name: cert + secret: + secretName: tls-secret + pvcs: + dind: + storageClassName: dind-local-volumes-runner-codefresh + volumeSize: 30Gi + reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id' + tolerations: + - key: codefresh.io + operator: Equal + value: dinds + effect: NoSchedule +{% endraw %} +{% endhighlight %} + +### Custom Global Environment Variables +You can add your own environment variables in the runtime environment, so that all pipeline steps will have access to it. A typical example would be a shared secret that you want to pass to the pipeline. + +Under the `runtimeScheduler` block you can add an additional element with named `userEnvVars` that follows the same syntax as [secret/environment variables](https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-environment-variables). + +`runtime.yaml` +{% highlight yaml %} +{% raw %} +... +runtimeScheduler: + userEnvVars: + - name: GITHUB_TOKEN + valueFrom: + secretKeyRef: + name: github-token + key: token +... +{% endraw %} +{% endhighlight %} + +### Custom Volume Mounts +You can add your own volume mounts in the runtime environment, so that all pipeline steps have access to the same set of external files. A typical example of this scenario is when you want to make a set of SSL certificates available to all your pipelines. Rather than manually download the certificates in each pipeline, you can provide them centrally on the runtime level. + +Under the `dockerDaemonScheduler` block you can add two additional elements with names `userVolumeMounts` and `userVolumes` (they follow the same syntax as normal k8s `volumes` and `volumeMounts`) and define your own global volumes. + +`runtime.yaml` +{% highlight yaml %} +{% raw %} +... +dockerDaemonScheduler: + userVolumeMounts: + my-cert: + name: cert + mountPath: /etc/ssl/cert + readOnly: true + userVolumes: + my-cert: + name: cert + secret: + secretName: tls-secret +... +{% endraw %} +{% endhighlight %} + +### Debug Timeout Duration + +The default timeout for [debug mode]({{site.baseurl}}/docs/configure-ci-cd-pipeline/debugging-pipelines/) is 14 minutes, and even if the user is actively working, it is still 14 minutes. To change the duration of the debugger, you will need to update your Runtime Spec for the runtime you would like to change. To change the default duration, you will need to add `DEBUGGER_TIMEOUT` to the environment variable. The value you pass is a string value that will define the timeout in minutes. For example, you can pass '30', which will be 30 minutes. + +Under `.runtimeScheduler`, add an `envVars` section, then add `DEBUGGER_TIMEOUT` under `envVars` with the value you want. + +```yaml +... +runtimeScheduler: + envVars: + DEBUGGER_TIMEOUT: '30' +... +``` + +### Volume Reusage Policy + +The behavior of how the volumes are reused depends on volume selector configuration. +`reuseVolumeSelector` option is configurable in runtime environment spec. + +The following options are available: + +* `reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName'` - determined PV can be used by **ANY** pipeline of your account (it's a **default** volume selector). +* `reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id'` - determined PV can be used only by a **single pipeline**. +* `reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id,io.codefresh.branch_name'` - determined PV can be used only by **single pipeline AND single branch**. +* `reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id,trigger'` - determined PV can be used only by **single pipeline AND single trigger**. + +For approach `codefresh-app,io.codefresh.accountName`: + +* Benefit: less PVs --> lower cost (since any PV can be used by any pipeline, then, the cluster would need to keep less PVs in its pool of PVs for Codefresh) +* Downside: since the PV can be used by any pipeline, then, the PVs could have assets and info from different pipelines, thus reducing the probability of cache, + +For approach `codefresh-app,io.codefresh.accountName,pipeline_id`: + +* Benefit: more probability of cache (no "spam" from other pipelines) +* Downside: more PVs to keep (higher cost) + + +To change volume selector get runtime yaml spec and under `dockerDaemonScheduler.pvcs.dind` block specify `reuseVolumeSelector`: + +```yaml + pvcs: + dind: + volumeSize: 30Gi + reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id' +``` + +## Runtime Cleaners + +### Key points + +* Codefresh pipelines require disk space for: + * [Pipeline Shared Volume](https://codefresh.io/docs/docs/yaml-examples/examples/shared-volumes-between-builds/) (`/codefresh/volume`, implemented as [docker volume](https://docs.docker.com/storage/volumes/)) + * Docker containers - running and stopped + * Docker images and cached layers +* To improve performance, `volume-provisioner` is able to provision previously used disk with docker images and pipeline volume from previously running builds. It improves performance by using docker cache and decreasing I/O rate. +* Least recently docker images and volumes should be cleaned to avoid out-of-space errors. +* There are several places where pipeline volume cleanup is required, so there are several kinds of cleaner. + +### Cleaners + +* [IN-DIND cleaner](https://github.com/codefresh-io/dind/tree/master/cleaner) - deletes extra docker containers, volumes, images in **dind pod** +* [External volumes cleaner](https://github.com/codefresh-io/runtime-cluster-monitor/blob/master/charts/cf-monitoring/templates/dind-volume-cleanup.yaml) - deletes unused **external** PVs (EBS, GCE/Azure disks) +* [Local volumes cleaner](https://github.com/codefresh-io/dind-volume-utils/blob/master/local-volumes/lv-cleaner.sh) - deletes **local** volumes in case node disk space is close to the threshold + +*** + +#### IN-DIND cleaner + +**Purpose:** Removes unneeded *docker containers, images, volumes* inside kubernetes volume mounted to the dind pod + +**Where it runs:** Running inside each dind pod as script + +**Triggered by:** SIGTERM and also during the run when disk usage (cleaner-agent ) > 90% (configurable) + +**Configured by:** Environment Variables which can be set in Runtime Environment configuration + +**Configuration/Logic:** [README.md](https://github.com/codefresh-io/dind/tree/master/cleaner#readme) + +Override `dockerDaemonScheduler.envVars` on Runtime Environment if necessary (the following are **defaults**): + +```yaml +dockerDaemonScheduler: + envVars: + CLEAN_PERIOD_SECONDS: '21600' # launch clean if last clean was more than CLEAN_PERIOD_SECONDS seconds ago + CLEAN_PERIOD_BUILDS: '5' # launch clean if last clean was more CLEAN_PERIOD_BUILDS builds since last build + IMAGE_RETAIN_PERIOD: '14400' # do not delete docker images if they have events since current_timestamp - IMAGE_RETAIN_PERIOD + VOLUMES_RETAIN_PERIOD: '14400' # do not delete docker volumes if they have events since current_timestamp - VOLUMES_RETAIN_PERIOD + DISK_USAGE_THRESHOLD: '0.8' # launch clean based on current disk usage DISK_USAGE_THRESHOLD + INODES_USAGE_THRESHOLD: '0.8' # launch clean based on current inodes usage INODES_USAGE_THRESHOLD +``` + +*** + +#### External volumes cleaner + +**Purpose:** Removes unused *kubernetes volumes and related backend volumes* + +**Where it runs:** On Runtime Cluster as CronJob +(`kubectl get cronjobs -n codefresh -l app=dind-volume-cleanup`). Installed in case the Runner uses non-local volumes (`Storage.Backend != local`) + +**Triggered by:** CronJob every 10min (configurable), part of [runtime-cluster-monitor](https://github.com/codefresh-io/runtime-cluster-monitor/blob/master/charts/cf-monitoring/templates/dind-volume-cleanup.yaml) and runner deployment + +**Configuration:** + +Set `codefresh.io/volume-retention` annotation on Runtime Environment: + +```yaml +dockerDaemonScheduler: + pvcs: + dind: + storageClassName: dind-ebs-volumes-runner-codefresh + reuseVolumeSelector: 'codefresh-app,io.codefresh.accountName,pipeline_id' + volumeSize: 32Gi + annotations: + codefresh.io/volume-retention: 7d +``` + +Override environment variables for `dind-volume-cleanup` cronjob if necessary: + +* `RETENTION_DAYS` (defaults to 4) +* `MOUNT_MIN` (defaults to 3) +* `PROVISIONED_BY` (defaults to `codefresh.io/dind-volume-provisioner`) + +About *optional* `-m` argument: + +* `dind-volume-cleanup` to clean volumes that were last used more than `RETENTION_DAYS` ago +* `dind-volume-cleanup-m` to clean volumes that were used more than a day ago, but mounted less than `MOUNT_MIN` times + +*** + +#### Local volumes cleaner + +**Purpose:** Deletes local volumes in case node disk space is close to the threshold + +**Where it runs:** On each node on runtime cluster as DaemonSet `dind-lv-monitor`. Installed in case the Runner use local volumes (`Storage.Backend == local`) + +**Triggered by:** Starts clean if disk space usage or inodes usage is more than thresholds (configurable) + +**Configuration:** + +Override environment variables for `dind-lv-monitor` daemonset if necessary: + +* `VOLUME_PARENT_DIR` - default `/var/lib/codefresh/dind-volumes` +* `KB_USAGE_THRESHOLD` - default 80 (percentage) +* `INODE_USAGE_THRESHOLD` - default 80 + +## ARM Builds + +With hybrid runner it's possibe to run native ARM64v8 builds. + +>**Note:** Running both amd64 and arm64 images within the same pipeline - it is not possible. We do not support multi-architecture builds. One runtime configuration - one architecture. Considering one pipeline can map only to one runtime, it is possible to run either amd64 or arm64, but not both within a one pipeline + +The following scenario is an example of how to set up ARM Runner on existing EKS cluster: + +**Step 1 - Preparing nodes** + +Create new ARM nodegroup: + +```shell +eksctl utils update-coredns --cluster +eksctl utils update-kube-proxy --cluster --approve +eksctl utils update-aws-node --cluster --approve + +eksctl create nodegroup \ +--cluster \ +--region \ +--name \ +--node-type \ +--nodes <3>\ +--nodes-min <2>\ +--nodes-max <4>\ +--managed +``` + +Check nodes status: + +```shell +kubectl get nodes -l kubernetes.io/arch=arm64 +``` + +Also it's recommeded to label and taint the required ARM nodes: + +```shell +kubectl taint nodes arch=aarch64:NoSchedule +kubectl label nodes arch=arm +``` + +**Step 2 - Runner installation** + +Use [values.yaml](https://github.com/codefresh-io/venona/blob/release-1.0/venonactl/example/values-example.yaml) to inject `tolerations`, `kube-node-selector`, `build-node-selector` into the Runtime Environment spec. + +`values-arm.yaml` + +```yaml +... +Namespace: codefresh + +### NodeSelector --kube-node-selector: controls runner and dind-volume-provisioner pods +NodeSelector: arch=arm + +### Tolerations --tolerations: controls runner, dind-volume-provisioner and dind-lv-monitor +Tolerations: +- key: arch + operator: Equal + value: aarch64 + effect: NoSchedule +... +######################################################## +### Codefresh Runtime ### +### ### +### configure engine and dind pods ### +######################################################## +Runtime: +### NodeSelector --build-node-selector: controls engine and dind pods + NodeSelector: + arch: arm +### Tolerations for engine and dind pods + tolerations: + - key: arch + operator: Equal + value: aarch64 + effect: NoSchedule +... +``` + +Install the Runner with: + +```shell +codefresh runner init --values values-arm.yaml --exec-demo-pipeline false --skip-cluster-integration true +``` + +**Step 3 - Post-installation fixes** + +Change `engine` image version in Runtime Environment specification: + +```shell +# get the latest engine ARM64 tag +curl -X GET "https://quay.io/api/v1/repository/codefresh/engine/tag/?limit=100" --silent | jq -r '.tags[].name' | grep "^1.*arm64$" +1.136.1-arm64 +``` + +```shell +# get runtime spec +codefresh get re $RUNTIME_NAME -o yaml > runtime.yaml +``` + +under `runtimeScheduler.image` change image tag: + +```yaml +runtimeScheduler: + image: 'quay.io/codefresh/engine:1.136.1-arm64' +``` + +```shell +# patch runtime spec +codefresh patch re -f runtime.yaml +``` + +For `local` storage patch `dind-lv-monitor-runner` DaemonSet and add `nodeSelector`: + +```shell +kubectl edit ds dind-lv-monitor-runner +``` + +```yaml + spec: + nodeSelector: + arch: arm +``` + +**Step 4 - Run Demo pipeline** + +Run a modified version of the *CF_Runner_Demo* pipeline: + +```yaml +version: '1.0' +stages: + - test +steps: + test: + stage: test + title: test + image: 'arm64v8/alpine' + commands: + - echo hello Codefresh Runner! +``` + +## Troubleshooting + +For troubleshooting refer to the [Knowledge Base](https://support.codefresh.io/hc/en-us/sections/4416999487762-Hybrid-Runner) + +## What to read next + +* [Codefresh installation options]({{site.baseurl}}/docs/installation/installation-options/) +* [Codefresh On-Premises]({{site.baseurl}}/docs/administration/codefresh-on-prem/) +* [Codefresh API]({{site.baseurl}}/docs/integrations/codefresh-api/) diff --git a/_docs/runtime/git-sources.md b/_docs/installation/gitops/git-sources.md similarity index 64% rename from _docs/runtime/git-sources.md rename to _docs/installation/gitops/git-sources.md index 2b95dc542..932a77078 100644 --- a/_docs/runtime/git-sources.md +++ b/_docs/installation/gitops/git-sources.md @@ -1,23 +1,24 @@ --- -title: "Add Git Sources to runtimes" -description: "" -group: runtime +title: "Add Git Sources to GitOps Runtimes" +description: "Manage Git Sources storing resources" +group: installation +sub_group: gitops toc: true --- -A Git Source is the equivalent of an Argo CD application that tracks a Git repository and syncs the desired state of the repo to the destination K8s cluster. In addition to application resources, the Git Source can store resources for Codefresh runtimes, and CI/CD entities such as delivery pipelines, Workflow Templates, workflows, and applications. +A Git Source is the equivalent of an Argo CD application that tracks a Git repository and syncs the desired state of the repo to the destination K8s cluster. In addition to application resources, the Git Source can store resources for GitOps Runtimes, and CI/CD entities such as delivery pipelines, Workflow Templates, workflows, and applications. -Provisioning a runtime automatically creates a Git Source that stores resources for the runtime and for the demo CI pipelines that are optionally installed with the runtime. Every Git Source is associated with a Codefresh runtime. A runtime can have one or more Git Sources. You can add Git Sources at any time, to the same or to different runtimes. +Provisioning a GitOps Runtime automatically creates a Git Source that stores resources for the Runtime and for the demo CI pipelines that are optionally installed with the Runtime. Every Git Source is associated with a GitOps Runtime. You can add more Git Sources at any time, to the same or to different Runtimes. -Once you create a Git Source for a runtime, you can store resources for CI/CD entities associated with that runtime. For example, when creating pipelines or applications, you can select the Git Source to which to store manifest definitions. +Once you create a Git Source for a GitOps Runtime, you can store resources for CI/CD entities associated with it. For example, when creating pipelines or applications, you can select the Git Source to which to store manifest definitions. -### View Git Sources and definitions +## View Git Sources and definitions Drill down on a runtime in List View to see its Git Sources. -1. In the Codefresh UI, go to the [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"} page. -1. From the **List View** (the default), select a runtime name, and then select the **Git Sources** tab. +1. In the Codefresh UI, go to the [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes){:target="\_blank"} page. +1. From the **List View** (the default), select a Runtime name, and then select the **Git Sources** tab. {% include image.html @@ -26,20 +27,20 @@ Drill down on a runtime in List View to see its Git Sources. url="/images/runtime/git-source-list.png" alt="Git Sources in runtime" caption="Git Sources in runtime" - max-width="30%" + max-width="60%" %} {:start="3"} 1. To go to the repo tracked by the Git Source, in the Repo column, mouse over the repo name and select **Go to Git repo**. 1. To see the definitions for the Git Source, select the three dots at the end of the row. -### Create a Git Source -Create Git Sources for any provisioned runtime. The Git Sources are available to store resources for pipelines or applications when you create them. +## Create a Git Source +Create Git Sources for any provisioned Runtime. The Git Sources are available to store resources for pipelines or applications when you create them. >Make sure you are in the List View to create Git Sources. -1. In the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes**){:target="\_blank"}. -1. In the List View, select the runtime for which to add a Git Source, and then select the **Git Sources** tab. +1. In the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes**){:target="\_blank"}. +1. In the List View, select the Runtime for which to add a Git Source, and then select the **Git Sources** tab. 1. Select **Create Git Sources**, and in the Create Git Source panel, define the definitions for the Git Source: {% include @@ -49,14 +50,14 @@ Create Git Sources for any provisioned runtime. The Git Sources are available t url="/images/runtime/create-git-source.png" alt="Create Git Source" caption="Create Git Source" - max-width="30%" + max-width="60%" %} * **Git Source Name**: The name of the Git Source, which must be unique within the cluster. * **Source**: The Git repo with the desired state, tracked by the Git Source, and synced to the destination cluster. * **Repository**: Mandatory. The URL to the Git repo. * **Branch**: Optional. The specific branch within the repo to track. - * **Path**: Optional. The specific path within the repo, and branch, if one is specified, to track. + * **Path**: Optional. The specific path within the repo, and branch if one is specified, to track. * **Destination**: The destination cluster with the actual state to which to apply the changes from the **Source**. * **Namespace**: The namespace in the destination cluster to which to sync the changes. @@ -69,12 +70,12 @@ Create Git Sources for any provisioned runtime. The Git Sources are available t {:start="4"} 1. Select **+ Create Git Source**. -### Edit Git Source definitions +## Edit Git Source definitions Edit an existing Git Source by changing the source and destination definitions. > You cannot change the name of the Git Source. -1. In the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes**){:target="\_blank"}. -1. From the **List View** (the default), select the runtime with the Git Source, and then select the **Git Sources** tab. +1. In the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes**){:target="\_blank"}. +1. From the **List View** (the default), select the Runtime with the Git Source, and then select the **Git Sources** tab. 1. In the row with the Git Source to edit, select the three dots, and then select **Edit** in the panel that appears. {% include @@ -89,13 +90,13 @@ Edit an existing Git Source by changing the source and destination definitions. {:start="4"} 1. Change the **Source** and **Destination** definitions for the Git Source, and select **Save**. -### View/download logs for a Git Source -View online logs for any Git Source associated with a runtime, and if needed, download the log file for offline viewing and analysis. -Online logs show up to 1000 of the most recent events (lines), updated in real time. Downloaded logs include all the events from the application launch to the date and time of download. +## View/download logs for a Git Source +View online logs for any Git Source associated with a Runtime, and if needed, download the log file for offline viewing and analysis. +Online logs show up to 1000 of the most recent events (lines), updated in real time. Downloaded logs include all the events, from the application launch to the date and time of download. -1. In the Codefresh UI, go to [Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes**){:target="\_blank"}. -1. From the **List View** (the default), select the runtime with the Git Source, and then select the **Git Sources** tab. -1. In the row with the Git Source foe which to view/download logs, select the three dots, and then select **View Logs**. +1. In the Codefresh UI, go to [GitOps Runtimes](https://g.codefresh.io/2.0/account-settings/runtimes**){:target="\_blank"}. +1. From the **List View** (the default), select the Runtime with the Git Source, and then select the **Git Sources** tab. +1. In the row with the Git Source for which to view/download logs, select the three dots, and then select **View Logs**. {% include image.html @@ -104,7 +105,7 @@ Online logs show up to 1000 of the most recent events (lines), updated in real t url="/images/runtime/git-source-view-logs.png" alt="Edit Git Source" caption="Edit Git Source" - max-width="30%" + max-width="60%" %} {:start="4"} @@ -126,7 +127,8 @@ Online logs show up to 1000 of the most recent events (lines), updated in real t 1. To download the log, click **Download**. The file is downloaded with `.log` extension. -### What to read next -[Manage runtimes]({{site.baseurl}}/docs/runtime/monitor-manage-runtimes/) -[Recover runtimes]({{site.baseurl}}/docs/runtime/runtime-recovery/) +## Related articles +[Monitoring & managing GitOps Runtimes]({{site.baseurl}}/docs/installation/gitops/monitor-manage-runtimes/) +[Shared configuration repo]({{site.baseurl}}/docs/reference/shared-configuration) + diff --git a/_docs/runtime/hosted-runtime.md b/_docs/installation/gitops/hosted-runtime.md similarity index 61% rename from _docs/runtime/hosted-runtime.md rename to _docs/installation/gitops/hosted-runtime.md index 0a08ba3bc..11bd496ea 100644 --- a/_docs/runtime/hosted-runtime.md +++ b/_docs/installation/gitops/hosted-runtime.md @@ -1,18 +1,29 @@ --- -title: "Set up a hosted runtime environment" -description: "" -group: runtime +title: "Hosted GitOps Runtime setup" +description: "Provision Hosted GitOps environment" +group: installation +sub_group: gitops toc: true --- -If you have Codefresh's Hosted GitOps, set up your hosted environment, and you are all ready to leverage extensive CD Ops capabilities. -Read about [Hosted GitOps]({{site.baseurl}}/docs/incubation/intro-hosted-runtime/). +Set up your environment with the Hosted GitOps Runtime to leverage extensive CD capabilities. + -### Where to start with Hosted GitOps -If you have not provisioned a hosted runtime, Codefresh presents you with the setup instructions in the **Home** dashboard. +## System requirements for Hosted GitOps Runtimes +{: .table .table-bordered .table-hover} +| Item | Requirement | +| -------------- | -------------- | +|Kubernetes cluster | Server version 1.18 and higher to which to deploy applications| +|Git provider | {::nomarkdown}
Tip: To check the server version, run:
kubectl version --short.{:/}| +| Ingress controller| Configured on Kubernetes cluster and exposed from the cluster. {::nomarkdown}
Supported and tested ingress controllers include: