Argo-Diff

Application written in go that provides Github status checks and Pull Request comments for changes to Kubernetes manifests when those manifests are delivered via ArgoCD:

Overview

Argo-diff is designed to receive webhook notifications from Gihtub for push and pull_request events. When events are received, it queries the ArgoCD API to pull manifests for the Argo application(s) configured for the repository that is the source of the push or pull request. It will fetch live manifests deployed to your k8s cluster, predicted manifests generated at the revision of the change, and generate a diff.

If ArgoCD cannot generate manifests for the revision of the change, argo-diff will set its status check to a failure for that associated commit.

If the event is for a pull request, argo-diff will comment on the associated pull request with markdown displaying the diff of the manifests.

Argo-diff will not run (but may still produce status checks) in the following situations:

• Base branch of the pull request (branch in which the PR will be merged) is not the target revision for the Argo application. (eg: Your Argo application targets production, but your PR is to be merged into dev.)
• Any git push that is not a branch. (eg: When tags are pushed, they are completely ignored by argo-diff.)
• When your application has an Automated Sync Policy configured, and commits are pushed to that application's target branch. (ie: There's no need to run argo-diff, when ArgoCD will automatically pick up the change.)

Why argo-diff?

There are multiple ways to produce diff previews of proposed manifest changes to ArgoCD applications in Github Pull Requests, but they often require per-repository configuration. Because argo-diff is triggered by Github event notifications, and because argo-diff auto-discovers ArgoCD applications configured in the source code repository, no configuration is needed when repositories and/or ArgoCD applications are added and removed.

Screenshots

Below are screenshots of Github Pull Request comments generated by argo-diff.

Simple comment of healthy application

Long ling in diff truncated

Application out of sync

Deploying

• Access to the Github API will need to be granted either via a Personal Access Token (good for small environments or a proof of concept) or a Github App.
  • If you using a Personal Access Token, it is recommended to create a separate Github account in your organization specifically for argo-diff. Argo diff will look for the PAT in the env var GITHUB_PERSONAL_ACCESS_TOKEN.
    • Permissions for the PAT should match the permissions specified below for Github Apps.
  • If you using a Github App, you'll need to create a new application in your organization under Developer Settings:
    • Name can be argo-diff.
    • Homepage URL can be the base URL of your argocd instance or the base URL of what will be your webhook URL.
    • Callback URL should be empty, as should Setup URL.
    • Webhooks should not be active.
    • The webhook URL can be the URL you will configure Github webhooks below.
    • Permissions:
      • Administration: Read-only
      • Commit statuses: Read and write
      • Metadata: Read-only
      • Pull requests: Read and write
    • Where can this GitHub App be installed? -> Only on this account
    • Upon creating the Github App, take note of the App ID and then generate a new Private Key, which will download a .pem file locally.
    • Install App to your organization. Settings for the installion will have the URL formatted like: /organizations/<ORG_NAME>/settings/installations/<INSTALLATION_ID>. Take note of the installation id on the URL.
    • Argo Diff will look for the API client configuration in the following environment variables: GITHUB_APP_ID, GITHUB_APP_INSTALLATION_ID, and GITHUB_APP_PRIVATE_KEY (the latter of which should be the path to the .pem file generator above)
• Create a user in your ArgoCD instance. This user should have read-only access to all applications:
  • For example, in policy.csv: g, argo-diff, role:ci and p, role:ci, applications, get, *, allow
  • This user shouldn't need a password but does need an API token to be generated.
• Generate a webhook secret that will be shared both by the argo-diff deployment and Github webhook config.
• Deploy to Kubernetes:
  • The recommended approach is to use the Helm chart included in charts/argo-diff/:

    $ helm install my-release oci://ghcr.io/vince-riv/chart/argo-diff

  • You can also use the example manifests in the docs/k8s/ directory to deploy argo-diff to the argocd namespace of your Kubernetes cluster. An Ingress or IngressRoute will need to be added to allow webhooks in from Github to the /webhook endpoint on the argo-diff Service.
• Configure organizational (or perhaps just repository level?) webhook notifications to argo-diff. The Payload URL should map the ingress configured in your cluster, and the secret should be the webhook secret previously generated. Invididual event types to configure:
  • Issue comments (allows for a comment of argo diff to re-trigger argo-diff on the pull request)
  • Pull requests
  • Pushes (optional - enabling will trigger argo-diff to add commit statuses when branches are pushed)
• After the webhook is activated, the ping event should be received and verified by argo-diff and this will validate connectivity from Github to argo-diff

Limitations

This is still in a proof-of-concept and alpha/beta version state, so there are some known limitations.

• Changes to Secrets are not displayed. A future enhancement could flag that secrets are changing without displaying the contents. [#52]
• When many Argo applications are served by a single repository, performance may be slow. Manifests for each Argo application are fetched sequentially, so this could result in argo-diff statuses and/or comments taking minutes to complete.

Running locally

Set environment variables used by argo-diff and then execute go run cmd/main.go.

For example, you can place these in a file called .env.sh:

GITHUB_PERSONAL_ACCESS_TOKEN='github_pat_XXXX'
ARGOCD_AUTH_TOKEN='ABCDEFGHIJKLMNOPQRSTUVWXYZ'
ARGOCD_SERVER_ADDR='argocd.your.domain:443'
ARGOCD_UI_BASE_URL='https://argocd.your.domain'
APP_ENV='dev'

Then source it and execute go run:

$ set -o allexport ; . .env.sh ; set +o allexport
$ go run cmd/main.go

To send requests, you can copy webhook request headers and payloads to temp/curl-headers.txt and temp/curl-payload.json and use the post-local.sh script to ship them to the local server.

There is also the /dev endpoint that gets enabled when APP_ENV=dev - this endpoint is handled by devHandler() in main.go. This bypasses webhook processing is triggers argo-diff for specific event. When in APP_ENV is dev, argo-diff will comment but not set status checks.

The json request data should be POSTed to /dev should look like this:

{
    "ignore": false,
    "owner": "GITHUB_ORG_NAME",
    "repo": "REPOSITORY_NAME",
    "default_ref": "main",
    "commit_sha": "LONG_SHA_OF_COMMIT",
    "pr": 123,
    "change_ref": "BRANCH_NAME_OF_PR",
    "base_ref": "main"
}

Description of those fields:

• ignore: tells argo-diff to ignore the event
• owner: Github organization name
• repo: Github repository name
• default_ref: the default branch of the repository
• commit_sha: the sha that triggered the event - should be HEAD of the branch of the PR if you want it to comment
• pr: pull request number (duh)
• change_ref: the source branch of the PR (the feature branch)
• base_ref: the branch to which the PR is getting merged

Development Notes

This was originally developed by @vrivellino as a way to learn Go. Its functionality replicates that of an internal tool written by smart people at a previous job.