Hari Sekhon - Diagrams-as-Code

Diagrams-as-Code using the awesome D2 language, MermaidJS, Python diagrams and Graphviz.

Diagrams shown below are automatically (re)generated by GitHub Actions CI/CD 😎

I read an article that said:

the ability to create meaningful diagrams is the pinnacle of communication skills as an engineer

Documentation

Gist from Knowledge-Base repo full of links to Diagrams technologies and Icon Sets.

Diagrams

They say a picture is worth a thousand words...

This Repo's Creation & GitHub Actions CI/CD to auto-(re)generate diagrams from code changes

github_actions_cicd.py:

Open README.md to enlarge:

github_actions_cicd.d2:

GitHub Flow with Jira ticket integration

Prefix Git branches with Jira ticket numbers in Jira's AA-NNN format for GitHub Pull Requests to automatically appear in Jira tickets (see this doc):

%% https://mermaid.js.org/syntax/gitgraph.html#gitgraph-specific-configuration-options
%% https://htmlcolorcodes.com/
%%{ init: {
        "logLevel": "debug",
        "theme": "dark",
        "themeVariables": {
            "git0": "#839192",
            "git1": "#2874A6",
            "gitInv0": "#FFFFFF",
            "gitBranchLabel0": "#FFFFFF",
            "commitLabelColor": "#FFFFFF"
        }
    }
}%%
gitGraph
    commit
    commit id: "branch"
    branch AA-NNN-my-feature-branch
    checkout AA-NNN-my-feature-branch
    commit id: "add code"
    commit id: "refine code"
    checkout main
    merge AA-NNN-my-feature-branch id: "merge PR" type: HIGHLIGHT tag: "2023.15 release"
    commit
    commit

Git - why you shouldn't use long-lived branches

* Environment Branches may be one of the few exceptions but requires workflow discipline.

See Also: 100+ scripts for Git and the major Git repo providers like GitHub, GitLab, Bitbucket, Azure DevOps in my DevOps-Bash-tools repo.

%% https://mermaid.js.org/syntax/gitgraph.html#gitgraph-specific-configuration-options
%% https://htmlcolorcodes.com/
%%{ init: {
        "logLevel": "debug",
        "theme": "dark",
        "gitGraph": {
            "mainBranchName": "master"
        },
        "themeVariables": {
            "git0": "#839192",
            "git1": "#C0392B ",
            "git2": "#2E86C1",
            "gitInv0": "#FFFFFF",
            "gitBranchLabel0": "#FFFFFF",
            "commitLabelColor": "#FFFFFF"
        }
    }
}%%
gitGraph
    commit  id: "commit 1"
    commit id: "branch"
    branch long-lived-branch
    checkout long-lived-branch
    commit id: "50 clever commits"
    checkout master
    commit id: "commit 2"
    checkout long-lived-branch
    commit id: "too clever"
    checkout master
    commit id: "commit 3"
    checkout long-lived-branch
    commit id: "too long"
    checkout master
    commit id: "commit 4"
    checkout long-lived-branch
    commit id: "try to merge back"
    checkout master
    merge long-lived-branch id: "Merge Conflict!!" type: REVERSE
    checkout long-lived-branch
    commit id: "trying to fix"
    commit id: "still trying to fix"
    commit id: "struggling to fix"
    commit id: "ask Hari for help"
    branch fixes-branch-to-send-to-naughty-colleague
    checkout fixes-branch-to-send-to-naughty-colleague
    commit id: "fix 1"
    commit id: "fix 2"
    commit id: "fix 3"
    commit id: "could have been working on better things!"
    checkout long-lived-branch
    merge fixes-branch-to-send-to-naughty-colleague id: "merge fixes" type: HIGHLIGHT
    commit id: "more commits"
    commit id: "because this branch only had 105 commits already"
    checkout master
    merge long-lived-branch id: "Finallly Merged!" type: HIGHLIGHT
    commit id: "Please never do that again"

AWS Web Traffic Classic

aws_web_traffic_classic.py:

Azure Active Directory Single Sign-On

I've administered Azure Active Directory at a couple of companies and integrated a variety of applications including GitHub Enterprise Cloud, AWS IAM Identity Center (formerly AWS SSO), Jenkins, ArgoCD, Keycloak, Hubspot etc using the typical OIDC or SAML integration mechanisms.

azure_ad_aws_github_keycloak.d2:

Jenkins CI/CD on Kubernetes

A production Jenkins on Kubernetes I built for a client with auto-spawning agents for horizontal scaling and integration with Docker, SonarQube, Clair, Grype and Trivy for code & container scanning.

GitHub repo: HariSekhon/Kubernetes-configs
- Jenkins
- Clair
- SonarQube
- Trivy server
GitHub repo: HariSekhon/Jenkins
- Advanced Jenkinsfile
- Groovy Shared Library with the code & container scanning functions
  - clair.groovy
  - grype.groovy
  - trivy.groovy, trivyFS.groovy, trivyImages.groovy
  - gcrDockerAuth.groovy, garDockerAuth.groovy
  - and others in vars/, and don't forget about the epic Jenkinsfile

jenkins_kubernetes_cicd.d2:

screenshot:

GCP Cloudflare Web Architecture GKE

A production internet customer facing website and apps replatform to Google Kubernetes Engine I did for an internet startup client using:

GitHub repo: HariSekhon/Terraform
GitHub repo: HariSekhon/Kubernetes-configs
- External DNS automatic DNS record creation in Cloudflare for any Kubernetes ingresses
- External Secrets pulling into Kubernetes from GCP Secret Manager

There are Cloudflare API scripts in the HariSekhon/DevOps-Bash-tools repo.

gcp_cloudflare_web_architecture_gke.py:

GCP Malware Scanner with ClamAV

A variation using Kubernetes and Cloud Functions of this GCP malware scanner solution architecture:

https://cloud.google.com/architecture/automate-malware-scanning-for-documents-uploaded-to-cloud-storage

gcp_malware_scanner.d2:

Traditionally:

kubernetes_on_premise.d2:

with MetalLB:

GitHub repo: HariSekhon/Kubernetes-configs
- MetalLB

Is it just me or do MetaLB think they're Starfleet? (compare their logos)

kubernetes_on_premise_metallb.d2:

Traefik Kubernetes Ingress on GKE

A Traefik deployment I did for a client.

GitHub repo: HariSekhon/Kubernetes-configs
- Traefik
- Traefik Hub Agent

kubernetes_traefik_ingress_gke.py:

kubernetes_traefik_ingress_gke.d2:

Kong API Gateway on Kubernetes (AWS EKS)

A Kong API Gateway deployment I did for a client.

GitHub repo: HariSekhon/Kubernetes-configs
- Kong
- Cert Manager
- ArgoCD

kubernetes_kong_api_gateway_eks.py:

OpenTSDB on Kubernetes and HBase

A high scale production OpenTSDB replatform I did to Kubernetes for a client, ingesting 9 billion data points per day and serving 3 million queries per day.

I also had to do advanced performance tuning of their production HBase cluster which was suffering from frequent outages at this scale due to being set up by a non-SME on the wrong hardware (I had to make do with the existing hardware of course).

This was the second client I did in-depth performance tuning of HBase for - I've published a selection of useful HBase tools - see hbase_*.py and opentsdb_*.py in HariSekhon/DevOps-Python-tools.

opentsdb_kubernetes_hbase.d2:

Devs Test in Production

Iirc I created and stuck this meme pic of The Most Interesting Man in the World on the wall of my tech dept back in 2011 while leading the infra team of an internet Ad Tech company doing several production releases a day. We literally did test in production using a small fraction of live internet traffic via canary deployments.

test_in_production.d2:

code_commit_push.d2:

Git - Environment Branches

At least they don't only test in Production!

Another internet facing client refused to use tagging because they didn't want to have to think up version or release numbers for their website releases.

Not everybody likes environment branches, but they worked in production for over 2 years and they are easy to use.

Also, contrary to some naysayers it's quite easy to diff environment branches as everything should be in Git, so you can get a very quick and easy difference between your environments in a single git diff command. It's also easy to automate backporting hotfixes to lower environments:

GitHub repo: HariSekhon/Jenkins
- gitMerge.groovy
- gitMergePipeline.groovy

%%{ init: {
        "logLevel": "debug",
        "theme": "dark",
        "gitGraph": {
            "mainBranchName": "dev"
        },
        "themeVariables": {
            "git0": "red",
            "git1": "blue ",
            "git2": "green",
            "gitInv0": "#FFFFFF",
            "gitBranchLabel0": "#FFFFFF",
            "commitLabelColor": "#FFFFFF"
        }
    }
}%%

gitGraph
    branch staging
    branch production

    checkout dev
    commit id: "commit 1"

    checkout staging
    commit id: "QA fix 1 "

    checkout production
    commit id: "hotfix commit"

    checkout dev
    commit id: "commit 2"

    checkout staging
    merge dev id: "fast-forward merge" tag: "CI/CD + QA Tests"

    checkout production
    merge staging id: "fast-forward merge " tag: "v2023.1 Release (CI/CD)"


    checkout dev
    commit id: "commit 3"

    checkout staging
    commit id: "QA fix 2 "

    #checkout production
    #commit id: "commit 3  "

    checkout dev
    commit id: "commit 4"

    checkout staging
    merge dev id: "fast-forward merge 2" tag: "CI/CD + QA Tests"

    checkout production
    merge staging id: "fast-forward merge 2 " tag: "v2023.2 Release (CI/CD)"


    checkout dev
    commit id: "commit 5"

    checkout staging
    commit id: "QA fix 3 "

    #checkout production
    #commit id: "commit 5  "

    checkout dev
    commit id: "commit 6"

    checkout staging
    merge dev id: "fast-forward merge 3" tag: "CI/CD + QA Tests"

    checkout production
    merge staging id: "fast-forward merge 3 " tag: "v2023.3 Release (CI/CD)"

Note: I did eventually move this client to tagged releases using YYYY.NN release format, just incrementing NN which is a no brainer (githubNextRelease.groovy). It turns out the developers had eventually started using releases in Jira labelled as YYYY.NN to track which tickets were going into which production deployment, so when I pushed for this, it made sense to them finally as not being too great an inconvenience! It's also easy to automate by creating GitHub Releases in Jenkins (githubCreateRelease.groovy).

LucidChart - GCP Architecture

A sample architecture I did for a client for us to talk through, which was similar to what they had in mind (I won the gig).

This is the only diagram not as code (here for historical interest). I would embed the interactive live diagram but GitHub markdown doesn't allow HTML iframes so this is the png.

Web Basics

When you're trying to explain to your kids how the internet works...

web_basics.d2:

Network - Layer 3 - Local - ARP

network_layer3_local.d2:

Network - Layer 3 - Remote - IP

network_layer3_remote.d2:

Samples Revamped

These are reworked from Python diagrams and Cloudgram examples.

AWS Load Balanced Web Farm

aws_load_balanced_web_farm.py:

AWS Clustered Web Services

aws_clustered_web_services.py:

Advanced Web Services Open Source

advanced_web_services_open_source.py:

GCP Pub/Sub Analytics

gcp_pubsub_analytics.py:

AWS Event Processing

aws_event_processing.py:

AWS Serverless Image Processing

aws_serverless_image_processing.py:

Build from Source

Install D2, Graphviz, Python3 and 'diagrams' pip module:

git clone https://github.com/HariSekhon/Diagrams-as-Code diagrams

cd diagrams

make install

Create all the .png and .svg diagrams in the images/ dir:

make

Generate only the D2 svg diagrams:

make d2

Generate only the Python png diagrams:

make py

Create any single D2 diagram by running the d2 script file:

./jenkins_kubernetes_docker.d2

Create any single Python diagram and have it open automatically by running the python script file:

./gcp_cloudflare_web_architecture_gke.py

Templates

The templates/diagram.d2 and templates/diagram.py show the basics of each language.

They are a good starting point for creating your own diagrams, and come pre-loaded with many useful icons, links to docs and links to icon sets.

Name		Name	Last commit message	Last commit date
Latest commit History 1,007 Commits
.github/workflows		.github/workflows
bash-tools @ 7c5c954		bash-tools @ 7c5c954
custom		custom
images		images
screenshots		screenshots
setup		setup
templates @ 6f9e993		templates @ 6f9e993
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
.gitmodules		.gitmodules
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
advanced_web_services_open_source.py		advanced_web_services_open_source.py
aws_clustered_web_services.py		aws_clustered_web_services.py
aws_event_processing.py		aws_event_processing.py
aws_load_balanced_web_farm.py		aws_load_balanced_web_farm.py
aws_serverless_image_processing.py		aws_serverless_image_processing.py
aws_web_service_db_cluster.py		aws_web_service_db_cluster.py
aws_web_traffic_classic.py		aws_web_traffic_classic.py
azure-pipelines.yml		azure-pipelines.yml
azure_ad_aws_github_keycloak.d2		azure_ad_aws_github_keycloak.d2
bitbucket-pipelines.yml		bitbucket-pipelines.yml
code_commit_push.d2		code_commit_push.d2
gcp_cloudflare_web_architecture_gke.py		gcp_cloudflare_web_architecture_gke.py
gcp_malware_scanner.d2		gcp_malware_scanner.d2
gcp_pubsub_analytics.py		gcp_pubsub_analytics.py
github_actions_cicd.d2		github_actions_cicd.d2
github_actions_cicd.py		github_actions_cicd.py
jenkins_kubernetes_cicd.d2		jenkins_kubernetes_cicd.d2
kics.config		kics.config
kubernetes_deployment_hpa_ingress.py		kubernetes_deployment_hpa_ingress.py
kubernetes_external_traffic_policy.d2		kubernetes_external_traffic_policy.d2
kubernetes_kong_api_gateway_eks.py		kubernetes_kong_api_gateway_eks.py
kubernetes_on_premise.d2		kubernetes_on_premise.d2
kubernetes_on_premise_metallb.d2		kubernetes_on_premise_metallb.d2
kubernetes_stateful_architecture.py		kubernetes_stateful_architecture.py
kubernetes_traefik_ingress_gke.d2		kubernetes_traefik_ingress_gke.d2
kubernetes_traefik_ingress_gke.py		kubernetes_traefik_ingress_gke.py
network_layer3_local.d2		network_layer3_local.d2
network_layer3_remote.d2		network_layer3_remote.d2
old_web_architecture.py		old_web_architecture.py
opentsdb_kubernetes_hbase.d2		opentsdb_kubernetes_hbase.d2
opentsdb_kubernetes_hbase.py		opentsdb_kubernetes_hbase.py
rabbitmq_broker_with_custom_icon.py		rabbitmq_broker_with_custom_icon.py
requirements.txt		requirements.txt
sonar-project.properties		sonar-project.properties
spark_hadoop.py		spark_hadoop.py
template.d2		template.d2
template.py		template.py
test_in_production.d2		test_in_production.d2
test_in_production.py		test_in_production.py
web_basics.d2		web_basics.d2

License

HariSekhon/Diagrams-as-Code

Folders and files

Latest commit

History

Repository files navigation