Hari Sekhon - Knowledge Base from 20 years in DevOps, Linux, Cloud, Big Data, Security, AWS, GCP etc

Work in progress to gradually migrate my huge private personal knowledge base into this public repo.

Star and watch for updates.

This repo focuses on advanced knowledge, tips & tricks, performance tuning, troubleshooting, debugging, workarounds & fixes.

Much more advanced knowledge is captured in code and code comments. References to my major GitHub Repos are provided throughout where relevant.

References to official docs and some classic books are provided for those who want to thoroughly cover the basics.

Index

Most of these pages are far from fully ported and populated yet from my private knowledge base pages, some are stubs pending migration.

Hit the watch button on the top right for updates.

GitHub Repos

Full of open source tools and scripts used throughout the rest of these docs to make it easier to install and operate these technologies:

• DevOps-Bash-tools - 1000+ Bash scripts, advanced shell environment customizations, quick installation scripts and dot configs
• DevOps-Python-tools - 80+ Python programs for DevOps & Data
• DevOps-Perl-tools - 25+ Perl programs for DevOps
• DevOps-Golang-tools - a handful of Golang programs
• Kubernetes-configs - production ready checklist and deployment configs for many Kubernetes applications
• SQL-scripts - 100+ database scripts
• Templates - templates for many common file formats for programming languages, build systems and DevOps configs

Linux / Unix

• Linux
  • Debian
  • Ubuntu
  • Redhat
  • Alpine
• Mac
  • Homebrew
• Shell
  • Bash
  • .envrc / direnv

Core Services - DNS, DHCP, SSH

• SSH
  • SSH Tunnelling
• DNS
• DHCP

Editors

• Editors
• Vim
• IntelliJ

Architecture

• Diagrams-as-Code
  • D2 lang

Git

• Git
• GitHub TODO
• GitLab
• Azure DevOps TODO
• Bitbucket TODO

Cloud

• AWS
• GCP
• Azure
• Digital Ocean
• PaaS
• MinIO

Docker

• Docker
• Dockerfile TODO
• Docker Compose
• Docker Swarm TODO
• Podman
• HariSekhon/Dockerfiles repo

Kubernetes

• Kubernetes
• AWS EKS TODO
• GKE TODO
• AKS TODO

On-premise Kubernetes

• K3s TODO
• K3d TODO
• Rancher
• RKE2
• Portworx

Kubernetes Applications & Lifecycle Management

• Helm TODO
• Kustomize
• ArgoCD
• Cert Manager
• HariSekhon/Kubernetes-configs

Container Security

• Container Scanning
• Trivy TODO
• Grype TODO
• Clair

Programming

• Programming
• Perl
• Python
• Anaconda TODO
• Golang TODO
• R TODO
• JVM:
  • Java TODO
  • JVM Performance Tuning
  • Groovy
  • Scala TODO
• Regular Expressions

Configuration Management

• Terraform TODO
• Ansible
• Puppet TODO

Build Systems

• Build Systems
• Make
• Maven TODO
• Gradle TODO
• SBT

Artifact Registries

• Artifact Registries
• Artifactory
• Nexus TODO

CI/CD

• CI/CD
• Self-Hosted CI/CD:
  • Jenkins
    • Jenkins-on-Kubernetes
    • HariSekhon/Jenkins - master Jenkinsfile template and Groovy Shared Library functions
  • TeamCity TODO
    • HariSekhon/TeamCity-CI
  • Concourse
  • GoCD TODO
• Git Repo Providers hosted CI/CD:
  • GitHub Actions TODO
  • GitLab CI TODO
  • Azure DevOps TODO
  • Bitbucket Pipelines TODO
• 3rd Party hosted CI/CD:
  • CircleCI
  • TravisCI
  • Drone.io

Code Scanning

• Code Scanning
• Code Quality
• SonarQube TODO
• CodeClimate
• Codefresh

Internet Services

• Web
• HTTP Proxying
• Cloudflare

Databases / RDBMS

• SQL TODO
• SQL scripts
• Data Warehousing TODO
• MySQL TODO
• PostgreSQL TODO

MPP Databases

Massively Parallel Processing databases - horizontally scaling databases:

• Greenplum
• Impala TODO
• Presto TODO
• Apache Drill TODO

Hosted MPP Databases

• Distributed SQL
• Snowflake
• Google BigQuery TODO
• AWS Redshift TODO

See also the Hadoop Ecosystem section further down.

Database Management

• LiquiBase TODO
• AtlasGo

NoSQL

• Cassandra TODO
• Couchbase TODO
• HBase TODO
• MongoDB TODO

Caching

• Redis TODO
• Memcached TODO
• Oracle Coherence

Search

• Search
• Elasticsearch
• Solr
• Logstash TODO

Pub/Sub & Message Queues

• Kafka
• Celery
• for Kinesis see AWS

Data

• Data
• Big Data
• Data Warehousing TODO

Hadoop Ecosystem

• Spark TODO
• Kafka
• Hadoop TODO
  • Kerberos Hadoop TODO
  • YARN TODO
  • HDFS TODO
• Pig TODO
• Hive TODO
• Impala TODO
• Presto TODO
• Apache Drill TODO
• Storm TODO
• ZooKeeper

Data Science

• Data Science TODO
• Python
  • Anaconda TODO
• R TODO
• Octave TODO

Time Series Databases

• OpenTSDB TODO
  • TCollector TODO
• InfluxDB

Performance Engineering

• Performance Engineering
• Stress Testing

Security

• Security TODO
• SSL TODO
• Kerberos TODO
• FreeIPA TODO
• Penetration Testing TODO

Virtualization

• Virtualization
• VirtualBox
• Vagrant
• Packer
• OpenStack TODO

Hardware

• Hardware
• Storage TODO

Networking

• Networking TODO
• Cisco TODO
• Juniper TODO
• Foundry TODO

Load Balancing

• HAProxy
• F5

Monitoring

• Monitoring
• Nagios Plugins

Misc

• DevOps Misc TODO

• Dev Misc TODO

• Open Source

• Windows

• AI

• Travel Consulting

• Banking

See Also

For tonnes of great free tech programs and scripts, see also:

• DevOps Bash Tools - 1000+ DevOps Bash Scripts, Advanced .bashrc, .vimrc, .screenrc, .tmux.conf, .gitconfig, CI configs & Utility Code Library - AWS, GCP, Kubernetes, Docker, Kafka, Hadoop, SQL, BigQuery, Hive, Impala, PostgreSQL, MySQL, LDAP, DockerHub, Jenkins, Spotify API & MP3 tools, Git tricks, GitHub API, GitLab API, BitBucket API, Code & build linting, package management for Linux / Mac / Python / Perl / Ruby / NodeJS / Golang, and lots more random goodies

• DevOps Python Tools - 80+ DevOps CLI tools for AWS, GCP, Hadoop, HBase, Spark, Log Anonymizer, Ambari Blueprints, AWS CloudFormation, Linux, Docker, Spark Data Converters & Validators (Avro / Parquet / JSON / CSV / INI / XML / YAML), Elasticsearch, Solr, Travis CI, Pig, IPython

• SQL Scripts - 100+ SQL Scripts - PostgreSQL, MySQL, AWS Athena, Google BigQuery

• Jenkins - Advanced Jenkinsfile & Jenkins Groovy Shared Library

• GitHub-Actions - GitHub Actions master template & GitHub Actions Shared Workflows library

• Templates - dozens of Code & Config templates - AWS, GCP, Docker, Jenkins, Terraform, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions Workflows, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml, M4 etc.

• Kubernetes configs - Kubernetes YAML configs - Best Practices, Tips & Tricks are baked right into the templates for future deployments

• Terraform - Terraform templates for AWS / GCP / Azure / GitHub management

• The Advanced Nagios Plugins Collection - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)

• Nagios Plugin Kafka - Kafka API pub/sub Nagios Plugin written in Scala with Kerberos support

• DevOps Perl Tools - 25+ DevOps CLI tools for Hadoop, HDFS, Hive, Solr/SolrCloud CLI, Log Anonymizer, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Dockerfiles & SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Ambari FreeIPA Kerberos, Datameer, Linux...

• HAProxy Configs - 80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, Cloudera, Hortonworks, MapR, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, ZooKeeper, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, SSH, RabbitMQ, Redis, Riak, Rancher etc.

• Dockerfiles - 50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Mesos, Consul, Riak, OpenTSDB, Jython, Advanced Nagios Plugins & DevOps Tools repos on Alpine, CentOS, Debian, Fedora, Ubuntu, Superset, H2O, Serf, Alluxio / Tachyon, FakeS3

• HashiCorp Packer templates - Linux automated bare-metal installs and portable virtual machines OVA format appliances using HashiCorp Packer, Redhat Kickstart, Debian Preseed and Ubuntu AutoInstaller / Cloud-Init

• Diagrams-as-Code - Cloud & Open Source architecture diagrams with Python & D2 source code provided - automatically regenerated via GitHub Actions CI/CD - AWS, GCP, Kubernetes, Jenkins, ArgoCD, Traefik, Kong API Gateway, Nginx, Redis, PostgreSQL, Kafka, Spark, web farms, event processing...

Pre-built Docker images are available for those repos (which include this one as a submodule) and the "docker available" icon above links to an uber image which contains all my github repos pre-built. There are Centos, Alpine, Debian and Ubuntu versions of this uber Docker image containing all repos.