Isaac Gazimbe - DevOps Golang Tools

git.io/go-tools

DevOps Golang Tools

All programs have --help to list the available options.

See Also:

• DevOps Bash Tools
• DevOps Python Tools
• DevOps Perl Tools
• Advanced Nagios Plugins Collection

repos which contain hundreds of Cloud, Big Data, NoSQL, Web, Linux and other DevOps infrastructure tools. Isaac Gazimbe

Cloud & Big Data Contractor, Johanessburg South Africa

https://www.linkedin.com/in/isaac-gazimbe-2154b015b?lipi=urn%3Ali%3Apage%3Ad_flagship3_profile_view_base_contact_details%3BfqwmHWlSQsCxrWg%2FUOEfcg%3D%3D

(you're welcome to connect with me on LinkedIn)

Make sure you run make update if updating and not just git pull as you will often need the latest library submodule and possibly new upstream libraries

Quick Start

Ready to run Docker image

All programs and their pre-compiled dependencies can be found ready to run on DockerHub.

List all programs:

docker run harisekhon/go-tools

Run any given program:

docker run harisekhon/go-tools <program> <args>

Automated Build from source

installs git, make, pulls the repo and build the binaries:

curl -L https://git.io/go-bootstrap | sh

or manually:

git clone https://github.com/harisekhon/devops-golang-tools go-tools
cd go-tools
make

The make command automates building the go binaries which can then be copied around to other systems of the same family, eg. Linux or Mac.

Alternatively there is shebang magic which means each .go program can be called directly like a script and it'll runtime compile and execute instantly like a scripted language. This is a neat trick for quick usage and testing built on go run, but for frequent use the compiled binaries are usually the way to go.

Detailed Build Instructions are available further down.

Usage

All programs come with a --help switch which includes a program description and the list of command line options.

Environment variables are supported for convenience and also to hide credentials from being exposed in the process list eg. $PASSWORD, $TRAVIS_TOKEN. These are indicated in the --help descriptions in brackets next to each option and often have more specific overrides with higher precedence eg. $AMBARI_HOST, $HBASE_HOST take priority over $HOST.

DevOps Golang Tools - Inventory

• Linux:
  • uniq2.go - like uniq but you don't have to sort first and it preserves the ordering
  • httpfirst.go - returns the first http/https url address argument to respond (fastest multi-threaded reply using go channels). More sophisticated version is find_active_server.py in the DevOps Python tools repo which can handle multi-master clusters, tcp sockets, regex etc.
  • colors.go - prints a table of terminal colors and their escape codes for doing fancy shell stuff
  • welcome.go - cool spinning welcome message greeting your username and showing last login time and user to put in your shell's .profile (there are also Python and Perl versions in my DevOps Python Tools and DevOps Perl Tools repos)

Detailed Build Instructions

Manual Setup

Enter the go-tools directory and run git submodule init and git submodule update to fetch my library repo:

git clone https://github.com/harisekhon/devops-golang-tools go-tools
cd go-tools
git submodule init
git submodule update
go build

Mac OS X

The automated build also works on Mac OS X but you'll need to install Apple XCode (on recent Macs just typing git is enough to trigger Xcode install).

I also recommend you get HomeBrew to install other useful tools and libraries you may need like OpenSSL for development headers and tools such as wget (these are installed automatically if Homebrew is detected on Mac OS X):

bash-tools/setup/install_homebrew.sh

Configuration for Strict Domain / FQDN validation

Strict validations include host/domain/FQDNs using TLDs which are populated from the official IANA list is done via my PyLib library submodule - see there for details on configuring this to permit custom TLDs like .local, .intranet, .vm, .cloud etc. (all already included in there because they're common across companies internal environments).

Updating

Run make update. This will git pull and then git submodule update which is necessary to pick up corresponding library updates.

Testing

Continuous Integration is run on this repo with tests for success and failure scenarios:

• integration tests of the top level programs using the libraries for things like option parsing
• functional tests for the top level programs using local test data and Docker containers

To trigger all tests run:

make test

which will start with the underlying libraries, then move on to top level integration tests and functional tests using docker containers if docker is available.

Contributions

Patches, improvements and even general feedback are welcome in the form of GitHub pull requests and issue tickets.

See Also

• DevOps Bash Tools - 700+ DevOps Bash Scripts, Advanced .bashrc, .vimrc, .screenrc, .tmux.conf, .gitconfig, CI configs & Utility Code Library - AWS, GCP, Kubernetes, Docker, Kafka, Hadoop, SQL, BigQuery, Hive, Impala, PostgreSQL, MySQL, LDAP, DockerHub, Jenkins, Spotify API & MP3 tools, Git tricks, GitHub API, GitLab API, BitBucket API, Code & build linting, package management for Linux / Mac / Python / Perl / Ruby / NodeJS / Golang, and lots more random goodies

• SQL Scripts - 100+ SQL Scripts - PostgreSQL, MySQL, AWS Athena, Google BigQuery

• DevOps Python Tools - 80+ DevOps CLI tools for AWS, GCP, Hadoop, HBase, Spark, Log Anonymizer, Ambari Blueprints, AWS CloudFormation, Linux, Docker, Spark Data Converters & Validators (Avro / Parquet / JSON / CSV / INI / XML / YAML), Elasticsearch, Solr, Travis CI, Pig, IPython

• DevOps Perl Tools - 25+ DevOps CLI tools for Hadoop, HDFS, Hive, Solr/SolrCloud CLI, Log Anonymizer, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Dockerfiles & SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Ambari FreeIPA Kerberos, Datameer, Linux...

• The Advanced Nagios Plugins Collection - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)

• HAProxy Configs - 80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, Cloudera, Hortonworks, MapR, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, ZooKeeper, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, SSH, RabbitMQ, Redis, Riak, Rancher etc.

• Templates - dozens of Code & Config templates - AWS, GCP, Docker, Jenkins, Terraform, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions Workflows, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml, M4 etc.

• Kubernetes configs - Kubernetes YAML configs - Best Practices, Tips & Tricks are baked right into the templates for future deployments

• Dockerfiles - 50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Mesos, Consul, Riak, OpenTSDB, Jython, Advanced Nagios Plugins & DevOps Tools repos on Alpine, CentOS, Debian, Fedora, Ubuntu, Superset, H2O, Serf, Alluxio / Tachyon, FakeS3

• PyLib - Python library

• Perl Lib - Perl library

Stargazers over time

git.io/go-tools