Skip to content
master
Go to file
Code

README.md


(5/29) NEW GUIDE AVAILABLE ILLUSTRATING EXAMPLES AND WALKTHROUGH OF AWS CLI QUERIES; CHECK OUT AWS_CLI_Queries_Walkthrough.md


Useful Documentation, Code, and Tools for use on AWS

  1. The AWS Code Examples Repository contains sample code for coding solutions in every AWS Service in every SDK available
    https://github.com/awsdocs/aws-doc-sdk-examples

  2. The Official Source of AWS Feature Updates
    https://aws.amazon.com/new/?whats-new-content-all.sort-by=item.additionalFields.postDateTime&whats-new-content-all.sort-order=desc

  3. Query EC2 Instance Attributes such as instanceId, Region, privateIp address, and more using this python module
    https://github.com/adamchainz/ec2-metadata

  4. Run bash scripts on AWS Lambda, also available as a container image
    https://github.com/gkrizek/bash-lambda-layer

  5. Return only the relevant datapoints you need by scripting with the aws-cli and jq
    https://medium.com/circuitpeople/aws-cli-with-jq-and-bash-9d54e2eabaf1
    https://blog.scottlowe.org/2017/08/15/quick-reference-common-aws-cli-commands/
    …as well as SQL
    https://github.com/lebinh/aq

  6. Comprehensive Collection of DevOps Scripts
    https://github.com/HariSekhon/DevOps-Bash-tools

  7. CLI tool that recommends an instance type based on your compute and memory requirements
    https://github.com/aws/amazon-ec2-instance-selector

  8. The verified AWS-Lab repo on Github has 450+ Amazon-approved solutions documented and coded for many common issues devs face
    https://github.com/awslabs

  9. Self-Described ‘practical guide to AWS’; hasn’t been updated in awhile but contains a solid central source of key terms and concepts
    https://github.com/open-guides/og-aws

  10. A curated list of awesome AWS libraries, open source repos, guides, blogs, and other resources. Definitely the gold standard in aws knowledge-sharing
    https://github.com/donnemartin/awesome-aws

  11. Hands on labs and code to help you learn, measure, and build using architectural best practices in AWS
    https://github.com/awslabs/aws-well-architected-labs

Using an AWS Service or Tool for the First Time:

Elastic Compute Cloud (EC2)

Official AWS Documentation Pages

Changing the instance type
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-resize.html

Placement groups
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html

aws ec2 modify-instance-placement
https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-placement.html

EC2 Scheduler
https://github.com/amazon-archives/ec2-scheduler/blob/master/documentation/ec2-scheduler.pdf

Working with security groups in Amazon EC2
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/ec2-example-security-group.html

Collection of workshops to demonstrate best practices in using Amazon EC2 Spot Instances
https://github.com/awslabs/ec2-spot-workshops

amazon-ec2-utils contains a set of utilities and settings for Linux deployments in EC2
https://github.com/aws/amazon-ec2-utils

Setting the Time for Your Linux Instance
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html

Amazon Web Services Elastic Compute Cloud (EC2) Rescue for Linux is a python-based tool that allows for the automatic diagnosis of common problems found on EC2 Linux instances.

https://github.com/awslabs/aws-ec2rescue-linux

Listing and Filtering Your Resources

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Filtering.html

Community Articles with Walkthroughs and Guides

EC2 instance metadata proxy for linux containers
https://github.com/dump247/ec2metaproxy

Spot Instances usage in batch jobs
https://github.com/hariboddu/AWS/blob/master/project.py

Playing the AWS ec2 Spot Market
https://medium.com/@mrpowers/playing-the-aws-ec2-spot-market-74b703454f4f

An easy interface to query the EC2 metadata API, with caching.
https://github.com/adamchainz/ec2-metadata

Linux and Server Admin
Password Authentication

How to prevent cloud-init from disable ssh PasswordAuthentication?
https://ask.openstack.org/en/question/26817/how-to-prevent-cloud-init-from-disable-ssh-passwordauthentication/

How to set “PasswordAuthentication” as yes through automation
https://askubuntu.com/questions/988845/how-to-set-passwordauthentication-as-yes-through-automation

Repositories

CentOS epel repository listed more than once
https://unix.stackexchange.com/questions/424099/centos-epel-repository-listed-more-than-once

Micellaneous Installation Guides

How To: Install Homebrew on Amazon Linux
https://garrettsyhampton.wordpress.com/2019/10/02/how-to-install-homebrew-on-amazon-linux/

Homebrew Formulae – Python 3.8
https://formulae.brew.sh/formula/python@3.8

How to install Python 3.8 on Ubuntu?
https://dev.to/mortoray/how-to-install-python-3-8-on-ubuntu-1bp4

User data to install and run Docker in an Ubuntu AWS instance
http://www.markcortada.com/blogs/nuevasTecnologiasEN/post/2017/05/21/User-data-to-install-and-run-Docker-in-an-Ubuntu-(AWS)-instance.aspx

How to Install Package Updates on Ubuntu via Command-Line
https://tecadmin.net/install-updates-on-ubuntu-via-command-line/

Updating instance software on your Amazon Linux instance
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-updates.html

I want to install a Graphical User Interface (GUI) in my Amazon EC2 instance running Amazon Linux 2. How do I do this?
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-2-install-gui/

How To Set Up GUI On Amazon EC2 Ubuntu server
https://comtechies.com/how-to-set-up-gui-on-amazon-ec2-ubuntu-server.html

Recover Deleted Files

How to Recover deleted files in Ubuntu through TestDisk
https://vitux.com/how-to-recover-deleted-files-in-ubuntu-through-testdisk/

Elastic Block Storage (EBS)

Official AWS Documentation Pages

Amazon EBS–optimized instances
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-optimized.html

Mapping between EC2 volume and your mounted filesystem
https://unix.stackexchange.com/questions/523839/mapping-between-ec2-volume-and-your-mounted-filesystem

Amazon EBS encryption
https://github.com/awsdocs/amazon-ec2-user-guide/blob/6420aa9d3d85edbe98b15eef07ea433ef3b6a78e/doc_source/EBSEncryption.md

Encrypting unencrypted resources
https://github.com/awsdocs/amazon-ec2-user-guide/blob/6420aa9d3d85edbe98b15eef07ea433ef3b6a78e/doc_source/EBSEncryption.md#encrypt-unencrypted

Using encryption with EBS-backed AMIs
https://github.com/awsdocs/amazon-ec2-user-guide/blob/f8fd01ab75b1f1ab054a7293a6b61157dce26ba6/doc_source/AMIEncryption.md

Detaching an Amazon EBS volume from a Linux instance
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-detaching-volume.html

Amazon EBS fast snapshot restore
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-fast-snapshot-restore.html

EC2 Instance Metadata Query Tool – A simple bash script that uses curl to query the EC2 instance Metadata from within a running EC2 instance.
https://aws.amazon.com/code/ec2-instance-metadata-query-tool/

Community Articles with Walkthroughs and Guides

Amazon EBS volumes: How to Shrink ’em Down to Size
https://cloudacademy.com/blog/amazon-ebs-shink-volume/

How to downsize a root EBS volume on AWS EC2 (Amazon Linux)
https://medium.com/@andtrott/how-to-downsize-a-root-ebs-volume-on-aws-ec2-amazon-linux-727c00148f61

How to reduce volume of an AWS EC2 instance
https://medium.com/@rahulian/how-to-reduce-volume-of-an-aws-ec2-instance-d493ec891698

Decrease the size of EBS volume in your EC2 instance
https://medium.com/@m.yunan.helmy/decrease-the-size-of-ebs-volume-in-your-ec2-instance-ea326e951bce

How to SHRINK an Amazon AWS EBS volume safely with no data loss
http://thehunk.blogspot.com/2017/04/how-to-shrink-amazon-aws-ebs-volume.html

Extending a Linux file system after resizing a volume
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html

How to encrypt EBS volumes of a running EC2 instance?
https://hackernoon.com/how-to-encrypt-ebs-volumes-of-a-running-ec2-instance-a2236b0b45

How to get latest Snapshot for a volume in AWS using API
https://stackoverflow.com/questions/55045588/how-to-get-latest-snapshot-for-a-volume-in-aws-using-api

how to get the latest aws volume snapshot id number using python/boto?
https://stackoverflow.com/questions/28221043/how-to-get-the-latest-aws-volume-snapshot-id-using-python-boto

Lambda for snapshotting EBS, only keeping latest snapshot
https://github.com/kkleidal/latest-snapshot-ebs/blob/master/backup-ebs-volumes.py

Encrypt EBS volumes from AWS EC2 instances
https://github.com/jbrt/ec2cryptomatic

Encrypt EBS volumes from AWS EC2 instances with a serverless StepFunctions machine
https://github.com/jbrt/ec2cryptomatic-serverless

ec2_encrypt_ebs
https://github.com/ffturan/ec2_encrypt_ebs/blob/e2bf95bd4a740db599f1df3be4546dcde22a37d8/README.md

Iterate through each attached volume and encrypt it for EC2 – root
https://github.com/indrahrp/s4misc/blob/375c07f8f53467840b35808b4e6a3facc8bb4390/ec2_root_encryption1.py

Iterate through each attached volume and encrypt it for EC2. – one instance
https://github.com/indrahrp/s4misc/blob/375c07f8f53467840b35808b4e6a3facc8bb4390/encrypt-oneinstance.py

Tools to manage AWS EBS volumes and snapshots
https://github.com/juliogonzalez/ebs-tools

Serverless Backup for your EBS Volumes on AWS, using AWS Lambda
https://github.com/devlinrcg/ebs-serverless-backup

Tag EBS Volumes with Names Similar to the hosts they’re attached too
https://github.com/undeadops/ebsTagger

I do not have permission to access files on the mounted volume
http://nfs.sourceforge.net/nfs-howto/ar01s07.html#pemission_issues

Key Management Service (KMS)

Official AWS Documentation Pages

AWS Key Management Service Best Practices (Whitepaper)
https://d0.awsstatic.com/whitepapers/aws-kms-best-practices.pdf

Share Custom Encryption Keys More Securely Between Accounts by Using AWS Key Management Service
https://aws.amazon.com/blogs/security/share-custom-encryption-keys-more-securely-between-accounts-by-using-aws-key-management-service/

How do I share my customer master keys (CMKs) across multiple AWS accounts?
https://aws.amazon.com/premiumsupport/knowledge-center/share-cmk-account/

Allowing users in other accounts to use a CMK
https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying-external-accounts.html

AWS KMS API permissions: Actions and resources reference
https://docs.aws.amazon.com/kms/latest/developerguide/kms-api-permissions-reference.html

Community Articles with Walkthroughs and Guides

Systems Manager (SSM)

Official AWS Documentation Pages

The open source version of the AWS Systems Manager user guide. You can submit feedback & requests for changes by submitting issues in this repo or by making proposed changes & submitting a pull request.
https://github.com/awsdocs/aws-systems-manager-user-guide

How do I install AWS Systems Manager Agent (SSM Agent) on an Amazon EC2 Linux instance at launch?
https://aws.amazon.com/premiumsupport/knowledge-center/install-ssm-agent-ec2-linux/

The AWS-UpdateWindowsAmi document enables you to automate image maintenance tasks on your Amazon Windows AMIs without having to author the workflow in JSON or YAML.
https://github.com/awsdocs/aws-systems-manager-user-guide/blob/master/doc_source/automation-walk-patch-windows-ami-cli.md

How patches are installed
https://docs.aws.amazon.com/systems-manager/latest/userguide/patch-manager-how-it-works-installation.html

Query for the latest Amazon Linux AMI IDs using AWS Systems Manager Parameter Store
https://aws.amazon.com/blogs/compute/query-for-the-latest-amazon-linux-ami-ids-using-aws-systems-manager-parameter-store/

A utility to convert your Python or Bash programs into SSM documents (and optionally - CloudFormation templates, containing those documents).
https://github.com/awslabs/aws-systems-manager-document-generator

Automation system variables

https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-variables.html

Systems Manager Automation actions reference

https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-actions.html

Running Automation workflows in multiple AWS Regions and accounts

https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-automation-multiple-accounts-and-regions.html

Running an Automation workflow by using delegated administration
https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-walk-security-delegated.html

Attach an IAM instance profile to an EC2 instance
https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-launch-managed-instance.html

Community Articles, Forums, Repos with Code and Guides

Terraform module to automate patching of AMI templated EC2 Linux instances in a specify schedule
https://github.com/osodevops/aws-terraform-module-ssm-automated-patching

A command line utility enabling SSH-like interactive sessions and continuous output for long running commands via the AWS SSM API.
https://github.com/node40/smsh

AWS Storage Gateway

Official AWS Documentation Pages

Requirements
https://docs.aws.amazon.com/storagegateway/latest/userguide/Requirements.html

Performance
https://docs.aws.amazon.com/storagegateway/latest/userguide/Performance.html

awslabs – aws-cloudformation-templates/aws/services/CloudFormation/HostnameChangeRHEL-Userdata.template
https://github.com/awslabs/aws-cloudformation-templates/blob/master/aws/services/CloudFormation/HostnameChangeRHEL-Userdata.template

Community Articles, Forums, Repos with Code and Guides

Creating AWS File Gateway as an NFS storage for your EC2 instance.
https://medium.com/tensult/creating-aws-file-gateway-as-an-nfs-storage-for-your-ec2-instance-57c141c76409

Very very slow performance…
https://forums.aws.amazon.com/thread.jspa?threadID=250901

Community GitHub Repos

FUSE-based file system backed by Amazon S3
https://github.com/s3fs-fuse/s3fs-fuse

IAM

Official AWS Documentation Pages

Switching to an IAM Role (AWS API)
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-api.html

How can I configure a Lambda function to assume a role from another AWS account?
https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-assume-iam-role/

Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html

Community Articles with Walkthroughs and Guides

A tool for quickly evaluating IAM permissions in AWS.
https://github.com/nccgroup/PMapper

Awsume is a convenient way to manage session tokens and assume role credentials.
https://awsu.me/

Auto-Scaling Groups

Official AWS Documentation Pages
Community Articles with Walkthroughs and Guides

AWS Auto Scaling based on Memory Utilization in CloudFormation
https://medium.com/@lvthillo/aws-auto-scaling-based-on-memory-utilization-in-cloudformation-159676b6f4d6

Elastic Load Balancer (ELB)

Official AWS Documentation Pages
Community Articles with Walkthroughs and Guides

What is Amazon Elastic Load Balancer (ELB)
https://hackernoon.com/what-is-amazon-elastic-load-balancer-elb-16cdcedbd485

CloudWatch

Official AWS Documentation Pages

Viewing Available Metrics
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/viewing_metrics_with_cloudwatch.html

Why isn’t my CloudWatch GetMetricStatistics API call returning data points?
https://aws.amazon.com/premiumsupport/knowledge-center/cloudwatch-getmetricstatistics-data/

Ingesting High-Cardinality Logs and Generating Metrics with CloudWatch Embedded Metric Format
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html

Using Metric Math
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html

Amazon CloudWatch Embedded Metric Format Client Library
https://github.com/awslabs/aws-embedded-metrics-python

Probabilistic time series modeling in Python
https://github.com/awslabs/gluon-ts

Push metrics and logs to CloudWatch from any system that can run Python
https://github.com/awslabs/amazon-cloudwatch-publisher

A CLI tool to merge schema files, and validate operations against the merged GraphQL Schema. With features like adding your own custom validation rules.
https://github.com/awslabs/graphql-schema-utilities

Graph Metrics Manually on a CloudWatch Dashboard
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/add_old_metrics_to_graph.html

Amazon CloudWatch Embedded Metric Format Client Library
https://github.com/awslabs/aws-embedded-metrics-python/tree/master/examples

Building an Amazon CloudWatch Dashboard Outside of the AWS Management Console
https://aws.amazon.com/blogs/devops/building-an-amazon-cloudwatch-dashboard-outside-of-the-aws-management-console/

Amazon CloudWatch metrics for Amazon EBS

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cloudwatch_ebs.html

CLOUDWATCH DASHBOARD
https://reinvent2019.aws-management.tools/mgt415/en/cont/3cloudwatch.html

Amazon EBS Metrics
https://github.com/awsdocs/amazon-ec2-user-guide-windows/blob/befad90a6b8c71abbec34335ea0ea4e6dad715e9/doc_source/using_cloudwatch_ebs.md#ebs-volume-metrics

Community Articles with Walkthroughs and Guides

access to custom cloudwatch metrics using lambda+boto3?
https://serverfault.com/questions/824510/access-to-custom-cloudwatch-metrics-using-lambdaboto3

Metrics exporter for Amazon AWS CloudWatch
https://github.com/prometheus/cloudwatch_exporter

[ARCHIVE] DSL for querying against metrics stored in Amazon CloudWatch
https://github.com/fsprojects/zzarchive-Amazon.CloudWatch.Selector

Store custom CloudWatch metrics in a cost-effective way
https://github.com/alpha-lambda/cloudwatch-telemetry

USING CLOUDWATCH TO MONITOR AWS S3 BUCKETS
https://www.opsdash.com/blog/aws-s3-cloudwatch-monitoring.html

Redash is our take on freeing the data within our company in a way that will better fit our culture and usage patterns.
https://github.com/Dorpolo/redash/tree/f09179178a8f3dba2297a50cab723b59ccbe1040

Zabbix Scripts & templates for monitoring AWS Services
https://github.com/Enrise/zabbix-cloudwatch/blob/ee571260055c6c8d2be43e532c929a2d005c7e44/externalscripts/cloudfront_stats.py

AWS CloudWatch Fluent Metrics
https://pypi.org/project/cloudwatch-fluent-metrics/

Start up script used to create Cloudwatch monitoring and alarms for RAM.
https://github.com/1Strategy/ram-monitoring-script

A Dockerized version of Ice, Netflix’s AWS Usage tool.
https://github.com/jonbrouse/docker-ice

AWS Cloudwatch to Spreadsheet
https://github.com/petezybrick/awscwxls

The Node module to draw charts for AWS CloudWatch metrics
https://github.com/jeka-kiselyov/aws-cloudwatch-chart

CloudWatch to AWS Lambda to Slack Channel Alerts and Charts. Posts CloudWatch Alerts via SNS topic via Lambda function to Slack channel. And draws charts for CPUUtilization metrics(both for EC2 abd RDS).
https://gist.github.com/jeka-kiselyov/3f3801a165cab9e4a9fd

Just dump all the CloudWatch metrics.
https://github.com/mogproject/cloudwatch-dump

Queries AWS for metrics of running EC2 instances.
https://github.com/kgryte/aws-ec2-metrics

An API to export Cloudwatch Metrics to Prometheus exposition format
https://github.com/marcelocrnunes/cwexporter/tree/5101e26e16328c92959e32e2a058210195c8d8ac

Allowing to display your AWS Cloudwatch dashboards on your website or locally
https://github.com/Dlas1212/AWSCloudwatchExternalDashboards

Bring Amazon AWS Cloudwatch metrics into Graphite
https://github.com/edasque/cloudwatch2graphite

Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.
https://github.com/eric-erki/Make-Your-Company-Data-Driven./tree/944adb95ba29bf241fa9a0af9a80734f3afbb3e3

Send metrics from AWS Cloudwatch to Sumo Logic
https://github.com/gadouryd/AWS-Scripts/tree/1df2e439f300c821ad19d302ec56c438580c34dd

Extremely Useful AWS CLI shell scripts to be used with CloudWatch
https://github.com/mikeoscarson/aswadmin/tree/a2fea10904a841221630ad4a0a7bedd7d91d6d8f/cloudwatchmetrics

20+ Grafana dashboards for AWS CloudWatch metrics: EC2, Lambda, S3, ELB, EMR, EBS, SNS, SES, SQS, RDS, EFS, ElastiCache, Billing, API Gateway, VPN, Step Functions, Route 53, CodeBuild, …
https://github.com/monitoringartist/grafana-aws-cloudwatch-dashboards

Lambda to create cloudwatch metric filters, custom metrics and dashboard
https://github.com/nikhilswagle/cloudwatch-monitoring-lambda

Relational Database Service (RDS)

Official AWS Documentation Pages

Importing Data into Oracle on Amazon RDS
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Procedural.Importing.html

Community Articles, Forums, Repos with Code and Guides

Database Migration Service (DMS)

Official AWS Documentation Pages

Using an Oracle Database as a Source for AWS DMS
https://github.com/awsdocs/aws-dms-user-guide/blob/master/doc_source/CHAP_Source.Oracle.md

Best practices for migrating an Oracle database to Amazon RDS PostgreSQL or Amazon Aurora PostgreSQL: Source database considerations for the Oracle and AWS DMS CDC environment
https://aws.amazon.com/blogs/database/best-practices-for-migrating-an-oracle-database-to-amazon-rds-postgresql-or-amazon-aurora-postgresql-source-database-considerations-for-the-oracle-and-aws-dms-cdc-environment/

How can I troubleshoot AWS DMS endpoint connectivity failures?
https://aws.amazon.com/premiumsupport/knowledge-center/dms-endpoint-connectivity-failures/

Create AWS CloudFormation templates for AWS DMS tasks using Microsoft Excel
https://aws.amazon.com/blogs/database/create-aws-cloudformation-templates-for-aws-dms-tasks-using-microsoft-excel/

Community Articles, Forums, Repos with Code and Guides

Using AWS Data Migration Service with RDS
https://linuxacademy.com/guide/13043-using-aws-data-migration-service-with-rds/

Creates an example AWS DMS for replicating an (on-prem) Oracle database to a cloud-based Postgres database
https://github.com/GeoscienceAustralia/aws-dms-example

Sample code that reads Microsoft Excel workbook/CSV File for the details required to create a DMS task CloudFormation template
https://github.com/aws-samples/dms-cloudformation-templates-generator/tree/v1.0

AWS Simple Storage Service (S3)

Official AWS Documentation Pages

Why am I getting Access Denied errors on ListObjects actions when I run a sync command on my Amazon S3 bucket?
https://aws.amazon.com/premiumsupport/knowledge-center/s3-access-denied-listobjects-sync/

Community Articles, Forums, Repos with Code and Guides

Print usage stats of S3 buckets
https://github.com/marpada/s3_stats/blob/master/s3_stats.py

Amazon QuickSight

Official AWS Documentation Pages

Amazon QuickSight Python3 SDK
https://docs.aws.amazon.com/quicksight/latest/user/quicksight-sdk-python.html

Advanced analytics with table calculations in Amazon QuickSight
https://aws.amazon.com/blogs/big-data/advanced-analytics-with-table-calculations-in-amazon-quicksight/

Python3 Example of Embedding a Dashboard with Amazon QuickSight
https://github.com/awsdocs/amazon-quicksight-user-guide/blob/master/doc_source/embedded-dashboards-example-python.md

A QuickSight dashboard embedding sample for web apps.
https://github.com/aws-samples/amazon-quicksight-embedding-sample

ARN Formats
https://docs.aws.amazon.com/quicksight/latest/APIReference/qs-arn-format.html

Amazon QuickSight Embedded Analytics
https://aws.amazon.com/quicksight/features-embedding/

Evolve your analytics with Amazon QuickSight’s new APIs and theming capabilities
https://aws.amazon.com/blogs/big-data/evolve-your-analytics-with-amazon-quicksights-new-apis-and-theming-capabilities/

Community Articles, Forums, Repos with Code and Guides

AWS Quicksight - Quick Guide
https://www.tutorialspoint.com/aws_quicksight/aws_quicksight_quick_guide.htm

Tools that make it easier to analyze AWS Cost and Usage reports. Initial version provides support for Athena and QuickSight.
https://github.com/concurrencylabs/aws-cost-analysis

Amazon QuickSight, Amazon Athena and Amazon Spectrum workshop. Workshop will focus on ingesting data into Athena & Spectrum, combining it with other data sources, and visualizing it in QuickSight.
https://github.com/avinashnid/QuicksightAthenaSpectrum

The objective of the project is to build a data pipeline which helps in extracting insights from the data with faster turnaround time.
https://github.com/Nikhilgupta1891/AWS-Lambda-Athena-QuickSight

Python3 Example for Getting the Dashboard URL from the App Server
https://docs.aws.amazon.com/quicksight/latest/user/embedded-dashboards-example-python.html

HTML Example for Embedding a Dashboard with Amazon QuickSight Using the URL from the App Server
https://docs.aws.amazon.com/quicksight/latest/user/embedded-dashboards-example-html.html

AWS Glue

Official AWS Documentation Pages

Find Matches Examples
https://github.com/aws-samples/aws-glue-samples/tree/master/findmatches

FAQ for AWS Glue
https://github.com/aws-samples/aws-glue-samples/blob/master/FAQ_and_How_to.md

Utilities for AWSGlue
https://github.com/aws-samples/aws-glue-samples/tree/master/utilities

Samples of AWS Glue Implementation
https://github.com/aws-samples/aws-glue-samples

Dynamic Dataframes
https://github.com/aws-samples/aws-glue-samples/blob/master/examples/data_cleaning_and_lambda.md

Community Articles, Forums, Repos with Code and Guides

AWS ECS

Official AWS Documentation Pages

The Amazon ECS CLI enables users to create and manage their ECS projects, from source code, infrastructure to operations.
https://github.com/aws/amazon-ecs-cli-v2

Docker Basics for Amazon ECS
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html

The role of AWS Fargate in the container world
https://aws.amazon.com/blogs/containers/the-role-of-aws-fargate-in-the-container-world/

Community Articles, Forums, Repos with Code and Guides

Full IAC to build jenkins master (AMI, master & slave docker image), slave cluster in different regions on EC2, ECS EC2 & fargate on AWS

https://github.com/linhkikuchi/aws-jenkins-cluster-ecs-fargate

Fargate CLI
https://somanymachines.com/fargate/

How to use AWS Fargate and Lambda for long-running processes in a Serverless app
https://www.serverless.com/blog/serverless-application-for-long-running-process-fargate-lambda/

AWS Fargate Labs
https://github.com/koundykarthik/stylelabs

Deploy serverless containers to the cloud from your command line (Go-based Fargate CLI)
https://github.com/turnerlabs/fargate

Copies all the stuff in account, region “a” to account, region “b” in regards to ECR. A quick lift and shift deal.
https://github.com/DarklyAutumnClad/CopyAWSCLIECRRepositoriesCrossAccountCrossRegion

Testing ground for container orchestration on AWS
https://github.com/angelokurtis/amazon-ecs-labs

CloudFormation

Official AWS Documentation Pages

Best Practices
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/best-practices.html

The cfn-init helper script reads template metadata from the AWS::CloudFormation::Init key and acts accordingly to, Fetch and parse metadata from AWS CloudFormation, Install packages, Write files to disk, Enable/disable and start/stop services
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-init.html

Templates exemplifying the conditional creation of resources based on environment
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-sample-templates.html

Run arbitrary python code in your CloudFormation templates
https://github.com/aws-cloudformation/aws-cloudformation-macros/tree/eef5948e26104d7a7e81c1be9d1ed336c79940c1/PyPlate

A collection of useful CloudFormation templates
https://github.com/awslabs/aws-cloudformation-templates

Community Articles, Forums, Repos with Code and Guides

Lambda

Official AWS Documentation Pages

Blank AWS Lambda Function with Python Runtime
https://github.com/awsdocs/aws-lambda-developer-guide/tree/master/sample-apps/blank-python

Community Articles, Forums, Repos with Code and Guides

Get AWS account id inside Lambda function programmatically - python
https://gist.github.com/aws-scripting-guy/83d901527fa7adc271da

Run Bash scripts in AWS Lambda via Layers
https://github.com/gkrizek/bash-lambda-layer

Amazon Web Services Modules (aws-modules, awsm’s) contain pre-written, isolated functions ready to run on one or multiple AWS Lambda functions. Some examples are: functions that send out emails, register users or handle webhooks from other services.
https://github.com/awsm-org/awsm

The AWS Lambda container image converter tool (img2lambda) repackages container images (such as Docker images) into AWS Lambda function deployment packages and Lambda layers.
https://github.com/awslabs/aws-lambda-container-image-converter

Elastic Kubernetes Service (EKS)

Official AWS Publication
Community Sourced

A curated list of awesome tools for Amazon EKS
https://github.com/realvz/awesome-eks

General Utilities

Official AWS Publication

The AWS Deployment Framework (ADF) is an extensive and flexible framework to manage and deploy resources across multiple AWS accounts and regions based on AWS Organizations.
https://github.com/awslabs/aws-deployment-framework

Create CloudFormation templates from existing infrastructure – this version of the CloudFormer solution can be used to launch the solution into an existing VPC.
https://github.com/awslabs/aws-support-tools/tree/master/CloudFormation/CloudFormer

Serverless app designed for any customer with two or more accounts to view resources across accounts/regions in simple single pane of glass website
https://github.com/awslabs/aws-multi-account-viewer

AWS-SDK-IOS-Samples
https://github.com/awslabs/aws-sdk-ios-samples

Imputation of missing values in tables.
https://github.com/awslabs/datawig

A solution which enables a multilingual conversational platform, facilitating the creation of applications using voice and text
https://github.com/awslabs/serverless-bot-framework

This repo contains a simple application that consists of three microservices. Each application is deployed using different Compute options on AWS.
https://github.com/aws-samples/aws-microservices-deploy-options

The AWS CDK is a new software development framework from AWS with the sole purpose of making it fun and easy to define cloud infrastructure in your favorite programming language and deploy it using AWS CloudFormation.
https://cdkworkshop.com/

Official AWS CDK Roadmap
https://github.com/orgs/aws/projects/7

Collection of AWS Scripts
https://github.com/marcomoscatiello/aws_scripts

Community Sourced

A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.
https://github.com/donnemartin/awesome-aws

Rich Morrow’s Personal Compilation of Helpful AWS Links
http://aws-materials.s3.amazonaws.com/index.html

Use Boto3 to Assume a Role in another AWS Account
https://www.slsmk.com/use-boto3-to-assume-a-role-in-another-aws-account/

Building Beautiful Command Line Interfaces with Python
https://codeburst.io/building-beautiful-command-line-interfaces-with-python-26c7e1bb54df

All Datasets used as part of 538.com articles – good repo to read sample data from
https://github.com/fivethirtyeight/data

This script will process cloudtrail records to create a sqlite db of resources and their creators, and then use that sqlitedb to tag the resources with their creator’s name.
https://cloudcustodian.io/docs/tools/c7n-trailcreator.html

A tool to load bulk simulation data into AWS IoT Analytics
https://pypi.org/project/onica-iotloader/

This is a set of boilerplate scripts describing the normalized script pattern that GitHub uses in its projects. While these patterns can work for projects based on any framework or language, these particular examples are for a simple Ruby on Rails application.
https://github.com/github/scripts-to-rule-them-all

On this lab we will convert our infrastructure into code by using CloudFormation to stand up the same stack from last week.
https://github.com/coolx28/DevSecOps/blob/c18f57b7ecbbff7e442b5ab34913b3b279da22ed/Week-5/labs/LAB-1.md

AI and Machine Learning with Kubeflow, Amazon EKS, and SageMaker
https://github.com/data-science-on-aws/workshop

amplify-chime-react
https://github.com/mats16/amplify-chime-react/tree/f1a19708f13246efa318583dec72a15ad5c93c86

Boto and LibCloud to automate AWS and OpenStack related tasks (CloudWatch/AutoScaling/Glacier/Volumes/EC2/S3/SNS/etc)
https://github.com/rsd-raul/Cloud-Computing-Project/tree/634514e55827830aae25f1a4460750f9a3e01482

awstools
https://github.com/iMilnb/awstools

Devops_wordpress_Notes
https://github.com/sandeepmchary/Devops_wordpress_Notes/blob/1d4603ac98b7b47b681967f32508a8c91ac38360/Udemy/Aws-automation-with-boto3/Session-4/Sesssion-4.txt

List of open source tools for AWS security: defensive, offensive, auditing, DFIR, etc.
https://github.com/osodevops/my-arsenal-of-aws-security-tools

A repository of useful functions (offensive & defensive) to interact with AWS services
https://github.com/carnal0wnage/weirdAAL/wiki

Project to manage AWS Organization using boto3
https://github.com/osodevops/aws-orgs

Cartography is a Python tool that consolidates infrastructure assets and the relationships between them in an intuitive graph view powered by a Neo4j database.
https://github.com/lyft/cartography

Rules engine for cloud security, cost optimization, and governance, DSL in yaml for policies to query, filter, and take actions on resources
https://github.com/cloud-custodian/cloud-custodian/

Cloud Custodian Policies
https://github.com/kvskmech/Cloud-Custodian-Policies/tree/ac5390a353db654e348a12b5b6d4300967168484/policies

Windows Subsystem for Linux

Awesome list dedicated to Windows Subsystem for Linux
https://github.com/sirredbeard/Awesome-WSL#wsl-tools

Amazon Linux as a WSL Instance. Supports multiple install.
https://github.com/yosukes-dev/AmazonWSL

Setting up Ubuntu (WSL) for Linux GUI Apps
https://x410.dev/cookbook/wsl/setting-up-wsl-for-linux-gui-apps/

Targeting the Windows Subsystem for Linux from Visual Studio
https://devblogs.microsoft.com/cppblog/targeting-windows-subsystem-for-linux-from-visual-studio/

Tutorials, Certification and Trainings

General Algorithms, Data Structures, Big O, etc

Big O Cheatsheet
https://www.bigocheatsheet.com/

Data Structures
https://sinon.org/algorithms//#data-structures

AWS CLI

Environment variables to configure the AWS CLI
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html

Getting credentials from EC2 instance metadata
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-metadata.html

AWS CLI Cheatsheet
https://gist.github.com/apolloclark/b3f60c1f68aa972d324b

How to run a script or a command on EC2 instance via AWS CLI?
https://devops.stackexchange.com/questions/3264/how-to-run-a-script-or-a-command-on-ec2-instance-via-aws-cli

AWS-Python-Utilities
https://github.com/hms-dbmi/aws-python-utilities

yq is a lightweight and portable command-line YAML processor
https://mikefarah.gitbook.io/yq/

AWS CLI v2 Docker image
https://aws.amazon.com/blogs/developer/aws-cli-v2-docker-image/

AWS CLI with jq and Bash – The CLI is utilitarian, but a little jq sauce makes it beautiful
https://medium.com/circuitpeople/aws-cli-with-jq-and-bash-9d54e2eabaf1

AWS CLI with jq make things easy
https://www.lewuathe.com/aws-cli-with-jq-make-things-easy.html

jq in-browser test queries
https://jqplay.org/

AWS Lambda Layer for AWS CLI
https://github.com/aws-samples/aws-lambda-layer-awscli

Examples of querying AWS through ‘–query’ flag with AWS-CLI, as well as through SQL and other means

Querying Instances using Filter on Tag Values
https://gist.github.com/zharling/0c1049dc362a17d1bd0ef0c1d2fabec8

Useful AWS command line tools using JMESPath and jq
https://gist.github.com/micklove/76b2cb13ea4d86f4b5f4ec42b0f9939b

AWS-CLI Cheatsheet
https://gist.github.com/davidmoremad/6db1981d37ed6b6481a29b91929a4fa4

AWS-CLI commands(Region, AZ, IAM, EC2)
https://gist.github.com/nvogel/21a36a9a44548aa2835b2ea9ffb8032e

AWS-CLI commands examples (Autoscaling and EC2)
https://gist.github.com/namuan/4d612f75e3d757935fe60aa530de7aa6

Query AWS resources with SQL
https://github.com/lebinh/aq

Quick Reference to Common AWS CLI Commands
https://blog.scottlowe.org/2017/08/15/quick-reference-common-aws-cli-commands/

Controlling command output from the AWS CLI
https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.html

How can I list Amazon EBS volume or snapshot information for my Amazon EC2 instance by using the AWS CLI?
https://aws.amazon.com/premiumsupport/knowledge-center/ebs-volume-snapshot-ec2-instance/

awsq provides a way to query and filter data from aws ec2 describe-instances
https://github.com/drewkhoury/awsq

JMESPath exploration tool in the terminal
https://github.com/jmespath/jmespath.terminal

aegea - Amazon Web Services Operator Interface
https://github.com/kislyuk/aegea

Shell Scripting

Bash Scripting Tutorial - 5. If Statements
https://ryanstutorials.net/bash-scripting-tutorial/bash-if-statements.php

Passing Arguments to Bash Functions
https://linuxize.com/post/bash-functions/#passing-arguments-to-bash-functions

Script to fetch an MFA token for you to use temporary aws access credentials
https://github.com/asagage/aws-mfa-script

AWK command in Unix/Linux with examples
https://www.geeksforgeeks.org/awk-command-unixlinux-examples/

Bash Read Comma Separated CSV File on Linux / Unix
https://www.cyberciti.biz/faq/unix-linux-bash-read-comma-separated-cvsfile/

How to do integer & float calculations, in bash or other languages/frameworks?
https://unix.stackexchange.com/questions/40786/how-to-do-integer-float-calculations-in-bash-or-other-languages-frameworks

Is there any way to copy all lines from open file to clipboard in VI editor?
https://stackoverflow.com/questions/1620018/copy-all-the-lines-to-clipboard

How can I preview HTML documents from the command line?
https://askubuntu.com/questions/58416/how-can-i-preview-html-documents-from-the-command-line

Bash script to easily set shell environment with temporary access tokens using a config file and the AWS credentials file
https://github.com/rik2803/aws-sts-assumerole

DevOps Bash Tools
https://github.com/HariSekhon/DevOps-Bash-tools

The GNU Awk User’s Guide
https://www.gnu.org/software/gawk/manual/gawk.html

A CLI tool and go library which recommends instance types based on resource criteria like vcpus and memory

https://github.com/aws/amazon-ec2-instance-selector

Containers

Open Container Initiative – Creating open standards around container technology
https://github.com/opencontainers

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://github.com/aws/containers-roadmap

Docker

HOW TO KEEP DOCKER CONTAINERS RUNNING
http://bigdatums.net/2017/11/07/how-to-keep-docker-containers-running/

Docker Certified Associate
https://www.whizlabs.com/learn/course/docker-certified-associate/

Simple Docker image with the AWS CLI
https://github.com/mikesir87/aws-cli-docker

How To Install and Use Docker on Ubuntu 18.04
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04

User data to install and run Docker in an Ubuntu AWS instance
http://www.markcortada.com/blogs/nuevasTecnologiasEN/post/2017/05/21/User-data-to-install-and-run-Docker-in-an-Ubuntu-(AWS)-instance.aspx

Examples using the Docker Engine SDKs and Docker API
https://docs.docker.com/engine/api/sdk/examples/

docker build
https://docs.docker.com/engine/reference/commandline/build/

How To SSH Into A Running Docker Container And Run Commands
https://phoenixnap.com/kb/how-to-ssh-into-docker-container

Dockerize your Python Application
https://runnable.com/docker/python/dockerize-your-python-application

This project provides source code of example applications for people to get started using AWS Serverless Application Repository.
https://github.com/aws-samples/serverless-app-examples/blob/master/python/hello-world-python3/lambda_function.py

How to containerize your legacy apps without sweating
https://www.cio.com/article/3191502/how-to-containerize-your-legacy-apps-without-sweating.html

Podman

What is Podman? Podman is a daemonless container engine for developing, managing, and running OCI Containers on your Linux System. Containers can either be run as root or in rootless mode. Simply put: alias docker=podman.
https://podman.io/

Podman Installation Instructions
https://podman.io/getting-started/installation

How to run Podman on Windows with WSL2
https://www.redhat.com/sysadmin/podman-windows-wsl2

libpod is a library used to create container pods. Home of Podman.
https://github.com/containers/libpod

Configuring container networking with Podman
https://www.redhat.com/sysadmin/container-networking-podman

containerd is available as a daemon for Linux and Windows. It manages the complete container lifecycle of its host system, from image transfer and storage to container execution and supervision to low-level storage to network attachments and beyond.
https://containerd.io/

Building, Running, and Managing Containers with RHEL8
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/building_running_and_managing_containers/index?extIdCarryOver=true&sc_cid=701f2000001OH7JAAW#using_standard_red_hat_base_images

Unprivileged Sandbox Tool
https://github.com/containers/bubblewrap

A quick guide to podman and toolbox in Debian (and maybe Ubuntu)
https://dev.to/bureado/a-quick-guide-to-podman-and-toolbox-in-debian-5672

Python

defaultdict
https://docs.python.org/3/library/collections.html#defaultdict-examples

comp3321 NSA Python Training Document
https://archive.org/details/comp3321/page/n387/mode/2up/search/device

Leetcode Algorithms
https://github.com/JBielan/leetcode-py-js/tree/master/python

Number Of Islands Tutorial (Python)
https://colorfulcodesblog.wordpress.com/2018/09/06/number-of-islands-tutorial-python/

heapq — Heap queue algorithm
https://docs.python.org/3.8/library/heapq.html

Python Sorting Algorithms
https://www.tutorialspoint.com/python_data_structure/python_sorting_algorithms.htm

LRU-Cache
https://leetcode.com/problems/lru-cache/discuss/164037/Solution-(Python)-using-Doubly-Linked-List-and-Map-with-Detailed-Explanation

Stop pip from failing on single package when installing with requirements.txt
https://stackoverflow.com/questions/22250483/stop-pip-from-failing-on-single-package-when-installing-with-requirements-txt

An easy interface to query the EC2 metadata API, with caching.
https://pypi.org/project/ec2-metadata/

Parse ARNs using Python
https://pypi.org/project/arnparse/

Wrap a WSGI application in an AWS Lambda handler function for running on API Gateway or an ALB.
https://pypi.org/project/apig-wsgi/

A Simple and Scalable Logging Utility With a Beautiful Visualization Dashboard
https://pypi.org/project/ml-logger/

bunker is a command line program for setting up an ec2 in AWS for remote development or as a backup. It can clone your git repos, and transfer ignored files from your machine to the ec2.
https://pypi.org/project/aws-bunker/

A collection of useful decorators for making AWS Lambda handlers
https://pypi.org/project/lambda-decorators/

Boto3 (Python SDK for AWS)

Async client for aws services using botocore and aiohttp
https://pypi.org/project/aiobotocore/

Async AWS SDK for Python
https://aioboto3.readthedocs.io/en/latest/usage.html

Resources – Boto3
https://github.com/boto/boto3/blob/develop/docs/source/guide/resources.rst

boto3 quick hands-on
https://gist.github.com/itorres/0ce27bc1bbfe17f863f2559602a3bc1c

Pandas on AWS
https://github.com/awslabs/aws-data-wrangler

Migration from Boto2.x to Boto3
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/migrationec2.html

A basic boto3 based tool for retrieving information from running EC2 instances.
https://gist.github.com/dastergon/b4994c605f76d528d0c4

Manipulate and parse data in Python using dataframes, also includes info on handling csv and Excel files (.xlsx, .xlsm)

xlsxwriter – Adjust excel reference cells in loop when creating formula
https://stackoverflow.com/questions/25183305/python-xlsxwriter-adjust-excel-reference-cells-in-loop-when-creating-formula

openpyxl – Accessing many cells; Ranges of cells can be accessed using slicing:
https://openpyxl.readthedocs.io/en/latest/tutorial.html#accessing-many-cells

openpyxl – Sheet: Data Access
http://docs.pyexcel.org/en/latest/tutorial.html

Examples of automation of excel via python, and related useful things
https://github.com/richard512/python-Excel-Automation

Ingest Excel data automatically into Amazon QuickSight
https://aws.amazon.com/blogs/big-data/ingest-excel-data-automatically-into-amazon-quicksight/

How to Export Pandas DataFrame to an Excel File
https://datatofish.com/export-dataframe-to-excel/

Removing duplicate entries in a csv file using a python script
https://bytes.com/topic/python/answers/738404-removing-duplicate-entries-csv-file-using-python-script

Pandas – Python | Pandas dataframe.drop_duplicates()
https://www.geeksforgeeks.org/python-pandas-dataframe-drop_duplicates/

Joining DataFrames in Pandas
https://www.datacamp.com/community/tutorials/joining-dataframes-pandas

How To Append Rows to a Pandas dataframe using a for loop in Python
https://kite.com/python/answers/how-to-append-rows-to-a-pandas-dataframe-using-a-for-loop-in-python

A Python extension for writing structured and reusable inline HTML.
https://github.com/dropbox/pyxl

Apply and Lambda usage in pandas
https://towardsdatascience.com/apply-and-lambda-usage-in-pandas-b13a1ea037f7

About

This repo is meant to be a place where AWS concepts, documentation, guides, and code can be shared freely. All credit for work is attributed, let me know if I missed something.

Resources

License

Releases

No releases published

Packages

No packages published
You can’t perform that action at this time.