(5/29) NEW GUIDE AVAILABLE ILLUSTRATING EXAMPLES AND WALKTHROUGH OF AWS CLI QUERIES; CHECK OUT AWS_CLI_Queries_Walkthrough.md
Reach out about having a specific topic you’d like to see added; also feel free to edit and add to this list if you’d like. Over time, I’ll refine this list and try to present the most useful references from the links in-page
-
The AWS Code Examples Repository contains sample code for coding solutions in every AWS Service in every SDK available
https://github.com/awsdocs/aws-doc-sdk-examples -
The Official Source of AWS Feature Updates
https://aws.amazon.com/new/?whats-new-content-all.sort-by=item.additionalFields.postDateTime&whats-new-content-all.sort-order=desc -
Query EC2 Instance Attributes such as instanceId, Region, privateIp address, and more using this python module
https://github.com/adamchainz/ec2-metadata -
Run bash scripts on AWS Lambda, also available as a container image
https://github.com/gkrizek/bash-lambda-layer -
Return only the relevant datapoints you need by scripting with the aws-cli and jq
https://medium.com/circuitpeople/aws-cli-with-jq-and-bash-9d54e2eabaf1
https://blog.scottlowe.org/2017/08/15/quick-reference-common-aws-cli-commands/
…as well as SQL
https://github.com/lebinh/aq -
Comprehensive Collection of DevOps Scripts
https://github.com/HariSekhon/DevOps-Bash-tools -
CLI tool that recommends an instance type based on your compute and memory requirements
https://github.com/aws/amazon-ec2-instance-selector -
The verified AWS-Lab repo on Github has 450+ Amazon-approved solutions documented and coded for many common issues devs face
https://github.com/awslabs -
Self-Described ‘practical guide to AWS’; hasn’t been updated in awhile but contains a solid central source of key terms and concepts
https://github.com/open-guides/og-aws -
A curated list of awesome AWS libraries, open source repos, guides, blogs, and other resources. Definitely the gold standard in aws knowledge-sharing
https://github.com/donnemartin/awesome-aws -
Hands on labs and code to help you learn, measure, and build using architectural best practices in AWS
https://github.com/awslabs/aws-well-architected-labs
Using an AWS Service or Tool for the First Time:
-
AWS-CLI (v1, v2)
- Documentation Guide for Installation and Configuration
- Official Github Repo
- CLI Reference
-
Boto3 (Python SDK for AWS)
- Official Github Repo
- API Reference
-
AWS ECS CLI
- Official Github Repo
-
AWS Cloud Deployment Kit (CDK)
- Official Github Repo
-
Official AWS SDK for Go
- Official Github Repo
-
AWS Serverless Application Model (SAM)
- Official Github Repo
Changing the instance type
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-resize.html
Placement groups
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html
aws ec2 modify-instance-placement
https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-placement.html
EC2 Scheduler
https://github.com/amazon-archives/ec2-scheduler/blob/master/documentation/ec2-scheduler.pdf
Working with security groups in Amazon EC2
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/ec2-example-security-group.html
Collection of workshops to demonstrate best practices in using Amazon EC2 Spot Instances
https://github.com/awslabs/ec2-spot-workshops
amazon-ec2-utils contains a set of utilities and settings for Linux deployments in EC2
https://github.com/aws/amazon-ec2-utils
Setting the Time for Your Linux Instance
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html
Amazon Web Services Elastic Compute Cloud (EC2) Rescue for Linux is a python-based tool that allows for the automatic diagnosis of common problems found on EC2 Linux instances.
https://github.com/awslabs/aws-ec2rescue-linux
Listing and Filtering Your Resources
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Filtering.html
EC2 instance metadata proxy for linux containers
https://github.com/dump247/ec2metaproxy
Spot Instances usage in batch jobs
https://github.com/hariboddu/AWS/blob/master/project.py
Playing the AWS ec2 Spot Market
https://medium.com/@mrpowers/playing-the-aws-ec2-spot-market-74b703454f4f
An easy interface to query the EC2 metadata API, with caching.
https://github.com/adamchainz/ec2-metadata
How to prevent cloud-init from disable ssh PasswordAuthentication?
https://ask.openstack.org/en/question/26817/how-to-prevent-cloud-init-from-disable-ssh-passwordauthentication/
How to set “PasswordAuthentication” as yes through automation
https://askubuntu.com/questions/988845/how-to-set-passwordauthentication-as-yes-through-automation
CentOS epel repository listed more than once
https://unix.stackexchange.com/questions/424099/centos-epel-repository-listed-more-than-once
How To: Install Homebrew on Amazon Linux
https://garrettsyhampton.wordpress.com/2019/10/02/how-to-install-homebrew-on-amazon-linux/
Homebrew Formulae – Python 3.8
https://formulae.brew.sh/formula/python@3.8
How to install Python 3.8 on Ubuntu?
https://dev.to/mortoray/how-to-install-python-3-8-on-ubuntu-1bp4
User data to install and run Docker in an Ubuntu AWS instance
http://www.markcortada.com/blogs/nuevasTecnologiasEN/post/2017/05/21/User-data-to-install-and-run-Docker-in-an-Ubuntu-(AWS)-instance.aspx
How to Install Package Updates on Ubuntu via Command-Line
https://tecadmin.net/install-updates-on-ubuntu-via-command-line/
Updating instance software on your Amazon Linux instance
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-updates.html
I want to install a Graphical User Interface (GUI) in my Amazon EC2 instance running Amazon Linux 2. How do I do this?
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-2-install-gui/
How To Set Up GUI On Amazon EC2 Ubuntu server
https://comtechies.com/how-to-set-up-gui-on-amazon-ec2-ubuntu-server.html
How to Recover deleted files in Ubuntu through TestDisk
https://vitux.com/how-to-recover-deleted-files-in-ubuntu-through-testdisk/
Amazon EBS–optimized instances
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-optimized.html
Mapping between EC2 volume and your mounted filesystem
https://unix.stackexchange.com/questions/523839/mapping-between-ec2-volume-and-your-mounted-filesystem
Amazon EBS encryption
https://github.com/awsdocs/amazon-ec2-user-guide/blob/6420aa9d3d85edbe98b15eef07ea433ef3b6a78e/doc_source/EBSEncryption.md
Encrypting unencrypted resources
https://github.com/awsdocs/amazon-ec2-user-guide/blob/6420aa9d3d85edbe98b15eef07ea433ef3b6a78e/doc_source/EBSEncryption.md#encrypt-unencrypted
Using encryption with EBS-backed AMIs
https://github.com/awsdocs/amazon-ec2-user-guide/blob/f8fd01ab75b1f1ab054a7293a6b61157dce26ba6/doc_source/AMIEncryption.md
Detaching an Amazon EBS volume from a Linux instance
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-detaching-volume.html
Amazon EBS fast snapshot restore
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-fast-snapshot-restore.html
EC2 Instance Metadata Query Tool – A simple bash script that uses curl to query the EC2 instance Metadata from within a running EC2 instance.
https://aws.amazon.com/code/ec2-instance-metadata-query-tool/
Amazon EBS volumes: How to Shrink ’em Down to Size
https://cloudacademy.com/blog/amazon-ebs-shink-volume/
How to downsize a root EBS volume on AWS EC2 (Amazon Linux)
https://medium.com/@andtrott/how-to-downsize-a-root-ebs-volume-on-aws-ec2-amazon-linux-727c00148f61
How to reduce volume of an AWS EC2 instance
https://medium.com/@rahulian/how-to-reduce-volume-of-an-aws-ec2-instance-d493ec891698
Decrease the size of EBS volume in your EC2 instance
https://medium.com/@m.yunan.helmy/decrease-the-size-of-ebs-volume-in-your-ec2-instance-ea326e951bce
How to SHRINK an Amazon AWS EBS volume safely with no data loss
http://thehunk.blogspot.com/2017/04/how-to-shrink-amazon-aws-ebs-volume.html
Extending a Linux file system after resizing a volume
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html
How to encrypt EBS volumes of a running EC2 instance?
https://hackernoon.com/how-to-encrypt-ebs-volumes-of-a-running-ec2-instance-a2236b0b45
How to get latest Snapshot for a volume in AWS using API
https://stackoverflow.com/questions/55045588/how-to-get-latest-snapshot-for-a-volume-in-aws-using-api
how to get the latest aws volume snapshot id number using python/boto?
https://stackoverflow.com/questions/28221043/how-to-get-the-latest-aws-volume-snapshot-id-using-python-boto
Lambda for snapshotting EBS, only keeping latest snapshot
https://github.com/kkleidal/latest-snapshot-ebs/blob/master/backup-ebs-volumes.py
Encrypt EBS volumes from AWS EC2 instances
https://github.com/jbrt/ec2cryptomatic
Encrypt EBS volumes from AWS EC2 instances with a serverless StepFunctions machine
https://github.com/jbrt/ec2cryptomatic-serverless
ec2_encrypt_ebs
https://github.com/ffturan/ec2_encrypt_ebs/blob/e2bf95bd4a740db599f1df3be4546dcde22a37d8/README.md
Iterate through each attached volume and encrypt it for EC2 – root
https://github.com/indrahrp/s4misc/blob/375c07f8f53467840b35808b4e6a3facc8bb4390/ec2_root_encryption1.py
Iterate through each attached volume and encrypt it for EC2. – one instance
https://github.com/indrahrp/s4misc/blob/375c07f8f53467840b35808b4e6a3facc8bb4390/encrypt-oneinstance.py
Tools to manage AWS EBS volumes and snapshots
https://github.com/juliogonzalez/ebs-tools
Serverless Backup for your EBS Volumes on AWS, using AWS Lambda
https://github.com/devlinrcg/ebs-serverless-backup
Tag EBS Volumes with Names Similar to the hosts they’re attached too
https://github.com/undeadops/ebsTagger
I do not have permission to access files on the mounted volume
http://nfs.sourceforge.net/nfs-howto/ar01s07.html#pemission_issues
AWS Key Management Service Best Practices (Whitepaper)
https://d0.awsstatic.com/whitepapers/aws-kms-best-practices.pdf
Share Custom Encryption Keys More Securely Between Accounts by Using AWS Key Management Service
https://aws.amazon.com/blogs/security/share-custom-encryption-keys-more-securely-between-accounts-by-using-aws-key-management-service/
How do I share my customer master keys (CMKs) across multiple AWS accounts?
https://aws.amazon.com/premiumsupport/knowledge-center/share-cmk-account/
Allowing users in other accounts to use a CMK
https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-modifying-external-accounts.html
AWS KMS API permissions: Actions and resources reference
https://docs.aws.amazon.com/kms/latest/developerguide/kms-api-permissions-reference.html
The open source version of the AWS Systems Manager user guide. You can submit feedback & requests for changes by submitting issues in this repo or by making proposed changes & submitting a pull request.
https://github.com/awsdocs/aws-systems-manager-user-guide
How do I install AWS Systems Manager Agent (SSM Agent) on an Amazon EC2 Linux instance at launch?
https://aws.amazon.com/premiumsupport/knowledge-center/install-ssm-agent-ec2-linux/
The AWS-UpdateWindowsAmi document enables you to automate image maintenance tasks on your Amazon Windows AMIs without having to author the workflow in JSON or YAML.
https://github.com/awsdocs/aws-systems-manager-user-guide/blob/master/doc_source/automation-walk-patch-windows-ami-cli.md
How patches are installed
https://docs.aws.amazon.com/systems-manager/latest/userguide/patch-manager-how-it-works-installation.html
Query for the latest Amazon Linux AMI IDs using AWS Systems Manager Parameter Store
https://aws.amazon.com/blogs/compute/query-for-the-latest-amazon-linux-ami-ids-using-aws-systems-manager-parameter-store/
A utility to convert your Python or Bash programs into SSM documents (and optionally - CloudFormation templates, containing those documents).
https://github.com/awslabs/aws-systems-manager-document-generator
Automation system variables
https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-variables.html
Systems Manager Automation actions reference
https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-actions.html
Running Automation workflows in multiple AWS Regions and accounts
Running an Automation workflow by using delegated administration
https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-walk-security-delegated.html
Attach an IAM instance profile to an EC2 instance
https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-launch-managed-instance.html
Terraform module to automate patching of AMI templated EC2 Linux instances in a specify schedule
https://github.com/osodevops/aws-terraform-module-ssm-automated-patching
A command line utility enabling SSH-like interactive sessions and continuous output for long running commands via the AWS SSM API.
https://github.com/node40/smsh
Requirements
https://docs.aws.amazon.com/storagegateway/latest/userguide/Requirements.html
Performance
https://docs.aws.amazon.com/storagegateway/latest/userguide/Performance.html
awslabs – aws-cloudformation-templates/aws/services/CloudFormation/HostnameChangeRHEL-Userdata.template
https://github.com/awslabs/aws-cloudformation-templates/blob/master/aws/services/CloudFormation/HostnameChangeRHEL-Userdata.template
Creating AWS File Gateway as an NFS storage for your EC2 instance.
https://medium.com/tensult/creating-aws-file-gateway-as-an-nfs-storage-for-your-ec2-instance-57c141c76409
Very very slow performance…
https://forums.aws.amazon.com/thread.jspa?threadID=250901
FUSE-based file system backed by Amazon S3
https://github.com/s3fs-fuse/s3fs-fuse
Switching to an IAM Role (AWS API)
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-api.html
How can I configure a Lambda function to assume a role from another AWS account?
https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-assume-iam-role/
Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html
A tool for quickly evaluating IAM permissions in AWS.
https://github.com/nccgroup/PMapper
Awsume is a convenient way to manage session tokens and assume role credentials.
https://awsu.me/
AWS Auto Scaling based on Memory Utilization in CloudFormation
https://medium.com/@lvthillo/aws-auto-scaling-based-on-memory-utilization-in-cloudformation-159676b6f4d6
What is Amazon Elastic Load Balancer (ELB)
https://hackernoon.com/what-is-amazon-elastic-load-balancer-elb-16cdcedbd485
Viewing Available Metrics
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/viewing_metrics_with_cloudwatch.html
Why isn’t my CloudWatch GetMetricStatistics API call returning data points?
https://aws.amazon.com/premiumsupport/knowledge-center/cloudwatch-getmetricstatistics-data/
Ingesting High-Cardinality Logs and Generating Metrics with CloudWatch Embedded Metric Format
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html
Using Metric Math
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html
Amazon CloudWatch Embedded Metric Format Client Library
https://github.com/awslabs/aws-embedded-metrics-python
Probabilistic time series modeling in Python
https://github.com/awslabs/gluon-ts
Push metrics and logs to CloudWatch from any system that can run Python
https://github.com/awslabs/amazon-cloudwatch-publisher
A CLI tool to merge schema files, and validate operations against the merged GraphQL Schema. With features like adding your own custom validation rules.
https://github.com/awslabs/graphql-schema-utilities
Graph Metrics Manually on a CloudWatch Dashboard
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/add_old_metrics_to_graph.html
Amazon CloudWatch Embedded Metric Format Client Library
https://github.com/awslabs/aws-embedded-metrics-python/tree/master/examples
Building an Amazon CloudWatch Dashboard Outside of the AWS Management Console
https://aws.amazon.com/blogs/devops/building-an-amazon-cloudwatch-dashboard-outside-of-the-aws-management-console/
Amazon CloudWatch metrics for Amazon EBS
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cloudwatch_ebs.html
CLOUDWATCH DASHBOARD
https://reinvent2019.aws-management.tools/mgt415/en/cont/3cloudwatch.html
Amazon EBS Metrics
https://github.com/awsdocs/amazon-ec2-user-guide-windows/blob/befad90a6b8c71abbec34335ea0ea4e6dad715e9/doc_source/using_cloudwatch_ebs.md#ebs-volume-metrics
access to custom cloudwatch metrics using lambda+boto3?
https://serverfault.com/questions/824510/access-to-custom-cloudwatch-metrics-using-lambdaboto3
Metrics exporter for Amazon AWS CloudWatch
https://github.com/prometheus/cloudwatch_exporter
[ARCHIVE] DSL for querying against metrics stored in Amazon CloudWatch
https://github.com/fsprojects/zzarchive-Amazon.CloudWatch.Selector
Store custom CloudWatch metrics in a cost-effective way
https://github.com/alpha-lambda/cloudwatch-telemetry
USING CLOUDWATCH TO MONITOR AWS S3 BUCKETS
https://www.opsdash.com/blog/aws-s3-cloudwatch-monitoring.html
Redash is our take on freeing the data within our company in a way that will better fit our culture and usage patterns.
https://github.com/Dorpolo/redash/tree/f09179178a8f3dba2297a50cab723b59ccbe1040
Zabbix Scripts & templates for monitoring AWS Services
https://github.com/Enrise/zabbix-cloudwatch/blob/ee571260055c6c8d2be43e532c929a2d005c7e44/externalscripts/cloudfront_stats.py
AWS CloudWatch Fluent Metrics
https://pypi.org/project/cloudwatch-fluent-metrics/
Start up script used to create Cloudwatch monitoring and alarms for RAM.
https://github.com/1Strategy/ram-monitoring-script
A Dockerized version of Ice, Netflix’s AWS Usage tool.
https://github.com/jonbrouse/docker-ice
AWS Cloudwatch to Spreadsheet
https://github.com/petezybrick/awscwxls
The Node module to draw charts for AWS CloudWatch metrics
https://github.com/jeka-kiselyov/aws-cloudwatch-chart
CloudWatch to AWS Lambda to Slack Channel Alerts and Charts. Posts CloudWatch Alerts via SNS topic via Lambda function to Slack channel. And draws charts for CPUUtilization metrics(both for EC2 abd RDS).
https://gist.github.com/jeka-kiselyov/3f3801a165cab9e4a9fd
Just dump all the CloudWatch metrics.
https://github.com/mogproject/cloudwatch-dump
Queries AWS for metrics of running EC2 instances.
https://github.com/kgryte/aws-ec2-metrics
An API to export Cloudwatch Metrics to Prometheus exposition format
https://github.com/marcelocrnunes/cwexporter/tree/5101e26e16328c92959e32e2a058210195c8d8ac
Allowing to display your AWS Cloudwatch dashboards on your website or locally
https://github.com/Dlas1212/AWSCloudwatchExternalDashboards
Bring Amazon AWS Cloudwatch metrics into Graphite
https://github.com/edasque/cloudwatch2graphite
Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.
https://github.com/eric-erki/Make-Your-Company-Data-Driven./tree/944adb95ba29bf241fa9a0af9a80734f3afbb3e3
Send metrics from AWS Cloudwatch to Sumo Logic
https://github.com/gadouryd/AWS-Scripts/tree/1df2e439f300c821ad19d302ec56c438580c34dd
Extremely Useful AWS CLI shell scripts to be used with CloudWatch
https://github.com/mikeoscarson/aswadmin/tree/a2fea10904a841221630ad4a0a7bedd7d91d6d8f/cloudwatchmetrics
20+ Grafana dashboards for AWS CloudWatch metrics: EC2, Lambda, S3, ELB, EMR, EBS, SNS, SES, SQS, RDS, EFS, ElastiCache, Billing, API Gateway, VPN, Step Functions, Route 53, CodeBuild, …
https://github.com/monitoringartist/grafana-aws-cloudwatch-dashboards
Lambda to create cloudwatch metric filters, custom metrics and dashboard
https://github.com/nikhilswagle/cloudwatch-monitoring-lambda
Importing Data into Oracle on Amazon RDS
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Procedural.Importing.html
Using an Oracle Database as a Source for AWS DMS
https://github.com/awsdocs/aws-dms-user-guide/blob/master/doc_source/CHAP_Source.Oracle.md
Best practices for migrating an Oracle database to Amazon RDS PostgreSQL or Amazon Aurora PostgreSQL: Source database considerations for the Oracle and AWS DMS CDC environment
https://aws.amazon.com/blogs/database/best-practices-for-migrating-an-oracle-database-to-amazon-rds-postgresql-or-amazon-aurora-postgresql-source-database-considerations-for-the-oracle-and-aws-dms-cdc-environment/
How can I troubleshoot AWS DMS endpoint connectivity failures?
https://aws.amazon.com/premiumsupport/knowledge-center/dms-endpoint-connectivity-failures/
Create AWS CloudFormation templates for AWS DMS tasks using Microsoft Excel
https://aws.amazon.com/blogs/database/create-aws-cloudformation-templates-for-aws-dms-tasks-using-microsoft-excel/
Using AWS Data Migration Service with RDS
https://linuxacademy.com/guide/13043-using-aws-data-migration-service-with-rds/
Creates an example AWS DMS for replicating an (on-prem) Oracle database to a cloud-based Postgres database
https://github.com/GeoscienceAustralia/aws-dms-example
Sample code that reads Microsoft Excel workbook/CSV File for the details required to create a DMS task CloudFormation template
https://github.com/aws-samples/dms-cloudformation-templates-generator/tree/v1.0
Why am I getting Access Denied errors on ListObjects actions when I run a sync command on my Amazon S3 bucket?
https://aws.amazon.com/premiumsupport/knowledge-center/s3-access-denied-listobjects-sync/
Print usage stats of S3 buckets
https://github.com/marpada/s3_stats/blob/master/s3_stats.py
Amazon QuickSight Python3 SDK
https://docs.aws.amazon.com/quicksight/latest/user/quicksight-sdk-python.html
Advanced analytics with table calculations in Amazon QuickSight
https://aws.amazon.com/blogs/big-data/advanced-analytics-with-table-calculations-in-amazon-quicksight/
Python3 Example of Embedding a Dashboard with Amazon QuickSight
https://github.com/awsdocs/amazon-quicksight-user-guide/blob/master/doc_source/embedded-dashboards-example-python.md
A QuickSight dashboard embedding sample for web apps.
https://github.com/aws-samples/amazon-quicksight-embedding-sample
ARN Formats
https://docs.aws.amazon.com/quicksight/latest/APIReference/qs-arn-format.html
Amazon QuickSight Embedded Analytics
https://aws.amazon.com/quicksight/features-embedding/
Evolve your analytics with Amazon QuickSight’s new APIs and theming capabilities
https://aws.amazon.com/blogs/big-data/evolve-your-analytics-with-amazon-quicksights-new-apis-and-theming-capabilities/
AWS Quicksight - Quick Guide
https://www.tutorialspoint.com/aws_quicksight/aws_quicksight_quick_guide.htm
Tools that make it easier to analyze AWS Cost and Usage reports. Initial version provides support for Athena and QuickSight.
https://github.com/concurrencylabs/aws-cost-analysis
Amazon QuickSight, Amazon Athena and Amazon Spectrum workshop. Workshop will focus on ingesting data into Athena & Spectrum, combining it with other data sources, and visualizing it in QuickSight.
https://github.com/avinashnid/QuicksightAthenaSpectrum
The objective of the project is to build a data pipeline which helps in extracting insights from the data with faster turnaround time.
https://github.com/Nikhilgupta1891/AWS-Lambda-Athena-QuickSight
Python3 Example for Getting the Dashboard URL from the App Server
https://docs.aws.amazon.com/quicksight/latest/user/embedded-dashboards-example-python.html
HTML Example for Embedding a Dashboard with Amazon QuickSight Using the URL from the App Server
https://docs.aws.amazon.com/quicksight/latest/user/embedded-dashboards-example-html.html
Find Matches Examples
https://github.com/aws-samples/aws-glue-samples/tree/master/findmatches
FAQ for AWS Glue
https://github.com/aws-samples/aws-glue-samples/blob/master/FAQ_and_How_to.md
Utilities for AWSGlue
https://github.com/aws-samples/aws-glue-samples/tree/master/utilities
Samples of AWS Glue Implementation
https://github.com/aws-samples/aws-glue-samples
Dynamic Dataframes
https://github.com/aws-samples/aws-glue-samples/blob/master/examples/data_cleaning_and_lambda.md
The Amazon ECS CLI enables users to create and manage their ECS projects, from source code, infrastructure to operations.
https://github.com/aws/amazon-ecs-cli-v2
Docker Basics for Amazon ECS
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html
The role of AWS Fargate in the container world
https://aws.amazon.com/blogs/containers/the-role-of-aws-fargate-in-the-container-world/
Full IAC to build jenkins master (AMI, master & slave docker image), slave cluster in different regions on EC2, ECS EC2 & fargate on AWS
https://github.com/linhkikuchi/aws-jenkins-cluster-ecs-fargate
Fargate CLI
https://somanymachines.com/fargate/
How to use AWS Fargate and Lambda for long-running processes in a Serverless app
https://www.serverless.com/blog/serverless-application-for-long-running-process-fargate-lambda/
AWS Fargate Labs
https://github.com/koundykarthik/stylelabs
Deploy serverless containers to the cloud from your command line (Go-based Fargate CLI)
https://github.com/turnerlabs/fargate
Copies all the stuff in account, region “a” to account, region “b” in regards to ECR. A quick lift and shift deal.
https://github.com/DarklyAutumnClad/CopyAWSCLIECRRepositoriesCrossAccountCrossRegion
Testing ground for container orchestration on AWS
https://github.com/angelokurtis/amazon-ecs-labs
Best Practices
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/best-practices.html
The cfn-init helper script reads template metadata from the AWS::CloudFormation::Init key and acts accordingly to, Fetch and parse metadata from AWS CloudFormation, Install packages, Write files to disk, Enable/disable and start/stop services
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-init.html
Templates exemplifying the conditional creation of resources based on environment
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-sample-templates.html
Run arbitrary python code in your CloudFormation templates
https://github.com/aws-cloudformation/aws-cloudformation-macros/tree/eef5948e26104d7a7e81c1be9d1ed336c79940c1/PyPlate
A collection of useful CloudFormation templates
https://github.com/awslabs/aws-cloudformation-templates
Blank AWS Lambda Function with Python Runtime
https://github.com/awsdocs/aws-lambda-developer-guide/tree/master/sample-apps/blank-python
Get AWS account id inside Lambda function programmatically - python
https://gist.github.com/aws-scripting-guy/83d901527fa7adc271da
Run Bash scripts in AWS Lambda via Layers
https://github.com/gkrizek/bash-lambda-layer
The AWS Lambda container image converter tool (img2lambda) repackages container images (such as Docker images) into AWS Lambda function deployment packages and Lambda layers.
https://github.com/awslabs/aws-lambda-container-image-converter
A curated list of awesome tools for Amazon EKS
https://github.com/realvz/awesome-eks
The AWS Deployment Framework (ADF) is an extensive and flexible framework to manage and deploy resources across multiple AWS accounts and regions based on AWS Organizations.
https://github.com/awslabs/aws-deployment-framework
Create CloudFormation templates from existing infrastructure – this version of the CloudFormer solution can be used to launch the solution into an existing VPC.
https://github.com/awslabs/aws-support-tools/tree/master/CloudFormation/CloudFormer
Serverless app designed for any customer with two or more accounts to view resources across accounts/regions in simple single pane of glass website
https://github.com/awslabs/aws-multi-account-viewer
AWS-SDK-IOS-Samples
https://github.com/awslabs/aws-sdk-ios-samples
Imputation of missing values in tables.
https://github.com/awslabs/datawig
A solution which enables a multilingual conversational platform, facilitating the creation of applications using voice and text
https://github.com/awslabs/serverless-bot-framework
This repo contains a simple application that consists of three microservices. Each application is deployed using different Compute options on AWS.
https://github.com/aws-samples/aws-microservices-deploy-options
The AWS CDK is a new software development framework from AWS with the sole purpose of making it fun and easy to define cloud infrastructure in your favorite programming language and deploy it using AWS CloudFormation.
https://cdkworkshop.com/
Official AWS CDK Roadmap
https://github.com/orgs/aws/projects/7
Collection of AWS Scripts
https://github.com/marcomoscatiello/aws_scripts
A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.
https://github.com/donnemartin/awesome-aws
Rich Morrow’s Personal Compilation of Helpful AWS Links
http://aws-materials.s3.amazonaws.com/index.html
Use Boto3 to Assume a Role in another AWS Account
https://www.slsmk.com/use-boto3-to-assume-a-role-in-another-aws-account/
Building Beautiful Command Line Interfaces with Python
https://codeburst.io/building-beautiful-command-line-interfaces-with-python-26c7e1bb54df
All Datasets used as part of 538.com articles – good repo to read sample data from
https://github.com/fivethirtyeight/data
This script will process cloudtrail records to create a sqlite db of resources and their creators, and then use that sqlitedb to tag the resources with their creator’s name.
https://cloudcustodian.io/docs/tools/c7n-trailcreator.html
A tool to load bulk simulation data into AWS IoT Analytics
https://pypi.org/project/onica-iotloader/
This is a set of boilerplate scripts describing the normalized script pattern that GitHub uses in its projects. While these patterns can work for projects based on any framework or language, these particular examples are for a simple Ruby on Rails application.
https://github.com/github/scripts-to-rule-them-all
On this lab we will convert our infrastructure into code by using CloudFormation to stand up the same stack from last week.
https://github.com/coolx28/DevSecOps/blob/c18f57b7ecbbff7e442b5ab34913b3b279da22ed/Week-5/labs/LAB-1.md
AI and Machine Learning with Kubeflow, Amazon EKS, and SageMaker
https://github.com/data-science-on-aws/workshop
amplify-chime-react
https://github.com/mats16/amplify-chime-react/tree/f1a19708f13246efa318583dec72a15ad5c93c86
Boto and LibCloud to automate AWS and OpenStack related tasks (CloudWatch/AutoScaling/Glacier/Volumes/EC2/S3/SNS/etc)
https://github.com/rsd-raul/Cloud-Computing-Project/tree/634514e55827830aae25f1a4460750f9a3e01482
awstools
https://github.com/iMilnb/awstools
Devops_wordpress_Notes
https://github.com/sandeepmchary/Devops_wordpress_Notes/blob/1d4603ac98b7b47b681967f32508a8c91ac38360/Udemy/Aws-automation-with-boto3/Session-4/Sesssion-4.txt
List of open source tools for AWS security: defensive, offensive, auditing, DFIR, etc.
https://github.com/osodevops/my-arsenal-of-aws-security-tools
A repository of useful functions (offensive & defensive) to interact with AWS services
https://github.com/carnal0wnage/weirdAAL/wiki
Project to manage AWS Organization using boto3
https://github.com/osodevops/aws-orgs
Cartography is a Python tool that consolidates infrastructure assets and the relationships between them in an intuitive graph view powered by a Neo4j database.
https://github.com/lyft/cartography
Rules engine for cloud security, cost optimization, and governance, DSL in yaml for policies to query, filter, and take actions on resources
https://github.com/cloud-custodian/cloud-custodian/
Cloud Custodian Policies
https://github.com/kvskmech/Cloud-Custodian-Policies/tree/ac5390a353db654e348a12b5b6d4300967168484/policies
Awesome list dedicated to Windows Subsystem for Linux
https://github.com/sirredbeard/Awesome-WSL#wsl-tools
Amazon Linux as a WSL Instance. Supports multiple install.
https://github.com/yosukes-dev/AmazonWSL
Setting up Ubuntu (WSL) for Linux GUI Apps
https://x410.dev/cookbook/wsl/setting-up-wsl-for-linux-gui-apps/
Targeting the Windows Subsystem for Linux from Visual Studio
https://devblogs.microsoft.com/cppblog/targeting-windows-subsystem-for-linux-from-visual-studio/
Big O Cheatsheet
https://www.bigocheatsheet.com/
Data Structures
https://sinon.org/algorithms//#data-structures
Environment variables to configure the AWS CLI
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
Getting credentials from EC2 instance metadata
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-metadata.html
AWS CLI Cheatsheet
https://gist.github.com/apolloclark/b3f60c1f68aa972d324b
How to run a script or a command on EC2 instance via AWS CLI?
https://devops.stackexchange.com/questions/3264/how-to-run-a-script-or-a-command-on-ec2-instance-via-aws-cli
AWS-Python-Utilities
https://github.com/hms-dbmi/aws-python-utilities
yq is a lightweight and portable command-line YAML processor
https://mikefarah.gitbook.io/yq/
AWS CLI v2 Docker image
https://aws.amazon.com/blogs/developer/aws-cli-v2-docker-image/
AWS CLI with jq and Bash – The CLI is utilitarian, but a little jq sauce makes it beautiful
https://medium.com/circuitpeople/aws-cli-with-jq-and-bash-9d54e2eabaf1
AWS CLI with jq make things easy
https://www.lewuathe.com/aws-cli-with-jq-make-things-easy.html
jq in-browser test queries
https://jqplay.org/
AWS Lambda Layer for AWS CLI
https://github.com/aws-samples/aws-lambda-layer-awscli
Querying Instances using Filter on Tag Values
https://gist.github.com/zharling/0c1049dc362a17d1bd0ef0c1d2fabec8
Useful AWS command line tools using JMESPath and jq
https://gist.github.com/micklove/76b2cb13ea4d86f4b5f4ec42b0f9939b
AWS-CLI Cheatsheet
https://gist.github.com/davidmoremad/6db1981d37ed6b6481a29b91929a4fa4
AWS-CLI commands(Region, AZ, IAM, EC2)
https://gist.github.com/nvogel/21a36a9a44548aa2835b2ea9ffb8032e
AWS-CLI commands examples (Autoscaling and EC2)
https://gist.github.com/namuan/4d612f75e3d757935fe60aa530de7aa6
Query AWS resources with SQL
https://github.com/lebinh/aq
Quick Reference to Common AWS CLI Commands
https://blog.scottlowe.org/2017/08/15/quick-reference-common-aws-cli-commands/
Controlling command output from the AWS CLI
https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.html
How can I list Amazon EBS volume or snapshot information for my Amazon EC2 instance by using the AWS CLI?
https://aws.amazon.com/premiumsupport/knowledge-center/ebs-volume-snapshot-ec2-instance/
awsq provides a way to query and filter data from aws ec2 describe-instances
https://github.com/drewkhoury/awsq
JMESPath exploration tool in the terminal
https://github.com/jmespath/jmespath.terminal
Bash Scripting Tutorial - 5. If Statements
https://ryanstutorials.net/bash-scripting-tutorial/bash-if-statements.php
Passing Arguments to Bash Functions
https://linuxize.com/post/bash-functions/#passing-arguments-to-bash-functions
Script to fetch an MFA token for you to use temporary aws access credentials
https://github.com/asagage/aws-mfa-script
AWK command in Unix/Linux with examples
https://www.geeksforgeeks.org/awk-command-unixlinux-examples/
Bash Read Comma Separated CSV File on Linux / Unix
https://www.cyberciti.biz/faq/unix-linux-bash-read-comma-separated-cvsfile/
How to do integer & float calculations, in bash or other languages/frameworks?
https://unix.stackexchange.com/questions/40786/how-to-do-integer-float-calculations-in-bash-or-other-languages-frameworks
Is there any way to copy all lines from open file to clipboard in VI editor?
https://stackoverflow.com/questions/1620018/copy-all-the-lines-to-clipboard
How can I preview HTML documents from the command line?
https://askubuntu.com/questions/58416/how-can-i-preview-html-documents-from-the-command-line
Bash script to easily set shell environment with temporary access tokens using a config file and the AWS credentials file
https://github.com/rik2803/aws-sts-assumerole
DevOps Bash Tools
https://github.com/HariSekhon/DevOps-Bash-tools
The GNU Awk User’s Guide
https://www.gnu.org/software/gawk/manual/gawk.html
A CLI tool and go library which recommends instance types based on resource criteria like vcpus and memory
https://github.com/aws/amazon-ec2-instance-selector
Open Container Initiative – Creating open standards around container technology
https://github.com/opencontainers
This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://github.com/aws/containers-roadmap
HOW TO KEEP DOCKER CONTAINERS RUNNING
http://bigdatums.net/2017/11/07/how-to-keep-docker-containers-running/
Docker Certified Associate
https://www.whizlabs.com/learn/course/docker-certified-associate/
Simple Docker image with the AWS CLI
https://github.com/mikesir87/aws-cli-docker
How To Install and Use Docker on Ubuntu 18.04
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04
User data to install and run Docker in an Ubuntu AWS instance
http://www.markcortada.com/blogs/nuevasTecnologiasEN/post/2017/05/21/User-data-to-install-and-run-Docker-in-an-Ubuntu-(AWS)-instance.aspx
Examples using the Docker Engine SDKs and Docker API
https://docs.docker.com/engine/api/sdk/examples/
docker build
https://docs.docker.com/engine/reference/commandline/build/
How To SSH Into A Running Docker Container And Run Commands
https://phoenixnap.com/kb/how-to-ssh-into-docker-container
Dockerize your Python Application
https://runnable.com/docker/python/dockerize-your-python-application
This project provides source code of example applications for people to get started using AWS Serverless Application Repository.
https://github.com/aws-samples/serverless-app-examples/blob/master/python/hello-world-python3/lambda_function.py
How to containerize your legacy apps without sweating
https://www.cio.com/article/3191502/how-to-containerize-your-legacy-apps-without-sweating.html
What is Podman? Podman is a daemonless container engine for developing, managing, and running OCI Containers on your Linux System. Containers can either be run as root or in rootless mode. Simply put: alias docker=podman.
https://podman.io/
Podman Installation Instructions
https://podman.io/getting-started/installation
How to run Podman on Windows with WSL2
https://www.redhat.com/sysadmin/podman-windows-wsl2
libpod is a library used to create container pods. Home of Podman.
https://github.com/containers/libpod
Configuring container networking with Podman
https://www.redhat.com/sysadmin/container-networking-podman
containerd is available as a daemon for Linux and Windows. It manages the complete container lifecycle of its host system, from image transfer and storage to container execution and supervision to low-level storage to network attachments and beyond.
https://containerd.io/
Building, Running, and Managing Containers with RHEL8
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/building_running_and_managing_containers/index?extIdCarryOver=true&sc_cid=701f2000001OH7JAAW#using_standard_red_hat_base_images
Unprivileged Sandbox Tool
https://github.com/containers/bubblewrap
A quick guide to podman and toolbox in Debian (and maybe Ubuntu)
https://dev.to/bureado/a-quick-guide-to-podman-and-toolbox-in-debian-5672
defaultdict
https://docs.python.org/3/library/collections.html#defaultdict-examples
comp3321 NSA Python Training Document
https://archive.org/details/comp3321/page/n387/mode/2up/search/device
Leetcode Algorithms
https://github.com/JBielan/leetcode-py-js/tree/master/python
Number Of Islands Tutorial (Python)
https://colorfulcodesblog.wordpress.com/2018/09/06/number-of-islands-tutorial-python/
heapq — Heap queue algorithm
https://docs.python.org/3.8/library/heapq.html
Python Sorting Algorithms
https://www.tutorialspoint.com/python_data_structure/python_sorting_algorithms.htm
Stop pip from failing on single package when installing with requirements.txt
https://stackoverflow.com/questions/22250483/stop-pip-from-failing-on-single-package-when-installing-with-requirements-txt
An easy interface to query the EC2 metadata API, with caching.
https://pypi.org/project/ec2-metadata/
Parse ARNs using Python
https://pypi.org/project/arnparse/
Wrap a WSGI application in an AWS Lambda handler function for running on API Gateway or an ALB.
https://pypi.org/project/apig-wsgi/
A Simple and Scalable Logging Utility With a Beautiful Visualization Dashboard
https://pypi.org/project/ml-logger/
bunker is a command line program for setting up an ec2 in AWS for remote development or as a backup. It can clone your git repos, and transfer ignored files from your machine to the ec2.
https://pypi.org/project/aws-bunker/
A collection of useful decorators for making AWS Lambda handlers
https://pypi.org/project/lambda-decorators/
Async client for aws services using botocore and aiohttp
https://pypi.org/project/aiobotocore/
Async AWS SDK for Python
https://aioboto3.readthedocs.io/en/latest/usage.html
Resources – Boto3
https://github.com/boto/boto3/blob/develop/docs/source/guide/resources.rst
boto3 quick hands-on
https://gist.github.com/itorres/0ce27bc1bbfe17f863f2559602a3bc1c
Pandas on AWS
https://github.com/awslabs/aws-data-wrangler
Migration from Boto2.x to Boto3
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/migrationec2.html
A basic boto3 based tool for retrieving information from running EC2 instances.
https://gist.github.com/dastergon/b4994c605f76d528d0c4
Manipulate and parse data in Python using dataframes, also includes info on handling csv and Excel files (.xlsx, .xlsm)
xlsxwriter – Adjust excel reference cells in loop when creating formula
https://stackoverflow.com/questions/25183305/python-xlsxwriter-adjust-excel-reference-cells-in-loop-when-creating-formula
openpyxl – Accessing many cells; Ranges of cells can be accessed using slicing:
https://openpyxl.readthedocs.io/en/latest/tutorial.html#accessing-many-cells
openpyxl – Sheet: Data Access
http://docs.pyexcel.org/en/latest/tutorial.html
Examples of automation of excel via python, and related useful things
https://github.com/richard512/python-Excel-Automation
Ingest Excel data automatically into Amazon QuickSight
https://aws.amazon.com/blogs/big-data/ingest-excel-data-automatically-into-amazon-quicksight/
How to Export Pandas DataFrame to an Excel File
https://datatofish.com/export-dataframe-to-excel/
Removing duplicate entries in a csv file using a python script
https://bytes.com/topic/python/answers/738404-removing-duplicate-entries-csv-file-using-python-script
Pandas – Python | Pandas dataframe.drop_duplicates()
https://www.geeksforgeeks.org/python-pandas-dataframe-drop_duplicates/
Joining DataFrames in Pandas
https://www.datacamp.com/community/tutorials/joining-dataframes-pandas
How To Append Rows to a Pandas dataframe using a for loop in Python
https://kite.com/python/answers/how-to-append-rows-to-a-pandas-dataframe-using-a-for-loop-in-python
A Python extension for writing structured and reusable inline HTML.
https://github.com/dropbox/pyxl
Apply and Lambda usage in pandas
https://towardsdatascience.com/apply-and-lambda-usage-in-pandas-b13a1ea037f7