Skip to content

Commit

Permalink
only modify site and readme (#2512)
Browse files Browse the repository at this point in the history
  • Loading branch information
depombo committed Jul 10, 2023
1 parent e67cbfb commit bbeb653
Show file tree
Hide file tree
Showing 19 changed files with 63 additions and 123 deletions.
10 changes: 5 additions & 5 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ Common tests are inside `test/common/` directory. This tests run sequentially an

### Modules

Modules tests are inside `test/modules/` directory. Currently, every module has a dedicated test account with credentials given via Github Actions secrets and every test run picks an AWS regions within that test account via the [CI setup](https://github.com/iasql/iasql/blob/main/.github/workflows/ci.yml#L84). Modules tests run in parallel with each other but sequentially within each module file and each use a specific account per module to avoid rate limits per account. To add a new test:
Modules tests are inside `test/modules/` directory. Currently, every module has a dedicated test account with credentials given via Github Actions secrets and every test run picks an AWS regions within that test account via the [CI setup](https://github.com/alantech/iasql/blob/main/.github/workflows/ci.yml#L84). Modules tests run in parallel with each other but sequentially within each module file and each use a specific account per module to avoid rate limits per account. To add a new test:

- Create the test file inside `test/modules`. The current pattern to name the file is `aws-<aws-service>-integration`.
- Run the tests. It will parallelize the test runs by file.
Expand All @@ -163,15 +163,15 @@ Modules tests are inside `test/modules/` directory. Currently, every module has

### 1. Release the newest version

Go to ["Version: Release Newest" tab](https://github.com/iasql/iasql/actions/workflows/release.yml) in the repository Actions tab. Then click the "Run Workflow" button to mark the newest version as released.
Go to ["Version: Release Newest" tab](https://github.com/alantech/iasql/actions/workflows/release.yml) in the repository Actions tab. Then click the "Run Workflow" button to mark the newest version as released.

### 2. Deploy Production

Go to ["Deploy to Production" tab](https://github.com/iasql/iasql/actions/workflows/production.yml) in the repository Actions tab. Next click the "Run Workflow" button. Then type in the new version number to deploy and finally click the green "Run workflow" button.
Go to ["Deploy to Production" tab](https://github.com/alantech/iasql/actions/workflows/production.yml) in the repository Actions tab. Next click the "Run Workflow" button. Then type in the new version number to deploy and finally click the green "Run workflow" button.

### 3. Begin development of the next version

Go to ["Version: Start New Development" tab](https://github.com/iasql/iasql/actions/workflows/release-develop.yml) in the repository Actions tab. Next click the "Run Workflow" button. Then type in the new version number for the next release and finally click the green "Run workflow" button. It will create a commit with that version with a `-beta` appended to it to make it clear this and following commits are not for users to work with directly.
Go to ["Version: Start New Development" tab](https://github.com/alantech/iasql/actions/workflows/release-develop.yml) in the repository Actions tab. Next click the "Run Workflow" button. Then type in the new version number for the next release and finally click the green "Run workflow" button. It will create a commit with that version with a `-beta` appended to it to make it clear this and following commits are not for users to work with directly.

### 4. Develop!

Expand Down Expand Up @@ -217,7 +217,7 @@ This method is the same as commit, with the difference that it runs with a `dryR

### Read-Diff-Execute Loop (apply and sync)

The [`apply`](https://github.com/iasql/iasql/blob/main/src/modules/iasql_functions/iasql.ts#L859) and [`sync`](https://github.com/iasql/iasql/blob/main/src/modules/iasql_functions/iasql.ts#L1098) functions are very similar dual `do-while` loops that execute the following steps:
The [`apply`](https://github.com/alantech/iasql/blob/main/src/modules/iasql_functions/iasql.ts#L859) and [`sync`](https://github.com/alantech/iasql/blob/main/src/modules/iasql_functions/iasql.ts#L1098) functions are very similar dual `do-while` loops that execute the following steps:

1. For every `mapper` provided (it will depend if we are in the first apply phase of the commit where we only have the mappers of the modules that have changes or the second where we will have all of them), load all database records into their associated `entities` using the database `read` function from the `mapper`.
2. Recreate all possible changes that might have occurred after the current `iasql_commit` started in order to exclude them during this apply execution.
Expand Down
18 changes: 3 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,7 @@

&nbsp;

[![Integration Tests](https://github.com/iasql/iasql/actions/workflows/tests.yml/badge.svg)](https://github.com/iasql/iasql/actions/workflows/tests.yml)
[![Dashboard Tests](https://github.com/iasql/iasql/actions/workflows/tests-dashboard.yml/badge.svg)](https://github.com/iasql/iasql/actions/workflows/tests-dashboard.yml)
[![Docs](https://img.shields.io/badge/docs-docusaurus-blue)](https://iasql.com/docs)
[![Discord](https://img.shields.io/badge/discord-iasql-purple)](https://discord.com/invite/machGGczea)
[![Twitter](https://img.shields.io/badge/twitter-iasql-9cf)](https://www.twitter.com/iasql)

---

Expand All @@ -30,14 +26,6 @@

To get started with IaSQL using our documentation. Visit [iasql.com/docs](https://iasql.com/docs)

## 💬 Community, Support, and Questions

Reach out on Discord

<a href="https://discord.com/invite/machGGczea">
<img src="https://discord.com/assets/ff41b628a47ef3141164bfedb04fb220.png" height="40px" />
</a>

## 🔨 Use cases

- [Quickly setup ECR+ECS+ELB using our ejectable abstraction for those services](https://iasql.com/blog/ecs-simplified)
Expand All @@ -48,14 +36,14 @@ Reach out on Discord

## ☁️ Cloud Providers

AWS is our main focus at the moment, but we plan to support GCP, Azure and other cloud providers soon. Let us know if you need a specific AWS service and we might be able to prioritize it!
AWS is our main focus at the moment, but we plan to support GCP, Azure, and other cloud providers soon. Let us know if you need a specific AWS service and we might be able to prioritize it!

### AWS coverage

An IaSQL module roughly maps to an AWS service. Check out our supported modules in this [part](https://iasql.com/docs/modules/) of our docs.

AWS services that are going to be added next can be found [here](https://github.com/iasql/iasql/issues?q=is%3Aissue+is%3Aopen+label%3A%22cloud+coverage%22). Let us know if you would like to see one that is not listed!
AWS services that are going to be added next can be found [here](https://github.com/alantech/iasql/issues?q=is%3Aissue+is%3Aopen+label%3A%22cloud+coverage%22). Let us know if you would like to see one that is not listed!

## 🚀 Contribute

To contribute, visit [Contributing.md](https://github.com/iasql/iasql/blob/main/CONTRIBUTING.md)
To contribute, visit [Contributing.md](https://github.com/alantech/iasql/blob/main/CONTRIBUTING.md)
10 changes: 5 additions & 5 deletions postmortems/001 - Postmortem for v0.0.19 Deployment Failure.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@ The `v0.0.19` release could not be deployed despite the test suite succeeding. A
## Timeline

- **2022-09-05**: Release day for `v0.0.19`, rushing to get desired features out the door, but were unable to. Release was delayed rather than skip these features.
- **2022-09-07**: [Actual attempted release for `v0.0.19`](https://github.com/iasql/iasql-engine/tree/v0.0.19), tests are passing on the `main` branch, and manual testing locally did not uncover any issues. There *was* an error in the staging deploy earlier in the day, but it was assumed caused by other experiments. After the `staging` IaSQL database was cleaned up, further staging deploys were successful, so it was assumed that the `production` deployment would be the same. Instead the `production` deploy failed, and an investigation began.
- **2022-09-09**: [Actual root cause found and fixed, and new test written to verify no regressions occur](https://github.com/iasql/iasql-engine/pull/1241). A few bugs unrelated to the root cause in the `default_aws_region` function are also fixed in the process.
- **2022-09-07**: [Actual attempted release for `v0.0.19`](https://github.com/alantech/iasql-engine/tree/v0.0.19), tests are passing on the `main` branch, and manual testing locally did not uncover any issues. There *was* an error in the staging deploy earlier in the day, but it was assumed caused by other experiments. After the `staging` IaSQL database was cleaned up, further staging deploys were successful, so it was assumed that the `production` deployment would be the same. Instead the `production` deploy failed, and an investigation began.
- **2022-09-09**: [Actual root cause found and fixed, and new test written to verify no regressions occur](https://github.com/alantech/iasql-engine/pull/1241). A few bugs unrelated to the root cause in the `default_aws_region` function are also fixed in the process.
- **2022-09-12**: Release day for `v0.0.20`, but being extra cautious and it was also put on hold because of an error in the Prisma Example end-to-end test, and investigation into this occurs.
- **2022-09-13**: [Improved logging in the Prisma example deployment](https://github.com/iasql/iasql-engine/pull/1270), but cause of this issue not found.
- **2022-09-14**: [Root cause of Prisma end-to-end test found](https://github.com/iasql/iasql-engine/pull/1272): The test account was not properly cleaned. Adding a pre-clean script however, also broke things, and an issue in the `delete_all_records` postgres function was discovered. Adding `aws_regions` to a blacklist to that function resolved that issue. Deployment of `v0.0.20` successfully occurred after that and the Prisma end-to-end test fix was not truly necessary as `delete_all_records` is not meant for general consumption (yet).
- **2022-09-13**: [Improved logging in the Prisma example deployment](https://github.com/alantech/iasql-engine/pull/1270), but cause of this issue not found.
- **2022-09-14**: [Root cause of Prisma end-to-end test found](https://github.com/alantech/iasql-engine/pull/1272): The test account was not properly cleaned. Adding a pre-clean script however, also broke things, and an issue in the `delete_all_records` postgres function was discovered. Adding `aws_regions` to a blacklist to that function resolved that issue. Deployment of `v0.0.20` successfully occurred after that and the Prisma end-to-end test fix was not truly necessary as `delete_all_records` is not meant for general consumption (yet).
- **2022-09-16**: A strange behavior was detected in the production instance. `main` was not clean at that point and it was assumed it was related to that, so the code was cleaned and an out-of-band production deploy was done, and the problem was apparently resolved.
- **2022-09-19**: An investigation determined that was not the root cause, [the issue was reproduced and then fixed](https://github.com/iasql/iasql-engine/pull/1297) and a second out-of-band deploy occurred, resolving the outage completely.
- **2022-09-19**: An investigation determined that was not the root cause, [the issue was reproduced and then fixed](https://github.com/alantech/iasql-engine/pull/1297) and a second out-of-band deploy occurred, resolving the outage completely.

## Detection

Expand Down
2 changes: 1 addition & 1 deletion site/blog/optimizations/save-on-ecr.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ date: 2023-02-15
tags: [optimizations]
---

IaSQL is an [open-source](https://github.com/iasql/iasql) software tool that creates a two-way connection between an unmodified PostgreSQL database and an AWS account so you can manage your infrastructure from a database. In this post, we are going to learn how untagged ECR images can rack up your AWS bill unnecessarily and how to get rid of unused repository images with a single query in IaSQL: `DELETE FROM repository_images WHERE tag = '<untagged>';` <!--truncate--> Every time you want to deploy a new version of your code to an ECS container or EKS pod, you build and push a new docker image from your code into an ECR container repository. The latest image in an ECR repository is the only one in use. Stale images remain in the repository and accumulate over long periods by normal usage of ECS, or EKS, as there is typically no workflow to delete untagged images. However, there is a gotcha. AWS charges $0.10 per GB per month for images stored in private or public ECR repositories per the [pricing page](https://aws.amazon.com/ecr/pricing/). This doesn't sound like a lot, but for context, the size of the IaSQL docker image is around 2 GB. We have a CI job that deploys our staging environment every time we land a pull request to the main branch of our repository. This added up to 845 deploys over only a few months. Those 845 images, which have since been deleted, summed up to roughly 1700 GB which was $170 per month in our case. However, companies with lots of microservices on ECS or large docker containers can have many gigabytes of unused storage that can come out to hundreds or thousands of dollars per month of unnecessary AWS spend.
IaSQL is an [open-source](https://github.com/alantech/iasql) software tool that creates a two-way connection between an unmodified PostgreSQL database and an AWS account so you can manage your infrastructure from a database. In this post, we are going to learn how untagged ECR images can rack up your AWS bill unnecessarily and how to get rid of unused repository images with a single query in IaSQL: `DELETE FROM repository_images WHERE tag = '<untagged>';` <!--truncate--> Every time you want to deploy a new version of your code to an ECS container or EKS pod, you build and push a new docker image from your code into an ECR container repository. The latest image in an ECR repository is the only one in use. Stale images remain in the repository and accumulate over long periods by normal usage of ECS, or EKS, as there is typically no workflow to delete untagged images. However, there is a gotcha. AWS charges $0.10 per GB per month for images stored in private or public ECR repositories per the [pricing page](https://aws.amazon.com/ecr/pricing/). This doesn't sound like a lot, but for context, the size of the IaSQL docker image is around 2 GB. We have a CI job that deploys our staging environment every time we land a pull request to the main branch of our repository. This added up to 845 deploys over only a few months. Those 845 images, which have since been deleted, summed up to roughly 1700 GB which was $170 per month in our case. However, companies with lots of microservices on ECS or large docker containers can have many gigabytes of unused storage that can come out to hundreds or thousands of dollars per month of unnecessary AWS spend.

How do you get rid of untagged ECR images though? Deleting hundreds, or thousands, of container images across ECR repositories is quite tedious through the AWS console, which involves multiple clicks per deleted image. There are a few other options. IaC tools require state file manipulation, or it is not possible at all as ECR images are not typically modeled in the infrastructure declaration as they are often generated via CI and are quite numerous. Cloud Query and Steampipe let you query your cloud and inspect your ECR images, but do not let you delete the images or any part of your cloud account for that matter as they are read-only. The most common solution is often a script that calls the AWS SDK or CLI directly. IaSQL lets you delete untagged images to eliminate unnecessary AWS costs using the following query:

Expand Down
4 changes: 2 additions & 2 deletions site/blog/tutorials/deploy-static-website.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ authors: [mtp1376, depombo]
tags: [tutorial]
---

Did you know you can deploy a static website using a SQL REPL? In this post, we'll show you how to use IaSQL to deploy a static website from your GitHub repository to AWS S3 + Cloudfront services using only SQL queries. IaSQL is an [open-source](https://github.com/iasql/iasql) software tool that creates a two-way connection between an unmodified PostgreSQL database and an AWS account so you can manage your infrastructure from a database.
Did you know you can deploy a static website using a SQL REPL? In this post, we'll show you how to use IaSQL to deploy a static website from your GitHub repository to AWS S3 + Cloudfront services using only SQL queries. IaSQL is an [open-source](https://github.com/alantech/iasql) software tool that creates a two-way connection between an unmodified PostgreSQL database and an AWS account so you can manage your infrastructure from a database.

We will create and configure an S3 bucket to serve our static website. To enable support for HTTPS, we'll also add a CloudFront distribution. We will also leverage CodeBuild to automatically build the files for our project and copy them to the S3 bucket created already.

Expand Down Expand Up @@ -69,7 +69,7 @@ FROM
This is going to upload a file named `hello.txt` in our bucket whose content is `Hello IaSQL!`.

:::note
You can read the code for [`s3_upload_object` RPC](https://github.com/iasql/iasql/blob/c70f068c7520baf00cea9ddd3a76b8c6dbd2b23b/src/modules/aws_s3/rpcs/s3_upload_object.ts#L27-L33) as well as all other IaSQL modules and RPCs in our GitHub [repository](https://github.com/iasql/iasql/) to see how they work.
You can read the code for [`s3_upload_object` RPC](https://github.com/alantech/iasql/blob/c70f068c7520baf00cea9ddd3a76b8c6dbd2b23b/src/modules/aws_s3/rpcs/s3_upload_object.ts#L27-L33) as well as all other IaSQL modules and RPCs in our GitHub [repository](https://github.com/alantech/iasql/) to see how they work.
:::

Let's see if we can access our file using the S3 bucket URL. It should be as follows:
Expand Down
6 changes: 3 additions & 3 deletions site/blog/tutorials/django.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ In this tutorial, we will run [Django SQL migrations](https://docs.djangoproject

<!--truncate-->

IaSQL is an [open-source](https://github.com/iasql/iasql) software tool that creates a two-way connection between an unmodified PostgreSQL database and an AWS account so you can manage your infrastructure from a database. The code for this tutorial lives in this part of the [repository](https://github.com/iasql/iasql/tree/main/examples/ecs-fargate/django/app/infra/migrations/0003_initial.py).
IaSQL is an [open-source](https://github.com/alantech/iasql) software tool that creates a two-way connection between an unmodified PostgreSQL database and an AWS account so you can manage your infrastructure from a database. The code for this tutorial lives in this part of the [repository](https://github.com/alantech/iasql/tree/main/examples/ecs-fargate/django/app/infra/migrations/0003_initial.py).

## Start managing an AWS account with a PostgreSQL IaSQL db

Expand Down Expand Up @@ -95,7 +95,7 @@ If the function call is successful, it will return a virtual table with a record

## Connect to the PostgreSQL db and provision cloud resources in your AWS account

1. Get a local copy of the [ECS Fargate examples](https://github.com/iasql/iasql/tree/main/examples/ecs-fargate)
1. Get a local copy of the [ECS Fargate examples](https://github.com/alantech/iasql/tree/main/examples/ecs-fargate)

2. (Optional) Create and activate a virtual environment to install python dependencies

Expand Down Expand Up @@ -215,7 +215,7 @@ All of these steps will be done in a CodeBuild project in your AWS account. To u
```sql
SELECT
ecr_build (
'https://github.com/iasql/iasql/', -- replace with your own Github repo if you want to use your own codebase
'https://github.com/alantech/iasql/', -- replace with your own Github repo if you want to use your own codebase
(
SELECT
id
Expand Down
Loading

0 comments on commit bbeb653

Please sign in to comment.