Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add new aws recommendations #6820

Merged
merged 17 commits into from
Jun 18, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
845c48f
add new aws recommendations
joeyouss May 23, 2024
453ec37
added savings
joeyouss Jun 4, 2024
4bad4ee
added new diagram
joeyouss Jun 6, 2024
c01b1d1
added new diagram
joeyouss Jun 6, 2024
bbf1806
Update docs/cloud-cost-management/5-use-ccm-cost-governance/asset-gov…
joeyouss Jun 7, 2024
3d39b04
Update docs/cloud-cost-management/5-use-ccm-cost-governance/asset-gov…
joeyouss Jun 7, 2024
2ec43f9
Update docs/cloud-cost-management/5-use-ccm-cost-governance/asset-gov…
joeyouss Jun 7, 2024
32aabd9
Update docs/cloud-cost-management/5-use-ccm-cost-governance/asset-gov…
joeyouss Jun 7, 2024
92823be
Update docs/cloud-cost-management/5-use-ccm-cost-governance/asset-gov…
joeyouss Jun 7, 2024
e1c8905
Update docs/cloud-cost-management/5-use-ccm-cost-governance/asset-gov…
joeyouss Jun 7, 2024
1b538a0
Update docs/cloud-cost-management/5-use-ccm-cost-governance/asset-gov…
joeyouss Jun 7, 2024
c6a4f15
Update docs/cloud-cost-management/5-use-ccm-cost-governance/asset-gov…
joeyouss Jun 7, 2024
02bd052
Update docs/cloud-cost-management/5-use-ccm-cost-governance/asset-gov…
joeyouss Jun 7, 2024
3a9a77f
Update docs/cloud-cost-management/5-use-ccm-cost-governance/asset-gov…
joeyouss Jun 7, 2024
04ecafb
Update docs/cloud-cost-management/5-use-ccm-cost-governance/asset-gov…
joeyouss Jun 7, 2024
7e4dc32
Update docs/cloud-cost-management/5-use-ccm-cost-governance/asset-gov…
joeyouss Jun 7, 2024
b305a58
Update docs/cloud-cost-management/5-use-ccm-cost-governance/asset-gov…
joeyouss Jun 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ You can see a list of all recommendations offered by Harness for each Cloud prov

| Cloud | Cost Correlation | First Class Region Filter Support | Recommendations | Multi-Policy | Autostopping (EC2/VM/Instance) |
|-------|------------------|---------------------------------|------------------|--------------|--------------------------------|
| AWS | `aws.ec2`, `aws.ebs`, `aws.rds`, `aws.ebs-snapshot`, `aws.elastic-ip`, `aws.elb` | Yes ✅ | Yes ✅ | Yes ✅ | Yes ✅ |
| AWS | `aws.ec2`, `aws.ebs`, `aws.rds`, `aws.ebs-snapshot`, `aws.elastic-ip`, `aws.elb`, `cache-cluster`, `s3`, `redshift`, `redshift-snapshot` | Yes ✅ | Yes ✅ | Yes ✅ | Yes ✅ |
| GCP | `gcp.instance`, `gcp.disk`, `gcp.snapshot`, `gcp.sql-instance`, `gcp.image` | No ❌ | Yes ✅ | Yes ✅ | No ❌ |
| Azure | Every Resource in Billing Report | Yes ✅ | Yes ✅ | Yes ✅ | No ❌ |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -215,3 +215,259 @@ policies:

---

### Recommendation: under-utilized-elasticcache-cluster
**Description:** Delete underutilised cache cluster with CPU utilisation less than 5% in last 7 days.
joeyouss marked this conversation as resolved.
Show resolved Hide resolved

**Policy Used:**

```yaml
policies:
- name: under-utilized-elasticcache-cluster
resource: cache-cluster
description: |
Delete underutilised cache cluster with CPU utilisation less than 5% in last 7 days
filters:
- type: metrics
name: CPUUtilization
days: 7
period: 86400
value: 5
op: less-than
actions:
- type: delete
skip-snapshot: false
```

**Savings Computed:** The policy identifies a list of resources on which potential savings are calculated by summing up cost of each resource for last 30 days.
joeyouss marked this conversation as resolved.
Show resolved Hide resolved

**Permissions Required:**
- **Dry Run:**
- ```elasticache:DescribeCacheClusters```
- **Run Once:**
- ```elasticache:DescribeCacheClusters```
- ```elasticache:DeleteCacheCluster```
- ```elasticache:DeleteReplicationGroup```

---
joeyouss marked this conversation as resolved.
Show resolved Hide resolved

### Recommendation: s3-lifecycle-configuration

**Description:** Configure lifecycle for s3 buckets wherever it is absent which would help to reduce storage spend
joeyouss marked this conversation as resolved.
Show resolved Hide resolved

**Policy Used:**

```yaml
policies:
- name: s3-lifecycle-configuration
resource: aws.s3
description: |
Configure lifecycle for s3 buckets wherever it is absent which would help to reduce storage spend
filters:
- type : value
key : Lifecycle
value : absent
actions:
- type: configure-lifecycle
rules:
- ID: harness-default-lifecycle
Status: Enabled
Filter:
Prefix: ''
Expiration:
ExpiredObjectDeleteMarker: True
AbortIncompleteMultipartUpload:
DaysAfterInitiation: 7
NoncurrentVersionExpiration:
NoncurrentDays: 30
NewerNoncurrentVersions: 6
```

**Savings Computed**:
iamanmolm marked this conversation as resolved.
Show resolved Hide resolved
To estimate the percentage cost savings from the given S3 lifecycle policies, we need to look at the specific actions and apply some reasonable assumptions. Here's a step-by-step approach:

1. Abort Incomplete Multipart Uploads after 7 days:
- Assumption: 5% of all uploads are incomplete and are not cleaned up without this policy.
- Cost Impact: Each incomplete multipart upload that is aborted saves the storage cost of the data uploaded so far.

2. Expire Noncurrent Versions after 30 days (keeping 6 versions):
- Assumption: Each object has, on average, 10 noncurrent versions stored. Expiring noncurrent versions after 30 days, keeping only the latest 6, will delete 4 out of every 10 noncurrent versions.
- Cost Impact: Deleting 40% of noncurrent versions reduces the total storage used by these versions.

**Example Calculation**

Let's assume the following for a single S3 bucket:

**Total Storage Used**: 1 TB (1,024 GB) in the S3 Standard storage class.

**Storage Distribution:**
- Current versions: 50% (512 GB)
- Noncurrent versions: 40% (410 GB)
- Incomplete multipart uploads: 10% (102 GB)

**Without Lifecycle Policy**
joeyouss marked this conversation as resolved.
Show resolved Hide resolved

**Cost for S3 Standard:** $0.023 per GB per month


<DocImage path={require('../static/cal1.png')} width="50%" height="50%" title="Click to view full size image" />


**With Lifecycle Policy**

**Abort Incomplete Multipart Uploads:**
- Removes 5% of total storage (5% of 1,024 GB = 51 GB)
- New incomplete multipart uploads storage: 51 GB (102 - 51)

**Expire Noncurrent Versions:**
- Reduces noncurrent versions by 40% (40% of 410 GB = 164 GB)
- New noncurrent versions storage: 246 GB (410 - 164)

<DocImage path={require('../static/cal2.png')} width="50%" height="50%" title="Click to view full size image" />


**Total Savings**
<DocImage path={require('../static/savings1.png')} width="40%" height="40%" title="Click to view full size image" />

References:
- [AWS S3 Lifecycle Policies](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html)
- [AWS S3 Pricing](https://aws.amazon.com/s3/pricing/)
- [Managing S3 Costs](https://aws.amazon.com/blogs/storage/optimizing-costs-with-amazon-s3-lifecycle-configurations/)

**Permissions Required:**
- **Dry Run:**
- ```s3:GetLifecycleConfiguration```
- **Run Once:**
- ```s3:GetLifecycleConfiguration```
- ```s3:PutLifecycleConfiguration```

---
joeyouss marked this conversation as resolved.
Show resolved Hide resolved

### Recommendation: s3-intelligent-tiering-configuration

**Description:** Configure intelligent tiering for s3 buckets wherever it is disabled which would help to reduce storage spend.
joeyouss marked this conversation as resolved.
Show resolved Hide resolved

**Policy Used:**

```yaml
policies:
- name: s3-intelligent-tiering-configuration
resource: aws.s3
description: |
Configure intelligent tiering for s3 buckets wherever it is disabled which would help to reduce storage spend.
filters:
- not:
- type: intelligent-tiering
attrs:
- Status: Enabled
actions:
- type: set-intelligent-tiering
Id: harness-default
IntelligentTieringConfiguration:
Id: harness-default
Status: Enabled
Tierings:
- Days: 90
AccessTier: ARCHIVE_ACCESS
- Days: 180
AccessTier: DEEP_ARCHIVE_ACCESS
```

**Savings Computed:**
iamanmolm marked this conversation as resolved.
Show resolved Hide resolved
- **Frequent Access Tier:** This tier is equivalent in cost to the standard S3 storage, so no savings here.
- **Infrequent Access Tier:** Data not accessed for 30 days moves here, saving approximately 45% compared to standard S3 storage​.
- **Archive Instant Access Tier:** Data not accessed for 90 days moves here, with savings up to 68% compared to standard storage​.
joeyouss marked this conversation as resolved.
Show resolved Hide resolved
- **Archive Access Tier:** If configured, data not accessed for 90 days can move here, offering around 71% savings​.
- **Deep Archive Access Tier:** Data not accessed for 180 days can be moved to this tier, providing up to 95% savings.

**Example Calculation**

Assume you have 1 TB of data stored in S3 standard storage:

<DocImage path={require('../static/example1.png')} width="60%" height="60%" title="Click to view full size image" />

**Example Scenario**

If 20% of your data transitions to the Infrequent Access tier after 30 days, 20% moves to Archive Access after 90 days, and 10% moves to Deep Archive Access after 180 days, your costs might look like this:

<DocImage path={require('../static/example2.png')} width="50%" height="50%" title="Click to view full size image" />

This results in a cost savings of approximately 32.76% compared to keeping all data in standard S3 storage ($23.00 per month vs. $15.463 per month).

References:
- [AWS intelligent tiering](https://aws.amazon.com/s3/storage-classes/intelligent-tiering/)
- [AWS S3 Pricing](https://aws.amazon.com/s3/pricing/)

**Permissions Required:**
- **Dry Run:**
- ```s3:GetBucketIntelligentTieringConfiguration```
- **Run Once:**
- ```s3:GetBucketIntelligentTieringConfiguration```
- ```s3:PutIntelligentTieringConfiguration```

---
joeyouss marked this conversation as resolved.
Show resolved Hide resolved

### Recommendation: delete-underutilized-redshift-cluster

**Description:** Delete redshift cluster where CPU Utilization is less than 5% for last 7 days
joeyouss marked this conversation as resolved.
Show resolved Hide resolved

**Policy Used:**

```yaml
policies:
- name: delete-underutilized-redshift-cluster
resource: redshift
description: |
Delete redshift cluster where CPU Utilization is less than 5% for last 7 days
filters:
- type: metrics
name: CPUUtilization
days: 7
period: 86400
value: 5
op: less-than
actions:
- delete
```

**Savings Computed:** The policy identifies a list of resources on which potential savings are calculated by summing up cost of each resource for last 30 days.
joeyouss marked this conversation as resolved.
Show resolved Hide resolved

**Permissions Required:**
- **Dry Run:**
- ```redshift:DescribeClusters```
- **Run Once:**
- ```redshift:DescribeClusters```
- ```redshift:DeleteCluster```

---
joeyouss marked this conversation as resolved.
Show resolved Hide resolved

### Recommendation: delete-redshift-old-manual-snapshots

**Description:** Delete all redshift snapshot older than 35 days with lifetime retention period
joeyouss marked this conversation as resolved.
Show resolved Hide resolved

**Policy Used:**

```yaml

policies:
- name: delete-redshift-old-manual-snapshots
resource: redshift-snapshot
description: |
Delete all redshift snapshot older than 35 days with lifetime retention period
filters:
- "ManualSnapshotRetentionPeriod": -1
- type: age
days: 35
op: gt
actions:
- delete
```
**Savings Computed:** The policy identifies a list of resources on which potential savings are calculated by summing up cost of each resource for last 30 days.
joeyouss marked this conversation as resolved.
Show resolved Hide resolved

**Permissions Required:**
- **Dry Run:**
- ```redshift:DescribeClusterSnapshots```
- **Run Once:**
- ```redshift:DeleteClusterSnapshot```
- ```redshift:DescribeClusterSnapshots```

---
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading