feat: Add log retention on VPC flow logs#4164
Conversation
|
🍕 Here are the new binary sizes!
|
Codecov Report
@@ Coverage Diff @@
## mainline #4164 +/- ##
============================================
+ Coverage 69.21% 69.27% +0.05%
============================================
Files 250 250
Lines 35885 35987 +102
Branches 264 264
============================================
+ Hits 24838 24930 +92
- Misses 9854 9862 +8
- Partials 1193 1195 +2
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
There was a problem hiding this comment.
I wonder if it's worthy doing something similar to workload's log retention: , that we give users the option to write:
network:
vpc:
flow_log:
retention: 21 to specify their preferred retention, and if it isnetwor.vpc.flow_log: true we give it a default of 14 .
Just throwing this random thought here - perhaps no one needs to configure the retention and that a default of 14 is enough. What do yall think?
paragbhingre
left a comment
There was a problem hiding this comment.
Looks good, just a nit and a question.
efekarakus
left a comment
There was a problem hiding this comment.
Sweet! Looks great!
I have some clarification questions to @dannyrandall regarding the Union type and based on his feedback, we can then ship it!
This PR implements methods to read environment addons from the workspace in `copilot/environments/addons` folder. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the Apache 2.0 License.
Previously, turning on [access logs](https://aws.github.io/copilot-cli/docs/manifest/environment/#http-public-access-logs) for the ALB would result in the error like: ``` Access Denied for bucket: app-env-elbaccesslogsbucket-1234. Please check S3bucket permission (Service: AmazonElasticLoadBalancing; Status Code: 400; Error Code: InvalidConfigurationRequest; Request ID: id; Proxy: null) ``` This is because Cloudformation would update the ALB's settings _before_ creating the bucket policy that allows the ALB to send logs to the bucket. Now, we add an explicit dependency to the ALB on the bucket policy if access logs is enabled so that the policy is created/applied before attempting to put objects in the bucket. --- By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the Apache 2.0 License.
This PR implements a new API for the `workspace` package. With the new API, a workspace client that gets returned to the caller always have the `copilotDirAbs` field in it. This means that the method calls can assume non-empty value of that field. This addresses the proposal made in aws#4147 (comment) By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the Apache 2.0 License.
As well as removing deprecated references By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the Apache 2.0 License.
aws#4113) This PR resolves aws#815 For example, if you run `copilot app init my-proj` in `us-east-1`, then run `copilot app init my-proj` in `us-west-2` the second init will fail. This happens because CloudFormation creates an adminrole for the application of the form `{app-name}-adminrole`. CloudFormation can't create two roles with the same name since roles are global. Currently, CloudFormation errors out saying failed to create: [AdministrationRole] `stack {app-name}-infrastructure-roles did not complete successfully and exited with status ROLLBACK_COMPLETE` This PR will do the following 1. Check to see if the application name exists in SSM 2. If it doesn't, check to see if the role exists 3. If the role exists, but the application doesn't exist shows an error: `application named {app-name} already exists in another region` Case 1: If you run `copilot app init test-project` in `us-east-1` and If run `copilot app init test-project `in `us-west-2` Then an error message shows `application named test-project already exists in another region` Case 2: If there is already a role existing in the account with same name as `applicationname-adminrole` then we will error out `IAM admin role {applicationname}-adminrole already exists in this account` By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the Apache 2.0 License.
Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.24.0 to 1.24.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/onsi/gomega/blob/master/CHANGELOG.md">github.com/onsi/gomega's changelog</a>.</em></p> <blockquote> <h2>1.24.1</h2> <h3>Fixes</h3> <ul> <li>maintain backward compatibility for Eventually and Consisntetly's signatures [4c7df5e]</li> <li>fix small typo (<a href="https://github-redirect.dependabot.com/onsi/gomega/issues/601">#601</a>) [ea0ebe6]</li> </ul> <h3>Maintenance</h3> <ul> <li>Bump golang.org/x/net from 0.1.0 to 0.2.0 (<a href="https://github-redirect.dependabot.com/onsi/gomega/issues/603">#603</a>) [1ba8372]</li> <li>Bump github.com/onsi/ginkgo/v2 from 2.4.0 to 2.5.0 (<a href="https://github-redirect.dependabot.com/onsi/gomega/issues/602">#602</a>) [f9426cb]</li> <li>fix label-filter in test.yml [d795db6]</li> <li>stop running flakey tests and rely on external network dependencies in CI [7133290]</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/onsi/gomega/commit/3eef0d7813628f9e4bcfb93e6dd3e8cd12342cd6"><code>3eef0d7</code></a> v1.24.1</li> <li><a href="https://github.com/onsi/gomega/commit/4c7df5e4410ca8808b0c7d783a67276c7d66bd1d"><code>4c7df5e</code></a> maintain backward compatibility for Eventually and Consisntetly's signatures</li> <li><a href="https://github.com/onsi/gomega/commit/1ba837273b0a831eb63517e7b420b6dbb77ba9a7"><code>1ba8372</code></a> Bump golang.org/x/net from 0.1.0 to 0.2.0 (<a href="https://github-redirect.dependabot.com/onsi/gomega/issues/603">#603</a>)</li> <li><a href="https://github.com/onsi/gomega/commit/f9426cb65c62d6aba984200a75552119dca077c8"><code>f9426cb</code></a> Bump github.com/onsi/ginkgo/v2 from 2.4.0 to 2.5.0 (<a href="https://github-redirect.dependabot.com/onsi/gomega/issues/602">#602</a>)</li> <li><a href="https://github.com/onsi/gomega/commit/ea0ebe63c4f7ebba21cbb115e3653427b34750f8"><code>ea0ebe6</code></a> fix small typo (<a href="https://github-redirect.dependabot.com/onsi/gomega/issues/601">#601</a>)</li> <li><a href="https://github.com/onsi/gomega/commit/d795db6ca6927dacbd7f46d20ee54023b46542c1"><code>d795db6</code></a> fix label-filter in test.yml</li> <li><a href="https://github.com/onsi/gomega/commit/7133290814e1dd9bc139f8cf189f8c0f8a55f457"><code>7133290</code></a> stop running flakey tests and rely on external network dependencies in CI</li> <li>See full diff in <a href="https://github.com/onsi/gomega/compare/v1.24.0...v1.24.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>
…4177) Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.131 to 1.44.136. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p> <blockquote> <h1>Release v1.44.136 (2022-11-11)</h1> <h3>Service Client Updates</h3> <ul> <li><code>service/glue</code>: Updates service documentation <ul> <li>Added links related to enabling job bookmarks.</li> </ul> </li> <li><code>service/iot</code>: Updates service API and documentation <ul> <li>This release add new api listRelatedResourcesForAuditFinding and new member type IssuerCertificates for Iot device device defender Audit.</li> </ul> </li> <li><code>service/license-manager</code>: Updates service API and documentation</li> <li><code>service/marketplace-catalog</code>: Updates service API and documentation</li> <li><code>service/rekognition</code>: Updates service API and documentation <ul> <li>Adding support for ImageProperties feature to detect dominant colors and image brightness, sharpness, and contrast, inclusion and exclusion filters for labels and label categories, new fields to the API response, "aliases" and "categories"</li> </ul> </li> <li><code>service/securityhub</code>: Updates service documentation</li> <li><code>service/ssm-incidents</code>: Updates service API and documentation</li> </ul> <h1>Release v1.44.135 (2022-11-10)</h1> <h3>Service Client Updates</h3> <ul> <li><code>service/autoscaling</code>: Updates service documentation <ul> <li>This release adds a new price capacity optimized allocation strategy for Spot Instances to help customers optimize provisioning of Spot Instances via EC2 Auto Scaling, EC2 Fleet, and Spot Fleet. It allocates Spot Instances based on both spare capacity availability and Spot Instance price.</li> </ul> </li> <li><code>service/ec2</code>: Updates service API and documentation <ul> <li>This release adds a new price capacity optimized allocation strategy for Spot Instances to help customers optimize provisioning of Spot Instances via EC2 Auto Scaling, EC2 Fleet, and Spot Fleet. It allocates Spot Instances based on both spare capacity availability and Spot Instance price.</li> </ul> </li> <li><code>service/ecs</code>: Updates service API, documentation, and examples <ul> <li>This release adds support for task scale-in protection with updateTaskProtection and getTaskProtection APIs. UpdateTaskProtection API can be used to protect a service managed task from being terminated by scale-in events and getTaskProtection API to get the scale-in protection status of a task.</li> </ul> </li> <li><code>service/es</code>: Updates service API and documentation <ul> <li>Amazon OpenSearch Service now offers managed VPC endpoints to connect to your Amazon OpenSearch Service VPC-enabled domain in a Virtual Private Cloud (VPC). This feature allows you to privately access OpenSearch Service domain without using public IPs or requiring traffic to traverse the Internet.</li> </ul> </li> <li><code>service/resource-explorer-2</code>: Updates service documentation</li> <li><code>service/scheduler</code>: Updates service API, documentation, paginators, and examples</li> </ul> <h1>Release v1.44.134 (2022-11-09)</h1> <h3>Service Client Updates</h3> <ul> <li><code>service/connect</code>: Updates service API and documentation</li> <li><code>service/connectcases</code>: Updates service API and documentation</li> <li><code>service/ec2</code>: Updates service API <ul> <li>Amazon EC2 Trn1 instances, powered by AWS Trainium chips, are purpose built for high-performance deep learning training. u-24tb1.112xlarge and u-18tb1.112xlarge High Memory instances are purpose-built to run large in-memory databases.</li> </ul> </li> <li><code>service/groundstation</code>: Updates service API, documentation, and paginators</li> <li><code>service/mediapackage-vod</code>: Updates service API and documentation</li> <li><code>service/transcribe-streaming</code>: Updates service API and documentation</li> </ul> <h1>Release v1.44.133 (2022-11-08)</h1> <h3>Service Client Updates</h3> <ul> <li><code>service/acm</code>: Updates service API and documentation <ul> <li>Support added for requesting elliptic curve certificate key algorithm types P-256 (EC_prime256v1) and P-384 (EC_secp384r1).</li> </ul> </li> <li><code>service/billingconductor</code>: Updates service API, documentation, and paginators</li> <li><code>service/ec2</code>: Updates service API and documentation</li> </ul> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/aws/aws-sdk-go/commit/00c5fc72f87f4eebe2fc25400d9283ca2bceb0ea"><code>00c5fc7</code></a> Release v1.44.136 (2022-11-11) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4620">#4620</a>)</li> <li><a href="https://github.com/aws/aws-sdk-go/commit/2d26930f04c9ef7e02cc6ad143c26c779b338c8e"><code>2d26930</code></a> Release v1.44.135 (2022-11-10) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4619">#4619</a>)</li> <li><a href="https://github.com/aws/aws-sdk-go/commit/b4fc3aa52b054aaad4d16f31f10e3a5b05a2e203"><code>b4fc3aa</code></a> Release v1.44.134 (2022-11-09) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4617">#4617</a>)</li> <li><a href="https://github.com/aws/aws-sdk-go/commit/b682b31ee89b7310b4ccad3e434aa8be70cf5fe4"><code>b682b31</code></a> Release v1.44.133 (2022-11-08) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4616">#4616</a>)</li> <li><a href="https://github.com/aws/aws-sdk-go/commit/ddc6ad3b4da199468c2ef42b3ebbc08bb40c3d3a"><code>ddc6ad3</code></a> Release v1.44.132 (2022-11-07) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4615">#4615</a>)</li> <li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.131...v1.44.136">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>
should help with some debugging. - https://gitter.im/aws/copilot-cli?at=636cb648473cf96648dbcbb9 - https://gitter.im/aws/copilot-cli?at=636c024ca34b51121122d168 By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the Apache 2.0 License.
This PR resolves aws#3200 With this PR we will have a new field `count` to the Manifest of RequestDrivenService. ```yaml count: 'high-availability/3' ``` In order to use this field customers have to provide `AutoScalingConfigurationName/AutoScalingConfigurationRevision` as input to`count`. `count: AutoScalingConfigurationName/AutoScalingConfigurationRevision` This field will help customers to reference existing autoscaling configurations for different services. By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the Apache 2.0 License.
…`spot_from` equal (aws#4187) Addresses: https://gitter.im/aws/copilot-cli?at=637244e418f21c023bb057cc Previously, the logic to scale into Fargate Spot only applied if the `spot_from` value was greater than the `min` value. So in edge case of users having equal values greater than 1 for those fields, as in the case linked to above, all of the tasks would have Fargate Spot as their capacity provider, rather than the `spot_from` value determining the threshold for Fargate vs Fargate Spot. This kind of config is somewhat atypical, as the `min` value isn't guaranteed if that number task has Fargate Spot as a capacity provider, but if users do set up their scaling this way, at least we are honoring the `spot_from` value. With this change: `min: 3` and `spot_from: 3` --> 2 Fargate, 1 Spot `min: 2` and `spot_from: 2` --> 1 Fargate, 1 Spot `min: 1` and `spot_from: 1` --> 0 Fargate, 1 Spot `min: 0` and `spot_from: 0` --> 0 Fargate, 1 Spot By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the Apache 2.0 License.
…s#4189) This PR contains three changes: 1. A generic `Union` transformer - Union types no longer have to be added individually to the transformers list, as this transformer will handle all cases. 2. When unmarshalling, `Union` now uses `reflect.IsZero` to determine if a value hasn't been set. This means types used in a Union will need to be pointers if the zero value of a type means its been set (e.g., `key: false` will be considered unset unless the Union type is `*bool`) 3. ~Enables strict unmarshaling of types embedded in `Union`. This solves an issue where invalid yaml for a struct results in a zero value, requiring most struct types to have their own `IsZero` function to handle this case. For example, take this struct and unmarshal the below yaml into `Union[*string, complex]`:~ By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the Apache 2.0 License.
This PR resolves #4162 the retention for a log group of VPC FlowLog i.e, The number of days to retain the log events in the specified log group.
If the flowlogs is
onThe default value for retention of log events in log group is set to 14 days[2 weeks].
If the customer want to have a specific retention rate then Environment Manifest can be modified to
Here are the possible values for retention in days: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-loggroup.html
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the Apache 2.0 License.