feat(cli): watch streams resources' CloudWatch logs to the terminal #18159

corymhall · 2021-12-23T18:39:35Z

This adds a new --logs flag on cdk watch which is set to true by
default. Watch will monitor all CloudWatch Log groups in the application
and stream the log events back to the users terminal.

re #18122

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

gitpod-io · 2021-12-23T18:39:37Z

packages/aws-cdk/lib/api/hotswap/monitor/logs-monitor.ts

corymhall · 2021-12-23T18:43:28Z

This is what the logs currently look like. Definitely looking for feedback on formatting.

skinny85

I think this will be an amazing feature - awesome job @corymhall!

My main feedback:

Let's make the logs show by default.
I don't think waiting for a resource to be hotswapped to show its logs is the correct experience. Let's show all of the logs of supported resources instead.
Let's work on the log presentation, and try to reduce the delay in surfacing them to the minimum.
Some code needs some polishing, as far as the split of responsibilities in the APIs (see my comments inline).

packages/aws-cdk/bin/cdk.ts

packages/aws-cdk/lib/api/aws-auth/sdk.ts

packages/aws-cdk/lib/api/hotswap-deployments.ts

packages/aws-cdk/lib/api/hotswap/monitor/logs-monitor.ts

skinny85 · 2021-12-28T00:29:06Z

packages/aws-cdk/lib/api/hotswap/monitor/logs-monitor.ts

+  public print(event: CloudWatchLogEvent): void {
+    this.stream.write(
+      util.format(
+        '%s %s %s %s',


This is way, way too much info on the left side. For sure you can kill the log stream from there, I think it's not important to show it. I would even change the log group name to the physical name of the Lambda, will make it clearer where the logs are coming from, and take less space.

Also, maybe think about using separators like [] to make the logs clearer? Something like this perhaps:

[HackathonExampleStack-NodeJsFunction6DD2A8DD-adO4KmEjQSuq] 3:48:32 PM: REPORT RequestId: b333acdd-f56b-403a-9b88-2a3bc07a62b9 Duration: 17.37 ms Billed Duration: 18 ms Memory Size: 128 MB Max Memory Used: 56 MB Init Duration: 185.21 ms

This is partially updated. Still working on the final version.

packages/aws-cdk/lib/api/hotswap/monitor/logs-monitor.ts

skinny85

I like the general approach! A few more comments. The main thing is that I think we should try to unchain this functionality from hotswapping - I think it should be available to all cdk watch customers, regardless of whether they use hotswapping, or not (I don't see this functionality being tied to hotswapping in any significant way).

packages/aws-cdk/bin/cdk.ts

packages/aws-cdk/lib/api/hotswap-deployments.ts

packages/aws-cdk/lib/api/hotswap/monitor/logs-monitor.ts

packages/aws-cdk/lib/api/hotswap/cloudwatch-logs.ts

packages/aws-cdk/lib/api/hotswap/monitor/logs-monitor.ts

packages/aws-cdk/lib/api/hotswap/cloudwatch-logs.ts

packages/aws-cdk/lib/api/hotswap/monitor/logs-monitor.ts

packages/aws-cdk/lib/api/hotswap-deployments.ts

This adds a new `--logs` flag on `cdk watch` which is set to false by default. When this flag is provided (set to true) watch will fetch any CloudWatch Logs log events for any hotswapped Lambda Function starting at the time the function was hotswapped. Once `cdk watch --logs` is run, any Lambda function that is hotswapped is added to a list of "monitored" resources. This list is maintained through full CloudFormation deployments. So for example if an application had 3 Lambda functions, after running `cdk watch --logs` none of the functions are monitored by default. 1. Change function code for Function1 Function1's CloudWatch log group is now being monitored 2. Change function code for Function2 Function2's CloudWatch log group is now being monitored 3. Change a non-hotswappable property which triggers a full CloudFormation deployment Function1 & Function2 are still being monitored re #18122

Co-authored-by: Adam Ruka <adamruka85@gmail.com>

updated to use the `print` method that we use elsewhere in the cli also switched back to using `setTimeout` instead of `setInterval` because `setInterval` sometimes caused events to be processed more than once.

corymhall · 2021-12-29T21:17:32Z

@skinny85 I've updated the PR to include all hotswappable resources and it now monitors logs independent of hotswap. Some things I'm still thinking through:

Should we or should we not filter what log groups we monitor? I think we should to some degree (we don't want to monitor VPC Flow Logs for example). Right now I am filtering for only resources that we support hotswapping for, but I don't think that is the correct level. We also might want to somehow filter out Lambda Functions that are used as custom resource providers.
Still working through the "experience" aspect of this feature. Some things I added today:
- Pause the monitor while we are performing a deployment so we don't have application logs intermixed with deployment logs. After the deployment is complete, the monitor will read and print all logs since the last time we processed logs.
- Limit to last 100 log events
- Updated to use the logger that we use elsewhere in the cli which eliminates some of the display bugs I was running into.

skinny85

This is starting to look great!

We need to iterate on the names, and on the locations of the particular files, a little bit more.

To answer your questions:

Should we or should we not filter what log groups we monitor? I think we should to some degree (we don't want to monitor VPC Flow Logs for example). Right now I am filtering for only resources that we support hotswapping for, but I don't think that is the correct level. We also might want to somehow filter out Lambda Functions that are used as custom resource providers.

Like I said in #18159 (comment), I think we should filter out the Log Groups we don't want (like the VPC flow logs), instead of keeping only the logs we deem "interesting".

Still working through the "experience" aspect of this feature. Some things I added today:

Pause the monitor while we are performing a deployment so we don't have application logs intermixed with deployment logs. After the deployment is complete, the monitor will read and print all logs since the last time we processed logs.

I actually think it's fine if we show the deployment logs during deployment - I assume those would be mainly the logs of our Custom Resources, which might actually make diagnosing problems with them easier when they fail.

Unless that screws up the progress bar we render during deployment, in which case yeah, that's a good call 🙂.

packages/aws-cdk/bin/cdk.ts

packages/aws-cdk/lib/api/cloudformation-deployments.ts

packages/aws-cdk/lib/api/hotswap-deployments.ts

packages/aws-cdk/lib/api/hotswap/cloudwatch-logs.ts

packages/aws-cdk/lib/api/hotswap/monitor/logs-monitor.ts

packages/aws-cdk/lib/cdk-toolkit.ts

corymhall · 2021-12-30T18:40:38Z

I actually think it's fine if we show the deployment logs during deployment - I assume those would be mainly the logs of our Custom Resources, which might actually make diagnosing problems with them easier when they fail.

Unless that screws up the progress bar we render during deployment, in which case yeah, that's a good call 🙂.

Yeah it was messing with the progress bar during deployment. We'll still get the logs from our CustomResources, they'll just be printed after the deployment finishes.

corymhall · 2021-12-30T18:43:33Z

Current experience:

resource types

skinny85

This is amazing. The experience of the feature is awesome, and the code looks perfect to me. I'd want to wait for #18277 to be merged first, and then update this PR to also use the lookup Role for new-style synthesis, but I'm giving my "ShipIt" right now, and just adding the pr/do-not-merge label.

Awesome work @corymhall!

packages/aws-cdk/lib/api/aws-auth/sdk-provider.ts

This reverts commit 3f3547a.

skinny85

I'm still good with this, just one suggestion.

skinny85 · 2022-01-11T00:15:20Z

packages/aws-cdk/lib/api/cloudformation-deployments.ts

+  * then return those and note that we are returning the default credentials. The calling
+  * function can then decide to use them or fallback to another role.
+  */
+export async function prepareSdkWithLookupRoleFor(


Any chance we can make this a method of SdkProvider?

I originally had it as a method of SdkProvider, but after discussing with @rix0rrr we decided to put it here.

Yeah, the SdkProvider shouldn't know about CloudFormationStackArtifacts or the fact that SSM parameters hold bootstrap stack versions.

rix0rrr

Love this feature. I just want more of it😅

rix0rrr · 2022-01-11T15:04:22Z

packages/aws-cdk/lib/api/cloudformation-deployments.ts

+  * then return those and note that we are returning the default credentials. The calling
+  * function can then decide to use them or fallback to another role.
+  */
+export async function prepareSdkWithLookupRoleFor(


Yeah, the SdkProvider shouldn't know about CloudFormationStackArtifacts or the fact that SSM parameters hold bootstrap stack versions.

rix0rrr · 2022-01-11T15:06:09Z

packages/aws-cdk/lib/api/evaluate-cloudformation-template.ts

+  listStackResources(): Promise<AWS.CloudFormation.StackResourceSummary[]>;
+}
+
+export class LazyListStackResources implements ListStackResources {


Nit: it's more like CachedListStackResources than Lazy*.

~~Is there ever a chance we will list the stack resources before a particular resource is created, and then never update the list to account for the new resource?~~

Aha I guess since we only use this for hotswapping, that will never add new resources so doesn't apply.

In general, the PR description could do with a Cliff's Notes of the approach taken, biggest challenges, design decisions and refactorings. It's very helpful when evaluating a change.

In general, the PR description could do with a Cliff's Notes of the approach taken, biggest challenges, design decisions and refactorings. It's very helpful when evaluating a change.

That is a good suggestion! This particular file was existing functionality that was moved here, which I could have called out.

Nit: it's more like CachedListStackResources than Lazy*.

It is actually Lazy though (it will not make a service call unless you call listResources()).

That is true for most methods though 🤣

Not sure what you mean. I think a very sensible non-lazy implementation would be to make the service call in the constructor of the class, and not in listResources() (it would be simpler, for one thing).

rix0rrr · 2022-01-11T15:08:19Z

packages/aws-cdk/lib/api/evaluate-cloudformation-template.ts

+      try {
+        return await this.evaluateCfnExpression(physicalNameInCfnTemplate);
+      } catch (e) {
+        // If we can't evaluate the resource's name CloudFormation expression,


What is this function used for? To predict physical names, before deployment happens?

Can't we just always list from the currently deployed stack?

I think the intent here was to avoid a lookup if we could ( cc @skinny85 )

Yes, this is used in case we can get the physical name from the template, and in this way save a service call (not super relevant here, but very relevant for hotswapping, and this class is used in both places).

rix0rrr · 2022-01-11T15:11:45Z

packages/aws-cdk/lib/api/logs/find-cloudwatch-logs.ts

+const IGNORE_LOGS_RESOURCE_TYPES = ['AWS::EC2::FlowLog', 'AWS::CloudTrail::Trail', 'AWS::CodeBuild::Project'];
+
+// Resource types that will create a CloudWatch log group with a specific name if one is not provided.
+// The keys are CFN resource types, and the values are the names of the physical name property of that resource.


Don't you need to account for the prefix? Or are you guaranteeing that it will always be /aws/<servicename>/ ?

Because I can assure you, AWS don't roll that way 🤣

I could update this to include the prefix to more future proof this.

rix0rrr · 2022-01-11T15:12:29Z

packages/aws-cdk/lib/api/logs/find-cloudwatch-logs.ts

+export async function findCloudWatchLogGroups(
+  sdkProvider: SdkProvider,
+  stackArtifact: cxapi.CloudFormationStackArtifact,
+): Promise<{ env: cxapi.Environment, sdk: ISDK, logGroupNames: string[] }> {


Declare the return struct separately please.

rix0rrr · 2022-01-11T15:22:36Z

packages/aws-cdk/lib/api/logs/logs-monitor.ts

+      do {
+        const response = await logGroupsAccessSettings.sdk.cloudWatchLogs().filterLogEvents({
+          logGroupName: logGroupName,
+          nextToken,


How do nextToken and startTime interact? Don't we need to hold on to the token as well?

the filterLogEvents operation returns as many log events after the startTime as can fit in 1 MB (up to 100 events since we specified that limit). If the first 90 events took up the 1 MB then we would receive a nextToken to retrieve the final 10. In the case where we receive the nextToken we still want to use the same startTime, we only want to update the startTime for the next time we process events (the next time we tick().

Interesting (and odd). But okay!

Ended up removing the nextToken since the intent is to just provide a sampling of 100 events if we have > 100.

rix0rrr · 2022-01-11T15:23:20Z

packages/aws-cdk/lib/api/logs/logs-monitor.ts

+import { ISDK } from '../aws-auth';
+
+// how often we should read events from CloudWatchLogs
+const SLEEP = 2_000;


We probably need some backoff as well, eh?

What do you mean? Maybe I should reword this comment, it is the amount of time we wait after processing events before we start processing new events. So we:

read all events starting with startTime for all log groups

print all the events

SLEEP

start back at 1

I'm just a little concerned about overwhelming CloudWatch Logs with filterLogEvents calls, and getting throttled.

Typically, in a throttling event you do an exponential backoff to avoid overloading the backend service. I suppose there's an argument to be made that the SDK's retry settings would take care of this for us... but if that's your argument I'd like to see that explicitly stated somewhere.

I added a comment here to note that sleep is also how we are handling backoff. Now that we are not using nextToken we will only make 1 request per log group. If any of the requests are throttled the SDKs builtin retry/backoff will take effect. If that doesn't resolve it, it will throw an error, wait for SLEEP and then try again.

rix0rrr · 2022-01-11T15:26:03Z

packages/aws-cdk/lib/cdk-toolkit.ts

@@ -304,10 +310,12 @@ export class CdkToolkit {
    // --------------                --------  'cdk deploy' done  --------------  'cdk deploy' done  --------------
    let latch: 'pre-ready' | 'open' | 'deploying' | 'queued' = 'pre-ready';

+    const cloudWatchLogMonitor = options.traceLogs ? new CloudWatchLogEventMonitor() : undefined;


So honestly, this doesn't have a lot to do with watch, does it?

We're not monitoring changes that get triggered during watch, or even as a result of watch.

We're watching for activity that most likely occurs because a user clicks around in their UI.

Shouldn't this be a separate CLI command primarily, and only incidentally something that can be activated as part of cdk watch ?

cc @skinny85

We don't have currently any continuous running command in our CLI beyond watch, so I don't see where it would fit besides watch, and it slots into watch beautifully.

If you want to create a separate command for this (cdk logs, or something), I think that's reasonable, but that's a separate task and PR.

rix0rrr · 2022-01-11T15:26:57Z

packages/aws-cdk/lib/api/logs/logs-monitor.ts

+   * Print out a cloudwatch event
+   */
+  private print(event: CloudWatchLogEvent): void {
+    print(util.format('[%s] %s %s',


Wouldn't this need some spacing/alignment here to make this readable?

How long are these log group names typically?

Also, do we need the log stream name in there?

Wouldn't this need some spacing/alignment here to make this readable?

I'm not sure, what type of spacing/alignment are you thinking? The current experience looks fine to me. I think the colors help to make it easy to read.

How long are these log group names typically?

They can be pretty long (for example /aws/lambda/AnotherHotswapTestingAppStack-Handler886CB40B-zR3TWjodpJbS). I'm not sure how to trim it down though. Someone could just have a really long stack name and then there would be no way to differentiate the log groups in a stack.

Also, do we need the log stream name in there?

I did originally include the log stream name, but we decided it made the output even longer without really adding much.

In my opinion, the current experience is fine. I would be even happier with substituting the name of the Log Group (which tends to be very long) with the physical name of the resource it corresponds too, but there's some complexity there, so I',m fine with it as-is.

I think the log group stream name is not needed, and, when combined with the log group name and the date, takes up 80% of a 150-character line.

rix0rrr · 2022-01-11T15:27:43Z

packages/aws-cdk/lib/api/logs/logs-monitor.ts

+  private print(event: CloudWatchLogEvent): void {
+    print(util.format('[%s] %s %s',
+      colors.blue(event.logGroupName),
+      colors.yellow(event.timestamp.toLocaleTimeString()),


What timezone are these timestamps in? Are they correct? Wouldn't we want more control over the formatting?

The timestamp of the event is using UTC (expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC) so this should just format those to the users locale. So for example someone in the US would get something like 12/11/2012, 7:00:00 PM.

Yes, everyone will see the time printed in their local timezone, which I think is what you want here.

Pretty sure toLocaleTimeString() does not do timezone conversion, and I think native d/t formatting is not a great idea.

But it will do for now, I suppose.

I definitely see the logs in PST, so it's either toLocaleTimeString(), or perhaps CloudWatch returns them in the correct timezone...?

I tested on machines with different timezones and I get whatever the local time is.

rix0rrr

Conditionally approved on the passing of a method, and having thought about backoff.

log groups from the deployed template rather than by parsing the current template

skinny85 · 2022-01-12T19:41:36Z

packages/aws-cdk/lib/api/hotswap-deployments.ts

@@ -40,7 +39,7 @@ export async function tryHotswapDeployment(
    account: resolvedEnv.account,
    region: resolvedEnv.region,
    partition: (await sdk.currentAccount()).partition,
-    urlSuffix: sdk.getEndpointSuffix,
+    urlSuffix: sdk.getEndpointSuffix(resolvedEnv.region),


Nope! Please, leave this as-is, and pass a lambda function there, like Rico suggested.

This is so that we avoid as many service calls as we can during hotswapping.

ahhh sorry, fixed!

skinny85

👍

mergify · 2022-01-12T20:39:16Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

mergify · 2022-01-12T21:25:21Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2022-01-12T21:25:24Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
Commit ID: d875b92
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…ws#18159) This adds a new `--logs` flag on `cdk watch` which is set to `true` by default. Watch will monitor all CloudWatch Log groups in the application and stream the log events back to the users terminal. re aws#18122 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

github-actions bot added the package/tools Related to AWS CDK Tools or CLI label Dec 23, 2021

github-actions bot assigned rix0rrr Dec 23, 2021

mergify bot added the contribution/core This is a PR that came from AWS. label Dec 23, 2021

corymhall commented Dec 23, 2021

View reviewed changes

packages/aws-cdk/lib/api/hotswap/monitor/logs-monitor.ts Outdated Show resolved Hide resolved

corymhall requested a review from skinny85 December 23, 2021 18:43

comcalvi assigned skinny85 and unassigned rix0rrr Dec 23, 2021

skinny85 suggested changes Dec 28, 2021

View reviewed changes

corymhall commented Dec 28, 2021

View reviewed changes

packages/aws-cdk/lib/api/hotswap/monitor/logs-monitor.ts Outdated Show resolved Hide resolved

skinny85 reviewed Dec 29, 2021

View reviewed changes

corymhall and others added 4 commits December 29, 2021 12:46

simplifying some stuff

151b41d

Apply suggestions from code review

d290fee

Co-authored-by: Adam Ruka <adamruka85@gmail.com>

refactoring to include all hotswap resources

a13dafb

corymhall force-pushed the corymhall/hotswap-logs branch from 0eb6ce3 to a13dafb Compare December 29, 2021 12:49

corymhall added 3 commits December 29, 2021 14:19

updates

ad1b4a4

monitor logs regardless of whether we are hotswapping or not

18fbefd

pause the logs monitor during a deployment

5bba356

updated to use the `print` method that we use elsewhere in the cli also switched back to using `setTimeout` instead of `setInterval` because `setInterval` sometimes caused events to be processed more than once.

skinny85 suggested changes Dec 30, 2021

View reviewed changes

updates

94fc4c8

corymhall added 2 commits December 30, 2021 19:20

update to include all cloudwatch log groups and only exclude certain

9e0bef8

resource types

updating readme

809d3d8

corymhall changed the title ~~feat(cli): cdk watch can now stream lambda function logs to the terminal~~ feat(cli): cdk watch can now stream cloudwatch logs to the terminal Dec 30, 2021

github-actions bot assigned rix0rrr Dec 30, 2021

reset tracked log groups prior to each deployment

cb078bf

skinny85 added the pr/do-not-merge This PR should not be merged at this time. label Jan 7, 2022

skinny85 approved these changes Jan 7, 2022

View reviewed changes

corymhall added 2 commits January 10, 2022 10:46

Merge branch 'master' into corymhall/hotswap-logs

c38de39

using lookup role for reading cloudwatch logs

3f3547a

corymhall commented Jan 10, 2022

View reviewed changes

packages/aws-cdk/lib/api/aws-auth/sdk-provider.ts Outdated Show resolved Hide resolved

corymhall added 2 commits January 10, 2022 16:38

Revert "using lookup role for reading cloudwatch logs"

5043f4a

This reverts commit 3f3547a.

use lookup role for reading events from cloudwatch logs

965f963

corymhall removed the pr/do-not-merge This PR should not be merged at this time. label Jan 10, 2022

corymhall requested a review from rix0rrr January 10, 2022 19:01

skinny85 approved these changes Jan 11, 2022

View reviewed changes

rix0rrr reviewed Jan 11, 2022

View reviewed changes

updates based on review comments

e46e971

corymhall requested a review from rix0rrr January 11, 2022 19:51

corymhall added 2 commits January 11, 2022 14:53

Merge branch 'master' into corymhall/hotswap-logs

4369e08

switching from colors to chalk

15f3e91

rix0rrr approved these changes Jan 12, 2022

View reviewed changes

rix0rrr added the pr/do-not-merge This PR should not be merged at this time. label Jan 12, 2022

refactoring findCloudWatchLogGroups to lookup the list of CloudWatch

2c07e8f

log groups from the deployed template rather than by parsing the current template

skinny85 reviewed Jan 12, 2022

View reviewed changes

fixing urlSuffix

87d8d9d

skinny85 approved these changes Jan 12, 2022

View reviewed changes

corymhall removed the pr/do-not-merge This PR should not be merged at this time. label Jan 12, 2022

Merge branch 'master' into corymhall/hotswap-logs

d875b92

mergify bot merged commit a9038ae into aws:master Jan 12, 2022

jumic mentioned this pull request Jan 14, 2022

feat(ec2): add Hpc6a instances #18445

Merged

tmokmss mentioned this pull request Jan 15, 2022

(cli): cdk watch shows "...messages supressed..." logs infinitely #18451

Closed

feat(cli): watch streams resources' CloudWatch logs to the terminal #18159

feat(cli): watch streams resources' CloudWatch logs to the terminal #18159

Conversation

corymhall commented Dec 23, 2021 • edited

gitpod-io bot commented Dec 23, 2021

corymhall commented Dec 23, 2021

skinny85 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skinny85 left a comment

Choose a reason for hiding this comment

corymhall commented Dec 29, 2021

skinny85 left a comment

Choose a reason for hiding this comment

corymhall commented Dec 30, 2021

corymhall commented Dec 30, 2021

skinny85 left a comment

Choose a reason for hiding this comment

skinny85 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rix0rrr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skinny85 Jan 12, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rix0rrr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skinny85 left a comment

Choose a reason for hiding this comment

mergify bot commented Jan 12, 2022

mergify bot commented Jan 12, 2022

aws-cdk-automation commented Jan 12, 2022

AWS CodeBuild CI Report

corymhall commented Dec 23, 2021 •

edited

skinny85 Jan 12, 2022 •

edited