CNCF AWS accounts for public Lambda layer

[wangzlei]

OpenTelemetry Lambda SIG wants to publish public Lambda layer in AWS accounts for integration test, soaking and distro. Lambda users can play OTel by consuming these public Lambda layers freely like downloading java dependencies from Maven repo in Java development.

cc @alolita @mwto @codeboten

[alolita]

@wangzlei I'd like to better understand what the security implications are of providing a public Lambda layer with using a CNCF AWS account vs. using an AWS account that AWS provides. Let's add this to the agenda for the upcoming Wed SIG meeting.

[wangzlei]

OTel Lambda SIG meeting today discussed this topic. Here answer @alolita 's question.

Why OTel Lambda need CNCF account.

From customer's perspective there is not difference between using a CNCF AWS account vs. using an AWS account that AWS provides. The only special is account id is showing in Lambda layer ARN, the pattern is "arn:aws:lambda:<Region>:<Account>:layer:<LayerName>:<Version>"

We known who provides account, who pays the bill and has absolute power. When AWS upstreams OTel Lambda project to OpenTelemetry community, the project lost ADOT(AWS Distro OTel) label and is owned by OpenTelemetry community, AWS would be contributor and maintainer but not the only one. If AWS provides account to OTel Lambda Repo for carrying Lambda layer:

• Standing in OTel's shoe, both code and Lambda layer should have been OTel's assets. But OTel does not own the account, not sure if the account owner would make mistake to break the functionality.
• From AWS's point of view, account is in external Repo. Though account owner can limit the permission of credential, but still cannot control what the credential is used for.
  So, if AWS provides account for OTel Lambda, both OTel and AWS have security concern. The ideal way is OTel Lambda using CNCF account, both code and account are managed by OTel.

[wangzlei]

Cost estimation

To publish public Lambda layer we need a CI/CD workflow to cover integration test, soaking, canary, etc. That needs AWS services Lambda, API Gateway, CloudWatch, Xray and S3.
Monthly bill is about $170, it contains hourly canary test and weekly soaking for up to 30 AWS commercial regions and up to 10 programming language OTel&Lambda may support in the future(at the moment we only support Java/Python/JS/.Net/Go).
please refer the rough cost estimation for AWS account in OTel Lambda: https://calculator.aws/#/estimate?id=8fe783773beb2a0f0cf16b3470d88253b9434282

[wangzlei]

What account does OTel Lambda want

OTel Lambda CI/CD will run integration/soaking/canary test by AWS services Lambda, XRay, CloudWatch, CloudFormation, S3, API Gatewa, deploy public Lambda layer in AWS Lambda. If OTel Lambda will get an IAM user derived from a shared CNCF-OTel account, the easiest way is to have AdministratorAccess permission, if it is against security policy, have to to limit the permission precisely, please grant these 6 services FullAccess permissions.

Because AWS Lambda layer is regional resource, has to be deployed to every region respectively. CN regions(Beijing and Ningxia) are isolated from normal regions, CN AWS account and normal AWS account cannot access with each other, we need 2 accounts, one for CN regions and one for normal regions. As the best practice we also want to separate Test and Prod if possible, the test account is for integration/soaking/canary test, the Prod account is only for carrying Public Lambda Layer and run smoke test before change to be public access.

To sum up, in the best practice we need 4 accounts(IAM users) for:

1. standard region Test
2. standard region Prod
3. CN region Test
4. CN region Prod

The simple solution is combining test and prod, we need at least 2 accounts(IAM users) for:

1. standard region
2. CN region

[SergeyKanzhelev]

was it resolved? Assigned to @mtwo and @alolita to resolve

[trask]

@wangzlei @open-telemetry/lambda-extension-maintainers is this still something that the OTel Lambda SIG would like to set up? is the cost estimate above still valid? thx!

[Aneurysm9]

@trask I'll look to validate that estimate, but it seems roughly correct. I do think it would be good to have access to CNCF-owned accounts for CI and releases.

[trask]

@Aneurysm9 I noticed that the CloudFormation service is mentioned above, but is not included in the estimate, can you update the estimate to include that as well if it's needed?

[Aneurysm9]

CloudFormation is a free service. There is no charge for using it, only for the resources that it is used to deploy.

It's been somewhat hard to accurately validate the estimate using our existing testing environment as it is shared with other test infrastructure that would not be necessary for the Lambda SIG. The only significant Lambda-related expense that I've been able to identify as out-of-line with this estimate is related to provisioned concurrency test functions that were not properly cleaned up following testing. Otherwise, all of these expense estimates appear appropriately conservative with the potential exception of S3 which may be on the order of $10/mo instead of $2.50, depending on retained storage size.

[cartersocha]

I asked @tedsuo to open a service desk ticket with the CNCF. Apparently only people on this list have access to do that.

[mtwo]

Filed a service desk ticket to track this: https://cncfservicedesk.atlassian.net/servicedesk/customer/portal/1/CNCFSD-1556?created=true

[cartersocha]

@Aneurysm9 @wangzlei do we need 2 or 4 accounts?

The 2 account approach seems better to me but open to suggestions

[mtwo]

Updated the ticket to reflect this

[cartersocha]

Could we please get a status update on the ticket? @carlosalberto, @tedsuo, @mtwo ?

[mtwo]

Following up, thanks for the ping! They asked if our cost estimates listed here are still correct and are waiting for us to reply.

[cartersocha]

I think Anthony addressed this in his previous comment but we'll double check. @Aneurysm9 @bryan-aguilar could yall please confirm the cost estimates or rerun the numbers based on current usage?

[Aneurysm9]

Nothing has changed from a cost estimate perspective since this comment.

[cartersocha]

Thanks Anthony! @mtwo confirmed it is the same!

[mtwo]

Replied back on the ticket!

[mtwo]

Got a reply back from the CNCF. They've asked if we'd like them to apply this to our existing AWS account: cncf-opentelemetry-aws@lists.cncf.io.

Two questions:

• Do we want to use the existing account for this? I assume yes, but I wanted to double-check.
• Can we confirm that we have access to the existing account? @Aneurysm9 @alolita @cartersocha or anyone else, do you have the credentials?

[cartersocha]

No credentials on my end!

[Aneurysm9]

I do not have credentials for accessing that account. I'm fine with using an existing account if it is the path of least resistance.

[mtwo]

Turns out that the CNCF manages the account for us. I'll tell them to make the necessary changes!

[mtwo]

I've asked them to give access to @cartersocha , @Aneurysm9 , and @codeboten

[cartersocha]

We haven’t received any emails yet

[cartersocha]

@mtwo the delay on getting access the account is starting to really affect sig output and ability to deliver. Any way we can push this along?

[mtwo]

Interesting, following up with the CNCF now

[mtwo]

I've pinged the ticket, I'll post here as soon as I get a response

[mtwo]

Just got the credentials, I'm sharing them with each of you via the CNCF Slack