aws - Lambda output_dir - synchronize variable interpolation with notify message and metadata.json

[llange]

Describe the feature

Context

We're using Lambda deployment mode of policies, on AWS.
We're deploying on multiple accounts with c7n-org.
The execution output is stored in a deeply hierarchized S3 bucket:

output_dir: s3://MY_S3_BUCKET/c7n-output/{account_id}/{region}/{policy_name}/{now:%Y/%m/%d/%H/%M}/

At each execution, the following items are stored:

• custodian-run.log
• metadata.json
• resources.json
• parse_errors.json (sometime)
• opted_out.json (sometime)
• denied.json (sometime)

Feature Request

1. In the notification part of the policies, we would like to send a message containing a link to this bucket, (and/or be able to access the same variables available in the output_dir variable expansion logic).
2. When building reports based on the execution output (files in output_dir, mainly metadata.json and resources.json) we also want to be able to reference this output_dir path.
3. More generally, we would like that (most of) the variables available for output_dir, the notify message, and the metadata.json file are the same (same value - and maybe in the future same name also).

This way, it's easier to "cross-reference" the same variables in those 3 different contexts.

Use-case

• We have a web interface that is able to dynamically build an execution report from an execution output, given a path to an S3 execution report. (e.g: https://my-report-website/view#s3://MY_S3_BUCKET/c7n-output/11111111/us-east-1/pol1/2022/11/28/20/45/ )
• In the notify part of the policies, we send message notifications (slack, teams, email, ...) with (almost) only a link to the dynamic report. (like the example link above)
• For the moment, it's not possible to base the output_dir on the date, because now in output_dir is not equals to the execution_start date in the notification message (they can differ by a few seconds ; they are not referencing strictly the same moment)
• For the moment, it's not possible to base the output_dir on the execution id, as uuid in output_dir is generated independently of execution_id of the notify message and do not have the same value.

Variables

If I'm not mistaken, the variables available for output_dir variable expansion are:

• account_id
• now : utcnow() at the moment of created the output_dir path via get_output_path
• policy_name
• region
• uuid : a newly-generated UUID, that seems unrelated to other UUIDs available

For the notify message sent in the SQS, the following fields are available in the message:

• account_id
• account
• action
• event
• execution_id
• execution_start
• partition
• policy
• region
• resources
• version

In the output metadata file metadata.json, we can access the following similar information:

• api-stats
• config (containing account_id, region, output_dir in template (variables not replaced) form, ...)
• execution.duration
• execution.end_time
• execution.id
• execution.start
• metrics
• policy (containing policy.name, etc...)
• sys-stats
• version

Summary of the proposed changes / additions

output_dir	notify SQS message	metadata.json	comment
-	-	execution.duration	Could be useful to add to notify message.
-	-	metrics	N/A
-	-	sys-stats	N/A
-	-	api-stats	N/A
-	account	-	Can be useful to add to metadata.json / not sure for output_dir
-	execution_id	execution.id	Should be added to output_dir.
-	execution_start	execution.start	Should be added to output_dir.
-	partition	-	May be add it to metadata.json ?
-	policy	-	N/A
-	region	config.region	OK
-	resources	-	N/A
-	version	version	OK
account_id	account_id	config.account_id	OK
now	-	-	(not exactly the same as execution.start)
policy_name	policy.name	policy.name	OK
uuid	-	-	(not exactly the same as execution.id)
-	output_dir_final	output_dir_final	(The full path - variables resolved)

Please comment on the usefulness, and I'll try to do the implementation.

Extra information or context

No response