c7n_logexporter serverless implementation not working.

[AbhishekSinha47]

Hi @kapilt ,

For serverless usage of c7n_logexporter:
After following the steps as mentioned in https://github.com/cloud-custodian/cloud-custodian/tree/master/tools/c7n_logexporter , below the series of issues that happened:

1. Error : Runtime.ImportModuleError: Unable to import module 'exporter': No module named 'exporter' : Reason : In the Makefile we were removing all the .py files under Package
   I removed that line. Error was solved.
2. Error : multiple requirements were missing like 'PyYAML' etc in the requirements.txt file, added those and the errors were gone.
3. Now when I'm deploying, I'm getting error that 'handler' is missing in exporter.py . We have mentioned exporter.handler in the cfn.yml template for the Lambda function, but no such handler function exists in the exporter.py file. Can you please give a solution to this??

[AbhishekSinha47]

@kapilt Also, wanted to know: for cross-accounts:

4. through cloud formation template, Makefile deploys Lambda function to the master account alone? and this Lambda should have cross account role?
   OR
5. Cloud formation template needs to deploy lambda function to each of the sub-accounts?

Can you explain the working flow? as in the Readme file nothing seems to be explained.

[kapilt]

4 and 5, the lambda only exists in the destination account, the tool setups cross account streaming destinations for cloud watch log groups using the native features of those groups, ie log groups -> destination -> kinesis -> [optional lambda]

alternatively you can setup to a kinesis firehose destination and send direct to s3/elastic/etc without a lambda.

[kapilt]

the lambda usage for log exporter is pretty specialized around vpc to make, its also completely standalone with no dependencies that aren't already present n lambda, so your references 1,2,3 seem like your doing something wrong.. ie the lambda function here is a reformatter for flow logs to preserve compatibility with log group exports and streaming and is self contained to this module https://github.com/cloud-custodian/cloud-custodian/blob/master/tools/c7n_logexporter/c7n_logexporter/flowdeliver.py

[kapilt]

looking back through the readme, it does seem like its misleading about server less usage, so leaving this open till that is updated to clarify intent there.

[AbhishekSinha47]

Hey @kapilt ! I have followed the readme for the implementations. Here are my list of queries(Correct me if I'm wrong in my understanding):

1. in the cfn.yml file: as handler function its mentioned exporter.handler, but actually the handle function exists in flowdeliver.py. So ideally, it should be flowdeliver.handle in cfn.yml file.
2. The lambda function deployed in the destination/master account should have cross account access role. What permissions should be added to this Lambda role?
3. Even If I'm running the CLI command for multiple accounts: I'm getting error like :

   ERROR:c7n-log-exporter:Error on account xxx-staging err: Log group {'logGroupName':    '/aws/yyyy/xxxx', 'creationTime': 1554383451229, 'metricFilterCount': 0, 'arn': 'arn:aws:logs:us-east-1:xxxxx:log-group:/aws/yyyy/xxxx':*', 'storedBytes': 71066, 'exportStart': datetime.datetime(2019, 4, 4, 13, 10, 51, 229000)} not found.
   

Though the log group /aws/yyyy/xxxx in the xxx-staging account exists.

[elzayat]

@abhishekFreshworks for 2 i think you don't need lambda cross account access as long as you have bucket policy for the master bucket. and lambda execution role in target accounts have access to write to this bucket