Upgrading to 3.77.0 & Python 3.9 Results in Import Error

[kgochenour]

Describe what happened:
We use the DD provided cloudformation stack to deploy the Lambda Log forwarder. However it was on version 3.59.0. We upgraded to 3.77.0 using the latest.yaml template. The update went through, but suddenly the lambda failed all invocations with the following:

[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': cannot import name '_rand' from 'ddtrace.internal' (/var/task/ddtrace/internal/__init__.py)
Traceback (most recent call last):

Manually changing the run time from Python 3.9 to 3.8 fixed this issue.

As a result we rolled back to 3.73.0 which is the last Python 3.8 deployment, and everything is working OK.

Describe what you expected:
Expected to run latest.yaml Cloudformation template and the Log Forwarder to work.

Steps to reproduce the issue:
Deployed latest.yaml as an update to our CloudFormation stack.

[RaphaelAllier]

Hello,

I've opened a ticket to track this internally, in the meantime 3.73 should be stable

[RaphaelAllier]

Hello @kgochenour, when updating did the Layer version get updated as well ?
I redeployed the latest version from scratch using the CF template in our sandbox and didn't encounter this import issue.
EDIT: Installation from scratch worked but updating through CF with the latest.yaml yields your issue, we're looking into it

[kgochenour]

@RaphaelAllier I opened a ticket with DD (internal ticket 1289207) on this, and we found that the CloudFormation template was not actually updating the Lambda code. The stack would change the Tag for the forwarder version, and did update the runtime. However in the logs, and in DD itself, the lambda reported as being on an old version.

I found that by running the cloudformation template update, then manually updating the lambda code with the datadog forwarder code, that it would work. But this meant that all future updates would need to be done this way. Which is odd.

My internal ticket closed because I did not respond due to being away. But basically we werent able to figure out why this happened.

[RaphaelAllier]

@kgochenour Hello. Following up as I was on PTO, we identified internally that there was indeed an issue with the Cloudformation update path. We're trying to pinpoint the issue and will post updates when we have a fix.

The issue seems to be located in the CF update path. Newer installations work fine with InstallAsLayer set to true as the update only changes the layer (which contains the whole Forwarder code). However, older installations may see a conflict of dependencies that we're trying to investigate.
Thanks for the ticket number, I'll also have a look there

[RaphaelAllier]

After some troubleshooting I think we've narrowed down the issue here

To recap:

• Version 3.73.0 runs in Python 3.8
• Latest version (3.83.0) runs in Python 3.9
  New installations of the Datadog Forwarder don't contain code in the Lambda itself, but in a layer that contains all the business logic. This is controlled in the Cloudformation template by the parameter InstallAsLayer set to true by default. We recommend using this approach, although a non layered installation works just as well.

We tested various installations/setup and Cloudformation upgrade patterns.

• Using CF, deploy a 3.73.0 Forwarder with a non layered installation (Code is packaged in a .zip file on the Lambda). CF updates to the latest template work fine, and both 3.73.0 and latest version send logs to our intake.
• Using CF, deploy a 3.73.0 Forwarder with a layered installation (Code is packaged in a layer). CF updates to the latest template work fine, and both versions send logs to our intake
• Similar to what is described in the support ticket in this issue, we tried mixing Cloudformation updates with manual .zip code updates. Deploying a 3.73.0 Forwarder with the layer approach through CF and manually updating the .zip code to the latest version (picking a release .zip) will cause dependency conflicts to appear and show the error mentioned in this issue.

When the Forwarder is deployed with its code packaged in the layer, Cloudformation updates will only make a call to UpdateFunctionConfiguration, which won't update the source code of the Lambda. If the function code has been manually updated, CF won't take that into account when the stack is re-applied, which may lead to errors.

Additionally, a workaround to sanitise the installation is to reapply the CF template twice, once with InstallAsLayer: false and the second time with InstallAsLayer: true, which should clean up the Lambda's source code.

Alternatively, fresh installation (not Cloudformation upgrades) of the latest version of the template are functional

[anthonyangel]

Hi @RaphaelAllier Do you have an upgrade guide? From reading the docs it looks like InstallAsLayer defaults to true, but my reading of the post above suggests that it should be possible to upgrade with the following steps:

1. deploy 3.73, InstallAsLayer:true (ie default)
2. deploy 3.73, InstallAsLayer:false
3. deploy 3.73, InstallAsLayer:true
4. deploy 3.91, InstallAsLayer:true

Please can you confirm, or provide alternate instructions?

[RaphaelAllier]

Hello @anthonyangel
If your installation is currently using the version 3.73.0 and may have been manually upgraded in the past, steps to "sanitise" the installation (if needed, regular CF upgrades should work fine by default) should be:

Additional steps to make sure the forwarder code is completely packaged in the layer only, avoiding dependency conflicts

template URL: https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/3.73.0.yaml

• deploy same CF template version (3.73.0) with InstallAsLayer: false
• deploy same CF template version (3.73.0) with InstallAsLayer: true

From this step, your forwarder shouldn't have any code except in the Lambda layer attached to it (which is the standard way of deploying the Datadog Forwarder)

Regular upgrade process

template URL: https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml

• upgrade to the latest available CF template

To clarify, regular CF upgrades should always work fine. The errors listed in this issue we think resulted from a mix of CloudFormation upgrades and manual upgrades (like getting the last zip code from the latest release and manually installing it in the AWS console)