Elastic APM python agent is not capturing traces/transactions for Azure Python Functions.

[fxdgear]

Describe the bug:

I read the docs and followed the instructions. https://www.elastic.co/guide/en/apm/agent/python/current/azure-functions-support.html

Specifically the part about using extensions https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference-python?tabs=asgi%2Capplication-level&pivots=python-mode-configuration#using-extensions

where I added the PYTHON_ENABLE_WORKER_EXTENSIONS=1 in my app settings.

Then when I added this line of code:

from elasticapm.contrib.serverless.azure import ElasticAPMExtension

ElasticAPMExtension.configure()

To my function_app.py which is where all the other functions get registered. I added it at the top of the file, before registering functions and after the imports.

I have verified that the env vars are getting populated into my function by logging them and seeing them in the logs.

I have also seen some dependencies show up. For example when I was testing my app function I was curling the elastic apm url and getting back a 200. But then when I looked at the APM screen in kibana I could see that my apm service was showing up (yay) but it had no transactions. It only had a dependency on the requests.get request I made to the elastic apm endpoint.

SO I know that the apm agent is doing something and connecting to elastic, but it's not doing it correctly and it's not able to capture traces.

To Reproduce

see description

Environment (please complete the following information)

• OS: Linux via azure functions
• Python version: 3.10
• Framework and version [e.g. Django 2.1]: Azure Functions
• APM Server version: v8.13.2
• Agent version: elastic_apm-6.22.0-py2.py3

We are using the blueprint framework in the azure function sdk https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference-python?tabs=asgi%2Capplication-level&pivots=python-mode-decorators#blueprints

[xrmx]

Thanks for reporting, any chance you can check if this is something Blueprint specific?

[fxdgear]

Thanks for reporting, any chance you can check if this is something Blueprint specific?

I'm not sure how I would do that tbh. which is why i came here to ask :)

[xrmx]

@fxdgear I mean to try tracing an usual azure function and not one that comes from a blueprint.

[fxdgear]

@xrmx I want to also note that when I run the app locally using func start and I have the APM env vars added to my local.settings.json it does work. It is only not working when deployed to azure.

so I can verify that it does work with blueprint. But I don't know what's happening in azure itself to keep it from working.

[xrmx]

I'm trying to replicate the issue but I'm not even able to have azure run the latest version of the function code I've uploaded.

[xrmx]

Using azure cli tools deploy is fine and I get transactions recorded. This is my function:

import azure.functions as func
import datetime
import json
import logging

from elasticapm.contrib.serverless.azure import ElasticAPMExtension

app = func.FunctionApp(http_auth_level=func.AuthLevel.ANONYMOUS)

ElasticAPMExtension.configure()

bp = func.Blueprint()

@bp.route(route="hello")
def hello(req: func.HttpRequest) -> func.HttpResponse:
    has_client = ElasticAPMExtension.client is not None
    logging.info(f'Python HTTP trigger function processed a request. has_client={has_client}')

    name = req.params.get('name')
    if not name:
        try:
            req_body = req.get_json()
        except ValueError:
            pass
        else:
            name = req_body.get('name')

    return func.HttpResponse(f"Hello, {name}. This HTTP triggered function executed successfully. has_client={has_client}")

app.register_functions(bp)

And I got the transactions recorded:

[fxdgear]

@xrmx That's awesome.

Can you do one last test and create a function_app.py which does the registering of you functions.
And then pu your hello in a separate file.

I added the elasticAPMExtention.configure() to the function_app.py file.

# function_app.py

import azure.functions as func
from bp_hello import bp as hello_bp
from elasticapm.contrib.serverless.azure import ElasticAPMExtension
ElasticAPMExtension.configure()

app.register_functions(hello_bp)

And then your function would be in

# bp_hello.py

import azure.functions as func
import datetime
import json
import logging

bp = func.Blueprint()

@bp.route(route="hello")
def hello(req: func.HttpRequest) -> func.HttpResponse:
    # not sure what to do here to check for this? 
    has_client = ElasticAPMExtension.client is not None
    logging.info(f'Python HTTP trigger function processed a request. has_client={has_client}')

    name = req.params.get('name')
    if not name:
        try:
            req_body = req.get_json()
        except ValueError:
            pass
        else:
            name = req_body.get('name')

    return func.HttpResponse(f"Hello, {name}. This HTTP triggered function executed successfully. has_client={has_client}")

[fxdgear]

oh I think I know what I did wrong. I was adding the elasticapm capture span decorator at the top of my function.

since it's already instrumented using the configure I don't need to decorate my functions.