Codeguru Profiler Fails with Custom Java Runtime?

[brcolow]

I am getting the following sequence of log messages when attempting to run CodeGuru on my custom java runtime which can be found here: https://github.com/brcolow/lambda-java-runtime

START RequestId: 65e66e2e-3e90-4a6f-8900-79faef4cc681 Version: $LATEST 
Apr 03, 2021 9:11:32 PM software.amazon.codeguruprofilerjavaagent.LambdaProfilerUtils attemptToCreateProfiler 
INFO: CodeGuru profiling group: as2 in AWS region: us-west-2 will be used for profiling your lambda function 
Apr 03, 2021 9:11:34 PM software.amazon.codeguruprofilerjavaagent.Profiler startWithProfilingCommand 
INFO: Starting the profiler : Profiler{profilingGroupName=as2, isRunning=false, isProfiling=false} 
Apr 03, 2021 9:11:34 PM software.amazon.codeguruprofilerjavaagent.ProfilingCommandExecutor scheduleProfiling 
INFO: Profiling scheduled, sampling rate is PT1S 
Apr 03, 2021 9:11:35 PM software.amazon.codeguruprofilerjavaagent.ProfilingCommandWithPause isFleetTypeIncorrect
INFO: CodeGuru Profiler agent was found to be running on a compute platform other than AWS Lambda. Profiling will be terminated. Please refer to CodeGuru Profiler's documentation to learn more about how to profile AWS Lambda functions
Apr 03, 2021 9:11:35 PM software.amazon.codeguruprofilerjavaagent.ProfilingCommand flush
INFO: Dropping the profile as it could not be reported 

It is running on AWS Lambda and I have the two environment variables properly set, and the permissions set to the same thing as the lambda's execution role.

I am invoking the profiler as follows:

public void handleRequest(InputStream input, OutputStream output, TestContext context) throws IOException {
                response = LambdaProfiler.profile(input, null, (i, o) -> {
                    return handleInboundMessage(new HttpEvent(event.getHttpMethod(),
                            headers, base64EncodedBodyOptional.get(), event.getPath()), context);
                });
}

I have found the error in the source code, but I have no idea what FleetInfo is or how it is set:

protected boolean isFleetTypeIncorrect() {
        FleetInstanceType fleetType = (FleetInstanceType)this.profilerFinalParameters.getAgentMetadata().getFleetInfo().map(FleetInfo::getFleetInstanceType).orElse(FleetInstanceType.Unknown);
        if (!FleetInstanceType.AWSLambda.equals(fleetType)) {
            LOG.info("CodeGuru Profiler agent was found to be running on a compute platform other than AWS Lambda. Profiling will be terminated. Please refer to CodeGuru Profiler's documentation to learn more about how to profile AWS Lambda functions");
            return true;
        } else {
            return false;
        }
}

I wonder if maybe it is because of the AWS_EXECUTION_ENV variable being different on a custom runtime (i.e. provided versus java11 - though not sure why codeguru-profiler-java-agent is checking that).

Thank you.

[pandpara]

Hi @brcolow!

Thanks for reaching out!

Can you please confirm a few settings in your lambda function?

1. AWS_EXECUTION_ENV is set in your lambda setup. You can read the environment variable and print it out and see if its set.
2. AWS_LAMBDA_FUNCTION_MEMORY_SIZE is set in your lambda setup. You can read the environment variable and print it out and see if its set.

Thanks!

[brcolow]

@pandpara Thanks for the response.

Here are the requested environment variables:

AWS_EXECUTION_ENV: null
AWS_LAMBDA_FUNCTION_MEMORY_SIZE: 512

The first one looks like it could be the culprit. What should I set it to in my custom java runtime so that codeguru knows it's on AWS Lambda?

[pandpara]

Hi @brcolow !

AWS_EXECUTION_ENV is a reserved env variable[1], you cannot set it.

Yes, and we do depend on value of this environment variable to decide whether you are using the correct compute platform(Lambda in your case)

1. https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html#configuration-envvars-runtime

[brcolow]

@pandpara Okay. So is there anyway to use codeguru profiler with a custom (java) runtime?

[brcolow]

I see that it is being set by the default AWS Java runtime: https://github.com/aws/aws-lambda-java-libs/blob/311f97cc47b414633c62ee96d9df3e8404e14d69/aws-lambda-java-runtime-interface-client/src/main/java/com/amazonaws/services/lambda/runtime/api/client/util/EnvWriter.java

Why can I not set it in my custom runtime? I understand it is reserved for Lambda's that run on the default runtimes, but it shouldn't be reserved for custom runtimes? Or perhaps there would be a better way for codeguru profiler to check if it's on Lambda but a custom runtime?

[pandpara]

You can try to set it actually!

[brcolow]

So I now have my Lambda reporting the following:

AWS_EXECUTION_ENV: AWS_Lambda_java11
AWS_LAMBDA_FUNCTION_MEMORY_SIZE: 512

But I am still getting the same error:

INFO: CodeGuru Profiler agent was found to be running on a compute platform other than AWS Lambda. Profiling will be terminated. Please refer to CodeGuru Profiler's documentation to learn more about how to profile AWS Lambda functions

Are you sure that is how codeguru profiler determines it is on AWS Lambda?

[brcolow]

@pandpara Any ideas?

[brcolow]

Hmmm..looking at the decompiled source code more closely I wonder if it is because getInvokedFunctionArn of the Context object is null. Let me experiment...

[brcolow]

Yep. That was it. So to summarize - in order to use codeguru profiler agent on a Java custom runtime the following three conditions must be met:

1.) AWS_EXECUTION_ENV environment variable must be set to AWS_Lambda_java8 or AWS_Lambda_java11.
2.) AWS_LAMBDA_FUNCTION_MEMORY_SIZE environment must be set (to some number - ideally the memory size of the lambda function).
3.) The Context object passed to LambdaProfiler.profile must be non-null and must implement the getInvokedFunctionArn method - which returns the correct ARN. This can be set in the runtime by fetching Lambda-Runtime-Invoked-Function-Arn HTTP header from the /runtime/invocation/next resource.

Thanks for your help.

[brcolow]

Although codeguru profiler does now run and I have run the Lambda many times (in 5 minute intervals as according to docs) it still says setup required - I wonder if this is because of the custom runtime aspect?

[pandpara]

Hi @brcolow

Can you provide the profiling group name? I can check if we have gotten the profile or not.

[brcolow]

You want the profiling group ARN? Not sure what "name" you are referring to? Thank you.

[pandpara]

Yes, profiling group ARN would suffice. It has profiling group name in it.

[brcolow]

@pandpara

arn:aws:codeguru-profiler:us-west-2:961619659401:profilingGroup/as2

Thank you!

[brcolow]

@pandpara Did you have any time to look at this?

[brcolow]

@pandpara Ping :)

[pandpara]

Hi @brcolow

I don't see any profile submitted yet. Do you see from the logs that a profile has been submitted successfully?

[brcolow]

@pandpara Do you know if running on a custom runtime is supported? Maybe we should start there?

[mirelap-amazon]

Hi, sorry for the late traction on this.

Our documentation does not state that we support custom runtimes and I'm not aware of any testing for it. Thanks for all your findings, this is helpful for us.

Either if not working, profiler should log if it report or not. Could you please enable more logs and let us know what the agent prints? Maybe something like this. To enable the logs, enable the TRACE for software.amazon.codeguruprofilerjavaagent, documented here.

[mirelap-amazon]

Closing due to inactivity and due to the fact that our documentation doesn't specify yet anything about custom runtime. If interested again, please open a new issue, and sorry for the late traction on this. Thank you.