Lambda benchmark on comparing Node.js, ADOT layer

[mugli]

Hey team,

Being a regular user of the OTel collector, Rotel lambda layer looks like an amazing direction for the resource constrained use case. I haven't tried it yet, was curious how the performance difference looks like using Node.js (instead of Python), but haven't seen a benchmark on that.

[1] Any plans to maintain a Node.js lambda benchmark similar to https://github.com/streamfold/python-lambda-benchmark?

Few additional questions related to benchmark,

[2] Saw the rotel lambda extension mentioned in this issue: aws-observability/aws-otel-lambda#228
How does the performance compare with the ADOT layer? I also couldn't find a benchmark (elsewhere) comparing ADOT with OTel lambda, will ADOT performance be in the similar ballpark of the OTel layer? Or, asking differently, any chance of adding ADOT layer to the benchmark comparison? Rotel seems to be the only lambda layer for OTel prioritizing performance first, the expectations to see that comparison here comes from that :)

[3] For auto-instrumentation, Rotel lambda layer needs additional language support extension, where the ADOT layer merged the collector and language support extensions together. Does this difference have impact on cold-start time, or is it just convenience for the ADOT design decision? Any reason not to use the language support extension in the Python lambda benchmark?

[mheffner]

Hey @mugli ,

These are great questions, let me address them separately:

[1] Not at the moment, though we could likely stand something up without a lot of trouble. As I'll explain more below, the goal with the benchmarks was primarily to isolate the impact the collector component has on cold starts, separate from the individual runtimes and/or auto-instrumentation. We do have a Node.js Lambda example app which can give some idea of cold start times on Node.js, but it's not setup as a full benchmark suite.

[2] We didn't initially test against ADOT because the version of the collector it bundles is quite old, relative to the pace of OTel collector releases. The test configuration file we were using were not compatible with it so I wasn't able to get it to start.

Given that ADOT is a downstream packaging of the opentelemetry-lambda distribution, I wouldn't expect a significant difference in performance. However, it wouldn't hurt to figure out the right back-dated config file format and include it in the test matrix anyways.

[3] I don't see a reason for the bundled layers other then convenience. Rotel takes the approach that the OTel Lambda project takes by splitting the collector from the auto-instrumentation layers. In both cases the auto-instrumentation layers use the exec wrapper method to initialize, so they should have similar performance behavior.

We didn't use the auto-instrumentation layers in our benchmarking because we wanted to isolate the performance overhead of the collector separate from the instrumentation. The auto-instrumentation performance can vary a lot based on the complexity of the app and how many dependencies must be instrumented. Some users may also not rely on auto-instrumentation or have runtimes that don't support it. Therefore, we wanted a "clean room" test that compared just the Collector vs. Rotel (and Datadog) apart from auto-instrumentation.