Performance issue

[isma3ilovich]

Hi,

I am performing perf testing of my application which integrates opentelemtry v1.0.0 and i have a huge impact of openetelemty on application response time.
I'm using BatchSpanProcessor with the following configuration
options.max_queue_size = 2048; options.schedule_delay_millis = std::chrono::milliseconds(30000); options.max_export_batch_size = 1000;

How to reduce opentelemetry impact ?

Thanks.

[esigo]

Hi, thanks for bringing this up.
Have you checked Performance best practices?

[isma3ilovich]

Hi @esigo Thanks for your reply.

Yes, I only have a few spans by Microservice (3 to 4 spans) and I use the time event for logging.
Using MultiSpanProcessor of BatchProcessor can reduce the overhead?

[esigo]

Hi @isma3ilovich,
What exporter are you using?
Would it be possible to share some timings before and after using OpenTelemetry.

[isma3ilovich]

Hi @esigo ,

I'm using OtlpExporter .

The test case are :

1. Test 1 : 150 transactions per second using SimpleSpanProcessor : response time is more than 1s per transaction
2. Test 2 : 150 transactions per second without OpenTelemetry : response time is about 100ms per transaction
3. Test 3 : 150 transactions per second using BatchSpanProcessor (default parameters): response time is about 400ms per transaction

[esigo]

For a few spans these numbers doesn't make sense to me. We had some fixes for OtlpHttpExporter, it might help to use the new version of OpenTelemetry.
In prod you might need to use sampler to reduce the impact.

I'm not sure if we have benchmark number reported somewhere.
Tagging @lalitb, if you would like to add more here.

[lalitb]

I did a quick benchmark test for otlp exporters using SimpleProcessor on my local machine, and locally running otlp collector, and the result is:

• OTLP gRPC exporter: ~4000 spans exported within ~2.6ms
• OTLP HTTP exporter: ~3200 spans exported within ~5.8ms

Code to benchmark (basically tweaking the otlp example to provide benchmark): https://github.com/open-telemetry/opentelemetry-cpp/compare/main...lalitb:otlp-benchmark-1?expand=1
Detailed results

$ ./example_otlp_grpc
2022-01-04 15:10:53
Running ./example_otlp_grpc
Run on (20 X 2808.01 MHz CPU s)
CPU Caches:
  L1 Data 32K (x10)
  L1 Instruction 32K (x10)
  L2 Unified 256K (x10)
  L3 Unified 20480K (x1)
Load Average: 0.21, 0.34, 0.40
-------------------------------------------------------
Benchmark             Time             CPU   Iterations
-------------------------------------------------------
BM_otlp_test    2689120 ns       632688 ns         1107

$ ./example_otlp_http
2022-01-04 15:11:01
Running ./example_otlp_http
Run on (20 X 2808.01 MHz CPU s)
CPU Caches:
  L1 Data 32K (x10)
  L1 Instruction 32K (x10)
  L2 Unified 256K (x10)
  L3 Unified 20480K (x1)
Load Average: 0.25, 0.35, 0.40
-------------------------------------------------------
Benchmark             Time             CPU   Iterations
-------------------------------------------------------
BM_otlp_test    5882389 ns       889489 ns          801

Each iteration above creates 4 spans. Haven't tested the batch processor, but it should give better results with default config parameters.

@isma3ilovich - Would be interesting if you can figure out which part of the pipeline is causing the performance issue for you. Could it be network latency with otlp-collector?

[isma3ilovich]

@lalitb @esigo thanks for your reply.

I will upgrade to the latest version v1.1.1 then i will supervise network latency to identify which part of pipeline is causing the issue .
but i'm facing the flowing compilation error for v1.1.1 .

In file included from /pwrcard/home/src/inc/pwc_otel_common_tracer.h:13:0, from /pwrcard/home/src/libs/libhsmgrpc/pwc_hsm_service_impl.cc:2: /usr/local/include/opentelemetry/exporters/otlp/otlp_grpc_exporter.h:10:74: fatal error: opentelemetry/proto/collector/trace/v1/trace_service.grpc.pb.h: No such file or directory #include "opentelemetry/proto/collector/trace/v1/trace_service.grpc.pb.h"

I use to compile && cmake -DBUILD_SHARED_LIBS=ON -DWITH_OTLP=ON -DBUILD_TESTING=OFF -DWITH_OTLP_HTTP=OFF .. \ && make install \

[lalitb]

I use to compile && cmake -DBUILD_SHARED_LIBS=ON -DWITH_OTLP=ON -DBUILD_TESTING=OFF -DWITH_OTLP_HTTP=OFF .. \ && make install \

Can you try the latest build, looks like #1131 should be fixing this issue.

[ThomsonTan]

Another option to avoid update local source is try add -DWITH_OTLP_GRPC=ON and make sure it is a clean build.

[isma3ilovich]

Thanks @lalitb & @ThomsonTan .
Now the issue is at link level /bin/ld: cannot find -lopentelemetry_exporter_otlp_grpc
The used command is make -DBUILD_SHARED_LIBS=ON -DWITH_OTLP=ON -DWITH_OTLP_GRPC=ON -DBUILD_TESTING=OFF -DWITH_OTLP_HTTP=OFF install

[lalitb]

Now the issue is at link level /bin/ld: cannot find -lopentelemetry_exporter_otlp_grpc

Can't reproduce this, did you do try a clean build?

[isma3ilovich]

@lalitb it seems that the option WITH_OTLP_GRPC still always off , i changed it in CMakList,now the compilation is OK.
Thanks.