How to instrument NGINX with OpenTelemetry

[harshalschaudhari]

Want to enable the OpenTelemetry in ingress-nginx

How to instrument NGINX with OpenTelemetry

I followed the documentation (https://opentelemetry.io/blog/2022/instrument-nginx/) to integrate Jaeger tracing with Kubernetes ingress-nginx, but I'm encountering an error. Could you please help me resolve it?

Error

 2024/05/16 06:46:05 [emerg] 1#1: dlopen() "/opt/opentelemetry-webserver-sdk/WebServerModule/Nginx/1.16.0/ngx_http_opentelemetry_module.so" failed (/opt/opentelemetry-webserver-sdk/WebServerModule/Nginx/1.16.0/ngx_http_opentelemetry_module.so: cannot open shared object file: No such file or directory) in /etc/nginx/nginx.conf:1
nginx-1                 | nginx: [emerg] dlopen() "/opt/opentelemetry-webserver-sdk/WebServerModule/Nginx/1.16.0/ngx_http_opentelemetry_module.so" failed (/opt/opentelemetry-webserver-sdk/WebServerModule/Nginx/1.16.0/ngx_http_opentelemetry_module.so: cannot open shared object file: No such file or directory) in /etc/nginx/nginx.conf:1

Steps as below

Install the module for NGINX

Start from an empty directory. Create a file called Dockerfile and copy the following content into it:

FROM nginx:1.23.1
RUN apt-get update ; apt-get install unzip
ADD https://github.com/open-telemetry/opentelemetry-cpp-contrib/releases/download/webserver%2Fv1.0.3/opentelemetry-webserver-sdk-x64-linux.tgz /opt
RUN cd /opt ; unzip opentelemetry-webserver-sdk-x64-linux.tgz.zip; tar xvfz opentelemetry-webserver-sdk-x64-linux.tgz
RUN cd /opt/opentelemetry-webserver-sdk; ./install.sh
ENV LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/opentelemetry-webserver-sdk/sdk_lib/lib
RUN echo "load_module /opt/opentelemetry-webserver-sdk/WebServerModule/Nginx/1.23.1/ngx_http_opentelemetry_module.so;\n$(cat /etc/nginx/nginx.conf)" > /etc/nginx/nginx.conf
COPY opentelemetry_module.conf /etc/nginx/conf.d

Next, create another file called opentelemetry_module.conf and copy the following content into it:

NginxModuleEnabled ON;
NginxModuleOtelSpanExporter otlp;
NginxModuleOtelExporterEndpoint localhost:4317;
NginxModuleServiceName DemoService;
NginxModuleServiceNamespace DemoServiceNamespace;
NginxModuleServiceInstanceId DemoInstanceId;
NginxModuleResolveBackends ON;
NginxModuleTraceAsError ON;

With the Dockerfile and NGINX config in place, build your docker image

# Build
docker build -t nginx-otel-arm:v1 --platform linux/arm64 .

# Map a tag
docker tag nginx-otel-arm:v1 harshalschaudhari/nginx-otel-arm:v1.0

# Push docker image
docker push harshalschaudhari/nginx-otel-arm:v1.0

and run the container using docker-compose up:

version: '3.8'
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - '16686:16686'
  collector:
    image: otel/opentelemetry-collector:latest
    command: ['--config=/etc/otel-collector-config.yaml']
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
  nginx:
    image: harshalschaudhari/nginx-otel-arm:v1.0
    volumes:
      - ./opentelemetry_module.conf:/etc/nginx/conf.d/opentelemetry_module.conf
    ports:
      - 8080:80
  user-service-api:
    ports:
      - '5010:8080'
      - '5011:443'
    container_name: user-service-api
    image: harshalschaudhari/user-service-api:v1.0
  calculator-service-api:
    ports:
      - '5020:8080'
      - '5021:443'
    container_name: calculator-service-api
    image: harshalschaudhari/calculator-service-api:v1.0

docker-compose up, nginx log as below

nginx-1                 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
nginx-1                 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
nginx-1                 | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
nginx-1                 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx-1                 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx-1                 | /docker-entrypoint.sh: Configuration complete; ready for start up
nginx-1                 | 2024/05/16 06:46:05 [emerg] 1#1: dlopen() "/opt/opentelemetry-webserver-sdk/WebServerModule/Nginx/1.16.0/ngx_http_opentelemetry_module.so" failed (/opt/opentelemetry-webserver-sdk/WebServerModule/Nginx/1.16.0/ngx_http_opentelemetry_module.so: cannot open shared object file: No such file or directory) in /etc/nginx/nginx.conf:1
nginx-1                 | nginx: [emerg] dlopen() "/opt/opentelemetry-webserver-sdk/WebServerModule/Nginx/1.16.0/ngx_http_opentelemetry_module.so" failed (/opt/opentelemetry-webserver-sdk/WebServerModule/Nginx/1.16.0/ngx_http_opentelemetry_module.so: cannot open shared object file: No such file or directory) in /etc/nginx/nginx.conf:1
nginx-1 exited with code 1

Additional Information

I have tried the multiple options as well such as https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentelemetry/

How to set up OpenTelemetry module
also tried https://docs.nginx.com/nginx/admin-guide/dynamic-modules/opentelemetry/

Expectation: Trace should be start from Nginx span.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[longwuyuan]

/remove-kind feature
/kind support

From the issue description you have provided, its not certain if you reported this issue in the context that is related to this ingress-nginx controller.

You wrote that you tried https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentelemetry/ so you will deep dive into those specifics as copy/paste data that shows that this documented link is broken.

Your description of creating a dockerfile and and a image from that dockerfile is not related to this project

(1) So first please look at the new bug report template and answer those questions from the template by editing the issue description here.

(2) Secondly deep dive into this document https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentelemetry/ and copy/paste all the information like configuration, state, logs, screenshots and describe your actions done to troubleshoot. That will give info to a reader here as to what is wrong with this project's code and releases.

Once you have done the above, please re-open this issue. Normally many issues stay open without any actionable item in them and they are piling up here so harder to track of real actionable issues. I will close this issue for now so you can re-open after the required info is added.

/triage needs-information
/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to this:

/remove-kind feature
/kind support

From the issue description you have provided, its not certain if you reported this issue in the context that is related to this ingress-nginx controller.

You wrote that you tried https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentelemetry/ so you will deep dive into those specifics as copy/paste data that shows that this documented link is broken.

Your description of creating a dockerfile and and a image from that dockerfile is not related to this project

(1) So first please look at the new bug report template and answer those questions from the template by editing the issue description here.

(2) Secondly deep dive into this document https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/opentelemetry/ and copy/paste all the information like configuration, state, logs, screenshots and describe your actions done to troubleshoot. That will give info to a reader here as to what is wrong with this project's code and releases.

Once you have done the above, please re-open this issue. Normally many issues stay open without any actionable item in them and they are piling up here so harder to track of real actionable issues. I will close this issue for now so you can re-open after the required info is added.

/triage needs-information
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.