Add OTEL tracing export

[ibizaman]

Hello,

I was curious how interested you are in adding tracing with the OTEL libraries. I know we already have tracing thanks to PHP’s xdebug and I have already used it and it does the job. But the experience could be so much better.

At work we use the Golang OTEL library and, although I wasn’t convinced in the beginning, I must admit having traces flowing automatically from our apps to our Grafana instance (using Tempo) and having logs, metrics and traces all linked together is really, really great.

I checked the PHP OTEL library and they are even more mature than the Go ones. Also, it looks like we can have automatic instrumentation like xdebug does.

I did not try implementing anything yet and instead of dumping a PR out of the blue, I wanted to know if you were interested in having a PR adding tracing using the PHP OTEL library. If so, I’ve got no timeline guarantees but I’ll start work on this.

Btw, this all came from me wanting to squeeze out more performance from the WebDAV Nextcloud implementation.

EDIT: looking at some of the bugs and issues raised here and in the forum, I’m convinced a lot of these could be diagnosed thanks to traces.

EDIT2: also I’m maybe not seeing the full picture here. How do ppl usually track down performance issues in Nextcloud? So there’s xdebug, some amount of monitoring metrics, some logs, then more monitoring on external things used by Nextcloud like disk I/O, CPU and memory usage, etc. Apart from xdebug, I don’t see these as telling you exactly where the issue lies but instead as helpers to pinpoint where to look. Am I missing something?

[jpb0418]

I went down the rabbit hole trying to get traces and I got it for the most part I think. Had to build a custom image and some extra files in nextcloud but overall it's not to bad. I get traces from traefik all the down to redis/db/io/php calls. Not a clue if someone else has gotten traces working but I could not find a good write up. I'm still not %100 sure if everything is needed, lots of random stuff for testing but I did some cleanup and found things that did and didn't break and it still works so I'm not touching it. It also added some cookies and headers for long term session tracking and per click tracing to see all calls made from clicking a button and traceid in header for easier debugging

FROM nextcloud:32.0.3-fpm-alpine

RUN echo "@testing https://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories

RUN apk add --no-cache wget ca-certificates git \
 && wget -q -O /usr/local/bin/install-php-extensions https://github.com/mlocati/docker-php-extension-installer/releases/latest/download/install-php-extensions \
 && chmod +x /usr/local/bin/install-php-extensions \
 && apk add --no-cache --virtual .build-deps autoconf gcc g++ make libtool pkgconfig \
 && /usr/local/bin/install-php-extensions opentelemetry \
 && apk del .build-deps \
 && rm -f /usr/local/bin/install-php-extensions

COPY --from=composer:2 /usr/bin/composer /usr/local/bin/composer

WORKDIR /otel
RUN /usr/local/bin/php -m | grep -i opentelemetry

RUN /usr/local/bin/php /usr/local/bin/composer require --no-interaction --no-progress --prefer-dist \
    open-telemetry/sdk \
    open-telemetry/exporter-otlp \
    open-telemetry/opentelemetry-auto-io \
    open-telemetry/opentelemetry-auto-pdo \
    open-telemetry/opentelemetry-auto-doctrine \
    open-telemetry/opentelemetry-auto-guzzle \
    open-telemetry/opentelemetry-auto-psr18 \
    open-telemetry/opentelemetry-auto-curl \
    open-telemetry/opentelemetry-auto-psr3 \
    mismatch/opentelemetry-auto-redis \
 && /usr/local/bin/php /usr/local/bin/composer dump-autoload -o \
 && rm -rf /root/.composer/cache

---

opentelemetry.ini: |
   ; OpenTelemetry PHP extension configuration
   auto_prepend_file=/otel/prepend.php
   opentelemetry.display_warnings=Off
   opentelemetry.validate_hook_functions=Off
   opentelemetry.attr_hooks_enabled=Off

  prepend.php: |
    <?php
    if (PHP_SAPI === 'cli') { return; }

    require_once '/otel/vendor/autoload.php';

    $uri = $_SERVER['REQUEST_URI'] ?? '';

    function uuidv4(): string {
      $b = random_bytes(16);
      $b[6] = chr((ord($b[6]) & 0x0f) | 0x40);
      $b[8] = chr((ord($b[8]) & 0x3f) | 0x80);
      $hex = bin2hex($b);
      return sprintf('%s-%s-%s-%s-%s',
        substr($hex, 0, 8),
        substr($hex, 8, 4),
        substr($hex, 12, 4),
        substr($hex, 16, 4),
        substr($hex, 20, 12)
      );
    }

    function isSecureRequest(): bool {
      $xfp = strtolower($_SERVER['HTTP_X_FORWARDED_PROTO'] ?? '');
      if ($xfp === 'https') return true;
      if (!empty($_SERVER['HTTPS']) && $_SERVER['HTTPS'] !== 'off') return true;
      if (($_SERVER['SERVER_PORT'] ?? '') === '443') return true;
      return false;
    }

    // Heuristic for top-level document navigations
    $secFetchDest = strtolower($_SERVER['HTTP_SEC_FETCH_DEST'] ?? '');
    $accept       = strtolower($_SERVER['HTTP_ACCEPT'] ?? '');
    $isDocumentNav = ($secFetchDest === 'document')
      || (str_contains($accept, 'text/html') && (($_SERVER['REQUEST_METHOD'] ?? 'GET') === 'GET'));


    $cookieDomain = 'DOMAIN';
    $isSecure = isSecureRequest();

    // 1) session cookie (long-lived)
    $sessionId = $_COOKIE['session_trace_id'] ?? '';
    if ($sessionId === '') {
      $sessionId = uuidv4();
      @setcookie('session_trace_id', $sessionId, [
        'expires'  => time() + 60*60*24*30, // 30 days
        'path'     => '/',
        'domain'   => $cookieDomain,
        'secure'   => $isSecure,
        'httponly' => true,
        'samesite' => 'Lax',
      ]);
    }

    // 2) page cookie (rotated on document navigation)
    $pageId = $_COOKIE['page_id'] ?? '';
    if ($isDocumentNav) {
      $pageId = uuidv4();
      @setcookie('page_id', $pageId, [
        'expires'  => 0, // session cookie
        'path'     => '/',
        'secure'   => $isSecure,
        'httponly' => true,
        'samesite' => 'Lax',
      ]);
    }

    // Build carrier from FastCGI->PHP headers
    $carrier = [];
    foreach ($_SERVER as $k => $v) {
      if (str_starts_with($k, 'HTTP_')) {
        $name = strtolower(str_replace('_', '-', substr($k, 5)));
        $carrier[$name] = $v;
      }
    }

    $incomingTraceparent = $_SERVER['HTTP_TRACEPARENT'] ?? '';
    $traceId = null;

    $span = null;
    $scope = null;
    $tp = null;

    try {
      $parent = \OpenTelemetry\API\Trace\Propagation\TraceContextPropagator::getInstance()->extract(
        $carrier,
        \OpenTelemetry\Context\Propagation\ArrayAccessGetterSetter::getInstance()
      );

      $tp = \OpenTelemetry\API\Globals::tracerProvider();
      $tracer = $tp->getTracer('nextcloud-request', '1.0.0');

      $name = sprintf('%s %s', $_SERVER['REQUEST_METHOD'] ?? 'HTTP', $uri ?: '/');
      $span = $tracer->spanBuilder($name)
        ->setSpanKind(\OpenTelemetry\API\Trace\SpanKind::KIND_SERVER)
        ->setParent($parent)
        ->startSpan();

      $traceId = $span->getContext()->getTraceId();
      $scope = $span->activate();

      $span->setAttribute('http.method', $_SERVER['REQUEST_METHOD'] ?? null);
      $span->setAttribute('http.target', $uri ?: null);
      $span->setAttribute('net.peer.ip', $_SERVER['REMOTE_ADDR'] ?? null);
      $span->setAttribute('session.id', $sessionId);
      if ($pageId !== '') $span->setAttribute('page.id', $pageId);
      $span->setAttribute('page.is_document', $isDocumentNav);

    } catch (\Throwable $e) {
      // fallback: parse trace id from traceparent if present
      if ($incomingTraceparent) {
        $parts = explode('-', $incomingTraceparent);
        if (isset($parts[1]) && preg_match('/^[0-9a-f]{32}$/i', $parts[1])) {
          $traceId = strtolower($parts[1]);
        }
      }
      if ($traceId === null) {
        $traceId = bin2hex(random_bytes(16));
      }
    }

    // === Always emit headers so php-fpm access.log can capture them ===
    @header('X-Trace-Id: ' . $traceId);


    // Optional: make readable in browser JS for XHR/fetch
    @header('Access-Control-Expose-Headers: X-Trace-Id');
    register_shutdown_function(static function () use ($span, $scope, $tp): void {
      try { if ($scope) { $scope->detach(); } } catch (\Throwable $e) {}
      try { if ($span) { $span->end(); } } catch (\Throwable $e) {}
      if (is_object($tp) && method_exists($tp, 'forceFlush')) {
        try { $tp->forceFlush(); } catch (\Throwable $e) {}
      }

        - name: OTEL_SERVICE_NAME
          value: nextcloud-app
        - name: OTEL_PROPAGATORS
          value: tracecontext,baggage

        - name: OTEL_PHP_AUTOLOAD_ENABLED
          value: "true"
        - name: OTEL_PHP_AUTOLOAD_PATH
          value: /otel/vendor/autoload.php

        - name: OTEL_TRACES_EXPORTER
          value: otlp
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: http/protobuf
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://192.168.40.141:4318
        - name: OTEL_LOG_LEVEL
          value: DEBUG
        - name: OTEL_TRACES_SAMPLER
          value: "always_on"

        - name: php-opentelemetry-ini
          mountPath: /usr/local/etc/php/conf.d/10-opentelemetry.ini
          subPath: opentelemetry.ini
        - name: otel-bootstrap
          mountPath: /otel/prepend.php
          subPath: prepend.php

  nextcloud.conf: |
      # =========================
      # OpenTelemetry (ngx_otel_module)
      # =========================
      # Export spans to Tempo (OTLP/gRPC).
      otel_exporter {
        endpoint 192.168.40.141:4317;
      }

      otel_service_name nextcloud-nginx;

      # =========================
      # Nextcloud nginx config
      # =========================
      upstream php-handler {
        server 127.0.0.1:9000;
      }

      map $arg_v $asset_immutable {
        default "";
        ""      "";
        ~.+     "immutable";
      }

      # Choose Cloudflare or fallback to X-Real-IP
      map $http_cf_connecting_ip $use_cf_ip {
        default 0;
        ""      0;
        ~.+     1;
      }
      map $use_cf_ip $real_client_ip {
        1 $http_cf_connecting_ip;
        0 $http_x_real_ip;
      }
      map $otel_parent_sampled $otel_flags {
        0 "00";
        1 "01";
      }


      map "$otel_trace_id$otel_span_id" $otel_traceparent {
        default "00-$otel_trace_id-$otel_span_id-$otel_flags";
      }
      map $upstream_http_x_trace_id $log_trace_id {
        default $upstream_http_x_trace_id;
        ""      $otel_trace_id;
      }

      types {
        application/javascript mjs;
      }

      server {
        listen 8080;
        server_name drive.DOMAIN;
        root /var/www/html;
        index index.php index.html /index.php$request_uri;

        otel_trace on;
        otel_trace_context propagate; 
        # Real IP / Proxy trust
        set_real_ip_from 192.168.40.96;
        set_real_ip_from 127.0.0.1;
        set_real_ip_from 10.0.0.0/8;
        set_real_ip_from 192.168.0.0/16;
        real_ip_header X-Forwarded-For;
        real_ip_recursive on;

        # Security headers
        add_header X-Robots-Tag "noindex,nofollow" always;
        add_header X-Frame-Options "SAMEORIGIN" always;
        add_header X-Content-Type-Options "nosniff" always;
        add_header Referrer-Policy "no-referrer-when-downgrade" always;
        add_header Strict-Transport-Security "max-age=15552000; includeSubDomains; preload" always;

        client_max_body_size 0;
        proxy_request_buffering off;
        fastcgi_request_buffering off;

        # ----------------------------
        # Static files
        # ----------------------------
        location ~ ^/(?:css|js|mjs|svg|gif|ico|jpg|jpeg|png|webp|wasm|ttf|otf|woff2?)$ {
          try_files $uri =404;
          expires 6M;
          add_header Cache-Control "public, max-age=15768000, $asset_immutable";
        }

        location = /robots.txt {
          allow all;
          log_not_found off;
        }

        location ^~ /.well-known {
          location = /.well-known/carddav { return 301 /remote.php/dav; }
          location = /.well-known/caldav  { return 301 /remote.php/dav; }
          try_files $uri $uri/ =404;
        }

        location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)(?:/|$) { return 404; }
        location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) { return 404; }

        location ~ \.php(?:$|/) {
          rewrite ^/(?!index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+|.+/richdocumentscode(_arm64)?/proxy) /index.php$request_uri;

          fastcgi_split_path_info ^(.+?\.php)(/.*)$;
          set $path_info $fastcgi_path_info;

          try_files $fastcgi_script_name =404;
          include fastcgi_params;

          fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
          fastcgi_param PATH_INFO $path_info;

          fastcgi_param REMOTE_ADDR $real_client_ip;
          fastcgi_param HTTP_X_FORWARDED_FOR $http_x_forwarded_for;
          fastcgi_param HTTP_X_REAL_IP $http_x_real_ip;

          fastcgi_param HTTP_TRACEPARENT $otel_traceparent;
          fastcgi_param HTTP_TRACESTATE  $http_tracestate;
          fastcgi_param OTEL_SESSION_ID $cookie_session_trace_id;
          fastcgi_param OTEL_PAGE_ID    $cookie_page_id;
          fastcgi_pass php-handler;
        }


        location ~ ^/(?:remote|DAV|webdav|public|ocs|cron|index|status)\.php/ {


          try_files $uri =404;
          include fastcgi_params;

          fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
          fastcgi_param modHeadersAvailable true;
          fastcgi_param front_controller_active true;

          fastcgi_param REMOTE_ADDR $real_client_ip;
          fastcgi_param HTTP_X_FORWARDED_FOR $http_x_forwarded_for;
          fastcgi_param HTTP_X_REAL_IP $http_x_real_ip;

          fastcgi_param HTTP_TRACEPARENT $otel_traceparent;
          fastcgi_param HTTP_TRACESTATE  $http_tracestate;
          fastcgi_param OTEL_SESSION_ID $cookie_session_trace_id;
          fastcgi_param OTEL_PAGE_ID    $cookie_page_id;
          fastcgi_pass php-handler;
          fastcgi_intercept_errors on;
          fastcgi_request_buffering off;
        }
        location = /tracecheck.php {


          include fastcgi_params;
          fastcgi_param SCRIPT_FILENAME $document_root/tracecheck.php;
          fastcgi_param PATH_INFO "";

          fastcgi_param HTTP_TRACEPARENT $otel_traceparent;
          fastcgi_param HTTP_TRACESTATE  $http_tracestate;

          fastcgi_param REMOTE_ADDR $real_client_ip;
          fastcgi_param HTTP_X_FORWARDED_FOR $http_x_forwarded_for;
          fastcgi_param HTTP_X_REAL_IP $http_x_real_ip;
          fastcgi_param OTEL_SESSION_ID $cookie_session_trace_id;
          fastcgi_param OTEL_PAGE_ID    $cookie_page_id;
          fastcgi_pass php-handler;
          fastcgi_intercept_errors on;
          fastcgi_request_buffering off;
        }


        location / {
          # otel_trace_context extract;
          # otel_span_name "nextcloud.http $request_method $uri";

          try_files $uri $uri/ /index.php$request_uri;
        }
      }

  php-fpm.conf: |
    [global]
    error_log = /var/log/nextcloud/error.log
    [www]
    access.log = /var/log/nextcloud/access.log
    access.format = "%R - %u %t \"%m %r\" %s %f %{milli}dms trace_id=%{X-Trace-Id}o session_id=%{OTEL_SESSION_ID}e page_id=%{OTEL_PAGE_ID}e"
    php_admin_flag[log_errors] = on
    php_admin_value[error_log] = /var/log/nextcloud/php-error.log
    php_admin_flag[display_errors] = off
    user = www-data
    group = www-data
    listen = 0.0.0.0:9000
    clear_env = no
    pm = dynamic
    pm.max_children = 50
    pm.start_servers = 5
    pm.min_spare_servers = 5
    pm.max_spare_servers = 20
    pm.max_requests = 500
    php_admin_value[post_max_size] = 0
    php_admin_value[upload_max_filesize] = 0
    php_admin_value[max_execution_time] = 3600
    php_admin_value[max_input_time] = 3600

  nginx.conf: |
    # main context
    user  nginx;
    load_module modules/ngx_otel_module.so;
    worker_processes  auto;

    error_log  /var/log/nginx/error.log warn;
    pid        /var/run/nginx.pid;

    events { worker_connections  1024; }

    http {
      include       /etc/nginx/mime.types;
      default_type  application/octet-stream;

    log_format main
      '$remote_addr - $remote_user [$time_local] '
      '"$request" $status $body_bytes_sent '
      '"$http_referer" "$http_user_agent" '
      'xff="$http_x_forwarded_for" '
      'rt=$request_time uct=$upstream_connect_time uht=$upstream_header_time urt=$upstream_response_time '
      'up="$upstream_addr" us=$upstream_status '
      'traceparent="$http_traceparent" '
      'trace_id="$log_trace_id" span_id="$otel_span_id" sampled=$otel_parent_sampled '
      'session_id="$cookie_session_trace_id" page_id="$cookie_page_id"';

      access_log  /var/log/nginx/access.log  main;

      sendfile        on;
      keepalive_timeout  65;

      include /etc/nginx/conf.d/*.conf;
    }

[ibizaman]

This is wonderful @jpb0418 Thanks for doing the hard work. Feel free to work on it if you want but otherwise I’ll work on a PR for it. I’ll post a message here when I start so we don’t do double work.

[jpb0418]

Some more ai SLOP that seems to work lol, cron tracing. create LOTS of spans >1000 seems like a whole lot of DB activity.
Hopefully this is helpful, lots of throwing noodles at the wall to see what sticks lol. Someone that knows what there doing would have to do some clean up

  prepend-cli.php: |
    <?php
    declare(strict_types=1);

    if (PHP_SAPI !== 'cli') { return; }


    $autoload = '/otel/vendor/autoload.php';
    if (is_file($autoload)) {
        require_once $autoload;
    }

    use OpenTelemetry\API\Globals;
    use OpenTelemetry\API\Trace\SpanKind;
    use OpenTelemetry\API\Trace\StatusCode;
    use OpenTelemetry\API\Trace\Propagation\TraceContextPropagator;
    use OpenTelemetry\Context\Propagation\ArrayAccessGetterSetter;
    use OpenTelemetry\SemConv\TraceAttributes;

    // Build a carrier from environment variables if you want to link CLI spans to an upstream trace
    // You can pass TRACEPARENT/TRACESTATE in the environment when invoking occ/cron.
    $carrier = [];
    $tpEnv = getenv('TRACEPARENT');
    $tsEnv = getenv('TRACESTATE');
    if (is_string($tpEnv) && $tpEnv !== '') { $carrier['traceparent'] = $tpEnv; }
    if (is_string($tsEnv) && $tsEnv !== '') { $carrier['tracestate']  = $tsEnv; }

    $span = null;
    $scope = null;
    $tp = null;

    try {
        $parent = TraceContextPropagator::getInstance()->extract(
            $carrier,
            ArrayAccessGetterSetter::getInstance()
        );

        $tp = Globals::tracerProvider();
        $tracer = $tp->getTracer('nextcloud-cli', '1.0.0');

        $argv = $_SERVER['argv'] ?? [];
        $script = (string)($argv[0] ?? 'php');
        $args = array_slice($argv, 1);

        // Nice span names for Nextcloud
        $name = 'cli ' . basename($script);
        if (str_ends_with($script, '/occ') || basename($script) === 'occ') {
            $name = 'occ' . (isset($args[0]) ? (' ' . $args[0]) : '');
        } elseif (basename($script) === 'cron.php') {
            $name = 'cron.php';
        } elseif ($args) {
            $name .= ' ' . $args[0];
        }

        $span = $tracer->spanBuilder($name)
            ->setSpanKind(SpanKind::KIND_INTERNAL)
            ->setParent($parent)
            ->startSpan();

        $scope = $span->activate();

        // Attributes (helpful in Tempo/Grafana)
        $cmdline = trim($script . ' ' . implode(' ', array_map('strval', $args)));

        $span->setAttribute(TraceAttributes::PROCESS_COMMAND, $script);
        $span->setAttribute(TraceAttributes::PROCESS_COMMAND_ARGS, $args);
        $span->setAttribute('process.command_line', $cmdline);
        $span->setAttribute('php.sapi', PHP_SAPI);
        $span->setAttribute('php.version', PHP_VERSION);
        $span->setAttribute('nextcloud.cli.script', $script);

        // If you want, add a cheap “job id” for grouping:
        // $span->setAttribute('job.id', bin2hex(random_bytes(8)));

        // Record uncaught exceptions/errors
        set_exception_handler(static function (\Throwable $e) use (&$span): void {
            if ($span) {
                $span->recordException($e);
                $span->setStatus(StatusCode::STATUS_ERROR, $e->getMessage());
            }
            // Re-throw so the process still exits non-zero as usual
            throw $e;
        });

        set_error_handler(static function (int $severity, string $message, string $file, int $line) use (&$span): bool {
            if ($span) {
                $span->recordException(new \ErrorException($message, 0, $severity, $file, $line));
                $span->setStatus(StatusCode::STATUS_ERROR, $message);
            }
            return false; // allow normal PHP error handling too
        });

    } catch (\Throwable $e) {
        // If OTel isn't ready, we just do nothing and let CLI run normally.
    }

    register_shutdown_function(static function () use (&$span, &$scope, &$tp): void {
        try {
            if ($scope) { $scope->detach(); }
        } catch (\Throwable $e) {}

        try {
            if ($span) {
                // Mark span error if process exits with non-zero and we didn’t already set status
                $code = (int) (is_array(error_get_last()) ? 1 : 0);
                // NOTE: we can't reliably read exit code here unless you manage it yourself,
                // but error_get_last() at least marks fatal errors.
                if ($code !== 0) {
                    $span->setStatus(\OpenTelemetry\API\Trace\StatusCode::STATUS_ERROR);
                }
                $span->end();
            }
        } catch (\Throwable $e) {}

        try {
            if (is_object($tp) && method_exists($tp, 'forceFlush')) {
                $tp->forceFlush();
            }
        } catch (\Throwable $e) {}
    });

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nextcloud-cron
  namespace: nextcloud
spec:
  schedule: "*/5 * * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          securityContext:
            fsGroup: 82
          containers:
            - name: nextcloud-cron
              image: CUSTOMDOCKERIMAGE
              imagePullPolicy: IfNotPresent
              command:
                - php
                - -d
                - auto_prepend_file=/otel/prepend-cli.php
                - -f
                - /var/www/html/cron.php
              env:
                - name: PUID
                  value: "82"
                - name: PGID
                  value: "82"
                - name: HOME
                  value: "/var/www"
                - name: PGSSLMODE
                  value: "require"

                - name: OTEL_SERVICE_NAME
                  value: nextcloud-cron
                - name: OTEL_PROPAGATORS
                  value: tracecontext,baggage
                - name: OTEL_PHP_AUTOLOAD_ENABLED
                  value: "true"
                - name: OTEL_PHP_AUTOLOAD_PATH
                  value: /otel/vendor/autoload.php
                - name: OTEL_TRACES_EXPORTER
                  value: otlp
                - name: OTEL_EXPORTER_OTLP_PROTOCOL
                  value: http/protobuf
                - name: OTEL_EXPORTER_OTLP_ENDPOINT
                  value: http://192.168.40.141:4318
                - name: OTEL_LOG_LEVEL
                  value: DEBUG
                - name: OTEL_TRACES_SAMPLER
                  value: "always_on"
              securityContext:
                runAsUser: 82
                runAsGroup: 82
              volumeMounts:
                - name: nextcloud-data
                  mountPath: /var/www/html
                - name: otel-prepend-cli
                  mountPath: /otel/prepend-cli.php
                  subPath: prepend-cli.php

          volumes:
            - name: nextcloud-data
              persistentVolumeClaim:
                claimName: nextcloud-data-pvc
            - name: otel-prepend-cli
              configMap:
                name: nextcloud-nginx-config
                items:
                  - key: prepend-cli.php
                    path: prepend-cli.php

[cbcoutinho]

I'm very much looking forward to this PR landing.

My 2c: I would suggest allowing opentelemtry collectors to manage trace filtering rather than having an opinionated OTEL tracing module in Nextcloud filter what developers think would be interesting to trace. That way App developers and NC admins can decide what to trace and how often, rather than have limitations imposed by NC itself. In practice, this would mean that Nextcloud OTEL support would generate probably more traces than is necessary for most use-cases, and then this PR should include documentation on how that filtering would work.

It's my understanding that there is no parallel to "log levels" for traces, and instead the volume of traces can be managed via Sampling, Span Attributes, and Status Codes. I guess we could include something similar to "log levels" or similar to span attributes, and in that case I would suggest standardizing this across all apps so that there is at least something that all apps can leverage to unify how traces are managed across all apps