[@azure/monitor-opentelemetry] - Requests with status code 304 are shown as failed in application insights

[despgiat]

• Package Name: @azure/monitor-opentelemetry
• Package Version: 1.4.0
• Operating system:
• nodejs
  • version: 18.16.0
• browser
  • name/version: macOS
• typescript
  • version: 5.4.5

Describe the bug
Currently some requests are treated and shown as errors, e.g. 304 which is a redirect / cache-hit. The majority of the errors that we get are 304 requests. Moreover, we don't get any information about these requests in application insights (in the first screenshot below you can see how the the dashboard looks like when the 304 is clicked to reveal more information about the errors). Also, these requests are not shown as errors in transaction search, and when they are filtered (ex. by response code), they do not show in the failed requests graph anymore.

To Reproduce
Our setup is similar to this example:

import { AzureMonitorOpenTelemetryOptions, useAzureMonitor } from '@azure/monitor-opentelemetry';
import { metrics, trace } from '@opentelemetry/api';
import { registerInstrumentations } from '@opentelemetry/instrumentation';
import { ExpressInstrumentation } from '@opentelemetry/instrumentation-express';
import { HttpInstrumentationConfig } from '@opentelemetry/instrumentation-http';
import { MongoDBInstrumentationConfig } from '@opentelemetry/instrumentation-mongodb';
import { Resource } from '@opentelemetry/resources';
import { SEMRESATTRS_SERVICE_NAME } from '@opentelemetry/semantic-conventions';
import { IncomingMessage } from 'http';
import { env } from '../env';

const httpInstrumentationConfig: HttpInstrumentationConfig = {
  enabled: true,
  ignoreIncomingRequestHook: (request: IncomingMessage) => {
    // Ignore OPTIONS incoming requests
    if (request.method === 'OPTIONS') {
      return true;
    }
    return false;
  },
};

const mongoInstrumentationConfig: MongoDBInstrumentationConfig = {
  enabled: true,
  enhancedDatabaseReporting: true,
};

export const initializeTelemetry = () => {
  const customResource = Resource.EMPTY;
  customResource.attributes[SEMRESATTRS_SERVICE_NAME] = 'Backend';

  const options: AzureMonitorOpenTelemetryOptions = {
    resource: customResource,
    azureMonitorExporterOptions: {
      connectionString: env.APPLICATION_INSIGHTS_CONNECTION_STRING,
    },
    instrumentationOptions: {
      http: httpInstrumentationConfig,
      mongoDb: mongoInstrumentationConfig,
    },
  };

  useAzureMonitor(options);
  addOpenTelemetryInstrumentation();
};

/**
 * Add additional OpenTelemetry instrumentation that is not bundled with Azure OpenTelemetry Distro
 */
const addOpenTelemetryInstrumentation = () => {
  const instrumentations = [new ExpressInstrumentation()];
  registerInstrumentations({
    tracerProvider: trace.getTracerProvider(),
    meterProvider: metrics.getMeterProvider(),
    instrumentations: instrumentations,
  });
};

Expected behavior
The requests with status code 304 are not shown as errors in application insights.

Screenshots

Additional context
We played around a bit and tried to set custom attributes to the http requests using a SpanEnrichingProcessor but it seems that no custom attributes could be added to these failed requests.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @hectorhdzg @JacksonWeber @ramthi.

[JacksonWeber]

@despgiat I'm not able to reproduce this situation of 304 status codes being shown as errors. Requests marked as response code 304 are being shown as I expect in the transaction blade when using your example code with an express server that returns a 304 response.

Can you provide more information on where these 304 response codes are being returned from or more details for reproducing your scenario? Thanks!

[p-hlp]

@JacksonWeber Guess I can chime in here. You can use this repo for repro. It was initially used for a different problem with application insights js sdk/azure otel distro, but has the same issue with 304's showing up under failures. You can find detailed steps for reproducing the scenario in the readme. For the setup - personally I used two ai resources (web client/api) in same workspace, however a shared one also works.

[JacksonWeber]

@p-hlp Thank you for doing this work to repro. I'll recreate this scenario and work on debug as soon as I can.

[p-hlp]

@JacksonWeber Are there any updates by chance? It's a bit of an annoying problem. Otherwise anything I could do on my side to support with getting this going?

[JacksonWeber]

@p-hlp Apologies I haven't had much time to look into this earlier. Investigating this afternoon and I'll provide an update ASAP.