OpenAI: Cannot read properties of null

[vanstinator]

Expected behaviour
Bumping dd-trace from 4.2.0 to 4.4.0 should work.

Actual behaviour
Instead our applications are failing to start up with the following error:

TypeError: Cannot read properties of null (reading 'dogstatsd')
    at Object.module.exports.init (/Users/justin/projects/media-providers/node_modules/dd-trace/packages/datadog-plugin-openai/src/services.js:14:24)
    at new OpenApiPlugin (/Users/justin/projects/media-providers/node_modules/dd-trace/packages/datadog-plugin-openai/src/index.js:22:42)
    at PluginManager.loadPlugin (/Users/justin/projects/media-providers/node_modules/dd-trace/packages/dd-trace/src/plugin_manager.js:76:35)
    at PluginManager._loadedSubscriber (/Users/justin/projects/media-providers/node_modules/dd-trace/packages/dd-trace/src/plugin_manager.js:65:12)
    at Channel.publish (node:diagnostics_channel:56:9)

Steps to reproduce

This seems similar to #3296

Our repo is a fairly convoluted monorepo, so I can't trivially pull out relevant pieces to share. If the issue is non-obvious I'm happy to spend the time putting together an example repo that replicates the behavior.

We're calling tracer.init in a require hook before we do anything else, and any tracer.use calls are happening after we call tracer.init.

Environment

• Operation system: macOS
• Node.js version: 18.16.1
• Tracer version: 4.2.0
• Agent version: N/A
• Relevant library versions: openai 3.3.0

[tlhunter]

Thanks for pointing this out. It looks like we failed to test if DogStatsD is enabled in the tracer before using it. I'll work on a fix.

[tlhunter]

This should be fixed in tracer versions v4.6.0 / v3.27.0 / v2.40.0.

[vanstinator]

Amazing. Thank you!

[vanstinator]

@tlhunter This is still happening with more or less the same stracktrace, reading the original and current error more closely it looks like tracerConfig is null, so the if (tracerConfig.dogstatsd) check will fail too, because the config doesn't exist.

Presumably there's nothing on the public API for the library for me to do other than tracer.init?

[vanstinator]

Can we get this re-opened?

[GeoffreyPlitt]

+1
Still happening for me in 4.7.0

[tlhunter]

@vanstinator @GeoffreyPlitt When these errors happen, are your applications making a call to openai immediately in the process, like some sort of script or batch process? Or are they making the request later, like within an HTTP request handler?

[GeoffreyPlitt]

I'm not using OpenAI, just dd-trace. No script or batch process, it's a NodeJS web server.

Uncaught TypeError: Cannot read properties of null (reading 'dogstatsd')
at Object.module.exports.init (node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/datadog-plugin-openai/src/services.js:13:20)
at new OpenApiPlugin (node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/datadog-plugin-openai/src/index.js:22:42)
at PluginManager.loadPlugin (node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/dd-trace/src/plugin_manager.js:76:35)
at PluginManager._loadedSubscriber (node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/dd-trace/src/plugin_manager.js:65:12)
at Channel.publish (node:diagnostics_channel:56:9)
at /node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/datadog-instrumentations/src/helpers/register.js:47:25
at safeHook (node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/datadog-instrumentations/src/helpers/hook.js:28:12)
at Module.Hook.Module.require (node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/dd-trace/src/ritm.js:134:33)
at require (node:internal/modules/cjs/helpers:110:18)
at Object. (node_modules/.pnpm/openai@3.2.1_debug@4.3.4/node_modules/openai/dist/index.js:26:14)
at Module._compile (node:internal/modules/cjs/loader:1256:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
at Object.newLoader [as .js] (node_modules/.pnpm/pirates@4.0.5/node_modules/pirates/lib/index.js:141:7)
at Module.load (node:internal/modules/cjs/loader:1119:32)
at Module._load (node:internal/modules/cjs/loader:960:12)
at Function.updatedLoad [as _load] (app/lib/ts-require-patch.ts:59:20)
at Module.require (node:internal/modules/cjs/loader:1143:19)
at Module.Hook.Module.require (node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/dd-trace/src/ritm.js:85:33)
at require (node:internal/modules/cjs/helpers:110:18)
at Object. (app/controllers/generative.ts:16:17)
at Module._compile (node:internal/modules/cjs/loader:1256:14)
at Module._compile (node_modules/.pnpm/pirates@4.0.5/node_modules/pirates/lib/index.js:136:24)
at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
at Object.newLoader [as .ts] (node_modules/.pnpm/pirates@4.0.5/node_modules/pirates/lib/index.js:141:7)
at Module.load (node:internal/modules/cjs/loader:1119:32)
at Module._load (node:internal/modules/cjs/loader:960:12)
at Function.updatedLoad [as _load] (app/lib/ts-require-patch.ts:59:20)
at Module.require (node:internal/modules/cjs/loader:1143:19)
at Module.Hook.Module.require (node_modules/.pnpm/dd-trace@4.7.0/node_modules/dd-trace/packages/dd-trace/src/ritm.js:85:33)

[tlhunter]

@GeoffreyPlitt this is super concerning! Can you send me an email of your package-lock.json or yarn.lock files, or even the output of npm ls --depth=10? My email address is my github username {at} datadog.com.

I'm curious if openai is some sort of transient dependency of the project.

@vanstinator I was also assuming you are using openai too. Is this not the case?

[GeoffreyPlitt]

@tlhunter If this was a personal project I would, but I don't think I can send the entire package lockfile for my client.

I was also surprised to find dd-trace has dependencies on OpenAI, but if you search the code, you'll find OpenAI plugin stuff under "packages":
https://github.com/search?q=repo%3ADataDog%2Fdd-trace-js+datadog-plugin-openai&type=code

[GeoffreyPlitt]

Update: my project does use OpenAI for a little AI demo, and apparently dd-trace found that and started running the code in datadog-plugin-openai. I wish it left things alone.

I think most people would find it absurd that dd-trace contains an OpenAI plugin. Like, what the heck is that doing in here?

But even if it somehow has a reason to be here, is there a way to disable it?

[tlhunter]

dd-trace doesn't ship with the openai package and otherwise doesn't provide any machine learning functionality. It does instrument the package if it's loaded by an application, much like we instrument the postgres and redis packages when an app loads them.

As a temporary fix this environment variable should help: DD_TRACE_DISABLED_INSTRUMENTATIONS=openai

I'm working on a better solution which doesn't require the environment variable.

[GeoffreyPlitt]

DD_TRACE_DISABLED_INSTRUMENTATIONS=openai worked perfectly-- thanks! We can close this.

[vanstinator]

Sorry for the delay here.

We had a bug in how we were implementing tracer.init. The call was wrapped in a conditional, and in the cases where that code path was not being hit we were seeing the openai failure.

I submitted a patch to our internal systems to always call tracer.init and control whether or not we're actually tracing with the DD_TRACE_ENABLED environment variable.

[GeoffreyPlitt]

Ended up reverting our upgrade. Our project has way too many entry points to use this env-var band-aid.

@tlhunter do you have a branch I can follow for the true bugfix?

[GeoffreyPlitt]

@tlhunter Can we re-open this issue until the bug is fixed?

[tlhunter]

@GeoffreyPlitt @vanstinator Sorry for not updating the ticket but this should have been fixed on July 21st with the release of the following tracer versions: v4.8.0, v3.29.0, and v2.42.0. Please upgrade to at least one of those versions. If the problem persists I can open this issue up again.

[GeoffreyPlitt]

Thank you! Will test