OpenAI client prevents process from gracefully terminating

[punkpeye]

Confirm this is a Node library issue and not an underlying OpenAI API issue

• This is an issue with the Node library

Describe the bug

When using OpenAI client to stream completions, the process fails to shutdown gracefully.

To Reproduce

Stream and attempt to gracefully shutdown.

Active handles show instances of agentkeepalive, which is coming from OpenAI SDK.

{
  resource: {
    _parent: '[Object: TCP]',
    _parentWrap: null,
    _secureContext: '[Object: SecureContext]',
    onerror: '[Function: onerror]',
    onhandshakedone: '[Function: ]',
    onhandshakestart: '[Function: noop]',
    onkeylog: '[Function: onkeylog]',
    onnewsession: '[Function: onnewsessionclient]',
    onocspresponse: '[Function: onocspresponse]',
    reading: true
  },
  stacks: [
    'node:internal/async_hooks:202:43 [emitInitNative]',
    'node:_tls_wrap:699:24 [TLSSocket._wrapHandle]',
    'node:_tls_wrap:570:18 [TLSSocket]',
    'node:_tls_wrap:1766:19 [connect]',
    'node:https:173:22 [createConnection]',
    '/Users/frank/Developer/glama/node_modules/.pnpm/agentkeepalive@4.6.0/node_modules/agentkeepalive/lib/https_agent.js:29:49 [createConnection]',
    'node:_http_agent:324:26 [createSocket]',
    'node:_http_agent:276:10 [addRequest]',
    'node:_http_client:347:16 [ClientRequest]'
  ],
  type: 'TLSWRAP'
},

Code snippets

OS

macOS

Node version

22

Library version

4.83.0

[RobertCraigie]

Thanks for the report, would you be able to share a reproduction?

[punkpeye]

The problem here is the httpAgent.

Here is the abstraction I've used to solve it when implementing https://glama.ai

import 'openai/shims/web';
import { shutdownHandler } from '#app/services/shutdownHandler.server';
import { remember } from '@epic-web/remember';
import { Agent } from 'node:http';
import OpenAI, { type ClientOptions } from 'openai';

const httpAgent = remember('httpAgent', () => {
  return new Agent({
    keepAlive: true,
  });
});

shutdownHandler.registerShutdownHandler({
  name: 'openai:httpAgent',
  run: async () => {
    httpAgent.destroy();
  },
  weight: 500,
});

export const createOpenAIClient = (options: ClientOptions) => {
  return new OpenAI({
    ...options,
    httpAgent,
  });
};

[ac-mmi]

@RobertCraigie In my PR ,I modified the _iterSSEMessages() function to ensure the stream is properly closed by calling response.body.cancel() in the finally block, preventing lingering connections and allowing for a graceful shutdown.

[RobertCraigie]

wait you're using shims/web? I think httpAgent just won't work at all in that case...

[punkpeye]

wait you're using shims/web? I think httpAgent just won't work at all in that case...

I wasn't before this issue. That's what I added.

I thought shims/web makes it use node's native fetch agent?

[punkpeye]

ah, just double checked the docs. Looks like it should be openai/shims/node. Thanks for calling it out.

[RobertCraigie]

Did switching to shims/node fix the issue?

We're also going to be changing how this works slightly in v5 to avoid this issue where you'll pass fetchOptions: { httpAgent } instead and you should get type errors if it isn't supported as long as your global types are accurate (new release coming soon with this change included). As such I'm going to close this issue.