Specifying http proxy changed in Node@18 and 20

[waldekmastykarz]

Since v18, Node ships with a native fetch client. This client requires a different way to specify an HTTP proxy than what we've been supporting in our SDKs so far.

To specify an HTTP proxy for Node's fetch, you need to use the following code (tested on Node v18.19 and v20.11):

import { ProxyAgent } from 'undici';
import { Client } from '@microsoft/microsoft-graph-client';

const dispatcher = new ProxyAgent(process.env.https_proxy);
const fetchOptions = {
  dispatcher
};

Client.initWithMiddleware({
  middleware,
  fetchOptions
});

What's noteworthy:

• we specify HTTP proxy using dispatcher instead of agent (agent is ignored on requests, which makes it hard to debug)
• the ProxyAgent dispatcher comes from undici which seems to be maintained by Node folks. The ProxyAgent is incompatible with other agents that we used to use in the past.

We should update our FetchOptions interface to support specifying dispatcher.

[baywet]

Thanks for reaching out.

@sebastienlevert and I were chatting about that earlier and he mentioned he made it work. I'll let him provide the specifics when he has time (as you know I'm traveling right now and don't have time to dig into this)

[sebastienlevert]

Just to confirm... This is for Graph JS v3, right? We donc have the middleware initialization anymore IIRC.

As we won't be adding new features over there, I highly doubt we'll get to this.

[waldekmastykarz]

This change affects any code that relies on Node's fetch. I noticed it in Graph JS SDK v3 but if the new TypeScript SDK also uses Node.js fetch, then it would also have the same issue.

[baywet]

Since this initial post, our plans have changed multiple times.

We now planning a full depreciation of the "old" JS SDK, which means the "new" TypeScript SDK and kiota dependencies won't have any dependencies on it.

The client factory now supports providing a fetch callback

Consumers should simply need to do something like this:

const proxy = new HttpsProxyAgent('http://168.63.76.32:3128');
const customFetch  = (request: string, init: RequestInit) => fetch(request, {...init, agent: proxy});
const httpClient = KiotaClientFactory.create(customFetch); // additional middleware configuration here
const requestAdapter = new DefaultRequestAdapter(authenticationProvider, undefined, undefined, httpClient);
const apiClient = new APIClient(requestAdapter);

Of course, we can add shorthands in graph core and service libraries to reduce the number of calls here, the important thing is the capability is available natively.

We might also want to advise customers to take an additional dependency on https-proxy-agent, more context

Assuming this tab gets updated (it should really say JavaScript to be consistent with generated doc snippets), does that address your concern @waldekmastykarz ?

[waldekmastykarz]

Yes it does! I appreciate you being mindful of this requirement, that many large organization need for their apps.

[jasonjoh]

@baywet I'm having no luck getting your example proxy code to work. The problem is around RequestInit. It seems like Node's adoption of undici is a mess.

I started with code like this:

// Create a new HTTPS proxy agent (from https-proxy-agent)
const proxyAgent = new HttpsProxyAgent('http://localhost:8888');

// Create a custom fetch callback
const customFetch = async (request: string, init: RequestInit) =>
  fetch(request, { ...init, agent: proxyAgent });

// Create an HttpClient with custom fetch callback
var httpClient = KiotaClientFactory.create(customFetch);

Two errors with this code:

• 'RequestInit' is not defined. eslint(no-undef)
• Object literal may only specify known properties, and 'agent' does not exist in type 'RequestInit'.

Ignoring the eslint error for now, I checked Intellisense for RequestInit:

So I tried importing RequestInit from 'undici-types'. That cleared up the eslint error but there's still no agent. However, I notice that the Intellisense for the RequestInit that fetch is expecting differs from the interface definition from 'undici-types':

export interface RequestInit {
  method?: string
  keepalive?: boolean
  headers?: HeadersInit
  body?: BodyInit | null
  redirect?: RequestRedirect
  integrity?: string
  signal?: AbortSignal | null
  credentials?: RequestCredentials
  mode?: RequestMode
  referrer?: string
  referrerPolicy?: ReferrerPolicy
  window?: null
  dispatcher?: Dispatcher
  duplex?: RequestDuplex
}

So it seems like the fetch that's resolving in my project is expecting a different RequestInit than what the 'undici-types' package provides, even though my understanding is that Node's fetch IS the undici one??

So I then explicitly import fetch from undici-types and tried passing proxyAgent to the dispatcher property, since it seems that's how undici supports proxy (reference). It seems that HttpsProxyAgent isn't compatible with the Dispatcher interface - no problem, I can switch to undici's ProxyAgent.

Now Kiota's client factory isn't happy:

src/snippets/customClients.ts:87:46 - error TS2345: Argument of type '(request: string, init: import("C:/Source/Repos/msgraph-snippets-typescript/node_modules/undici-types/fetch").RequestInit) => Promise<import("C:/Source/Repos/msgraph-snippets-typescript/node_modules/undici-types/fetch").Response>' is not assignable to parameter of type '(request: string, init: RequestInit) => Promise<Response>'.
  Types of parameters 'init' and 'init' are incompatible.
    Type 'RequestInit' is not assignable to type 'import("C:/Source/Repos/msgraph-snippets-typescript/node_modules/undici-types/fetch").RequestInit'.
      Types of property 'body' are incompatible.
        Type 'BodyInit | null | undefined' is not assignable to type 'BodyInit | undefined'.
          Type 'ReadableStream<any>' is not assignable to type 'BodyInit | undefined'.

Does anyone have a working bit of code for using a proxy?

[baywet]

I have meant to come back to this issue for over a month now but failed the time to do so.
@rkodev when you have couple of minutes, can you please look into this?