fix(ClientRequest): preserve headers casing on mocked responses #463

mikicho · 2023-09-30T19:56:25Z

There are a few typing issues, but I would like to hear what you think about this solution first.
Do you know how can we find the Symbol more elegantly? I'm not very familiar with Symbols and Symbol.for('headers list') returns undefined

closes #448

Some thoughts:

While this solution probably will work for good, it relies on an "unofficial" behavior of Node.
Perhaps we could add a function that allows users to set the 'rawHeaders', although it may be a bit clumsy.
We can also solve it from Nock's side by sending a tempered version of the Response object which does not return lowercase headers. but IMO the solution should be in this lib.

kettanaito · 2023-10-01T09:31:43Z

src/interceptors/ClientRequest/NodeClientRequest.test.ts

@@ -315,3 +315,36 @@ it('does not send request body to the original server given mocked response', as
  const text = await getIncomingMessageBody(response)
  expect(text).toBe('mock created!')
 })
+
+it.only('does not lowercase the rawHeaders', async () => {


This test should be unmarked as .only.

src/interceptors/ClientRequest/NodeClientRequest.ts

kettanaito

While this approach may work, I recommend against it (see my reasoning in the comments).

Instead, I propose we find some middle-ground in exposing the original response to the consumer (the response listener). Historically, Interceptors used to expose the original request instance in the request listener, which looked like this:

type RequestListener = (
  request: Request,
  ref: ClientRequest | XMLHttpRequest
) => void

This was to give the consumer the means to access and operate on that original instance, need be. I've then moved away from this call signature, finding it too dynamic and confusing for the consumer.

But I think we can bring it back, although partially. I dislike that the ref was a completely different instance based on the request-issuing module. I think Interceptors must expose requests/responses in a unified way. I think the next unified way to represent requests/responses is HTTP message.

What do you say if we create and expose an HTTP message based on the request/response in the respective listeners?

// Approximate object shape.
interface HttpMessage {
  status: number
  statusText: string
  headers: Record<string, string[] | string>
  body?: string
}

interceptor.on('response', ({ response, httpMessage }) => {
  httpMessage.headers // raw headers
})

The httpMessage will be a custom abstraction respecting the HTTP message specification and describing the request/response as it was sent/received. This would be your primary way to access raw headers, as an example.

I'm currently working on something very closely related, and I can abstract the HttpMessage class into a separate package and we can consume it in Interceptors then.

Having access to the HTTP message also provides more data to the consumer, as they are now able to do thing like calculating the headers/body size.

mikicho · 2023-10-01T11:04:41Z

I dislike that the ref was a completely different instance

Agree!

This would be your primary way to access raw headers, as an example.

Do you mean that the response hook no longer would be read-only?
If the response remains read-only, I believe that including this in the request hook would be more appropriate. This is because we must ensure that the headers are set correctly in the response instance that the user receives in the callback.

kettanaito · 2023-10-01T13:06:23Z

The response event remains read-only. What I meant is to expose another property in the listener to that event called httpMessage, which will be an abstraction over the originally sent HTTP message. That way the consumer can operate with the response using its Fetch API Response representation as well as the "raw" HTTP message representation (your use case, to access the raw response headers). Neither allow modifying the response.

mikicho · 2023-10-01T13:13:05Z

@kettanaito maybe I'm missing something, but this way, the IncomingMessage instance (the response ) will still have lowercased header names. In the rawHeaders property.

kettanaito · 2023-10-01T16:53:37Z

@mikicho, no because the HTTP message will be created from the original IncomingMessage not its fetch representation. That way, the HTTP message will contain the raw headers as they came, which seems to be the expected behavior here. That way we have a more low-level representation of responses (and even requests, if we want to) without exposing a variadic request instance reference, which would be different for each request client.

I've published the package to GitHub (ref), will publish to NPM soon. You can take a look to see if it'd work for this use case.

mikicho · 2023-10-01T17:00:22Z

What about mocked responses?
I think I'm missing something.

The current flow is:

someone calls the req.end() function
Call to all request event listeners
Check if someone called to responseWith in one of the listeners
If yes, use the mocked response which is a Response instance with lowercased header names.

What do I miss?

kettanaito · 2023-10-01T17:12:38Z

@mikicho, since mock responses are declared using the Fetch API Response class, there is no concept of raw headers there. The response headers will be available by their raw name and by their normalized name (lowercase). Generally, response headers will be case-insensitive.

This will make your test pass in both cases:

// Assert normalized response headers.
const normalizedHeaders = Object.fromEntries(response.headers.entries())
expect(normalizedHeaders).toEqual(...)

// Assert raw response headers.
expect(response.headers.get('X-Custom-Header')).toBe('Expected Value')

But I think I understand the issue here. Since it's Response -> IncomingMessage translation, you also want to make sure that ClientRequest receives the IncomingMessage object with the right headers, which are case-sensitive (speaking of rawHeaders).

I will think about it and reply on this tomorrow.

kettanaito · 2023-10-03T07:56:26Z

Apologies for not understanding the requirements here sooner. I think I've finally wrapped my head around them. Yes, I think your original proposal to tap into Headers's symbols may be our best bet here given the constraints of Response.

Let me show you how we can access that symbol a bit better...

kettanaito · 2023-10-03T08:28:30Z

src/interceptors/ClientRequest/NodeClientRequest.ts

-    if (headers) {
+    // Try extracting the raw headers from the headers instance.
+    // If not possible, fallback to the headers instance as-is.
+    const rawHeaders = getRawFetchHeaders(headers) || headers


We will try to read the raw headers from the internal Headers symbols but if those are not present for any reason, fallback to the headers instance as-is.

kettanaito · 2023-10-03T08:29:01Z

src/utils/getRawFetchHeaders.ts

+
+  const headersMap = getValueBySymbol<
+    Map<string, { name: string; value: string }>
+  >('headers map', headersList)


I've noticed the raw headers map is buried a bit deeper in the symbols:

Headers - Symbol(headers list) - Symbol(headers map)

This seems to be specific to a Node.js version. This is where the raw headers map is for me on 18.14.0 but, apparently, it wasn't there on 18.8.0 used in the CI. I've updated the Node version on CI, let's see how it fares.

I confirm that's due to the Node.js difference in versions:

On 18.14.0, headers map is Map<normalizedName, { name: rawName, value: string }>

On 18.8.0, the same headers map symbol is Map<normalizedName, value>. Looks like older versions of Node doesn't even expose the raw header name in any way.

I think they do in the Symbol(headers list).entires (not the function entries)

I fixed this by checking what values headersMap actually has. If it's not an object with a name property, return undefined.

@mikicho, unfortunately, .entries() on headers list contains already normalized header names in Node.js 18.8.0:

const h = new Headers([['aBc, 'value']]) getValueBySymbol('headers list', h).entries() // [ ['abc', 'value'] ]

kettanaito · 2023-10-03T08:29:26Z

src/utils/getValueBySymbol.ts

+  const ownSymbols = Object.getOwnPropertySymbols(source)
+
+  const symbol = ownSymbols.find((symbol) => {
+    return symbol.description === symbolName


I took your original lookup logic just wrapped it in a utility function.

kettanaito · 2023-10-03T08:29:47Z

test/modules/http/compliance/http-res-raw-headers.test.ts

+  interceptor.dispose()
+})
+
+it('preserves the original mocked response headers casing in "rawHeaders"', async () => {


I've also moved the raw headers test from NodeClientRequest.test.ts to a new compliance test. I think it makes sense to be here.

kettanaito

This looks good from my side now. @mikicho, let me know if we should merge this.

mikicho · 2023-10-03T18:26:31Z

@kettanaito LGTM! Thanks! ❤️
I still need to verify the support limit with Nock's maintainer(s?), but I think I can make a good workaround until 18.14 is widely used and even deprecated.

Just a quick note, I see the value of sticking with a common and familiar object like Response, but it can also restrict us in unforeseen ways. It might be beneficial to consider a solution that can help us avoid these limitations moving forward.

kettanaito · 2023-10-04T09:40:25Z

It might be beneficial to consider a solution that can help us avoid these limitations moving forward.

You may be right. But the way I see it—the Fetch API is extremely powerful and is the future of JavaScript. It's limitations are imposed only by our familiarity with, frankly, archaic ways of dealing with network requests in Node.js (looking at you, http.IncomingMessage). Things like header sensitivity have little practice use since most specs declare headers as case-insensitive. I'd even go as far as to say that writing tests for rawHeaders is testing Node's implementation details but I understand that the line between those is rather blurred (has been blurred for years when using Node).

kettanaito · 2023-10-08T10:13:05Z

Released: v0.25.7 🎉

This has been released in v0.25.7!

📄 Release notes
📦 npm package

Make sure to always update to the latest version (npm i @mswjs/interceptors@latest) to get the newest features and bug fixes.

Predictable release automation by @ossjs/release.

mikicho added 2 commits September 30, 2023 22:52

fix raw headers

821a587

better test name

c82ce84

kettanaito reviewed Oct 1, 2023

View reviewed changes

src/interceptors/ClientRequest/NodeClientRequest.ts Outdated Show resolved Hide resolved

kettanaito requested changes Oct 1, 2023

View reviewed changes

revert curr solution + remove only

a0dade3

mikicho changed the title ~~fix(ClientRequest) don't lowercase the rawHeaders~~ fix(ClientRequest): don't lowercase the rawHeaders Oct 1, 2023

kettanaito changed the title ~~fix(ClientRequest): don't lowercase the rawHeaders~~ fix(ClientRequest): preserve headers casing on mocked responses Oct 3, 2023

kettanaito added 2 commits October 3, 2023 10:24

fix(ClientRequest): preserve mocked response headers casing

187c9e6

test(ClientRequest): move raw headers casing test to compliance

62c8102

kettanaito reviewed Oct 3, 2023

View reviewed changes

kettanaito added 4 commits October 3, 2023 10:31

chore(ci): bump node to 18.14.0

91be9f7

chore: debug

0424cf0

fix(getRawFetchHeaders): account for older versions of node

9b1ab33

test(getRawFetchHeaders): test multi-value headers

19dd0ba

kettanaito approved these changes Oct 3, 2023

View reviewed changes

kettanaito added the release candidate label Oct 3, 2023

kettanaito merged commit fa5a34a into mswjs:main Oct 4, 2023
1 check passed

mikicho deleted the Michael/fix-raw-headers/448 branch October 4, 2023 10:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(ClientRequest): preserve headers casing on mocked responses #463

fix(ClientRequest): preserve headers casing on mocked responses #463

mikicho commented Sep 30, 2023 •

edited

Loading

kettanaito Oct 1, 2023

kettanaito left a comment •

edited

Loading

mikicho commented Oct 1, 2023

kettanaito commented Oct 1, 2023

mikicho commented Oct 1, 2023 •

edited

Loading

kettanaito commented Oct 1, 2023 •

edited

Loading

mikicho commented Oct 1, 2023

kettanaito commented Oct 1, 2023

kettanaito commented Oct 3, 2023

kettanaito Oct 3, 2023

kettanaito Oct 3, 2023

kettanaito Oct 3, 2023

kettanaito Oct 3, 2023

mikicho Oct 3, 2023 •

edited

Loading

kettanaito Oct 3, 2023

kettanaito Oct 3, 2023 •

edited

Loading

kettanaito Oct 3, 2023

kettanaito Oct 3, 2023

kettanaito left a comment

mikicho commented Oct 3, 2023

kettanaito commented Oct 4, 2023

kettanaito commented Oct 8, 2023

fix(ClientRequest): preserve headers casing on mocked responses #463

fix(ClientRequest): preserve headers casing on mocked responses #463

Conversation

mikicho commented Sep 30, 2023 • edited Loading

Choose a reason for hiding this comment

kettanaito left a comment • edited Loading

Choose a reason for hiding this comment

mikicho commented Oct 1, 2023

kettanaito commented Oct 1, 2023

mikicho commented Oct 1, 2023 • edited Loading

kettanaito commented Oct 1, 2023 • edited Loading

mikicho commented Oct 1, 2023

kettanaito commented Oct 1, 2023

kettanaito commented Oct 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikicho Oct 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kettanaito Oct 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kettanaito left a comment

Choose a reason for hiding this comment

mikicho commented Oct 3, 2023

kettanaito commented Oct 4, 2023

kettanaito commented Oct 8, 2023

Released: v0.25.7 🎉

mikicho commented Sep 30, 2023 •

edited

Loading

kettanaito left a comment •

edited

Loading

mikicho commented Oct 1, 2023 •

edited

Loading

kettanaito commented Oct 1, 2023 •

edited

Loading

mikicho Oct 3, 2023 •

edited

Loading

kettanaito Oct 3, 2023 •

edited

Loading