Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement net.Socket interceptor #515

Open
wants to merge 76 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
b0f0e37
feat: implement socket interceptor
kettanaito Mar 3, 2024
4f48ecd
chore: guard against missing streams
kettanaito Mar 3, 2024
ac28a3c
chore: replay all suppressed events
kettanaito Mar 3, 2024
6053cfc
fix: respect auth, content-length assumption
kettanaito Mar 3, 2024
f8294ad
fix: set "credentials" to "same-origin"
kettanaito Mar 3, 2024
b9be9e6
chore: import order
kettanaito Mar 3, 2024
d8a6075
fix: always set request body stream if allowed
kettanaito Mar 3, 2024
0a67263
feat: implement "respondWith()"
kettanaito Mar 3, 2024
cff7820
chore: import "HTTPParser" from "node:_http_common"
kettanaito Mar 4, 2024
5d9a812
chore: abstract http message parser
kettanaito Mar 4, 2024
f7fded8
chore: introduce SocketWrap class
kettanaito Mar 5, 2024
4058080
test: add http compliance test suite
kettanaito Mar 5, 2024
1c74f23
fix: improve Socket compliance, add events tests
kettanaito Mar 5, 2024
a755028
test: migrate other http tests to socket interceptor
kettanaito Mar 5, 2024
095548c
feat(wip): agent-based socket interception
kettanaito Mar 9, 2024
d3df8e0
fix(MockHttpSocket): do not destroy the request stream
kettanaito Mar 9, 2024
cdf6c25
fix(MockHttpSocket): handle undefined writes on .end()
kettanaito Mar 9, 2024
3364c8e
feat: implement MockHttpsAgent
kettanaito Mar 9, 2024
7904858
fix: free the request parser on socket finish
kettanaito Mar 9, 2024
9c7794e
test: use new interceptor in response tests
kettanaito Mar 9, 2024
fa5827f
fix(MockHttpAgent): error with "Network error" on Response.error()
kettanaito Mar 9, 2024
48b3036
test: use the new interceptor in rest of response tests
kettanaito Mar 9, 2024
5d9d422
test: add all http tests
kettanaito Mar 9, 2024
00bb31a
chore: delete old "ClientRequestInterceptor"
kettanaito Mar 9, 2024
3c7b902
chore: add "_http_common.d.ts"
kettanaito Mar 10, 2024
aaa59b8
test: http-req-callback
kettanaito Mar 10, 2024
a81e947
fix(MockHttpAgent): support modifying request headers
kettanaito Mar 10, 2024
3dac2a0
test: fix invalid http-req-write callback tests
kettanaito Mar 10, 2024
bfa651b
test(http-request-without-options): remove non-mocked cases
kettanaito Mar 10, 2024
01ad60a
fix(MockHttpAgent): write headers for responses without body
kettanaito Mar 10, 2024
81f4cfc
fix(MockHttpSocket): preserve mocked response header name casing
kettanaito Mar 10, 2024
d41d16f
fix: implement the "response" event
kettanaito Mar 10, 2024
6e303ec
chore: remove logs
kettanaito Mar 10, 2024
6b4d06c
test(ClientRequest): adjust statusMessage assertion for inferred code
kettanaito Mar 10, 2024
640c9bf
fix(ClientRequest): restore https from https module
kettanaito Mar 10, 2024
dd53781
chore: remove unused "createRequest"
kettanaito Mar 10, 2024
4da4341
test: try fixing other tests
kettanaito Mar 10, 2024
635d12f
chore: use "NODE_TLS_REJECT_UNAUTHORIZED" flag for testing vs "httpsA…
kettanaito Mar 10, 2024
4972019
fix: update @types/node to support Readable.toWeb()
kettanaito Mar 10, 2024
46e51e4
test: fix "intecept/fetch" body assertions
kettanaito Mar 10, 2024
78d4ed3
fix: use "URL" from "node:url" and cast to string
kettanaito Mar 10, 2024
8fdb4b5
fix(MockHttpSocket): rely on "x-request-id" for event deduplication
kettanaito Mar 11, 2024
caeb168
chore: remove "SocketInterceptor"
kettanaito Mar 11, 2024
deff618
test: add mocked response body for HEAD request
kettanaito Mar 11, 2024
298d4ee
fix(MockHttpSocket): support empty ReadableStream in mocked responses
kettanaito Mar 11, 2024
13aea5b
fix(MockHttpSocket): emit correct events for TLS connections
kettanaito Mar 11, 2024
68ad529
chore: remove console.logs
kettanaito Mar 11, 2024
2cb79f3
test(MockSocket): add unit tests
kettanaito Mar 11, 2024
848ef00
docs: elaborate on when we free the parsers
kettanaito Mar 11, 2024
ac36cc6
docs: remove jest mention from the test
kettanaito Mar 11, 2024
81691ef
Merge branch 'main' into feat/yet-another-socket-interceptor
kettanaito Mar 21, 2024
700d34f
test: increase timeout for http-socket-timeout test
kettanaito Mar 21, 2024
230c238
chore: skip follow-redirect-http test
kettanaito Mar 21, 2024
2b1a5d7
fix: support Readable as request body (#527)
kettanaito Mar 21, 2024
3348dd7
fix(ClientRequest): spread object args to support "follow-redirects" …
kettanaito Mar 30, 2024
035bbe4
Merge branch 'main' into feat/yet-another-socket-interceptor
kettanaito Mar 31, 2024
92343e6
fix(MockHttpSocket): rely on internal request id header name
kettanaito Mar 31, 2024
69cffee
Merge branch 'main' into feat/yet-another-socket-interceptor
kettanaito Apr 12, 2024
824998b
fix(MockHttpSocket): exhaust .write() callbacks for mocked requests (…
kettanaito Apr 12, 2024
b49a316
Merge branch 'main' into feat/yet-another-socket-interceptor
kettanaito Apr 15, 2024
66c6046
fix: add "address()" on mock Socket (#549)
mikicho Apr 16, 2024
b170116
Merge branch 'main' into feat/yet-another-socket-interceptor
kettanaito Apr 16, 2024
6783a28
Merge branch 'main' into feat/yet-another-socket-interceptor
kettanaito Apr 17, 2024
ab8656e
docs: edit the algorithms section
kettanaito Apr 17, 2024
085d1ec
fix(MockHttpSocket): handle response stream errors (#548)
kettanaito Apr 17, 2024
430c65e
fix(MockHttpSocket): forward tls socket properties (#556)
kettanaito Apr 17, 2024
fc73636
chore: clean up ClientRequest utils (#557)
kettanaito Apr 18, 2024
e52851c
test: add `setTimeout` tests (#558)
kettanaito Apr 18, 2024
c362b39
test(MockHttpSocket): add "signal" tests (#559)
kettanaito Apr 18, 2024
8ff98db
fix(MockHttpSocket): set tls socket properties on init (#561)
mikicho Apr 21, 2024
f099479
Merge branch 'main' into feat/yet-another-socket-interceptor
kettanaito Apr 24, 2024
27ed6cd
merge main - resolve conflicts
mikicho Apr 28, 2024
5bbf61a
fix(ClientRequest): support "unhandledException" event
kettanaito Apr 29, 2024
b1481a3
fix: respect IPv6 hostnames and family (#571)
mikicho Apr 30, 2024
664a363
Merge branch 'main' into feat/yet-another-socket-interceptor
kettanaito Apr 30, 2024
7ab764e
test(http): add "connect" and "secureConnect" tests (#576)
kettanaito Jun 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
95 changes: 56 additions & 39 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,81 +11,98 @@ This library supports intercepting the following protocols:

## Motivation

While there are a lot of network communication mocking libraries, they tend to use request interception as an implementation detail, giving you a high-level API that includes request matching, timeouts, retries, and so forth.
While there are a lot of network mocking libraries, they tend to use request interception as an implementation detail, giving you a high-level API that includes request matching, timeouts, recording, and so forth.

This library is a strip-to-bone implementation that provides as little abstraction as possible to execute arbitrary logic upon any request. It's primarily designed as an underlying component for high-level API mocking solutions such as [Mock Service Worker](https://github.com/mswjs/msw).
This library is a barebones implementation that provides as little abstraction as possible to execute arbitrary logic upon any request. It's primarily designed as an underlying component for high-level API mocking solutions such as [Mock Service Worker](https://github.com/mswjs/msw).

### How is this library different?

A traditional API mocking implementation in Node.js looks roughly like this:

```js
import http from 'http'

function applyMock() {
// Store the original request module.
const originalHttpRequest = http.request

// Rewrite the request module entirely.
http.request = function (...args) {
// Decide whether to handle this request before
// the actual request happens.
if (shouldMock(args)) {
// If so, never create a request, respond to it
// using the mocked response from this blackbox.
return coerceToResponse.bind(this, mock)
}

// Otherwise, construct the original request
// and perform it as-is (receives the original response).
return originalHttpRequest(...args)
import http from 'node:http'

// Store the original request function.
const originalHttpRequest = http.request

// Override the request function entirely.
http.request = function (...args) {
// Decide if the outgoing request matches a predicate.
if (predicate(args)) {
// If it does, never create a request, respond to it
// using the mocked response from this blackbox.
return coerceToResponse.bind(this, mock)
}

// Otherwise, construct the original request
// and perform it as-is.
return originalHttpRequest(...args)
}
```

This library deviates from such implementation and uses _class extensions_ instead of module rewrites. Such deviation is necessary because, unlike other solutions that include request matching and can determine whether to mock requests _before_ they actually happen, this library is not opinionated about the mocked/bypassed nature of the requests. Instead, it _intercepts all requests_ and delegates the decision of mocking to the end consumer.
The core philosophy of Interceptors is to _run as much of the underlying network code as possible_. Strange for a network mocking library, isn't it? Turns out, respecting the system's integrity and executing more of the network code leads to more resilient tests and also helps to uncover bugs in the code that would otherwise go unnoticed.

Interceptors heavily rely on _class extension_ instead of function and module overrides. By extending the native network code, it can surgically insert the interception and mocking pieces only where necessary, leaving the rest of the system intact.

```js
class NodeClientRequest extends ClientRequest {
async end(...args) {
// Check if there's a mocked response for this request.
// You control this in the "resolver" function.
const mockedResponse = await resolver(request)

// If there is a mocked response, use it to respond to this
// request, finalizing it afterward as if it received that
// response from the actual server it connected to.
class XMLHttpRequestProxy extends XMLHttpRequest {
async send() {
// Call the request listeners and see if any of them
// returns a mocked response for this request.
const mockedResponse = await waitForRequestListeners({ request })

// If there is a mocked response, use it. This actually
// transitions the XMLHttpRequest instance into the correct
// response state (below is a simplified illustration).
if (mockedResponse) {
this.respondWith(mockedResponse)
this.finish()
// Handle the response headers.
this.request.status = mockedResponse.status
this.request.statusText = mockedResponse.statusText
this.request.responseUrl = mockedResponse.url
this.readyState = 2
this.trigger('readystatechange')

// Start streaming the response body.
this.trigger('loadstart')
this.readyState = 3
this.trigger('readystatechange')
await streamResponseBody(mockedResponse)

// Finish the response.
this.trigger('load')
this.trigger('loadend')
this.readyState = 4
return
}

// Otherwise, perform the original "ClientRequest.prototype.end" call.
return super.end(...args)
// Otherwise, perform the original "XMLHttpRequest.prototype.send" call.
return super.send(...args)
}
}
```

By extending the native modules, this library actually constructs requests as soon as they are constructed by the consumer. This enables all the request input validation and transformations done natively by Node.js—something that traditional solutions simply cannot do (they replace `http.ClientRequest` entirely). The class extension allows to fully utilize Node.js internals instead of polyfilling them, which results in more resilient mocks.
> The request interception algorithms differ dramatically based on the request API. Interceptors acommodate for them all, bringing the intercepted requests to a common ground—the Fetch API `Request` instance. The same applies for responses, where a Fetch API `Response` instance is translated to the appropriate response format.

This library aims to provide _full specification compliance_ with the APIs and protocols it extends.

## What this library does

This library extends (or patches, where applicable) the following native modules:
This library extends the following native modules:

- `http.get`/`http.request`
- `https.get`/`https.request`
- `XMLHttpRequest`
- `fetch`
- `WebSocket`

Once extended, it intercepts and normalizes all requests to the Fetch API `Request` instances. This way, no matter the request source (`http.ClientRequest`, `XMLHttpRequest`, `window.Request`, etc), you always get a specification-compliant request instance to work with.

You can respond to the intercepted request by constructing a Fetch API Response instance. Instead of designing custom abstractions, this library respects the Fetch API specification and takes the responsibility to coerce a single response declaration to the appropriate response formats based on the request-issuing modules (like `http.OutgoingMessage` to respond to `http.ClientRequest`, or updating `XMLHttpRequest` response-related properties).
You can respond to the intercepted HTTP request by constructing a Fetch API Response instance. Instead of designing custom abstractions, this library respects the Fetch API specification and takes the responsibility to coerce a single response declaration to the appropriate response formats based on the request-issuing modules (like `http.OutgoingMessage` to respond to `http.ClientRequest`, or updating `XMLHttpRequest` response-related properties).

## What this library doesn't do

- Does **not** provide any request matching logic;
- Does **not** decide how to handle requests.
- Does **not** handle requests by default.

## Getting started

Expand Down
47 changes: 47 additions & 0 deletions _http_common.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
declare var HTTPParser: {
new (): HTTPParser<number>
REQUEST: 0
RESPONSE: 1
readonly kOnHeadersComplete: unique symbol
readonly kOnBody: unique symbol
readonly kOnMessageComplete: unique symbol
}

export interface HTTPParser<ParserType extends number> {
new (): HTTPParser<ParserType>

[HTTPParser.kOnHeadersComplete]: ParserType extends 0
? RequestHeadersCompleteCallback
: ResponseHeadersCompleteCallback
[HTTPParser.kOnBody]: (chunk: Buffer) => void
[HTTPParser.kOnMessageComplete]: () => void

initialize(type: ParserType, asyncResource: object): void
execute(buffer: Buffer): void
finish(): void
free(): void
}

export type RequestHeadersCompleteCallback = (
versionMajor: number,
versionMinor: number,
headers: Array<string>,
idk: number,
path: string,
idk2: unknown,
idk3: unknown,
idk4: unknown,
shouldKeepAlive: boolean
) => void

export type ResponseHeadersCompleteCallback = (
versionMajor: number,
versionMinor: number,
headers: Array<string>,
method: string | undefined,
url: string | undefined,
status: number,
statusText: string,
upgrade: boolean,
shouldKeepAlive: boolean
) => void
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@
"@types/express-rate-limit": "^6.0.0",
"@types/follow-redirects": "^1.14.1",
"@types/jest": "^27.0.3",
"@types/node": "^16.11.26",
"@types/node": "18",
"@types/node-fetch": "2.5.12",
"@types/supertest": "^2.0.11",
"@types/ws": "^8.5.10",
Expand Down Expand Up @@ -198,4 +198,4 @@
"path": "./node_modules/cz-conventional-changelog"
}
}
}
}