PDS pipethrough optimizations #2770
Conversation
matthieusieben
force-pushed
the
msieben/micro-optimizations
branch
4 times, most recently
from
7d9350c to
f629fc7
Compare
matthieusieben
force-pushed
the
msieben/micro-optimizations
branch
from
956c637 to
343dc79
Compare
matthieusieben
changed the title
matthieusieben
force-pushed
the
msieben/micro-optimizations
branch
from
904663b to
10da838
Compare
|
Still need to make sure tests pass and add some more |
| @@ -43,15 +43,6 @@ export function ssrfFetchWrap<C = FetchContext>({ | |||
| } | |||
|
|
|||
| if (url.protocol === 'http:' || url.protocol === 'https:') { | |||
| // @ts-expect-error non-standard option | |||
| if (request.dispatcher) { | |||
There was a problem hiding this comment.
dispatcher was never exposed on Request
|
|
||
| if (allowUnknownTld !== true && !isValidDomain(url.hostname)) { |
There was a problem hiding this comment.
Will be performed through the lookup fn
packages/xrpc-server/src/server.ts
Outdated
| }) | ||
| } | ||
| if (isHandlerPipeThroughBuffer(outputUnvalidated)) { | ||
| setHeaders(res, outputUnvalidated) |
packages/xrpc-server/src/server.ts
Outdated
| } | ||
| if (!Object.values(v['headers']).every(isString)) { | ||
| return false | ||
| const createValidator = |
There was a problem hiding this comment.
I think we avoided zod here because it can be kinda heavy duty for validation of stuff that is often invalid. I know it used to handle this by throwing and catching errors which can be rough on Node so we tried to keep it out of the happy/critical path of our server stuff.
But I also have some vague memory that they fixed this & made it faster? Any idea? Or any idea how these compare?
There was a problem hiding this comment.
You are right, validating here is way overkill. Since we know the input is of type HandlerOutput, all we need is to discriminate based on that. We can even make some of the checks we currently do disappear !
packages/xrpc-server/src/server.ts
Outdated
| const createValidator = | ||
| <T>(schema: ZodSchema<T>) => | ||
| (v: unknown): v is T => | ||
| schema.safeParse(v).success |
There was a problem hiding this comment.
we have a helper function for this that we use in most places in common-web/check
check.is(v, schema)
There was a problem hiding this comment.
Added a check.create(schema) as the semantics I needed are a bit different
packages/xrpc-server/src/util.ts
Outdated
| return nsid.endsWith('/') ? nsid.slice(0, -1) : nsid // trim trailing slash | ||
| // /!\ Hot path | ||
|
|
||
| const originalUrl = ('originalUrl' in req && req.originalUrl) || req.url |
There was a problem hiding this comment.
could we rewrite as ternary? may be a bit easier to read on first glance
| @@ -23,42 +30,52 @@ export const getLocalLag = (local: LocalRecords): number | undefined => { | |||
| return Date.now() - new Date(oldest).getTime() | |||
| } | |||
|
|
|||
| export const handleReadAfterWrite = async <T>( | |||
| export const pipethroughReadAfterWrite = async <T>( | |||
packages/pds/src/context.ts
Outdated
| const safeAgent = new undici.Agent({ | ||
| allowH2: cfg.fetch.allowHTTP2, // This is experimental | ||
| headersTimeout: cfg.fetch.headersTimeout, | ||
| maxResponseSize: cfg.fetch.maxResponseSize, |
There was a problem hiding this comment.
are we cool re-using these config values for both the safeFetch & safeAgent?
only issue I could think of is if we have different tradeoffs on maxResponseSize
packages/pds/src/context.ts
Outdated
| @@ -256,12 +263,36 @@ export class AppContext { | |||
| appviewCdnUrlPattern: cfg.bskyAppView?.cdnUrlPattern, | |||
| }) | |||
|
|
|||
| const safeAgent = new undici.Agent({ | |||
There was a problem hiding this comment.
I almost want to call this proxyAgent or safeProxyAgent or something similar to give the sense of what it's for since it's only use there
packages/pds/src/pipethrough.ts
Outdated
| reqInit: RequestInit, | ||
| ): Promise<Response> => { | ||
| let res: Response | ||
| type Accept = [name: string, flags: { q: string }] |
There was a problem hiding this comment.
thoughts on
type Accept = {
name: string
q: string
}
instead of the array syntax? seems more clear & type-safe
There was a problem hiding this comment.
The idea of this code is to be agnostic of the accept-* header being negotiated.
Consider for example:
Accept-Patch: text/example;charset=utf-8, text/example
In that case, the Accept type could be generalized as such:
type Accept = [name: string, flags: Record<string, string>]When there is an obvious case for generalization, I like to write generic code. I guess this boils down to coding style preferences.
I can make this code more straight forward if you prefer or if you think this is too much YAGNI ?
What type safety difference are you thinking of ?
matthieusieben
force-pushed
the
msieben/micro-optimizations
branch
from
205bbae to
f1639a5
Compare
No description provided.