Update serve-static.ts

[Hoodgail]

Mistakes from the original code.

node-server/src/serve-static.ts

Line 29 in a685db9

const content = readFileSync(path)

This will load the whole file into memory.
If you attempt to serve an MP4 that is 1GB in size, 1GB will be loaded into memory. As a result, if you attempt to load that file 10 times and your server or computer only has 10GB of RAM, your memory will run out and your client will fail.
Use streams and range headers as a solution.
See Range headers for more information.

---

node-server/src/serve-static.ts

Lines 16 to 18 in a685db9

	if (c.finalized) {
	await next()
	}

The request will go on to the next handler if it is finished, but the code at the bottom will continue to execute as if the request was never concluded in the first place.

---

node-server/src/serve-static.ts

Line 31 in a685db9

const mimeType = getMimeType(path)

This issue has nothing to do with this repo, but it was missing a LOT of mimetypes, like MP4 videos, gltf, glb 3d files etc, i made a separate pull request to its repo at honojs/hono#827

[Hoodgail]

Works as expected

[Hoodgail]

I Found one issue with my pull request

206 partial content is not supported by the cache API, i'll fix it later, i forgot that the 206 status code is only sent once the client adds the the range header on the requests after the initial one, 200 is sent to the initial response

I'll fix when i'm available later.

[yusukebe]

Hi @Hoodgail !

Thank you for the PR. Please do three things:

1. Prettify source code. npx prettier src/serve-static.ts --write
2. Fix the type error as shown below.
3. Write proper tests.

[Hoodgail]

@yusukebe I made the first two fixes, but the test functions are already valid

[yusukebe]

Thanks. But, we have to add the following tests:

  it('Should return correct headers and data with range headers', async () => {
    let res = await request(server).get('/static/plain.txt').set('range', '0-9')
    expect(res.status).toBe(206)
    expect(res.headers['content-type']).toBe('text/plain; charset=utf-8')
    expect(res.headers['content-length']).toBe('10')
    expect(res.headers['content-range']).toBe('bytes 0-9/17')
    expect(res.text.length).toBe(10)
    expect(res.text).toBe('This is pl')

    res = await request(server).get('/static/plain.txt').set('range', '10-16')
    expect(res.status).toBe(206)
    expect(res.headers['content-type']).toBe('text/plain; charset=utf-8')
    expect(res.headers['content-length']).toBe('7')
    expect(res.headers['content-range']).toBe('bytes 10-16/17')
    expect(res.text.length).toBe(7)
    expect(res.text).toBe('ain.txt')
  })

And we also have to consider the case of exceptions.

await request(server).get('/static/plain.txt').set('range', '0-17')

Then it will be aborted.

[Hoodgail]

[yusukebe]

@Hoodgail

Thanks. Then how about this case?

it.only('Should not be aborted', async () => {
  await request(server).get('/static/plain.txt').set('range', '0-17')
})

This will fail with "aborted".

I think we have to handle the error in some way.

[mindplay-dk]

This is a good next step towards production ready static file server middleware. 👍

We need to eventually consider cache-headers (If-Modified-Since, ETag, etc.) in order to have predictable performance and reliable interop with e.g. cache proxy servers and CDNs etc. - there's a very complete implementation for Koa here that could be referenced.

I think, ideally, we should consider something like this proposal in order to avoid writing and maintaining complex static file-server logic for every platform - the proposition there is about interop between frameworks, but the same problem appears to apply to middleware even within the framework, and might could be addressed by something like this.

Don't let me distract you too much from just getting a working file-server going for Node though - that's still a useful step, and once that's proven, maybe we could start to think about extracting/abstracting the platform differences to achieve a production-ready static file service across platforms. 🤔

[Hoodgail]

@yusukebe I don't understand what you mean, shoulnt it be aborted after the all of the content is received? Unless there is a keep-alive header

[tangye1234]

Temperially I use a code snippet like this, I think serve-static is not production ready, event if it is almost getting close.
I only need to send a single file, so this code is not a replacement, while it may be a reference I can contribute :)

import { createReadStream, Stats } from 'fs'
import { stat } from 'fs/promises'
import type { Context } from 'hono'
import { getMimeType } from 'hono/utils/mime'
// Here I use Readable.toWeb, which is supported only in nodejs 18+, which can be replaced by many other means
import { Readable } from 'stream'

export function fileToResponse(
  c: Context<any, any, any>,
  path: string,
  type?: string
) {
  return stat(path)
    .then(stat => stream(c, path, stat, type), () => c.notFound())
    .catch((e: any) =>
      c.json({
        success: false,
        message: String(e?.message || 'Unknown error'),
      }, 500)
    )
}

function stream(
  c: Context<any, any, any>,
  path: string,
  stat: Stats,
  type?: string
) {
  if (!stat.isFile()) {
    return c.notFound()
  }

  const size = stat.size
  type = type || getMimeType(path) || 'application/octet-stream'

  const headers = {
    'Content-Type': type,
    'Content-Length': String(size),
  } as {
    'Content-Type': string
    'Content-Length': string
    'Content-Range'?: string
  }

  if (size === 0) {
    return c.body(null, 200, headers)
  }

  const range = c.req.header('range') || ''

  if (!range) {
    return c.body(Readable.toWeb(createReadStream(path)), 200, headers)
  }

  const m = range.match(/^(?<units>[^=]+)=(?<start>\d*)-(?<end>\d*)$/)

  const { units, start, end } = m?.groups || {}
  if (!m || !units) {
    return c.body(null, 400)
  }

  if (m[0].length !== range.length) {
    // not support multipart range request
    return c.body(null, 400)
  }

  if (units !== 'bytes') {
    return c.body(null, 400)
  }

  if (!start && !end) {
    // FIXME supported?
    return c.body(Readable.toWeb(createReadStream(path)), 200, headers)
  }

  if (!start) {
    const len = Math.min(Number(end), size)
    if (len === 0) {
      return c.body(null, 400)
    }

    if (len === size) {
      return c.body(Readable.toWeb(createReadStream(path)), 200, headers)
    }

    const s = size - len
    const e = size - 1

    headers['Content-Length'] = String(len)
    headers['Content-Range'] = `bytes ${s}-${e}/${size}`

    return c.body(
      Readable.toWeb(createReadStream(path, { start: s, end: e })),
      206,
      headers
    )
  }

  const e = Math.min(Number(end || size - 1), size)
  const s = Math.min(Number(start), e)
  const len = e - s + 1
  if (len === 0) {
    return c.body(null, 400)
  }

  if (len === size) {
    return c.body(Readable.toWeb(createReadStream(path)), 200, headers)
  }

  headers['Content-Length'] = String(len)
  headers['Content-Range'] = `bytes ${s}-${e}/${size}`

  return c.body(
    Readable.toWeb(createReadStream(path, { start: s, end: e })),
    206,
    headers
  )
}

[that-ambuj]

Hi @Hoodgail, I was the last person that worked on the serve-static middleware and I had built it from reference using Deno's serve-static middleware from the main repository. I see that my implementation had some issues but I'm also glad that I could learn from my mistakes. Since the current implementation was taken from the main repo, I think that you should apply these improvements there too.

[schonert]

Any news on this?

[yusukebe]

@schonert

Nothing. I'll be addressing this issue now.

[yusukebe]

Hi @Hoodgail

I apologize for the delay.

I've commented on the edge case. If you agree with my points, could you please revise your code and resolve the conflicts? Then, I can approve it.

[yusukebe]

We have to add the following code if the client range header value exceeds the data size:

if (stat.size < end - start + 1) {
  end = stat.size - 1
}

And should add the tests:

it('Should return correct headers and data if client range exceeds the data size', async () => {
  const res = await request(server).get('/static/plain.txt').set('range', '0-20')
  expect(res.status).toBe(206)
  expect(res.headers['content-type']).toBe('text/plain; charset=utf-8')
  expect(res.headers['content-length']).toBe('17')
  expect(res.headers['content-range']).toBe('bytes 0-16/17')
  expect(res.text.length).toBe(17)
  expect(res.text).toBe('This is plain.txt')
})

[mindplay-dk]

The implementation posted by @tangye1234 looks really clean and readable, with good separation of concerns - each function "does one thing and does it well". Should be very easy to unit test as well.

It would be great to have both the stream and fileToResponse functions not only as implementation details in a static file server, but exported - as these would be really helpful in specialized middleware besides static file-servers, e.g. caches or proxies to other servers or cloud file storage, image servers, and so on.

@tangye1234 with regards to your // FIXME supported?, yes, responding to a range request without a range header is permitted - servers are not required to support range, and the range header in the response indicates to the client that header was accepted and that the response is in fact partial; clients do not (must not) assume they will get a range response.

[yusukebe]

The PR #63 based on this PR is merge now. Thanks!