netlify dev + custom 404 page = unicode rendering problems

[jbrains]

Describe the bug

I’m creating a custom 404 page for my Jekyll site that uses some HTML5 entities (“smart” single quote, non-breaking space). There appears to be a Unicode problem when I preview the site using netlify dev compared to when I use Jekyll’s own web server (webrick, I guess).

When serving the site with webrick/jekyll serve, the text looks normal. When serving the site with netlify dev, there is the usual Unicode problem where the smart single quote is rendered as ’.

All my other pages are rendered correctly when serving the site with netlify dev. So far I’ve only seen this problem on the custom 404 page. All the pages have the same <meta charset> tag set to UTF-8.

I do not notice any difference between the generated 404.html file and (for example) the generated index.html that might cause them to be rendered differently, so I conclude that this points to a defect somewhere.

To Reproduce

I have not tried to produce a minimal failing example, so I don't know which of these steps are irrelevant.

Steps to reproduce the behavior:

1. Create a Jekyll project.
2. Create a custom 404 Markdown document ($SOURCE_ROOT/404.md) and add text that uses an apostrophe, such as "can't".
3. Create a welcome Markdown document ($SOURCE_ROOT/index.md) and add text that uses an apostrophe.
4. Add a rule to _redirects that ensures that the custom 404 page will handle HTTP status 404 responses.
5. netlify build. Notice that the HTML pages look correct in a text editor.
6. netlify dev, visit both pages, and notice that the 404 page renders incorrectly ("can’t"), although the welcome page renders correctly.
7. jekyll serve, visit both pages, and notice that the 404 page renders correctly.

Configuration

• If possible, please copy/paste below your netlify.toml.

[build]
  command = "jekyll build"
  publish = "openshift-www.jbrains.ca/www"

• Please enter the following command in a terminal and copy/paste its output:

npx envinfo --system --binaries --npmPackages netlify-cli --npmGlobalPackages netlify-cli

  System:
    OS: Linux 5.8 Pop!_OS 20.10
    CPU: (8) x64 Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
    Memory: 10.83 GB / 31.24 GB
    Container: Yes
    Shell: 5.8 - /usr/bin/zsh
  Binaries:
    Node: 15.7.0 - ~/.n/bin/node
    Yarn: 1.22.10 - ~/.npm-packages/bin/yarn
    npm: 7.4.3 - ~/.n/bin/npm
    Watchman: 4.9.0 - /usr/bin/watchman
  npmGlobalPackages:
    netlify-cli: 3.10.2

Expected behavior

Both the welcome page and the 404 page should be rendered with the correct Unicode character set.

CLI Output

The CLI output looks normal.

◈ Netlify Dev ◈
◈ Ignored general context env var: LANG (defined in process)
◈ Ignored general context env var: LANGUAGE (defined in process)
◈ Starting Netlify Dev with jekyll
Configuration file: /home/jbrains/Workspaces/www.jbrains.ca/_config.yml
            Source: /home/jbrains/Workspaces/www.jbrains.ca/source
       Destination: /home/jbrains/Workspaces/www.jbrains.ca/openshift-www.jbrains.ca/www
 Incremental build: disabled. Enable with --incremental
      Generating... 
                    done in 1.291 seconds.
 Auto-regeneration: enabled for 'source/'
    Server address: http://127.0.0.1:4000/
  Server running... press ctrl-c to stop.

   ┌─────────────────────────────────────────────────┐
   │                                                 │
   │   ◈ Server now ready on http://localhost:8888   │
   │                                                 │
   └─────────────────────────────────────────────────┘

Additional context

I have not noticed this problem before, so I can't provide any clues as to whatever this behavior has been recently introduced by a change. Sorry!

[TrentonAdams]

Is this not entirely about the content-type header? e.g. text/html;charset=utf-8. One is properly setting it and the other is not.

[jbrains]

@TrentonAdams Confirmed.

netlify dev sets the content-type header to content-type: text/html; charset=ISO-8859-1 instead of UTF-8.

[jbrains]

If someone could point me in the general direction of where to set this response header, I would try to submit a pull request for this problem.

[Morendil]

Same issue as #979, it seems, so not likely to be a recent change…

[Morendil]

It looks as if this might be a Jekyll issue, based on jekyll/jekyll#8005 - the difference in behavior between the two might be due to headers being rewritten (perhaps in a different order) or some such. (ETA: it seems that Jekyll returns a constant for Content-Type… not much wiggle room for a bug here. Maybe it gets rewritten later? A subtle hint from that issue is that the header is "utf-8" in lowercase for non-error pages and "UTF-8" in uppercase for error pages.)

[TrentonAdams]

@jbrains

netlify dev sets the content-type header to content-type: text/html; charset=ISO-8859-1 instead of UTF-8.

Tweet successful! hehe. I've run into that a few times working at a University people put in all sorts of characters we need to support.

[jbrains]

I noticed this difference in behavior, when running a site in netlify dev:

• Visit the 404 page directly (URI /404 with status OK) => content-type header set to UTF-8 and the page renders correctly
• Visit the 404 page indirectly (request with status "Not Found") => content-type header set to ASCII and the page renders incorrectly

When running a site in jekyll serve, in both cases, the content-type header is set to ASCII first, then to UTF-8. I presume that UTF-8 wins by being last in the headers list.

Now I wonder whether there are two related problems: one that accounts for the difference under netlify dev and one that accounts for the extraneous header under jekyll serve.

[jbrains]

The jekyll serve command adds its own content-type header, starting with jekyll/jekyll@3af22f1 which purported to fix issue jekyll/jekyll#2289

The problem now seems that while this patch helps when we serve with jekyll serve, for this to work in general, each method of serving the custom 404 page would need to be aware of the problem and set the content-type header appropriately. Is there a way to push this decision closer to the right place? Where is the ideal place to do this?

This seems like maybe a WEBrick problem and the jekyll serve patch is trying to fix that. That might mean that netlify dev has absolutely nothing to do with this. I'll investigate that now and update this issue once I feel confident.

[jbrains]

I am quite sure now that this is a WEBrick problem. The content-type header seems to be set differently in at least three situations:

• OK/200 when serving a file from the document root => Content-Type: text/html with no charset specified
• OK/200 when serving the built-in index page => Content-Type: text/html; charset="UTF-8"
• Not Found/404 => Content-Type: text/html; charset=ISO-8859-1

¯\_(ツ)_/¯

At a minimum, it seems that Netlify Dev does not contribute to this problem; it's all WEBrick 1.7.0, so I'm closing the issue. I hope that all this detail helps someone in the future.

[jbrains]

Oh, but wait... something strange is still happening with Netlify Dev. The content-type header I get depends on whether I explicitly go through jekyll serve. I have the default ports set up: 4000 for jekyll serve and 8888 for netlify dev.

• http://localhost:8888 => only the content-type header with ASCII, presumably going directly to WEBrick
• http://localhost:4000 => also the extra content-type header from jekyll serve that specifies UTF-8

So it appears that netlify dev handles the 404 response before jekyll serve can handle it. Or something. Is this as it should be?