fix(utils): remove non-ASCII limitation for path normalization

[birjj]

Pre-flight checklist

• I have read the Contributing Guidelines on pull requests.
• If this is a code change: I have written unit tests and/or added dogfooding pages to fully verify the new behavior.
• If this is a new API or substantial change: the PR has an accompanying issue (closes #0000) and the maintainers have approved on my working plan.

Motivation

Fixes #8124

The issue specifically pertains to index file detection for paths that contain non-ASCII letters (e.g. "æ"). This failed previously, due to a test for non-ASCII characters when normalizing file paths. This test originated in the sindresorhus/slash package, and was copied over when the code was integrated into docusaurus - but doesn't appear to match how Windows works.

See that issue for further discussion, or sindresorhus/slash#19 for discussion on limitation in the originating package. The limitation was removed in the originating package in sindresorhus/slash#20

Test Plan

Updated tests in this repo, and verified by modifying node_modules directly in https://github.com/birjj/docusaurus-repro-8124 that it works for the specific issue encountered:

For the file structure

docs
 ├─ abc
 │  └─ index.md
 └─ æøå
    └─ index.md

Test links

Deploy preview: https://deploy-preview-8137--docusaurus-2.netlify.app/ (note that this isn't relevant, as the issue is Windows-specific)

Related issues/PRs

#8124
sindresorhus/slash#19
sindresorhus/slash#20

[facebook-github-bot]

Hi @birjj!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

[netlify]

✅ [V2]

Name	Link
🔨 Latest commit	70d31a3
🔍 Latest deploy log	https://app.netlify.com/sites/docusaurus-2/deploys/63400b333b0c0000092d1f6d
😎 Deploy Preview	https://deploy-preview-8137--docusaurus-2.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[github-actions]

⚡️ Lighthouse report for the deploy preview of this PR

URL	Performance	Accessibility	Best Practices	SEO	PWA	Report
/	🟠 75	🟢 98	🟢 100	🟢 100	🟠 80	Report
/docs/installation	🟠 73	🟢 100	🟢 100	🟢 100	🟢 90	Report

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[Josh-Cena]

There's another escapePath utility we use to get around the issue. I think we can merge the two now.

[slorber]

Also questioned this part of the code before, see some past refs here:

• image file name can be in Chinese or other languages #4730 (comment)
• add needCheckAscii param as second param sindresorhus/slash#16 (comment)

Was wondering: is there a way to prevent this issue to resurface again with CI? Any idea to share? (we have Wndows CI but no e2e tests infra yet)

We have a website/_dogfood folder to have tests, is there a way to make the Docusaurus website build crash just using some md files + windows?

---

There's another escapePath utility we use to get around the issue. I think we can merge the two now.

The implementation looks quite different, if we want to dedup I'd rather do that in another dedicated PR as this looks more risky 😰

/**
 * When you have a path like C:\X\Y
 * It is not safe to use directly when generating code
 * For example, this would fail due to unescaped \:
 * `<img src={require('${filePath}')} />`
 * But this would work: `<img src={require('${escapePath(filePath)}')} />`
 *
 * posixPath can't be used in all cases, because forward slashes are only valid
 * Windows paths when they don't contain non-ascii characters, and posixPath
 * doesn't escape those that fail to be converted.
 */
export function escapePath(str: string): string {
  const escaped = JSON.stringify(str);

  // Remove the " around the json string;
  return escaped.substring(1, escaped.length - 1);
}

[birjj]

I agree that escapePath is probably best handled in a separate PR - and by someone more familiar with the codebase ;)

Was wondering: is there a way to prevent this issue to resurface again with CI? Any idea to share? (we have Wndows CI but no e2e tests infra yet)

We have a website/_dogfood folder to have tests, is there a way to make the Docusaurus website build crash just using some md files + windows?

I don't believe there is a way to make it crash on build, unless there's some specific edge case handling of paths that crash if they aren't converted correctly - in which case that behavior probably wouldn't be good to rely on anyway. Since slugs and document IDs aren't affected, I doubt there is any intended check that would fail.

If there is a way to test site functionality, we could test whether the index is detected correctly (#8124). Unfortunately, as far as I can see, that isn't possible with the dogfood system.

[slorber]

Can't find a way to test it locally (I don't have windows) nor to make it fail=>pass in the CI so I'll have to trust it works 🤪

Thanks 👍