Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Escaped URLs are not processed #323

Closed
Yoshanuikabundi opened this issue Aug 9, 2021 · 3 comments · Fixed by #369
Closed

Escaped URLs are not processed #323

Yoshanuikabundi opened this issue Aug 9, 2021 · 3 comments · Fixed by #369
Assignees
@JustinBeckwith
Labels
bug Something isn't working released

Comments

@Yoshanuikabundi
Copy link

Hi,

Linkinator doesn't process escaped URLs correctly (or at all).

The following HTML:

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>Test</title>
</head>
<body>

<ol>
    <li><a href="this_%3a_is_a_colon.html"> Test an escaped colon </a></li>
    <li><a href="this_:_is_a_colon.html"> Test an unescaped colon </a></li>
    <li><a href="this_%20_is_a_space.html"> Test an escaped space </a></li>
    <li><a href="this_ _is_a_space.html"> Test an unescaped space </a></li>

</body>
</html>

Located in index.html of the following file tree:

index.html
this_:_is_a_colon.html
this_ _is_a_space.html

Gives the following output:

npx linkinator index.html
🏊‍♂️ crawling index.html
[200] index.html
[200] this_:_is_a_colon.html
[404] this_%3a_is_a_colon.html
[404] this_%20_is_a_space.html

  [200] index.html
index.html
  [200] this_:_is_a_colon.html
  [404] this_%3a_is_a_colon.html
  [404] this_%20_is_a_space.html
ERROR: Detected 2 broken links. Scanned 4 links in 0.028 seconds.

The escaped URLs are reported as 404s, and the link with a literal space in it is not reported at all.

In Firefox, all four links work. As you can see, this means that there is no way to specify a link that has a space in it in a way that linkinator will accept.

The file tree described above is available here
linkinator-url-escape.zip.

Thanks!
@Yoshanuikabundi Yoshanuikabundi mentioned this issue Aug 9, 2021
Merged
@antoniancu
Copy link

I have a similar issue with links that contain a period in their URL. For example: https://forum.unity.com/forums/profiler-previews.199 resolves, but Linkinator reports it as BROKEN with a status code of 0.

@JustinBeckwith
Copy link
Owner

@antoniancu I think you're experiencing an entirely different problem. That url you posted is weird, and I think may rely on some kind of cookie being persisted across requests. Even curl can't seem to follow it:

 $ curl --location  https://forum.unity.com/forums/profiler-previews.199
curl: (47) Maximum (50) redirects followed

If you run linkinator with --verbosity DEBUG, you can get at the underlying stack trace causing the failure:

{
    "message": "maximum redirect reached at: https://forum.unity.com/register/genesis?state=O4X89tUhHQVJ1RyOrYYzq6f96L3H3xfATg0XCeL1%3B%2Fforums%2Fprofiler-previews.199&error=login_required",
    "type": "max-redirect",
    "config": {
      "method": "HEAD",
      "url": "https://forum.unity.com/forums/profiler-previews.199",
      "headers": {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"
      },
      "responseType": "stream",
      "timeout": null
    }
  },
  {
    "message": "maximum redirect reached at: https://forum.unity.com/register/genesis?state=G2fHCIvy2CTzLXASzxJNksxAsQ3Glxq0oYhr8zi7%3B%2Fforums%2Fprofiler-previews.199&error=login_required",
    "type": "max-redirect",
    "config": {
      "method": "GET",
      "url": "https://forum.unity.com/forums/profiler-previews.199",
      "responseType": "stream",
      "headers": {
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36"
      },
      "timeout": null
    }
  },

I would have to do some serious digging to try and figure out how cookie persistence works across requests (if that's even a thing we want to do), and I just think that would overcomplicate things beyond being useful. Please feel free to open a separate issue to track this if you like though!

@JustinBeckwith JustinBeckwith added the bug Something isn't working label Dec 18, 2021
@JustinBeckwith JustinBeckwith self-assigned this Dec 18, 2021
@JustinBeckwith JustinBeckwith mentioned this issue Dec 18, 2021
Merged
@github-actions
Copy link

🎉 This issue has been resolved in version 3.0.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JustinBeckwith JustinBeckwith

Labels
bug Something isn't working released
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: decode path parts in local web server JustinBeckwith/linkinator
3 participants
@JustinBeckwith @antoniancu @Yoshanuikabundi