Crates.io returns 404 when not specifying Accept: text/html

[kud1ing]

• awesome-rust uses awesome_bot to check links. Unfortunately links to crates.io return 404.
• curl https://crates.io/keywords/cassandra receives a 404, too.

Current summary

• This is confirmed as a bug that still exists as long as this issue is open. We do not need any further confirmations of the issue at this time.
• What we do need is a PR fixing the problem, those are quite welcome
• The workaround for this issue is to pass an Accept header of text/html.

[carols10cents]

I think we are doing something nonstandard, but I'm not sure what. Until we figure this out, a workaround is specifying HTML as the content type you want:

curl -v 'https://crates.io/crates/assert_approx_eq' -H 'Accept: text/html'

returns a 200 for me. Hope that helps!

[kud1ing]

crates.io was whitelisted in rust-unofficial/awesome-rust#310
I'd like to revert this in the end so that we can verify links to crates, categories and keywords.

[Nemo157]

Yeah, this is caused by

crates.io/src/dist.rs

Lines 38 to 52 in a20eb96

	// Second, if we're requesting html, then we've only got one page so
	// serve up that page. Otherwise proxy on to the rest of the app.
	let wants_html = req.headers()
	.find("Accept")
	.map(|accept| accept.iter().any(|s| s.contains("html")))
	.unwrap_or(false);
	if wants_html {
	self.dist.call(&mut RequestProxy {
	other: req,
	path: Some("/index.html"),
	method: None,
	})
	} else {
	self.handler.as_ref().unwrap().call(req)
	}

Interestingly requesting any URL with Accept: text/html will return 200 since the server doesn't know what the valid routes are (e.g. curl -H 'Accept: text/html' -I https://crates.io/foo/bar)

I don't think there's any way to have a bot validate URLs on crates.io without executing the javascript and checking if it loads the page successfully. At least, while crates.io does client-side only rendering, if server-side rendering of at least the initial page is ever added that should then know whether the URL being requested is valid or not.

[locks]

FYI, you can track server-side rendering efforts at #819.

[AaronFriel]

This is still not fixed.

[carols10cents]

@AaronFriel Yes, that's why this issue is still open.

[AaronFriel]

Would a commit altering the wants_html test to include wildcards be accepted? I'm just not sure why this has languished and I'm guessing there is something I'm missing.

[carols10cents]

Yes, I am not sure of the scope of the solution needed or the effects of various solutions on the frontend and backend-- please give it a try and let me know if you have questions!

[mark-i-m]

Any progress on this?

[locks]

@mark-i-m this is indirectly being tracked at #204

[luciusmagn]

Any updates on this?

[jtgeibel]

I've recently added PR #1788 which would impact this bug. This PR would make it so that the behavior is consistent, regardless of if an Accept: text/html header was sent or not. All such requests will now return a status 200 with the Ember index html (as is currently done for browsers and other clients that set the header).

This will in effect invert the issue described here. Instead of always returning a 404 with a JSON response, the site will now always respond with a status 200 with static HTML. Instead of false negatives for crates that do exists, there would be false positives for crates that do not exist.