Blacklist rust-ci.org

[akubera]

rust-ci DOT org is no longer a legitimate website for hosting rust-related content, and is now an entrypoint for a dubious ad-network. Looks like there are ~61 tomls on github using that site for hosting documentation. I don't know if there is a blacklisting procedure in place, but adding that site may be a good idea.

[carols10cents]

How should we implement blacklisting?

• Should we not display the documentation link at all?
• Should we display the link but with a warning that it's to a known spam site?
• Should we display "documentation link has been blacklisted" text instead of the link?

That takes care of existing crates that link to a blacklisted domain; should we also disallow blacklisted domains on cargo publish to prevent more versions referencing this and alert crate authors of the problem? Should that be a warning or an error?

[carols10cents]

I'm seeing 35 crates that have rust-ci.org in their documentation link in crates.io's database:

alfred
apply_pub
capture
cc
cld2-sys
cpuid
crc24
duktape
epsilonz
epsilonz_algebra
fastcgi
fingertree
fmod
free
free_macros
GSL
jlens
leveldb-rs
monad_macros
noisy
number_words
pcre
pipes
rclist
rope
rust-GSL
rust-assimp
rust-iteratorcomprehensions
superchan
tailrec
tau
tinycdb
touptek
tweetust_macros
underscore

[carols10cents]

Excerpts from discussion on Reddit:

Perhaps this isn't the best forum for this, but I've found at least one link on crates.io that led to ads/malware. The crate (rustless) seemed to have a custom domain, which I'm guessing expired and then picked up by the kind of people that pick up old domain names to monetize any traffic.

Has there ever been discussion over using a whitelist of allowed domains to host rust documentation so that users can feel safe that clicking on the Documentation link will lead to a known-safe site?

---

I feel like this could be better accomplished by a blacklist than a whitelist. I definitely don't want to have malware linked on crates.io, but I also don't think we should be forcing developers to use one of a curated list of 'safe' community-maintained sites which may or may not be just as likely to be dropped as the developer's personal site.

I've hosted docs for the very few crates I publish on my website, and if they had been using rust-ci (which was popular when I was first developing them), the links would be in a much worse state than they are now.

---

The whitelist is more effort to mark a site good in the first place but that's arguably a good thing, it's a nudge to use an already-whitelisted source that the community already tacitly endorses. And if the whitelist add policy is liberal, it's not a significant obstacle to people who feel strongly about self-hosting.

---

It's still an obstacle. Someone publishing their docs on their domain would have to:

• Learn about the whitelist policy
• Request that their domain be added to the whitelist
• Someone would have to approve their request

All of which could take some time. At what point would we enforce this? Would we prevent cargo publish if your documentation domain wasn't in the whitelist? Show a warning on cargo publish and not display your documentation link on crates.io? If we would prevent publish, that seems like a pretty high barrier. If we just wouldn't show the documentation link, crate authors might not even notice, which just hurts crate users since there is legitimate documentation available but they can't get to it.

---

My only point is that if the goal is for users to be confident they can click a link on crates.io without landing on a malware site, a blacklist isn't effective, as it can only address links that have already been demonstrated as bad. There's no prevention, just reaction. It's completely legitimate for other concerns to outweigh this one, but the tradeoff should be acknowledged.

A middle ground might be to decorate links with some kind of whitelisted icon. Or, conversely, you could do an icon or insert an interstitial page for non-whitelisted sites.

[carols10cents]

As I see it, we've got multiple problems to solve here, with different stakeholders:

• Crate users should feel safe and confident when clicking on a link from crates.io that they won't end up on a page with malware/ads/spam.
• Crate users should be able to access legitimate documentation.
• Crate authors should be able to host their docs on a custom domain without having unreasonable barriers (discovering the barriers, logistics of navigating the barriers, waiting for approval) placed in their way.
• Crates.io maintainers and/or mod team should not end up spending a large portion of their volunteer time checking/approving/denying links.

I'm not sure if it's possible to satisfy all of these constraints! I'm currently liking a little warning icon for links of unknown quality, that could link to an explanation about whatever policy we end up going with.

[Beefki]

Perhaps the best solution would be a soft whitelist coupled with a hard blacklist?

A flag attached to the docs when they're hosted on a whitelisted site would give the user the ability to gauge for themselves what risks they're willing to take. Adding a hard blacklist for known bad sites would just ensure that when widely used sites like rust-ci go down they could be blocked.

It would be more work, but I think it's the best way to offer developers freedom while protecting the users from malware and ads

[carols10cents]

I'm still not sure the best way to implement this, but here's a way that I think we could try and iterate on. Let's just start with a list of blocked domains first:

• In the backend, have a hardcoded list of blocked domains in the code rather than in the database, start with rust-ci.org
• In the JSON for each crate, where we currently send: documentation: "[url]", if the url is in the list of blocked domains, do not send the documentation URL, instead send documentation: null (as if the crate does not have a documentation link specified)
• Add documentation to, perhaps, http://doc.crates.io/manifest.html that we block rust-ci.org (Is this the best place for this? I mostly want to have somewhere to link to explaining why a crate's documentation link isn't showing up)

[vignesh-sankaran]

I'd be happy to take this up @carols10cents

[remram44]

Hi,
While implementing blacklisting is probably a good idea and should be done soon, I think it is easy to remove the rust-ci.org links right now. This issue has been opened for 4 months and links to this dubious, potentially harmful website are still up. Not to mention they are making a monetary profit from crates.io pointing to it.

[carols10cents]

Hi, it is not easy. Sorry!

[carols10cents]

And #816 has been deployed! Thank you @vignesh-sankaran, and thank you everyone for your patience!