Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal for default crate recommendation ranking #1824

Merged
merged 15 commits into from Apr 29, 2017

Conversation

@carols10cents
Copy link
Member

@carols10cents carols10cents commented Dec 19, 2016

Rendered!

@shepmaster worked on this too :)

1. nom
2. combine
3. and 4. peg and lalrpop, in some order
5. peresil

This comment has been minimized.

@christophebiocca

christophebiocca Dec 19, 2016

Markdown is renumbering this one.

This comment has been minimized.

@carols10cents

carols10cents Dec 19, 2016
Author Member

UGH MARKDOWN NO.

@ticki
Copy link
Contributor

@ticki ticki commented Dec 19, 2016

This seems like an obvious way to encourage careful documentation. It would be nice if it was a strict requirements for crates to achieve some rank to have complete documentation on every public item.

@tomaka
Copy link

@tomaka tomaka commented Dec 19, 2016

I disagree with the "Maintenance" section.
I prefer to release a new version of my libraries every two months or so, with a large number of fixes and new features each time, instead of releasing a patch for every single fix or new feature.
Release fixes and features in batches doesn't make my libraries less maintained, but it would greatly diminish their ranking.

@badboy
Copy link
Member

@badboy badboy commented Dec 19, 2016

I'm with @tomaka on the Maintenance part. Especially for small, self-contained crates this would be a huge disadvantage. Apart from bug fixes some simply might not need releases that often, while still providing value.

@withoutboats
Copy link
Contributor

@withoutboats withoutboats commented Dec 19, 2016

I love that this is even an RFC - its really cool that we're able to have a very public, community-driven conversation about something like how crates.io ranks crates.

I'm not sure displaying a numeric score is a great idea; I do worry that it encourages gamification (of course, people will optimize for the measurement by virtue of it being a measurement, no matter how its displayed). I guess I'm concerned people will compare close percentages as if that's authoritative when if two crates are ranked very close together its probably a wash.

Maybe some sort of more vague statement where the 2 or 3 highest ranking categories receive superlative statements.

@carols10cents
Copy link
Member Author

@carols10cents carols10cents commented Dec 19, 2016

@tomaka and @badboy, thanks!! Are there any changes to the formula that you think would be better, or is there another way to indicate whether a crate is maintained or not? Or should we not try to mix this into the ranking at all?

@tomaka
Copy link

@tomaka tomaka commented Dec 19, 2016

When it comes to documentation, if you take these two examples:

The first one is clearly much better documented, yet they would both get the same score.

But I agree that it's not easy to come up with a way to measure that objectively.

@carols10cents
Copy link
Member Author

@carols10cents carols10cents commented Dec 19, 2016

@withoutboats thanks! ❤️ I'm glad this is an RFC too, I'm looking forward to discussing this with everyone :)

WDYT about the emoji ☀️🌤⛅️🌥☁️🌧? What if we just showed the emoji?

I agree that not inferring significance when there isn't much is a problem; I do want it to be transparent why a crate is ranked higher than another, though. I also want to be able to show crate authors how they can make their crates better, if they choose to. Definite tension here.

@tomaka
Copy link

@tomaka tomaka commented Dec 19, 2016

@tomaka and @badboy, thanks!! Are there any changes to the formula that you think would be better, or is there another way to indicate whether a crate is maintained or not? Or should we not try to mix this into the ranking at all?

For me the only way to know whether a crate is well maintained is the average time between when a bug report is opened and when it gets fixed (or marked as wontfix for example).
But of course that can't be measured automatically.

For me it makes sense that a crate gets frequent releases before it reaches 1.0. A crate blocked at version 0.1.0 for six months clearly isn't maintained, because if there was nothing to change in the crate then it should be at version 1.0 already. But once you reach 1.0 maybe you don't release anything because there is nothing to release.

@badboy
Copy link
Member

@badboy badboy commented Dec 19, 2016

For me it makes sense that a crate gets frequent releases before it reaches 1.0. A crate blocked at version 0.1.0 for six months clearly isn't maintained, because if there was nothing to change in the crate then it should be at version 1.0 already.

That would be handled by rating >=1.0 versions higher than <1.0, right?

this category easier.

[nom]: https://crates.io/crates/nom
[peresil]: https://github.com/docopt/docopt.rs

This comment has been minimized.

@mgattozzi

mgattozzi Dec 19, 2016
Member

This links the wrong library and can be confusing. I looked up the library mentioned and it makes more sense since this is a parsing library.
https://github.com/shepmaster/peresil

This comment has been minimized.

@carols10cents

carols10cents Dec 19, 2016
Author Member

ooooops lol. I was using different examples at first and missed updating this link. Good catch!

@ticki
Copy link
Contributor

@ticki ticki commented Dec 19, 2016

@withoutboats

I'm not sure displaying a numeric score is a great idea; I do worry that it encourages gamification (of course, people will optimize for the measurement by virtue of it being a measurement, no matter how its displayed). I guess I'm concerned people will compare close percentages as if that's authoritative when if two crates are ranked very close together its probably a wash.

Is gamification necessarily bad? I mean, if it isn't overly competitive and inaccesible, I don't think it is a bad thing.

It is important though that it doesn't end up being something causing more issues (stress, discomfort, etc.) than benefits (fun, quality, etc.).

@shepmaster
Copy link
Member

@shepmaster shepmaster commented Dec 19, 2016

@tomaka @badboy one thing to remember is that the proposed ranking is based on what current, actual humans who use crates (and contributed to the survey) believe makes a crate better. It may or may not align with what authors think makes their crate better.

To create a strawman example, if I believe that placing an image of a cat 🐈 in my docs makes a crate better but the majority of users of crates believe that a dog 🐕 signifies a good crate, then it doesn't really matter what I think; the majority of people won't use it.

@shepmaster
Copy link
Member

@shepmaster shepmaster commented Dec 19, 2016

That would be handled by rating >=1.0 versions higher than <1.0, right?

@badboy I believe that @tomaka is suggesting that the "# releases in the last X time periods" score could be adjusted or rebalanced if the crate has reached 1.0, as they believe the number of releases should be lower than a pre-1.0 crate.

@carols10cents
Copy link
Member Author

@carols10cents carols10cents commented Dec 19, 2016

To create a strawman example, If I believe that placing an image of a cat 🐈 in my docs makes a crate better but the majority of users of crates believe that a dog 🐕 signifies a good crate, then it doesn't really matter what I think; the majority of people won't use it.

But whatever ranking we choose will definitely have an effect on both what users think is good and what authors think is good, so we should choose carefully! Some people might not bother with the analyses they do now; if we have a ranking, they might just trust the ranking.

One example of something that was mentioned in the survey a lot was looking at the number of dependencies a crate has, and preferring crates with fewer dependencies. While I have had bad experiences managing dependencies and dependencies of dependencies, Cargo makes it so much easier to manage dependencies than in other systems programming languages. For this reason, I don't think this is a measure we should use to rank crates, so I didn't include it in the formula.

@shepmaster
Copy link
Member

@shepmaster shepmaster commented Dec 19, 2016

Is gamification necessarily bad? I mean, if it isn't overly competitive and inaccesible, I don't think it is a bad thing.

As a frequent Stack Overflow contributor, I'm also fine with gamification in general. My main concerns are around meta-gaming: If people want "100% doc coverage", they can just add /// lol to everything. This doesn't help anyone, and is likely to actively hurt any potential users of the crate, while the crate sits higher up in the rankings.

Is this currently a problem in our community? I don't believe so. Will it be in 6, 12, 24, 36 months? That's harder to answer.

Perhaps the right risk to take is to try out something now that can be changed later, based on new information gained along the way.

- Maintenance
- Quality

Feeding those signals are related measures of:

This comment has been minimized.

@nagisa

nagisa Dec 19, 2016
Contributor

“Feeding” in this sentence seems to make no sense to me.

This comment has been minimized.

where there isn't really one. We've considered using letter grades, but those
often have emotional associations (F means you're a failure), when it should be
just an indicator of reality and not a value judgment. So we're also proposing
an option of an emoji scale and are open to other proposals:

This comment has been minimized.

@sgrif

sgrif Dec 19, 2016
Contributor

Seems like a 5-star system would be more easily recognizable and fit the same level of coarseness.

<tr>
<th>Crate</th>
<th>Downloads all time (~year)</th>
<th>Downloads in last 90 days (~6 mo)</th>
<th>Crate</th>
<th>Releases in last year</th>
<th>Releases in last 6 mo</th>
<th>Releases in last 1 mo</th>

This comment has been minimized.

@sgrif

sgrif Dec 19, 2016
Contributor

Do you think that 3 months serves the same purpose as 1 month here? Rating based on releases/commits in a 30 day period punishes projects where the maintainers take vacations.

This comment has been minimized.

@sgrif

sgrif Dec 19, 2016
Contributor

It would also mean that projects that release on the same cycle as rust itself (6 weeks) would see this score fluctuate wildly.

@ticki
Copy link
Contributor

@ticki ticki commented Dec 19, 2016

@shepmaster I might have overseen something then. I thought the ranking here was manual.

@carols10cents
Copy link
Member Author

@carols10cents carols10cents commented Dec 20, 2016

@shepmaster I might have overseen something then. I thought the ranking here was manual.

So the ranking that survey responders made was entirely manual based on their own metrics. We ranked these 5 crates pseudo-manually as well, to show how our proposed formula would automatically rank them. This RFC's goal is to not create extra work for anyone on an ongoing basis; the "Alternatives" section does mention that we could add curation or social voting/rating/reviewing features instead.

Does that clear anything up? I'm not sure exactly which statement of @shepmaster's you're replying to...

Copy link
Contributor

@nagisa nagisa left a comment

Would be nice to at least have HTML interjections gotten rid of

<td>F</td>
<td>🌧</td>
</tr>
</table>

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016
Contributor

This is very obnoxious for readers of the plain-text form. Please use one of the supported markdown table formats instead:

| Percentage  | Letter grade | Emoji |
| ----------- | ------------ | ----- |
| ≥ 90%       | A            |  ☀️   |
| 80-89%      | B            | 🌤    |
| 70-79%      | C            | ⛅️    |
| 60-69%      | D            | 🌥    |
| 50-59%      | E            | ☁️     |
| ≤ 49%       | F            | 🌧    |

which would end up looking like this:

Percentage Letter grade Emoji
≥ 90% A ☀️
80-89% B 🌤
70-79% C ⛅️
60-69% D 🌥
50-59% E ☁️
≤ 49% F 🌧

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016
Contributor

I’m considerably against using any letter association, mostly because they have no meaning to me but I’d also would HATE the academic-gradey feeling it has.

Emoji is a bad idea primarily because I, with a good sight, am having great trouble discerning between 3 of the pictures and rain doesn’t necessarily associate with bad to me nor do I find sunny to associate with “perfect”.

Some alternatives that come to mind:

  • colour coding; is terrible because of colour blindness concerns;
  • instead of binning crates into buckets, just explicitly note qualities a crate has: “popular parsing crate”, “credible serialization crate”, “easy to use high performance computing crate” etc. While not every crate would receive a nice label, you’d achieve similar results and avoid issues with developers of crates which do not make the cut.
  • Even then if you insist on bucketing, just write out the percentage explicitly. Beats either of the letter or emoji-coding.

This comment has been minimized.

@Keats

Keats Dec 20, 2016

Agree on all that, never used letter rating anywhere in my life and I have no idea what the emoji are really conveying. Also the >= 90% looks completely different from than the others on my machine.

What's wrong with a 5 stars rating if you really want to show some kind of rating?

This comment has been minimized.

@carols10cents

carols10cents Dec 20, 2016
Author Member

This is very obnoxious for readers of the plain-text form. Please use one of the supported markdown table formats instead:

Sorry about that! It was easier for me to write, without having to worry about the column widths, and I thought it wouldn't matter since HTML is valid Markdown. I'll convert them tomorrow.

This comment has been minimized.

@sgrif

sgrif Dec 20, 2016
Contributor

The column widths don't actually matter for markdown (though for plain text readers of course it's quite helpful)

This comment has been minimized.

@hgallagher1993

hgallagher1993 Dec 20, 2016

@nagisa

instead of binning crates into buckets, just explicitly note qualities a crate has: “popular parsing crate”, “credible serialization crate”, “easy to use high performance computing crate” etc. While not every crate would receive a nice label, you’d achieve similar results and avoid issues with developers of crates which do not make the cut.

Maybe instead of a third party giving a crate a label which would most likely be argued ad infinitum, crate owners/creators themselves could be given the option to tag their crate as something like; "Production ready" or "work in progress, intended for production" or "not intended for production". So if you searched for serialization crate you'd only get hits like Serde and not something that was never intended to be used by anyone else. Better to completely forget about the abandoned, unusable ones than have them clog up the search results where the majority have a tag of "not unusable" or "abandoned" and only ones tagged for production get reviewed for quality.

This comment has been minimized.

@ghost

ghost Dec 20, 2016

Having the crate owner declare the current project state as you describe, would allow for a better solution to the varying activity problem. For instance a project declared as "work in progress" should be getting more activity than a crate that has been in "maintenance" for a year.

This comment has been minimized.

@ghost

ghost Dec 20, 2016

As for whether grade, letter or cloud emoji,
I think we all could strive to get 5 Krustys! 🦀🦀🦀🦀🦀

This comment has been minimized.

@joshtriplett

joshtriplett Dec 21, 2016
Member

For accessibility and clarity, please don't use the emoji as the primary indicator. I think I'd rather see separate badges for each point, not a fuzzy representation of the aggregate score.

top-level items are extremely well documented, 170/195 or 87%. Our
definition of "top-level" counts the overall crate as an item. We think our
doc coverage POC can be modified to report this number.
- Would need to unpack and run this on each package version in a background

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016
Contributor

This is tricky due to #[cfg()]. docs.rs currently is building every single crate fully, although unnecessarily, but cargo model isn’t exactly compatible with desire to avoid running arbitrary code in order to figure out documentation for a crate.

job started by a publish; then save the percentage in crates.io's database.

- In the crate root documentation, presence of a section headed with the word
"Example" and containing a codeblock

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016
Contributor

What about “Examples”, “Use cases”, “Common usage” and similar synonymous phrases? What if example is, say, for some GUI tool provided by crate (e.g. video tutorial demonstrating use of an UI builder of some sort)?

opportunity to encourage at least one to be present reliably.
- Increases the doc percentage score by 5%

- Presence of files in `/examples`

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016
Contributor

Similar comment. What if people put their “examples” into directories with semantically equivalent meaning? What if the example files are in root of the repository? In src/ and linked to from documentation?

This comment has been minimized.

@est31

est31 Dec 20, 2016
Contributor

examples/ is already a standardized directory, as cargo run --example <name> runs examples/<name>.rs.

[cargo doc-coverage]: https://crates.io/crates/cargo-doc-coverage
[examples]: https://github.com/rust-lang/cargo/issues/2760

<table>

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016
Contributor

Another HTML table T_T

- Number of releases in the last year - 10%
- Number of releases in the last 6 mo - 30%
- Number of releases in the last month - 60%
- Yanked versions are not counted.

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016
Contributor

What about the essentially “completed” utility crates such as, say, wrapper crates for machine instructions? Lack of releases does not imply lack of maintenance.

This comment has been minimized.

@nrc

nrc Dec 20, 2016
Member

Is second this concern

This comment has been minimized.

@steveklabnik

steveklabnik Dec 20, 2016
Member

Yes, this one is tough. There are a number of small, focused crates that will rarely ever get an update, but are good crates and should be used.

This comment has been minimized.

@withoutboats

withoutboats Dec 20, 2016
Contributor

Such crates would be compared against alternatives for the same purpose, right? It doesn't seem like a big issue, because all of them would have similar scores.

This comment has been minimized.

@sgrif

sgrif Dec 20, 2016
Contributor

Is there any competition in that space? Are we worried about how libc or winapi are going to be ranked?

This comment has been minimized.

@joshtriplett

joshtriplett Dec 21, 2016
Member

I don't think we should use turnover/churn as a metric of goodness or badness. Some crates have frequent releases due to instability and churn; other crates have frequent releases because of regular maintenance. Some crates have rare releases because nobody maintains them; other crates have rare releases because they do their job and don't need further changes.

This comment has been minimized.


- Stable version number
- >= 1.0.0 ranks higher than < 1.0.0
- >= 1.0.0 increases the maintenance score by 5%.

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016
Contributor

I fail to see benefit of this scoring trigger. It is trivial to set the version number to 1.0.0 on first release and publish breaking releases everyday by increasing the left-most digit instead.

The best this will achieve is reinforcing the crates.io ecosystem to move from the currently quite common 0.x.y versions to x.y.0.

This comment has been minimized.

@steveklabnik

steveklabnik Dec 20, 2016
Member

I'm a big proponent of this. And if it doesn't matter, and 0.x.y is the same as x.y.0, then I don't know why there's an objection.

This comment has been minimized.

@withoutboats

withoutboats Dec 20, 2016
Contributor

It would be ideal to downrank crates with overfrequent major version increments, but that could encourage breaking changes in 1.x releases, which is the opposite of what we want.

hover, with a link to a more thorough explanation. We like the information
density in the way [npms][] displays scores:

<img src="http://i.imgur.com/yadRNyy.png" src="Example npms score circles" width="442" />

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016
Contributor

Markdown syntax please. At least you’ll avoid nonsensical HTML where alt text goes into the src attribute :)

![Example npms score circles](http://i.imgur.com/yadRNyy.png)
@brson
Copy link
Contributor

@brson brson commented Dec 20, 2016

Very well written RFC. Thank you.

@est31
Copy link
Contributor

@est31 est31 commented Dec 20, 2016

Some ideas of improvement:

  • "Percentage of docs comments" is no good measure, at least not how its implemented right now. A crate to make requests with a function with_url and no docs is better than one with a function u and docs that explain that u stands for url. Just making an on/off check for whether there is documentation for some public item or not is too simple IMO, as in crates with well named identifiers, single line documentation is almost useless. Far more useful is the documentation that follows, so that you know about precise behaviour of the functions/structs. So I'd suggest to only count multi line documentation, and disregard any single line one, or value it much less than multi line documentation (maybe 10% or sth). I also highly appreciate example code either in examples/ or in the root as doc comment, as this is very helpful.

  • Remove the >=1.0.0 check, it only maps of how well the crate authors think their crate is, not how well it actually is. All this does is that all crates will release their initial version as 1.0.0. Yes, semver mandates that after 1.0.0 you don't do breaking changes in minor versions, but crate authors can just simply do major versions instead. Scoring by the number of owners is a good measure though I think, as the more people are actually maintaining a crate, the more likely its to live on once a maintainer gets bored or busy with other things.

  • The number of downloads score is a good indicator for popularity IMO, as it also includes bots, which means the crate gets used by projects with high quality assurance systems. Obviously its a bit spammy, and maybe one should value automatic downloads different from human downloads (whatever qualifies as that), but I think overall a good measure.

I think overall, the score shouldn't be something overemphasized, as the methodology is quite flawed in comparison to manual involvement, like the manual moderation of stack stack overflow.

@withoutboats
Copy link
Contributor

@withoutboats withoutboats commented Dec 20, 2016

The big concern with gamification is 'min/maxing' behavior. We don't want people writing crappy doc comments or releasing pointless updates just to keep their 'score' up. Obscuring exactly what your "score" is can help avoid people seeing that if they released another version, they could go up a ranking.

About 10 years ago I played a great online game called Puzzle Pirates which did its best to discourage min/maxing by obscuring your raw score with fuzzy statements; all the games were ranked with an ELO system but none of them showed your score. I think this is pretty effective.

🌟 This crate has very complete documentation.
This crate has few downloads.
This crate releases frequently.

These needn't just be a raw equivalence to the ranking algorithm but could even have a certain degree of 'badginess' like there could be a special note about being above 1.0.

This gamification creates a tension with everything we measure, of course. For example, one thing I value in a crate is infrequent breaking changes, but if we measure that we might encourage users to release breaking changes using non-breaking semver!

(We could give people an interface for "committing to semver," given them a 'badge' and upranking crates with that commitment. This doesn't need to be policed - like taking their badge away if the violate semver - but it creates additional buy-in to the idea that following semver matters.)

@camlorn
Copy link

@camlorn camlorn commented Dec 20, 2016

I was summoned here via IRC. Unicode has major screen reader concerns, to wit that you can't expect it to read. Of the options proposed by @nagisa, just showing the percent is the one I'd go for.

If we want to use symbols, the only way to make them reliably accessible is to use graphics with alt text. While I wish I did live in a world with full unicode awareness from my AT, this is not going to happen anytime soon.

@nrc nrc added the T-dev-tools label Dec 20, 2016
@robojeb
Copy link

@robojeb robojeb commented Dec 20, 2016

@tomaka I don't know if you can really assume a crate pre-1.0 that doesn't change is unmaintained, at least without looking at its dependencies. For example I wrote a PRNG crate and I don't think that I should go 1.0 because the Rand crate isn't 1.0 yet. It doesn't make sense for me to go "stable" if my dependencies are unstable. So my crate doesn't change much any more.

@mgattozzi
Copy link
Member

@mgattozzi mgattozzi commented Mar 25, 2017

Like @carols10cents I'm strongly opposed to using ML. It's difficult to get right, it's difficult to maintain, and for the amount of resources the Rust community has, it's not worth the time nor do we have a decent data set for it. It's not some magic bullet that a lot of people think it is and it's not as easy as dumping data into a neural net to learn.

@ticki
Copy link
Contributor

@ticki ticki commented Mar 25, 2017

@mgattozzi I'm not claiming it's a magic bullet in any way. I agree it's hard, but I believe it's easier than the manual approach.

@carols10cents
Copy link
Member Author

@carols10cents carols10cents commented Mar 26, 2017

Please open a new RFC if you'd like to propose adding machine learning to recommend crates; I won't be changing this RFC to add it.

@retep998:

I think we should at least provide a way for users to provide input on the quality of specific crates. Even if we don't display that information, we can aggregate it and analyze it and decide what to do in the future.

I'm not going to spend time implementing and maintaining a data collection mechanism if we don't have a specific use for that data :-/

@aturon
Copy link
Member

@aturon aturon commented Mar 27, 2017

To echo what others have said, machine learning would constitute an entirely separate proposal, with a lot of its own complexities and downsides. As I emphasized in the motion to FCP, we can and should take an iterative approach here, and the RFC as it currently stands is something relatively straightforward to implement, that we can land within the constraints of this year's roadmap. I think it's also a clear improvement over the status quo, and has been through a number of rounds of iteration in this discussion.

The RFC includes extensive research, both in terms of prior art, and in terms of what the Rust community specifically values. It also lays out several assumptions/design constraints at the outset. After these many rounds of discussion, I personally think it's reached a quite reasonable point in the design space balancing between the goals suggested by the research, technical feasibility for a first-cut implementation, and various concerns about gaming and proxy measures.

In the interest of trying to drive things toward some kind of conclusion here, I think it's best to keep discussion to any final concerns about (1) the takeaways from the research or (2) the proposed design, in terms of its balance between the research-derived goals and what makes sense to try in a first-round effort here.

@rfcbot
Copy link

@rfcbot rfcbot commented Apr 17, 2017

🔔 This is now entering its final comment period, as per the review above. 🔔

@mark-i-m
Copy link
Member

@mark-i-m mark-i-m commented Apr 18, 2017

Thanks @carols10cents for the great (and much needed) RFC!

This is a really long thread, so I just wanted to post a few links to the summary posts scattered therein for those just reading the thread now (like me)...

@ruuda's comments here and here summarize the majority of the thread pretty well, I think.

@aturon's FCP summary of the RFC is here.

I was also intrigued by scattered proposals to take influence from ranking systems already in common use, such as Amazon, StackOverflow, and GitHub stars/trending. I won't bikeshed here, but I think these are worth considering for inspiration as this goes through iterations...

@ghost
Copy link

@ghost ghost commented Apr 21, 2017

If the goal is that "Rust should provide easy access to high quality crates" then badges and ranking are really just bells and whistles. The guts of the problem is to find a way to present information about all relevent crates so that a potential user can evaluate them easily. If the front page of crates.io had a keyword search that produced a list, ranked by the last 90 days downloads, with a a five line explanation drawn from the top of README.md, this would be ideal for me. The keyword search might give me 10 or 20 options, and at 5 lines each it wouldn't be much trouble at all to scan through. The first 5 lines would probably give a good general indication of quality of the documentation.

@Razican
Copy link

@Razican Razican commented Apr 26, 2017

The rendered RFC only talks about Coveralls. I personally have had a lot of issues with Coveralls configuration and recently have moved to Codecov in all my projects. They even provide Rust templates, and of course, badges.

@carols10cents carols10cents added this to FCP in Tracker Apr 26, 2017
@carols10cents
Copy link
Member Author

@carols10cents carols10cents commented Apr 26, 2017

@Razican

The rendered RFC only talks about Coveralls. I personally have had a lot of issues with Coveralls configuration and recently have moved to Codecov in all my projects. They even provide Rust templates, and of course, badges.

Interesting! I didn't know about Codecov. The implementation will definitely be extensible-- for example, someone sent in a PR to add a badge for GitLab CI status to go along with the Travis and Appveyor badges, and it was a pretty straightforward change.

@rfcbot
Copy link

@rfcbot rfcbot commented Apr 27, 2017

The final comment period is now complete.

@aturon
Copy link
Member

@aturon aturon commented Apr 29, 2017

Huzzah! This RFC has been merged, and is tracked here. As has been emphasized throughout, this design is a starting point, and we'll expect to do follow-up evaluation and iteration.

Thanks everyone who participated in this in-depth discussion, and especially @carols10cents for writing such a thoughtful RFC and sticking it out through 258 comments :-)

@shepmaster shepmaster deleted the integer32llc:crates-io-ranking branch Apr 29, 2017
@gbutler69
Copy link

@gbutler69 gbutler69 commented Jun 4, 2017

Couple of thoughts. A crate should receive a higher score based on the number of other crates on crates.io that depend upon it. A create should receive a lower score based on the number of other crates it depends upon. Depending on lower-ranked vs higher-ranked crates should have a larger negative impact on a crate's rating. Having higher-ranked crates vs. lower-ranked crates depending upon a crate should have a larger positive impact on the ranking of the crate.

@mark-i-m
Copy link
Member

@mark-i-m mark-i-m commented Jun 4, 2017

@le-jzr
Copy link

@le-jzr le-jzr commented Jun 4, 2017

Those two categories should be separate anyway. Unless you are searching the database just for fun (or reviewing, or whatever), you either want specifically a library, or specifically an application.

I think @gbutler69 has the right idea.

@le-jzr
Copy link

@le-jzr le-jzr commented Jun 4, 2017

But the score interaction would have to be broken up by individual influences, or otherwise stratified. Naive implementation could lead to nasty feedback cycles.

@MoSal
Copy link

@MoSal MoSal commented Jun 4, 2017

@gbutler69
Note that this RFC has already been merged. And it's not clear where re-evaluation and iteration discussions are going to take place (probably not here in this PR).

A crate should receive a higher score based on the number of other crates on crates.io that depend upon it.

Agreed.

A create should receive a lower score based on the number of other crates it depends upon.

This will encourage bad practices:

  • Copying code from other crates instead of depending on them.
  • Re-implementing functionality available in other crates poorly.

In other words, this will encourage people to work against the crate ecosystem, not work with it.

Depending on lower-ranked vs higher-ranked crates should have a larger negative impact on a crate's rating.

Again. This will encourage bad practices. For example, this will discourage people from releasing modular crates. Fearing the lower-level crates will have a low-score which will reflect poorly on the higher-level ones.

Having higher-ranked crates vs. lower-ranked crates depending upon a crate should have a larger positive impact on the ranking of the crate.

Could be useful. But this might have complex interactions with other factors. And the implementation details are not clear. For example:

  • Should the average score of all dependants be considered?
    • Or just the top 5?
    • Or just the one with the highest score?
  • Should there be a minimum number of dependants for this factor to be considered?
    • What is that number?
  • How much weight should be given to this factor?
  • Should a high-ranking dependant stop depending on foo, how much of a negative impact should foo suffer based on that event alone?

I think the straight forward factors I took into consideration in cargo-esr are enough to paint a rough picture of how much a crate is relevantly depended on.

@carols10cents
Copy link
Member Author

@carols10cents carols10cents commented Jun 5, 2017

Note that this RFC has already been merged. And it's not clear where re-evaluation and iteration discussions are going to take place (probably not here in this PR).

Yes, since this RFC has been merged, large changes proposed to this should get a new RFC. Small changes can be proposed in an issue on crates.io's repo, and the re-evaluation and iteration discussions will probably take place on crates.io's repo as well.

I'm unsubscribing from this thread, thank you all for your discussion!

@joshtriplett
Copy link
Member

@joshtriplett joshtriplett commented Nov 14, 2018

I wanted to pass along a comment I saw in a private forum discussing Rust:

Rust's package site, crates.io, has a nice search that has the most sensible result sorting I've seen, ranking by a combination of relevance and recent, as opposed to overall, download numbers, which is a good way of finding "the library that does what I want and also is maintained and used."

@Centril Centril added the A-ranking label Nov 23, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.