Proposal for default crate recommendation ranking #1824

Merged
merged 15 commits into from Apr 29, 2017

Conversation

@carols10cents
Member

carols10cents commented Dec 19, 2016

Rendered!

@shepmaster worked on this too :)

text/0000-crates.io-default-ranking.md
+1. nom
+2. combine
+3. and 4. peg and lalrpop, in some order
+5. peresil

This comment has been minimized.

@christophebiocca

christophebiocca Dec 19, 2016

Markdown is renumbering this one.

@christophebiocca

christophebiocca Dec 19, 2016

Markdown is renumbering this one.

This comment has been minimized.

@carols10cents

carols10cents Dec 19, 2016

Member

UGH MARKDOWN NO.

@carols10cents

carols10cents Dec 19, 2016

Member

UGH MARKDOWN NO.

@ticki

This comment has been minimized.

Show comment
Hide comment
@ticki

ticki Dec 19, 2016

Contributor

This seems like an obvious way to encourage careful documentation. It would be nice if it was a strict requirements for crates to achieve some rank to have complete documentation on every public item.

Contributor

ticki commented Dec 19, 2016

This seems like an obvious way to encourage careful documentation. It would be nice if it was a strict requirements for crates to achieve some rank to have complete documentation on every public item.

@tomaka

This comment has been minimized.

Show comment
Hide comment
@tomaka

tomaka Dec 19, 2016

I disagree with the "Maintenance" section.
I prefer to release a new version of my libraries every two months or so, with a large number of fixes and new features each time, instead of releasing a patch for every single fix or new feature.
Release fixes and features in batches doesn't make my libraries less maintained, but it would greatly diminish their ranking.

tomaka commented Dec 19, 2016

I disagree with the "Maintenance" section.
I prefer to release a new version of my libraries every two months or so, with a large number of fixes and new features each time, instead of releasing a patch for every single fix or new feature.
Release fixes and features in batches doesn't make my libraries less maintained, but it would greatly diminish their ranking.

@badboy

This comment has been minimized.

Show comment
Hide comment
@badboy

badboy Dec 19, 2016

Member

I'm with @tomaka on the Maintenance part. Especially for small, self-contained crates this would be a huge disadvantage. Apart from bug fixes some simply might not need releases that often, while still providing value.

Member

badboy commented Dec 19, 2016

I'm with @tomaka on the Maintenance part. Especially for small, self-contained crates this would be a huge disadvantage. Apart from bug fixes some simply might not need releases that often, while still providing value.

@withoutboats

This comment has been minimized.

Show comment
Hide comment
@withoutboats

withoutboats Dec 19, 2016

Contributor

I love that this is even an RFC - its really cool that we're able to have a very public, community-driven conversation about something like how crates.io ranks crates.

I'm not sure displaying a numeric score is a great idea; I do worry that it encourages gamification (of course, people will optimize for the measurement by virtue of it being a measurement, no matter how its displayed). I guess I'm concerned people will compare close percentages as if that's authoritative when if two crates are ranked very close together its probably a wash.

Maybe some sort of more vague statement where the 2 or 3 highest ranking categories receive superlative statements.

Contributor

withoutboats commented Dec 19, 2016

I love that this is even an RFC - its really cool that we're able to have a very public, community-driven conversation about something like how crates.io ranks crates.

I'm not sure displaying a numeric score is a great idea; I do worry that it encourages gamification (of course, people will optimize for the measurement by virtue of it being a measurement, no matter how its displayed). I guess I'm concerned people will compare close percentages as if that's authoritative when if two crates are ranked very close together its probably a wash.

Maybe some sort of more vague statement where the 2 or 3 highest ranking categories receive superlative statements.

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Dec 19, 2016

Member

@tomaka and @badboy, thanks!! Are there any changes to the formula that you think would be better, or is there another way to indicate whether a crate is maintained or not? Or should we not try to mix this into the ranking at all?

Member

carols10cents commented Dec 19, 2016

@tomaka and @badboy, thanks!! Are there any changes to the formula that you think would be better, or is there another way to indicate whether a crate is maintained or not? Or should we not try to mix this into the ranking at all?

@tomaka

This comment has been minimized.

Show comment
Hide comment
@tomaka

tomaka Dec 19, 2016

When it comes to documentation, if you take these two examples:

The first one is clearly much better documented, yet they would both get the same score.

But I agree that it's not easy to come up with a way to measure that objectively.

tomaka commented Dec 19, 2016

When it comes to documentation, if you take these two examples:

The first one is clearly much better documented, yet they would both get the same score.

But I agree that it's not easy to come up with a way to measure that objectively.

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Dec 19, 2016

Member

@withoutboats thanks! ❤️ I'm glad this is an RFC too, I'm looking forward to discussing this with everyone :)

WDYT about the emoji ☀️🌤⛅️🌥☁️🌧? What if we just showed the emoji?

I agree that not inferring significance when there isn't much is a problem; I do want it to be transparent why a crate is ranked higher than another, though. I also want to be able to show crate authors how they can make their crates better, if they choose to. Definite tension here.

Member

carols10cents commented Dec 19, 2016

@withoutboats thanks! ❤️ I'm glad this is an RFC too, I'm looking forward to discussing this with everyone :)

WDYT about the emoji ☀️🌤⛅️🌥☁️🌧? What if we just showed the emoji?

I agree that not inferring significance when there isn't much is a problem; I do want it to be transparent why a crate is ranked higher than another, though. I also want to be able to show crate authors how they can make their crates better, if they choose to. Definite tension here.

@tomaka

This comment has been minimized.

Show comment
Hide comment
@tomaka

tomaka Dec 19, 2016

@tomaka and @badboy, thanks!! Are there any changes to the formula that you think would be better, or is there another way to indicate whether a crate is maintained or not? Or should we not try to mix this into the ranking at all?

For me the only way to know whether a crate is well maintained is the average time between when a bug report is opened and when it gets fixed (or marked as wontfix for example).
But of course that can't be measured automatically.

For me it makes sense that a crate gets frequent releases before it reaches 1.0. A crate blocked at version 0.1.0 for six months clearly isn't maintained, because if there was nothing to change in the crate then it should be at version 1.0 already. But once you reach 1.0 maybe you don't release anything because there is nothing to release.

tomaka commented Dec 19, 2016

@tomaka and @badboy, thanks!! Are there any changes to the formula that you think would be better, or is there another way to indicate whether a crate is maintained or not? Or should we not try to mix this into the ranking at all?

For me the only way to know whether a crate is well maintained is the average time between when a bug report is opened and when it gets fixed (or marked as wontfix for example).
But of course that can't be measured automatically.

For me it makes sense that a crate gets frequent releases before it reaches 1.0. A crate blocked at version 0.1.0 for six months clearly isn't maintained, because if there was nothing to change in the crate then it should be at version 1.0 already. But once you reach 1.0 maybe you don't release anything because there is nothing to release.

@badboy

This comment has been minimized.

Show comment
Hide comment
@badboy

badboy Dec 19, 2016

Member

For me it makes sense that a crate gets frequent releases before it reaches 1.0. A crate blocked at version 0.1.0 for six months clearly isn't maintained, because if there was nothing to change in the crate then it should be at version 1.0 already.

That would be handled by rating >=1.0 versions higher than <1.0, right?

Member

badboy commented Dec 19, 2016

For me it makes sense that a crate gets frequent releases before it reaches 1.0. A crate blocked at version 0.1.0 for six months clearly isn't maintained, because if there was nothing to change in the crate then it should be at version 1.0 already.

That would be handled by rating >=1.0 versions higher than <1.0, right?

text/0000-crates.io-default-ranking.md
+this category easier.
+
+[nom]: https://crates.io/crates/nom
+[peresil]: https://github.com/docopt/docopt.rs

This comment has been minimized.

@mgattozzi

mgattozzi Dec 19, 2016

Contributor

This links the wrong library and can be confusing. I looked up the library mentioned and it makes more sense since this is a parsing library.
https://github.com/shepmaster/peresil

@mgattozzi

mgattozzi Dec 19, 2016

Contributor

This links the wrong library and can be confusing. I looked up the library mentioned and it makes more sense since this is a parsing library.
https://github.com/shepmaster/peresil

This comment has been minimized.

@carols10cents

carols10cents Dec 19, 2016

Member

ooooops lol. I was using different examples at first and missed updating this link. Good catch!

@carols10cents

carols10cents Dec 19, 2016

Member

ooooops lol. I was using different examples at first and missed updating this link. Good catch!

@ticki

This comment has been minimized.

Show comment
Hide comment
@ticki

ticki Dec 19, 2016

Contributor

@withoutboats

I'm not sure displaying a numeric score is a great idea; I do worry that it encourages gamification (of course, people will optimize for the measurement by virtue of it being a measurement, no matter how its displayed). I guess I'm concerned people will compare close percentages as if that's authoritative when if two crates are ranked very close together its probably a wash.

Is gamification necessarily bad? I mean, if it isn't overly competitive and inaccesible, I don't think it is a bad thing.

It is important though that it doesn't end up being something causing more issues (stress, discomfort, etc.) than benefits (fun, quality, etc.).

Contributor

ticki commented Dec 19, 2016

@withoutboats

I'm not sure displaying a numeric score is a great idea; I do worry that it encourages gamification (of course, people will optimize for the measurement by virtue of it being a measurement, no matter how its displayed). I guess I'm concerned people will compare close percentages as if that's authoritative when if two crates are ranked very close together its probably a wash.

Is gamification necessarily bad? I mean, if it isn't overly competitive and inaccesible, I don't think it is a bad thing.

It is important though that it doesn't end up being something causing more issues (stress, discomfort, etc.) than benefits (fun, quality, etc.).

@shepmaster

This comment has been minimized.

Show comment
Hide comment
@shepmaster

shepmaster Dec 19, 2016

Member

@tomaka @badboy one thing to remember is that the proposed ranking is based on what current, actual humans who use crates (and contributed to the survey) believe makes a crate better. It may or may not align with what authors think makes their crate better.

To create a strawman example, if I believe that placing an image of a cat 🐈 in my docs makes a crate better but the majority of users of crates believe that a dog 🐕 signifies a good crate, then it doesn't really matter what I think; the majority of people won't use it.

Member

shepmaster commented Dec 19, 2016

@tomaka @badboy one thing to remember is that the proposed ranking is based on what current, actual humans who use crates (and contributed to the survey) believe makes a crate better. It may or may not align with what authors think makes their crate better.

To create a strawman example, if I believe that placing an image of a cat 🐈 in my docs makes a crate better but the majority of users of crates believe that a dog 🐕 signifies a good crate, then it doesn't really matter what I think; the majority of people won't use it.

@shepmaster

This comment has been minimized.

Show comment
Hide comment
@shepmaster

shepmaster Dec 19, 2016

Member

That would be handled by rating >=1.0 versions higher than <1.0, right?

@badboy I believe that @tomaka is suggesting that the "# releases in the last X time periods" score could be adjusted or rebalanced if the crate has reached 1.0, as they believe the number of releases should be lower than a pre-1.0 crate.

Member

shepmaster commented Dec 19, 2016

That would be handled by rating >=1.0 versions higher than <1.0, right?

@badboy I believe that @tomaka is suggesting that the "# releases in the last X time periods" score could be adjusted or rebalanced if the crate has reached 1.0, as they believe the number of releases should be lower than a pre-1.0 crate.

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Dec 19, 2016

Member

To create a strawman example, If I believe that placing an image of a cat 🐈 in my docs makes a crate better but the majority of users of crates believe that a dog 🐕 signifies a good crate, then it doesn't really matter what I think; the majority of people won't use it.

But whatever ranking we choose will definitely have an effect on both what users think is good and what authors think is good, so we should choose carefully! Some people might not bother with the analyses they do now; if we have a ranking, they might just trust the ranking.

One example of something that was mentioned in the survey a lot was looking at the number of dependencies a crate has, and preferring crates with fewer dependencies. While I have had bad experiences managing dependencies and dependencies of dependencies, Cargo makes it so much easier to manage dependencies than in other systems programming languages. For this reason, I don't think this is a measure we should use to rank crates, so I didn't include it in the formula.

Member

carols10cents commented Dec 19, 2016

To create a strawman example, If I believe that placing an image of a cat 🐈 in my docs makes a crate better but the majority of users of crates believe that a dog 🐕 signifies a good crate, then it doesn't really matter what I think; the majority of people won't use it.

But whatever ranking we choose will definitely have an effect on both what users think is good and what authors think is good, so we should choose carefully! Some people might not bother with the analyses they do now; if we have a ranking, they might just trust the ranking.

One example of something that was mentioned in the survey a lot was looking at the number of dependencies a crate has, and preferring crates with fewer dependencies. While I have had bad experiences managing dependencies and dependencies of dependencies, Cargo makes it so much easier to manage dependencies than in other systems programming languages. For this reason, I don't think this is a measure we should use to rank crates, so I didn't include it in the formula.

@shepmaster

This comment has been minimized.

Show comment
Hide comment
@shepmaster

shepmaster Dec 19, 2016

Member

Is gamification necessarily bad? I mean, if it isn't overly competitive and inaccesible, I don't think it is a bad thing.

As a frequent Stack Overflow contributor, I'm also fine with gamification in general. My main concerns are around meta-gaming: If people want "100% doc coverage", they can just add /// lol to everything. This doesn't help anyone, and is likely to actively hurt any potential users of the crate, while the crate sits higher up in the rankings.

Is this currently a problem in our community? I don't believe so. Will it be in 6, 12, 24, 36 months? That's harder to answer.

Perhaps the right risk to take is to try out something now that can be changed later, based on new information gained along the way.

Member

shepmaster commented Dec 19, 2016

Is gamification necessarily bad? I mean, if it isn't overly competitive and inaccesible, I don't think it is a bad thing.

As a frequent Stack Overflow contributor, I'm also fine with gamification in general. My main concerns are around meta-gaming: If people want "100% doc coverage", they can just add /// lol to everything. This doesn't help anyone, and is likely to actively hurt any potential users of the crate, while the crate sits higher up in the rankings.

Is this currently a problem in our community? I don't believe so. Will it be in 6, 12, 24, 36 months? That's harder to answer.

Perhaps the right risk to take is to try out something now that can be changed later, based on new information gained along the way.

text/0000-crates.io-default-ranking.md
+- Maintenance
+- Quality
+
+Feeding those signals are related measures of:

This comment has been minimized.

@nagisa

nagisa Dec 19, 2016

Contributor

“Feeding” in this sentence seems to make no sense to me.

@nagisa

nagisa Dec 19, 2016

Contributor

“Feeding” in this sentence seems to make no sense to me.

This comment has been minimized.

text/0000-crates.io-default-ranking.md
+where there isn't really one. We've considered using letter grades, but those
+often have emotional associations (F means you're a failure), when it should be
+just an indicator of reality and not a value judgment. So we're also proposing
+an option of an emoji scale and are open to other proposals:

This comment has been minimized.

@sgrif

sgrif Dec 19, 2016

Contributor

Seems like a 5-star system would be more easily recognizable and fit the same level of coarseness.

@sgrif

sgrif Dec 19, 2016

Contributor

Seems like a 5-star system would be more easily recognizable and fit the same level of coarseness.

text/0000-crates.io-default-ranking.md
+ <tr>
+ <th>Crate</th>
+ <th>Downloads all time (~year)</th>
+ <th>Downloads in last 90 days (~6 mo)</th>
text/0000-crates.io-default-ranking.md
+ <th>Crate</th>
+ <th>Releases in last year</th>
+ <th>Releases in last 6 mo</th>
+ <th>Releases in last 1 mo</th>

This comment has been minimized.

@sgrif

sgrif Dec 19, 2016

Contributor

Do you think that 3 months serves the same purpose as 1 month here? Rating based on releases/commits in a 30 day period punishes projects where the maintainers take vacations.

@sgrif

sgrif Dec 19, 2016

Contributor

Do you think that 3 months serves the same purpose as 1 month here? Rating based on releases/commits in a 30 day period punishes projects where the maintainers take vacations.

This comment has been minimized.

@sgrif

sgrif Dec 19, 2016

Contributor

It would also mean that projects that release on the same cycle as rust itself (6 weeks) would see this score fluctuate wildly.

@sgrif

sgrif Dec 19, 2016

Contributor

It would also mean that projects that release on the same cycle as rust itself (6 weeks) would see this score fluctuate wildly.

@ticki

This comment has been minimized.

Show comment
Hide comment
@ticki

ticki Dec 19, 2016

Contributor

@shepmaster I might have overseen something then. I thought the ranking here was manual.

Contributor

ticki commented Dec 19, 2016

@shepmaster I might have overseen something then. I thought the ranking here was manual.

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Dec 20, 2016

Member

@shepmaster I might have overseen something then. I thought the ranking here was manual.

So the ranking that survey responders made was entirely manual based on their own metrics. We ranked these 5 crates pseudo-manually as well, to show how our proposed formula would automatically rank them. This RFC's goal is to not create extra work for anyone on an ongoing basis; the "Alternatives" section does mention that we could add curation or social voting/rating/reviewing features instead.

Does that clear anything up? I'm not sure exactly which statement of @shepmaster's you're replying to...

Member

carols10cents commented Dec 20, 2016

@shepmaster I might have overseen something then. I thought the ranking here was manual.

So the ranking that survey responders made was entirely manual based on their own metrics. We ranked these 5 crates pseudo-manually as well, to show how our proposed formula would automatically rank them. This RFC's goal is to not create extra work for anyone on an ongoing basis; the "Alternatives" section does mention that we could add curation or social voting/rating/reviewing features instead.

Does that clear anything up? I'm not sure exactly which statement of @shepmaster's you're replying to...

@nagisa

Would be nice to at least have HTML interjections gotten rid of

text/0000-crates.io-default-ranking.md
+ <td>F</td>
+ <td>🌧</td>
+ </tr>
+</table>

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016

Contributor

This is very obnoxious for readers of the plain-text form. Please use one of the supported markdown table formats instead:

| Percentage  | Letter grade | Emoji |
| ----------- | ------------ | ----- |
| ≥ 90%       | A            |  ☀️   |
| 80-89%      | B            | 🌤    |
| 70-79%      | C            | ⛅️    |
| 60-69%      | D            | 🌥    |
| 50-59%      | E            | ☁️     |
| ≤ 49%       | F            | 🌧    |

which would end up looking like this:

Percentage Letter grade Emoji
≥ 90% A ☀️
80-89% B 🌤
70-79% C ⛅️
60-69% D 🌥
50-59% E ☁️
≤ 49% F 🌧
@nagisa

nagisa Dec 20, 2016

Contributor

This is very obnoxious for readers of the plain-text form. Please use one of the supported markdown table formats instead:

| Percentage  | Letter grade | Emoji |
| ----------- | ------------ | ----- |
| ≥ 90%       | A            |  ☀️   |
| 80-89%      | B            | 🌤    |
| 70-79%      | C            | ⛅️    |
| 60-69%      | D            | 🌥    |
| 50-59%      | E            | ☁️     |
| ≤ 49%       | F            | 🌧    |

which would end up looking like this:

Percentage Letter grade Emoji
≥ 90% A ☀️
80-89% B 🌤
70-79% C ⛅️
60-69% D 🌥
50-59% E ☁️
≤ 49% F 🌧

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016

Contributor

I’m considerably against using any letter association, mostly because they have no meaning to me but I’d also would HATE the academic-gradey feeling it has.

Emoji is a bad idea primarily because I, with a good sight, am having great trouble discerning between 3 of the pictures and rain doesn’t necessarily associate with bad to me nor do I find sunny to associate with “perfect”.

Some alternatives that come to mind:

  • colour coding; is terrible because of colour blindness concerns;
  • instead of binning crates into buckets, just explicitly note qualities a crate has: “popular parsing crate”, “credible serialization crate”, “easy to use high performance computing crate” etc. While not every crate would receive a nice label, you’d achieve similar results and avoid issues with developers of crates which do not make the cut.
  • Even then if you insist on bucketing, just write out the percentage explicitly. Beats either of the letter or emoji-coding.
@nagisa

nagisa Dec 20, 2016

Contributor

I’m considerably against using any letter association, mostly because they have no meaning to me but I’d also would HATE the academic-gradey feeling it has.

Emoji is a bad idea primarily because I, with a good sight, am having great trouble discerning between 3 of the pictures and rain doesn’t necessarily associate with bad to me nor do I find sunny to associate with “perfect”.

Some alternatives that come to mind:

  • colour coding; is terrible because of colour blindness concerns;
  • instead of binning crates into buckets, just explicitly note qualities a crate has: “popular parsing crate”, “credible serialization crate”, “easy to use high performance computing crate” etc. While not every crate would receive a nice label, you’d achieve similar results and avoid issues with developers of crates which do not make the cut.
  • Even then if you insist on bucketing, just write out the percentage explicitly. Beats either of the letter or emoji-coding.

This comment has been minimized.

@Keats

Keats Dec 20, 2016

Agree on all that, never used letter rating anywhere in my life and I have no idea what the emoji are really conveying. Also the >= 90% looks completely different from than the others on my machine.

What's wrong with a 5 stars rating if you really want to show some kind of rating?

@Keats

Keats Dec 20, 2016

Agree on all that, never used letter rating anywhere in my life and I have no idea what the emoji are really conveying. Also the >= 90% looks completely different from than the others on my machine.

What's wrong with a 5 stars rating if you really want to show some kind of rating?

This comment has been minimized.

@carols10cents

carols10cents Dec 20, 2016

Member

This is very obnoxious for readers of the plain-text form. Please use one of the supported markdown table formats instead:

Sorry about that! It was easier for me to write, without having to worry about the column widths, and I thought it wouldn't matter since HTML is valid Markdown. I'll convert them tomorrow.

@carols10cents

carols10cents Dec 20, 2016

Member

This is very obnoxious for readers of the plain-text form. Please use one of the supported markdown table formats instead:

Sorry about that! It was easier for me to write, without having to worry about the column widths, and I thought it wouldn't matter since HTML is valid Markdown. I'll convert them tomorrow.

This comment has been minimized.

@sgrif

sgrif Dec 20, 2016

Contributor

The column widths don't actually matter for markdown (though for plain text readers of course it's quite helpful)

@sgrif

sgrif Dec 20, 2016

Contributor

The column widths don't actually matter for markdown (though for plain text readers of course it's quite helpful)

This comment has been minimized.

@hgallagher1993

hgallagher1993 Dec 20, 2016

@nagisa

instead of binning crates into buckets, just explicitly note qualities a crate has: “popular parsing crate”, “credible serialization crate”, “easy to use high performance computing crate” etc. While not every crate would receive a nice label, you’d achieve similar results and avoid issues with developers of crates which do not make the cut.

Maybe instead of a third party giving a crate a label which would most likely be argued ad infinitum, crate owners/creators themselves could be given the option to tag their crate as something like; "Production ready" or "work in progress, intended for production" or "not intended for production". So if you searched for serialization crate you'd only get hits like Serde and not something that was never intended to be used by anyone else. Better to completely forget about the abandoned, unusable ones than have them clog up the search results where the majority have a tag of "not unusable" or "abandoned" and only ones tagged for production get reviewed for quality.

@hgallagher1993

hgallagher1993 Dec 20, 2016

@nagisa

instead of binning crates into buckets, just explicitly note qualities a crate has: “popular parsing crate”, “credible serialization crate”, “easy to use high performance computing crate” etc. While not every crate would receive a nice label, you’d achieve similar results and avoid issues with developers of crates which do not make the cut.

Maybe instead of a third party giving a crate a label which would most likely be argued ad infinitum, crate owners/creators themselves could be given the option to tag their crate as something like; "Production ready" or "work in progress, intended for production" or "not intended for production". So if you searched for serialization crate you'd only get hits like Serde and not something that was never intended to be used by anyone else. Better to completely forget about the abandoned, unusable ones than have them clog up the search results where the majority have a tag of "not unusable" or "abandoned" and only ones tagged for production get reviewed for quality.

This comment has been minimized.

@karoofish

karoofish Dec 20, 2016

Having the crate owner declare the current project state as you describe, would allow for a better solution to the varying activity problem. For instance a project declared as "work in progress" should be getting more activity than a crate that has been in "maintenance" for a year.

@karoofish

karoofish Dec 20, 2016

Having the crate owner declare the current project state as you describe, would allow for a better solution to the varying activity problem. For instance a project declared as "work in progress" should be getting more activity than a crate that has been in "maintenance" for a year.

This comment has been minimized.

@karoofish

karoofish Dec 20, 2016

As for whether grade, letter or cloud emoji,
I think we all could strive to get 5 Krustys! 🦀🦀🦀🦀🦀

@karoofish

karoofish Dec 20, 2016

As for whether grade, letter or cloud emoji,
I think we all could strive to get 5 Krustys! 🦀🦀🦀🦀🦀

This comment has been minimized.

@joshtriplett

joshtriplett Dec 21, 2016

Member

For accessibility and clarity, please don't use the emoji as the primary indicator. I think I'd rather see separate badges for each point, not a fuzzy representation of the aggregate score.

@joshtriplett

joshtriplett Dec 21, 2016

Member

For accessibility and clarity, please don't use the emoji as the primary indicator. I think I'd rather see separate badges for each point, not a fuzzy representation of the aggregate score.

text/0000-crates.io-default-ranking.md
+ top-level items are extremely well documented, 170/195 or 87%. Our
+ definition of "top-level" counts the overall crate as an item. We think our
+ doc coverage POC can be modified to report this number.
+ - Would need to unpack and run this on each package version in a background

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016

Contributor

This is tricky due to #[cfg()]. docs.rs currently is building every single crate fully, although unnecessarily, but cargo model isn’t exactly compatible with desire to avoid running arbitrary code in order to figure out documentation for a crate.

@nagisa

nagisa Dec 20, 2016

Contributor

This is tricky due to #[cfg()]. docs.rs currently is building every single crate fully, although unnecessarily, but cargo model isn’t exactly compatible with desire to avoid running arbitrary code in order to figure out documentation for a crate.

text/0000-crates.io-default-ranking.md
+ job started by a publish; then save the percentage in crates.io's database.
+
+- In the crate root documentation, presence of a section headed with the word
+ "Example" and containing a codeblock

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016

Contributor

What about “Examples”, “Use cases”, “Common usage” and similar synonymous phrases? What if example is, say, for some GUI tool provided by crate (e.g. video tutorial demonstrating use of an UI builder of some sort)?

@nagisa

nagisa Dec 20, 2016

Contributor

What about “Examples”, “Use cases”, “Common usage” and similar synonymous phrases? What if example is, say, for some GUI tool provided by crate (e.g. video tutorial demonstrating use of an UI builder of some sort)?

text/0000-crates.io-default-ranking.md
+ opportunity to encourage at least one to be present reliably.
+ - Increases the doc percentage score by 5%
+
+- Presence of files in `/examples`

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016

Contributor

Similar comment. What if people put their “examples” into directories with semantically equivalent meaning? What if the example files are in root of the repository? In src/ and linked to from documentation?

@nagisa

nagisa Dec 20, 2016

Contributor

Similar comment. What if people put their “examples” into directories with semantically equivalent meaning? What if the example files are in root of the repository? In src/ and linked to from documentation?

This comment has been minimized.

@est31

est31 Dec 20, 2016

Contributor

examples/ is already a standardized directory, as cargo run --example <name> runs examples/<name>.rs.

@est31

est31 Dec 20, 2016

Contributor

examples/ is already a standardized directory, as cargo run --example <name> runs examples/<name>.rs.

text/0000-crates.io-default-ranking.md
+[cargo doc-coverage]: https://crates.io/crates/cargo-doc-coverage
+[examples]: https://github.com/rust-lang/cargo/issues/2760
+
+<table>

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016

Contributor

Another HTML table T_T

@nagisa

nagisa Dec 20, 2016

Contributor

Another HTML table T_T

text/0000-crates.io-default-ranking.md
+ - Number of releases in the last year - 10%
+ - Number of releases in the last 6 mo - 30%
+ - Number of releases in the last month - 60%
+ - Yanked versions are not counted.

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016

Contributor

What about the essentially “completed” utility crates such as, say, wrapper crates for machine instructions? Lack of releases does not imply lack of maintenance.

@nagisa

nagisa Dec 20, 2016

Contributor

What about the essentially “completed” utility crates such as, say, wrapper crates for machine instructions? Lack of releases does not imply lack of maintenance.

This comment has been minimized.

@nrc

nrc Dec 20, 2016

Member

Is second this concern

@nrc

nrc Dec 20, 2016

Member

Is second this concern

This comment has been minimized.

@steveklabnik

steveklabnik Dec 20, 2016

Member

Yes, this one is tough. There are a number of small, focused crates that will rarely ever get an update, but are good crates and should be used.

@steveklabnik

steveklabnik Dec 20, 2016

Member

Yes, this one is tough. There are a number of small, focused crates that will rarely ever get an update, but are good crates and should be used.

This comment has been minimized.

@withoutboats

withoutboats Dec 20, 2016

Contributor

Such crates would be compared against alternatives for the same purpose, right? It doesn't seem like a big issue, because all of them would have similar scores.

@withoutboats

withoutboats Dec 20, 2016

Contributor

Such crates would be compared against alternatives for the same purpose, right? It doesn't seem like a big issue, because all of them would have similar scores.

This comment has been minimized.

@sgrif

sgrif Dec 20, 2016

Contributor

Is there any competition in that space? Are we worried about how libc or winapi are going to be ranked?

@sgrif

sgrif Dec 20, 2016

Contributor

Is there any competition in that space? Are we worried about how libc or winapi are going to be ranked?

This comment has been minimized.

@joshtriplett

joshtriplett Dec 21, 2016

Member

I don't think we should use turnover/churn as a metric of goodness or badness. Some crates have frequent releases due to instability and churn; other crates have frequent releases because of regular maintenance. Some crates have rare releases because nobody maintains them; other crates have rare releases because they do their job and don't need further changes.

@joshtriplett

joshtriplett Dec 21, 2016

Member

I don't think we should use turnover/churn as a metric of goodness or badness. Some crates have frequent releases due to instability and churn; other crates have frequent releases because of regular maintenance. Some crates have rare releases because nobody maintains them; other crates have rare releases because they do their job and don't need further changes.

This comment has been minimized.

text/0000-crates.io-default-ranking.md
+
+- Stable version number
+ - >= 1.0.0 ranks higher than < 1.0.0
+ - >= 1.0.0 increases the maintenance score by 5%.

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016

Contributor

I fail to see benefit of this scoring trigger. It is trivial to set the version number to 1.0.0 on first release and publish breaking releases everyday by increasing the left-most digit instead.

The best this will achieve is reinforcing the crates.io ecosystem to move from the currently quite common 0.x.y versions to x.y.0.

@nagisa

nagisa Dec 20, 2016

Contributor

I fail to see benefit of this scoring trigger. It is trivial to set the version number to 1.0.0 on first release and publish breaking releases everyday by increasing the left-most digit instead.

The best this will achieve is reinforcing the crates.io ecosystem to move from the currently quite common 0.x.y versions to x.y.0.

This comment has been minimized.

@steveklabnik

steveklabnik Dec 20, 2016

Member

I'm a big proponent of this. And if it doesn't matter, and 0.x.y is the same as x.y.0, then I don't know why there's an objection.

@steveklabnik

steveklabnik Dec 20, 2016

Member

I'm a big proponent of this. And if it doesn't matter, and 0.x.y is the same as x.y.0, then I don't know why there's an objection.

This comment has been minimized.

@withoutboats

withoutboats Dec 20, 2016

Contributor

It would be ideal to downrank crates with overfrequent major version increments, but that could encourage breaking changes in 1.x releases, which is the opposite of what we want.

@withoutboats

withoutboats Dec 20, 2016

Contributor

It would be ideal to downrank crates with overfrequent major version increments, but that could encourage breaking changes in 1.x releases, which is the opposite of what we want.

text/0000-crates.io-default-ranking.md
+hover, with a link to a more thorough explanation. We like the information
+density in the way [npms][] displays scores:
+
+<img src="http://i.imgur.com/yadRNyy.png" src="Example npms score circles" width="442" />

This comment has been minimized.

@nagisa

nagisa Dec 20, 2016

Contributor

Markdown syntax please. At least you’ll avoid nonsensical HTML where alt text goes into the src attribute :)

![Example npms score circles](http://i.imgur.com/yadRNyy.png)
@nagisa

nagisa Dec 20, 2016

Contributor

Markdown syntax please. At least you’ll avoid nonsensical HTML where alt text goes into the src attribute :)

![Example npms score circles](http://i.imgur.com/yadRNyy.png)
@brson

This comment has been minimized.

Show comment
Hide comment
@brson

brson Dec 20, 2016

Contributor

Very well written RFC. Thank you.

Contributor

brson commented Dec 20, 2016

Very well written RFC. Thank you.

@est31

This comment has been minimized.

Show comment
Hide comment
@est31

est31 Dec 20, 2016

Contributor

Some ideas of improvement:

  • "Percentage of docs comments" is no good measure, at least not how its implemented right now. A crate to make requests with a function with_url and no docs is better than one with a function u and docs that explain that u stands for url. Just making an on/off check for whether there is documentation for some public item or not is too simple IMO, as in crates with well named identifiers, single line documentation is almost useless. Far more useful is the documentation that follows, so that you know about precise behaviour of the functions/structs. So I'd suggest to only count multi line documentation, and disregard any single line one, or value it much less than multi line documentation (maybe 10% or sth). I also highly appreciate example code either in examples/ or in the root as doc comment, as this is very helpful.

  • Remove the >=1.0.0 check, it only maps of how well the crate authors think their crate is, not how well it actually is. All this does is that all crates will release their initial version as 1.0.0. Yes, semver mandates that after 1.0.0 you don't do breaking changes in minor versions, but crate authors can just simply do major versions instead. Scoring by the number of owners is a good measure though I think, as the more people are actually maintaining a crate, the more likely its to live on once a maintainer gets bored or busy with other things.

  • The number of downloads score is a good indicator for popularity IMO, as it also includes bots, which means the crate gets used by projects with high quality assurance systems. Obviously its a bit spammy, and maybe one should value automatic downloads different from human downloads (whatever qualifies as that), but I think overall a good measure.

I think overall, the score shouldn't be something overemphasized, as the methodology is quite flawed in comparison to manual involvement, like the manual moderation of stack stack overflow.

Contributor

est31 commented Dec 20, 2016

Some ideas of improvement:

  • "Percentage of docs comments" is no good measure, at least not how its implemented right now. A crate to make requests with a function with_url and no docs is better than one with a function u and docs that explain that u stands for url. Just making an on/off check for whether there is documentation for some public item or not is too simple IMO, as in crates with well named identifiers, single line documentation is almost useless. Far more useful is the documentation that follows, so that you know about precise behaviour of the functions/structs. So I'd suggest to only count multi line documentation, and disregard any single line one, or value it much less than multi line documentation (maybe 10% or sth). I also highly appreciate example code either in examples/ or in the root as doc comment, as this is very helpful.

  • Remove the >=1.0.0 check, it only maps of how well the crate authors think their crate is, not how well it actually is. All this does is that all crates will release their initial version as 1.0.0. Yes, semver mandates that after 1.0.0 you don't do breaking changes in minor versions, but crate authors can just simply do major versions instead. Scoring by the number of owners is a good measure though I think, as the more people are actually maintaining a crate, the more likely its to live on once a maintainer gets bored or busy with other things.

  • The number of downloads score is a good indicator for popularity IMO, as it also includes bots, which means the crate gets used by projects with high quality assurance systems. Obviously its a bit spammy, and maybe one should value automatic downloads different from human downloads (whatever qualifies as that), but I think overall a good measure.

I think overall, the score shouldn't be something overemphasized, as the methodology is quite flawed in comparison to manual involvement, like the manual moderation of stack stack overflow.

@withoutboats

This comment has been minimized.

Show comment
Hide comment
@withoutboats

withoutboats Dec 20, 2016

Contributor

The big concern with gamification is 'min/maxing' behavior. We don't want people writing crappy doc comments or releasing pointless updates just to keep their 'score' up. Obscuring exactly what your "score" is can help avoid people seeing that if they released another version, they could go up a ranking.

About 10 years ago I played a great online game called Puzzle Pirates which did its best to discourage min/maxing by obscuring your raw score with fuzzy statements; all the games were ranked with an ELO system but none of them showed your score. I think this is pretty effective.

🌟 This crate has very complete documentation.
This crate has few downloads.
⭐️ This crate releases frequently.

These needn't just be a raw equivalence to the ranking algorithm but could even have a certain degree of 'badginess' like there could be a special note about being above 1.0.

This gamification creates a tension with everything we measure, of course. For example, one thing I value in a crate is infrequent breaking changes, but if we measure that we might encourage users to release breaking changes using non-breaking semver!

(We could give people an interface for "committing to semver," given them a 'badge' and upranking crates with that commitment. This doesn't need to be policed - like taking their badge away if the violate semver - but it creates additional buy-in to the idea that following semver matters.)

Contributor

withoutboats commented Dec 20, 2016

The big concern with gamification is 'min/maxing' behavior. We don't want people writing crappy doc comments or releasing pointless updates just to keep their 'score' up. Obscuring exactly what your "score" is can help avoid people seeing that if they released another version, they could go up a ranking.

About 10 years ago I played a great online game called Puzzle Pirates which did its best to discourage min/maxing by obscuring your raw score with fuzzy statements; all the games were ranked with an ELO system but none of them showed your score. I think this is pretty effective.

🌟 This crate has very complete documentation.
This crate has few downloads.
⭐️ This crate releases frequently.

These needn't just be a raw equivalence to the ranking algorithm but could even have a certain degree of 'badginess' like there could be a special note about being above 1.0.

This gamification creates a tension with everything we measure, of course. For example, one thing I value in a crate is infrequent breaking changes, but if we measure that we might encourage users to release breaking changes using non-breaking semver!

(We could give people an interface for "committing to semver," given them a 'badge' and upranking crates with that commitment. This doesn't need to be policed - like taking their badge away if the violate semver - but it creates additional buy-in to the idea that following semver matters.)

@camlorn

This comment has been minimized.

Show comment
Hide comment
@camlorn

camlorn Dec 20, 2016

I was summoned here via IRC. Unicode has major screen reader concerns, to wit that you can't expect it to read. Of the options proposed by @nagisa, just showing the percent is the one I'd go for.

If we want to use symbols, the only way to make them reliably accessible is to use graphics with alt text. While I wish I did live in a world with full unicode awareness from my AT, this is not going to happen anytime soon.

camlorn commented Dec 20, 2016

I was summoned here via IRC. Unicode has major screen reader concerns, to wit that you can't expect it to read. Of the options proposed by @nagisa, just showing the percent is the one I'd go for.

If we want to use symbols, the only way to make them reliably accessible is to use graphics with alt text. While I wish I did live in a world with full unicode awareness from my AT, this is not going to happen anytime soon.

@nrc nrc added the T-dev-tools label Dec 20, 2016

@robojeb

This comment has been minimized.

Show comment
Hide comment
@robojeb

robojeb Dec 20, 2016

@tomaka I don't know if you can really assume a crate pre-1.0 that doesn't change is unmaintained, at least without looking at its dependencies. For example I wrote a PRNG crate and I don't think that I should go 1.0 because the Rand crate isn't 1.0 yet. It doesn't make sense for me to go "stable" if my dependencies are unstable. So my crate doesn't change much any more.

robojeb commented Dec 20, 2016

@tomaka I don't know if you can really assume a crate pre-1.0 that doesn't change is unmaintained, at least without looking at its dependencies. For example I wrote a PRNG crate and I don't think that I should go 1.0 because the Rand crate isn't 1.0 yet. It doesn't make sense for me to go "stable" if my dependencies are unstable. So my crate doesn't change much any more.

@Keats

This comment has been minimized.

Show comment
Hide comment
@Keats

Keats Dec 20, 2016

In general, I disagree with number of recent releases being used as metrics for "goodness". Losing ranks because your crate is stable doesn't seem like a good thing.

Also for the docs, what if the person prefers to do a github page/a site for the project rather than documenting through rustdoc? I always look for docs in readme/examples first and rustdoc last.

Lastly for the downloads, I would only count direct downloads of the crate. Dependencies downloaded should not change their score or at least be weighted them differently (if possible).

Keats commented Dec 20, 2016

In general, I disagree with number of recent releases being used as metrics for "goodness". Losing ranks because your crate is stable doesn't seem like a good thing.

Also for the docs, what if the person prefers to do a github page/a site for the project rather than documenting through rustdoc? I always look for docs in readme/examples first and rustdoc last.

Lastly for the downloads, I would only count direct downloads of the crate. Dependencies downloaded should not change their score or at least be weighted them differently (if possible).

@DanielKeep

This comment has been minimized.

Show comment
Hide comment
@DanielKeep

DanielKeep Dec 20, 2016

Glad to see the results of that survey being put to good use; this is good work. And now...

*puts Devil's Advocate hat on firmly*

Looks like it's time for another round of Game That System! *cheering crowd*

We have created a proof-of-concept cargo doc-coverage tool to count the number of public items and the percentage of those that have/don't have documentation.

/// Spiddles a huk.
pub fn spiddle_huk() { ... }

Max score, piece of cake. I don't even need examples if I hit 100% right off the bat. And that's even assuming I go to the effort of re-stating the item name; I could just put in TODO (or any other placeholder any hypothetical blacklist hasn't yet accounted for).

Last released version date: newer is better.

Allow me to present cargo maintain, which bumps the minor version number and publishes automatically. Whack a call to it in a once-every-15-days scheduled task, and you have a crate with a perfect maintenance score!

Hell, add an -R recursion switch and just run it on all the crates.

Number of downloads weighted by time across all versions.

I don't think I even need to point out how to game this one. What's great is that once you game it enough to push your ranking up, it will start to naturally snowball, and it just gets easier!

So three metrics, all trivially easy to subvert; sign me up!

*takes Devil's Advocate hat off*

<whingeing can-skip="yes">

The problem I have with these is that they're not incentivising the things they claim, but rather things which are only coincidentally related to the desired property. Yes, a well-documented crate will have many things being documented... but the inverse relationship doesn't hold.

Of course, all the actually desired properties are either subjective or pathologically hard to measure.

I suppose my other major issue is that I don't see this affecting the "search for a good crate" process in a positive fashion. I mean, unless I can trust these scores, I'd only be doing myself a disservice by using them. I think the one thing you can say is that bad crates will probably have low scores; high scores do not imply a good crate, though.

This makes me think that instead of having a "score", just have badges with minimum cut-offs. Does this crate have a decent proportion of its items documented, yes/no? Does this crate have examples, yes/no? This avoids the whole question of weighting, and makes gaming the system a little harder. It's also a lot clearer. You don't have to teach people what the score may or may not mean; they can just look at the badges and immediately see (assuming consistent alt text).

There's also a potential middle-ground in having some kind of basic overall "has a minimum number of badges" badge that can be expanded for more detail.

Also, I just straight-up don't like activity at all. I've written crates that haven't gotten updates in ages because they don't need them. I wrote them to do a thing, and they do that thing as well today as they did when they were last updated. Why should I be punished for writing good code in the first place?

To be clear, yes, I will check the update history when evaluating crates, but only as a very low-priority and very rough measure. This is talking about formalising it and making a clear, loud statement to the community: "update or be punished". I agree that measuring "time to fix" on bug reports is a much better measure, but that gets into questions of how to monitor this, and what the difference between a bug, a feature request, and a tracking issue are.

Before I forget, I also have to protest the use of emoji for a scale. On my machine, half of them don't render with the same appearance (solid black symbols versus colour icons), and the second-to-highest one looks more cloudy than the fourth-to-highest. If you desperately need to use iconography, just stick to stars.

</whingeing>

In lieu of hypothetical, accurate metrics, I think a better system would be badges for relatively simple ones that are presented not as marks of quality, but as just baseline features most crates should probably have. Perhaps:

  • Does it have a decent level of documentation present?
  • Does it have examples?
  • Is it in the top (5%, 10%, 25%) of downloaded crates?

As a final note, I like the idea of being able to "favourite" authors/maintainers. I'm not hot on other forms of manual user annotation, on the basis that such annotations can get out of date quickly, though it's certainly an alluring idea.

Edit: Oh, I totally forgot about the >=1.0.0 thing. Suffice to say, if you pull that, I'll just start all my crates at 1.0.0, because why wouldn't I?

DanielKeep commented Dec 20, 2016

Glad to see the results of that survey being put to good use; this is good work. And now...

*puts Devil's Advocate hat on firmly*

Looks like it's time for another round of Game That System! *cheering crowd*

We have created a proof-of-concept cargo doc-coverage tool to count the number of public items and the percentage of those that have/don't have documentation.

/// Spiddles a huk.
pub fn spiddle_huk() { ... }

Max score, piece of cake. I don't even need examples if I hit 100% right off the bat. And that's even assuming I go to the effort of re-stating the item name; I could just put in TODO (or any other placeholder any hypothetical blacklist hasn't yet accounted for).

Last released version date: newer is better.

Allow me to present cargo maintain, which bumps the minor version number and publishes automatically. Whack a call to it in a once-every-15-days scheduled task, and you have a crate with a perfect maintenance score!

Hell, add an -R recursion switch and just run it on all the crates.

Number of downloads weighted by time across all versions.

I don't think I even need to point out how to game this one. What's great is that once you game it enough to push your ranking up, it will start to naturally snowball, and it just gets easier!

So three metrics, all trivially easy to subvert; sign me up!

*takes Devil's Advocate hat off*

<whingeing can-skip="yes">

The problem I have with these is that they're not incentivising the things they claim, but rather things which are only coincidentally related to the desired property. Yes, a well-documented crate will have many things being documented... but the inverse relationship doesn't hold.

Of course, all the actually desired properties are either subjective or pathologically hard to measure.

I suppose my other major issue is that I don't see this affecting the "search for a good crate" process in a positive fashion. I mean, unless I can trust these scores, I'd only be doing myself a disservice by using them. I think the one thing you can say is that bad crates will probably have low scores; high scores do not imply a good crate, though.

This makes me think that instead of having a "score", just have badges with minimum cut-offs. Does this crate have a decent proportion of its items documented, yes/no? Does this crate have examples, yes/no? This avoids the whole question of weighting, and makes gaming the system a little harder. It's also a lot clearer. You don't have to teach people what the score may or may not mean; they can just look at the badges and immediately see (assuming consistent alt text).

There's also a potential middle-ground in having some kind of basic overall "has a minimum number of badges" badge that can be expanded for more detail.

Also, I just straight-up don't like activity at all. I've written crates that haven't gotten updates in ages because they don't need them. I wrote them to do a thing, and they do that thing as well today as they did when they were last updated. Why should I be punished for writing good code in the first place?

To be clear, yes, I will check the update history when evaluating crates, but only as a very low-priority and very rough measure. This is talking about formalising it and making a clear, loud statement to the community: "update or be punished". I agree that measuring "time to fix" on bug reports is a much better measure, but that gets into questions of how to monitor this, and what the difference between a bug, a feature request, and a tracking issue are.

Before I forget, I also have to protest the use of emoji for a scale. On my machine, half of them don't render with the same appearance (solid black symbols versus colour icons), and the second-to-highest one looks more cloudy than the fourth-to-highest. If you desperately need to use iconography, just stick to stars.

</whingeing>

In lieu of hypothetical, accurate metrics, I think a better system would be badges for relatively simple ones that are presented not as marks of quality, but as just baseline features most crates should probably have. Perhaps:

  • Does it have a decent level of documentation present?
  • Does it have examples?
  • Is it in the top (5%, 10%, 25%) of downloaded crates?

As a final note, I like the idea of being able to "favourite" authors/maintainers. I'm not hot on other forms of manual user annotation, on the basis that such annotations can get out of date quickly, though it's certainly an alluring idea.

Edit: Oh, I totally forgot about the >=1.0.0 thing. Suffice to say, if you pull that, I'll just start all my crates at 1.0.0, because why wouldn't I?

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Dec 20, 2016

Member

Thank you everyone for your comments so far! What I'm hearing most is that:

  • Both emoji and letter grades are not ideal. I'm thinking about the fuzzy statements @withoutboats mentioned and tomorrow I'm going to see what I can work up using those instead.
  • The version release formula as proposed isn't using useful time intervals: the latest version should be counted as "recently updated" if it was made in the last 6 weeks, at the shortest, to match those who release with Rust, and perhaps longer.
  • There's also a concern about "finished" crates that are maintained, but just don't need much maintenance. I'm actually not sure how someone unfamiliar with the crate would be able to distinguish these definitively by hand right now, without the crate explicitly saying as much in its README/documentation. I would love ideas on what we could do for this case.
  • There's concern over the effects that having a >= 1.0.0 bonus will have on the meaning of version numbers. While I understand that it might incentivize people to start at 1.0.0 instead of 0.1.0, don't we want to incentivize having an ecosystem that is at least perceived to be more stable? This recent urlo thread is relevant.
  • The doc coverage lint might not work with crates that have a build.rs and/or use cfg
  • Doc coverage isn't meaningful enough to be useful, since it's a binary "is this item documented or not" and doesn't capture how much documentation per item there might be, or documentation in a README, etc.
  • Maybe we shouldn't do ranking at all and only do badges! :)

I'm off to bed, looking forward to discussing this further! <3

Member

carols10cents commented Dec 20, 2016

Thank you everyone for your comments so far! What I'm hearing most is that:

  • Both emoji and letter grades are not ideal. I'm thinking about the fuzzy statements @withoutboats mentioned and tomorrow I'm going to see what I can work up using those instead.
  • The version release formula as proposed isn't using useful time intervals: the latest version should be counted as "recently updated" if it was made in the last 6 weeks, at the shortest, to match those who release with Rust, and perhaps longer.
  • There's also a concern about "finished" crates that are maintained, but just don't need much maintenance. I'm actually not sure how someone unfamiliar with the crate would be able to distinguish these definitively by hand right now, without the crate explicitly saying as much in its README/documentation. I would love ideas on what we could do for this case.
  • There's concern over the effects that having a >= 1.0.0 bonus will have on the meaning of version numbers. While I understand that it might incentivize people to start at 1.0.0 instead of 0.1.0, don't we want to incentivize having an ecosystem that is at least perceived to be more stable? This recent urlo thread is relevant.
  • The doc coverage lint might not work with crates that have a build.rs and/or use cfg
  • Doc coverage isn't meaningful enough to be useful, since it's a binary "is this item documented or not" and doesn't capture how much documentation per item there might be, or documentation in a README, etc.
  • Maybe we shouldn't do ranking at all and only do badges! :)

I'm off to bed, looking forward to discussing this further! <3

@aturon aturon added the T-libs label Dec 20, 2016

@nagisa

This comment has been minimized.

Show comment
Hide comment
@nagisa

nagisa Dec 20, 2016

Contributor

One of the ideas floating in my head is deriving some of the rating from reverse dependencies. Consider some crate (henceforth goodcrate) using a library e.g. image. Now this goodcrate has a lot of downloads and in general is a very well regarded crate (whatever criteria is used to rate this crate).

That probably means, to some extent, that image is also a good crate. To me it seems like this is a similar criterion to download count, but much more difficult to game and may actually mean something. It is not applicable at all to crates with [bin], though.

Contributor

nagisa commented Dec 20, 2016

One of the ideas floating in my head is deriving some of the rating from reverse dependencies. Consider some crate (henceforth goodcrate) using a library e.g. image. Now this goodcrate has a lot of downloads and in general is a very well regarded crate (whatever criteria is used to rate this crate).

That probably means, to some extent, that image is also a good crate. To me it seems like this is a similar criterion to download count, but much more difficult to game and may actually mean something. It is not applicable at all to crates with [bin], though.

@DanielKeep

This comment has been minimized.

Show comment
Hide comment
@DanielKeep

DanielKeep Dec 20, 2016

@nagisa Perhaps we could call this metric "crate rank". No, 'tis a silly name...

@nagisa Perhaps we could call this metric "crate rank". No, 'tis a silly name...

@est31

This comment has been minimized.

Show comment
Hide comment
@retep998

This comment has been minimized.

Show comment
Hide comment
@retep998

retep998 Dec 20, 2016

Member

I strongly disagree with any sort of score system based on the metrics as proposed. So many of those metrics are easy to game, and not necessarily a measure of actual quality.

What I would agree with is the system proposed by @nagisa. Maybe in conjunction with some way for sapient entities to provide input on which crates they think are good quality to boostrap that system.

We considered using number of other crates an author has, but that would skew heavily towards retep998.

Many of your metrics would skew heavily in either direction for winapi. Number of documented items? 0. Examples? 0 (but there's a huge amount of crates using it that can be considered examples). Number of reverse dependencies? In the hundreds. Number of downloads? Ridiculously large. Meanwhile it is still < 1.0, has only one owner (nobody else wants the burden of maintaining it), and doesn't release very often.

Member

retep998 commented Dec 20, 2016

I strongly disagree with any sort of score system based on the metrics as proposed. So many of those metrics are easy to game, and not necessarily a measure of actual quality.

What I would agree with is the system proposed by @nagisa. Maybe in conjunction with some way for sapient entities to provide input on which crates they think are good quality to boostrap that system.

We considered using number of other crates an author has, but that would skew heavily towards retep998.

Many of your metrics would skew heavily in either direction for winapi. Number of documented items? 0. Examples? 0 (but there's a huge amount of crates using it that can be considered examples). Number of reverse dependencies? In the hundreds. Number of downloads? Ridiculously large. Meanwhile it is still < 1.0, has only one owner (nobody else wants the burden of maintaining it), and doesn't release very often.

@cramertj

This comment has been minimized.

Show comment
Hide comment
@cramertj

cramertj Dec 20, 2016

Member

"While I understand that it might incentivize people to start at 1.0.0 instead of 0.1.0, don't we want to incentivize having an ecosystem that is at least perceived to be more stable? "

Personally, I think it's valuable to know whether the developer plans on making significant breaking changes before they consider their crate "stable". Currently, this is often indicated by a leading '0'.

Incentivizing crate authors to use 'x.y.0' instead of '0.x.y' won't make the ecosystem more stable-- it'll make it harder to tell which crates aren't stable.

Member

cramertj commented Dec 20, 2016

"While I understand that it might incentivize people to start at 1.0.0 instead of 0.1.0, don't we want to incentivize having an ecosystem that is at least perceived to be more stable? "

Personally, I think it's valuable to know whether the developer plans on making significant breaking changes before they consider their crate "stable". Currently, this is often indicated by a leading '0'.

Incentivizing crate authors to use 'x.y.0' instead of '0.x.y' won't make the ecosystem more stable-- it'll make it harder to tell which crates aren't stable.

@shahn

This comment has been minimized.

Show comment
Hide comment
@shahn

shahn Dec 20, 2016

The most important things to me are a manually curated, complete and accurate ChangeLog, documentation that is more than API docs (more like a how and why document about the purpose of the crate), a 100% reliable way to link up versions on crates.io with the actual git tag they're derived from. None of these are easy to verify/rank automatically (the last point on that list would be, if crates.io supported it). I fear that the proposed system is highly gameable and creates an illusion of quality that most crates cannot match in practice. I would rather depend on a crate that releases every three months with a well-documented set of new features than one where every pull request triggers an upload of the crate without care in the release process.

shahn commented Dec 20, 2016

The most important things to me are a manually curated, complete and accurate ChangeLog, documentation that is more than API docs (more like a how and why document about the purpose of the crate), a 100% reliable way to link up versions on crates.io with the actual git tag they're derived from. None of these are easy to verify/rank automatically (the last point on that list would be, if crates.io supported it). I fear that the proposed system is highly gameable and creates an illusion of quality that most crates cannot match in practice. I would rather depend on a crate that releases every three months with a well-documented set of new features than one where every pull request triggers an upload of the crate without care in the release process.

@kbknapp

This comment has been minimized.

Show comment
Hide comment
@kbknapp

kbknapp Mar 23, 2017

Since we're starting with the proposed metrics, what are the thoughts/possibility of making this slight change?

To count docs use: $ tokei src/ | rg 'Rust' | awk '{print $5}'
To count code use: $ tokei src/ | rg 'Rust' | awk '{print $4}'

This gives some big benefits but still retains the proposed metrics. The biggest two gains are:

  • This correctly includes /*! */ style comments which several very prominent crates use, whereas the system as proposed would exclude them, hurting the docs score.
  • This only counts lines of Rust code (not blank lines, integration tests, benches, automatically deducts comments, etc) giving a more accurate count

With this method, here's how the scores would look:

  • combine:

    • 1,475 lines of documentation
    • 105 lines in README.md
    • 3565 lines of Rust
    • (1475 + 105) / 3565 = 1580/3565 = .44
  • nom:

    • 2,930 lines of documentation
    • 374 lines in README.md
    • 9,870 lines of Rust
    • (2930 + 374) / 9870 = 3354 / 9870 = .34
  • peresil:

    • 180 lines of documentation
    • 20 lines in README.md
    • 808 lines of Rust
    • (180 + 20) / 808 = 200/808 = .25
  • lalrpop:

    • 12,627 lines of documentation
    • 116 lines in ../README.md
    • 45,073 lines of Rust
    • (12627 + 116) / 45073 = 12743/45073 = .28
  • peg:

    • 3 lines of documentation
    • 214 lines in README.md
    • 820 lines of Rust
    • (3 + 214) / 820 = 217/820 = .27

Downsides

  • Counts regular comments (although this should be a very small number in comparison), but I believe it's still far more accurate
  • Still counts unit tests, but the proposal as it stands does this also
  • Doesn't count `examples/, but should these be counted towards code (negatively affecting score), or docs (positively affecting score)? This could be added though.

tokei is a Rust tool for counting lines of code and comments very fast.

kbknapp commented Mar 23, 2017

Since we're starting with the proposed metrics, what are the thoughts/possibility of making this slight change?

To count docs use: $ tokei src/ | rg 'Rust' | awk '{print $5}'
To count code use: $ tokei src/ | rg 'Rust' | awk '{print $4}'

This gives some big benefits but still retains the proposed metrics. The biggest two gains are:

  • This correctly includes /*! */ style comments which several very prominent crates use, whereas the system as proposed would exclude them, hurting the docs score.
  • This only counts lines of Rust code (not blank lines, integration tests, benches, automatically deducts comments, etc) giving a more accurate count

With this method, here's how the scores would look:

  • combine:

    • 1,475 lines of documentation
    • 105 lines in README.md
    • 3565 lines of Rust
    • (1475 + 105) / 3565 = 1580/3565 = .44
  • nom:

    • 2,930 lines of documentation
    • 374 lines in README.md
    • 9,870 lines of Rust
    • (2930 + 374) / 9870 = 3354 / 9870 = .34
  • peresil:

    • 180 lines of documentation
    • 20 lines in README.md
    • 808 lines of Rust
    • (180 + 20) / 808 = 200/808 = .25
  • lalrpop:

    • 12,627 lines of documentation
    • 116 lines in ../README.md
    • 45,073 lines of Rust
    • (12627 + 116) / 45073 = 12743/45073 = .28
  • peg:

    • 3 lines of documentation
    • 214 lines in README.md
    • 820 lines of Rust
    • (3 + 214) / 820 = 217/820 = .27

Downsides

  • Counts regular comments (although this should be a very small number in comparison), but I believe it's still far more accurate
  • Still counts unit tests, but the proposal as it stands does this also
  • Doesn't count `examples/, but should these be counted towards code (negatively affecting score), or docs (positively affecting score)? This could be added though.

tokei is a Rust tool for counting lines of code and comments very fast.

@kbknapp

This comment has been minimized.

Show comment
Hide comment
@kbknapp

kbknapp Mar 23, 2017

I've downloaded every crate on crates.io (latest verion of the crate) and run them through the above two methods.

The raw results as well as the top 20% for each method can be seen here.

kbknapp commented Mar 23, 2017

I've downloaded every crate on crates.io (latest verion of the crate) and run them through the above two methods.

The raw results as well as the top 20% for each method can be seen here.

@kbknapp

This comment has been minimized.

Show comment
Hide comment
@kbknapp

kbknapp Mar 25, 2017

The results have been fixed, and sorted by name to make things easier to find. I've also added a diff. I'd recommend using xsv (an amazing CSV tool) to play with the data.

kbknapp commented Mar 25, 2017

The results have been fixed, and sorted by name to make things easier to find. I've also added a diff. I'd recommend using xsv (an amazing CSV tool) to play with the data.

@bluss

This comment has been minimized.

Show comment
Hide comment
@bluss

bluss Mar 25, 2017

README.rst scores zero 👎

bluss commented Mar 25, 2017

README.rst scores zero 👎

@ticki

This comment has been minimized.

Show comment
Hide comment
@ticki

ticki Mar 25, 2017

Contributor

There is a lot of things going on in this thread, and I doubt we can agree on a manually designed algorithm, so I propose using machine learning.

Here's the approach I'm thinking of:

When a person uploads their first crate, cargo will prompt them for being a survey respondent. If a crate lacks of some number of survey responses and the user depends on it, the user will be prompted prompted with a simple scale (say 1-100) of how good a crate is in total (considering documentation, API, etc. etc.).

This scale is trained against the data points of the crate in a neural network (or maybe just basic regression). Following points are considered:

  • Time it takes to close issues.
  • Number of tests.
  • % comments
  • % documentation
  • Number of unique downloads (tracked through a bloom filter to improve anonymity).
  • Number of reverse dependencies.
  • Average rating of reverse dependencies.
  • Number of dependencies.
  • Average rating of dependencies.

and so on.

The crate's rating is then produced based on the NN trained on this data set (data points → survey) and the score is represented as a number, maybe or maybe not expoosed to the user, but certainly used in search ordering.

Contributor

ticki commented Mar 25, 2017

There is a lot of things going on in this thread, and I doubt we can agree on a manually designed algorithm, so I propose using machine learning.

Here's the approach I'm thinking of:

When a person uploads their first crate, cargo will prompt them for being a survey respondent. If a crate lacks of some number of survey responses and the user depends on it, the user will be prompted prompted with a simple scale (say 1-100) of how good a crate is in total (considering documentation, API, etc. etc.).

This scale is trained against the data points of the crate in a neural network (or maybe just basic regression). Following points are considered:

  • Time it takes to close issues.
  • Number of tests.
  • % comments
  • % documentation
  • Number of unique downloads (tracked through a bloom filter to improve anonymity).
  • Number of reverse dependencies.
  • Average rating of reverse dependencies.
  • Number of dependencies.
  • Average rating of dependencies.

and so on.

The crate's rating is then produced based on the NN trained on this data set (data points → survey) and the score is represented as a number, maybe or maybe not expoosed to the user, but certainly used in search ordering.

@est31

This comment has been minimized.

Show comment
Hide comment
@est31

est31 Mar 25, 2017

Contributor

Trying to solve this with machine learning with training based on the rating sounds like a good idea, although I do think we should take different points for consideration. For example, test coverage is a far better (still not perfect) metric for number of tests, as I prefer to do large tests with multiple test cases inside, and about the time it takes to close issues, its a bad metric as well, as most projects track their feature requests as open issues (htop I think does not, they close every feature request immediately). Also, evil maintainers can just stick the NOTABUG label onto valid issues and they would be rewarded. % comments and % documentation sounds some good metric, but obviously sometimes code is self explaining, and I prefer that to code that puts all explanation into the comments.

Number of downloads is skewed towards already working crates.

I'm not too familiar with ML, but can we feed the entire source code of the crate to the network somehow and then let it predict the rating?

Contributor

est31 commented Mar 25, 2017

Trying to solve this with machine learning with training based on the rating sounds like a good idea, although I do think we should take different points for consideration. For example, test coverage is a far better (still not perfect) metric for number of tests, as I prefer to do large tests with multiple test cases inside, and about the time it takes to close issues, its a bad metric as well, as most projects track their feature requests as open issues (htop I think does not, they close every feature request immediately). Also, evil maintainers can just stick the NOTABUG label onto valid issues and they would be rewarded. % comments and % documentation sounds some good metric, but obviously sometimes code is self explaining, and I prefer that to code that puts all explanation into the comments.

Number of downloads is skewed towards already working crates.

I'm not too familiar with ML, but can we feed the entire source code of the crate to the network somehow and then let it predict the rating?

@ticki

This comment has been minimized.

Show comment
Hide comment
@ticki

ticki Mar 25, 2017

Contributor

@est31

The whole point of ML is that you don't have to worry about ANYTHING of that. Etablishing the correlation between say lines of test and survey response is entirely automatic. Same goes for number of downloads: It shouldn't be biased as it is all about rating.

For sorting properly, you could use unique downloads (maybe logarithm of it) times the rating or something like that.

Contributor

ticki commented Mar 25, 2017

@est31

The whole point of ML is that you don't have to worry about ANYTHING of that. Etablishing the correlation between say lines of test and survey response is entirely automatic. Same goes for number of downloads: It shouldn't be biased as it is all about rating.

For sorting properly, you could use unique downloads (maybe logarithm of it) times the rating or something like that.

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Mar 25, 2017

Member

I am opposed to adding machine learning; it's way too complex to implement and maintain. Furthermore, the results will be hard to understand.

Member

carols10cents commented Mar 25, 2017

I am opposed to adding machine learning; it's way too complex to implement and maintain. Furthermore, the results will be hard to understand.

@retep998

This comment has been minimized.

Show comment
Hide comment
@retep998

retep998 Mar 25, 2017

Member

@ticki What if we simply used the user's survey results directly instead of passing it through a machine learning filter?

Member

retep998 commented Mar 25, 2017

@ticki What if we simply used the user's survey results directly instead of passing it through a machine learning filter?

@ticki

This comment has been minimized.

Show comment
Hide comment
@ticki

ticki Mar 25, 2017

Contributor

@retep998 That would mean that it's imossible to score new or small crates, as they have no or few users.

Contributor

ticki commented Mar 25, 2017

@retep998 That would mean that it's imossible to score new or small crates, as they have no or few users.

@kornelski

This comment has been minimized.

Show comment
Hide comment
@kornelski

kornelski Mar 25, 2017

Contributor

Yeah, survey is going to be a sparse dataset, so it's not usable directly. However, it still could be useful for evaluation and tuning of ranking metrics. It's probably a separate research project, outside scope of this rfc.

Contributor

kornelski commented Mar 25, 2017

Yeah, survey is going to be a sparse dataset, so it's not usable directly. However, it still could be useful for evaluation and tuning of ranking metrics. It's probably a separate research project, outside scope of this rfc.

@retep998

This comment has been minimized.

Show comment
Hide comment
@retep998

retep998 Mar 25, 2017

Member

I think we should at least provide a way for users to provide input on the quality of specific crates. Even if we don't display that information, we can aggregate it and analyze it and decide what to do in the future.

Member

retep998 commented Mar 25, 2017

I think we should at least provide a way for users to provide input on the quality of specific crates. Even if we don't display that information, we can aggregate it and analyze it and decide what to do in the future.

@mgattozzi

This comment has been minimized.

Show comment
Hide comment
@mgattozzi

mgattozzi Mar 25, 2017

Contributor

Like @carols10cents I'm strongly opposed to using ML. It's difficult to get right, it's difficult to maintain, and for the amount of resources the Rust community has, it's not worth the time nor do we have a decent data set for it. It's not some magic bullet that a lot of people think it is and it's not as easy as dumping data into a neural net to learn.

Contributor

mgattozzi commented Mar 25, 2017

Like @carols10cents I'm strongly opposed to using ML. It's difficult to get right, it's difficult to maintain, and for the amount of resources the Rust community has, it's not worth the time nor do we have a decent data set for it. It's not some magic bullet that a lot of people think it is and it's not as easy as dumping data into a neural net to learn.

@ticki

This comment has been minimized.

Show comment
Hide comment
@ticki

ticki Mar 25, 2017

Contributor

@mgattozzi I'm not claiming it's a magic bullet in any way. I agree it's hard, but I believe it's easier than the manual approach.

Contributor

ticki commented Mar 25, 2017

@mgattozzi I'm not claiming it's a magic bullet in any way. I agree it's hard, but I believe it's easier than the manual approach.

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Mar 26, 2017

Member

Please open a new RFC if you'd like to propose adding machine learning to recommend crates; I won't be changing this RFC to add it.

@retep998:

I think we should at least provide a way for users to provide input on the quality of specific crates. Even if we don't display that information, we can aggregate it and analyze it and decide what to do in the future.

I'm not going to spend time implementing and maintaining a data collection mechanism if we don't have a specific use for that data :-/

Member

carols10cents commented Mar 26, 2017

Please open a new RFC if you'd like to propose adding machine learning to recommend crates; I won't be changing this RFC to add it.

@retep998:

I think we should at least provide a way for users to provide input on the quality of specific crates. Even if we don't display that information, we can aggregate it and analyze it and decide what to do in the future.

I'm not going to spend time implementing and maintaining a data collection mechanism if we don't have a specific use for that data :-/

@aturon

This comment has been minimized.

Show comment
Hide comment
@aturon

aturon Mar 27, 2017

Member

To echo what others have said, machine learning would constitute an entirely separate proposal, with a lot of its own complexities and downsides. As I emphasized in the motion to FCP, we can and should take an iterative approach here, and the RFC as it currently stands is something relatively straightforward to implement, that we can land within the constraints of this year's roadmap. I think it's also a clear improvement over the status quo, and has been through a number of rounds of iteration in this discussion.

The RFC includes extensive research, both in terms of prior art, and in terms of what the Rust community specifically values. It also lays out several assumptions/design constraints at the outset. After these many rounds of discussion, I personally think it's reached a quite reasonable point in the design space balancing between the goals suggested by the research, technical feasibility for a first-cut implementation, and various concerns about gaming and proxy measures.

In the interest of trying to drive things toward some kind of conclusion here, I think it's best to keep discussion to any final concerns about (1) the takeaways from the research or (2) the proposed design, in terms of its balance between the research-derived goals and what makes sense to try in a first-round effort here.

Member

aturon commented Mar 27, 2017

To echo what others have said, machine learning would constitute an entirely separate proposal, with a lot of its own complexities and downsides. As I emphasized in the motion to FCP, we can and should take an iterative approach here, and the RFC as it currently stands is something relatively straightforward to implement, that we can land within the constraints of this year's roadmap. I think it's also a clear improvement over the status quo, and has been through a number of rounds of iteration in this discussion.

The RFC includes extensive research, both in terms of prior art, and in terms of what the Rust community specifically values. It also lays out several assumptions/design constraints at the outset. After these many rounds of discussion, I personally think it's reached a quite reasonable point in the design space balancing between the goals suggested by the research, technical feasibility for a first-cut implementation, and various concerns about gaming and proxy measures.

In the interest of trying to drive things toward some kind of conclusion here, I think it's best to keep discussion to any final concerns about (1) the takeaways from the research or (2) the proposed design, in terms of its balance between the research-derived goals and what makes sense to try in a first-round effort here.

@rfcbot

This comment has been minimized.

Show comment
Hide comment
@rfcbot

rfcbot Apr 17, 2017

🔔 This is now entering its final comment period, as per the review above. 🔔

rfcbot commented Apr 17, 2017

🔔 This is now entering its final comment period, as per the review above. 🔔

@mark-i-m

This comment has been minimized.

Show comment
Hide comment
@mark-i-m

mark-i-m Apr 18, 2017

Contributor

Thanks @carols10cents for the great (and much needed) RFC!

This is a really long thread, so I just wanted to post a few links to the summary posts scattered therein for those just reading the thread now (like me)...

@ruuda's comments here and here summarize the majority of the thread pretty well, I think.

@aturon's FCP summary of the RFC is here.

I was also intrigued by scattered proposals to take influence from ranking systems already in common use, such as Amazon, StackOverflow, and GitHub stars/trending. I won't bikeshed here, but I think these are worth considering for inspiration as this goes through iterations...

Contributor

mark-i-m commented Apr 18, 2017

Thanks @carols10cents for the great (and much needed) RFC!

This is a really long thread, so I just wanted to post a few links to the summary posts scattered therein for those just reading the thread now (like me)...

@ruuda's comments here and here summarize the majority of the thread pretty well, I think.

@aturon's FCP summary of the RFC is here.

I was also intrigued by scattered proposals to take influence from ranking systems already in common use, such as Amazon, StackOverflow, and GitHub stars/trending. I won't bikeshed here, but I think these are worth considering for inspiration as this goes through iterations...

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Apr 21, 2017

If the goal is that "Rust should provide easy access to high quality crates" then badges and ranking are really just bells and whistles. The guts of the problem is to find a way to present information about all relevent crates so that a potential user can evaluate them easily. If the front page of crates.io had a keyword search that produced a list, ranked by the last 90 days downloads, with a a five line explanation drawn from the top of README.md, this would be ideal for me. The keyword search might give me 10 or 20 options, and at 5 lines each it wouldn't be much trouble at all to scan through. The first 5 lines would probably give a good general indication of quality of the documentation.

ghost commented Apr 21, 2017

If the goal is that "Rust should provide easy access to high quality crates" then badges and ranking are really just bells and whistles. The guts of the problem is to find a way to present information about all relevent crates so that a potential user can evaluate them easily. If the front page of crates.io had a keyword search that produced a list, ranked by the last 90 days downloads, with a a five line explanation drawn from the top of README.md, this would be ideal for me. The keyword search might give me 10 or 20 options, and at 5 lines each it wouldn't be much trouble at all to scan through. The first 5 lines would probably give a good general indication of quality of the documentation.

@Razican

This comment has been minimized.

Show comment
Hide comment
@Razican

Razican Apr 26, 2017

The rendered RFC only talks about Coveralls. I personally have had a lot of issues with Coveralls configuration and recently have moved to Codecov in all my projects. They even provide Rust templates, and of course, badges.

Razican commented Apr 26, 2017

The rendered RFC only talks about Coveralls. I personally have had a lot of issues with Coveralls configuration and recently have moved to Codecov in all my projects. They even provide Rust templates, and of course, badges.

@carols10cents carols10cents added this to FCP in Tracker Apr 26, 2017

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Apr 26, 2017

Member

@Razican

The rendered RFC only talks about Coveralls. I personally have had a lot of issues with Coveralls configuration and recently have moved to Codecov in all my projects. They even provide Rust templates, and of course, badges.

Interesting! I didn't know about Codecov. The implementation will definitely be extensible-- for example, someone sent in a PR to add a badge for GitLab CI status to go along with the Travis and Appveyor badges, and it was a pretty straightforward change.

Member

carols10cents commented Apr 26, 2017

@Razican

The rendered RFC only talks about Coveralls. I personally have had a lot of issues with Coveralls configuration and recently have moved to Codecov in all my projects. They even provide Rust templates, and of course, badges.

Interesting! I didn't know about Codecov. The implementation will definitely be extensible-- for example, someone sent in a PR to add a badge for GitLab CI status to go along with the Travis and Appveyor badges, and it was a pretty straightforward change.

@rfcbot

This comment has been minimized.

Show comment
Hide comment
@rfcbot

rfcbot Apr 27, 2017

The final comment period is now complete.

rfcbot commented Apr 27, 2017

The final comment period is now complete.

@aturon aturon referenced this pull request in rust-lang/rust Apr 29, 2017

Open

Tracking issue for RFC 1824: Proposal for default crate recommendation ranking #41616

5 of 9 tasks complete

@aturon aturon merged commit 38c59ee into rust-lang:master Apr 29, 2017

@aturon

This comment has been minimized.

Show comment
Hide comment
@aturon

aturon Apr 29, 2017

Member

Huzzah! This RFC has been merged, and is tracked here. As has been emphasized throughout, this design is a starting point, and we'll expect to do follow-up evaluation and iteration.

Thanks everyone who participated in this in-depth discussion, and especially @carols10cents for writing such a thoughtful RFC and sticking it out through 258 comments :-)

Member

aturon commented Apr 29, 2017

Huzzah! This RFC has been merged, and is tracked here. As has been emphasized throughout, this design is a starting point, and we'll expect to do follow-up evaluation and iteration.

Thanks everyone who participated in this in-depth discussion, and especially @carols10cents for writing such a thoughtful RFC and sticking it out through 258 comments :-)

@shepmaster shepmaster deleted the integer32llc:crates-io-ranking branch Apr 29, 2017

@nasa42 nasa42 referenced this pull request in rust-unofficial/awesome-rust May 4, 2017

Closed

Transfer #289

@gbutler69

This comment has been minimized.

Show comment
Hide comment
@gbutler69

gbutler69 Jun 4, 2017

Couple of thoughts. A crate should receive a higher score based on the number of other crates on crates.io that depend upon it. A create should receive a lower score based on the number of other crates it depends upon. Depending on lower-ranked vs higher-ranked crates should have a larger negative impact on a crate's rating. Having higher-ranked crates vs. lower-ranked crates depending upon a crate should have a larger positive impact on the ranking of the crate.

Couple of thoughts. A crate should receive a higher score based on the number of other crates on crates.io that depend upon it. A create should receive a lower score based on the number of other crates it depends upon. Depending on lower-ranked vs higher-ranked crates should have a larger negative impact on a crate's rating. Having higher-ranked crates vs. lower-ranked crates depending upon a crate should have a larger positive impact on the ranking of the crate.

@mark-i-m

This comment has been minimized.

Show comment
Hide comment
@mark-i-m

mark-i-m Jun 4, 2017

Contributor
Contributor

mark-i-m commented Jun 4, 2017

@le-jzr

This comment has been minimized.

Show comment
Hide comment
@le-jzr

le-jzr Jun 4, 2017

Those two categories should be separate anyway. Unless you are searching the database just for fun (or reviewing, or whatever), you either want specifically a library, or specifically an application.

I think @gbutler69 has the right idea.

le-jzr commented Jun 4, 2017

Those two categories should be separate anyway. Unless you are searching the database just for fun (or reviewing, or whatever), you either want specifically a library, or specifically an application.

I think @gbutler69 has the right idea.

@le-jzr

This comment has been minimized.

Show comment
Hide comment
@le-jzr

le-jzr Jun 4, 2017

But the score interaction would have to be broken up by individual influences, or otherwise stratified. Naive implementation could lead to nasty feedback cycles.

le-jzr commented Jun 4, 2017

But the score interaction would have to be broken up by individual influences, or otherwise stratified. Naive implementation could lead to nasty feedback cycles.

@MoSal

This comment has been minimized.

Show comment
Hide comment
@MoSal

MoSal Jun 4, 2017

@gbutler69
Note that this RFC has already been merged. And it's not clear where re-evaluation and iteration discussions are going to take place (probably not here in this PR).

A crate should receive a higher score based on the number of other crates on crates.io that depend upon it.

Agreed.

A create should receive a lower score based on the number of other crates it depends upon.

This will encourage bad practices:

  • Copying code from other crates instead of depending on them.
  • Re-implementing functionality available in other crates poorly.

In other words, this will encourage people to work against the crate ecosystem, not work with it.

Depending on lower-ranked vs higher-ranked crates should have a larger negative impact on a crate's rating.

Again. This will encourage bad practices. For example, this will discourage people from releasing modular crates. Fearing the lower-level crates will have a low-score which will reflect poorly on the higher-level ones.

Having higher-ranked crates vs. lower-ranked crates depending upon a crate should have a larger positive impact on the ranking of the crate.

Could be useful. But this might have complex interactions with other factors. And the implementation details are not clear. For example:

  • Should the average score of all dependants be considered?
    • Or just the top 5?
    • Or just the one with the highest score?
  • Should there be a minimum number of dependants for this factor to be considered?
    • What is that number?
  • How much weight should be given to this factor?
  • Should a high-ranking dependant stop depending on foo, how much of a negative impact should foo suffer based on that event alone?

I think the straight forward factors I took into consideration in cargo-esr are enough to paint a rough picture of how much a crate is relevantly depended on.

MoSal commented Jun 4, 2017

@gbutler69
Note that this RFC has already been merged. And it's not clear where re-evaluation and iteration discussions are going to take place (probably not here in this PR).

A crate should receive a higher score based on the number of other crates on crates.io that depend upon it.

Agreed.

A create should receive a lower score based on the number of other crates it depends upon.

This will encourage bad practices:

  • Copying code from other crates instead of depending on them.
  • Re-implementing functionality available in other crates poorly.

In other words, this will encourage people to work against the crate ecosystem, not work with it.

Depending on lower-ranked vs higher-ranked crates should have a larger negative impact on a crate's rating.

Again. This will encourage bad practices. For example, this will discourage people from releasing modular crates. Fearing the lower-level crates will have a low-score which will reflect poorly on the higher-level ones.

Having higher-ranked crates vs. lower-ranked crates depending upon a crate should have a larger positive impact on the ranking of the crate.

Could be useful. But this might have complex interactions with other factors. And the implementation details are not clear. For example:

  • Should the average score of all dependants be considered?
    • Or just the top 5?
    • Or just the one with the highest score?
  • Should there be a minimum number of dependants for this factor to be considered?
    • What is that number?
  • How much weight should be given to this factor?
  • Should a high-ranking dependant stop depending on foo, how much of a negative impact should foo suffer based on that event alone?

I think the straight forward factors I took into consideration in cargo-esr are enough to paint a rough picture of how much a crate is relevantly depended on.

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Jun 5, 2017

Member

Note that this RFC has already been merged. And it's not clear where re-evaluation and iteration discussions are going to take place (probably not here in this PR).

Yes, since this RFC has been merged, large changes proposed to this should get a new RFC. Small changes can be proposed in an issue on crates.io's repo, and the re-evaluation and iteration discussions will probably take place on crates.io's repo as well.

I'm unsubscribing from this thread, thank you all for your discussion!

Member

carols10cents commented Jun 5, 2017

Note that this RFC has already been merged. And it's not clear where re-evaluation and iteration discussions are going to take place (probably not here in this PR).

Yes, since this RFC has been merged, large changes proposed to this should get a new RFC. Small changes can be proposed in an issue on crates.io's repo, and the re-evaluation and iteration discussions will probably take place on crates.io's repo as well.

I'm unsubscribing from this thread, thank you all for your discussion!

@sgrif sgrif referenced this pull request in rust-lang/crates.io Jun 18, 2017

Closed

Suggest append a voting #786

@carols10cents carols10cents referenced this pull request in rust-lang/crates.io Sep 7, 2017

Closed

Clean up outdated #1043

@carols10cents carols10cents referenced this pull request in rust-lang/crates.io Sep 20, 2017

Closed

considerations on parking #1063

@carols10cents carols10cents referenced this pull request in rust-lang/cargo Jan 9, 2018

Open

cargo new: create a README #3506

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment