There has to be a way to automate this

[frigginglorious]

For github in particular, It probably could be automated from an API.

Or has anyone tried to add a shell script that parses pages like the contributor page (eg https://github.com/kentcdodds/all-contributors/graphs/contributors) that updates and replaces a contributor table at the bottom of README.md?

Maybe not. Doing it that way would be hardly foolproof, and difficult to organize.

[kentcdodds]

The challenge with fully automating it is that you'd have trouble automating when someone posts a blogpost, gives a talk, is constructively reviewing a PR, or answering questions on Stack Overflow (just to name a few examples, not to mention the difficulty with telling the difference between a code contribution and a design/test contribution).

However, I think that we can make it easier to maintain this list and @jfmengels is working on that 👍

[chrissimpkins]

While I really hesitate to propose additional repository meta files, a JSON or YAML file spec could help here and tools could be built around it.

Ugh, did I just say that...

[kentcdodds]

I agree that a config spec would be nice. Here's a start maybe: all-contributors/cli#1

[chrissimpkins]

@kentcdodds @jfmengels looks like it is coming together already!

[jfmengels]

Trying to :)

[glasnt]

I think I might be able to help here :)

I've made a tool specifically for automating the collecting of contributors from the GitHub API, called octohatrack (article about the app).

The algorithm is fairly straight forward, but it takes a while to parse for larger repos.

I've also little apps that show how to get at least some of the contributors for Stackoverflow tags and Wikipedia posts under the LABHR, if that helps :)

The visualisation method that @leesdolphin and I have been working on at https://labhr.github.io/js-hatrack/ doesn't have as much visual differentiation as this version, but works on a binary code/non-coding method.

[kentcdodds]

Very cool @glasnt! Thanks for chiming in!

I just ran it on my more popular repo angular-formly and it came up with hundreds of contributors. Super cool!

I think the real challenge with a fully automated solution is as I stated earlier:

The challenge with fully automating it is that you'd have trouble automating when someone posts a blogpost, gives a talk, is constructively reviewing a PR, or answering questions on Stack Overflow (just to name a few examples, not to mention the difficulty with telling the difference between a code contribution and a design/test contribution).

I'm just not sure that we can go 100% automated. I think the best is to develop a tool as @jfmengels is working on to make it easier to generate the markdown, and then encourage contributors to add themselves (and their contributions) to the list.

[glasnt]

@kentcdodds

So that's always going to be a problem. The major hubs can be automated, but it's not going to pick up everyone.

You can have 'rss feed planets' and such, but there's no a lot that beats the maintainers actively attributing help.

The future work I have planned for hatrack is to automate the github hubs, and then have a manual addition process, specifically for acknowledgements outside of GitHub. I think having a CONTRIBUTORS page that feeds into that might be super useful, and your categorisation stuff on top makes it extra fancy :)

[chrissimpkins]

@glasnt Very cool project Katie. It seems that you've already given a good deal of thought to something that (to my knowledge) we haven't resolved which is how to bridge high contributor volume, mature projects to this specification. This will prove to be a challenge.

Anyone have thoughts about this? From what I understand about @jfmengels's tool, it will be perfect for moderated, semi-automated list generation for low contributor volume projects at any stage of the project history, and for all projects from early stages in the project history where you have the opportunity to build the list prospectively. How do more mature, higher volume projects that already have numerous previous contributors reach into their history to support the spec?

[jfmengels]

@glasnt Cool project!
I'm working on all-contributors/cli#1 (soon in the shelves), so any tool that can fetch contributor data and write it in the configuration file will work with my tool. So if we build tools that find contributors well (I don't know how effective @glasnt's project currently is at that), I think we can build a nice toolchain.
Please file issues on my repo if you think of cool/useful things :)

[burodepeper]

Hi there, just wanted to say hi! ; ) (hey Chris!)

Actually, I was looking into/for something like this, so yay! I have to look deeper in to what has already been said and done, but thus far I find the suggested things rather (visually) verbose, though I recognize that that is fully my own opinion.

I'm curious what you all think about the order in which contributors appear. I might have skimmed over some previous remarks, but automation doesn't really allow for the subtle nuances of "intrinsic value added to a project". Even more so when it comes to sources that are non-public or non-trivial; random conversations via chat/email, social-media attention, perhaps even a seemingly unrelated thing a friend mentions during a bar-crawl. It's a little off-topic, but what are your views on adding those people in the first place?

[kentcdodds]

@burodepeper, thanks for your interest in the spec! I think that as you read up on the spec and the comments here your feelings are shared :-) What you describe is basically the reason this spec was created.

[mbad0la]

@kentcdodds @frigginglorious @chrissimpkins @jfmengels , please have a look at this atom package i'm working on. This helps to automate the all-contributors-spec inside atom text editor.

[kentcdodds]

Very cool! Do you wanna add it to the README?

[mbad0la]

Sure! I'll send a PR.

[boneskull]

I'm curious what you all think about the order in which contributors appear.

alphabetical 😄

[kentcdodds]

Actually, on the JavaScript Air website, I recently had (almost) all guest/panelist images appear in random order each time the site is generated. We could do the same for this project. Randomize the contributors each time the table is generated. This would obviously only be reasonable with the CLI.

[mbad0la]

@kentcdodds @boneskull , having a randomised contributors' ordering seems like a good idea!
Btw, just a thought. Will it make sense to keep the repository maintainers out of the randomisation process? Let's always assign them the initial blocks in our table?

[boneskull]

So, I'm interested in getting this to work for Mocha. I've explored writing a tool to get at this type of dataset. It works similarly to octohatrack--you make a bunch of API calls and aggregate things.

But that's really bending the GitHub API in a direction it doesn't want to go. Sure, you can grab all this data at once and throw it in a cache. How can you efficiently process new data that comes in? You can't really lean on the Events API because of its limitations. You basically have to pull down all of the issues, PRs, wiki pages and commits again, then check if anything is newer than last time you ran the pulls.

It took me three tries--running it once, waiting 5 minutes, running again, waiting 25 minutes, then running a third time--using octohatrack to pull down all of the Mocha data, because I exceeded my request limit with my OAuth token. Granted, octohatrack could be optimized in a couple places, but this is not a fast or cheap operation any way you look at it.

Having gone through this song-and-dance with the API, I'm thinking the only solution that scales properly is going to be hook-based. You must run a service, and when something happens, let GitHub tell you. You will still have to do the initial pull of data.

Of course, that only covers GitHub! We're not talking about blog posts, webcasts, SO answers, etc. You may be able to automate some of it, but you're not necessarily going to have access to hooks on any given service--especially if your project doesn't own the user account. Even if you do, you'd want throw everything in a pile for manual review. Some sort of Bayesian classifier which could assist with the hooks or periodic searches or whathaveyou. Oh, and then there's BitBucket.

I feel like the spec, as written, simply won't scale to large projects; it's too ambitious. For large projects, this must be automated. If it must be automated, then it has to start with machines. The questions I'd ask then are:

• What public APIs can we use?
• Where can we use web hooks? If our service is event-based, we'll scale better.
• Which types of contributions are too "fuzzy" to easily automate? These should wait until after we cover the "easy" types.

FWIW, there is demand for this type of data amongst projects. It's important to make it visible, not just for the OSS community, but also for companies who may sponsor projects, or want more insight into their own (and their employees' activities). I don't have a lot of bandwidth, but I'm more than happy to think further about how to build out a service that accomplishes this.

[boneskull]

I'd like to clarify this point: if the project doesn't own the data, it's going to be tough to get. If Joe Contributor writes a blog post about your project, how will you know?

UPDATE: It still isn't very clear. But I hope you understand what I'm trying to say. Understanding a user wrote a blog post about your project is a very different thing from a computer understanding the same. Maybe Palantir would be interested in lending us some tech. 😝

[glasnt]

Hey @boneskull, I have a few things that might help you here:

• the ability to add 'custom' contributors to octohatrack that I mentioned earlier up in this thread has been implemented, which allows for the use case of 'Someone has blogged about this'
• There are a number of tools already in the LABHR suite for wikipedia and stackoverflow that leverage existing aggregations
• There have been a number of changes in GitHub for user-profile contribution attribution, so there may or may not be more project-profile contribution attribution later. Specifically, adding comment-level contributions. I highly doubt this will ever extend natively to contributions outside GitHub
• While webhooks might work for events from a certain point, backfilling will also be a problem
• As much as we can automate the internet-layer stuff, there's always the meatsphere we can't automate, so there always needs to be a manual addition mechanism for a comprehensive list of contributors.

[boneskull]

@glasnt Thanks.

I understand the "custom" contributor functionality, but this is not easily automated, which is my point.

There have been a number of changes in GitHub for user-profile contribution attribution, so there may or may not be more project-profile contribution attribution later. Specifically, adding comment-level contributions. I highly doubt this will ever extend natively to contributions outside GitHub

I'm not sure I follow what this is about.

While webhooks might work for events from a certain point, backfilling will also be a problem

Yes, the initial pull needs to happen, which I mentioned above. But thereafter, it's expensive to "poll".

As much as we can automate the internet-layer stuff, there's always the meatsphere we can't automate, so there always needs to be a manual addition mechanism for a comprehensive list of contributors.

I suppose what I'm trying to ask then w/r/t the specification is "should we have one or more contribution types for which data cannot be automatically gathered"?

[glasnt]

I'm not sure I follow what this is about.

There's been a bunch of changes recently in the GitHub interface to allow for actual customisation of user profiles:

• https://github.com/blog/2191-pin-repositories-to-your-github-profile
• https://github.com/blog/2173-more-contributions-on-your-profile

As far as I'm aware, there's no changes at this level for project profiles, to date.

---

"should we have one or more contribution types that for which data cannot be automatically gathered"?

Within the scope of all-contributors, I can't comment.

[boneskull]

There's been a bunch of changes recently in the GitHub interface to allow for actual customisation of user profiles:

I was under the impression that this was just a Facebook-like feature where you could present yourself to others the way you want to be seen. That's how I use it anyway. 😝 Did they alter the API for this?

So I think what you're saying is that GitHub may do some of this work for us in the future, based on the changes they made to the user profiles. But that's speculation, because GitHub doesn't tend to give advance notice of new features, and there's certainly not much of a dialogue going on. Which is 👎.

"should we have one or more contribution types that for which data cannot be automatically gathered"?

Within the scope of all-contributors, I can't comment.

I meant this as a general question to anyone interested.

[kentcdodds]

I think that the qualitative nature of this spec prevents it from being 100% automated. However, I think that adding people to the list of contributors is fairly simple. What I've done so far is ask that someone add themselves to the list. I have it as part of my CONTRIBUTING.md. It's worked fairly well so far. Of course, I don't have a huge project like Mocha. But I think that if people see that they can be recognized for writing a blogpost, they'll be eager to add themselves (despite the required effort).

And honestly, they don't even have to pull down the repo to add themselves with the script. One look at .all-contributorsrc and they'll see what's going on (though you may need to document how people can get the URL for their profile picture). Then people can simply add themselves in a PR from github.com

[patcon]

Oh hey! This other proejct appeared on my radar c/o @jywarren @ebarry, and might be relevant (or worth collaborating with): https://github.com/mntnr/name-your-contributors

cc: @RichardLitt (related: mntnr/name-your-contributors#45) @

[kentcdodds]

That's awesome @patcon! I'll bet that we could use that for the bot (#58) if anyone ever gets around to it!

[RichardLitt]

I've been meaning to post here for a while - I've been waiting for time to look over this closer. name-your-contributors could definitely help. It was almost tailor made for this purpose.

Happy to collaborate and work together! What would be the easiest way to figure out how?

[RichardLitt]

@Berkmann18 The bot probably covers this. I haven't had the time. :(

[Berkmann18]

@RichardLitt It doesn't as it's not auto fetching from the repo, it does automate the config/README change process but it will still require a request for each contributors.

[RichardLitt]

That is less than ideal. I think we can change that. Let's keep working in mntnr/name-your-contributors#45?

[Berkmann18]

@RichardLitt Yeah, ideally we could get the auto-fetching on the CLI which would then help the bot (which relies on the CLI) to then create a PR with all the existing contributors.

[mrchief]

This sounds similar to all-contributors/cli#117. Is this being worked on? I was planning to pick up on 117 but don't want to duplicate efforts.

[Berkmann18]

@mrchief Yes it is and we're both working on it (cf. my comments on your draft PR).

[akhilmhdh]

Hey guys. I came across this problem recently so I thought of building an easier one. So here is GitHub action made by me to automate contributors list in readme.
(https://github.com/marketplace/actions/contribute-list)

[Berkmann18]

@akhilmhdh What does it do that the bot/cli doesn't do?

[akhilmhdh]

@Berkmann18 hey no offence the bot is really amazing. I build the action it if someone wants a more easier way to automate the readme in a more simpler way, in which nothing has to be done later. Contributors list gets updated automatically as contribution increase. And it doesn't have as feature rich as that one.

[Berkmann18]

@akhilmhdh That's nice but it seems to lack an important thing that All Contributors have which makes the action currently unsuitable for those who want to follow the specification.
You're more than welcome to provide and contribute with your GHA knowledge tho 😀.

[PGijsbers]

Sorry if this isn't the place to ask, or if it's already answered elsewhere. We just adopted the bot over at openml-python! Like others before, we now face to task of adding each existing contributor to the list. As far as I can tell automatically detecting and crediting contributors to an existing project is still work in progress (even when limiting to e.g. code contributions)?

[Berkmann18]

@PGijsbers Yup, have you looked at all-contributors/cli#196 (comment)?

[PGijsbers]

Thanks @Berkmann18! So the advice is to use that feature branch? (with its caveats)

[Berkmann18]

@PGijsbers Yes, and contribute (if you can and want to).

[JoshuaKGoldberg]

In the meantime, I wrote:

• https://github.com/JoshuaKGoldberg/all-contributors-auto-action: GitHub Action to automate some contributors & contribution types
• https://github.com/JoshuaKGoldberg/all-contributors-for-repository: Backing package to generate that list

Note that as has been mentioned, some contributions can't be auto-detected from Git history or even changelogs. Copying this warning from both tools' READMEs:

Warning This tool only sees contributions that can be detected from the last 500 events in GitHub's API. Don't forget to manually add in other forms of contributions!

[Berkmann18]

@JoshuaKGoldberg Does that handle all of the contribution types you can get off a repo?

[JoshuaKGoldberg]

@Berkmann18 please re-read my comment in full 😛 (typing something more useful now)

[JoshuaKGoldberg]

OH I just reparsed your comment, sorry if I was being too snarky @Berkmann18. I thought you were asking if it handles all possible contribution types in general. But now I think you were actually asking if it handles all the types one could glean from a repo?

If so: it's somewhat comprehensive but I'm sure there are points I missed. It hasn't (yet?) had any logic added around file extensions mapping to contribution types, e.g. .md -> docs.

[Berkmann18]

@JoshuaKGoldberg Yup, that's what I meant (just realised I didn't reply to that comment).