RFC: Create a listing of all packages

[aviks]

This creates a page as part of the julia documentation suite with the details of currently available packages as specified in the METADATA. As requested in #1896

There are a couple of issues for comments

1. This needs a Github OAuth token, since un-authenticated requests are limited to 60 per hour, which is not enough given the number of packages we have. Therefore, you need to specify the token as an environment variable GH_AUTH for the script to work.

   Once the auth token is specified, run make listpkg to create the latest listing. Note that this will update your metadata repository, and install the JSON package if you dont have it yet.

2. I have checked in the resultant packagelist.rst. This is up for debate, but I thought that the requirements above should not be made mandatory for everyone who wants to build the docs locally.

Note: Read on if you don't already have an access token you can use.

To create a github oauth token, you will need to have a registered github application. Once you have an application, you can then call

curl -i -u <gh-user> -d @gh_auth https://api.github.com/authorizations

This will prompt for your gh password using basic authentication. gh_auth is a local file containing:

{
  "scopes": ["public_repo"],
  "client_id":"<application client id>",
  "client_secret":"<application client secret>",
  "note": "For Julia Metadata Listings"
}

Registered applications and authorised token are visible within your github profile page.

Contact me privately if you wan to use my registered app to create a token.

To reiterate, you dont have to do all this to check out how the docs look. Just run make html, and you'll get a listing valid as of this writing.

[aviks]

This is how it looks, once built by sphinx. There will be a link from the index of the documentation.

[Keno]

It's missing the pictures. Can't live without the pictures! What happened to the pictures??

[ViralBShah]

I like this. Can we get the pictures though?

[aviks]

Better?

[ViralBShah]

This looks good. Leaving the pull request around a bit longer for others to comment.

[Keno]

Looks good to me too!

[pao]

With the avatar feature, no one will be able to take any packages of mine seriously. Of course no one is probably taking me seriously anyways.

This looks cool! Any thoughts on how well this will scale up in the everyone-uses-Julia future?

It would be nice to not have an explicit GitHub dependency here. Maybe enable more features if GitHub is the host? Being able to index METADATA.jl would still be useful in private/internal METADATA.jl forks which maintain internal packages and may be hosted in other ways.

(I'm okay with merge-and-extend-later here, btw. Just thinking a bit.)

[StefanKarpinski]

On the other hand, everyone will take my packages very seriously. Viking4Lyfe!!

This does look very nice and I think we can worry about the eventual scaling to millions of packages future a little later on. Let's do merge-and-extend. We need to have a round of figuring out how to factor out GitHub support with some kind of hook architecture and add support for other git hosting services like BitBucket, etc.

[timholy]

I can just tell, you are going to make me finally choose a picture.

[aviks]

Currently, if the package is not hosted on Github, you will get a 'Not provided" text for the package description and maintainer fields. However, the package name and its dependencies will still show in the listing, since that information is via METADATA

To support other repositories, its currently a matter of implementing an else block to pick out author names and package description. Of course, that can and should be abstracted out as more repositories are supported

As for scaling, yes, this is all in one page. Eventually, we'll need pagination and search. Hopefully, sphinx will be of some assistance for that.

[aviks]

Thanks @ViralBShah . This is now live on http://docs.julialang.org/en/latest/packages/packagelist/

[ViralBShah]

Thanks @aviks. That was fantastic.

[rennis250]

I really like the look of this, @aviks. Nice work! Would it be possible to somehow include @o-jasper's name on the following packages?

GetC
GLU
GLUT
SDL
OpenGL

If possible, I think it would be better to list his name before mine in the maintainer list. I wouldn't feel right if his name weren't listed on that page.

--Rob

[ViralBShah]

Perhaps we need a per-package AUTHORS file.

[StefanKarpinski]

Is there a GitHub API to get the contributors to a package?

On Tue, Jan 8, 2013 at 5:37 PM, Viral B. Shah notifications@github.comwrote:

Perhaps we need a per-package AUTHORS file.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1935#issuecomment-12021334.

[pao]

Repo collaborators can be listed (http://developer.github.com/v3/repos/collaborators/) but I don't see anything else. That said, a package might still wish to present a single maintainer even if the package is maintained by a GitHub Organization, and an AUTHORS file would also cover the non-GitHub case, so I'm kind of inclined in that direction.

[johnmyleswhite]

The idea of including an AUTHORS file makes think that we need a single metadata file associated with each package that contains authors, maintainers, etc..

[ViralBShah]

Yes, we should certainly have only a single metadata file, rather than having a bunch of these.

[aviks]

@rennis250 The idea behind the maintainer field was that it is simply the owner of the repository linked from METADATA. Its is a piece of factual information, saying who owns the repo I am installing from. I did not see that as any form of attribution. Repo maintainers WILL change over the lifetime of the package, and will not necessarily correspond to the author of the majority of the codebase. However, I believe it is important to highlight this upfront, since, that's the owner of the repo I'm installing from.

So in my mind, that field has nothing to do with attribution. Attribution is best done within the package itself, by whatever means necessary. Hence, every package in the listing links back to its github page and homepage (if provided).

Having said that, there is a github api to get a list of contributors to a package, helpfully sorted by the number of contributions. I'll do a mockup for everyone to see how it looks, but my concern is that the top level listing page should be compact and information rich. You dont want to have to scroll a lot for each package. And reST limits the amount of layout flexibility you can get.

[ViralBShah]

@aviks Could you push an update to the package listing page? We have a couple more - especially GLM by @dmbates which we want to use next week for the julia tutorial. It would be nice if it were in the list, so that students can browse around.

[timholy]

Do you want us splitting out our multi-module packages before the end of the weekend?

[ViralBShah]

I would prefer not splitting multi-module packages as a general rule, but only if you think it is meaningful, and would make it easier for someone to use the packages.

[timholy]

I guess I'm referring to your message to the users list:

I think it would be good to separate Plot out into a different package. I
guess it does make sense for every package to provide only one module to
the user, even though it may internally be organized as multiple modules.

Currently, JLD is part of the HDF5.jl package. So

using HDF5
using JLD

works, but just using JLD doesn't. Similarly, both IProfile and SProfile are in the Profile.jl package; you can't currently say using SProfile directly, it's got to be

require("Profile")
using SProfile

In this case, writing using Profile rather than require("Profile") fails, because there is (currently) no Profile module.

I'm wondering whether these are sufficient motivation to split the packages (I see arguments both ways).

[aviks]

#2023

[ViralBShah]

Thanks @aviks

[rennis250]

I understand your intentions, @aviks, and agree that it is important for a user to know the maintainer and location of the relevant package. However, as far as I can tell, in the majority of cases, the maintainer of a Julia package is also the main developer, so my only concern is that people would confuse me for the main developer. Maybe I could put together some Javascript that would expand/contract an authors section when moused-over/clicked? And, in this list we could have small avatars for each maintainer/developer?

I do like the idea of a package metadata file. With this, a function like Pkg.info() could be made that would provide a user with a quick idea of what a package does. At the moment, I often have to go off to METADATA.jl, find the author, go to their Github page, and find the right repo to see what a package does and decide whether I want to install it or not. That, or I could install it and hunt down the README. (I get the feeling that the average user isn't going to be too happy with either of these approaches.) Pkg.info() could make this much easier, especially given the non-descriptive names of some of the packages.

--Rob

[StefanKarpinski]

We could also move packages to live under o-jasper's repo instead.

On Wed, Jan 16, 2013 at 7:48 PM, Robert Ennis notifications@github.comwrote:

I understand your intentions, @aviks https://github.com/aviks, and
agree that it is important for a user to know the maintainer and location
of the relevant package. However, as far as I can tell, in the majority of
cases, the maintainer of a Julia package is also the main developer, so my
only concern is that people would confuse me for the main developer. Maybe
I could put together some Javascript that would expand/contract an authors
section when moused-over/clicked? And, in this list we could have small
avatars for each maintainer/developer?

I do like the idea of a package metadata file. With this, a function like
Pkg.info() could be made that would provide a user with a quick idea of
what a package does. At the moment, I often have to go off to METADATA.jl,
find the author, go to their Github page, and find the right repo to see
what a package does and decide whether I want to install it or not. That,
or I could install it and hunt down the README. (I get the feeling that the
average user isn't going to be too happy with either of these approaches.)
Pkg.info() could make this much easier, especially given the
non-descriptive names of some of the packages.

--Rob

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1935#issuecomment-12349065.

[rennis250]

I brought the subject up to him recently, but he's been away from email for a bit. I figured I would wait for his response, before he suddenly found some new repos in his possession, combined with me losing push privileges for some time. However, maybe I'm misunderstanding the effects of a transfer. Would I still retain push privileges, even though the repos have been transferred to his account? If I do retain push privileges, then I would just do the transfer now.

NOTE: I can't seem to find any info on whether I retain push rights after a transfer or not.

--Rob

[StefanKarpinski]

I don't think you can transfer between people, actually. He'd have to fork your repo and then you'd have to delete yours. Then we could update the METADATA. We should, however, also make it possible for there to be a list of authors.

[aviks]

Yes I'll have a revised listing page with all contributors soon.

[rennis250]

Ah, I wasn't sure, since the transfer form allows me to type in a user's name. Anyway, I'll figure it out with @o-jasper once we're back in touch. Thanks, @StefanKarpinski!

Glad to hear about the authors list. Thanks, @aviks!

Also, I didn't even think that a Pkg.info() function could be put together by grabbing info off of the current package list page. I might see if I can throw something together soon.

--Rob

[diegozea]

+1 to list of authors

A cite section on package info and a function like R's citation [ http://stat.ethz.ch/R-manual/R-patched/library/utils/html/citation.html ] inside Julia would be great too ;)

[aviks]

@rennis250 Your repositories do not seem to return any contributor data.

See: https://api.github.com/repos/rennis250/OpenGL.jl/contributors
vs
https://api.github.com/repos/aviks/JSON.jl/contributors

[Keno]

That's because @rennis250 uses a different email for his commits and his Github account, so the two are not linked.

[rennis250]

Ah, sorry, I'll fix that. When I had created my account, I was paranoid about spammer bots getting a hold of my email, but I can deal with that now.

BTW, I like the look of it, @aviks!

--Rob

[rennis250]

Alright, email is fixed and my username now shows as a link next to some commits. I can do a few dummy commits, if necessary.

[StefanKarpinski]

I wouldn't bother, it will fix itself.

[rennis250]

Thanks, @StefanKarpinski and @loladiro, it sorted itself out. Looks like everything is working for my account now, @aviks.

--Rob