Add 552 packages from emacsmirror (discussion)

[BenediktBroich]

Hello,

so i was trying to get some packages to melpa and found this emacsmirror Package List (552). The list ist described as:
"Many mirrored packages are actually missing from Melpa. Here you can find some lists of such packages, in case you would like to help making Melpa more complete."

I had the following idea:
I could write script that forks the given repository's, add a pull request for a workflow like melpazoid to these repos and automate the recipe commits to melpa.

So i wrote it: melpify

I can add these packages one by one and fix there issues, but i can also just add all the recipes so you can join.

What do you think about this? Any ideas?

Vote:
Just Do It! 👍
Stop it. Get some help. 👎

[milkypostman]

Thank you for sending this issue before sending the PRs. I am a 👎🏼 on this simply because I want to avoid cruft. I mean, there are other reasons too (see the PR checklist we have).

Probably at some point we need to look at the packages we have and say "If a package hasn't been downloaded in N years, let's get rid of it". But that's kinda the same thing here, where I think it's worth someone actively wanting a package as motivation for adding it. Because ultimately there is a maintenance cost to having many packages in MELPA.

I don't think you need help, this is a well-intentioned and appreciated approach to MELPA but I don't think it's healthy for the project.

I will leave this open for discussion incase someone else wants to chime in. Thanks again.

[Boruch-Baum]

I just took a small peek at the list (ie emacsmirror Package List (552 and can immediately report that it's not accurate. It reports package w3m as not being in melpa due to that project's choice of version control; however, the package is in melpa, and did switch to git version control several years ago.

So, it's a good thing you didn't blindly dive in and devote effort solely based upon the list. The list seems to have been created fairly recently, by @tarsius, so I'm mentioning him so he gets this as a heads-up.

[tarsius]

@Boruch-Baum Well I wouldn't call it "fairly recent" but "surprisingly outdated". Now that I have heard from someone actually using it, I will try to update it more often or maybe even automate it. Even so it might at times be a week or two behind.

By the way you looked at another list from the same page as the list Benedikt worked with. I have removed that list a while ago (but not published the html until now) because it indeed is not accurate/relevant anymore.

@BenediktBroich A more recent (though potentially also slightly outdated) version can be found at https://github.com/emacsmirror/epkg-reports.

The code-block used to generate that list comes with a second variant that is commented out because I considered it to be too slow. It seems fast enough to me now, so I just switched to that variant. Have another look. The list is sorted by upstream url but it might be more useful to sort by last-modified date. Changing the order should be easy if you use the org file instead of the html export. [Edit: Just changed to order by last-modified date by default.]

[tarsius]

Probably at some point we need to look at the packages we have and say "If a package hasn't been downloaded in N years, let's get rid of it"

@milkypostman Interesting. I wouldn't go that far but am happy to hear that you are open to removing some cruft. But we should probably discuss criteria for actually removing packages in a separate issue.

With size come new issues such as reduced discoverability, and "removing cruft" is one way to deal with that. Another would be to categorize available packages better and that is relevant here because @BenediktBroich is motivated to help Melpa and the the wider community I assume, and might be interested to work on that instead/also.

But back to "adding the missing packages". The list I published is (and was for a long time) preceded by this comment:

The below packages, which are available from the Emacsmirror, are missing from Melpa. Not all of those packages should be added to Melpa, and if you decide to submit recipes for some of them, then please first make sure they meet the minimal requirements.

I have now added "(in fact most of them shouldn't)" and also a link to this conversation.

Also note that this list excludes any packages that are available from GNU Elpa (but I haven't gotten around to dealing with NonGNU Elpa yet).

I don't think you need help, this is a well-intentioned and appreciated approach to MELPA but I don't think it's healthy for the project.

I agree that adding all/most of these packages would be a mistake, but IMO adding a few dozen of them might be a positive.

@BenediktBroich I would recommend you only look at recently modified packages that you find interesting and pass the initial smell test. For those packages you could then proceed like this:

1. Search Melpa for past issues about that package. Maybe it was already removed or rejected for some reason.
2. Use @riscy's review tools and open issues and/or pull-requests in the upstream repository to get the issues fixed.
3. If and only if upstream is happy about these contributions and only after all or nearly all of them have been addressed, then proceed to opening a pull-request to add the package to Melpa. Even then @riscy will have to invest some time doing another review, but he will surely appreciate it if all obvious issues have already been taken care of.

Doing (2) is useful even for packages that don't get added to Melpa, including for packages that haven't been touched in a few years, but keep in mind that maintainers of stale packages might be less enthusiastic about cleanup pull-requests than those still actively working on their packages.

Looking at the recently modified packages I see a few that should not be added. For example: Mu4e is certainly a high quality package but it depends on a binary and that and the elisp are best installed together using the system package manager or manually. Emir is used to maintain the Emacsmirror, so naturally it is mirrored there, but I am its only user so it should not be included in Melpa. jcs090218 (jcs-elpa) has written a dozen or more packages to do fuzzy matching and we should not add them all to Melpa as I expect most of them are going to end up not being actively maintained. There is no package named Dired-Hacks on Melpa, but that is because the Emacsmirror and Melpa deal differently with repositories that contain multiple packages; Melpa distributes each separately, while the Emacsmirror mirrors the repository as a whole. See "Shared repositories" for more such packages.

[BenediktBroich]

Thanks for all you reply's.

@tarsius The epkg-reports is perfect for finding new packages. Next to the modified date I think about adding Github stars and forks to better decide what project would be good to add to melpa. And i will create a checklist with all the points to check and where to check them.

I would be good to have a list of deleted packages, as well as duplicates (function wise) and not responding, or no longer mainained. @milkypostman @Boruch-Baum Do you have a list of packages that were already deleted from MELPA? Is there a way to organize this.

[tarsius]

I would be good to have a list of deleted packages

git log  --pretty=format: --name-only --diff-filter=D recipes | sort -u

(This also includes packages that have since been added back.)

as well as duplicates (function wise)

For that someone has to go through all packages, spending a few minutes to analyze and categorize each, while improving the categories in the process. It would be nice to have the result of that available but someone has to do it first. Of course one could start small by only categorizing a subset of packages or using only basic categories such as "major-mode for a specific programming language" / "additional functionality for a certain programming language" > "the language", "general purpose text editing utility", "client for a certain service" > "the service"; while allowing for some level of mis-categorization. Are you volunteering? 😄 Of course it would be best if multiple people worked on that and if they repeatedly exchanged notes to improve how packages are classified and go back to the already classified packages where necessary...

[Boruch-Baum]

On 2022-02-08 12:59, Benedikt Broich wrote: ... ***@***.*** ***@***.*** Do you have a list of packages that were already deleted from MELPA? Is there a way to organize this.

Not me. This is all beyond my expertise.

…

-- hkp://keys.gnupg.net CA45 09B5 5351 7C11 A9D1 7286 0036 9E45 1595 8BC0

[BenediktBroich]

I am working on it branch.

[tarsius]

So you are adding the ability to browse by keyword? That seems like a useful feature (which already exists in Emacs).

If you also use this to perform the categorization, then there are better tools (Emacs).

[tarsius]

In summary, some of these package should be added, but to avoid overwhelming the reviewer, the person who opens the prs (one per package), should try to resolve any issues from the checklist. Most of these packages should not be added.