Remove all Emacswiki packages

[tarsius]

Since everyone here (#2342) agrees that the Emacswiki packages should be removed because they pose a huge security risk, let's just do it. We pretty much decided to do this years ago and we tried to get maintainers to move their packages. Waiting a few more years won't make any difference except that it leaves the community at risk.

---

Update

Status Packages from the Emacswiki are still available from Melpa, but they are no longer being updates, because melpa/package-build#9 has been merged into Melpa.

TODO Here is a list of ongoing efforts to get packages of the Emacswiki:

• #5034 Getting Drew to commit to (a) Git repository/-ies Needs a volunteer
• #5020 Moving rubikitch's packages to github
• #2342 Deprecate all emacswiki packages. Mostly replaced by this issue, except:
• #2342 (comment) asks github-using maintainers who have packages on the Emacsmirror to migrate. That issue has gotten unwieldy, so I will probably replace it with one or more new issues and start pinging.

You might want to just skip ahead to #5008 (comment). On the other hand, if you want the full context, then read the stuff inbetween - as well as all the other issues labeled emacswiki ;-)

[purcell]

I'm strongly in favour of this, and it certainly makes sense to just bite the bullet and do it. However, it will dramatically impact the day-to-day experience of many people who rely on certain very popular packages that are not available elsewhere. So I think we need to publicise this ahead of time and give a timeline. It would also be really helpful if @kensanata would be willing to end-of-life the hosting of code on Emacswiki on a compatible timeline - is that something you'd consider, Alex?

[purcell]

Additionally, this will presumably cause a bunch of other packages to become uninstallable, as a result of depending on packages in this list.

[purcell]

It would also be really helpful if @kensanata would be willing to end-of-life the hosting of code on Emacswiki on a compatible timeline - is that something you'd consider, Alex?

A half-way step to this would be to block the "raw" view of files on the emacswiki, so that files could be viewed in an html wrapper, but not easily downloaded in their original form.

[jgkamat]

While this is a positive step for security, it's really annoying to suddenly find that your dotfiles no longer properly work on a new computer (arguably when you need them to work the most, since you're busy setting other stuff up).

Would it be possible to freeze and copy the emacswiki packages to github or similar git hosting, instead of deleting them entirely? I guess that would raise the issue of who would maintain the packages then...

[tarsius]

this will presumably cause a bunch of other packages to become uninstallable, as a result of depending on packages in this list.

We have already frozen those: #2342 (comment) ff. Maybe there are new ones, I'll check.

it will dramatically impact the day-to-day experience of many people who rely on certain very popular packages that are not available elsewhere.

Could you give me a list of the download counts? We could freeze the most popular ones like we have done for the ones being depended on by non-wiki packages.

[technomancy]

While this is a positive step for security, it's really annoying to suddenly find that your dotfiles no longer properly work on a new computer

If your dotfiles depend on software downloaded from a wiki, they're already broken. Better to be broken in a way that is obvious than one that is hidden.

If this doesn't get merged what's to stop someone from just deleting them from the emacswiki directly instead?

[purcell]

Could you give me a list of the download counts? We could freeze the most popular ones like we have done for the ones being depended on by non-wiki packages.

Is this sufficient? - https://melpa.org/download_counts.json

[tarsius]

Is this sufficient?

Yes. I forgot that exists - it was late 😉

[tarsius]

I think we should immediately stop updating any packages from the wiki, which can be easily done by editing package-build-checkout. That way existing malware, if any, won't disappear, but at least no new ("let's do it before the hole gets fixed") attacks would be possible.

[tarsius]

I have updated the statistics at https://emacsmirror.net/stats/emacswiki.html.

[dgutov]

yaoddmuse |   | company

This line is not very accurate at least: yaoddmuse is a soft dependency, company works just fine without it.

[tarsius]

This line is not very accurate at least: yaoddmuse is a soft dependency, company works just fine without it.

I am aware of that. From #2342 (comment):

It does not include any indirect dependers and it is based on automatically extracted dependency information, not the Package-Requires header.

Despite these limitations I still think this information is useful.

[tarsius]

I have updated the table. Soft dependencies are now marked as such.

[aplaice]

Looking at the dependencies as seen by MELPA (using https://melpa.org/recipes.json and https://melpa.org/archive.json), gives a similar picture (this obviously does not include soft dependencies):

dependee	depender
filesets+	helm-filesets
font-lock+	all-the-icons
goto-chg	evil
header2	org-readme
help-fns+	nu-mode
hexrgb	paper-theme
highlight	cider-eval-sexp-fu
highlight	eval-sexp-fu
highlight	evil-search-highlight-persist
highlight	floobits
highlight	nrepl-eval-sexp-fu
highlight	php-boris-minor-mode
highlight	sonic-pi
lib-requires	org-readme
thingatpt+	el-spice
yaoddmuse	org-readme

The greatest (and by far the most important) discrepancy is that evil (!) depends on goto-chg.

The only other differences are that org-readme depends on several other wiki packages, though I'm not sure whether they're really hard dependencies and, according to MELPA, jabber does not have a hard depency on hexrgb (indeed, installing jabber does not install hexrgb).

This obviously does not list all indirect dependencies (which would include everything that depends on evil.

[syl20bnr]

Opened an issue on the evil repo for the goto-chg problem.
emacs-evil/evil#922

[wasamasa]

goto-chg is particularly embarassing as I've found the code depending on it to be broken in the presence of the only other dependency, undo-tree. This fork works with one or two modifications so it would need to be added to our organization to make it a replacement package.

@purcell Would you accept if I forked that package under the emacs-evil organization, modified it so that it works correctly with Evil and replaced the existing MELPA package with it? This would obviously require that it stays a drop-in replacement that works correctly both with and without undo-tree, but I'm sure that can be arranged.

[tarsius]

The greatest (and by far the most important) discrepancy is that evil (!) depends on goto-chg.

That's because the evil repository contains a copy of goto-chg.el. In such cases my tools don't look for another package that also provides it.

@purcell Would you accept if I forked that package under the emacs-evil organization, modified it so that it works correctly with Evil and replaced the existing MELPA package with it?

I would very much welcome that. Actually I think that I have proposed that to the previous maintainer (or maybe I just intended to do it, or only talked about it here).

[purcell]

@purcell Would you accept if I forked that package under the emacs-evil organization, modified it so that it works correctly with Evil and replaced the existing MELPA package with it?

Yes, definitely.

[wasamasa]

OK, done.

@tarsius Don't forget removing goto-chg from this PR to resolve the conflict.

[kensanata]

I think the hosting of code on Emacs Wiki is a small but positive feature. Distributing this code using a package manager might be something that requires an extra effort you are not willing to spend (and neither am I) but removing this feature would result in a net negative for me and that is why I don't think that is a good idea.

[tarsius]

Since this is trending on r/emacs...

I think we should immediately stop updating any packages from the wiki

I have implemented that in melpa/package-build#9, but that still has to be merge into Melpa. I think it is a good solution in the short run.

[tekktonic]

@kensantana the issue isn't that it's hard to maintain but that there is no verifiability and it's therefore a security risk. I can go onto emacswiki and modify a common package to recursively delete $HOME or worse and anyone unlucky enough to pull that package before it's fixed is screwed

EDIT: Oops, missed some nuance here; this is what I get for reading github issues on my phone on the bus home :)

[tarsius]

@tekktonic: He (@kensanata) already acknowledged this risk. He went on making a distinction between the risk of making packages with originate from the wiki available through a package manager (i.e. allowing to "pull" them (without being aware of the original source)) and the benefits of making code available on a wiki (implying that this is not as risky because users know what's going on).

I think we should let that side-issue rest for the time being and concentrate on getting the Melpa issue resolved first.

[kensanata]

Yeah, the "more work involved" I implied is something like keeping a hash of verified releases, and doing the actual verifying. It's something that a release manager would do. Whether the code author does it (which is what you would like to enforce—and which you implicitly trust all of them to do) or whether an independent release manager does it (which nobody is volunteering to do), or indeed relying on the end user to do it (basically this is what the wiki does) are simply different ways of solving the issue. As Jonas said, we all agreed that simply copying from the wiki and packaging the latest version doesn't do it. An additional organizational step is needed. Actual work by real people, sadly. :)

[purcell]

Alright, melpa/melpa#4802 is merged (which removes support for most of the non-git fetchers), and that means wiki is next in the firing line, ie. this PR.

@tarsius has submitted melpa/package-build#9, which will effectively prevent emacswiki packages from updating in future on MELPA. Merging it and using it for MELPA means that if we ever had to do a full rebuild from MELPA, the wiki packages would be dropped.

I think that's acceptable, and we can establish a timeline for removing them anyway and finally merging this PR. I might suggest three months? Perhaps it would be good to somehow highlight wiki packages more prominently on the web site and (ideally) in Emacs so that people will know to avoid/migrate them... ideas?

I've spent a bit of time figuring out if MELPA's overall build machinery will break when melpa/package-build#9 is incorporated, and I think we're safe, so I'll go ahead and apply those changes here. In any case, I modified our build script directly so that wiki packages will no longer be built (see b6b281c).

[purcell]

I've started the ball rolling by taking the only part of dired+ I use (extra font locking) and extracting it into a new package hosted on github.

[tarsius]

Thanks @purcell! I have summarized the current status in my initial post above.

[Fuco1]

This will destroy my dired config, but I am fully in favour. I think what @purcell did with the font lock is a great idea. I depend on some code from diredp so maybe I will just copy/extract the logic into dired-hacks or a separate package and swap the dependency to that.

Do we want to start tracking these "soft forks" of some of the wiki functionality so we can have a list for people who would want to migrate/fix their setups?

[etu]

Couldn't deps (like font-lock+) just be placed in a repository under /melpa until someone comes along and want to maintain it?

At least it could be used for keeping track of changes and bug reports. It doesn't matter if changes take alot of time to get through if you ask me.

[Fuco1]

@etu that would probably work, but it would require some review of the code to make sure there is no dangerous content in there---we can not assume that the packages are not already compromised if we do this mainly from the security point of view.

So if people start chopping the packages into smaller pieces there will be some sort of implicit review in that and at least some basic safety will be ensured. So it's mostly about the available resources I guess.

[etu]

So if I would create a github-organization for a emacs-wiki-packages-rescue project, and then copy all the code as initial commits without going through the code at all. And then ask people and myself to go through things to later submit to melpa after it's tested through and checked.

How would people feel about such project?

[tarsius]

How would people feel about such project?

I still think it would be better to write some tool for Drew that would allow him to upload to the wiki as before but which would at the same time push to package-specific git repository.

[etu]

@tarsius Oh... I had missed that that was an issue... What about something that scrapes emacs-wiki and makes pull requests in corresponding repositories automatically so we can have revisions and checks of what happens?

[tarsius]

What about something that scrapes emacs-wiki

The Emacsmirror already contains 678 such repositories. I update them every few weeks.

and makes pull requests in corresponding repositories automatically so we can have revisions and checks of what happens?

Do you want to perform these reviews for the next decade? I have been waiting for a decade for Drew to start using git. While maintaining tools specifically to deal with his packages (and five others or so). And I am done with it, by which I mean I will keep maintaining these tools, but I don't want to be involved in any efforts to write better tools - unless they solve the issue at the source, i.e. when Drew invokes the "upload" command.