New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove all Emacswiki packages #5008

Merged
merged 2 commits into from Jan 25, 2018

Conversation

Projects
None yet
@tarsius
Member

tarsius commented Sep 15, 2017

Update 2018-01-24 - The Finale

Removal of Emacswiki packages has long been requested and planned, yet we've all hung on because the convenience of useful code that's insecurely managed and distributed has consistently exceeded the motivation to mitigate the risks: prominent Emacswiki authors have not taken action, while library and starter kit authors and end users (myself included) have cheerfully continued to depend on Emacswiki-sourced packages.

MELPA stopped building/updating all these packages a while ago, with the intention of removing them once we had a plan to minimise disruption. In the meantime, we continued to serve up our last-built versions. We'd have preferred to publicly sunset our continued hosting of them, but having accidentally deleted those historic (probably-outdated) packages yesterday, we now won't be restoring them. We apologise for the disruption that the sudden removal has caused (and will continue to cause), but this would likely have been the case even if we'd announced a sunset date weeks in advance, and we hope that you'll understand our decision.

In hindsight, MELPA should probably never have distributed Emacswiki-sourced packages. When MELPA was starting off, incorporating those packages helped MELPA's growth, to both the community's benefit and its cost. Thanks to everyone who pushed us to resolve this!

Now that we all know better, let's get on with fixing things and making the package ecosystem better and safer. If Emacswiki authors move their code to a MELPA-supported SCM, we can re-add those packages to MELPA. Some authors won't be willing to do that, so the community will have to decide how much it values that code, and how to get it distributed.

Whatever steps are necessary along the way, we - the MELPA maintainers - will be happy to advise.

-@purcell


Update 2018-01-24

All packages were accidentally deleted, including non-Emacswiki packages. Those are in the process of being restored, but for the Emacswiki packages that is not currently possible. That probably means that we make this final soon, but still waiting for Steve to weight in. The discussion about this begins further down.



Update 2017-09-16

Status Packages from the Emacswiki are still available from Melpa, but they are no longer being updates, because melpa/package-build#9 has been merged into Melpa.

TODO Here is a list of ongoing efforts to get packages of the Emacswiki:

  • #5034 Getting Drew to commit to (a) Git repository/-ies Unlikely, we've tried for a decade.
  • #5020 Moving rubikitch's packages to github No response so far. Sent an email.
  • #2342 Deprecate all emacswiki packages. Mostly replaced by this issue, except:
  • #2342 (comment) asks github-using maintainers who have packages on the Emacsmirror to migrate. That issue has gotten unwieldy, so I will probably replace it with one or more new issues and start pinging.

Original issue text

Since everyone here (#2342) agrees that the Emacswiki packages should be removed because they pose a huge security risk, let's just do it. We pretty much decided to do this years ago and we tried to get maintainers to move their packages. Waiting a few more years won't make any difference except that it leaves the community at risk.

@purcell

This comment has been minimized.

Member

purcell commented Sep 15, 2017

I'm strongly in favour of this, and it certainly makes sense to just bite the bullet and do it. However, it will dramatically impact the day-to-day experience of many people who rely on certain very popular packages that are not available elsewhere. So I think we need to publicise this ahead of time and give a timeline. It would also be really helpful if @kensanata would be willing to end-of-life the hosting of code on Emacswiki on a compatible timeline - is that something you'd consider, Alex?

@purcell

This comment has been minimized.

Member

purcell commented Sep 15, 2017

Additionally, this will presumably cause a bunch of other packages to become uninstallable, as a result of depending on packages in this list.

@purcell

This comment has been minimized.

Member

purcell commented Sep 15, 2017

It would also be really helpful if @kensanata would be willing to end-of-life the hosting of code on Emacswiki on a compatible timeline - is that something you'd consider, Alex?

A half-way step to this would be to block the "raw" view of files on the emacswiki, so that files could be viewed in an html wrapper, but not easily downloaded in their original form.

@jgkamat

This comment has been minimized.

Contributor

jgkamat commented Sep 16, 2017

While this is a positive step for security, it's really annoying to suddenly find that your dotfiles no longer properly work on a new computer (arguably when you need them to work the most, since you're busy setting other stuff up).

Would it be possible to freeze and copy the emacswiki packages to github or similar git hosting, instead of deleting them entirely? I guess that would raise the issue of who would maintain the packages then...

@tarsius

This comment has been minimized.

Member

tarsius commented Sep 16, 2017

this will presumably cause a bunch of other packages to become uninstallable, as a result of depending on packages in this list.

We have already frozen those: #2342 (comment) ff. Maybe there are new ones, I'll check.

it will dramatically impact the day-to-day experience of many people who rely on certain very popular packages that are not available elsewhere.

Could you give me a list of the download counts? We could freeze the most popular ones like we have done for the ones being depended on by non-wiki packages.

@technomancy

This comment has been minimized.

technomancy commented Sep 16, 2017

While this is a positive step for security, it's really annoying to suddenly find that your dotfiles no longer properly work on a new computer

If your dotfiles depend on software downloaded from a wiki, they're already broken. Better to be broken in a way that is obvious than one that is hidden.

If this doesn't get merged what's to stop someone from just deleting them from the emacswiki directly instead?

@purcell

This comment has been minimized.

Member

purcell commented Sep 16, 2017

Could you give me a list of the download counts? We could freeze the most popular ones like we have done for the ones being depended on by non-wiki packages.

Is this sufficient? - https://melpa.org/download_counts.json

@tarsius

This comment has been minimized.

Member

tarsius commented Sep 16, 2017

Is this sufficient?

Yes. I forgot that exists - it was late 😉

@tarsius

This comment has been minimized.

Member

tarsius commented Sep 16, 2017

I think we should immediately stop updating any packages from the wiki, which can be easily done by editing package-build-checkout. That way existing malware, if any, won't disappear, but at least no new ("let's do it before the hole gets fixed") attacks would be possible.

tarsius added a commit that referenced this pull request Sep 16, 2017

Remove all Emacswiki packages
The Emacswiki is completely unsuitable as a source of automatically
generated and widely distributed packages.  Anyone can edit any
package on the Emacswiki without even having to create an account.

This is a huge security risk for the whole community.  And addressing
that is much more important than to avoid inconveniencing some users
by removing their favorite packages.

Also see #5008.
@tarsius

This comment has been minimized.

Member

tarsius commented Sep 16, 2017

I have updated the statistics at https://emacsmirror.net/stats/emacswiki.html.

@dgutov

This comment has been minimized.

Contributor

dgutov commented Sep 16, 2017

yaoddmuse |   | company

This line is not very accurate at least: yaoddmuse is a soft dependency, company works just fine without it.

@tarsius

This comment has been minimized.

Member

tarsius commented Sep 16, 2017

This line is not very accurate at least: yaoddmuse is a soft dependency, company works just fine without it.

I am aware of that. From #2342 (comment):

It does not include any indirect dependers and it is based on automatically extracted dependency information, not the Package-Requires header.

Despite these limitations I still think this information is useful.

@tarsius

This comment has been minimized.

Member

tarsius commented Sep 16, 2017

I have updated the table. Soft dependencies are now marked as such.

@aplaice

This comment has been minimized.

Contributor

aplaice commented Sep 16, 2017

Looking at the dependencies as seen by MELPA (using https://melpa.org/recipes.json and https://melpa.org/archive.json), gives a similar picture (this obviously does not include soft dependencies):

dependee depender
filesets+ helm-filesets
font-lock+ all-the-icons
goto-chg evil
header2 org-readme
help-fns+ nu-mode
hexrgb paper-theme
highlight cider-eval-sexp-fu
highlight eval-sexp-fu
highlight evil-search-highlight-persist
highlight floobits
highlight nrepl-eval-sexp-fu
highlight php-boris-minor-mode
highlight sonic-pi
lib-requires org-readme
thingatpt+ el-spice
yaoddmuse org-readme

The greatest (and by far the most important) discrepancy is that evil (!) depends on goto-chg.

The only other differences are that org-readme depends on several other wiki packages, though I'm not sure whether they're really hard dependencies and, according to MELPA, jabber does not have a hard depency on hexrgb (indeed, installing jabber does not install hexrgb).

This obviously does not list all indirect dependencies (which would include everything that depends on evil.

@syl20bnr

This comment has been minimized.

Contributor

syl20bnr commented Sep 16, 2017

Opened an issue on the evil repo for the goto-chg problem.
emacs-evil/evil#922

@wasamasa

This comment has been minimized.

Contributor

wasamasa commented Sep 16, 2017

goto-chg is particularly embarassing as I've found the code depending on it to be broken in the presence of the only other dependency, undo-tree. This fork works with one or two modifications so it would need to be added to our organization to make it a replacement package.

@purcell Would you accept if I forked that package under the emacs-evil organization, modified it so that it works correctly with Evil and replaced the existing MELPA package with it? This would obviously require that it stays a drop-in replacement that works correctly both with and without undo-tree, but I'm sure that can be arranged.

@tarsius

This comment has been minimized.

Member

tarsius commented Sep 16, 2017

The greatest (and by far the most important) discrepancy is that evil (!) depends on goto-chg.

That's because the evil repository contains a copy of goto-chg.el. In such cases my tools don't look for another package that also provides it.

@purcell Would you accept if I forked that package under the emacs-evil organization, modified it so that it works correctly with Evil and replaced the existing MELPA package with it?

I would very much welcome that. Actually I think that I have proposed that to the previous maintainer (or maybe I just intended to do it, or only talked about it here).

@purcell

This comment has been minimized.

Member

purcell commented Sep 17, 2017

@purcell Would you accept if I forked that package under the emacs-evil organization, modified it so that it works correctly with Evil and replaced the existing MELPA package with it?

Yes, definitely.

@wasamasa wasamasa referenced this pull request Sep 17, 2017

Merged

Pull goto-chg from GitHub #5012

3 of 6 tasks complete
@wasamasa

This comment has been minimized.

Contributor

wasamasa commented Sep 18, 2017

OK, done.

@tarsius Don't forget removing goto-chg from this PR to resolve the conflict.

tarsius added a commit that referenced this pull request Sep 18, 2017

Remove all Emacswiki packages
The Emacswiki is completely unsuitable as a source of automatically
generated and widely distributed packages.  Anyone can edit any
package on the Emacswiki without even having to create an account.

This is a huge security risk for the whole community.  And addressing
that is much more important than to avoid inconveniencing some users
by removing their favorite packages.

Also see #5008.

tarsius added a commit that referenced this pull request Sep 19, 2017

Remove all Emacswiki packages
The Emacswiki is completely unsuitable as a source of automatically
generated and widely distributed packages.  Anyone can edit any
package on the Emacswiki without even having to create an account.

This is a huge security risk for the whole community.  And addressing
that is much more important than to avoid inconveniencing some users
by removing their favorite packages.

Also see #5008.
@kensanata

This comment has been minimized.

kensanata commented Sep 22, 2017

@tarsius

This comment has been minimized.

Member

tarsius commented Sep 25, 2017

Since this is trending on r/emacs...

I think we should immediately stop updating any packages from the wiki

I have implemented that in melpa/package-build#9, but that still has to be merge into Melpa. I think it is a good solution in the short run.

@tekktonic

This comment has been minimized.

tekktonic commented Sep 25, 2017

@kensantana the issue isn't that it's hard to maintain but that there is no verifiability and it's therefore a security risk. I can go onto emacswiki and modify a common package to recursively delete $HOME or worse and anyone unlucky enough to pull that package before it's fixed is screwed

EDIT: Oops, missed some nuance here; this is what I get for reading github issues on my phone on the bus home :)

@tarsius

This comment has been minimized.

Member

tarsius commented Sep 25, 2017

@tekktonic: He (@kensanata) already acknowledged this risk. He went on making a distinction between the risk of making packages with originate from the wiki available through a package manager (i.e. allowing to "pull" them (without being aware of the original source)) and the benefits of making code available on a wiki (implying that this is not as risky because users know what's going on).

I think we should let that side-issue rest for the time being and concentrate on getting the Melpa issue resolved first.

@mattiasb

This comment has been minimized.

Contributor

mattiasb commented Jan 26, 2018

all-the-icons removed its dependency on font-lock+ yesterday fwiw.

tarsius added a commit that referenced this pull request Jan 26, 2018

Remove packages whose upstream repositories have disappeared
Like for the packages from the Emacswiki we can not rebuild these
packages after having deleted them accidentally.  In this case it
is not because the code to do so no longer exists, but because the
data no longer exists.

Re #4622.  Also see #5008.

vedang added a commit to vedang/el-spice that referenced this pull request Jan 28, 2018

Replace dependence on `thingatpt+' with `thingatpt'
MELPA has removed support for EmacsWiki packages.

See: melpa/melpa#5008
@vedang

This comment has been minimized.

Contributor

vedang commented Jan 28, 2018

I have updated el-spice to remove the dependency on thingatpt+. @tarsius : I can see the el-spice recipe in the melpa recipies folder, so I'm not sure there's anything else to do at the moment. Please let me know if I've missed anything.

@tarsius

This comment has been minimized.

Member

tarsius commented Jan 28, 2018

@mattiasb @vedang Thanks!

@vedang Nothing else should be required.

tarsius added a commit that referenced this pull request Feb 5, 2018

Remove all packages that depend on packages from the Emacswiki
These commits were removed in the parent commit (which see).

Only remove packages that hard `require' a package from the Emacswiki
at the top-level and that specify the dependency using the
Package-Requires header keyword.  There are some other packages that
soft require such a package and/or only require it when some command
is called and/or when some non-essential library is loaded, and that
do not specify a dependency using Package-Requires.

Also see #5008.

scoiatael added a commit to scoiatael/dotfiles that referenced this pull request Feb 10, 2018

Ambrevar added a commit to Ambrevar/melpa that referenced this pull request Feb 13, 2018

Remove all Emacswiki packages
The Emacswiki is completely unsuitable as a source of automatically
generated and widely distributed packages.  Anyone can edit any
package on the Emacswiki without even having to create an account.

This is a huge security risk for the whole community.  And addressing
that is much more important than to avoid inconveniencing some users
by removing their favorite packages.

Also see melpa#5008.

Ambrevar added a commit to Ambrevar/melpa that referenced this pull request Feb 13, 2018

Remove packages whose upstream repositories have disappeared
Like for the packages from the Emacswiki we can not rebuild these
packages after having deleted them accidentally.  In this case it
is not because the code to do so no longer exists, but because the
data no longer exists.

Re melpa#4622.  Also see melpa#5008.

jmquigley added a commit to jmquigley/emacs that referenced this pull request Feb 20, 2018

Updates package list based on new emacs wiki package exclusion
The configuration contains packages that are no longer supported by melpa.

melpa/melpa#5008

This included bookmarks+, hl-line+, redo+, and tidy.  I have copied those dependencies into the 3rd-party directory for now.  I will look for melpa replacements later.  This was a stop gap to fix the build problem I found today.

waymondo added a commit to waymondo/melpa that referenced this pull request Feb 20, 2018

Remove all Emacswiki packages
The Emacswiki is completely unsuitable as a source of automatically
generated and widely distributed packages.  Anyone can edit any
package on the Emacswiki without even having to create an account.

This is a huge security risk for the whole community.  And addressing
that is much more important than to avoid inconveniencing some users
by removing their favorite packages.

Also see melpa#5008.

waymondo added a commit to waymondo/melpa that referenced this pull request Feb 20, 2018

Remove packages whose upstream repositories have disappeared
Like for the packages from the Emacswiki we can not rebuild these
packages after having deleted them accidentally.  In this case it
is not because the code to do so no longer exists, but because the
data no longer exists.

Re melpa#4622.  Also see melpa#5008.

pjozog added a commit to pjozog/settings that referenced this pull request Feb 25, 2018

Updates to melpa required removing/manually downloading packages
See melpa/melpa#5008.

Also color-themes seems to be broken; as of this commit you need to

$ mkdir .emacs.d/elpa/color-theme-20070910.1007/themes

Otherwise startup will complain that directory 'themes' does not exist.

pjozog added a commit to pjozog/settings that referenced this pull request Feb 25, 2018

Updates to melpa required removing/manually downloading packages
See melpa/melpa#5008.

Also color-themes seems to be broken; as of this commit you need to

$ mkdir .emacs.d/elpa/color-theme-20070910.1007/themes

Otherwise startup will complain that directory 'themes' does not exist.

shackra added a commit to shackra/emacs that referenced this pull request Feb 25, 2018

chg: Dired+ ya no se carga desde Elpa
Esto debido a los ajustes en Melpa sobre archivos inseguros bajados desde la Wiki de Emacs

melpa/melpa#5008

@tarsius tarsius referenced this pull request Feb 27, 2018

Merged

Re-create backup-each-save #5335

5 of 6 tasks complete
@ninrod

This comment has been minimized.

Contributor

ninrod commented Mar 5, 2018

Gentleman I've just fixed the dependencies of exato so it only depends on evil now. All should be fine.

sufyanadam pushed a commit to sufyanadam/.emacs.d that referenced this pull request Mar 27, 2018

listx added a commit to listx/syscfg that referenced this pull request Apr 4, 2018

emacs: remove dependency on zoom-frm
zoom-frm was removed from MELPA in late 2017 (see
melpa/melpa#5008).

listx added a commit to listx/syscfg that referenced this pull request May 25, 2018

emacs: remove dependency on zoom-frm
zoom-frm was removed from MELPA in late 2017 (see
melpa/melpa#5008).

aptrik added a commit to aptrik/.emacs.d that referenced this pull request May 29, 2018

Removed old EmacsWiki packages
EmacsWiki packages are not supported by melpa anymore.
See melpa/melpa#5008

@purcell purcell referenced this pull request Nov 11, 2018

Closed

Add recipe for column-marker #5803

6 of 6 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment