Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

emacs-modes: replace melpa-generated with json format + updater in emacs lisp #59189

Merged
merged 1 commit into from Aug 2, 2019

Conversation

@bendlas
Copy link
Contributor

commented Apr 8, 2019

This approach has several differences with emacs2nix:

  • the updater uses a downloaded recipes.json and archive.json for commit information, it uses a local checkout only for hashing the recipes
  • the generated file is JSON
  • the updater is written in emacs lisp
  • prefetch errors are put into an error key in the JSON, for review + meta.broken attributes are generated from it

The updater re-uses the existing generated file to memoize prefetched content-sha256s for commits, thus prefetching should normally be quite fast. The main bottleneck seems to be the git log call to determine the latest commit id per recipe file.

I think having the updater written in emacs lisp increases the chances of getting it adopted by Nix' Emacs users, as well as possibilities to integrate more deeply with the melpa library.

To be clear, this is a review request. If people would rather stick with emacs2nix, I'd be happy to help with documenting / improving it.

cc

I want to get wide input from people interested in emacs in nixos, please excuse spam

#11503
#59153

@mdorman @phunehehe

TODO:

  • emulate old renames (e.g. 0blayout -> _0blayout)

  • handle invalid names (e.g. @)

  • add commitStable and sha256Stable and port melpa-stable-generated

  • add support for elpa and org

  • extract support code from melpa-packages.nix

  • Update instructions in melpa-packages.nix

  • Fix evaluation errors

  • Fix bitbucket sources

    if somebody could help track down the infinite recursion, that occurs when trying this .. <3

@matthewbauer

This comment has been minimized.

Copy link
Member

commented Apr 8, 2019

It would be good to collect some performance stats on this. Basically counting how long nix-instantiate emacsPackagesNg.magit and nix-env -qa -A emacsPackagesNg.

@bendlas

This comment has been minimized.

Copy link
Contributor Author

commented Apr 8, 2019

It would be good to collect some performance stats

I can get some, later. Right now I'm in the middle of a couple of rebuilds and don't wan't to GC. The PR should be in a perfectly usable state, if you want to take a look.

Subjectively, nix-instantiate performance seemed decent.

@adisbladis

This comment has been minimized.

Copy link
Member

commented Apr 9, 2019

This is awesome work! Thanks!
I'm all for simplifying generation of melpa packages. As it is now it's not easy to use emacs2nix successfully.

I'd like to get an opinion on this from @ttuegel before feeling comfortable about moving away from emacs2nix.

@adisbladis

This comment has been minimized.

Copy link
Member

commented Apr 9, 2019

@matthewbauer A completely unscientific test:

$ time nix-env -f '<nixpkgs>' -qa -A emacsPackagesNg

Master:

0.78user 0.08system 0:01.72elapsed 50%CPU (0avgtext+0avgdata 290204maxresident)k
0inputs+0outputs (0major+78099minor)pagefaults 0swaps

This PR:

0.69user 0.07system 0:01.38elapsed 55%CPU (0avgtext+0avgdata 279408maxresident)k
0inputs+0outputs (0major+73315minor)pagefaults 0swaps

$ time nix-instantiate '<nixpkgs>' -A emacsPackagesNg.magit

Master:

0.48user 0.05system 0:00.58elapsed 92%CPU (0avgtext+0avgdata 143172maxresident)k
0inputs+0outputs (0major+36232minor)pagefaults 0swaps

This PR:

0.38user 0.04system 0:00.47elapsed 90%CPU (0avgtext+0avgdata 122252maxresident)k
0inputs+0outputs (0major+29041minor)pagefaults 0swaps

While both of the results from this PR are faster it's imho a negligible difference.

@adisbladis adisbladis requested a review from etu Apr 9, 2019

@bendlas

This comment has been minimized.

Copy link
Contributor Author

commented Apr 9, 2019

While both of the results from this PR are faster it's imho a negligible difference.

Ha, seems like nix has a fast JSON reader built in ;-)

Any significant difference in performance would surprise me, since the computed nix expressions remained pretty much the same.

@bendlas

This comment has been minimized.

Copy link
Contributor Author

commented Apr 9, 2019

Here is a test script for timing nix-instantiate:

#!/bin/sh -ex

instantiate () {
    echo -n "Instantiated count "
    nix-instantiate -E "
      with import ./. { config = { allowBroken = true; }; };
      emacsPackagesNg.melpaPackages
    " 2>/dev/null | wc -l
}

(
    cd master
    echo "============================
          On master"
    nix-store --option keep-derivations false --gc 2>/dev/null
    time instantiate
)


(
    cd emacs-updater-elisp
    echo "============================
          On emacs-updater-elisp"
    nix-store --option keep-derivations false --gc 2>/dev/null
    time instantiate
)

the output on my machine is

============================================
          On master
12224 store paths deleted, 51.57 MiB freed
Instantiated count 4060

real    0m16.480s
user    0m4.346s
sys     0m0.331s
============================================
          On emacs-updater-elisp
12042 store paths deleted, 51.11 MiB freed
Instantiated count 4118

real    0m17.428s
user    0m4.518s
sys     0m0.363s

So it seems, we are a little bit slower, but times on my machine fluctuate heavily (15-25 seconds)
Differences in package count are due to this updater emitting broken packages as well.

@adisbladis

This comment has been minimized.

Copy link
Member

commented Apr 15, 2019

ping @ttuegel

@ttuegel

This comment has been minimized.

Copy link
Member

commented Apr 21, 2019

I'd like to get an opinion on this from @ttuegel before feeling comfortable about moving away from emacs2nix.

I am currently developing a new branch of emacs2nix with the aim of producing an overlay which updates automatically. I don't really like maintaining such a large set of generated packages in Nixpkgs itself. I also haven't been a very responsive maintainer (new baby, dissertation, new full-time job, upcoming new baby... excuses, excuses) so I'm not going to be offended if Nixpkgs moves away from emacs2nix. Right now the packages are not kept up-to-date, so I say: if this can distribute the work of package updates, and if it seems to work, then ship it!

@bendlas

This comment has been minimized.

Copy link
Contributor Author

commented Apr 30, 2019

@ttuegel thanks for your response. This PR isn't ready, yet, and as far as I'm concerned, it won't be before we can generate elpa and org packages as well. Now with your confirmation, I'll be glad to cherry-pick relevant commits into this PR.

I don't really like maintaining such a large set of generated packages in Nixpkgs itself.

This is a good point. The generator could have black-/whitelist capability, in order to maintain only a couple of necessary packages within nixpkgs and have the rest in overlays. Would this serve your purpose?

It sounds like you might still keep working on emacs2nix, if we were to make the switch ... would you consider adopting the .json format in that case?

@adisbladis

This comment has been minimized.

Copy link
Member

commented Apr 30, 2019

This PR isn't ready, yet, and as far as I'm concerned, it won't be before we can generate elpa and org packages as well.

Is this really true? In my experience the hard part about emacs packages is generating melpa. Org and elpa packages always worked perfectly with emacs2nix.
I would rather see this PR move ahead as-is and iterate on it later.

@bendlas

This comment has been minimized.

Copy link
Contributor Author

commented May 3, 2019

Is this really true? In my experience the hard part about emacs packages is generating melpa. Org and elpa packages always worked perfectly with emacs2nix.

This may be true, but it would introduce another split (emacsPackages, emacsPackagesNg, melpa). Also, as I already mentioned, the integration of the json format from the nix side is a horrible hack and needs to be cleaned up.

To be perfectly blunt: The issue for me is also about shared ownership. Replacing a solution where @ttuegel is the sole maintainer with a solution where I'm the sole maintainer makes little sense to me, neither from a community standpoint, nor from a standpoint of my own free time.

So either somebody is going to jump in to help push this along (if this means pushing it out as is, so be it, at least I'll not be the only one receiving tickets about it), or it will have to wait until I get it to a state where I'm happy about releasing it, myself.

@bendlas bendlas force-pushed the bendlas:emacs-updater-elisp branch from f4d7183 to 14c37aa Jun 7, 2019

@bendlas

This comment has been minimized.

Copy link
Contributor Author

commented Jun 8, 2019

I've updated to use extracted libraries and to handle melpa-stable. I think the melpa part should be mostly ready, now. Please give it a whirl.

Does anybody care about some changed package names? e.g. _0blayout now needs to be referenced as emacsPackages."0blayout". I think this is justified, because there might be more renames, that the old scheme might not have caught either, e.g. @, and I don't feel like reverse engineering it all, given that the user benefit would be dubious at best ..

@adisbladis adisbladis force-pushed the bendlas:emacs-updater-elisp branch from c2ccb15 to 4c74432 Jul 29, 2019

@adisbladis

This comment has been minimized.

Copy link
Member

commented Jul 29, 2019

@bendlas I've rebased and re-ran this. The generation works but building my configuration currently fails on cider-mode not passing sha256.

I intend to make this usable (and hopefully closer to a mergable state) in the next couple of days.

@adisbladis adisbladis force-pushed the bendlas:emacs-updater-elisp branch from 4c74432 to cbae708 Jul 30, 2019

@adisbladis

This comment has been minimized.

Copy link
Member

commented Jul 30, 2019

My errors were simply a side effect of running the generation on an unstable internet connection.
I re-ran the generation and building my emacs config with 74 packages works great :)

@bendlas How do you feel about merging me merging this (providing ofborg gives thumbs up)?

I think we just want to copy the name mappings from emacs2nix https://github.com/ttuegel/emacs2nix/blob/9143b374be02bd6220fd6317f1b72837ce8254bf/names.nix before we do the merge.

cc'ing a few other known emacs users for testing
@matthewbauer @ttuegel @etu @talyz @tazjin

@adisbladis

This comment has been minimized.

Copy link
Member

commented Jul 30, 2019

It seems I still have some evaluation errors to sort out. We're making headway!

edit: Bitbucket sources are broken. I believe this to be the only show stopper right now.
Everything seems to be working as intended.

@etu

etu approved these changes Jul 31, 2019

Copy link
Contributor

left a comment

I've just tried to run ./update-melpa, it took a while but the result seemed fine. And I can build packages: nix-build -A emacsPackagesNg.magit

@adisbladis

This comment has been minimized.

Copy link
Member

commented Aug 1, 2019

Unless anyone requests changes I plan to merge this in 24 hours.

edit: I have been running off of this PR for the last 2 days without issues (the previous evaluation errors for bitbucket was not affecting my personal configuration).

@Infinisil

This comment has been minimized.

Copy link
Member

commented Aug 1, 2019

I suggest cleaning up the commit history though, otherwise this could get annoying to run into while bisecting and it also doesn't adhere to the commit naming scheme we're trying to adhere to in nixpkgs

@adisbladis

This comment has been minimized.

Copy link
Member

commented Aug 1, 2019

@Infinisil That was always the plan :) In fact I'll do that now.

@adisbladis adisbladis force-pushed the bendlas:emacs-updater-elisp branch from e99e032 to f43ae5d Aug 1, 2019

@adisbladis adisbladis force-pushed the bendlas:emacs-updater-elisp branch from f43ae5d to e1b2d85 Aug 2, 2019

emacs-modes: replace melpa-generated with json format
This approach has several differences with emacs2nix:

- the updater uses a downloaded recipes.json and archive.json for commit information, it uses a local checkout only for hashing the recipes
- the generated file is JSON
- the updater is written in emacs lisp
- prefetch errors are put into an error key in the JSON, for review + meta.broken attributes are generated from it

The updater re-uses the existing generated file to memoize prefetched content-sha256s for commits, thus prefetching should normally be quite fast.

@adisbladis adisbladis force-pushed the bendlas:emacs-updater-elisp branch from e1b2d85 to d65f1b2 Aug 2, 2019

@adisbladis adisbladis merged commit e624d0e into NixOS:master Aug 2, 2019

1 check was pending

grahamcofborg-eval Cloning project
Details
@adisbladis

This comment has been minimized.

Copy link
Member

commented Aug 2, 2019

@bendlas Thanks for all your hard work in making this PR! I'm so happy about this change 😄 🎉

@bendlas

This comment has been minimized.

Copy link
Contributor Author

commented Aug 2, 2019

Awesome! Thanks for pushing this through! 🎉🎆

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.