This commit passes the full load-path to the child emacs process when doing byte-compilation. This should prevent the child process from failing to compile files that could have been compiled in the parent process.
Also, pass list of files to be byte-compiled as a lisp form on the
command line instead of stdin, and use a more future-proof method of
determining which elisp files the child process needs to load.
Passing the list of files to compile on the command line is what we used to do, I switched to using stdin to be able to fix command line length limits related bugs that in particular windows users were experiencing.
Is it possible to send in the load-path as the first line of stdin?
Actually, I really made two orthogonal changes here. The first is that I switched from passing via stdin to passing via command-line argument. The second is that instead of just passing a list of files, I'm actually passing a full expression to be evaluated. I propose that I keep the second change and scrap the first. That is, pass a complete form to be evaluated via stdin, and have the child process simply do something like (eval (read)).
This sounds a little like wanting to be too general, as the subprocess here is only crafted to be able to byte-compile files. It needs a load-path to do its job, I don't think it needs to be a full blown general emacs-lisp worker, like the one you can find in el-node if memory serves.
There are several things that the emacs subprocess needs: the load-path, a list of files to load (i.e. el-get-byte-compile.el), and finally the list of files to byte-compile. We could send these via stdin in a plist or something, but if we are sending (:load-path (XXX) :load-files (YYY) :byte-compile (ZZZ)), then by that point we may as well just send (progn (setq load-path (XXX)) (mapc load (YYY)) (mapc 'el-get-byte-compile-file-or-directory (ZZZ))).
(:load-path (XXX) :load-files (YYY) :byte-compile (ZZZ))
(progn (setq load-path (XXX)) (mapc load (YYY)) (mapc 'el-get-byte-compile-file-or-directory (ZZZ)))
Obviously my current implementation is bad because it creates really really ridiculously long command lines, which we don't want. So I'll fix it to use stdin instead.
We could also say have stdin look likes:
Yes that looks like a specialized protocol with labels, colons, empty lines and \n being meaningful, but somehow I still thing that would be better to have el-get-byte-compile a very specialized subprocess here.
Passing the entire load-path to the sub-process fixes the build for the anything package once #556 is applied. I'll go ahead and close #540.
It seems cleaner to pass the read representation of the list on stdin than to come up with a new protocol. The passed list doesn't have to be eval'able: ((:load-path . ("foo")) (:byte-compile . (...))
The load-files idea is a good one. My homegrown stuff for byte-compilation will load every file first if it needs byte-compilation just to be sure that all references are present. It may be useful to do that here as well.
Thanks for coding up acceptable equivalents. el-get's superb and this was the last blocker to me embracing it.
That should read:
"My homegrown stuff for byte-compilation will load each file that needs byte-compilation just to be sure that all references are present. It may be useful to do that here as well."
The code may be easier to understand:
(defun sj/recompile-newer-libs (&optional prefix)
"Re-byte-compile all libs with newer source in load-path that have paths
beginning with PREFIX. Returns alist of (FILE . ERROR) for libs that didn't
(let ((prefix (concat "^" (expand-file-name (or prefix sj/emacs-base-dir))))
(labels ((trim (prefix lib)
(replace-regexp-in-string prefix "..." lib nil t)))
(dolist (lib (sj/find-newer-libraries load-path))
(when (string-match-p prefix lib)
(sj/byte-compile-file lib 'force)
(push (trim prefix lib) compiled))
(error (push (cons (trim prefix lib) err) errors)))))
(message (cond ((or errors compiled)
(if errors (format "Failed: %s\n" errors) "")
(if compiled (format "Compiled: %s" compiled) "")))
(t "All libraries are up to date!")))))
(defun sj/load-and-byte-compile-library (library)
"Byte-compile a library after first locating it using load-path.
Loads the library file first."
(interactive "sLibrary name: ")
(unless (equal (substring library -3) ".el") ".el")))))
Pass load-path to byte-compiling child process
Fix typo naming el-get-all-symbol-files-1
Pass code to eval via stdin instead of command-line
Add test for byte-compilation
I've updated my branch to pass via stdin, but otherwise basically do the same thing. I added a test that installs yasnippet and ensures that it actually gets byte-compiled.
On reflection I can see how a property list of lists of filenames is better than inventing yet another protocol. I needed a good sleep :)
Well, I can rewrite it to pass a plist instead of passing code directly if you like.
Yes please just hand it data over stdin, adding to the existing mechanism only the ability to set the load-path and load a given list of files before byte compiling.
Pass a plist instead of code over stdin
Done. Test it with test/el-get-issue-541.el.
Looks mostly good, I'm wondering about el-get-byte-compile-for-subprocess which isn't in the patch. I'm also failing to see where the list of files to load is defined, which might well be the same problem.
Remove obsolete variables
Remove obsolete functions
Oops, I accidentally left those in after I removed the need for them. The "list of files to load" is currently just el-get-byte-compile.el, which I've just put in the command line itself, the way it was originally. There's no way to pass that via stdin without also passing some code to load it via --eval or stdin, since there's a bootstrapping issue otherwise.
Some packages apparently need to load specific files before they can byte-compile their sources, I think we should add support for that too. That would be another patch though, the current one seems ready to go!
Unless there's a way to auto-detect which files to load before byte-compiling, then I think the simplest solution to that problem would be adding a :pre-compile property to el-get-sources, which contains code that is run in the emacs subprocess before byte-compiling. Or it could just be a list of features to load before compiling or something.
Or those packages could just provide a :build property.
I think a :byte-compile-load property (maybe with a better name, it's too late here) would cut it, it's a very special need that few recipe have.