function with loopy fails on second call

[Luis-Henriquez-Perez]

I wrote this function to begin automating finding the load-path of mu4e. Weird thing is that it succeeds on the first call (it returns a path), but on every call afterwards it throws an error void-variable base-dir.

Notably when I just run the body of this function--aka the loopy loop--it always succeeds. I'm not sure exactly what's going on but i wanted to point this out.

(defun mu4e:load-path-guix ()
  "Return load-path for mu4e.
This assumes that you're on guix."
  (loopy (with (regexp "\\`[[:alnum:]]+-mu-\\(?:[[:digit:]]\\|\\.\\)+")
	       (base-dir "/gnu/store/"))
	 ((list file (directory-files base-dir))
	  (expr full-path (f-expand file base-dir))
	  (when (and (string-match-p regexp file) (f-dir-p full-path))
	    (expr mu4e-path (f-expand "share/emacs/site-lisp/" full-path))
	    (when (f-dir-p mu4e-path)
	      (return mu4e-path))))))

This is the corresponding backtrace.

Debugger entered--Lisp error: (void-variable base-dir)
  (directory-files base-dir)
  (let* ((list-2794 (directory-files base-dir))) (cl-block nil (while (consp list-2794) (setq file (car list-2794)) (setq full-path (f-expand file base-dir)) (when (and (string-match-p regexp file) (f-dir-p full-path)) (setq mu4e-path (f-expand "share/emacs/site-lisp/" full-path)) (when (f-dir-p mu4e-path) (cl-return-from nil mu4e-path))) (setq list-2794 (cdr list-2794))) nil))
  (let ((mu4e-path nil) (full-path nil) (file nil)) (let* ((list-2794 (directory-files base-dir))) (cl-block nil (while (consp list-2794) (setq file (car list-2794)) (setq full-path (f-expand file base-dir)) (when (and (string-match-p regexp file) (f-dir-p full-path)) (setq mu4e-path (f-expand "share/emacs/site-lisp/" full-path)) (when (f-dir-p mu4e-path) (cl-return-from nil mu4e-path))) (setq list-2794 (cdr list-2794))) nil)))
  (let* ((regexp "\\`[[:alnum:]]+-mu-\\(?:[[:digit:]]\\|\\.\\)+")) (let ((mu4e-path nil) (full-path nil) (file nil)) (let* ((list-2794 (directory-files base-dir))) (cl-block nil (while (consp list-2794) (setq file (car list-2794)) (setq full-path (f-expand file base-dir)) (when (and (string-match-p regexp file) (f-dir-p full-path)) (setq mu4e-path (f-expand "share/emacs/site-lisp/" full-path)) (when (f-dir-p mu4e-path) (cl-return-from nil mu4e-path))) (setq list-2794 (cdr list-2794))) nil))))
  (loopy (with (regexp "\\`[[:alnum:]]+-mu-\\(?:[[:digit:]]\\|\\.\\)+")) ((list file (directory-files base-dir)) (expr full-path (f-expand file base-dir)) (when (and (string-match-p regexp file) (f-dir-p full-path)) (expr mu4e-path (f-expand "share/emacs/site-lisp/" full-path)) (when (f-dir-p mu4e-path) (return mu4e-path)))))
  mu4e:load-path-guix()
  eval((mu4e:load-path-guix) nil)
  elisp--eval-last-sexp(t)
  eval-last-sexp(t)
  eval-print-last-sexp(nil)
  funcall-interactively(eval-print-last-sexp nil)
  call-interactively(eval-print-last-sexp nil nil)
  command-execute(eval-print-last-sexp)

[okamsn]

That is unexpected.

• Would you share how you've setup Guix and maybe Mu4e so that I can try running the function in the same environment? I don't use either.
• Does it still fail if you use the expanded form that the macro produces? I wouldn't expect a difference, but it might be worth checking.
• Are you running a compiled form of the function, or is this failing when it's just interpreted too?

(defun mu4e:load-path-guix ()
  "Return load-path for mu4e.
This assumes that you're on guix."
  (let* ((base-dir "/tmp/mu4e-test/")
         (regexp "\\`[[:alnum:]]+-mu-\\(?:[[:digit:]]\\|\\.\\)+"))
    (let ((list-458 (directory-files base-dir))
          (mu4e-path nil)
          (full-path nil)
          (file nil))
      (cl-block nil
        (while (consp list-458)
          (setq file (car list-458))
          (setq full-path (f-expand file base-dir))
          (when (and (string-match-p regexp file)
                     (f-dir-p full-path))
            (setq mu4e-path (f-expand "share/emacs/site-lisp/" full-path))
            (when (f-dir-p mu4e-path)
              (cl-return-from nil mu4e-path)))
          (setq list-458 (cdr list-458)))
        nil))))

EDIT:

Also, what version of Emacs are you using? I tried creating a similar function (which expands to the above code) on Emacs 28, and it keeps returning nil without error. The nil return value is expected, as I don't have the sought file path.

[Luis-Henriquez-Perez]

Would you share how you've setup Guix and maybe Mu4e so that I can try running the function in the same environment? I don't use either.

I don't think this has to do with Guix or Mu4e. In my code I just search for a path that happens to be a path in guix and that happens to be for Mu4e. I tried reproducing with this for example.

(defun mu4e:other-path ()
  "Return load-path for mu4e.
This assumes that you're on guix."
  (loopy (with (regexp "Documents")
	       (base-dir (f-full "~/")))
	 ((list file (directory-files base-dir))
	  (expr full-path (f-expand file base-dir)))))

Also I didn't repond to you immediately because i wanted to try it after
restarting emacs. I first tried it with commit 967f2cd and then I pulled and
recompiled to the latest version 312cfb2. I get embarrased if I can't
reproduce. But in this case I was able to consistently reproduce the problem.

Also, what version of Emacs are you using?

(emacs-version) ;;=>

"GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.20, cairo version 1.16.0)"

Does it still fail if you use the expanded form that the macro produces? I wouldn't expect a difference, but it might be worth checking.

No, it does not fail in that case. I post the code I evaluated.

(defun mu4e:other-path ()
  "Return load-path for mu4e.
This assumes that you're on guix."
  (loopy (with (regexp "Documents")
	       (base-dir (f-full "~/")))
	 ((list file (directory-files base-dir))
	  (expr full-path (f-expand file base-dir)))))

(mu4e:other-path)
;; second time produces an error

;; first time evaluated
nil

(defun mu4e:other-path-macroexpanded ()
  (let*
      ((base-dir
	(f-full "~/"))
       (regexp "Documents"))
    (let
	((full-path nil)
	 (file nil))
      (let*
	  ((list-45
	    (directory-files base-dir)))
	(cl-block nil
	  (while
	      (consp list-45)
	    (setq file
		  (car list-45))
	    (setq full-path
		  (f-expand file base-dir))
	    (setq list-45
		  (cdr list-45)))
	  nil)))))

(mu4e:other-path-macroexpanded)

;; several calls to eval-print-last-sexp
nil

nil

nil

nil

Are you running a compiled form of the function, or is this failing when it's just interpreted too?

I did not compile it. This is interpreted.

[okamsn]

I can reproduce this on Emacs 27.1, and will probably need to ask about this on the mailing list.

Like what you found, if I evaluate the function in IELM, it will return nil the first time and signal an error on all further calls until I re-evaluate the function again. This resets the behavior, but still signals an error on the second invocation.

The function value of the symbol (as gotten by symbol-function) is

(closure
 (t)
 nil "Return load-path for mu4e.
This assumes that you're on guix."
 (loopy
  (with
   (regexp "Documents")
   (base-dir
    (f-full "~/")))
  ((list file
         (directory-files base-dir))
   (expr full-path
         (f-expand file base-dir)))))

after evaluating the function, but

(closure
 (t)
 nil "Return load-path for mu4e.
This assumes that you're on guix."
 (loopy
  (with
   (regexp "Documents"))
  ((list file
         (directory-files base-dir))
   (expr full-path
         (f-expand file base-dir)))))

after invoking the function. As you can see, (base-dir (f-full "~/")) was removed from the function value, for some reason.

[okamsn]

I asked the question on the mailing list here: https://lists.gnu.org/archive/html/help-gnu-emacs/2021-02/msg00104.html.

Below is the reply of Stefan Monnier:

---

No, I wrote a macro like cl-loop here:

https://github.com/okamsn/loopy

I received a bug report that said that the above function would raise an
error after running a second time here:

In the current situation, the problem is:

(setq loopy--main-body (nreverse loopy--main-body)
      loopy--with-vars (nreverse loopy--with-vars))

which reverses the "with vars" received as arguments.
If you replace nreverse with reverse here, this particular problem
should disappear.

This said, this nreverse reverses the order of the bindings in (with ....)
which maybe isn't what you intended (I suspect that the nreverse here
is meant to reverse the bindings added to loopy--with-vars via
push).
So maybe a better solution is to replace

    (setq loopy--with-vars (cdr arg)))

with

    (setq loopy--with-vars (reverse (cdr arg))))

so that the subsequent nreverse puts them back in the original order.

---

I've made the change that he suggested, and the problem seems to be fixed. I still don't understand why it happens, or what is meant by the symbol's function value changing. I will write more when I get or am told more information.

[Luis-Henriquez-Perez]

On a basic level I know that nreverse is destructive so it somehow messes with loopy--with-vars. But I don't know the details of how that explains this specific case. What intrigues me is why it works for emacs 28 but not emacs 27.

[okamsn]

Related to this experience is the discussion at https://old.reddit.com/r/emacs/comments/lf8whe/elisp_question_let_variables/, where a user notes that changes to a constant structure defined in a let-form persist between function calls.

There, user wasamasa replies:

The problem is that quote allows an optimization where the object constructed at read time is reused. If you mutate that object (by means of setf), you end up mutating the object backing the literal and thereby, the definition of that function, giving it a different literal every time you run it.

I would have thought that loopy creates new lists each time it is run, but maybe that is not the case, as everything is treated like a symbol before being evaluated.

[Luis-Henriquez-Perez]

This seems to have been resolved, so I will close this.