Org is rebuilt even though no source files have changed (just the mtime of the lisp subdir)

[andersjohansson]

What's wrong

I often visit the source of org files, and even if I don’t change anything, the mtime of the lisp subdir in the org repo where all elisp files reside is often changed because flycheck creates temporary files that are later removed. The command for detecting changes which (as I understand it) should be something like find org -name .git -prune -o -newermt CACHEDMTIME -print will then output lisp and org will be rebuilt, even though no "real" changes to the source have occurred.

This got a lot more noticeable now that I’m testing out the emacs native-comp branch and all fans go off at max when recompiling all the org elisp to eln files 🙂.

I suppose this only happens for repos like org where the elisp files are in a subdir. Temporary and removed files in the main repo directory resulting in changed mtime for that directory is not registered as a change. Would it be feasible with more precision for the change detection? But than we would risk to miss what should be detected as a change in other repos? For example only looking at elisp files won’t work, as it’s that’s clearly not enough for for many packages that contain for example C code like pdf-tools)?

Directions to reproduce

I created a .emacs.d with only this init.el

(setq straight-use-package-by-default t
      straight-vc-git-default-clone-depth 1
      straight-cache-autoloads t
      straight-repository-branch "develop")

(defvar bootstrap-version)
(let ((bootstrap-file
       (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
      (bootstrap-version 5))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
        (url-retrieve-synchronously
         "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
         'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp)))
  (load bootstrap-file nil 'nomessage))

(straight-use-package 'use-package)

(use-package org
  :commands org-mode
  :demand t)

Then ran it:
emacs --quick --eval "(setq user-emacs-directory \"/home/aj/.emacs.d-test-straight\")" --eval "(setq user-init-file \"/home/aj/.emacs.d-test-straight/init.el\")" --load "/home/aj/.emacs.d-test-straight/init.el"

Straight, use-package and org is installed.
Quitting and restarting, everything works as expected.

touch ~/.emacs.d-test-straight/straight/repos/org/lisp/testfile
rm  ~/.emacs.d-test-straight/straight/repos/org/lisp/testfile

Restarting now triggers a rebuild of org.

Version information

• Emacs version: "27.0.90" (built from master commit 0b217c2bfde6da6529499c9526a460fcdbeec8f)
• Operating system: Linux 5.4.36-1-lts SMP Wed, 29 Apr 2020 16:23:03 +0000 x86_64 GNU/Linux

[andersjohansson]

Maybe I was mistaken about what the find command does. Just now I looked at some source files for helm, which for some reason I really don’t know ended up in the ".git" directory getting an updated mtime (none of the current files inside it are updated).

find ~/.emacs.d/straight/repos/helm -name .git -prune -o -newermt "2020-05-05 10:00" -print then outputs the base dir ~/.emacs.d/straight/repos/helm as changed, and a rebuild is started (more fan whine 🙂).

[raxod502]

I don't understand. Is the mtime of .git updated or the mtime of the entire local repository directory?

[andersjohansson]

The mtime of the entire local repository. I tested with watchexec 'env | grep WATCHEXEC' as suggested in #508 and got

WATCHEXEC_WRITTEN_PATH=/home/aj/.emacs.d/straight/repos/bm/flycheck_bm.el
WATCHEXEC_CREATED_PATH=/home/aj/.emacs.d/straight/repos/bm/flycheck_bm.el
WATCHEXEC_REMOVED_PATH=/home/aj/.emacs.d/straight/repos/bm/flycheck_bm.el

when visiting bm.el, so flycheck is the culprit, like in #508.
This resulted in /home/aj/.emacs.d/straight/repos/bm/ being reported as changed on startup and a rebuild was triggered.

So maybe this is a question of straight being able to accurately detect “real” or “relevant” changes to a repository, where a changed mtime of the repository directory because of added and removed files isn’t a relevant change, but triggers a rebuild with the current implementation. But it is of course a hard problem to do this in a good way.

[andersjohansson]

Closing as a duplicate of #508. Sorry for missing that before posting this.

[raxod502]

I suspect there is no workaround and we will have to prevent Flycheck from creating those temporary files. After all, it's not possible to tell what change caused the mtime on a directory to be updated.