Links to notes in subdirectories

[haditim]

Hello,
Thanks a lot for this awesome package. This is not an issue with denote, but my lack of elisp knowledge and I would appreciate if one could shine a bit of light in my path.

I needed to organize my notes into subdirectories as opposed to the default notes path. So I came up with following functions:

(require 'f)
(require 'denote)

(defun mynote--get-note-subdirs ()
  "Lists only names of subdirectories"
  (let ((subdir-names)
        (subdirs (f-directories denote-directory)))
    (dolist (item subdirs)
      (add-to-list 'subdir-names (file-name-nondirectory item)))
    subdir-names))

(defun mynote--set-denote-keywords ()
  "Sets `denote-keywords' based on subfolder structure"
  (setq denote-known-keywords (mynote--get-note-subdirs)))

(defun +mynote/new-subdir ()
  "Creates sub directory in the `denote-directory' for better organization"
  (interactive)
  (if-let (keyword (read-string "Subdir name: " nil))
      (let ((subdir (file-name-concat denote-directory keyword)))
        (let ((loc-file (file-name-concat subdir ".dir-locals.el")))
          (if (f-dir? subdir)
              (message (concat "directory " subdir " already exists!"))
            (progn
              (make-directory subdir)
              (if (f-file? loc-file)
                  (message (concat "file " loc-file " already exists!"))
                (progn
                  (make-empty-file loc-file)
                  (write-region "((nil . ((denote-directory . local))))" nil loc-file)))))
          (mynote--set-denote-keywords)))))

(defun +mynote/new-in-subdir ()
  "Call this function instead of `denote' to create note in a subfolder and set keywords"
  (interactive)
  (let* ((keyword (denote--keywords-prompt))
         (denote-directory (file-name-concat denote-directory keyword)))
    (denote
     (denote--title-prompt)
     keyword)))

(defun +mynote/new-in-subdir-with-date ()
  "Call this function instead of `denote-date' to create notes in subfolder with date"
  (interactive)
  (let* ((keyword (denote--keywords-prompt))
         (denote-directory (file-name-concat denote-directory keyword)))
    (denote-date
     (denote--date-prompt)
     (denote--title-prompt)
     keyword)))

(defun +mynote/browse-notes ()
  "Browse files from `denote-directory'"
  (interactive)
  (unless (bound-and-true-p denote-directory)
    (message "denote-directoy not defined"))
  (doom-project-browse (concat denote-directory "/")))

This allows me to create subdirectories and at the same time populate denote-known-keywords. Everything works as expected. However, when I link a note from any subdirectories, the link would not include the path and fails to open. Also backlinks are nil if links are from subdirectories.

FYI, I use doom emacs and this is the link to my full configuration.

Thanks again.

[protesilaos]

Hello @haditim,

Thanks a lot for this awesome package.

You are welcome!

This is not an issue with denote, but my lack of elisp knowledge [...]
However, when I link a note from any subdirectories. The link would
not include the path and fails to open. Also backlinks are nil if
links are from subdirectoeries.

This is an inherent constraint in the current design of Denote. There
is a hardcoded assumption that notes always exist inside a flat
directory.

There is the possibility to maintain distinct flat directories, by using
a directory-local variable for denote-directory, but this still means
that links/backlinks are inside a flat listing---the directories do not
have links between them.

The local variable setup is mentioned in this section of the manual:
https://protesilaos.com/emacs/denote#h:f34b172b-3440-446c-aec1-bf818d0aabfe.

I am open to the possibility of supporting subdirectories, though I am
still not sure what the best course of action is. Maybe we want to just
recursively search the denote-directory. Or, perhaps, we should have
a denote-directories which explicitly lists all the paths where notes
are stored. I tend to prefer the latter, as we then do not have to
worry about false positives, such as a subdir with attachments that are
not notes per se.

Any further thoughts from you or others are most welcome.

[jeanphilippegg]

Maybe we want to just
recursively search the denote-directory. Or, perhaps, we should have
a denote-directories which explicitly lists all the paths where notes
are stored.

I don't know which one would be the best, but I would like to suggest a 3rd option to consider if this gets implemented: Search recursively the denote-directory, excluding directories in denote-excluded-subdirectories for those who need to exclude subdirs. This is similar to how git works with its .gitignore.

[haditim]

@protesilaos thank you for the reply

The local variable setup is mentioned in this section of the manual: https://protesilaos.com/emacs/denote#h:f34b172b-3440-446c-aec1-bf818d0aabfe.

Yes, I followed the manual for local variable (it is exactly what my +mynote/new-subdir does)

I am open to the possibility of supporting subdirectories, though I am still not sure what the best course of action is. Maybe we want to just recursively search the denote-directory. Or, perhaps, we should have a denote-directories which explicitly lists all the paths where notes are stored. I tend to prefer the latter, as we then do not have to worry about false positives, such as a subdir with attachments that are not notes per se.

Any further thoughts from you or others are most welcome.

I personally cannot think of organizing all my notes into one flat file structure. I honestly don't know if other people would use categorizing per subfolders as heavily as I would do, but I think it'd be a good idea to support it. In that I also think what @ggjp suggested is good:

Search recursively the denote-directory, excluding directories in denote-excluded-subdirectories for those who need to exclude subdirs. This is similar to how git works with its .gitignore.

I think one can also automate the denote-excluded-subdirectories by checking if local variable denote-directory is set in any (sub) directory and whether it is different from the global one.

The way I went with my config also allows a dynamic keyword population and removes the need to manually set them in advance.

[protesilaos]

I personally cannot think of organizing all my notes into one flat
file structure. I honestly don't know if other people would use
categorizing per subfolders as heavily as I would do, but I think it'd
be a good idea to support it. In that I also think what @ggjp
suggested is good

Very well! Later today I will write a prototype and put it on a new
branch. We can then work on the technicalities.

[protesilaos]

Very well! Later today I will write a prototype and put it on a new
branch. We can then work on the technicalities.

This will require a bit more time. Expect an update tomorrow.

[jeanphilippegg]

@protesilaos About making denote aware of the subdirectories (while keeping links/backlinks still working), I was able to make it work in a private branch, but the fontification of the backlinks in denote-link--prepare-backlinks broke, so I had to comment it out. I have not implemented the excluded-subdirectories feature yet. I expect this to be easy to implement in a second step.

I just wanted to share this info with you, in case you have not started working on it and would rather wait a bit until the copyright assignment is done. (I have received the form to sign and will email it in a few hours.)

[protesilaos]

@ggjp Very well! I have something, but it is not final. Will stash it for now.

Do you want to prepare a PR or wait for your first one to be merged?

[jeanphilippegg]

My work might not be final either. I have created a pull request to support subdirectories, if you want to have a look at it. I have also listed the work that remains to be done.

[protesilaos]

The code from @ggjp is now in the subdir-support branch: https://github.com/protesilaos/denote/tree/subdir-support

Will iterate on it. Once their FSF paperwork is done, we can merge everything into main.

[protesilaos]

Just to note that the support for subdirectories is now in the main branch. I think everything should work now, though please report any further issues.

We then need to consider the case of id: link types, as discussed in issue 29.

[protesilaos]

With #29 done, I am closing this issue. I think the subdir functionalitty works now. Will continue testing it and prepare a new release as soon as possible.

[haditim]

Thanks a lot for the work on the issue. I was very busy over the weekend. I gave new changes a ride, but could not make it to work in a one hour time slot and had to give up on it. I will be trying to make this work once I have more time and report back.
Thanks again for all the efforts on this.

[protesilaos]

You are welcome! Please let me know if you need help.

[haditim]

This is to confirm that everything works as expected with denote-subdirectory and is exactly what I had in mind. Thanks again for all your efforts.

[protesilaos]

You are welcome!