id: links not working when denote has not been used before #29
Comments
|
After considering what was written on the mailing list, I think I will be setting |
|
Thank you @ggjp for reporting this! I copy the relevant feedback from @shrysr and my reply to it: |
|
I ping @kaushalmodi with whom we implemented the |
That's what I have seen as well. People update this variable in their emacs config. |
|
Searching through GitHub, I see that people add all files from Org roam directory to this variable in their emacs config. Files from denote directory can be appended to that list the same way. |
|
I don't think we need to remove the code for |
Agreed! I will now change the default value of |
|
To be clear: removing the |
|
Good! Don't forget to remove the lines about (not tested) |
A non-nil value requires extra setup and comes with potential downsides. We need to account for them and provide solutions where appropriate. Read issue 29 on the GitHub mirror: <#29>.
|
What I did now is to basically undo commit Though yes, the |
|
The problem with |
|
I am not following. What would be the issue if we remove the lines about |
Maybe I did not explain myself properly. I think they should be denote.el | 2 --
1 file changed, 2 deletions(-)
diff --git a/denote.el b/denote.el
index 909c861..50b0414 100644
--- a/denote.el
+++ b/denote.el
@@ -259,8 +259,6 @@ (defun denote-directory ()
(path (if (or (eq val 'default-directory) (eq val 'local)) default-directory val)))
(unless (file-directory-p path)
(make-directory path t))
- (when (require 'org-id nil :noerror)
- (setq org-id-extra-files (directory-files-recursively path "\.org$")))
(file-name-as-directory path)))
(defun denote--extract (regexp str &optional group)What we have now is not the right way to do it. Still, we need a working solution for users who will opt-in to the Am I now clear? Is what I am suggesting correct or did I misunderstand |
|
The misunderstanding might be coming from me. I was suggesting that the solution would be to not trying to provide a working solution for users of |
|
Would they not need to update EDIT: each "time", not "type" |
|
Ah, now I see your point! Maybe denote's code could make an exception and update it for them automatically on note creation only. I don't know other cases. |
|
I just noticed this part from the original comment of @ggjp:
Strictly speaking, this is an Org/user problem. Though we provide the Constantly updating My understanding right now is that org-roam requires the database to |
This is what I was implying. That we are, in fact, providing an option that is not viable long-term, but keeping the option for expert users who will be able to handle this. And we should warn about this clearly in the doc of that option. |
|
I do not know the internal workings of org-roam - but I think this understanding is correct, and can be verified from Section 6 in the org-roam manual. There is atleast 1 other package which used org-id for hierarchical linking, and has no database driven caching as far as I can remember - org-brain. I believe it was not very performant with some key functions with a large number of notes, as even mentioned in the README, and that aligns with my experience at that time, though it has been a few years since I used org-brain and I did not have a large number of notes then. However, I also recall there were interesting elisp based optimisations proposed on some issues (Example). If I remember right, the org-brain-visualise function was the one where all the org-ids would need to be parsed - Ref. I would not be able to judge whether these are viable in the long term and whether these solutions truly.. solved the problem. In terms of available solutions I know of - most people have taken the db or external program route. Example, Karl Voit's Memacs (driven by python iirc) and John Kitchin's org-db (sqlite indexing of org-headings during idle time). |
What you write here @ggjp and what @shrysr explained tells me that those expert users will need to be real experts. To put it concretely, I am an experienced Emacs user with no programming background, who has written several Emacs packages (including the modus-themes which are built into Emacs), but I have zero knowledge of using a db or of handling things with python and the like. So if I opt in to If each package needs to write its own optimisations and maintain its own cache, to me this shows that I wanted to support I think being a good Emacs citizen is a laudable principle. In this case, the right thing to do is to recommend the use of org-roam instead of trying to accommodate For some context here: the reason I never used org-roam is because (i) it is Org-specific whereas I write notes in different file types and (ii) I did not want to ever rely on a db or equivalent dependency. |
|
I am inclined to agree with @protesilaos in that, the core problem appears to be about optimising org-id in org.git, and that there is a large overhead in considering org-id driven links. I'm adding a comment from a slightly different point of view: As I read through #8, it seemed to me that the main points of the discussion leading to org-id based linking are :
Using org-id based linking is bound to get more complex as users start creating org-notes and linking not just atomic notes, but org headings within as well. As I recall, this was one of the key motivations between using org-id links in org-roam v2. This means not just 1 org-id per note, and even more over-head for every refresh of org-id locations. Personally, I interpret the driving principles of Denote, to be a little different from org-roam and it does not necessarily call for the benefits of org-id like interlinking of headlines, especially since note formats can be variable (md, txt, org) as already mentioned. For example - the focus is on being able to view any note using Though these notes are ideally meant to be about 'mindful note taking' - and @protesilaos has described his style as being able to compose coherent atomic notes - my mind for one - is messy and I do not always end up creating terribly informative, coherent atomic notes, and end up with a large number of short(er) notes, many of which may contain just a few lines - which may even be all I practically need on that topic. Org mode (and org-id) for such ultra-short notes is an overkill. imo, there is definitely value in being able to see that there are links between notes, create them painlessly and quickly access linked notes and ideally be able to 'link' to say, a paragraph in a note. It's worth noting that the minute things become slow-ish - the entire process is significantly hobbled. I think denote type links provides the above, and of course we can look at ways to improve, but, by Not relying on org-id - we can also consider putting efforts into adapting and enhancing a non-db driven enhanced keywords-within-notes linking - like that employed in https://howm.osdn.jp/ , in which it seems an external grep is an optional dependency, and imo much more acceptable and easily available than a db. I'm not sure how well this would scale (perhaps @kaushalmodi and @ggjp and others would care to weigh in?) - but I keep thinking that a db is not the only answer to this purpose, and org-id need not necessarily be viewed as critical. Canonical style linking, in my mind, is not critically important since it is not clear to me that the original intent behind org-mode envisioned backlinks and a very high degree of interlinking anyway. Denote links are not brittle like file and path based links from what I've seen, and denote links can be used to link to Org files as well, without org-id. |
The original idea was to support the 'org-id' library on the premise that it makes Denote a good Emacs citizen. However, discussions on the mailing list[0] and the GitHub mirror[1] have made it clear to me that 'org-id' is not consistent with Denote's emphasis on simplicity. To support the way 'org-id' works, we will eventually have to develop some caching mechanism, just how the org-roam package does it. This is because the variable 'org-id-extra-files' needs to be kept up-to-date whenever an operation on a file is performed. At scale, this sort of monitoring requires specialised software. Such a mechanism is outside the scope of Denote---if you need a db, use org-roam which is already great. [0] <https://lists.sr.ht/~protesilaos/denote/%3C8735fk4y1w.fsf%40hallac.net%3E#%3C877d4un73c.fsf@protesilaos.com%3E> [1] <#29> Quote of what I wrote on the GitHub mirror issue 29: [ggjp] This is what I was implying. That we are, in fact, providing an option that is not viable long-term, but keeping the option for expert users who will be able to handle this. And we should warn about this clearly in the doc of that option. [protesilaos] What you write here @ggjp and what @shrysr explained tells me that those expert users will need to be real experts. To put it concretely, I am an experienced Emacs user with no programming background, who has written several Emacs packages (including the modus-themes which are built into Emacs), but I have zero knowledge of using a db or of handling things with python and the like. So if I opt in to 'denote-link-use-org-id' I will eventually run into problems that my non-existent skills will prevent me from solving. At that point, I will just use org-roam which already handles this use-case in a competent way (and has a massive community to rely on in case I need further support). If each package needs to write its own optimisations and maintain its own cache, to me this shows that 'org-id' is not good enough for the time being: more work needs to be done in org.git to provide a universal solution. I wanted to support 'org-id' by default on the premise that Denote must be a good Emacs citizen which interoperates with the rest of the wider ecosystem. But if 'org-id' leaves something to be desired, then that goal is not worth pursuing: we add complexity to our code, offer an option that we cannot genuinely/adequately support, and make usage of it contingent on reading the docs and having a high level of expertise. I think being a good Emacs citizen is a laudable principle. In this case, the right thing to do is to recommend the use of org-roam instead of trying to accommodate 'org-id'. As such, I have now changed my mind and think we should remove what we previously added. For some context here: the reason I never used org-roam is because (i) it is Org-specific whereas I write notes in different file types and (ii) I did not want to ever rely on a db or equivalent dependency. <#29 (comment)>
|
I just created the @shrysr Thanks for the feedback! Indeed, the headline-based workflow is not intended for Denote. I think massive files with multiple headings introduce a dependency on Org or relevant specialised software. Same principle for not mixing notes with TODO items. My personal view is that those are bad habits that Org encourages or, to be more precise, they are good only in the right situation. Those points are covered in the manual's FAQ section: https://protesilaos.com/emacs/denote#h:da2944c6-cde6-4c65-8f2d-579305a159bb. Short notes of the sort you use are fine though: they work everywhere. About howm's approach, I am interested to learn more, though better not change the subject here. |
|
I am fine with removing org |
I think using the There is also a general design/maintenance consideration here. If we are writing "hidden options", we still need to test them as if they were formal user options (otherwise why are we providing them?). Better avoid that. Does this make sense or did I go too far? |
|
Sorry, I am away from computer for few days (long weekend in US). I'm just quickly reading this whole thread on phone. I'm alright with changing the org-id style linking to nil, and also removing the org-id-extra-files update from within denote as users can and should add that in their emacs config. But can the ID property still be left the default? That doesn't force the users to use org-id. But the org-id users can get the note linking work out of box (as they would have the org-id-extra-files set correctly in their config). There's no performance penalty for using the ID property. Thoughts? |
With
All of these work great:
|
|
Another thought: @protesilaos I don't want to cause any further trouble with this decision. I'd say just pick the approach that fits your use case and bandwidth for support, and gut feeling 👍 |
I understand your reasons and I think they are valid. I am fine with the option you choose. |
The original idea was to support the 'org-id' library on the premise that it makes Denote a good Emacs citizen. However, discussions on the mailing list[0] and the GitHub mirror[1] have made it clear to me that 'org-id' is not consistent with Denote's emphasis on simplicity. To support the way 'org-id' works, we will eventually have to develop some caching mechanism, just how the org-roam package does it. This is because the variable 'org-id-extra-files' needs to be kept up-to-date whenever an operation on a file is performed. At scale, this sort of monitoring requires specialised software. Such a mechanism is outside the scope of Denote---if you need a db, use org-roam which is already great. [0] <https://lists.sr.ht/~protesilaos/denote/%3C8735fk4y1w.fsf%40hallac.net%3E#%3C877d4un73c.fsf@protesilaos.com%3E> [1] <#29> Quote of what I wrote on the GitHub mirror issue 29: [ggjp] This is what I was implying. That we are, in fact, providing an option that is not viable long-term, but keeping the option for expert users who will be able to handle this. And we should warn about this clearly in the doc of that option. [protesilaos] What you write here @ggjp and what @shrysr explained tells me that those expert users will need to be real experts. To put it concretely, I am an experienced Emacs user with no programming background, who has written several Emacs packages (including the modus-themes which are built into Emacs), but I have zero knowledge of using a db or of handling things with python and the like. So if I opt in to 'denote-link-use-org-id' I will eventually run into problems that my non-existent skills will prevent me from solving. At that point, I will just use org-roam which already handles this use-case in a competent way (and has a massive community to rely on in case I need further support). If each package needs to write its own optimisations and maintain its own cache, to me this shows that 'org-id' is not good enough for the time being: more work needs to be done in org.git to provide a universal solution. I wanted to support 'org-id' by default on the premise that Denote must be a good Emacs citizen which interoperates with the rest of the wider ecosystem. But if 'org-id' leaves something to be desired, then that goal is not worth pursuing: we add complexity to our code, offer an option that we cannot genuinely/adequately support, and make usage of it contingent on reading the docs and having a high level of expertise. I think being a good Emacs citizen is a laudable principle. In this case, the right thing to do is to recommend the use of org-roam instead of trying to accommodate 'org-id'. As such, I have now changed my mind and think we should remove what we previously added. For some context here: the reason I never used org-roam is because (i) it is Org-specific whereas I write notes in different file types and (ii) I did not want to ever rely on a db or equivalent dependency. <#29 (comment)>
Thank you for the understanding! This is a difficult predicament to be in. I just merged the branch after making some final changes to the README.org. This note in particular: My apologies once again! |
I'm finding this sensible and tbh I forgot mentioning this in my rambling last night. I guess this addresses the benefits of canonical linking. Thanks @kaushalmodi.
This makes sense. Maybe I misunderstand things here, but can we not have denote style links to be default for (de)notes - and explicitly supported, while if they need to, users can still link denote org files via org-id to any other notes/files (and vice versa) -- in which case performance + testing for org-id driven linking is not within Denote's purview at all? Also to be clear @protesilaos - I'm just expressing thoughts, and especially since my deplorable elisp knowledge does not yet let me usefully contribute code - I'm quite fine with what you are comfortable with. |
|
I'm not sure if my earlier question was clear. Here is another attempt : I mean to ask that, can we not use the ID property for Denote Org files and have
|
|
@shrysr I will reply to you tomorrow morning. It is already past my bedtime and I feel I won't manage to write the whole thing now. I've been here practically since dawn. |
The formal support for My concern here is with managing expectations. If our Org notes are superficially the same as org-roam's, an unsuspecting user may think that Denote is an org-roam lite. We will thus get issues/requests, such as those already mentioned in this GitHub repo, about migrating from org-roam to Denote. While there are similarities, Denote is not a minimalist org-roam and I would not like to encourage the idea of treating the two as interchangeable. Doing things half-way-through is a way to create false expectations. A package on GNU ELPA must be usable by users of all skill levels. If the functionality we provide is incomplete and needs to be covered by user-level tweaks, we are excluding a portion of the user base while still assuming the maintenance burden. If someone trusts Denote to, say, write a 1000 notes, we do not want to surprise them after the fact. Imagine if the reported issues that triggered this change happened 6 months into one's daily usage of Denote: it wouldn't be nice. Setting the right expectations is a matter of responsibility: we let the user make a more informed choice and show respect for their time. It also makes it easier for me to keep Denote's scope in check by not supporting every little extra that Org implements. The premier Org extension is org-roam: we do not need another one (or, if we do, I am not the one to implement it). Some comments on the
There may be refinements to be made, which is okay as that is part of a maintainer's duties. |
|
Thanks for the explanation.
|
This issue is actually unrelated to the subdir branch, so I created a new issue to keep things separate.
How to reproduce:
id:link and click on it.Cause:
org-id-extra-filesis not set yet, because we set it in the functiondenote-directorywhich has not been called yet.I don't know how we should fix this. Is there a clean way to make sure that
org-id-extra-filesis set when a note is opened?Maybe the user should add this in their init file if they want
id:links:But maybe this is too much for a user. Alternatively, we could set this directly in the top-level of
denote.el.(In any case, we should append to that variable instead of overwriting it.)
UPDATE: Actually, I get the same behavior by default using
id:links in org files outside of denote: they do not work as-is. I suppose the user is expected to manually setorg-id-extra-files.The text was updated successfully, but these errors were encountered: