Added graph visualization of notes #921
Conversation
|
Thank you for opening your first PR! 🎉 We are very happy and would like to thank you very much for your contribution. If everything checks out, we'll make sure to review the PR as soon as possible and give feedback. In the meantime, to make the reviewing process as fast as possible, you can help us by checking the following things:
Furthermore, make sure that the linter does not complain, which will check your code on every new commit. If the linter task fails, make sure to run |
|
Hey, thanks a lot! That's quite a big thing, but, funny enough, almost implemented the way I would've done it, so for that, great! I do not have time to review it currently, and it won't ship with 1.7, but in the meantime what I've already noticed:
So long, thanks again for the PR and I'm looking forward to working on this after 1.7+my vacation! |
|
Hey @nathanlesage, I have taken a quick look on the application size and I noticed that the ➜ Zettlr git:(develop) du -sh /Applications/Zettlr.app/Contents/Resources/app.asar
107M /Applications/Zettlr.app/Contents/Resources/app.asar
➜ Zettlr git:(develop) du -sh /Applications/Slack.app/Contents/Resources/app.asar
2.3M /Applications/Slack.app/Contents/Resources/app.asar
➜ Zettlr git:(develop) du -h /Applications/Obsidian.app/Contents/Resources/*.asar
676K /Applications/Obsidian.app/Contents/Resources/app.asar
8.1M /Applications/Obsidian.app/Contents/Resources/obsidian.asarIf I understand correctly, the |
|
@Aigeruth Well. I see your point, but this is not Zettlr's fault. Here's what's taking up so much space:
Removing both would according to simple math result in an
Don't you think I have thought about that myself? Fact is:
This is already the case. Only a few scattered TL;DR: Neither the application size nor the way it actually is technically composed are something we could in any way really optimize. However, what the real performance bottlenecks are is quite easy to spot: My own, shitty and un-optimized code. That needs to be updated. But until very recently I did not have any help, and maintaining such a large codebase is very difficult alone. It will get better, but it will take time. Please feel free to join in, if you spot something that's worth optimizing! |
|
|
Hey @nathanlesage, I'm sorry if it came across that way, but I have meant in no way to imply that this is Zettlr's fault or that you are not aware of the performance recommendations of the Electron team. Thank you for explaining your point of view. Would you mind helping me with how you sized up mermaid.js, so I have the full picture here? I looked at the distributed versions of mermaid and it seems to be smaller to me (1.1M), but I may have missed something obvious: $ du -h source/node_modules/mermaid/dist/*
16K source/node_modules/mermaid/dist/index.html
408K source/node_modules/mermaid/dist/mermaid.core.js
688K source/node_modules/mermaid/dist/mermaid.core.js.map
4.4M source/node_modules/mermaid/dist/mermaid.js
4.8M source/node_modules/mermaid/dist/mermaid.js.map
1.1M source/node_modules/mermaid/dist/mermaid.min.js
5.1M source/node_modules/mermaid/dist/mermaid.min.js.mapThese bundled and minified versions already contain all necessary dependencies. I agree with you and probably there is not much to shave off from dictionaries and other assets. |
Don't worry, I was a little bit snappy myself, sorry for that!
Huh, this is surprising — but it may very well be incurred by the peer-dependencies of mermaid itself. I remember walking the extra mile and I just downloaded mermaid into an empty NPM project to see what additional dependencies it would in itself pull, and in the end the This is basically why I'm always hesitant in adding new dependencies, because I fear that the additional peer dependencies bloat up the overall size :/
Is there an additional package on NPM that I missed? Because I'd love having 1.1 meg over 40 megabytes, that's for sure. But I want to pull it in as an NPM package, and not having to create an additional CI run just to keep that dependency updated. It seems you've managed to only retrieve the mermaid dependency? (But maybe we should now transfer this discussion someplace else, maybe you could open an additional PR if you managed to strip the codebase of these 39 unnecessary MB of packages) |
|
And @JulienMir thanks for checking it out! I always thought d3 would be one of these monstrosities, but if you can keep it small, this might work out. How much additional overhead do the d3-modules incur? (+ we need to keep in mind to then maybe replace chartJS with d3 in the mid-term) Concerning your fourth point: No worries, it's pretty recent and you couldn't know! Keep your time, and after 1.7 is released, I'll come back to you (and if I forget, just ping me with a comment here) and we can have a look in-depth, as I'm really excited to add this functionality! |
|
Quick update, I have to apologize: Mermaid pulls in d3 as a dependency, hence Zettlr already has this thing as a transient dependency, so I waive my reservations! |
Good to hear! I tried to look into how to use the cache instead of the file system. Should the FSAL be a provider as well or should I initialize the LinkProvider outside of the |
|
Hmm … from the top of my head I'd argue that we might want to add a hook after the FSAL has loaded to initialize the link provider — I know that this trick has worked good in the past. Oh, even better: Why not add a |
|
I did as you suggested: I added a sync function that gets called on 'fsal-state-changed' signals. It indeed seems like the best solution |
Hey Julien,this is a pretty cool feature and I almost had started implementing the same thing if I didn't had found your pull request. Some remarks from my side:
Some ideas
|
|
Hey @nathanlesage, I hope you had a good vacation :) |
|
Hey, true! One of the reasons I didn't reply to this PR is that I have to really hard think again about whether this feature fits Zettlr, or not. The main reason being that I get a lot of comments along the lines of "Roam can do it, why not Zettlr?" or "Obsidian can do it, why not Zettlr?", and the thing is that I realize more and more that we're talking about two different types of application – one that is meant for heavy-lifting of writing text (Zettlr) and pure note-taking (such as Roam and Obsidian) apps. And the more I talk with other people about the feature-set of Zettlr, and the more I think about this myself, the more I get the impression that instead of copying over functionality for pure note-taking from other apps we should focus more on getting Zettlr en par with Word etc. Because, let's be honest: Note-taking apps are easy to build and it's much easier to get all of this running, but when you're dealing with longer texts such as papers or even whole book-chapters, this whole approach is prone to slow computing times, often breakages, and other stuff. Besides, while graph visualizations do sound super awesome, it's one of these things where you can implement it and get a ton of likes, but there's actually no surplus value added with them. They are bad for induction, are really CPU-heavy and do not boast productivity at all. This is why I'm seriously considering to not implement this, which does hurt me super hard, because there's so much dedication and work in it, but on the other side I have real anxiety that this project becomes just that inch too much where it suddenly becomes useless to everyone :( |
|
@nathanlesage In my opinion Zettlr would get a lot from this feature. There are plenty of reasons not to use Obsidian or Roam, and this enhancement bridge the missing gap I am missing for making Zettlr my main editor. |
|
But … do we really win anything from this graph? |
|
@nathanlesage here's my humble opinion on the whole topic. ProcessRight off the bat, I will note that I am biased towards adding this feature. Before prior research, I think that graph visualization is highly helpful for learning new concepts and documenting the learning process for reflection. During the compilation of the following hypothesis, I will do my best to adhere to waitbutwhy's thinking like a scientist(scroll down about a quarter of the way or search for to the heading "The Battle Over Our Beliefs") Please correct me if I misunderstand something you or one of your sources. Responses
That's all the analysis I'm able to do tonight. Ultamately, I think we can make use of a graph visualization by using it like a mind map and taking advantage of Zettlr interactive use case. If Zettlr doesn't want to be a note taking app then I guess people will have to find other alternatives. |
|
How about using cytoscape.js for this? I'm not a developer, but out of what I can read and understand, it's not a really huge library, there is a Vue.js component for it, can be installed to node.js. And most of all, it is very advanced, has many configuration options for those who need it, and it is actively developed... Another benefit would most likely be that it would be easy to export data to any network graph or knowledge graph system if needed... For those that use zettlr for research writing and don't wish to learn R or Python... Just a suggestion that might be in the scope of both usage and maintenance, since it's a stable library developed by the Cytoscape team... |
|
I have been working on a tool that analyses notes and wikilinks between them, and while it doesn't give you the information within Zettlr itself, you can actually redirect the output of the tool to Markdown files, to read with Zettlr. Or just read on the command-line. No visual output for now, sorry! The tool is called NoteExplorer, and among other things, you can list these kinds of notes:
It also lets you find broken links and insert backlinks into your notes, if you wish. Since this isn't at all a part of Zettlr, any questions or comments about NoteExplorer should be handled on its own GitHub pages, and not here. |
|
Emile, the developer of the Obsidian Advanced Graph told me if someone want to implement his project into any other project he would help where he could, and his project is active... It's a really great graph project, with lot of really helpful features planned, including a inline block code function for graphs with cytoscape.js (already working... If it is of any interest, check out his project: https://github.com/HEmile/obsidian-neo4j-graph-view Only thing you will be responsible for is the plugin integration, not the plugin it self... just a tips for something that actually work... |
|
One notable contribution to this debate (which is incredibly important to me as well, since if there was a graph feature in Zettlr I could opt entirely out of Obsidian), is that in the roadmap just published by @nathanlesage for V.2 (#1690) there is no mention of visualization or graphs. For me, that is the one missing piece of Zettlr to make backlinks functional. |
|
@tarahmarie Right now it's not on the roadmap because there's just too much to do, and I want the new version to be stable before engaging in such in endeavour. But nevertheless, seeing how many people still – after almost a year of me saying "No, no, no!" – still want such a feature, it's getting harder and harder to say "no". So here's a small ray of light for y'all: I will definitely integrate links deeper into the file descriptors, and once that works, I'd be willing to give visualisations a shot and see how such a network would fit into Zettlr's user interface. |
There was a problem hiding this comment.
So I've checked the last commit and suggested some changes to some parts of the code.
Disclaimer: I have not studied the whole codebase, and I have not tested your pull request (neither with nor without my suggestions). Oh, and I'd like to mention that I've been using Zettlr for about a week now (and liking it, TBH).
| @@ -159,6 +161,7 @@ async function parseFile (filePath, cache, parent = null) { | |||
|
|
|||
| // Finally, report the tags | |||
There was a problem hiding this comment.
| // Finally, report the tags | |
| // Finally, report the tags and links |
| @@ -249,6 +253,16 @@ function parseFileContents (file, content) { | |||
| } else { | |||
| file.id = '' // Remove the file id again | |||
| } | |||
|
|
|||
| // Parse links in the file | |||
| while ((match = linkRE.exec(mdWithoutCode)) != null) { | |||
There was a problem hiding this comment.
while ((match = linkRE.exec(mdWithoutCode)) != null) works fine, but consider using the new matchAll method because it's more readable and easier to iterate over.
See: String.prototype.matchAll() - JavaScript | MDN.
Note: matchAll is supported natively in Electron v12.0.1 ((Chromium v89, Node v14)) which this projects depends on.
| file.links.push({ 'name': file.name.replace(file.ext, ''), 'source': file.id, 'target': match[1] }) | ||
| } | ||
|
|
||
| // Always have atleast one link for identity |
There was a problem hiding this comment.
| // Always have atleast one link for identity | |
| // Always have at least one link for identity |
| this._globalLinkDatabase[link.source].name = link.name | ||
| } | ||
|
|
||
| // Some links might have no target |
There was a problem hiding this comment.
| // Some links might have no target | |
| // Some links might have no target (e.g. the null target we set for identity) |
| * @return {Object} An object containing all links. | ||
| */ | ||
| getLinkDatabase: () => { | ||
| return JSON.parse(JSON.stringify(this._globalLinkDatabase)) |
There was a problem hiding this comment.
We're returning a clone of the "link database" (maybe you want to mention this in the method's documentation comment).
Also, using JSON.parse(JSON.stringify(..)) to clone objects is inefficient especially for large objects or when repeated a lot. If getLinkDatabase() is used a lot, consider using a more efficient cloning approach/function.
| /** | ||
| * Returns a link (or all, if name was not given) | ||
| * @param {String} [name=null] The link to be searched for | ||
| * @return {Object} Either undefined (as returned by Array.find()) or the tag | ||
| */ | ||
| get (link = null) { | ||
| if (!link) { | ||
| return this._links | ||
| } | ||
|
|
||
| return this._links.find((elem) => { return (elem.source === link.source && elem.target === link.target && elem.name === link.name) }) | ||
| } | ||
|
|
||
| /** | ||
| * Add or change a given link. | ||
| * @param {String} name The link source's name | ||
| * @param {String} source The link source | ||
| * @param {String} target The link target | ||
| */ | ||
| set (name, source, target) { | ||
| let link = this.get({ 'name': name, 'source': source, 'target': target }) | ||
| // Either overwrite or add | ||
| if (!link) { | ||
| this._links.push({ 'name': name, 'source': source, 'target': target }) | ||
| } | ||
|
|
||
| this._save() | ||
|
|
||
| return this | ||
| } |
There was a problem hiding this comment.
The set method is not doing what it's claiming in its documentation ("Add or change a given link.").
It is only trying to add an element if it doesn't already exist.
const tmpLink = { 'name': name, 'source': source, 'target': target }
const gotLink = this.get(tmpLink)
if (link) {
// or maybe use: Object.assign(gotLink, tmpLink)
// SEE: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/assign
// gotLink.name = name // Isn't link.name read-only?
gotLink.source = source
gotLink.target = target
} else {
this._links.push(tmpLink)
}
// ...More importantly, the get method is not working as expected: get says it will return a Link object where its name attribute matches the passed String argument; however, we are passing (and it is using) a Link-shaped object. Worst, in find, it is testing whether all properties match (name, source, and target) as opposed to just name, and this results in the wrong behavior of returning the link only if it was not changed...
-
Fix: Make the predicate passed to
findtest only if(elem.name === link.name). -
Better: Make the
getfunction actually accept only a String argument calledname. This will make it make more sense and not force the caller to create temporary objects (like we're currently doing when calling it). -
Just saying: Having
getreturn different types depending on the input (a singleObject/nullvs anArray<Object>) feels meh. Maybe another method,getAll, should be responsible for returning all links if the caller intentionally wants them. (Plus it's more descriptive.)
| // Building node list | ||
| for (const a of Object.entries(data)) { | ||
| tmpNodes.push({ | ||
| 'id': a[0], | ||
| 'title': a[1].name ? a[1].name.replace(a[0], '') : 'Undefined', | ||
| 'inbound': a[1].inbound ? a[1].inbound.length : 0, | ||
| 'outbound': a[1].outbound ? a[1].outbound.length : 0 | ||
| }) | ||
| } | ||
|
|
||
| this.graph = { | ||
| 'nodes': tmpNodes, | ||
| 'links': tmpLinks | ||
| } |
There was a problem hiding this comment.
In the "Add nodes" part, you're calling .attr('r', function (d) { return 1 + d.inbound + d.outbound }).
When I first read it, I was like, "Aren't inbound and outbound arrays? Wouldn't calling this code return incorrect results like the following snippet?"
(1 + [2, 3] + [4, 5]) === '12,34,5'But it seems like LinkProvider's Node/Link objects are different from NoteNetwork's. In this case, I suggest using different names for them... by "them," I mean the types/objects, or at least rename the attributes from inbound to inboundCount and outbound to outboundCount.
| class LinkProvider { | ||
| /** | ||
| * Create the instance on program start and initially load the links. | ||
| * @param {FSALCache} cache a cache to store links |
There was a problem hiding this comment.
The constructor accepts no cache param.
| var tmpNodes = [] | ||
|
|
||
| // Building link list | ||
| for (const a of Object.entries(data)) { |
There was a problem hiding this comment.
Maybe consider renaming a to idNodePair and i to targetNodeId (if I got these correctly, that is).
| global.links = { | ||
| /** | ||
| * Adds an array of links to the database | ||
| * @param {Array} linkArray An array containing the links to be added |
There was a problem hiding this comment.
Aside: As a general suggestion about JSDoc comments, I believe it would be better if you could use generic types (e.g. Array<String>) instead of the non-generic version (e.g. Array). This would help others understand the code and (I guess) make the editor provide better hints and type-checking.
This is not really important as Zettlr is migrating to TypeScript anyways.
|
Work on this pull request makes my comment mostly useless, but imma post the following anyways: Before I knew of this project, I was considering forking the VSCode extension that @pcuci mentioned (to make it a standalone Node package). These "factlets" may interest you:
From @tchayen/markdown-links/src/utils.ts: export const getDot = (graph: Graph) => `digraph g {
${graph.nodes
.map((node) => ` ${node.id} [label="${node.label}"];`)
.join("\n")}
${graph.edges.map((edge) => ` ${edge.source} -> ${edge.target}`).join("\n")}
}`;You could do something similar to the following in Mermaid: export const getMermaid = (graph: Graph) => `graph TB {
${graph.nodes
.map((node) => ` ${node.id}(${node.label})`)
.join("\n")}
${graph.edges.map((edge) => ` ${edge.source} --> ${edge.target}`).join("\n")}
}`;
// TODO: Manually replace bidirectional arrows with one undirected arrow.
// Like, if `A --> B` and `B --> A` exist, output only one `A --- B`.
/*
const getCorrectLink = (nodeAId, nodeBId) => {
const existsEdgeBA = graph.edges.some(e => e.source === nodeBId && e.source.target === nodeAId)
const shouldOutput = (!existsEdgeBA) || (existsEdgeBA && nodeAId > nodeBId)
const arrowType = existsEdgeBA ? '---' : '-->'
return shouldOutput ? ` ${nodeAId} ${arrowType} ${nodeBId}` : ''
}
*/
|
|
@nathanlesage - Only thing I did was to ask Emile because his addon is something that will be active for a while, and it has a lot of customization options AND use Cytoscape.js that is under active development and is used by many projects... What you or any others want to do with that information, that's entirely up to you. |
|
Any updates on this feature? |
Maybe you should look at the work Emilie are doing for his addon Juggl for Obsidian, He is also thinking about making it more useful for other Editors by splitting the code of the graph from the Obsidian depended code... I think He would be really happy for any real help he can get, I have linked to his github repo just above your last comment, and he can be reached on his discord channel. If I don''t remember wrong, he also have a feature that makes it possible to create inline graphs using a code block. Personally I think it would be of great help for many users, even writers and maybe specially writers of technical documentation to have advanced features like this... Just a tips |
I just came across Juggl and was quite impressed with what it's doing. I came here to mention it in case it was of use. Its developer wrote that although it's currently implemented for Obsidian, it doesn't rely on that and could be ported. I saw your comment also mentioning Juggl, so I'll just add a link https://juggl.io/Juggl -- who knows, it might be an interesting combination someday. |
|
I am very late to the party, but I wanted to at least share my thougts that I had while reading through this issues. Zettlr is Open Source and Free, which makes it already better than tools à la Obsidian. Another aspect is that it respects sane Markdown defaults and integrating pandoc is a very sensible choice. It also does a really good job in making a plaintext workflow more accessible, not only by it's design and features, but also in the way it seeks to provide infomation (the docs, youtube videos, the community) and encourage users to contribute. I don't think that the crucial question is if Zettlr is either for notetaking or a replacement of Office software. Most people who regularly write may like the way to integrate there notes in their writing environment (btw I know a lot of people who use some kind of Office software for note taking). Imo the question is if Zettlr should have all features of a notetaking app, that aims to be this and nothing more. Imo Zettlr is a nice writing environment, that gives you some quality of life features, that are very valuable when you write longer texts in fields like humanities or even for journalists. As the basic idea of using Markdown and plaintext approches in general is independence from specific platforms and tools I don't think that Zettlr has to ship all features of apps that just do another job check also this reddit pos. Imo there is no problem in using various tools, scripts or maybe plugins (I read in antoher issue that there are no plans to add plugin support for now, however I think something like this may be an ideal example for something which could be implemented as a plugin and not as a core feature) to provide some features. I know there is the tendency to add as much nice features as possible to a program (I am looking at you Emacs), however I am not sure if this is always a good idea. While there may always be a debate if something deserves to be a 'core' feature or not it could clearly help to have a clear definition of the long term goals. My comment may be somehow off topic and maybe there is another plattform which is more suitable for this type of question. |
|
@lduktus well put. I would like to add (augment?) the following. On the issue of note-taking application and writing tool, I believe that Zettlr's emphasis on a nice writing environment is precisely what makes its dual-purpose as a note-taking tool so valuable. The more it excels for taking, managing, and developing notes, the better I can convert those to my finalized writing. Maybe I'm just restating what's been said though. It's just that I think what is best about Zettlr is how it enables those processes to be wed together. Those graphical/visualization tools are super interesting and might prove valuable in various ways. At the moment, I personally have absolutely no use for them--mostly a curious gimmick to me. Nevertheless, I like to follow what other people are doing with them because I suspect that there might eventually be some really good use that I just am not aware of yet. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
stale
bot
added
the
stale
|
I don't think this issue has lost any of its relevance; we're just waiting for the additions mentioned above. |
nathanlesage
removed
the
stale
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
stale
bot
added
the
stale
|
Is it someone else's turn to note that this issue is not stale, just awaiting the fixes above? :-) |
stale
bot
removed
the
stale
nathanlesage
added
the
pinned
|
@tarahmarie It's now pinned, so stalebot should leave it alone. |
|
@nathanlesage |
|
Still not at the top of the list – prior to this are some nasty bugs, the transition to Vue3 and TypeScript, and a full clean up of the code base. Currently estimating that I can actually get to this issue sometime in Spring 2022, provided that I can take a week off from work to get to the app |
|
Until then, you can try a manual visualisation of your Zettelkasten, e.g. with the help of https://github.com/mickael-menu/zk or https://github.com/joashxu/zetteltools and the original gists plus forks mentioned in https://github.com/mickael-menu/zk/issues/48#issuecomment-970612138 |
|
Since @JulienMir has unfortunately not been active on this PR for quite a while, this PR has now been superseded by 4c03381, which adds a graph view, building a little bit on the work he has done, but with some improvements with regard to data handling. A graph view is now part of Zettlr. |
Description
I really wanted the graphical view of the note network so I added it.
By selecting the button on the toolbar, a dialog pops up and displays a network of notes.
Each note is represented by a circle, links from one note to another are represented as edges between circles.
Each circle is labelled using the filename of the note (the extension and eventually the ID are removed first).
The more links (inbound and outbound) a note has, the bigger the radius of the circle representing it.
You can drag notes around to reorganize them.
Clicking on a note launches a search with the associated ID (therefore opening the note but also displaying connexe notes in the left panel).
All actions that can be done on the note (i.e. the circle) can be done on the associated label (its title)
Clicking and dragging the background lets you translate the whole network.
You can also zoom in and out with the mouse wheel.
As of now, the only information you can see besides notes and their relation is the centrality of a note.
This is indicated by the size and color of a node.
I hope this feature can grow into something more useful, displaying a variety of information.
Changes
I added a new dialog.
I changed the toolbar to display a new button (next to "tag cloud").
I modified the fsal-file to create and update a "database" containing all informations about note's relationships. This is linked to a provider.
Additional information
As it is the first time I ever did a PR, I did not want to overdo it by adding too much.
Most of the stuff I've done is based the "tag-cloud" functionality.
I copied and adapted almost all of it.
I might have added unnecessary code this way...
I did not manage to change textual informations (Like button labels, translated description, tool tips, etc.) despite adding all the necessary values in the different files (e.g. fr-FR.json, toolbar.json, etc.)
Tested on:
Windows 10 using yarn