Feat: graph visualization for directory structure

[davidde]

I recently discovered foam and I'm really liking it, especially since I have already settled on local plaintext markdown notes. The Roam Research "anti-hierarchy" approach might be great for proprietary solutions that abstract away the file system, but I'm not convinced it is the best approach for working with local files. As such, I am very happy that Foam aims to support both approaches.

I think the graph visualization is really cool, but less useful for people with well-structured note archives, where linking is of secondary importance. My first instinct was to add "toc" files in every directory. However, on second thought this might not be a great idea because of the maintenance burden. I should be able to just write about the stuff that matters without too much boilerplate maintenance.

Thinking some more about this, I believe this might be an ideal feature to support at the extension level; most people with local notes likely already have a directory structure of some sort, since things become unmanageable quickly without. It would be really nice to have built-in support for visualizing this structure. I feel this would bring utility beyond even what current proprietary solutions have to offer.

[riccardoferretti]

Hi @davidde, I think there are some interesting ideas there, a few people have mentioned some sort of "index" file per directory, and this feels like something that could be useful, although I am unclear about the shape or how to get to it.

For what regards a graph based on file path, in what way do you consider it to be better than the file explorer that comes with VsCode?

[riccardoferretti]

#421 is where we discuss having an index page

[davidde]

You're totally right that this kind of overlaps in functionality with the file explorer, and this is also why I was initially hesitant to propose it. Why I eventually did propose it, is because I currently feel like I'm "missing out" on this wonderful visualization tool that is already available. With this feature request I'm trying to gauge interest before spending too much time brainstorming an unwanted feature ;)

Why I feel this would be useful, is because directory hierarchy is actually very similar to linking; it indicates what is related. I usually do not bother to create explicit links where files are already organized by directory, and this means my current graph looks a bit meagre. Having both visualizations in a single graph, with maybe a checkbox to toggle the directory visualization on/off, feels like a winning combination to me.

One idea I had was simply a function that auto-generates these "toc" files I mentioned. A single toc file per directory, which itself would link to all its sibling files, as well as the toc files of its sibling directories. (Alternatively, this data could also be kept in foam.json or any other centralized location instead of individual toc files.) This visualization could have a different color to clarify the difference, and the "auto-generate" function could have a boolean config, making it easy to ignore if you don't care for it.

But then again, I'll totally understand if you feel this is redundant because it duplicates file explorer functionality. I'll probably experiment with toc files a bit and see if it's worth the trouble.

Thank you for mentioning #421. It sounds similar indeed, since what I am proposing is basically a visualization of such an index page. It could make sense to tie both together. I'll be keeping an eye on that issue.

[riccardoferretti]

@davidde I think your idea makes lots of sense in different ways, and it's something that Foam in a way can already support.

I agree with you that often times the path of a note gives some information about it, so it would be good to leverage that.
Currently you can achieve something like this using https://github.com/foambubble/foam/blob/master/docs/features/foam-local-plugins.md, and adding properties (e.g. the type property, which you can then style), or adding connections, .. when nodes are processed.

Also, it would actually be pretty straightforward to completely reuse the graph code with another one that is based on different metadata (in this case, file structure). The only change would be the "translation" of the Foam model to the graph model. If you are curious, just look at the generateGraphData in dataviz.ts.
In fact, another approach is to replace the File Explorer in the first place, with something that can leverage the Foam Model (e.g. that would allow us to change the icon based on the note type, integrate directory structure with links, ...). Now that in VsCode it's possible to have webview panels, this would totally be possible.

The reason why I am hesitant to put it in the extension itself is maintenance of the feature, this feels like would be a good extension on top of Foam - which will soon be possible as we lock in our data model v1.

To sum up, I think the idea is pointing in the right direction, will be good to get votes on it to see how much people care about it, and to see what comments we get here to think it through.

[davidde]

Sounds good. All great points.

I was hoping the graph code would be reusable. Now that I think about it, it might even lend itself well to enable a third visualization mode of #tags, which would be especially useful since tags can be used multiple times in the same document.

I'll look into the local plugins and generateGraphData.

[riccardoferretti]

paging @jmg-duarte as he has done some work on the graph, has some good ideas about how to move it forward and make it its own module - so might be interested in the thread, FYI

[maciejmatczak]

I would love to have directories act as nodes in a graph.

Currently I have my knowledge organized as:

knowledge/
  dev/
    python/
      profilers.md

... and so on, but that makes a profilers.md just floating, unrelated article. I am not sure if I should change my note taking style (I feel one weekend with Dendron changed me ;) ).

It's easier for me to navigate in VS Code through the folder relations I designed (ex.: CTRL-P, type py pro, first hint is correct) whether having everything in one bucket and keep relations in wiki links. How in such case I should search for python related profilers information (assuming I would like to have a similar articles for many languages).

File name conflicts though...

[bsecker]

I have a very hierarchical structure to my notes, much like @maciejmatczak, and have also been interested in having some sort of "index files" per subdirectory to make the implicit connections between similar notes in a directory explicit on the graph. So, I whipped up a quick python script for myself that walks through the directory tree and builds an index file with the same name as it's containing directory, and it has links to contained pages and subdirs.

There's some basic filtering (excluding certain folders and excluding folders with a README.md already in them). Figured I'd share the script in case anyone wanted to modify it for their purposes.

As an example from my wiki, It ends up generating index files that look like this:

---
type: readme
---

# Dev Ops

Subdirectories:
- **[Devops-Days-Conference](./Devops-Days-Conference/devops-days-conference.md)**
- **[LinuxConf-2021](./LinuxConf-2021/linuxconf-2021.md)**

Pages in this directory:
- [cool-bash-scripts](cool-bash-scripts.md)
- [functional-programming](functional-programming.md)
- [gitops](gitops.md)
- [hashicorp-vault](hashicorp-vault.md)
- [hosting](hosting.md)
- [infrastructure-as-code](infrastructure-as-code.md)
- [pki_notes](pki_notes.md)

Here's the effect it had on the foam graph:

The downside with this (as discussed earlier in this issue with the 'toc file' approach) is the maintainability aspect. I initially used gitlab CI to automatically run this script and let it overwrite existing files, which kept things up to date, but meant that if I ever wanted to write on the index page the changes would be lost.

It would be great to have this functionality natively in foam so I don't have to rely on my hacky script. The ideas discussed earlier sound like a great way forward!

[riccardoferretti]

this could be an interesting feature candidate, if someone proposes a PR for it, I would consider including it in core if it seems to make sense to me (no promises ;) ).
The place to augment the graph would be the dataviz.ts file.

And at minimum this seems like a good area to include in the Foam API, so that if we don't include this in core, it would still be accessible to a Foam plugin to achieve the same.

cc @jmg-duarte

[cliffordfajardo]

Related to visualizing hierachy's I've been using the the @antv/G6 graph visualization library for other projects at work.

One feature that G6 offer's is rendering "combos", which allows grouping you to group nodes. A combo can have other combo's inside as well. Below is a concept example, but for our use cases, imagine that the combo is used to represent a folder:

Codesanbox of super basic combo: https://codesandbox.io/s/g6-basic-combo-demo--foam-vscode-concept-1xxkd4

References

• @antv/G6 docs - Overview of combos
• @antv/G6 docs - Combo Theory
• @antv/G6 discussion - Combo with custom Icon
• @antv/G6 combos can be different colors, sizes, shapes etc

[KillyMXI]

ExcaliBrain (Obsidian plugin) uses folders as a separate kind of node similar to tags.

(And it also has other ideas to borrow...)