Support non-flat filesystem hierarchy

[NightMachinery]

I already have my notes in markdown, but I organize them in a hierarchy of directories. It seems to me that supporting this shouldn't be very hard? As I have encoded a lot of information in the hierarchy, I don't wish to move them to a flat structure. The hierarchical organization is also supported by generic software very well; I use WorkingCopy on iOS to edit my notes, and it's quite up to the task, even though it's just a general git client.

[NightMachinery]

After playing with neuron for a bit, I see its special sauce as its tree visualizer and the backlinking. I think this should be implementable on a normal directory, as it only needs to scan the directory and build a graph of the links. I think giving up the directory structure is a very costly move for people who don't like being limited to a specialized tool. I can now do things like this, with a few lines of shell:

Implementing such a thing with neuron would require a lot of special software, because the hierarchy is opaque. Of course, the flat structure has the advantage that you don't need to put things under a prespecified category; But my experience is that a lot of stuff lend themselves best to such top-down categorization. Since the two approaches can co-exist (everything goes by default into the uncategorized root, but the user can move them freely into subdirectories.), why not offer them both?

[srid]

Are you asking about supporting note files recursively (discarding the hierarchy)
? Or, are you asking about mapping the filesystem hierarchy as-is to neuron's folgezettel heterarchy?

[hazelweakly]

Neuron already supports hierarchical tags. Perhaps the file system tree could be mapped 1:1 with tags and otherwise zettels would link to each other normally?

[NightMachinery]

@jared-w commented on Jul 31, 2020, 5:22 PM GMT+4:30:

Neuron already supports hierarchical tags. Perhaps the file system tree could be mapped 1:1 with tags and otherwise zettels would link to each other normally?

This is a good idea. I think to facilitate the linking, neuron might need to change its linking scheme a bit. As far as I understand, currently, neuron uses IDs as filenames to make changing the title of notes costless. A better approach might be to allow both an ID and a title. The title won't be included in the links, only the ID. The title's only use is to allow interoperability with other tools. (For example, I use fzf to jump between my notes, and I need meaningful names for each note for this to work.) This title can be further used as the last tag in the hierarchy of tags. To illustrate:

/dev/zsh/testing/ZUnit.SOMEID.md would be a note with ID SOMEID and in the hierarchy dev -> zsh -> testing -> ZUnit.

[srid]

Neuron already supports hierarchical tags. Perhaps the file system tree could be mapped 1:1 with tags and otherwise zettels would link to each other normally?

This is what I was thinking. One could write an import script that converts the hierarchical notes directory, recursively, to a flat list of files each tagged accordingly, in addition to creating "portal zettels", for intermediate directories, that queries it's contents by tag.

So, for example, given /dev/zsh/testing/ZUnit.md, this would create 4 zettels:

• dev-zsh-testing-ZUnit.md tagged dev/zsh/testing
• dev-zsh-testing.md tagged dev/zsh (and queries dev/zsh/testing/*)
• dev-zsh.md tagged dev (and queries dev/zsh/*)
• dev.md (queries dev/*)

[srid]

@NightMachinary There is no good reason to change the ID scheme and complicate it unnecessarily. You can use title (slug) as the ID if you want. Neuron doesn't care as long as they are unique, and are alphanumeric (with hyphen and underscore). I recommend that you give the above approach a try.

[jaysnoddy]

I would like it if neuron could eventually support at least one level of subdirectories. I use four "bins" or subfolders that really seldom need cross-referencing among the bins. It was annoying to me if kept in the same folder. My bins/subdirectories are, as follows: Academic work, personal technology notes (e.g. program documentation, cheat sheets, boilerplate, etc.), Cooking/Recipes, and Current News. I have made some aliases to take away some of the complexity. However, serving a localhost for each seems messy. Others might, of course, have different bins. Others may only use neuron for one type of notes.

(Analog-analogy or pseudo-joke: Given the presence of slipnote paper, I would not be surprised if the Luhmann household used slipnotes for recipes, but I would be surprised if 'how to make Aunt Gertrude's apple strudel' ended up in the same zettlelkasten as his academic work.)

[srid]

@jaysnoddy I think the cross-links idea mentioned at #355 may very well cover your use case.

[bangedorrunt]

i think this would be cool if you could implement something like dendron. so we don't really sub folders. just name the file based on our convention

[fiatjaf]

I think this issue should be considered if neuron is to become a "complete" note-taking system that tries to support all kinds of workflows. If it wants to remain close to the Zettelkasten idea, then this is a distraction. I say this without deep knowledge of Zettelkasten, but I think doing less things is better.

[felixyz]

I would be perfectly happy if Neuron simply recursively picked up all markdown files from all subdirs and then proceeded to handle them just as if they were all in a single directory. Of course, then the problem is possible name clashes, so the id of foo/bar/qux.md would have to be foo/bar/qux. (Or Neuron could protest and demand that the files have different titles to use as id?)

A bit of sugar on top of that would be to the ability resolve links relative to the current file's location, like how module references work in most programming languages. So foo/bar/baz.md could refer to foo/bar/qux.md with [[qux]] or [[foo/bar/qux]], foo/guz.md could refer to it with [[bar/qux]] etc. This would make it easier if one decided to change the file hierarchy (although it would still not be free), and very intuitive I'd say, at least for programmers.

Finally, it would be a nice to have if Neuron would optionally tag all files with their path, so that qux would get the tag foo/bar.

For all of us who already keep our notes in a file tree, I think the above would let us get productive with Neuron right away without having to change anything about our existing notes.

[srid]

For those interested in this feature, I've pushed experimental support in the master branch. I'm not advertising it widely (yet) because this feature is likely to change, or even go away (if it proves to be problematic in some way). I'm publishing it for testing and feedback before we finalize anything.

To use this feature, install latest neuron and add recurseDir = True to your neuron.dhall.

EDIT: docs https://neuron.zettel.page/subdir.html

[felixyz]

Just tried this out and works great. As expected, I got warnings about id clashes. Are you leaning towards not doing anything to help resolve these? (By eg taking the file path into account?) I'd say that's reasonable, would introduce complexity to support what seems to be a rather small group of users who already have or want hierarchical file organization.

Auto-tagging might still be a helpful thing, and doesn't really affect anything else? (My suggestion above: "...optionally tag all files with their path, so that the file foo/bar/qux.md would get the tag foo/bar")

[srid]

A notion of "namespaced linking" could be considered. So [[qux]] can also be explicitly linked to as [[bar/qux]] (from a note inside foo/ directory) or [[foo/bar/qux]]. This should take cross links proposal in #355 into consideration. We should combine them, and come up with a simple proposal - that satisfies the two use cases. See also "linking strategy" of vscode-memo: svsool/memo#131

Auto-tagging is a separate feature, possibly to be implemented as a plugin. cf. https://github.com/b0o/neuron-extras#demo

[flyinggrizzly]

I've just started testing this out, and it may also be worth considering adding an option to ignore certain directories--I have a bin/ directory in my repo (with tiny scripts for running the local server and syncing the changes with git), and keep a readme.md file in there because I'm dumb and forget things... but that readme (along with any other MD file in the bin/ directory) shouldn't be considered a note.

[srid]

In that case, we can just replace both recurseDir and formats with a singular option:

patterns = [ "*.md" ]

^ That's the default, but it can be changed to [ "*.md", "foo/**.md", "bar/**.md" ]. That is, recurse - but only in ./foo and ./bar subfolders. A whitelist instead of blacklist. The original recurseDir behaviour can be specified using [ "**/*.md" ]. See https://hackage.haskell.org/package/filepattern-0.1.2/docs/System-FilePattern.html for spec for these patterns.

[flyinggrizzly]

I like consolidating into a single option, but might that not become burdensome if people add multiple directories? Instead of just adding it and then adding notes into it, you now also have to update the neuron config.

I don't know how this works for others, but I'm far more likely to want to add just one or two entries to a block-list than add every note-directory to an include-list.

It spins out another config option, but could it also work to have patterns and ignorePatterns? Like:

patterns = [ "**/*.md" ],
ignorePatterns = [ "bin/*" ],

[srid]

It spins out another config option, but could it also work to have patterns and ignorePatterns?

Sounds good to me. That's more ergonomic.

[flyinggrizzly]

Just a usage note--turning on recurseDir is surfacing a ghost README file that I can't get rid of for love nor money.

It looks like the default new project README file, so I assume it's coming in from Neuron's code itself, but tree . | grep README turns up nothing.

This isn't stopping me from using the feature for the moment, so not a big deal, but worth noting it's happening.

This is #141

[srid]

That's probably a different issue -- see #141

[flyinggrizzly]

Yup--wasn't seeing it at first because tree wasn't displaying dot-directories initially 😅

[srid]

There is a proposal for taking this feature to its natural next step by making the directories folgezettels. See #497

[srid]

I'm inclined to choose the following style of configuration for including/excluding Markdown files in a recursive traversal scenario and otherwise:

{ exclude = [".*", "README*"]
, ...
}

This example will read Markdown files recursively, excluding dotfiles (eg: .foo.md) and READMEs.

There is no "include" here, as I don't see a need for it; and then recursive traversal is enabled by default. The previous behaviour of looking for only top-level Markdown files can be resurrected by configuring as follows:

{ exclude = [".*", "README*", "*/*"]
}

(The third entry excludes every second level file/directory; effectively limiting notes to top-level)

[srid]

Now available: https://neuron.zettel.page/dirtree.html

Give it a try; the documentation can certainly be improved (PR welcome!).

[lambtho12]

I just read this discussion and I still have some questions (I am a new user of Neuron, so maybe I am just misconfiguring something)

I use Vim with the vimwiki plugin to write and do most of my production work, while Neuron is used to display the notes in a more usable way. As discussed here above, I also have a "rather flat" hierarchy of folders (one single level of directories after root). I use the dirtree plugin and it seems to work as intended.
Now I would like to use the "full path" of the notes when linking them. So a note called foo/bar.mdwould be linked with [[foo/bar]] or [[/foo/bar]] and not just [[bar]]. The reason is that by using the full path, I will also be able to navigate in the notes with vimwiki.

Currently if I do that, it will not find the note it is looking for. Is it something possible with neuron? Coudn't it be something worth adding?

[srid]

I'm certainly open to it as a PR if somebody is up for it, as long as it is optionally configurable in neuron.dhall.

[srid]

Alternatively, vimwiki can be somehow told to find notes under sub-directories? Both vscode-memo (vscode) and markdown-mode (emacs) does this already.

See jrblevin/markdown-mode#590

[srid]

@lambtho12 Emanote now implementes this. https://note.ema.srid.ca/

[srid]

Those that rely on filesystem hierarchy to structure their notes, I recommend trying out the neuron successor, Emanote.

You can use partial paths in links, eg: [[Foo/Bar]]. And also link to files using wiki-links, using same semantics.

[chris-sanders]

Emanote looks interesting, but I notice it lacks uptree or folgezettel support. Those are the primary reason I was looking at starting to use neuron. At this point should I be starting with neuron or have you stopped development and I should just consider Emanote?

I would be happy to use neuron as is today, but as above need support for linking into subfolders so my LSP engine can find links correctly for journal entries which are in their own subfolder.

[srid]

@chris-sanders Emanote recognizes folgezettel links. Next on my plate is to add uptree and z-index support (based on folgezettel). Once that's done, it will be ready as neuron replacement. See srid/emanote#25 for uptree

At this point should I be starting with neuron or have you stopped development [...]

I just put this page up officially announcing neuron to be in maintenance-only mode: https://neuron.zettel.page/next

and I should just consider Emanote?

I'd say yes. Or wait until uptree is implemented in Emanote (edit: should be done this weekend). The main thing that you will need to change is the query syntax. https://note.ema.srid.ca/guide/neuron

[chris-sanders]

Ok thanks I'll just start with ema then.

[srid]

@chris-sanders This is now available in emanote master. To use the neuron-like layout (with uptree), update emanote and add the following to your index.yaml:

https://github.com/srid/www.srid.ca/blob/8531793c8f4d6fca7341e4d336cf9222dc7ce7f4/index.yaml#L10

I just switched my own website over to it.

Try it out, and let me know how it works. Let's continue this discussion in Emanote github discussions area.

---

Demo https://www.srid.ca/haskell-new-project