Allow plugins to add links between notes

[HEmile]

In a plugin I'm going to develop with @SkepticMystic, we would like our plugin to manually add links between notes. This has also been requested in DataView use cases, where users expect the outputs of DV to link from the source note to the outputs of DV.

We might be able to hack around with resolvedLinks and the ReferenceCache, but that sounds challenging and unreliable.

[SkepticMystic]

So we would like there to be some method, probably on metadataCache, to create a link between A and B, even though no real link exists in the md content of either of these notes

[lishid]

I think I understand the use case, but I'm a bit hesitant on the implementation. Links are qualified as a plaintext with a source file and a dest file, and a start/end position for where the link is located in the file. resolvedLinks is a superficial cache/index for "are these two files linked" and is re-computed from the underlying link data in the metadata cache using various triggers.

Depending on where you want your link to show up, the injection process would possibly have to be done on file parse (which we currently do in a Worker, making it more challenging to inject), or after the fact.

Doing it on parse has the additional challenge of cache consistency with respect to plugin enable/disable. Suppose your plugin adds the additional information only on parse, then previously parsed files won't get the new information the moment your plugin is enabled. In addition, there's no way to cleanly disable/uninstall your plugin should it have lasting effects to the persistent metadata cache. If your plugin modifies the cache such that it causes Obsidian to crash on startup, then there's no fixing that unless one has a way to 'clear out' that cache (currently unnecessary as Obsidian's cache is internally consistent and self repairing).

The other way is to have your plugin compute and augment the cache on every boot up, which would require additional data structures to separately store what's original and what's added by the plugin. This could also make it such that current core/plugin features depending on the data structure to require an update to pickup the new format.

In either case, there are also additional challenges to also iron out, such as the "file rename link auto update" utility possibly getting confused when encountering plugin added fake links.

But coming back to the request - what do you mean by "adding links"? Are you hoping they'd show up in Graph view? Backlinks? Forward links? What "text" would it highlight when rendering the backlink result? How would other plugins interact with this data?

[HEmile]

I see I didn't respond here. It is good to know this background, it does indeed sound like a significant challenge.

But coming back to the request - what do you mean by "adding links"? Are you hoping they'd show up in Graph view? Backlinks? Forward links? What "text" would it highlight when rendering the backlink result? How would other plugins interact with this data?

Yes, to most of these. In particular the graph view, and it being in resolved links and referenceCache so other plugins (eg Juggl and Breadcrumbs) can handle them.

[nothingislost]

This is way too long but it is my attempt at explaining portions of the Obsidian caching system at a high level since it was hard for me to wrap my head around how things were working until I wrote it out. Feel free to point out any inaccuracies.

There are two types of cache, ephemeral and persistent.

The persistent cache holds aggregated results of the remark parser in an IndexedDB. For each file in the vault, the persistent cache holds a list of links, embeds, list items, sections, tags, frontmatter... and their respective positions in the document. This cache is primed from scratch one time, the first time you load Obsidian (or if the index is deleted). The persistent cache is continuously updated via various event triggers, such as rename, create, delete, etc.

It would be difficult to alter the content/population of the persistent cache for the reasons Licat details above. Since the cache is persistent, it would be difficult to undo any changes to the cached content that were made by a plugin. It would also cause side effects to all of the logic that currently relies on the persistent cache and makes assumptions about its contents and structure.

The ephemeral cache is initialized on application startup, leveraging the data from the persistent cache. The most interesting data in the ephemeral cache is the list of resolved and unresolved links. These data structures are used to build out the relationships between graph nodes and are also continuously updated via various event triggers.

The ephemeral cache seems easier for plugins to modify since, on plugin unload, the ephemeral cache could just be rebuilt on the fly. This isn't a cheap operation but it typically completes in under a second.

In the case of the resolved/unresolved link cache, a link resolver process iterates over all markdown files in the vault and checks to see if the links in each file resolves to an actual document. The components involved in this process are:

• A link resolver queue: MetadataCache.linkResolverQueue
• A link resolver orchestrator: MetadataCache.linkResolver()
• A link resolver worker: MetadataCache.resolveLinks(srcFilePath)

On app startup, all markdown files within the vault are passed into the linkResolverQueue and the results are stored in the resolved and unresolved link cache.

I see two use cases for modifying the ephemeral cache

1. Registering custom link resolvers

This is the use case I've been playing around with. In my case, I'm adding additional file resolution logic to getFirstLinkpathDest(linkText) that does a lookup to see if linkText resolves to any of the aliases currently defined within the vault. It will only resort to alias resolution if the default getFirstLinkpathDest logic does not resolve to an actual file.

With getFirstLinkpathDest patched, this allows bare [[aliases]] to work within the graph, embeds, backlinks, etc. This works because getFirstLinkpathDest is the method used to resolve link text pretty much everywhere within the app, including the link resolver process.

This method is useful when your links are already being parsed but you want to add additional logic to get them to resolve to the appropriate document.

2. Registering custom relationship builders

This use case is more in line with what Emile and SkepticMystic are asking for. In this case, a developer wants to establish their own relationship between documents. Use cases for custom relationships might be:

• There are already tangible links in the document that establish relationship but those links are inside blocks that do not allow link parsing (frontmatter, code blocks, etc).
• There are dynamically generated links in the document that the remark parser will just never see (see this Dataview issue)
• There are implicit relationships in the document defined through some custom logic. These relationships may be derived via frontmatter values or other means.

Ideally a custom relationship would show up in backlinks in addition to connecting up the graph nodes but the former would require changes to the persistent cache. If we focus on just the graph node connections for now, that might be easier to enable.

The challenge with building own relationships is accessing the data needed to drive your custom relationship logic in a performant way. If you are able to build your custom relationships using data already found in the persistent cache, such as frontmatter values, this would work well.

I've created an example of how you could create and manage your own graph node relationships using a frontmatter key called parent (à la breadcrumbs) here https://gist.github.com/nothingislost/23b6dbfcba6a48ad5e1c75e8bdee261f

If the data needed to build your custom relationships is not already cached, such as links inside of code blocks or dataview query results, then you'd need to first create and manage your own cache. Once you had the caching problem solved, the method referenced above could be used to inject your custom relationships.

This approach is brittle though and makes assumptions about how the graph view relationships are built internally. Having a more sanctioned way to add node relationships would make this easier to rely on.

tl;dr Modifying the persistent cache is hard but a simple interface for adding custom graph node relationships might be fairly painless to expose.

[dennisorlando]

A solution for this would be to cheat and create a .custom-plugin directory which becomes invisible in Obsidian. In there you can play with links and files as much as you want.
I think the only thing that has to be added for this to work is something to access ignored and not-ignored folders.
Something like "app.vault.unignorefolder" would be sufficient for this usecases. We would run it in when the plugin is loaded

EDIT: https://forum.obsidian.md/t/hide-select-vault-folders-from-obsidian/5744/26 already suggested something like this.
It would be as simple as "app.vault.hidefolder" in order to not show specific folders inside Obsidian's file explorer
I'm still not much familiar with the API which means I'm not sure if that feature has been already implemented.