Research Obsidian dataview approach to a markdown db #5
Assignees
Labels
Comments
43 tasks
|
@demenech please dump any notes you have so far and then let's pause this for now. |
Dump - 2023-03-22 (not reviewed)NotesWhat's the database structure?TL;DR:
The DB operations are defined in /** Load file metadata by path. */
public async loadFile(path: string): Promise<Cached<Partial<PageMetadata>> | null | undefined> {
return this.persister.getItem(this.fileKey(path)).then(raw => {
let result = raw as any as Cached<Partial<PageMetadata>>;
if (result) result.data = Transferable.value(result.data);
return result;
});
}The /** All extracted markdown file metadata obtained from a file. */
export class PageMetadata {
/** The path this file exists at. */
public path: string;
/** Obsidian-provided date this page was created. */
public ctime: DateTime;
/** Obsidian-provided date this page was modified. */
public mtime: DateTime;
/** Obsidian-provided size of this page in bytes. */
public size: number;
/** The day associated with this page, if relevant. */
public day?: DateTime;
/** The first H1/H2 header in the file. May not exist. */
public title?: string;
/** All of the fields contained in this markdown file - both frontmatter AND in-file links. */
public fields: Map<string, Literal>;
/** All of the exact tags (prefixed with '#') in this file overall. */
public tags: Set<string>;
/** All of the aliases defined for this file. */
public aliases: Set<string>;
/** All OUTGOING links (including embeds, header + block links) in this file. */
public links: Link[];
/** All list items contained within this page. Filter for tasks to get just tasks. */
public lists: ListItem[];
/** The raw frontmatter for this document. */
public frontmatter: Record<string, Literal>;
public constructor(path: string, init?: Partial<PageMetadata>) {
...
}
/** The name (based on path) of this file. */
public name(): string {
return getFileTitle(this.path);
}
/** The containing folder (based on path) of this file. */
public folder(): string {
return getParentFolder(this.path);
}
/** The extension of this file (likely 'md'). */
public extension(): string {
return getExtension(this.path);
}
/** Return a set of tags AND all of their parent tags (so #hello/yes would become #hello, #hello/yes). */
public fullTags(): Set<string> {
let result = new Set<string>();
for (let tag of this.tags) {
for (let subtag of extractSubtags(tag)) result.add(subtag);
}
return result;
}
/** Convert all links in this file to file links. */
public fileLinks(): Link[] {
// We want to make them distinct, but where links are not raw links we
// now keep the additional metadata.
let distinctLinks = new Set<Link>(this.links);
return Array.from(distinctLinks);
}
}Note that tags are typed as
/** The Obsidian 'link', used for uniquely describing a file, header, or block. */
export class Link {
/** The file path this link points to. */
public path: string;
/** The display name associated with the link. */
public display?: string;
/** The block ID or header this link points to within a file, if relevant. */
public subpath?: string;
/** Is this link an embedded link (!)? */
public embed: boolean;
/** The type of this link, which determines what 'subpath' refers to, if anything. */
public type: "file" | "header" | "block";
...
}And lists are defined as arrays of /** A list item inside of a list. */
export class ListItem {
/** The symbol ('*', '-', '1.') used to define this list item. */
symbol: string;
/** A link which points to this task, or to the closest block that this task is contained in. */
link: Link;
/** A link to the section that contains this list element; could be a file if this is not in a section. */
section: Link;
/** The text of this list item. This may be multiple lines of markdown. */
text: string;
/** The line that this list item starts on in the file. */
line: number;
/** The number of lines that define this list item. */
lineCount: number;
/** The line number for the first list item in the list this item belongs to. */
list: number;
/** Any links contained within this list item. */
links: Link[];
/** The tags contained within this list item. */
tags: Set<string>;
/** The raw Obsidian-provided position for where this task is. */
position: Pos;
/** The line number of the parent list item, if present; if this is undefined, this is a root item. */
parent?: number;
/** The line numbers of children of this list item. */
children: number[];
/** The block ID for this item, if one is present. */
blockId?: string;
/** Any fields defined in this list item. For tasks, this includes fields underneath the task. */
fields: Map<string, Literal[]>;
task?: {
/** The text in between the brackets of the '[ ]' task indicator ('[X]' would yield 'X', for example.) */
status: string;
/** Whether or not this task has been checked in any way (it's status is not empty/space). */
checked: boolean;
/** Whether or not this task was completed; derived from 'status' by checking if the field 'X' or 'x'. */
completed: boolean;
/** Whether or not this task and all of it's subtasks are completed. */
fullyCompleted: boolean;
};What is indexed? e.g. tags, tasks (where is code for this - see next item)
What code does the indexing/parsing?pages and markdown files
tags extraction e.g. #tag-name
tasks extraction
What's the query API?DataviewAPI query function (https://github.com/blacksmithgu/obsidian-dataview/blob/81ba6a0dd31d6562de852144112922bb33e084d9/src/api/plugin-api.ts#L264): public async query(
source: string | Query,
originFile?: string,
settings?: QueryApiSettings
): Promise<Result<QueryResult, string>> {
...
This function calls TODO |
rufuspollock
changed the title
|
Our intuition (though we should check) is you can't reuse outside obsidian. However, we can take inspiration. |
|
As we suspected this seems dependent on I installed the plugin and tried using it: and got following error. You have the following at the top of the index.js |
|
FIXED. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Obsidian dataview contains a sophisticated markdowndb index. its open source and we could learn from or even reuse some of.
In progress notes about obsidian where we could include these: https://datahub.io/notes/obsidian
Acceptance
The core question we want to answer:
importetc or is it somehow dependent on obsidian (NB: this is the preferred option if possible. let's not reinvent the wheel) 💬2023-11-09 preliminary research back in February here Accessing dataview API or database outside of obsidian blacksmithgu/obsidian-dataview#1811 ✅2023-11-10 i don't think this is possible - see Research Obsidian dataview approach to a markdown db #5 (comment)We have researched what dataview does https://github.com/blacksmithgu/obsidian-dataview work? specifically ...
#tag-nameThe text was updated successfully, but these errors were encountered: