Refactor code to have clean interface between parse layer and write to database and focus tests on parse layer

[rufuspollock]

Refactor to focus the code on the first steps to get a JS/TS objects representing essential "data". Create clear separation of concerns so that we can rapidly add new well tested features e.g. tag parsing, computed fields.

graph TD

markdown --remark-parse--> st[syntax tree]
st --extract features--> jsobj1[TS Object eg. File plus Metadata plus Tags plus Links]
jsobj1 --computing--> jsobj[TS Objects]
jsobj --convert to sql--> sqlite[SQLite markdown.db]
jsobj --write to disk--> json[JSON on disk in .markdowndb folder]
jsobj --tests--> testoutput[Test results]

Comment: we should write most of our tests at the js object level. Not at sqlite level.

Acceptance

See design sketch below.

• process.ts with a processFile function exists which given a path returns a File object or a derivative thereof ✅2023-11-15 this is done - see https://github.com/datopian/markdowndb/blob/8fcda1db4fc2b19eade703c25f91438eb6e69d30/src/lib/process.ts
  • supports frontmatter extraction ✅2023-11-15 see tests https://github.com/datopian/markdowndb/blob/8fcda1db4fc2b19eade703c25f91438eb6e69d30/src/lib/process.spec.ts
  • supports links ✅2023-11-15 ditto
• has tests ✅2023-11-15 see https://github.com/datopian/markdowndb/blob/8fcda1db4fc2b19eade703c25f91438eb6e69d30/src/lib/process.spec.ts

FUTURE

• Bonus: Split out the sqlite wrapper to write to sqlite in a separate file and refactored current code to use the processFile function

Comment: we can ignore sqlite right now in this branch not worry about refactoring that code to write to sqlite from JS objects. This would simplify code and allow us to use typescript for everything for now. It is easy to convert json to sqlite later if we want.

Notes

• Use micromark to parse the markdown file? ✅2023-11-10 ❌ just use remark-parse for now

Sample code

// FUTURE - to show how code path will run
function buildMarkdownSQLDb(folder) {
  walkFolder(folder)
  for each file in folder:
    parseFile => File
    storeFileInDb(File) // or storeAllFiles at once later
}

// what we want to add ...
function parseFile(path) returns File
  getKeyInfoOnFile(path)
  if(isMarkdown) {
    parseMarkdownFile(string) // returns Metadata, Links, Tags, Tasks
    // add this info to the FileInfo ...
  } else {
    return FileInfo
 } 
}

interface FileInfo {
  path
  
}

interface MarkdownFile extend File {
  // adds Metadata
  // adds Tags?
}

Schema

interface File {
  path: string; // relative path on disk, also will be primary key
  extension: string;
  metadata: MetaData | null; // frontmatter
}

interface Metadata {
  key: value
}

interface Link {
  link_type: "normal" | "embed";
  from: string; // path
  to: string; // to path or url
}

Design sketch for JSON on disk

NB: ❌ we don't need this atm

.markdowndb/
  files.json
  links.json

Define the schema for that in typescript

[rufuspollock]

FIXED. Completed in 8fcda1d

Took longer than estimated b/c confusion over whether we refactored existing code to use the new function.