Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Fix blocks (Lexer/Parser rework) #104

Closed
wants to merge 145 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
145 commits
Select commit Hold shift + click to select a range
b3ba38b
Enable noImplicitAny and strictNullChecks
GuiltyDolphin Jul 14, 2021
60fe5f5
Nit some formatting
GuiltyDolphin Jul 14, 2021
454fbaf
Add some tests for numberOfLines
GuiltyDolphin Jul 14, 2021
6ab1777
Switch to using Point and Position from unist
GuiltyDolphin Jul 14, 2021
6702dbf
Add jest configuration
GuiltyDolphin Jul 14, 2021
d30f0eb
Add some more numberOfLines tests
GuiltyDolphin Jul 14, 2021
1817b5d
Fix some bugs with toIndex
GuiltyDolphin Jul 14, 2021
4161f66
Ensure number of lines of empty string is 0
GuiltyDolphin Jul 14, 2021
d3a2ef9
Remove redundant type assertions
GuiltyDolphin Jul 14, 2021
2545e81
Rework location
GuiltyDolphin Jul 14, 2021
95a372c
Add tests for shift
GuiltyDolphin Jul 14, 2021
a216dc3
Add docstring for shift
GuiltyDolphin Jul 14, 2021
26d6b77
Add some tests for linePosition
GuiltyDolphin Jul 14, 2021
1836527
Add lengthOfLine helper
GuiltyDolphin Jul 14, 2021
3ffac1e
Fix up substring
GuiltyDolphin Jul 14, 2021
735eeab
Fix match
GuiltyDolphin Jul 14, 2021
283b817
Add TextKit type
GuiltyDolphin Jul 14, 2021
a5a55b5
Rewrite match/substring to not include end
GuiltyDolphin Jul 15, 2021
46f93b6
Add test for location inverse of toIndex
GuiltyDolphin Jul 15, 2021
9e8c48e
Add eof helper
GuiltyDolphin Jul 15, 2021
80ef3f2
Add eol helper
GuiltyDolphin Jul 15, 2021
93e305f
Add lastNonEOL helper
GuiltyDolphin Jul 15, 2021
c26b8a7
Fix incorrect linePosition calculation
GuiltyDolphin Jul 15, 2021
4d196ba
Add some more match tests
GuiltyDolphin Jul 15, 2021
889a86f
Refactor test helpers
GuiltyDolphin Jul 15, 2021
e1f97fb
Update orga for new text-kit
GuiltyDolphin Jul 16, 2021
d9c5c30
Relax version requirement for @types/unist
GuiltyDolphin Jul 16, 2021
790502e
Simplify match
GuiltyDolphin Jul 16, 2021
02b2efc
Specify known indices where possible
GuiltyDolphin Jul 16, 2021
cc42b28
Optimise toIndex when there is an offset present
GuiltyDolphin Jul 16, 2021
89b9510
Move distance into text-kit
GuiltyDolphin Jul 16, 2021
55f4119
Add some tests for carriage returns
GuiltyDolphin Jul 16, 2021
cd6b0a5
Add note about virtual point for eof
GuiltyDolphin Jul 16, 2021
c2a69c5
Add extra test
GuiltyDolphin Jul 16, 2021
da742ec
Split up TextKit (slightly) into extra/core
GuiltyDolphin Jul 16, 2021
cf1d9cd
Add bol helper
GuiltyDolphin Jul 16, 2021
d8cdda2
Add bof helper
GuiltyDolphin Jul 16, 2021
09fa48e
Add charAt helper
GuiltyDolphin Jul 16, 2021
e5b0a3c
Remove toIndex from exports
GuiltyDolphin Jul 16, 2021
503033a
Move linePosition out of Core
GuiltyDolphin Jul 16, 2021
e344e3e
Add fixPoint helper for fast point fixing
GuiltyDolphin Jul 17, 2021
0bd73ad
Add Point type to make use of line+col clear
GuiltyDolphin Jul 17, 2021
1422246
Update tryTo actions to have void return
GuiltyDolphin Jun 28, 2021
6ace35b
Nit some formatting
GuiltyDolphin Jun 29, 2021
1aeb62b
Add local definitions of Node, Parent, Literal
GuiltyDolphin Jun 29, 2021
8ee3321
Move paragraph builders into utility file
GuiltyDolphin Jun 29, 2021
c1782d7
Add sensible defaults to Extra type
GuiltyDolphin Jun 29, 2021
05a5aca
Add footnote helper types
GuiltyDolphin Jun 29, 2021
030a939
Add type for isWhitespaces
GuiltyDolphin Jun 29, 2021
dd734bf
Add testParse helper
GuiltyDolphin Jun 29, 2021
563750c
Nit
GuiltyDolphin Jun 29, 2021
f5b4852
Fix debug
GuiltyDolphin Jun 29, 2021
4081d86
Nit
GuiltyDolphin Jun 29, 2021
298113d
Simplify some function types
GuiltyDolphin Jun 29, 2021
43793e3
Add some parse tests for existing block behaviour
GuiltyDolphin Jun 29, 2021
5203a73
Fix unopened/unclosed blocks not being parsed
GuiltyDolphin Jun 29, 2021
a3099f1
Add test for unclosed property drawer casing
GuiltyDolphin Jun 29, 2021
6596765
Switch use of modify to use substring
GuiltyDolphin Jun 29, 2021
1273217
Refactor some tests
GuiltyDolphin Jun 29, 2021
e94e776
Ensure unopened drawers are treated as text
GuiltyDolphin Jun 29, 2021
809c1a0
Ensure unclosed drawers are treated as text
GuiltyDolphin Jun 29, 2021
d9dffea
Refactor drawer tests
GuiltyDolphin Jun 29, 2021
55264ff
Move drawer tests to new file
GuiltyDolphin Jun 29, 2021
7fde655
Move parser footnote tests to separate file
GuiltyDolphin Jun 29, 2021
e35d7b4
Ensure lists only contain list items
GuiltyDolphin Jun 29, 2021
b51316d
Pretend to use unist-node externally
GuiltyDolphin Jun 30, 2021
1df02e4
Remove old use of Node
GuiltyDolphin Jun 30, 2021
465622a
Revert "Pretend to use unist-node externally"
GuiltyDolphin Jul 1, 2021
34cd8f4
Add parse debug support for disabling colour
GuiltyDolphin Jul 3, 2021
dd4f7b1
Add initial support for parsing block contents
GuiltyDolphin Jul 3, 2021
305be45
Ensure greater blocks ignore name casing
GuiltyDolphin Jul 3, 2021
24042af
Simplify greater block name property
GuiltyDolphin Jul 3, 2021
754e26d
Ensure that special block contents are parsed
GuiltyDolphin Jul 3, 2021
32d73a5
Ensure block name can only be those from spec
GuiltyDolphin Jul 3, 2021
b0aa9c2
Fix typo
GuiltyDolphin Jul 3, 2021
0756b16
Add some explanatory comments
GuiltyDolphin Jul 3, 2021
2127efe
Add support for disabling lexer debug colouring
GuiltyDolphin Jul 3, 2021
76b31d0
Add tokenToText helper
GuiltyDolphin Jul 3, 2021
610d275
Fix some parenting
GuiltyDolphin Jul 3, 2021
9e64c53
Add initial support for parsing verse block bodies
GuiltyDolphin Jul 3, 2021
6040d1d
Add some fixes to markup parsing in verse blocks
GuiltyDolphin Jul 4, 2021
209cc08
Refactor block token tests
GuiltyDolphin Jul 5, 2021
94788bd
Refactor blank token tests
GuiltyDolphin Jul 5, 2021
2dc4f32
Refactor comment token tests
GuiltyDolphin Jul 5, 2021
0023d9c
Refactor drawer token tests
GuiltyDolphin Jul 5, 2021
2a97fe2
Refactor footnote token tests
GuiltyDolphin Jul 5, 2021
613727b
Add explicit orga dependency on @types/unist
GuiltyDolphin Jul 6, 2021
0a92d9f
Ensure DrawerEnd and Comment types are exported
GuiltyDolphin Jul 6, 2021
6457b56
Refactor headline token tests
GuiltyDolphin Jul 6, 2021
20fb9ff
Force specifying actionable property
GuiltyDolphin Jul 6, 2021
32daef6
Add tests for parsing of single character
GuiltyDolphin Jul 17, 2021
5e6979d
Fix lexing of newlines at end of block start/end
GuiltyDolphin Jul 17, 2021
75aa406
Add tests for BOL markup
GuiltyDolphin Jul 6, 2021
5a1aebe
Fix up some headline parsing based on spec
GuiltyDolphin Jul 6, 2021
b653e3c
Ensure priority can be any character but newline
GuiltyDolphin Jul 6, 2021
96fe8cf
Refactor horizontal rule token tests
GuiltyDolphin Jul 6, 2021
0429496
Refactor inline token tests
GuiltyDolphin Jul 6, 2021
db2dd3c
Clarify comment
GuiltyDolphin Jul 6, 2021
79ed6b5
Refactor keyword token tests
GuiltyDolphin Jul 6, 2021
48a13d2
Refactor list token tests
GuiltyDolphin Jul 6, 2021
edd48e2
Refactor planning token tests
GuiltyDolphin Jul 6, 2021
52273b9
Refactor table token tests
GuiltyDolphin Jul 6, 2021
253c016
Remove unused 'substring'
GuiltyDolphin Jul 6, 2021
f6e483d
Nit some formatting
GuiltyDolphin Jul 6, 2021
e138f2e
Rework heading and section parsing to match spec
GuiltyDolphin Jul 7, 2021
18c1400
Remove Section and Headline from Content
GuiltyDolphin Jul 7, 2021
b194640
Fix parser debugger borking on no position
GuiltyDolphin Jul 7, 2021
11b071c
Ensure that footnotes break footnotes
GuiltyDolphin Jul 7, 2021
55154a6
Allow footnotes to have no space before content
GuiltyDolphin Jul 7, 2021
2a1b664
Add pushMany helper
GuiltyDolphin Jul 7, 2021
e5cf543
Ensure Content type is exported
GuiltyDolphin Jul 7, 2021
0626f76
Remove use of any type
GuiltyDolphin Jul 7, 2021
8b80db8
Add some explicit typing
GuiltyDolphin Jul 7, 2021
0d870de
Add Child interface
GuiltyDolphin Jul 7, 2021
db0c28a
Migrate token types to separate file
GuiltyDolphin Jul 7, 2021
773fac6
Fix debug types
GuiltyDolphin Jul 7, 2021
912bda2
Refactor _text in util as it is in TokenI
GuiltyDolphin Jul 8, 2021
28500a4
Move AST builder helpers into parse utils
GuiltyDolphin Jul 8, 2021
f00aa67
Add versions of token builders to tokenize/util
GuiltyDolphin Jul 8, 2021
6d2189a
Switch test snapshot to use inline definition
GuiltyDolphin Jul 8, 2021
d055846
Add Char type for ensuring priority is one char
GuiltyDolphin Jul 8, 2021
7b2d007
Remove unused import
GuiltyDolphin Jul 8, 2021
ca95ef4
Ensure table cells support different objects
GuiltyDolphin Jul 8, 2021
6f908e9
Ensure pipes in markup in cells start new cell
GuiltyDolphin Jul 8, 2021
7959946
Add test for tables ending at first non-pipe line
GuiltyDolphin Jul 9, 2021
8928aa8
Add test for pipe not at start not starting table
GuiltyDolphin Jul 9, 2021
d31c78f
Fix positions being elided from lexer results
GuiltyDolphin Jul 9, 2021
bd37570
Fix missing positions in parser debugger
GuiltyDolphin Jul 9, 2021
ca7409c
Fix positions of table cells in parsing
GuiltyDolphin Jul 9, 2021
3fd4df9
Add some parsing combinators + switch over table
GuiltyDolphin Jul 9, 2021
34d11d6
Fix an implicit use of any
GuiltyDolphin Jul 10, 2021
5a2d42e
Switch orga to use strictNullChecks
GuiltyDolphin Jul 10, 2021
de3d72c
Update drawer parsing to use combinators
GuiltyDolphin Jul 10, 2021
ac41666
Improve type safety with eat
GuiltyDolphin Jul 10, 2021
32566ad
Update headline parser to use combinators
GuiltyDolphin Jul 12, 2021
60030e5
Ensure priority is parsed in heading
GuiltyDolphin Jul 12, 2021
5d44a1e
Move timestamp lexer into tokenize directory
GuiltyDolphin Jul 12, 2021
3d0e816
Add todo
GuiltyDolphin Jul 12, 2021
711ebed
Remove timestamp parser from global exports
GuiltyDolphin Jul 13, 2021
3b3910d
Update getChar to return a Char
GuiltyDolphin Jul 13, 2021
52d7fe4
Ensure POST is wrapped as a group
GuiltyDolphin Jul 13, 2021
6b019a6
Update some dependencies
GuiltyDolphin Jul 17, 2021
eb6ecb5
Nit
GuiltyDolphin Jul 17, 2021
a8b8d36
Add initial support for multi-line markup
GuiltyDolphin Jul 17, 2021
d39ae90
Add support for objects in certain text markup
GuiltyDolphin Jul 18, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@
"eslint-config-prettier": "^8.3.0",
"eslint-plugin-import": "^2.23.4",
"eslint-plugin-prettier": "^3.3.1",
"jest": "27.0.4",
"jest-cli": "27.0.4",
"jest": "^27.0.6",
"jest-cli": "^27.0.6",
"lerna": "^4.0.0",
"prettier": "^2.3.1",
"ts-jest": "^27.0.3",
Expand Down
1 change: 1 addition & 0 deletions packages/orga/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
"prepublishOnly": "yarn build"
},
"dependencies": {
"@types/unist": "^2.0.5",
"date-fns-tz": "^1.1.4",
"text-kit": "^2.4.9"
},
Expand Down
20 changes: 20 additions & 0 deletions packages/orga/src/char.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
import { Char } from 'text-kit';
export type { Char };

export const isChar = (c: string): c is Char => c.length === 1;

export function assertChar(c: string): asserts c is Char {
if (!isChar(c)) {
throw new Error('expected string of length 1');
}
};

export const char = (c: string): Char => {
assertChar(c);
return c;
}

export const charAt = (c: string, n: number): Char | null => {
const ch = c.charAt(n);
return ch ? char(ch) : null;
}
2 changes: 0 additions & 2 deletions packages/orga/src/index.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import defaultOptions, { ParseOptions } from './options'
import { parse as _parse } from './parse'
import { parse as parseTimestamp } from './timestamp'
import { tokenize } from './tokenize'
import { Document } from './types'

Expand All @@ -14,6 +13,5 @@ export const parse = (
}

export {
parseTimestamp,
ParseOptions,
}
53 changes: 33 additions & 20 deletions packages/orga/src/node.ts
Original file line number Diff line number Diff line change
@@ -1,61 +1,74 @@
import { Node, Position } from 'unist'
import { Position } from 'unist'
import { after, before, isEmpty } from './position'
import { Parent } from './types'
import { Child, Node, Parent } from './types'

const clone = ({ start, end }: Position): Position => ({
start: { ...start },
end: { ...end },
})

const adjustPosition = (parent: Parent) => (child: Node): void => {
const adjustPosition = (parent: Parent & Partial<Child>) => (child: Node): void => {
let dirty = false

if (!child.position) return
const cp = clone(child.position);
if (parent.position) {
parent.position = clone(parent.position);
const belowLowerBound = before(parent.position.start)
const aboveUpperBound = after(parent.position.end)

if (isEmpty(parent.position)) {
parent.position = clone(child.position)
parent.position = cp;
dirty = true
} else if (belowLowerBound(child.position.start)) {
parent.position.start = { ...child.position.start }
} else if (belowLowerBound(cp.start)) {
parent.position.start = cp.start;
dirty = true
}else if (aboveUpperBound(child.position.end)) {
parent.position.end = { ...child.position.end }
} else if (aboveUpperBound(cp.end)) {
parent.position.end = cp.end;
dirty = true
}
} else {
parent.position = clone(child.position)
parent.position = cp;
dirty = true
}

if (!!parent.parent && dirty) {
if ('parent' in parent && parent.parent && dirty) {
adjustPosition(parent.parent)(parent)
}
}

export const pushMany = <P extends Parent>(p: P) => (n: P['children']): P => {
n.forEach(n => push(p)(n));
return p;
}

export const push = <P extends Parent>(p: P) => (n: Node & P['children'][number]): P => {
export const push = <P extends Parent>(p: P) => (n: P['children'][number]): P => {
if (!n) return p
adjustPosition(p)(n)
const node = n as Parent
if (node) {
node.parent = p
if (n) {
n.parent = p
}
p.children.push(n)
return p
}

export const map = (transform: (n: Node) => any) => (node: Node) => {
export const setChildren = <P extends Parent>(p: P) => (ns: [Node, ...Node[]] & P['children']): P => {
adjustPosition(p)(ns[ns.length - 1])
for (const n of ns) {
n.parent = p;
}
p.children = ns;
return p
}

export const map = <T extends Node & Partial<Parent>>(transform: (n: Node & Partial<Parent>) => T) => (node: Node & Partial<Parent>): T => {

const result = {
type: node.type,
...transform(node),
}

if ((node as Parent).children) {
result.children = (node as Parent).children.map(map(transform))
if ('children' in node && node.children) {
result.children = node.children.map(map(transform))
}
return result
}
Expand All @@ -74,10 +87,10 @@ interface DumpContext {
// return [line].concat(rest)
// }

export const level = (node: Parent): number => {
export const level = (node: Child): number => {
let count = 0
let parent = node.parent
while (parent) {
while (parent && 'parent' in parent) {
count += 1
parent = parent.parent
}
Expand Down
Loading