Implementation gotchas/requirements

[bpj]

I'm thinking of trying a Perl implementation. Are there any general gotchas or requirements to be aware of?

[jgm]

I imagine the djot.js code would probably port over fairly comfortably.

[bpj]

I tried to build djot.js with make build but got lots of errors.

[jgm]

Did you do npm install first?
Try:

npm install
npm run build

[jgm]

I've updated the Makefile so it will npm install automatically if you don't have node_modules.

[bpj]

Did that. It works now. Thanks!

[bpj]

Is there any policy WRT dependencies for an implementation, e.g. when I do a Perl port will it be OK to rely on third party modules from CPAN?

[jgm]

There's no policy. I chose not to have any dependencies for the Lua version because I wanted to make it easy to embed in C, etc.

[bpj]

To judge by the the JSON schema parsing position data (start/end position, line, column) are not required in the AST. Does this mean that implementations are not required to keep track of (all) of them? In Perl each string has a pos($str) property where the last regex match against the string left off, and that is about it. I did some experiments with creating a list of lines' metadata with offset, line number, length etc. And then searching for the last item whose startpos is ≤ the start position of the match and it was only very slightly more efficient than counting the number of newlines by regex match¹, but I think the most efficient would be to only calculate the line position by regex match in case of reporting an error.

Going out of your way to keep track of line position throughout is a major slowdown and should probably at least be made optional — as in the parser is a Perl class instance which has a readonly track_pos option which is set in the constructor call. Normally only the start/end character position of a match would be stored (which is reasonably efficient).

Notably Perl doesn't have any match object but such info as the start/end pos of the match/captures has to be retrieved from a number of global variables which are overwritten as soon as you make another successful regex match. I once wrote a class which, if the constructor is called immediately after a successful regex match, collects and encapsulates the info from these various global variables into something resembling a Python match object, but it is not very efficient.

Footnotes

1. the rather tortuous idiom is my $newline_count = () = substr($$subj, 0, $match_start) =~ m{\n}g;, and this creates a list of "\n" strings, stores the length of the list and then discards the list, hence not very efficient! (If the subject string is of any length it is much more efficient to use a reference to it since strings are copied on assignment in Perl!) ↩

[matklad]

Another reason to treat positions as impl defined is to make sure ast itself has all the info. We don’t want filters to use positions to lookup data from the original string, AST should be enough for everything

[jgm]

Yes it's optional, in the sense that this is a feature of the implementation rather than the syntax spec.