Importing grammars

[bkeepers]

pegjs/pegjs#523 is a proposal (with cross references to previous discussions) for adding support for importing grammars. I don't see any further discussion on this repo since the fork from pegjs, so I was curious if anyone has thought more about it.

I might try some workarounds (like just concatenating the grammars in the build process) for now. I would be interested to hear if anyone has tried this or any other approach to sharing grammars.

[hildjj]

I'm ready to start thinking about this. I wonder if futagoza's syntax will still work after we used @ for plucking.

I'd like to think about long-term extensibility of the grammar syntax, since we're running out of non-letter ASCII characters to give special meaning to. We haven't used < or > yet, so we could do something XML like, for instance.

[Mingun]

Actually, I think that js-like imports looks better. I already implement grammar part of this in my fork:

import { rule1, rule2 as rule3 } from "grammar.peggy";

start = rule1 / rule3;

I never pushed that although. It is based on ranges syntax, so I've planned to propose it after ranges.

[hildjj]

Yes, js-like is much better.

[Mingun]

I've opened a draft #292, so anyone can play with the implementation

[reverofevil]

Great!

[hildjj]

Is the import path relative to the .peggy file, or relative to the generated .js file? I think the first, but Typescript uses the second, so I want to make sure we are explicit in our decision.

[hildjj]

Also for clarity, import and require in the top-level initializer is relative to the .js file created.

[Mingun]

I think that path in imports should be relative to the .peggy file. It always will be just an agreement, however, because how exactly to use paths is up to the pass implementing import, but in my existing implementation I use paths relative to .peggy. Generator could always rewrite them if needed.

[nene]

I would be really-really interested in this as I'm currently working on a grammar file that's over 5000 lines long, which is getting more and more inconvenient to work with as it grows... and I'm expecting it to grow at least 2x :(

[Mingun]

If a quick-and-dirty result is enough for you, you can port an existing implementation. The syntax part is already ported (but still should be reviewed carefully). If in addition you are lazy, you can use the trunk branch. Syntax is slightly different, and some features is working slightly different then in current master, but that could be irrelevant for you. I use it in my pet project many years.

[nene]

Thanks. I'll have a look.

[hildjj]

I've been thinking about this a lot, and I'm worried about how the imported file will be read. It should work on the web as well as in Node. There are a couple of paths I can think of:

• There's a pre-parse step (or a particular error thrown) that just gives filenames/URLs that are going to be imported. The caller can then do whatever environment-specific thing they want to retrieve the files, then pass them in to generate explicitly.
• Like above, but have a node wrapper that is carefully not imported into the web version. Also a web version that handles "https?:" URLs. If you're using a late enough version of node (19+, with fetch), that wrapper might call the web version for those URL types.
• Do a bunch of URL processing, determine the environment, and do the right thing. This is going to be painful and lead to a bunch of security bugs. It might suck in big swaths of the node runtime into the web bundle by mistake. Note that fetch doesn't work with "file:" URLs in node.

[Mingun]

I think we should just provide conception of loaders, then in a web the user should just implement path into files resolving algorithm. For example, it could be loading files into local storage (manually by the user, for example, Kaitai Struct IDE does that) then read from it.

[hildjj]

See #417 as a part of the solution. That PR does not preclude in-grammar syntax for imports. It does add grammar for parsing es6 imports in your topLevelIniitializer, so that all of the imports from library grammars can be moved to the top of the output. Those same rules could also be used to parse in-grammar import syntax one day.