Hi, I have worked a bit with Roslyn and their leading vs trailing trivia, trying to implement it on my own again with syntax tree rewrites, and from my experience so far it looks like it adds unnecessary overhead — I advice you to rather consider only one kind of trivia, just like they later did for TypeScript — leading trivia only and special "EOF" virtual token to conform to the invariant. It makes all kinds of rewriting logic a lot easier and deterministic compared to having to keep both of them — a lot of edge cases come up later, for instance when you have to merge or move the tokens between two lists — even normalize the tokens if you have [tok] [trail] [lead] [tok]. It came up in my case when I tried creating AST trees directly from code.
Here is a question I once asked and even roslyn creators answered regarding that topic: https://stackoverflow.com/questions/76976029/whats-the-design-decision-for-differentiating-leading-and-trailing-trivia-in-ro
I found it also a lot more intuitive to reason and write logic when instead of trivia attachment to token there was a "inbetween" kind of structure (where the invariant is "between any two tokens there is a possible empty list of trivia"), but this gets into a territory of new data structure design (although it's a very interesting topic, also probably one that could give a lot of performance improvements in the future, for instance if you had structure for tokens and sibling structure for trivia, that you "weave into" the first like a zipper — take token from syntax tree, take trivia list node from trivia tree and repeat — any non-syntactical refactorings could work solely on trivia tree without ever touching the syntax tree — syntax tree becomes immutable and it's only trivia tree which gets discarded) — just a free-floating topic.
If you have a counterexample I would be glad to learn when it's bad, as that could be useful for me as wrell.
Hi, I have worked a bit with Roslyn and their leading vs trailing trivia, trying to implement it on my own again with syntax tree rewrites, and from my experience so far it looks like it adds unnecessary overhead — I advice you to rather consider only one kind of trivia, just like they later did for TypeScript — leading trivia only and special "EOF" virtual token to conform to the invariant. It makes all kinds of rewriting logic a lot easier and deterministic compared to having to keep both of them — a lot of edge cases come up later, for instance when you have to merge or move the tokens between two lists — even normalize the tokens if you have
[tok] [trail] [lead] [tok]. It came up in my case when I tried creating AST trees directly from code.Here is a question I once asked and even roslyn creators answered regarding that topic: https://stackoverflow.com/questions/76976029/whats-the-design-decision-for-differentiating-leading-and-trailing-trivia-in-ro
I found it also a lot more intuitive to reason and write logic when instead of trivia attachment to token there was a "inbetween" kind of structure (where the invariant is "between any two tokens there is a possible empty list of trivia"), but this gets into a territory of new data structure design (although it's a very interesting topic, also probably one that could give a lot of performance improvements in the future, for instance if you had structure for tokens and sibling structure for trivia, that you "weave into" the first like a zipper — take token from syntax tree, take trivia list node from trivia tree and repeat — any non-syntactical refactorings could work solely on trivia tree without ever touching the syntax tree — syntax tree becomes immutable and it's only trivia tree which gets discarded) — just a free-floating topic.
If you have a counterexample I would be glad to learn when it's bad, as that could be useful for me as wrell.