You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently tree.tokenize retokenizes the code within the pos of the tree, which means that those tokens are not related to the ones used in the parser and we will not get any information about indentation. That information might particularly be useful to tools that need to deal with indentation for example scalafmt. There are ways to work around it, but they cause significant complexity.
It would be cool to somehow expose information about where indentation was detected. Some options:
Keep additional isIndented method in the Tree
Safe the tokens from parsing while also inserting appropriate Indentation.Indent and Indentation.Outdent tokens
Just safe scanner tokens inside ScalametaParser, so that some tools can use that explitely.
1
This is something that we haven't done for most cases and introduced additional Trees for anything that can have an additional Boolean value. However, we could argue that isIndented is similar to for example pos, in the sense that it can be needed in every Tree. This would not solve an issue with the tokens, but would mean that we wouldn't need to save the tokens from parsing.
2
This requires two things:
keeping the tokens, which is something that was not done previously and most likely for performance/memory reasons
creating a new end tokens collections accessible via the tokens.method we could just save each token after next() method
3
ScalametaParser does not need to be binary compatible and is not exposed to the users, so this would mean that we would introduce a level of coupling between Scalameta and any tools that would use it explicitely.
None of the above seems like the best solution, I would turn to 2., but I am unsure of the consequences to any kind of optimizations.
The text was updated successfully, but these errors were encountered:
Currently
tree.tokenize
retokenizes the code within the pos of the tree, which means that those tokens are not related to the ones used in the parser and we will not get any information about indentation. That information might particularly be useful to tools that need to deal with indentation for example scalafmt. There are ways to work around it, but they cause significant complexity.It would be cool to somehow expose information about where indentation was detected. Some options:
Tree
Indentation.Outdent
tokens1
This is something that we haven't done for most cases and introduced additional Trees for anything that can have an additional Boolean value. However, we could argue that
isIndented
is similar to for examplepos
, in the sense that it can be needed in every Tree. This would not solve an issue with the tokens, but would mean that we wouldn't need to save the tokens from parsing.2
This requires two things:
tokens.method
we could just save each token afternext()
method3
ScalametaParser does not need to be binary compatible and is not exposed to the users, so this would mean that we would introduce a level of coupling between Scalameta and any tools that would use it explicitely.
None of the above seems like the best solution, I would turn to 2., but I am unsure of the consequences to any kind of optimizations.
The text was updated successfully, but these errors were encountered: