Substitute preprocessor definitions into token stream

[AbigailBuccaneer]

Currently we just ignore the contents of all directives. #define should be able to actually define tokens to be expanded into another list of tokens. Ideally this would do function-like macros as well as variables, but variables only would be a good start.

[Darkyenus]

I have started working on this, in branch define_macro. So far it correctly parses simple #define directives (not function-like yet). Problem is, that it is not trivial (or at least I don't see how) to substitute one IDENTIFIER with defined tokens, so all the GLSLElement type classes have problems with it.

What I'd like to do is to somehow expand GLSLRedefinedPsiElement (see that branch) to the #defined tokens inside GLSLElementImpl or somewhere around that. This would also be a good place to strip all preprocessor tokens that got mixed in during parsing.

This approach is not perfect though, it will most likely fail for something like:

#define FOO + 3.0 +

float bar = 5.0 FOO 1.0;

because it will be too late to do that substitution and retain precise token boundaries. (Above should get parsed correctly though. EDIT: Yes it does)

[AbigailBuccaneer]

I've been having a look (via PsiViewer) at how CLion handles this for C++. I think we can do better than we're doing with regards to the PSI tree we generate.

Considering the code:

#define BAR (true)
// ...
BAR ? 1 : 2;

CLion generates a tree that looks something like this:

OCMacroCall
 └─ OCMacroReferenceElement
     └─ PsiElement(IDENTIFIER)
OCExpressionStatement
 ├─ OCConditionalExpression
 ├─ OCParenthesizedExpression
 │   ├─ ForeignLeaf(OCPunctuator:(: ()
 │   ├─ OCLiteralExpression
 │   │   └─ ForeignLeaf(OCKeyword:true: true)
 │   └─ ForeignLeaf(OCPunctuator:): ))
 ├─ PsiElement(OCPunctuator:?)
 ├─ OCLiteralExpression
 │   └─ PsiElement(INTEGER_LITERAL)
 ├─ PsiElement(OCPunctuator::)
 ├─ OCLiteralExpression
 │   └─ PsiElement(INTEGER_LITERAL)
 └─ PsiElement(OCPunctuator:;)

So it generates an OCMacroCall element, and then continues parsing the substituted tokens, but generates a ForeignLeaf wrapper for each one. This way, the parse tree is exactly the same as it would be after substitution, other than the fact that it uses ForeignLeafPsiElement elements for the substituted tokens.

[Darkyenus]

There is a lot to be improved in the preprocessor macro handling, but I don't have time and experience with/knowledge about psi and how it is implemented in IntelliJ. Their approach is what I attempted to do first, but I didn't find out how to "inject" those foreign tokens into the parsing token stream conveniently, without rewriting substantial amounts of code. However if you or anybody else figures that out, PRs are welcome.

[AbigailBuccaneer]

I'll investigate.