Handle preprocessor directives

[kymckay]

I probably need to introduce a lexical preprocessor to truly handle these since they'd be resolved before the sqf lexer takes over

• #include
• #define MACRO value
• #define MACRO_FUNC(ARG1, ARG2, ...)
• # (stringification)
• ## (token concatenation)
• \ (exactly before a newline, multi-line definition)
• #undef
• #if
• #ifdef
• #ifndef
• #else
• #endif

[kymckay]

One idea:

1. Implement the preprocessor as an input stream wrapper (the lexer can then consume a preprocessor or just a bare stream as it currently dose)
2. As the lexer consumes the preprocessor it does its thing (e.g. resolve macros to an internal buffer) and then returns the next character post-preprocessing

[kymckay]

On further consideration it may make sense to just extend the lexer itself to be "preprocessor aware" and perform tokenisation as-if preprocessing happens first.

One consideration for preprocessing is the position of reported errors. Some inspiration may be taken from the C preprocessor

[kymckay]

My current idea for #define:

• Use regex to capture the head, arguments and body of a macro, replacing the arguments with standardised variation like $1 corresponding to their index
• Store in a multi map (macro can be defined multiple times for different argument counts) where the head is the key:
  • The number of arguments
  • The position of the head token
  • The body of the macro
• Whenever an alpha character or _ is encountered, read the full ID and check if it is a macro
  • If it is (matches an entry in the multimap) check for arguments
    • if the number of arguments matches an entry in the multimap then regex replace them by index using previously injected $1 tokens, then resolve further macros within the string
    • if the number of arguments doesn't match TBD (test real behaviour)
  • If it isn't just store the string in the buffer and return the first PosChar as expected

[kymckay]

With da91d20 the bulk of directives are now handled in some form. The remaining #include directive has a separate issue open.