RFE: Create Observer/Callback layer. Make ucpp more convenient to use as a library.

[GoogleCodeExporter]

Using a preprocessor as a library typically involves not only to translate
input into output, but also to provide callback hooks, such as to report
macro definitions, macro evaluations (during macro call expansion),
processing include directives, etc.

In addition, it would be good to allow users to provide output-generating
functions so output can be streamed not into a file, but into any
user-provided callback. It would be particularly helpful for this if such
calls wouldn't be fed characters, but entire tokens.

If any of this sounds useful, I would be glad to provide more detailed
use-cases, or even patches.

Thanks,
           Stefan

Original issue reported on code.google.com by stefan.s...@gmail.com on 27 Feb 2009 at 8:11

[GoogleCodeExporter]

ucpp already allows some of the functionality you are asking for.  See sample.c 
(http://code.google.com/p/ucpp/source/browse/trunk/sample.c) for an example of 
how to use ucpp in what 
Thomas Pornin called lexer mode.  The model used however is the Polling 
technique (much like an X11 Event 
parser) where every "tick" generates a "token" (including PRAGMA tokens, 
INCLUDE tokens, WHITESPACE 
tokens, etc).  Output in lexer mode is up to the developer.  My concern is that 
some macro 
definitions/expansions happen without triggering a "token".  I can't say for 
sure because my use of the macro 
tokens has been limited in my applications.

For some uses of ucpp, I can see where an Observer/Callback pattern would be 
useful.  Writing a layer on top 
of lexer mode to implement this would not be difficult.  However, ucpp is a 
very low priority for me at the 
moment.  If you would like to submit a patch, I would gladly consider adding it.

Original comment by lpsan...@gmail.com on 28 Feb 2009 at 12:23

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Thanks for the quick reply !
I have been using ucpp for quite a while (as part of the Synopsis project:
http://synopsis.fresco.org), but always in stand-alone mode. Now I'm trying to 
use it
in lexer mode, with moderate success. I need ucpp to accurately reproduce the 
input
(as far as possible), as I use the output to regenerate the source, with markup.

However, while I'm able to get that using stand-alone mode, with lexer mode 
(i.e.
calling lex()), it seems some (pseudo-)tokens are amiss. In particular, for 
things
like macro-definitions, no newlines are generated, so all subsequent output is
shifted back.

Any idea how to fix this ? (I'm using SEMPER_FIDELIS.)

Thanks a lot !

Original comment by stefan.s...@gmail.com on 3 Mar 2009 at 2:44

[GoogleCodeExporter]

Ahh!  I understand now.  lexer mode is not "pure" enough for your purposes.  
It'll gobble up the macros and 
then and then expand them when it comes across them in text.  In macros.c, 
handle_define could probably be 
rewritten to respond to a new tune.h parameter allowing the macro to be 
processed as pseudo-tokens (#, 
NAME=define, NAME=macroName, etc.).

I do think this enhancement (let's call it, NO_MACRO_DEFINES), and an 
Observer/Callback pattern would be 
useful.  I'll capture these two enhancements in separate requests (so that they 
can be implemented and tested 
indepently).  Anything else, specifically, you are looking for?

BTW, re2c 0.13.5 (http://re2c.sf.net) has an example grammar (examples/pp-c.re) 
which seems to come 
close to what you need (at least for macro definition).  It "warns" on the 
macro continuation symbol but that 
could probably fixed in the grammar.

Original comment by lpsan...@gmail.com on 3 Mar 2009 at 6:51

• Changed title: RFE: Make ucpp more convenient to use as a library. Create Observer/Callback layer.
• Changed state: Accepted
• Added labels: Usability

[GoogleCodeExporter]

Original comment by lpsan...@gmail.com on 3 Mar 2009 at 6:52

• Changed title: RFE: Create Observer/Callback layer. Make ucpp more convenient to use as a library.

[GoogleCodeExporter]

Actually, I do not want preprocessor directives to be pushed to the user. I 
just want
ucpp to push the equivalent number of newlines, such that subsequent tokens 
appear on
the correct line.
if I operate in stand-alone mode, the output looks good. When operating in 
lexer mode
however (with SEMPER_FIDELIS in particular), some newlines are missing. So all 
I'm
asking for is that those missing newlines are added such that I can reconstruct 
the
output file myself using tokens obtained from lex() calls.

Thanks,
          Stefan

Original comment by stefan.s...@gmail.com on 3 Mar 2009 at 1:55

[GoogleCodeExporter]

I think I need a use-case.

Original comment by lpsan...@gmail.com on 3 Mar 2009 at 4:17

[GoogleCodeExporter]

My input file:

---
//A comment
#define Foo
//Another comment
---

Results in 4 tokens from lex():
1: "//A comment"
2: newline
3: "//Another comment"
4: newline

What I would expect is another newline between 2 and 3 above, at the place 
where the
Foo macro definition was read. (If the definition takes multiple lines, I 
obviously
need multiple newlines.

In fact, lexer.c contains this code:

    if (!(ls->flags & LEXER) && (ls->flags & KEEP_OUTPUT))
        for (; ls->line > ls->oline;) put_char(ls, '\n');

which, at opportune times, just fills the output with newlines until input and 
output
line-number match. I think something conceptually eqivalent is required in lexer
mode, at least if SEMPER_FIDELIS is defined.

An alternative would be to generate more #line markers so I can adjust for it. 
But
since ucpp can do without them in stand-alone mode, I think it should be 
straight
forward to fix the lexer mode to be able to live without them, too.

Thanks,
        Stefan

Original comment by stefan.s...@gmail.com on 3 Mar 2009 at 4:28

[GoogleCodeExporter]

Hello,

did you get a chance to look at this issue ? Do you think it is solvable ? I'm 
about
to decide whether or not I can use ucpp in lexer mode, and this issue is 
blocking me
right now. Thus I would appreciate any feedback.

Many thanks,
              Stefan

Original comment by stefan.s...@gmail.com on 6 Mar 2009 at 2:11

[GoogleCodeExporter]

I think that change is doable but the 15 minutes I put into it weren't enough 
to fix it.  If I have some time this 
weekend, I might try to get to it.  But I did put another 15 minutes into re2c 
and got something that might be 
useable.  Can you do without the macro expansion?  If you can, I have something 
using re2c that might work.  
600 is the return code for newline.

lpsantil$ cat ~/x.h
//* Macro start */
#define mMacro( x ) \
   s + \
   x
//* Macro end */
mMacro( 5 );


lpsantil$ ./cpp < ~/tmp/x.h
304 //* Macro start */
600 
502 #define
286 mMacro
40  (
286 x
41  )
92  \
600 
286 s
43  +
92  \
600 
286 x
600 
304 //* Macro end */
600 
286 mMacro
40  (
285 5
41  )
59  ;
600 
600 
315 void
286 func
40  (
289 int
286 a
41  )
59  ;
600 

Original comment by lpsan...@gmail.com on 6 Mar 2009 at 5:13

Attachments:

• cpp.re

[GoogleCodeExporter]

Thanks for looking into this !
No, I really need a preprocessor, not only a tokenizer, i.e. macro expansion is
required. Here is my real use-case:

For Synopsis, I need to preprocess C and C++ files. Doing this, I need to record
detailed information about

- include directives, as well as file context switches
- macro definitions, as well as macro calls

This works fine with ucpp in stand-alone mode.
However, while Synopsis will then parse the preprocessed file, and attach 
comments
(docstrings) to declarations, I now want to attach comments also to macro
declarations (so macros can be properly documented, too).
To do this, I want to use ucpp in lexer mode, so I can cache all 'comment' 
tokens,
until I either hit a macro declaration or an ordinary C/C++ token.

In either case I still want to generate the preprocessed output file, for which 
being
able to produce correct output line numbers is important (or else the parser 
will
report wrong line numbers if he runs into syntax errors).

I would gladly help to integrate the callback hooks I added into ucpp, but I 
want to
get the above working first. :-)

Thanks,
      Stefan

Original comment by stefan.s...@gmail.com on 6 Mar 2009 at 5:24

[GoogleCodeExporter]

Ping, any news on this ? (I need to decide whether to hack on my version of the 
code
to get the features I need, or whether I may wait and get a clean ucpp tree 
building.)

Thanks,
      Stefan

Original comment by stefan.s...@gmail.com on 16 Mar 2009 at 3:24