#define (simplest cases only?)

[aldanor]

I'm sorry if this has already been brought up but I haven't found it in issues or docs: can preprocessor directives like #define be handled? It looks like they can't. I'm not too familiar with libclang so this may well be something obvious.

Aside from dealing with compile time flags, most of the C libraries use #define to set constants, as in #define X (1), #define Y ((float)-X), which easily transforms to immutable. For a large percentage of headers, just handling constants would be sufficient to get them working.

However, C-defines can get ugly, weird and unpredictable and sure there's no way to handle all of them... though some can be parsed into immutable constants, some into functions #define f(a, b) ((a) + (b)).

Are there any plans regarding this?

Thanks.

[jacob-carlborg]

Yes, there are plans to handle this. Unfortunately libclang doesn't provide an API to do this yet. Although, the C++ libraries provides API's to do this. I, or someone else, just need to add an API to libclang for this. Contributions are welcome 😉.

[aldanor]

Hmm. What about this flag?

http://clang.llvm.org/doxygen/group__CINDEX__TRANSLATION__UNIT.html#ggab1e4965c1ebe8e41d71e90203a723fe9a9e1dd69c4a2ff7954dca8df41a63c0e6

[jacob-carlborg]

There's still no API available to get access to any of this information, as far as I know. I've asked in the Clang mailing list.

[aldanor]

I've skimmed through the clang repo, there is PreprocessingRecord class which seems to track all of the preprocessing, and the only place in code where seems to be invoked is in CompilerInstance::createPreprocessor(), only when the PreprocessorOptions.DetailedRecord flag is set to true (i.e. the CXTranslationUnit_DetailedPreprocessingRecord flag is passed). Why wouldn't that work, am I missing something? Hm...

[jacob-carlborg]

Even if I pass the CXTranslationUnit_DetailedPreprocessingRecord flag, which API should I use. Example:

#define foo 1

How to I get the value, 1?

[aldanor]

Ok, please mind my clang, I have never used it before. So here's goes the header, then the program output, then the C++ code itself.

// target.h
#define a (1)
#define f(x, y) ((x) + (y))

#ifdef __QWE__
        #define __QWEQWE__
#endif

int qwe(int x);

int qweqwe() {
        return 1;
}

Note the MACRO entries:

int 'int'
identifier 'qwe'
l_paren '('
int 'int'
identifier 'x'
r_paren ')'
semi ';'
int 'int'
identifier 'qweqwe'
l_paren '('
r_paren ')'
l_brace '{'
return 'return'
numeric_constant '1'
semi ';'
r_brace '}'
eof ''
target.h:1:9 -> MACRO: l_paren '('  numeric_constant '1'  r_paren ')'
target.h:2:9 -> MACRO: l_paren '('  l_paren '('  identifier 'x'  r_paren ')'  plus '+'  l_paren '('  identifier 'y'  r_paren ')'  r_paren ')'

The code (I used string comparison on source location, that's kinda ugly but I don't know clang API well enough to check if the macro is defined in the target file; you might be able to do this better):

int main()
{
    CompilerInstance ci;
    DiagnosticOptions diagnosticOptions;
    ci.createDiagnostics();
    llvm::IntrusiveRefCntPtr<TargetOptions> pto( new TargetOptions());
    pto->Triple = llvm::sys::getDefaultTargetTriple();
    TargetInfo *pti = TargetInfo::CreateTargetInfo(ci.getDiagnostics(), pto.getPtr());
    ci.setTarget(pti);
    ci.createFileManager();
    ci.createSourceManager(ci.getFileManager());
    ci.createPreprocessor();
    ci.getPreprocessorOpts().DetailedRecord = true;
    const FileEntry *pFile = ci.getFileManager().getFile("target.h");
    ci.getSourceManager().createMainFileID(pFile);
    ci.getPreprocessor().EnterMainSourceFile();
    ci.getDiagnosticClient().BeginSourceFile(ci.getLangOpts(),
                                             &ci.getPreprocessor());
    Token tok;
    do {
        ci.getPreprocessor().Lex(tok);
        if (ci.getDiagnostics().hasErrorOccurred())
            break;
        ci.getPreprocessor().DumpToken(tok);
        std::cerr << std::endl;
    } while ( tok.isNot(clang::tok::eof));
    ci.getDiagnosticClient().EndSourceFile();
    for (Preprocessor::macro_iterator I = ci.getPreprocessor().macro_begin(),
            E = ci.getPreprocessor().macro_end(); I != E; ++I) {
        MacroInfo *MI = I->second->getMacroInfo();
        SourceLocation loc = MI->getDefinitionLoc();
        if (!loc.isFileID())
            continue;
        std::string strLoc = loc.printToString(ci.getSourceManager());
        if (strLoc.find("target.h") != std::string::npos) {
            std::cerr << strLoc << " -> ";
            ci.getPreprocessor().DumpMacro(*MI);
        }
    }
    return 0;
}

[jacob-carlborg]

I'm using libclang, the C API.

[aldanor]

@jacob-carlborg Ok, if you look at the c-index-test tool in llvm/clang/tools/c-index-test, it uses libclang exclusively and not the C++ API (link to source).

$ c-index-test -test-load-source all target.h | grep target.h

// CHECK: target.h:1:9: macro definition=a Extent=[1:9 - 1:14]
// CHECK: target.h:2:9: macro definition=f Extent=[2:9 - 2:28]
// CHECK: target.h:8:5: FunctionDecl=qwe:8:5 Extent=[8:1 - 8:15]
// CHECK: target.h:8:13: ParmDecl=x:8:13 (Definition) Extent=[8:9 - 8:14]
// CHECK: target.h:10:5: FunctionDecl=qweqwe:10:5 (Definition) Extent=[10:1 - 12:2]
// CHECK: target.h:10:14: CompoundStmt= Extent=[10:14 - 12:2]
// CHECK: target.h:11:2: ReturnStmt= Extent=[11:2 - 11:10]
// CHECK: target.h:11:9: IntegerLiteral= Extent=[11:9 - 11:10]

So it can somehow magically find the macro definitions (see the first two lines)?

[jacob-carlborg]

Yeah, I noticed that it's possible to get information if a cursor is a macro. Although I cannot find an API to get more information than the name of the macro, the spelling.

[aldanor]

Yea, I guess you can get the locations but not the actual text (with libclang). So, it seems the only tangible option (apart from pulling the macro text based by locations and then hacking the strings semi-manually and figuring out what each #define means, which is always an option) is having an extra tool which is based on C++ clang API (like in the example above) whose only job is to produce AST for macros? Unless you have reasons for not involving the clang C++ stack :) Idk hot ugh, maybe I'm plain wrong and there's no way to make this work..

[jacob-carlborg]

As I said in my first reply. The correct solution is to extend the C API with the missing pieces to handle macros.

[jacob-carlborg]

BTW, the reason for not using the C++ API's is:

• DStep is written in D. It's much easier to interface D code with C than C++ code. I don't even know if the C++ support in D was usable when I started DStep
• The C API is stable. It's even ABI stable. I can use any version of libclang from 3.1 to the latest, without having to recompile or relink anything

[joakim-noah]

Until a C API shows up, it would be nice if you could put in some basic regexp to translate simple constant defines of the form #define foo 1 to enum foo = 1, and punt on the more complicated ones until the preprocessor APIs are worked out.

[ciechowoj]

I suppose it could be closed via #67 . However I may create a file Issue20.d with unit tests with the few examples of defines mentioned here.

[jacob-carlborg]

@ciechowoj should I close it now or do you want me to wait for the unit tests?

[ciechowoj]

Actually it was question for you, but I'll add unit tests first : )