clang_tokenize fails after reparsing when using precompiled preambles.

[llvmbot]

Bugzilla Link	10999
Resolution	FIXED
Resolved on	Sep 26, 2011 03:03
Version	trunk
OS	All
Attachments	test program, test input for program
Reporter	LLVM Bugzilla Contributor
CC	@akyrtzi,@DougGregor,@tkremenek

Extended Description

For a file starting with a comment or a blank line, libclang has the following behaviour:

When calling clang_tokenize after clang_parseTranslationUnit, all tokens are returned. When then calling clang_reparseTranslationUnit and clang_tokenize, no tokens get returned. In both cases the range is the full file (so starting at line 1).

When the file starts with a function definition, it works fine. Also, when turning off precompiled preambles, it also works fine.

Attached is a program (main.cpp) which parses and reparses test.cpp and prints the number of tokens after both parses. When the first line is removed from test.cpp, it works fine.

This was tested against r140369.

[tkremenek]

cloned to rdar://problem/10175093

[llvmbot]

patch for c-index-test.c to expose the problem
against r140369

[llvmbot]

c-index-test input

[llvmbot]

c-index-test input

[akyrtzi]

Fixed at r140519.