(stdio.h:36:25) invalid token at start of a preprocessor expression

[sephiroth74]

---

Description

With ndk 16.0.4293906 rc1 I now receive this build error:

  In file included from ../shared/source_library/boost/boost/config/platform/linux.hpp:15:
  In file included from ~/Library/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/cstdlib:86:
  In file included from ~/Library/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/stdlib.h:94:
  In file included from ~/Library/Android/sdk/ndk-bundle/sources/android/support/include/stdlib.h:32:
  In file included from ~/Library/Android/sdk/ndk-bundle/sysroot/usr/include/stdlib.h:36:
  In file included from ~/Library/Android/sdk/ndk-bundle/sysroot/usr/include/malloc.h:22:
  In file included from ~/Library/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/stdio.h:108:
  ~/Library/Android/sdk/ndk-bundle/sources/android/support/include/stdio.h:36:25: error: invalid token at start of a preprocessor expression
  #if __USE_FILE_OFFSET64 && __ANDROID_API__ < __ANDROID_API_N__
                          ^

Environment Details

NDK Version: 16.0.4293906 rc1
Build System: cmake (invoked from ./gradlew)
Host: Mac
Compiler: Clang
ABI: x86_64 and arm64-v8a
SDL: c++_shared
NDK API Level: 21

[DanAlbert]

If you change that to #if defined(__USE_FILE_OFFSET64) ... does it go away? I'm not able to actually reproduce this myself (on Linux, anyway, going to take a while to download the mac NDK).

[sephiroth74]

Yes, that error goes away.

However now this occurs:

  In file included from ~/boost/boost/filesystem.hpp:19:
  In file included from ~/boost/boost/filesystem/string_file.hpp:14:
  In file included from ~/boost/boost/filesystem/fstream.hpp:23:
  ~/Library/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/fstream:824:9: error: use of undeclared identifier 'fseeko'
      if (fseeko(__file_, __width > 0 ? __width * __off : 0, __whence))
/Users/crugnola/Library/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/fstream:197:5: note: in instantiation of member function 'std::__ndk1::basic_filebuf<char, std::__ndk1::char_traits<char> >::seekoff' requested here
      basic_filebuf();
      ^
  ~/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/fstream:1176:5: note: in instantiation of member function 'std::__ndk1::basic_filebuf<char, std::__ndk1::char_traits<char> >::basic_filebuf' requested here
      basic_ofstream();
      ^
  ~/boost/boost/filesystem/fstream.hpp:109:5: note: in instantiation of member function 'std::__ndk1::basic_ofstream<char, std::__ndk1::char_traits<char> >::basic_ofstream' requested here
   ~/boost/boost/filesystem/string_file.hpp:24:12: note: in instantiation of member function 'boost::filesystem::basic_ofstream<char, std::__ndk1::char_traits<char> >::basic_ofstream' requested here
    ofstream file;

[DanAlbert]

Okay, even on a Mac, building via Studio, I still can't repro this.

[DanAlbert]

Adding boost (fresh checkout from master) to the mix doesn't change things.

[enh]

isn't there some compiler flag that changes the interpretation of undefined macros? i'm guessing it's being replaced with 0 for us, but not for the submitter?

can we see the submitter's compiler flags?

[jmgao]

I think a header that's getting included before is doing #define __USE_FILE_OFFSET64, there's at least one example I could find in the android tree, zlib.

[DanAlbert]

I'm going to go ahead and add the defined check around that anyway, since it is a correct change, but unless you can send me your project I don't think I'm going to be able to figure out what test we're missing that could have caught this.

As for your other errors:

~/Library/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/fstream:824:9: error: use of undeclared identifier 'fseeko'

I wish the r16 blog post wasn't blocked behind process. Here's the thing I would be pointing you toward:

_FILE_OFFSET_BITS=64

tl;dr: Don’t set _FILE_OFFSET_BITS=64 if you want to keep the behavior present in old NDKs.
Historically, setting _FILE_OFFSET_BITS=64 in the NDK did nothing. This feature was not present in the deprecated headers at all. With unified headers, the NDK now has up to date headers with support for this feature.

_FILE_OFFSET_BITS=64 is a macro you can define in your application to get support for a 64-bit off_t in 32-bit code. This works by both making off_t 64-bit (by default it is 32-bit in 32-bit code) and by implicitly replacing calls to APIs like lseek with calls to lseek64.

Support for _FILE_OFFSET_BITS=64 was not added to Android in a single release. One API, lseek64, has always been in bionic. Most APIs were added in Lollipop, and a few more were not added until later releases.

If you’re targeting a release that does not support the 64-bit off_t variant of a function you are using and have set _FILE_OFFSET_BITS=64, the function will not be available. This is in contrast to the behavior for r15 and r15b (but matches r15c) where the functions were wrongly exposed with a 32-bit off_t that would be silently truncated.

Note that the 64-bit off_t APIs are still available without _FILE_OFFSET_BITS=64 under different names. For example, instead of lseek, call lseek64. Instead of off_t, use off64_t.

Finally, since this feature is new to the NDK with unified headers, if you just want to return to the pre-unified headers behavior, all you need to do is stop setting _FILE_OFFSET_BITS=64.

For more information about off_t ABI details in bionic, see the Bionic 32-bit ABI bugs doc.

[enh]

yeah, or -D__USE_FILE_OFFSET64 ... it's not clear to me from the snippet of the failure whether the submitter is building boost or building something that's using boost, and i'm beginning to suspect the latter...

[sephiroth74]

I'm building boost from the sources (1.60)

[enh]

here's how i build boost 1.64 from source: https://gist.github.com/enh/b2dc8e2cbbce7fffffde2135271b10fd

[sephiroth74]

The origin of the error from boost/libs/filesystem/src/operations.cpp:18:

#define __USE_FILE_OFFSET64 // but that is harmless on Windows and on POSIX

chaging the file sources/android/support/include/stdio.h from:
#if __USE_FILE_OFFSET64 && __ANDROID_API__ < __ANDROID_API_N__

into:
#if defined(__USE_FILE_OFFSET64) && __ANDROID_API__ < __ANDROID_API_N__

fix the issue

[enh]

fixed in master, cherrypick to r16 release: https://android-review.googlesource.com/#/c/platform/ndk/+/479579/

[enh]

will be fixed in r16beta2.

[rcdailey]

Not sure if I should open another issue for this. This seems very related (maybe the same issue) so I'll start here.

Here's my command line (using CMake):

cmake -G "Ninja" -D ANDROID_TOOLCHAIN=clang -D ANDROID_ABI=armeabi-v7a -D ANDROID_PLATFORM=android-15 -D ANDROID_STL=c++_shared -D CMAKE_BUILD_TYPE=Release -D CMAKE_TOOLCHAIN_FILE="%ANDROID_NDK%/build/cmake/android.toolchain.cmake"

I'm using android NDK r16b. I'm building boost from the following github repository (which supports building boost using CMake):

https://github.com/Orphis/boost-cmake

I get the following compiler error:

[490/1071] Building CXX object CMakeFiles/Boost_filesystem.dir/boost/boost_1_64_0/libs/filesystem/src/operations.cpp.o
FAILED: CMakeFiles/Boost_filesystem.dir/boost/boost_1_64_0/libs/filesystem/src/operations.cpp.o
E:\android\ndk\toolchains\llvm\prebuilt\windows-x86_64\bin\clang++.exe --target=armv7-none-linux-androideabi --gcc-toolchain=E:/android/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64 --sysroot=E:/android/ndk/sysroot  -DBOOST_ALL_NO_LIB=1 -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_FILESYSTEM_STATIC_LINK=1 -isystem E:/code/_external/third-party-source/boost/boost/boost_1_64_0 -isystem E:/android/ndk/sources/cxx-stl/llvm-libc++/include -isystem E:/android/ndk/sources/android/support/include -isystem E:/android/ndk/sources/cxx-stl/llvm-libc++abi/include -isystem E:/android/ndk/sysroot/usr/include/arm-linux-androideabi -D__ANDROID_API__=15 -g -DANDROID -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -fno-integrated-as -mthumb -Wa,--noexecstack -Wformat -Werror=format-security -std=c++11    -w -std=c++11 -MD -MT CMakeFiles/Boost_filesystem.dir/boost/boost_1_64_0/libs/filesystem/src/operations.cpp.o -MF CMakeFiles\Boost_filesystem.dir\boost\boost_1_64_0\libs\filesystem\src\operations.cpp.o.d -o CMakeFiles/Boost_filesystem.dir/boost/boost_1_64_0/libs/filesystem/src/operations.cpp.o -c E:/code/_external/third-party-source/boost/boost/boost_1_64_0/libs/filesystem/src/operations.cpp
E:/code/_external/third-party-source/boost/boost/boost_1_64_0/libs/filesystem/src/operations.cpp:1705:12: error: no member named 'truncate' in the global namespace
    error(!BOOST_RESIZE_FILE(p.c_str(), size) ? BOOST_ERRNO : 0, p, ec,
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
E:/code/_external/third-party-source/boost/boost/boost_1_64_0/libs/filesystem/src/operations.cpp:224:38: note: expanded from macro 'BOOST_RESIZE_FILE'
#   define BOOST_RESIZE_FILE(P,SZ)(::truncate(P, SZ)== 0)
                                   ~~^
1 error generated.

Is this the same issue? I found this issue by following through a series of other github issues that make these seem connected:

• compile problems on Android ndk r16 beta 1 boostorg/filesystem#56
• truncate function not available for Android Level 19 #516
• Compile error for Android Level API lower then 21 with NDK r16 boostorg/filesystem#65

[rcdailey]

Forgot to mention @DanAlbert just in case email notifications are turned off for closed issues. Please let me know if I should open a new issue.

[rprichard]

Could we create legacy inlines for truncate[64]? It's a system call with signature int truncate(const char *path, off_t length); -- i.e. it doesn't involve FILE*. I see an existing legacy inline, wait4, that's also just a system call.

I think we're generally trying to avoid having programs access system calls directly, in favor of accessing them via libc.so. Maybe that's a reason not to provide an inline.

[rcdailey]

@DanAlbert: Sorry to ping you again, I know you're busy. If you get a moment I'd love a response to my earlier post related to the boost filesystem truncate issue.

[DanAlbert]

@rcdailey your analysis looks correct to me.

[rcdailey]

@DanAlbert Based on that confirmation, then why do I still see the issue on r16b? @enh stated it was fixed in r16 beta 2. Where do I go from here? I haven't seen any further progress on this specific issue in months (I guess because everyone believes it to be fixed). I'm not sure where the discrepancy is. Not sure at this point if it's a boost.filesystem issue or an NDK issue.

[DanAlbert]

@rcdailey: Because unfortunately that's a boost bug, not an NDK bug.

[rcdailey]

@DanAlbert Quote from someone in the other thread:

From the preprocessed output, there is no declaration of ::truncate, which I think is supposed to be there regardless of whether we define _FILE_OFFSET_BITS or __USE_FILE_OFFSET64 or not. This is part of POSIX API, and, if I understand correctly, it was also provided by an older NDK for the target Android API version you use. So, the fact that it is missing seems to me a bug in NDK.

Granted, we shouldn't define __USE_FILE_OFFSET64 ourselves as this is an internal macro specific to libc. The public POSIX macro is _FILE_OFFSET_BITS, which is the one we should define. But this may be a workaround for some broken libc, I don't know. If this is __USE_FILE_OFFSET64 that breaks NDK then we should remove it in this case.

The filesystem issue says it's an NDK bug. The NDK issue says it's a boost bug. Honestly I don't know much about the actual issue or its root cause, but this back and forth can't go on forever can it? Where is the actual issue?

[enh]

this is the mistake:

if I understand correctly, it was also provided by an older NDK for the target Android API version you use

no, it wasn't. in older NDKs, _FILE_OFFSET_BITS and __USE_FILE_OFFSET64 were ignored. they just used to get a 32-bit off_t and 32-bit functions. the current NDK supports _FILE_OFFSET_BITS=64, but at the cost of various degrees of support at different API levels.

if they want their old behavior back 100%, just don't define _FILE_OFFSET_BITS/__USE_FILE_OFFSET64 if defined(__ANDROID__).

[enh]

(commented on the boost bug too.)

[pps83]

IMO, ndk headers are buggy (all that std::to_string, log2, 64 bit file io fails) that it's difficult to take serious stuff like NDK issue says it's a boost bug.

things like that don't even compile with current latest r16b NDK:

#define _FILE_OFFSET_BITS 64
#include <cstdio>

Lots of projects have all kinds of bug reports revolving around these issues and it's not even clear what the definite answer to all those, what all these projects need to do to handle these ndk/android issues and how to make code portable across different versions of NDK versions and api targets.

[DanAlbert]

That test case works fine if you use a supported STL. We even have a test case to prove it: https://android.googlesource.com/platform/ndk/+/master/tests/build/libcxx_headers_file_offset_bits_64/jni

[DanAlbert]

all that std::to_string

This also works if you use a supported STL. The bugs are in the deprecated components, not the NDK headers.

[pps83]

what is supported STL, is it in reference to std::to_string? are you referring to libc++?

[DanAlbert]

Yes.

[rcdailey]

In the case I've been linking to over on the boost side, I was using libc++ and it didn't work. Not sure if that's the same case Dan is talking about, though. I still haven't heard the NDK and boost developers come to a consensus on what the root cause of the issue is and where the bug lies. A patch was proposed on the boost side that still blames this on the NDK as a bug on their side, so I'm not sure what the long term resolution is.

[enh]

I still haven't heard the NDK and boost developers come to a consensus

it really comes down to whether the boost folks want to ignore reality or accept it :-)

[DanAlbert]

In the case I've been linking to over on the boost side, I was using libc++ and it didn't work.

Yes, that issue is a boost bug. The test case @pps83 gave is unrelated to that issue.

[pps83]

With r16b so much code fails to compile, that I'm starting to think that my NDK download was corrupted while downloading :) Just about every library in my project fails to compile. I guess, all of them had bugs?

C:/android-ndk-r16b/sources/cxx-stl/gnu-libstdc++/4.9/include\cstdlib:166:3: error: declaration conflicts with target of using declaration already in scope
  abs(long __i) { return __builtin_labs(__i); }
  ^
C:/android-ndk-r16b/sources/cxx-stl/llvm-libc++/include\stdlib.h:111:44: note: target of using declaration
inline _LIBCPP_INLINE_VISIBILITY long      abs(     long __x) _NOEXCEPT {return  labs(__x);}
                                           ^

And just about every project fails with these random erros in std headers, lots of google own projects fail with similar errors. Compile error fest. Is there some magic thing that should have been defined, like -DPLEASE_COMPILE=1?

[rprichard]

sources/cxx-stl/gnu-libstdc++/4.9/include has headers for gnustl.

sources/cxx-stl/llvm-libc++/include has headers for the libc++ STL.

You seem to have both sets of headers on the compiler search path at the same time. That won't work -- you need to use only a single STL to compile your code. Maybe your project's build system is adding one (or both) of the above directories using -I <path> or -isystem <path>?

[juanbtc]

i have the same problem

[armeabi-v7a] Compile++ thumb: cryptopp <= network.cpp
[armeabi-v7a] Compile++ thumb: cryptopp <= rc2.cpp
[armeabi-v7a] Compile++ thumb: cryptopp <= seal.cpp
[armeabi-v7a] Compile++ thumb: cryptopp <= square.cpp
In file included from /home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/src/posix/fs.cpp:26:
In file included from /home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/include/mega.h:33:
In file included from /home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/include/mega/types.h:45:
In file included from /home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/include/mega/posix/megasys.h:45:
/home/john/Android/Ndk/sources/cxx-stl/llvm-libc++/include/fstream:824:9: error: use of undeclared identifier 'fseeko'
    if (fseeko(__file_, __width > 0 ? __width * __off : 0, __whence))
        ^
/home/john/Android/Ndk/sources/cxx-stl/llvm-libc++/include/fstream:197:5: note: in instantiation of member function
      'std::__ndk1::basic_filebuf >::seekoff' requested here
    basic_filebuf();
    ^
/home/john/Android/Ndk/sources/cxx-stl/llvm-libc++/include/fstream:1019:14: note: in instantiation of member function
      'std::__ndk1::basic_filebuf >::basic_filebuf' requested here
    explicit basic_ifstream(const char* __s, ios_base::openmode __mode = ios_base::in);
             ^
/home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/src/posix/fs.cpp:1425:14: note: in instantiation of member function
      'std::__ndk1::basic_ifstream >::basic_ifstream' requested here
    ifstream infile(configFile);
             ^
In file included from /home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/src/posix/fs.cpp:26:
In file included from /home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/include/mega.h:33:
In file included from /home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/include/mega/types.h:45:
In file included from /home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/include/mega/posix/megasys.h:45:
/home/john/Android/Ndk/sources/cxx-stl/llvm-libc++/include/fstream:826:20: error: use of undeclared identifier 'ftello'
    pos_type __r = ftello(__file_);
                   ^
/home/john/Android/Ndk/sources/cxx-stl/llvm-libc++/include/fstream:842:9: error: use of undeclared identifier 'fseeko'
    if (fseeko(__file_, __sp, SEEK_SET))
        ^
/home/john/Android/Ndk/sources/cxx-stl/llvm-libc++/include/fstream:197:5: note: in instantiation of member function
      'std::__ndk1::basic_filebuf >::seekpos' requested here
    basic_filebuf();
    ^
/home/john/Android/Ndk/sources/cxx-stl/llvm-libc++/include/fstream:1019:14: note: in instantiation of member function
      'std::__ndk1::basic_filebuf >::basic_filebuf' requested here
    explicit basic_ifstream(const char* __s, ios_base::openmode __mode = ios_base::in);
             ^
/home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/src/posix/fs.cpp:1425:14: note: in instantiation of member function
      'std::__ndk1::basic_ifstream >::basic_ifstream' requested here
    ifstream infile(configFile);
             ^
In file included from /home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/src/posix/fs.cpp:26:
In file included from /home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/include/mega.h:33:
In file included from /home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/include/mega/types.h:45:
In file included from /home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/include/mega/posix/megasys.h:45:
/home/john/Android/Ndk/sources/cxx-stl/llvm-libc++/include/fstream:906:13: error: use of undeclared identifier 'fseeko'
        if (fseeko(__file_, -__c, SEEK_CUR))
            ^
/home/john/Android/Ndk/sources/cxx-stl/llvm-libc++/include/fstream:197:5: note: in instantiation of member function
      'std::__ndk1::basic_filebuf >::sync' requested here
    basic_filebuf();
    ^
/home/john/Android/Ndk/sources/cxx-stl/llvm-libc++/include/fstream:1019:14: note: in instantiation of member function
      'std::__ndk1::basic_filebuf >::basic_filebuf' requested here
    explicit basic_ifstream(const char* __s, ios_base::openmode __mode = ios_base::in);
             ^
/home/john/AndroidStudioProjects/android/app/src/main/jni/mega/sdk/src/posix/fs.cpp:1425:14: note: in instantiation of member function
      'std::__ndk1::basic_ifstream >::basic_ifstream' requested here
    ifstream infile(configFile);
             ^
4 errors generated.
make: *** [/home/john/AndroidStudioProjects/android/app/src/main/obj/local/armeabi-v7a/objs/megasdk/sdk/src/posix/fs.o] Error 1
make: *** Waiting for unfinished jobs....

[DanAlbert]

Did you try the fixes mentioned in this bug?

[rcdailey]

I personally was able to get past this issue using NDK r17 and the newest version of Boost. Several issues on the boost side were filed, this was a boost bug not an NDK problem from what I could deduce. I think it's fair to close the issue, although the topic has deviated a bit, so I'm not sure which issue we're tracking here anymore.