Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clang does not compile re2c 0.15.x #122

Closed
otaran opened this issue Nov 27, 2015 · 20 comments

Comments

@otaran
Copy link
Contributor

commented Nov 27, 2015

Steps To Reproduce

  1. Download https://github.com/skvadrik/re2c/archive/0.15.tar.gz, unpack and cd to root directory
  2. Run git init (otherwise ./re2c/autogen.sh will fail with error fatal: Not a git repository (or any of the parent directories): .git)
  3. Run cd re2c (otherwise ./re2c/autogen.sh will fail with error autoreconf: 'configure.ac' or 'configure.in' is required)
  4. Run ./autogen.sh
  5. Run ./configure
  6. Run make and observe

Expected Result

make runs fine and makes re2c binary

Actual Result

make fails with following error:

src/codegen/emit_dfa.cc:250:65: error: use of overloaded operator '<<' is ambiguous (with operand types 're2c::OutputFile' and 'const size_t'
      (aka 'const unsigned long'))
        o << indent(ind++) << "static void *" << opts->yyctable << "[" << conds << "] = {\n";
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~
./src/codegen/output.h:84:22: note: candidate function
        friend OutputFile & operator << (OutputFile & o, char c);
                            ^
./src/codegen/output.h:85:22: note: candidate function
        friend OutputFile & operator << (OutputFile & o, uint32_t n);
                            ^
./src/codegen/output.h:86:22: note: candidate function
        friend OutputFile & operator << (OutputFile & o, uint64_t n);
                            ^

Notes

Reproducible with re2c versions 0.15, 0.15.1 and 0.15.2

OS: Apple OS X El Capitan (10.11.1)
Developer Tools: Xcode 7.1.1 (7B1005)

$ clang++ --version
Apple LLVM version 7.0.0 (clang-700.1.76)
Target: x86_64-apple-darwin15.0.0
Thread model: posix
@skvadrik

This comment has been minimized.

Copy link
Owner

commented Nov 27, 2015

Hi Oleksii!

On my system size_t is a simple typedef, same as uint64_t, so compiler uses one of the overloads and everything is fine. The problem is described e.g. here. I'll make a fix, but I cannot check if it works on OS X before releasing 0.15.3. Would you be willing to help (checkout re2c from git and make sure it compiles)?

By the way, why would you ever need to run autogen.sh when building from tarball? The pregenerated ./configure should work.

@skvadrik skvadrik self-assigned this Nov 27, 2015

@skvadrik skvadrik added this to the 0.15.3 milestone Nov 27, 2015

skvadrik added a commit that referenced this issue Nov 27, 2015

Don't use 'operator <<' overloads with integral types: resolution is …
…platform-dependent.

See bug #122 "clang does not compile re2c 0.15.x".

Example of error on Mac OS X:
    src/codegen/emit_dfa.cc:250:65: error: use of overloaded operator '<<' is ambiguous (with operand types 're2c::OutputFile' and 'const size_t'
          (aka 'const unsigned long'))
            o << indent(ind++) << "static void *" << opts->yyctable << "[" << conds << "] = {\n";
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^  ~~~~~
    ./src/codegen/output.h:84:22: note: candidate function
            friend OutputFile & operator << (OutputFile & o, char c);
                                ^
    ./src/codegen/output.h:85:22: note: candidate function
            friend OutputFile & operator << (OutputFile & o, uint32_t n);
                                ^
    ./src/codegen/output.h:86:22: note: candidate function
            friend OutputFile & operator << (OutputFile & o, uint64_t n);
                            ^

On OS X 'size_t' is neither 'uint32_t' nor 'uint64_t', resolution is therefore ambiguous.
@skvadrik

This comment has been minimized.

Copy link
Owner

commented Nov 27, 2015

Fixed in HEAD (in this commit: 499537c).
I'm planning to release re2c-0.15.3 by the end of the day.

@otaran

This comment has been minimized.

Copy link
Contributor Author

commented Nov 27, 2015

Hi Ulya,

It did fix the original build error, but it revealed a bunch of new errors (error log is too log, please see http://pastebin.com/Yjmgiyuj). I think there is no point in releasing 0.15.3 because re2c is still can't be compiled on OS X. I will investigate why build started to fail all of a sudden.

@otaran

This comment has been minimized.

Copy link
Contributor Author

commented Nov 27, 2015

By the way, why would you ever need to run autogen.sh when building from tarball? The pregenerated ./configure should work.

There is no ./re2c/configure script in archive available at https://github.com/skvadrik/re2c/archive/0.15.2.tar.gz:

$ pwd
/private/tmp/re2c20151127-33742-2n848j/re2c-0.15.2
$ ls re2c
CHANGELOG       NO_WARRANTY     autogen.sh      build.sh        configure.ac    doc             run_tests.sh.in test
Makefile.am     README          bootstrap       build_mingw.sh  distcheck.sh    examples        src
@skvadrik

This comment has been minimized.

Copy link
Owner

commented Nov 27, 2015

There is no |./re2c/configure| script in archive available at
https://github.com/skvadrik/re2c/archive/0.15.2.tar.gz:

Seems that you downloaded wrong tarball. The right one is here
http://re2c.org/install/install.html (the same one on github page
https://github.com/skvadrik/re2c/releases). How did you get that link?

@otaran

This comment has been minimized.

Copy link
Contributor Author

commented Nov 27, 2015

Oops, you are right, I was downloading from link "Source Code" on https://github.com/skvadrik/re2c/releases, but actually I should have downloaded re2c-0.15.2.tar.gz archive directly.

@skvadrik

This comment has been minimized.

Copy link
Owner

commented Nov 27, 2015

Build failure on OS X is caused by the same problem: on OS X size_t is a type alias of unsigned long long, which is neither uint32_t (type alias of unsigned int), nor uint64_t (type alias of unsigned long) .

On my platform there is no ambiguity: size_t is unsigned long and uint64_t overload is exact match.

There is a similar problem when building on Windows: long is 32 bits there. Only I can catch errors while cross-compiling on Windows with Mingw, not so with OS X (I know of no way to cross-compile from Linux to OS X).

My way to resolve conflicts is to drop overloaded functions in favour of explicit functions. I'll try to come up with some test that will catch all such cases. Thanks for patience!

@otaran

This comment has been minimized.

Copy link
Contributor Author

commented Nov 28, 2015

There is more than that:

../src/parse/lex.re:169:38: error: expected unqualified-id
                                        else if (opts->target == opt_t::CODE)
                                                                        ^
src/parse/y.tab.h:58:14: note: expanded from macro 'CODE'
#define CODE 260
             ^
src/parse/lex.cc:444:9: warning: use of old-style cast [-Wold-style-cast]
        yych = (YYCTYPE)*++YYCURSOR;
               ^        ~~~~~~~~~~~
../src/parse/lex.re:152:38: error: expected unqualified-id
                                        else if (opts->target == opt_t::CODE)
                                                                        ^
src/parse/y.tab.h:58:14: note: expanded from macro 'CODE'
#define CODE 260
             ^

otaran added a commit to otaran/re2c that referenced this issue Nov 28, 2015

otaran added a commit to otaran/re2c that referenced this issue Nov 28, 2015

@otaran

This comment has been minimized.

Copy link
Contributor Author

commented Nov 28, 2015

I have managed to build re2c on OS X (see #123). Unfortunately built re2c binary fails the majority of tests (577 out of 975), most of them due to some kind of memory corruption:

./run_tests.sh: line 107:  9583 Segmentation fault: 11  $valgrind $wine ../../$re2c $switches "$outx" 2> "$outc.stderr" 1>&2
@skvadrik

This comment has been minimized.

Copy link
Owner

commented Nov 28, 2015

Hmm, interesting! On my platform bison (version 2.4.3) generates token definitions like this:

/* Tokens.  */
#ifndef YYTOKENTYPE
# define YYTOKENTYPE
   /* Put the tokens into the symbol table, so that GDB and other debuggers
      know about them.  */
   enum yytokentype {
     CLOSE = 258,
     CLOSESIZE = 259,
     CODE = 260,
     CONF = 261,
     ID = 262,
     FID = 263,
     FID_END = 264,
     NOCOND = 265,
     REGEXP = 266,
     SETUP = 267,
     STAR = 268
   };
#endif

Which is certainly different from #define CODE 260. The fix is easy (as you suggested, prefix token names with something). But out of curiosity, what version of bison do you use?

Next, test failures. I regularly run tests under valgrind to catch all memory errors and undefined behaviours, so I suspect that all failures stem from the same error and must be easy to fix. Can you provide some details? The easiest way is to choose a small test that fails and get some kind of backtrace, valgrind log or whatever mechanisms you use to debug segfaults on OS X. I greatly appreciate your help!

skvadrik added a commit that referenced this issue Nov 28, 2015

Fix "CODE" symbol collision on OS X (see #122)
On OS X bison generates token enums as CPP macro
constants (y.tab.h):
    #define CODE 260
while on my box it's
   enum yytokentype {
     ...
     CODE = 260,
     ...
   };

That #define causes symbol collision as:

    ../src/parse/lex.re:169:38: error: expected unqualified-id
                                            else if (opts->target == opt_t::CODE)
                                                                            ^
    src/parse/y.tab.h:58:14: note: expanded from macro 'CODE'
    #define CODE 260

Renamed enum entry to TOKEN_CODE.
@skvadrik

This comment has been minimized.

Copy link
Owner

commented Nov 28, 2015

Managed to reproduce test failures on FreeBSD-10.2. (578 out of 977 tests failed with segfaults, so the error must be the same as on OS X). Looking into it.

skvadrik added a commit that referenced this issue Nov 28, 2015

Fixed crashes of 'ostream& operator<< (ostream& os, const char* s)' o…
…n NULL.

Crashes observed on platforms OS X (clang-7.0.0) and FreeBSD-10.2 (clang-3.4).
First reported in bug #122 "clang does not compile re2c 0.15.x".

What caused NULL passed to 'operator <<': re2c always generates content of
header file (regardless of '-t --type-header' option), but the content is
dumped to file (and header filename initialized to non-NULL) only if the
option was enabled.

Fix: always initialize header filename to non-NULL string.
@skvadrik

This comment has been minimized.

Copy link
Owner

commented Nov 28, 2015

This commit 8a361fe fixes test crashes on FreeBSD-10.2 with clang-3.4. It is very likely to fix tests on OS X as well.

@skvadrik

This comment has been minimized.

Copy link
Owner

commented Nov 28, 2015

Some tests are still failing due to sed adding newlines in the end of file.

@otaran

This comment has been minimized.

Copy link
Contributor Author

commented Nov 28, 2015

Looks better now, only 27 out of 975 tests failed.

OS X comes with bison 2.3:

$ bison --version
bison (GNU Bison) 2.3
Written by Robert Corbett and Richard Stallman.

Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Latest bison (3.0.4) generates output that is completely different from code in repository so I didn't test if generated code will be built into valid re2c.

@otaran

This comment has been minimized.

Copy link
Contributor Author

commented Nov 28, 2015

Did you try running make vtests on FreeBSD? When I run make vtests on OS X, every single test fails. It looks like valgrind concatenates its own output to generated *.c files, thus *.c.diff are always not empty. I wonder if this is due to some valgrind configuration which is different on OS X.

@skvadrik

This comment has been minimized.

Copy link
Owner

commented Nov 28, 2015

Latest bison (3.0.4) generates output that is completely different from code in repository so I didn't test if generated code will be built into valid re2c.
I did try to build with bison-3.0.4, everything seems to be ok (builds and tests pass except those 27 expected failures).

Could you show an example of .c file with valgrind's output?

@otaran

This comment has been minimized.

Copy link
Contributor Author

commented Nov 28, 2015

@skvadrik

This comment has been minimized.

Copy link
Owner

commented Nov 28, 2015

Seems that it's an OS X specific memory leak in ctime:

==63510== 16 bytes in 1 blocks are still reachable in loss record 5 of 84
==63510==    at 0x100078EBB: malloc (in /usr/local/Cellar/valgrind/3.11.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==63510==    by 0x100577F84: _nc_table_new (in /usr/lib/system/libsystem_notify.dylib)
==63510==    by 0x1005738F7: _notify_init_globals (in /usr/lib/system/libsystem_notify.dylib)
==63510==    by 0x100581FC3: _os_once (in /usr/lib/system/libsystem_platform.dylib)
==63510==    by 0x100581F8F: _os_alloc_once (in /usr/lib/system/libsystem_platform.dylib)
==63510==    by 0x1005752DF: notify_register_check (in /usr/lib/system/libsystem_notify.dylib)
==63510==    by 0x10035009C: notify_register_tz (in /usr/lib/system/libsystem_c.dylib)
==63510==    by 0x10034FB07: tzsetwall_basic (in /usr/lib/system/libsystem_c.dylib)
==63510==    by 0x1003514DF: localtime (in /usr/lib/system/libsystem_c.dylib)
==63510==    by 0x100351662: ctime (in /usr/lib/system/libsystem_c.dylib)
==63510==    by 0x10000D5C8: re2c::output_version_time(std::__1::basic_ostream<char, std::__1::char_traits<char> >&) (output.cc:447)
==63510==    by 0x10000D561: re2c::OutputFile::wversion_time() (output.cc:133)
==63510==    by 0x10003191F: re2c::parse(re2c::Scanner&, re2c::Output&) (parser.ypp:576)
==63510==    by 0x1000262B0: main (main.cc:51)

re2c calls ctime on every input file (when it dumps generation date). We run valgrind with -q and concatenate valgrind's output and normal output. If there where no errors, the output remains untouched.

You can suppress this particular error using valgrind's suppressions mechanism: http://valgrind.org/docs/manual/manual-core.html#manual-core.suppress . Let me know if you need an example of .supp file.

Beware that there are ~70 expected failures with make vtests (those tests are for various re2c errors, they call exit(1) too early and fail to call a couple of destructors --- this causes memory leaks).

Also, it is perhaps better to run ./run_tests.sh -j1 --valgrind, as make vtests runs in multiple threads and there is one synthetized fat test that eats a lot of RAM under valgrind. I will fix it soon.

@skvadrik

This comment has been minimized.

Copy link
Owner

commented Nov 30, 2015

Fixed the rest of tests, see this commit: 3cf9335

Also, now it must be safe to run make vtests (they won't eat RAM and should run much faster). Still there are some 70 expected failures with valgrind (these are tests for re2c errors).

@otaran

This comment has been minimized.

Copy link
Contributor Author

commented Nov 30, 2015

Looks good, make tests passes all tests on OS X, make vtests still has some failures. I am closing this issue and looking forward to seeing 0.15.3 release. Thank you for your work, Ulya!

@otaran otaran closed this Nov 30, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.