chibi-scheme on Windows with msys2's mingw-w64 fails to build

[CinchBlue]

• The flags for __MINGW32__ and _WIN32 are not necessarily disjoint. On msys2's mingw-w64, you cannot correctly compile since lcmpstri will be defined twice. Therefore, one possible idea is to change that line (chibi/features.h , line 760):

  #elif defined(_WIN32) && !defined(__MINGW32__)
  

• Finally, including the headers for <unistd.h> and <fcntl.h> for main.c does nothing. This is because fcntl() is not provided on Windows and with mingw-w64.

[ashinn]

I could make this change but not test it, not having access to a windows machine. Could you test? Or send a patch?

[CinchBlue]

@ashinn I don't think I can fix it, that's the thing. I could test if it works on my machine by building from source if you direct me to the branch I need to pull. There's no substitution for that function on MinGW-w64 so I'm a bit stuck...

[ashinn]

You don't need the unistd.h or fcntl.h headers on Windows - in fact they should already be conditionally compiled out. These are only needed for the green threads, which are disabled on Windows. If they're getting included somewhere it's a recent bug and I can remove it.

The only change that should be needed is the one you already proposed.

[mkeeter]

@ashinn I'm interested in helping to get an MSYS2 build working out of the box.

So far, I've made this commit, which makes chibi-scheme.exe and libchibi-scheme.dll build.

There are two issues:

1. Running chibi-scheme.exe (through chibi-run) fails immediately with the error message

ERROR: undefined variable: |▒▒▒▒|

This is pretty weird – any ideas? Running with -q or -Q both get to a REPL and seem to work.

2. The build process fails after this point with many errors in filesystem.c. It looks like most of the compiled libs are POSIX-flavored – do you expect any of them to work on Windows / MSYS2 out of the box? If not, can you point me to where I should disable them in the build system?

[CinchBlue]

@ashinn Let me post what happens with MinGW:

$ make
gcc -c -DSEXP_USE_STRING_STREAMS=0 -DBUILDING_DLL -DSEXP_USE_INTTYPES -Iinclude  -Wall -g -g3 -O3  -o main.o main.c
In file included from include/chibi/sexp.h:17:0,
                 from include/chibi/eval.h:12,
                 from main.c:9:
include/chibi/features.h:763:0: warning: ignoring #pragma warning  [-Wunknown-pragmas]
 #pragma warning(disable:4146) /* unary minus operator to unsigned type */

In file included from C:/msys64/mingw64/x86_64-w64-mingw32/include/guiddef.h:148:0,
                 from C:/msys64/mingw64/x86_64-w64-mingw32/include/winnt.h:628,
                 from C:/msys64/mingw64/x86_64-w64-mingw32/include/minwindef.h:163,
                 from C:/msys64/mingw64/x86_64-w64-mingw32/include/windef.h:8,
                 from C:/msys64/mingw64/x86_64-w64-mingw32/include/windows.h:69,
                 from C:/msys64/mingw64/x86_64-w64-mingw32/include/rpc.h:16,
                 from C:/msys64/mingw64/x86_64-w64-mingw32/include/objbase.h:7,
                 from C:/msys64/mingw64/x86_64-w64-mingw32/include/shlwapi.h:13,
                 from include/chibi/features.h:767,
                 from include/chibi/sexp.h:17,
                 from include/chibi/eval.h:12,
                 from main.c:9:
C:/msys64/mingw64/x86_64-w64-mingw32/include/string.h: In function 'lstrcmpi':
C:/msys64/mingw64/x86_64-w64-mingw32/include/string.h:116:138: error: '__sizeMaxCompare' undeclared (first use in this function)
   __CRT_INLINE int __cdecl strncasecmp (const char *__sz1, const char *__sz2, size_t __sizeMaxCompare) { return _strnicmp (__sz1, __sz2, __sizeMaxCompare); }
                                                                                                                                          ^~~~~~~~~~~~~~~~
C:/msys64/mingw64/x86_64-w64-mingw32/include/string.h:116:138: note: each undeclared identifier is reported only once for each function it appears in
In file included from include/chibi/sexp.h:17:0,
                 from include/chibi/eval.h:12,
                 from main.c:9:
C:/msys64/mingw64/x86_64-w64-mingw32/include/string.h: At top level:
include/chibi/features.h:764:20: error: redefinition of 'lstrcmpi'
 #define strcasecmp lstrcmpi
                    ^
include/chibi/features.h:765:32: note: previous definition of 'lstrcmpi' was here
 #define strncasecmp(s1, s2, n) lstrcmpi(s1, s2)
                                ^
main.c: In function 'sexp_make_unblocking':
main.c:88:29: warning: implicit declaration of function 'fcntl' [-Wimplicit-function-declaration]
     sexp_port_flags(port) = fcntl(sexp_port_fileno(port), F_GETFL);
                             ^~~~~
main.c:88:59: error: 'F_GETFL' undeclared (first use in this function)
     sexp_port_flags(port) = fcntl(sexp_port_fileno(port), F_GETFL);
                                                           ^~~~~~~
main.c:89:33: error: 'O_NONBLOCK' undeclared (first use in this function)
   if (!(sexp_port_flags(port) & O_NONBLOCK))
                                 ^~~~~~~~~~
main.c:90:39: error: 'F_SETFL' undeclared (first use in this function)
     if (fcntl(sexp_port_fileno(port), F_SETFL, sexp_port_flags(port) | O_NONBLOCK) == 0)

There appears to be 2 parts to this:

1. mingw-w64 may have something wrong with its own string comparison function. I have no idea why the compilation is failing with an error in the function definition in the string.h file itself, but I don't think it's my fault.

2. Using fcntl in main.c causes a compilation on Windows.

[ashinn]

@mkeeter, thanks! I've applied your changes.

-q means it can load init-7.scm fine. If you can (import (srfi 1)) the module system and path handling are working as well. From the error you get it sounds like it's trying to load a binary file as source code. I'll try to provide you with a path to trace loading.

Not all C libraries are expected to work - I simply haven't had time to port everything. As a simple fix we can condition out code that doesn't currently compile on Windows.

@VermillionAzure, try the changes I pushed from @mkeeter.

[CinchBlue]

@ashinn I've done an alternative build on MSYS2 instead of MSYS2 MinGW-w64. It works fine.

So with POSIX emulation, Chibi will compile without much errors. However, when you remove POSIX compatibility (since MinGW-w64 will not support POSIX signals, fork(), etc.), Chibi will fail to build. So it will not build with MSYS2's MinGW-w64 current compatible shell environment and packages.

Additionally, some additional warnings and errors are triggered. I have submitted a patch that move around two lines of macro definitions and includes an error log file for the rest of the errors.

#398

[ashinn]

Yes, it's known much of the POSIX stuff won't work on Windows. These are all optional libraries - they would be nice to have, but I don't have time (or a Windows machine) to make this work.

[mkeeter]

@ashinn agreed that those libraries are optional, and they should be turned off for MinGW-w64 (at compile-time) so that a plain make runs without errors. I can do the patching and testing if you point me to where in the build system I can disable them!

[CinchBlue]

@mkeeter @ashinn

  25 CHIBI_COMPILED_LIBS =
  26 CHIBI_CRYPTO_COMPILED_LIBS = #lib/chibi/crypto/crypto$(SO)
  27 CHIBI_IO_COMPILED_LIBS = #lib/chibi/io/io$(SO)
  28 CHIBI_OPT_COMPILED_LIBS = #lib/chibi/optimize/rest$(SO) \
  29     #lib/chibi/optimize/profile$(SO)
  30 EXTRA_COMPILED_LIBS ?=
  31 COMPILED_LIBS = #$(CHIBI_COMPILED_LIBS)                               $(CHIBI_IO_COMPILED_LIBS) \
  32     #$(CHIBI_OPT_COMPILED_LIBS) $(CHIBI_CRYPTO_COMPILED_LIBS) \
  33     #$(EXTRA_COMPILED_LIBS) \
  34     #lib/srfi/18/threads$(SO) lib/srfi/27/rand$(SO) lib/srfi/33/      bit$(SO) \
  35     #lib/srfi/39/param$(SO) lib/srfi/69/hash$(SO) lib/srfi/95/        qsort$(SO) \
  36     #lib/srfi/98/env$(SO) lib/scheme/time$(SO)

If I change the Makefile at those line numbers to the content you see above, the build is successful, with the following command outputting the entire stdout/stderr output to log.txt:

make &> log.txt

log.txt

As you can see, I essentially removed all of the compiled libraries. But when I try to run it from the MSYS2 MinGW-w64 command line, I get:

ERROR: couldn't find include: "srfi/69/hash.dll"

I don't exactly understand this error message and why the compiled-library-less version of Chibi requires a SFRI DLL, but the core of chibi-scheme should not be using an SFRI, right? I also tried doing make install but that didn't terminate without errors. If I do make install &> install.txt, I get this log file:

install.txt

I hope this has helped,
Austin

[ashinn]

Hash tables are used by equal? in (scheme base), to check for cyclic structures.

You don't want to disable all compiled libs, only the ones that use posix. Definitely keep SRFIs 27, 33, 69, 95.

You can move the definition of COMPILED_LIBS to Makefile.detect, conditionally defining for windows.

[sbjaver]

Hi,
I was unable to find anyones conclusion to the undefined variable behavior discussed above?
I've built with mingw-w64, disabled building of some of the libraries, etc, and I can run chibi-scheme.exe -q, but if I launch without '-q', I also see the undefined variable issue:

ERROR: undefined variable: |MZ▒▒▒▒@▒▒▒|

Thank You
Joe

[okuoku]

It seems Chibi-scheme does assume sizeof(int) == sizeof(long) on _WIN32 which is incorrect on Win64 -- so it cannot intern symbols properly on it. I haven't tracked down where it is yet. As it works pretty well on Cygwin64 which shares mostly same ABI with Win64, just fiddling with typedefs around #ifdef _WIN32 should be enough though.

(I tried some https://github.com/okuoku/chibi-scheme/tree/win32 but it seems we have way too many things to stub-out because standard libraries depends heavily on POSIX and it depends 128bits arithmetic which is not supported on MSVC so I gave up..)

[okuoku]

Chibi-scheme does assume sizeof(int) == sizeof(long) on _WIN32 which is incorrect on Win64

Uh, it seems I found one:

@@ -1860,10 +1860,10 @@ sexp sexp_apply_writer(sexp ctx, sexp writer, sexp obj, sexp out) {

 sexp sexp_write_one (sexp ctx, sexp obj, sexp out) {
 #if SEXP_USE_HUFF_SYMS
-  unsigned long res;
+  sexp_uint_t res;
 #endif
-  unsigned long len, c;
-  long i=0;
+  sexp_uint_t len, c;
+  sexp_sint_t i=0;
 #if SEXP_USE_FLONUMS
   double f, ftmp;
 #endif

It is not enough though -- it seems symbol->string truncates output by some:

stubs/lib/../lib/srfi/144/math.c:335:32: warning: implicit declaration of function 'fpclas'; did you mean 'fpclass'? [-Wimplicit-function-declaration]
   res = sexp_make_integer(ctx, fpclas(sexp_flonum_value(arg0)));
                                ^~~~~~
                                fpclass

fpclas should be fpclass here. Other symbol output from chibi-genstatic truncated as well, NULL will be NU, sha_context will be sha_c ...

[okuoku]

Anyway, anyhow, finally ended up 3 series of pull-reqests #439 #438 #437 .

I haven't ported Makefiles to MinGW64 environment yet as I use CMake for my purpose https://github.com/okuoku/chibi-scheme-cmake ... (I'm not sure it would worthwhile tackle with it -- even visual studio now bundles CMake..)

[okuoku]

I was unable to find anyones conclusion to the undefined variable behavior discussed above?

@sbjaver : I think the undefined-variable-issue is caused by two problems:

1. Incomplete Win64 porting on chibi-scheme. That broke huffman-coded symbols (SEXP_USE_HUFF_SYMS) badly and symbols are not work on Win64. I hope I've addressed it with Win32: Fix win32 port #438
2. SEXP_USE_DL is not implemented at all on Win32/Win64. I (personally) don't use the feature so I've ported the buildsystem to CMake and simply amalgamated .dlls into executable as same for Emscripten

Unfortunately, I have no time to seriously adopt Makefile for Win32/Win64... I guess we could borrow the same strategy with Emscripten, or, just port SEXP_USE_DL on Win32/64 might be easier.

[sbjaver]

@okuoku thanks for the suggestions on this. I tried your patches, but in my awful mingw32-make environment. I also tried using your cmakelists with vs 2017 and had some issues. Would you mind giving me any tips you can on how you perform the static cmake build using visual studio?

[okuoku]

@sbjaver : I prepared a bit more sorted tree: https://github.com/okuoku/chibi-scheme/tree/for-issue393

I don't expect CMakeList.txt would get merged like this -- perhaps it would be better to have git-submodule'ed tree somewhere.

To build the branch,

1. checkout for-issue393 branch
2. Install CMake and Ninja on your PATH (Makefile generators are depends on sh which would not work as expected in the most cases)
3. mkdir build && cd build
4. cmake -G Ninja ..
5. ninja
6. ninja test

I only tried with 64bit version of Git for windows SDK https://github.com/git-for-windows/build-extra/releases (mingw64.exe).

... It seems to doesn't work on Win32. I'm investigating why.

[sbjaver]

Thanks again @okuoku - I'll give this a try.
FWIW -
I managed to track down part of the actual issue I'm seeing, which is again the "undefined variable" issue. It appears that it is simply because I had SEXP_USE_DL=0, sexp_load_op was asked to load ast.dll, and since I had the dynamic loading code compiled out, it tries the to load the .dll as text. Someone mentioned this theory somewhere I believe. The "MZ..." i confirmed is actually the head bytes of ast.dll.

I enabled SEXP_USE_DL (made some additional changes in gc_heap.c) to not call unavailable dlxxx functions) and was able to proceed futher. But then I eventually bumped back into needing to port filesystem to mingw.

[okuoku]

(Fixed 32bit build issue on okuoku@d8b2ba8 I will update pull request too. It's now compatible with VS2017's "open with visual studio")

simply because I had SEXP_USE_DL=0

Ah, yeah, it seems Chibi-scheme will try to autoload DLLs as a script file.. That should make sense enabling SEXP_USE_DL made some progress then.

But then I eventually bumped back into needing to port filesystem to mingw

#439 includes patch against filesystem.stub 735719d but it just stub-out APIs/fields that not exists in MSVCRT; I found procedures in (chibi filesystem) is a bit too low-level to interface with Windows...

[sbjaver]

Awesome! I'm able to build and run! Thanks!