New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] / Master issue: Compiling under Windows (VS2013) #696

Closed
voltagex opened this Issue May 7, 2014 · 146 comments

Comments

Projects
None yet
@voltagex
Copy link

voltagex commented May 7, 2014

I'm (sort-of) interested in getting this working under VS2013, so I'd have a native-built neovim instead of relying on MinGW or similar (although this may be a valid goal as well)

I'd like to collate issues here that are stopping compilation of neovim on Windows on VS2013, in lieu of GitHub having proper blocker/issue dependency support

For now running cmake -G Visual Studio 12 results in:

CMake Error at C:/Program Files (x86)/CMake 2.8/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108 (message):
Could NOT find LibUV (missing: LIBUV_LIBRARY LIBUV_INCLUDE_DIR)
Call Stack (most recent call first):
C:/Program Files (x86)/CMake 2.8/share/cmake-2.8/Modules   /FindPackageHandleStandardArgs.cmake:315 (_FPHSA_FAILURE_MESSAGE)
cmake/FindLibUV.cmake:90 (find_package_handle_standard_args)
CMakeLists.txt:51 (find_package)

Todo:

  • Attempt to build neovim using cmake and VS2013
  • Build & install libuv somewhere that cmake can find it
  • Port if_cscope to use libuv (#810 (comment))
  • start thinking about how to split platform code (os_unix.c os_msvc.c etc)
  • Add bracket to if statements (maybe?)
  • Start splitting compiler options in CMake
  • Refactor vim_FulllName
  • Add include guards: sys/time.h unistd.h
  • Reorder includes here and there, i.e. ensure platform includes and/or vim.h are included before anything else
  • Document the above
  • ???
  • Profit

Thanks to @equalsraf for the above


Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

@aktau

This comment has been minimized.

Copy link
Member

aktau commented May 7, 2014

Hmm, I thought cmake tried to download libuv if not present, or did that change, @jszakmeister?

@jszakmeister

This comment has been minimized.

Copy link
Member

jszakmeister commented May 7, 2014

It's a two-stage process. The third-party area contains the bits to download the deps, and the top-level is used to build and attempts to discover the correct bits. Now, because it's Windows we might need to alter the build step for some dependencies. So there might be several layers of stuff to get through.

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented May 7, 2014

see also: #328

I was planning on re-opening that issue, but this will do.

@voltagex

This comment has been minimized.

Copy link

voltagex commented May 8, 2014

@justinmk - apologies for the duplication. Maybe you could rename your issue a little (Building libuv on MinGW?) and I'll add it to the task list?

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented May 8, 2014

#328 was never about MinGW specifically, it was just about building on Windows in general. @equalsraf left some notes on that issue that may save some time for anyone that wants to continue where s/he left off.

Building libuv on MinGW is not a goal as far as I know. tarruda once pointed out the excellent fact that terminal frontends running in cygwin (for example) can send RPC requests to a nvim binary that was built in MSVC. So there is not much need for a MinGW build. Unless someone wants to work with libuv upstream, I consider that issue tabled.

@voltagex

This comment has been minimized.

Copy link

voltagex commented May 8, 2014

Understood.

@justinmk justinmk added this to the first release milestone May 24, 2014

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Jun 1, 2014

Cheers

Some notes in case someone is trying this as well. For MSVC I keep a branch with some fixes in my own fork of neovim (branch tb-windows2)

  • Windows has its own environment variables for libraries and include folders (LIB and INCLUDE) point those to libuv.lib, etc
  • I've built libuv from source and CMake can find it with some minor changes to FindLibUv.cmake
  • For curses I just used GNU-win32
  • Minor changes to LibIntl (libnames same as llibuv)
  • Building msgpack-c can be a problem in Windows (it assumes a an old version of vcbuild) I just cooked my own CMakeLists.txt to build it (equalsraf/msgpack-c branch:tb-win-cmake)
  • I Installed luajit and luarocks (along with lpeg and lua-cmsgpack) - just followed the instructions (maybe there is a package out there?)
  • I had to disable error checking compile flags for MSVC

Currently it starts building but immediately fails for various reasons

  • It tries to include os_unix_defs.h which in turn goes for sys/params.h
  • Same for sys/select.h
  • It throws a generic syntax error when dealing with the restricted keyword in garray.h

Nothing too surprising there. I'll try to do a more thorough check when I get some more time on windows.

@aktau

This comment has been minimized.

Copy link
Member

aktau commented Jun 1, 2014

It throws a generic syntax error when dealing with the restricted keyword in garray.h

Gah... I figured as much. Does this happen even with MSVC 2013? I was under the impression they were trying to do much better than before. They could've at least aliased __restrict to restrict. As far as I can see, __restrict has been in msvc since version 10: http://msdn.microsoft.com/en-us/library/5ft82fed.aspx.

We could perhaps do #define restrict __restrict for msvc only...

UPDATE: found a blog suggesting a workaround: http://diaryofagraphicsprogrammer.blogspot.be/2014/01/visual-studio-2013-c99-support.html

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Jun 1, 2014

  • yes MSVC 2013
  • The #define seems to work as intended
@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Jun 1, 2014

I almost forgot, If you encounter errors with an undefined struct timeval just include uv.h - it knows where to find it.

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Jun 1, 2014

Also MSVC does not seem to handle inline

#define inline __inline
@aktau

This comment has been minimized.

Copy link
Member

aktau commented Jun 1, 2014

That can thankfully all be wrapped up in some special msvc fixes define
blocks.

Am Sonntag, 1. Juni 2014 schrieb Rui Abreu Ferreira :

Also MSVC does not seem to handle inline

#define inline __inline


Reply to this email directly or view it on GitHub
#696 (comment).

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Jun 1, 2014

Got as far as edit.c until I started hitting some MSVC oddities

src\nvim\edit.c(2588) : error C2275: 'size_t' : illegal use of this type as an expression

which apparently means that the declaration of the variable is not at the start of the function (i.e. it assumes C89), moving the declaration to the start does indeed make the error go away.

I assumed MSVC support for C99 would be better than this. Or maybe I'm missing some compiler flag.

@aktau

This comment has been minimized.

Copy link
Member

aktau commented Jun 1, 2014

which apparently means that the declaration of the variable is not at the start of the function (i.e. it assumes C89), moving the declaration to the start does indeed make the error go away.

That's... very strange. Perhaps it does this in C-mode and somehow all these improvements that were listed in that article I linked are only when compiling as C++ (though that would make very little sense). How does it do if you force it to compile as C++? (EDIT: nevermind, we use too many things that won't compile under C++ anyway).

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Jun 1, 2014

In C++ mode I get some security errors that fail earlier, like

  src\nvim\buffer.c(1366) : error C2440: '=' : cannot convert from 'void *' to 'buf_T *'

Let me go and add a bunch of explicit casts ...

@aktau

This comment has been minimized.

Copy link
Member

aktau commented Jun 1, 2014

No need, you will need 100's of casts, better not to even try.

Maybe this can give us a head start: http://stackoverflow.com/questions/2938460/how-do-i-know-whether-c89-or-c99-is-applied-by-cl-exe?rq=1

@aktau

This comment has been minimized.

Copy link
Member

aktau commented Jun 1, 2014

Another post worth reading: http://blogs.msdn.com/b/vcblog/archive/2013/06/28/c-11-14-stl-features-fixes-and-breaking-changes-in-vs-2013.aspx

It mentions:

Additionally, some C99 Core Language features will be implemented in 2013 RTM:

  • C99 _Bool
  • C99 compound literals
  • C99 designated initializers
  • C99 variable declarations

This seems to be what we need. Now we just need to find out how to activate that.

@udp

This comment has been minimized.

Copy link

udp commented Jun 1, 2014

FWIW I find it a lot easier to compile modern C as C++ with MSVC.

@aktau

This comment has been minimized.

Copy link
Member

aktau commented Jun 1, 2014

FWIW I find it a lot easier to compile modern C as C++ with MSVC.

There are some things that we use that will probably make msvc choke in C++ mode, namely:

  1. Compound literals
  2. Designated initializers
  3. Not casting void * to the correct type (implicit void pointer cast)

So I don't think point-blank compiling as C++ will work.

@voltagex

This comment has been minimized.

Copy link

voltagex commented Jun 3, 2014

@aktau for _Bool, it appears something like https://bitbucket.org/cffi/cffi/commits/d993a73f4d604bb34b86e92aea06c48545f7a287 will work. Everything else I'm not sure yet.

@voltagex

This comment has been minimized.

Copy link

voltagex commented Jun 3, 2014

Could someone let me know what I should update in the task list for this issue?

@aktau

This comment has been minimized.

Copy link
Member

aktau commented Jun 3, 2014

Here's some new info on the Visual Studio 14 beta: http://blogs.msdn.com/b/vcblog/archive/2014/06/03/visual-studio-14-ctp.aspx

They implemented most of the C99 library now. No idea if they did anything about all the other things that prevent compilation though.

Could someone let me know what I should update in the task list for this issue?

@voltagex better ask @equalsraf, he seems to have a handle on things.

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jun 3, 2014

for _Bool, it appears something like https://bitbucket.org/cffi/cffi/commits/d993a73f4d604bb34b86e92aea06c48545f7a287 will work.

@voltagex We decided in #287 #328 that we would only support MSVC 2013+, which already provides bool for example.

@aktau

This comment has been minimized.

Copy link
Member

aktau commented Jun 11, 2014

By the way, I posted a comment about this on HN. Some answers are intriguing (such as clang-cl, which I had heard about before but didn't work for me at that time, about 6 months ago):

https://news.ycombinator.com/item?id=7874866

@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Jun 11, 2014

clang-cl original announcement: http://lists.cs.uiuc.edu/pipermail/cfe-dev/2013-June/030404.html

Consider the use case of generating a Visual Studio project file that uses clang for diagnostics and or codegen

Looks cool, but what is the advantage if we still need cl.exe and Windows headers?

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Jun 11, 2014

@justinmk The advantage would be that we could build C99 code which we cant do with MSVC (and apparently never will).

@aktau We don't necessarily need clang-cl, hopefully clang can be used directly. Some quick testing on my end using

cmake-gui -DCMAKE_C_COMPILER=clang

seems to work (clang+nmake :D). More importantly (at least for the clang build I'm using) is to target the correct arch

add_definitions(-target i686-windows-msvc)

otherwise I think it assumes some kind of gnu/mingw target and fails to find all the headers.

I didn't test any further, cmake started building but failed with some straightforward errors about uid_t and timeval.

PS: the clang output alone is well worth the effort of installing clang :D

@fwalch

This comment has been minimized.

Copy link
Member

fwalch commented Nov 5, 2014

Oh, okay, that's even better :-)

@ghost

This comment has been minimized.

Copy link

ghost commented Nov 13, 2014

@equalsraf, today VS 2013 Update 4 is released. They have also released a free full featured version called "Community Version": http://www.visualstudio.com/en-us/products/visual-studio-community-vs.

Also VS2015 preview is updated, which has even more C99, C++11, C++14 and even some C++17 features. If you can try it out in a separate VM, you might be able to strip away some guards headers and give some valuable feedback on broken language feature before the VS2015 gets RTM'd. 😄

See this video by Herb Sutter: http://channel9.msdn.com/Events/Visual-Studio/Connect-event-2014/311 and couple of new blogs on http://blogs.msdn.com/vcblog/.

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Nov 13, 2014

@jasonwilliams200OK having a look now, thanks for the heads up.

@aktau

This comment has been minimized.

Copy link
Member

aktau commented Nov 13, 2014

Also VS2015 preview is updated, which has even more C99, C++11, C++14 and even some C++17 features.

Very interesting @jasonwilliams200OK, however I looked at the pages you mentioned and couldn't find the C99 improvements. Do you have another link.

It's a great thing that MS is seemingly turning more open towards *nix, it will make our lives way easier. (see the open-sourcing of .NET earlier).

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Nov 14, 2014

Pushing (or rather re-re-pushing) initial support for third-party in native Windows, it already builds libuv, msgpack, luarocks (but not the dependencies, etc) and luajit.

The big problem now is that there are no targets to install in .deps/usr/(bin,lib) and luarocks is even worse because the directory structure is nothing like in Unix.

Some useful cmake tidbits I picked up along the way

  • elseif() saves a lot in verbosity when writing platform checks
  • ExternalProject can chain commands for you, so instead of CONFIGURE_COMMAND ./autoreconf && ./configure you can do CONFIGURE_COMMAND ./autoreconf COMMAND ./configure
  • It should be ok to use Unix paths in a command provided they are not inside quotes
  • If you want to call "make" but don't know if you are using make/ninja/etc you can use cmake --build <build-dir> instead (PROBLEM: this kills colored output)
@ghost

This comment has been minimized.

Copy link

ghost commented Nov 14, 2014

@aktau, yes the quick release lifecycles are great! .NET, Roslyn etc. are open sourced. Mono has already forked the corefx repository on GitHub. List is available here: https://github.com/microsoft/dotnet.

The union bug seems to be fixed. All three tests reported on connect in reproduction steps (under details) are passing.

Optimizing Compiler Version 18.00.31101 for x86
Incremental Linker Version 12.00.31101.0

Stefan T Lavavej (STL) will hopefully author a blog for complete set of changes. I asked about the C99 related changes in comments here: http://blogs.msdn.com/b/vcblog/archive/2014/11/12/visual-studio-2015-preview-now-available.aspx?PageIndex=2#comments (as (CC) 👈 don't know what I was thinking), and Eric asked me to watch the video. :)

There is the roadmap table in the video (at 35 sec). In VS 2015 RTM, there will be coverage till UNC in literals. That chart is same as what STL updated here: http://blogs.msdn.com/b/vcblog/archive/2014/08/21/c-11-14-features-in-visual-studio-14-ctp3.aspx.

VC++ Conformance Update

Note in the video VS "14" is replaced with VS 2015, which is the official name of next VS.

@ghost

This comment has been minimized.

Copy link

ghost commented Nov 14, 2014

If you want to call "make" but don't know if you are using make/ninja/etc you can use cmake --build instead (PROBLEM: this kills colored output)

@equalsraf, perhaps we can ask @bradking about that problem if it is going to be changed in v3.1.0 (final), since the issue tracker is turned off in the repository: https://github.com/Kitware/CMake.

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Nov 14, 2014

Seems to be intentional, since --build was intended for automated builds i.e. since there is no tty colors are disabled - https://www.mail-archive.com/cmake@cmake.org/msg26011.html

@bradking

This comment has been minimized.

Copy link

bradking commented Nov 14, 2014

Since CMake 3.0 the cmake --build command shares its own pipes with the launched build system. Now color output works.

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Nov 14, 2014

Awesome! --build helps a ton when using ExternalProject.

Maybe we can get rid of the MAKE_PRG bits lying around.

@ghost

This comment has been minimized.

@aktau

This comment has been minimized.

Copy link
Member

aktau commented Nov 18, 2014

@aktau, here is the blog we were looking for: http://blogs.msdn.com/b/vcblog/archive/2014/11/17/c-11-14-17-features-in-vs-2015-preview.aspx.

Cool, I'll put it on the top of my to-read list.

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Dec 2, 2014

So MSVC 2015 definitely fixes the union initializers issue. Current problem is that libuv (or gyp maybe) does not build with MSVC 2015 yet.

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Dec 3, 2014

Updated notes: https://gist.github.com/equalsraf/4685f7aef022a36c26d5

Some upstream todo items for when I get back to a Windows machine:

  1. need to be able to build libuv in MSVC 2015 - I can't get around the union initializer issues anymore (the macros make them too hard to change)
  2. luajit, need a way to set flags when building with msvcbuild.bat (for /MTd)

... or alternatively build all dependencies as DLLs instead of static libs.

PS: whatever you do with your lives never mix static libraries from MSVC 2013 and 2015, preposterous errors will follow.

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Dec 4, 2014

Sigh, building libuv as a shared lib also fails joyent/libuv#1382

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Dec 7, 2014

clang-cl in cmake almost worked for the main Neovim binary but failed for the dependencies.

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Dec 8, 2014

Interesting bug, MSVC generated output using windows style newlines (CRLF), the grammar at msgpack-gen.lua only recognised LF(\n) as the separator - the end result was that the generator only parsed the first function in each header file, i.e. nvim only had a couple of API functions available

I wonder:

  • is there any modern system that only uses CR (Mac OS used to)?
  • How do we test for this kind of error - the binary builds successfully, but the API is just not there?
@aktau

This comment has been minimized.

Copy link
Member

aktau commented Dec 9, 2014

is there any modern system that only uses CR (Mac OS used to)?

Not to my knowledge, it's either LF or CRLF. Mac OS 9 used CR but we specifically don't support that (and have been removing code pertaining to it).

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Dec 12, 2014

It seems the recent changes to ui.c define a new UI_CALL macro

// See http://stackoverflow.com/a/11172679 for a better explanation of how it
// works.
#define UI_CALL(...)                                              \
  do {                                                            \
    for (size_t i = 0; i < ui_count; i++) {                       \
      UI *ui = uis[i];                                            \
      UI_CALL_HELPER(CNT(__VA_ARGS__), __VA_ARGS__);              \
    }                                                             \
  } while (0)
#define CNT(...) SELECT_NTH(__VA_ARGS__, MORE, MORE, MORE, MORE, ZERO, ignore)
#define SELECT_NTH(a1, a2, a3, a4, a5, a6, ...) a6
#define UI_CALL_HELPER(c, ...) UI_CALL_HELPER2(c, __VA_ARGS__)
#define UI_CALL_HELPER2(c, ...) UI_CALL_##c(__VA_ARGS__)
#define UI_CALL_MORE(method, ...) ui->method(ui, __VA_ARGS__)
#define UI_CALL_ZERO(method) ui->method(ui)

e.g. called as

  UI_CALL(resize, width, height);

But in Windows this fails to build, because the macro (for this particular call) expands as follows

do { 
  for (size_t i = 0; i < ui_count; i++) { 
    UI *ui = uis[i];
    ui->resize, width, height(ui);
  } 
} while (0);

but height is not a function. The same call in Linux expands as

do {
  for (size_t i = 0; i < ui_count; i++) { 
    UI *ui = uis[i]; 
    ui->resize(ui, width, height); 
  } 
} while (0);
@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Dec 12, 2014

@tarruda wouldn't this be equivalent?

#define UI_CALL(x, ...)                                              \
  do {                                                            \
    for (size_t i = 0; i < ui_count; i++) {                       \
      UI *ui = uis[i];                                            \
      ui->x(ui, __VA_ARGS__);                                     \
    }                                                             \
  } while (0)
@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Dec 12, 2014

Aaaahhhh MSVC VA_ARGS

The C Standard specifies that at least one argument must be passed to the ellipsis, to ensure that the macro does not resolve to an expression with a trailing comma. The Visual C++ implementation will suppress a trailing comma if no arguments are passed to the ellipsis.

which is why my previous example actually worked (with MSVC). For gcc we can go with ##__VA_ARGS__ instead of __VA_ARGS__. Does clang work with ##__VA_ARGS__?

@aktau

This comment has been minimized.

Copy link
Member

aktau commented Dec 14, 2014

Does clang work with ##VA_ARGS

Yes, I'm pretty sure it does. Clang tries hard to be compatible with gcc (GNU-like) on this front (and for commandline options too).

@equalsraf

This comment has been minimized.

Copy link
Contributor

equalsraf commented Dec 14, 2014

Yes, I'm pretty sure it does. Clang tries hard to be compatible with gcc (GNU-like) on this front (and for commandline options too).

So given the MSVC behaviour, it seems ##VA_ARGS (or similar) is available on Clang, GCC, MSVC. That makes me happy 😄 . The following seems to work with GCC and MSVC (only difference is ##).

#ifdef MSVC
  #define UI_CALL(x, ...)                                              \
    do {                                                            \
      for (size_t i = 0; i < ui_count; i++) {                       \
        UI *ui = uis[i];                                            \
        ui->x(ui, __VA_ARGS__);                                     \
      }                                                             \
    } while (0)
#else
  #define UI_CALL(x, ...)                                              \
    do {                                                            \
      for (size_t i = 0; i < ui_count; i++) {                       \
        UI *ui = uis[i];                                            \
        ui->x(ui, ##__VA_ARGS__);                                     \
      }                                                             \
    } while (0)
#endif

We can also try defining ##__VA_ARGS__ as __VA_ARGS__ for MSVC (cant test ATM).

@justinmk justinmk referenced this issue Dec 28, 2014

Closed

Compiling under Windows (VS2015) #1749

18 of 21 tasks complete
@justinmk

This comment has been minimized.

Copy link
Member

justinmk commented Dec 28, 2014

This thread is getting too long to digest, I suggest adding new comments to #1749. When I finish collecting the relevant parts of this issue, I would like to close this in favor of #1749.

@ghost

This comment has been minimized.

Copy link

ghost commented Apr 14, 2015

#1749 has been open for a while, so I'll close this. My apologies if I missed something.

@ghost ghost closed this Apr 14, 2015

@neovim neovim locked and limited conversation to collaborators Aug 12, 2015

This issue was closed.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.