Can't install via homebrew-neovim if luarocks path is set explicitly

[telemachus]

I believe that this is related to the following two issues, but I wanted to post it here as well, in case other people are seeing the same problems.

Related issues:

• Building neovim fails (lua module mpack not found) neovim#4573
• Build failures on OSX libmpack/libmpack#2

Failied build output:

$ brew install --HEAD neovim
==> Installing neovim from neovim/neovim
==> Cloning https://github.com/neovim/neovim.git
Cloning into '/Library/Caches/Homebrew/neovim--git'...
remote: Counting objects: 2041, done.
remote: Compressing objects: 100% (1897/1897), done.
remote: Total 2041 (delta 159), reused 1373 (delta 127), pack-reused 0
Receiving objects: 100% (2041/2041), 7.30 MiB | 4.41 MiB/s, done.
Resolving deltas: 100% (159/159), done.
Checking connectivity... done.
==> Checking out branch master
==> Downloading https://github.com/libuv/libuv/archive/v1.8.0.tar.gz
==> Downloading from https://codeload.github.com/libuv/libuv/tar.gz/v1.8.0
######################################################################## 100.0%
==> Downloading https://github.com/msgpack/msgpack-c/archive/cpp-1.0.0.tar.gz
==> Downloading from https://codeload.github.com/msgpack/msgpack-c/tar.gz/cpp-1.0.0
######################################################################## 100.0%
==> Downloading https://raw.githubusercontent.com/neovim/deps/master/opt/LuaJIT-2.0.4
######################################################################## 100.0%
==> Downloading https://github.com/keplerproject/luarocks/archive/5d8a16526573b36d5b2
==> Downloading from https://codeload.github.com/keplerproject/luarocks/tar.gz/5d8a16
######################################################################## 100.0%
==> Downloading https://github.com/mauke/unibilium/archive/v1.2.0.tar.gz
==> Downloading from https://codeload.github.com/mauke/unibilium/tar.gz/v1.2.0
######################################################################## 100.0%
==> Downloading http://www.leonerd.org.uk/code/libtermkey/libtermkey-0.18.tar.gz
######################################################################## 100.0%
==> Downloading https://github.com/neovim/libvterm/archive/1b745d29d45623aa8d22a7b928
==> Downloading from https://codeload.github.com/neovim/libvterm/tar.gz/1b745d29d4562
######################################################################## 100.0%
==> Downloading https://github.com/jemalloc/jemalloc/releases/download/4.0.2/jemalloc
==> Downloading from https://github-cloud.s3.amazonaws.com/releases/13310527/17afd0b4
######################################################################## 100.0%
==> Building third-party dependencies.
==> cmake ../third-party -DUSE_BUNDLED_BUSTED=OFF -DUSE_BUNDLED_LUV=OFF -DUSE_EXISTIN
==> make VERBOSE=1
==> Building Neovim.
==> cmake .. -DCMAKE_C_FLAGS_RELEASE=-DNDEBUG -DCMAKE_CXX_FLAGS_RELEASE=-DNDEBUG -DCM
Last 15 lines from /Users/telemachus/Library/Logs/Homebrew/neovim/03.cmake:
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE  
-- Checking Lua interpreter /tmp/neovim20160417-5070-l9l8zn/deps-build/usr/bin/luajit
-- [/tmp/neovim20160417-5070-l9l8zn/deps-build/usr/bin/luajit] The 'lpeg' lua package is required for building Neovim
-- Checking Lua interpreter /usr/local/bin/lua
-- [/usr/local/bin/lua] The 'mpack' lua package is required for building Neovim
CMake Error at CMakeLists.txt:390 (message):
  A suitable Lua interpreter was not found.


-- Configuring incomplete, errors occurred!
See also "/tmp/neovim20160417-5070-l9l8zn/build/CMakeFiles/CMakeOutput.log".
See also "/tmp/neovim20160417-5070-l9l8zn/build/CMakeFiles/CMakeError.log".

READ THIS: https://git.io/brew-troubleshooting
If reporting this issue please do so at (not Homebrew/brew):
  https://github.com/neovim/homebrew-neovim/issues

These open issues may also help:
Error when trying to install neovim in my Ubuntu 14.04 virtual machine. https://github.com/neovim/homebrew-neovim/issues/134

[justinmk]

#147 (comment)

[telemachus]

@justinmk I've tried a full reinstall (including clearing the cache completely), and it does not help.

mpack currently doesn't install on OSX for me, and I suspect that's the underlying problem.

[justinmk]

Left a comment at libmpack/libmpack#2. I think we require one of luajit or lua 5.1 for now.

[telemachus]

luajit could still be trouble since if I'm reading the build output from above correctly, when it attempts to use luajit, that fails because of lpeg. (But maybe I'm misunderstanding what's happening at that point in the build.)

-- Checking Lua interpreter /tmp/neovim20160417-5070-l9l8zn/deps-build/usr/bin/luajit
-- [/tmp/neovim20160417-5070-l9l8zn/deps-build/usr/bin/luajit] The 'lpeg' lua package is required for building Neovim

[telemachus]

@justinmk Per your request in Gitter, here's a verbose log: https://gist.github.com/telemachus/fe078934942bdcc2a0e86319de5e1148

[telemachus]

After uninstalling everything related to Lua, I was finally able to build neovim. I'm still not sure what happened to my environment, but I'll close this for now.

[justinmk]

@telemachus Thanks for followup.

[telemachus]

@justinmk et al. I've just reopened this, because after some more testing I'm very confused.

Here's where things stand right now: I can install neovim, but only if I uninstall everything else related to Lua first.

I don't think that should be required, and obviously, the Travis CI testing won't catch it, since (I assume) each Travis build starts with a fresh clean image without a previous Lua install.

So, in a nutshell, my question is this: Is it expected that neovim should fail if you already have Lua installed on OSX? If not, what can I test in order to see what's wrong with my setup?

Thanks.

[jszakmeister]

So, in a nutshell, my question is this: Is it expected that neovim should fail if you already have Lua installed on OSX? If not, what can I test in order to see what's wrong with my setup?

Jumping in here real quick. I would say no, it's not expected. In fact, we got through some trouble to make sure that our deps are picked up before system ones.

Looking at your verbose log, it appears that something went wrong when checking for the necessary modules or the lpeg module was not built correctly for some reason:

-- Checking Lua interpreter /tmp/neovim-20160419-32334-17yn1fw/deps-build/usr/bin/luajit
-- [/tmp/neovim-20160419-32334-17yn1fw/deps-build/usr/bin/luajit] The 'lpeg' lua package is required for building Neovim
-- Checking Lua interpreter /usr/local/bin/lua
-- [/usr/local/bin/lua] The 'mpack' lua package is required for building Neovim
CMake Error at CMakeLists.txt:390 (message):
  A suitable Lua interpreter was not found.


-- Configuring incomplete, errors occurred!
See also "/tmp/neovim-20160419-32334-17yn1fw/build/CMakeFiles/CMakeOutput.log".
See also "/tmp/neovim-20160419-32334-17yn1fw/build/CMakeFiles/CMakeError.log".

It tried to use our toolchain first, but something isn't right with the lpeg package. And it looks like it built:

/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/lpeg.dir/build.make CMakeFiles/lpeg.dir/depend
cd /tmp/neovim-20160419-32334-17yn1fw/deps-build && /usr/local/Cellar/cmake/3.5.2/bin/cmake -E cmake_depends "Unix Makefiles" /tmp/neovim-20160419-32334-17yn1fw/third-party /tmp/neovim-20160419-32334-17yn1fw/third-party /tmp/neovim-20160419-32334-17yn1fw/deps-build /tmp/neovim-20160419-32334-17yn1fw/deps-build /tmp/neovim-20160419-32334-17yn1fw/deps-build/CMakeFiles/lpeg.dir/DependInfo.cmake --color=
Dependee "/tmp/neovim-20160419-32334-17yn1fw/deps-build/CMakeFiles/lpeg.dir/DependInfo.cmake" is newer than depender "/tmp/neovim-20160419-32334-17yn1fw/deps-build/CMakeFiles/lpeg.dir/depend.internal".
Dependee "/tmp/neovim-20160419-32334-17yn1fw/deps-build/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/tmp/neovim-20160419-32334-17yn1fw/deps-build/CMakeFiles/lpeg.dir/depend.internal".
Scanning dependencies of target lpeg
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/lpeg.dir/build.make CMakeFiles/lpeg.dir/build
[ 51%] Generating usr/lib/luarocks/rocks/lpeg
usr/bin/luarocks build lpeg CC=/usr/local/Library/ENV/4.3/clang LD=/usr/local/Library/ENV/4.3/clang
/usr/local/Library/ENV/4.3/clang -O2 -fPIC -I/tmp/neovim-20160419-32334-17yn1fw/deps-build/usr/include/luajit-2.0 -c lpcap.c -o lpcap.o
/usr/local/Library/ENV/4.3/clang -O2 -fPIC -I/tmp/neovim-20160419-32334-17yn1fw/deps-build/usr/include/luajit-2.0 -c lpcode.c -o lpcode.o
/usr/local/Library/ENV/4.3/clang -O2 -fPIC -I/tmp/neovim-20160419-32334-17yn1fw/deps-build/usr/include/luajit-2.0 -c lpprint.c -o lpprint.o
/usr/local/Library/ENV/4.3/clang -O2 -fPIC -I/tmp/neovim-20160419-32334-17yn1fw/deps-build/usr/include/luajit-2.0 -c lptree.c -o lptree.o
/usr/local/Library/ENV/4.3/clang -O2 -fPIC -I/tmp/neovim-20160419-32334-17yn1fw/deps-build/usr/include/luajit-2.0 -c lpvm.c -o lpvm.o
/usr/local/Library/ENV/4.3/clang -bundle -undefined dynamic_lookup -all_load -o lpeg.so -L/tmp/neovim-20160419-32334-17yn1fw/deps-build/usr/lib lpcap.o lpcode.o lpprint.o lptree.o lpvm.o
Updating manifest for /tmp/neovim-20160419-32334-17yn1fw/deps-build/usr/lib/luarocks/rocks
lpeg 1.0.0-1 is now built and installed in /tmp/neovim-20160419-32334-17yn1fw/deps-build/usr (license: MIT/X11)

So maybe something is wrong with our detection scheme or maybe lpeg is not being loaded correctly with the new toolchain. It could be that a dependency needs to be updated, or there is something that needs to be fixed upstream. I would start there first.

[telemachus]

So maybe something is wrong with our detection scheme or maybe lpeg is not being loaded correctly with the new toolchain. It could be that a dependency needs to be updated, or there is something that needs to be fixed upstream. I would start there first.

@jszakmeister To clarify: you're suggesting that I post this directly to neovim rather than here? That's fine if so, I just want to be sure. (My obvious worry being that they will simply say "Oh that must be related to hombrew or OSX...")

Thanks.

[jszakmeister]

@jszakmeister To clarify: you're suggesting that I post this directly to neovim rather than here? That's fine if so, I just want to be sure. (My obvious worry being that they will simply say "Oh that must be related to hombrew or OSX...")

Not necessarily. I don't think we can say either way yet. Homebrew alters the build environment pretty substantially, and it could be something Homebrew is doing too. What I'm suggesting is that you dig in a little and see if you can figure a little more out about what is failing in the lpeg detection. This might mean building Neovim yourself so you can capture a bit more about what is going on.

If it's too much to ask, then hopefully someone else can come along and help figure it out. Bandwidth is limited though, so any help you can provide is useful.

[jszakmeister]

FWIW, I just updated, installed Lua via Homebrew, and then installed Neovim via Homebrew with 0 issues. It detected my interpreter just fine:

-- Checking Lua interpreter /tmp/neovim-20160422-4395-r1gnqz/neovim-0.1.3/deps-build/usr/bin/luajit
-- Using the Lua interpreter /tmp/neovim-20160422-4395-r1gnqz/neovim-0.1.3/deps-build/usr/bin/luajit.

[telemachus]

@jszakmeister Ah, ok, I'm fine trying whatever, but to be honest I'm not sure how I would dig in myself. But I can start with what you suggest: building neovim itself (outside of homebrew) with lua installed via homebrew. Now that I think of it, I can also check what happens if I have lua installed but not lpeg. I can say right away that I tried manually building neovim over last weekend (when this problem began for me), and the build failed then too. So I suspect it's probably also an upstream problem. If I can confirm that kind of failure, I'll report directly to neovim and cc this issue.

For the moment, though, I'm going to leave this open since other people may find it helpful if they see failed builds with lua (and lpeg) installed.

Re: your update, maybe try uninstalling neovim, installing lpeg, and then reinstall neovim again? EDIT: Nope, it's worse than that. I have lua and only lpeg, and neovim will still build. It must be some larger combination of the rocks I had installed. I'll keep experimenting today and over the weekend and update this if I figure out anything.

[jszakmeister]

Ah, ok, I'm fine trying whatever, but to be honest I'm not sure how I would dig in myself. But I can start with what you suggest: building neovim itself (outside of homebrew) with lua. Now that I think of it, I can also check the variant of what happens �if I have lua installed, but not lpeg. I can say right away that I did try this earlier in this cycle, and the build failed then too. So I suspect it's probably also an upstream problem. If I can confirm that kind of failure, I'll report directly to neovim and cc this issue.

I wish I had more advice to give here. Often I start in the same boat and just keep scratching at the problem until flakes start coming off. :-)

Re: your update, maybe try uninstalling neovim, installing lpeg, and then reinstall neovim again?

I don't think it will help. It finds the deps-provided Lua first, and it satisfies everything, so it doesn't even look at the other Lua I installed. (time passes) So I tried it anyways, and I saw no difference. I picks up our Lua first as I suspected.

[telemachus]

I don't think it will help. It finds the deps-provided Lua first, and it satisfies everything, so it doesn't even look at the other Lua I installed. (time passes) So I tried it anyways, and I saw no difference. I picks up our Lua first as I suspected.

@jszakmeister I appreciate your taking the time. This is very tedious and no fun. I'm now in the process of going through the rocks I normally have installed one by one, trying to figure out what gums up the works—though I can't for the life of me see how a rock could break a separate build.

[telemachus]

@justinmk @jszakmeister Ok, I solved my problem. (I'm changing the title again, hopefully for the last time, to focus on the problem.)

The issue was this in my bash profile:

command -v luarocks >/dev/null && eval "$(luarocks path)"

That was set, per advice of luarocks (https://github.com/keplerproject/luarocks/wiki/Using-LuaRocks#Rocks_trees_and_the_Lua_libraries_path), because I had rocks installed outside of the default location at one point.

I'm not going to immediately close this though, because maybe this is something that homebrew's and/or neovim's build process should handle. That is, as part of clearing away a user's personal environment, maybe one (or both) should also handle the luarocks path. If so, I'm happy to post this as an issue for neovim as well. If people disagree and think it was just a bad setting on my part, I'll close this or one of you can. (Also, for what it's worth, I've had this setting for as long as I've been using neovim and installing it with brew, but I only began to have this problem last Friday. So something relevant may have changed recently.)

In any case, thanks for your patience and all your help.

[jszakmeister]

I'm not going to immediately close this though, because maybe this is something that homebrew's and/or neovim's build process should handle. That is, as part of clearing away a user's personal environment, maybe one (or both) should also handle the luarocks path.

I struggle with the concept of the build system clearing away the personal environment. One reason is because there's no easy way for us to do it, but another is because I'm not sure it's always the right thing to do. Distros pass quite a lot through the environment building packages, and we'd potentially get in the way of that. The downside is that things can misbehave.

In short, I'm not sure how we want to handle this, or if we even do want to handle it. Perhaps if the lua interpreter that we pick is our deps version, maybe we could clear the associated environment variable?

(Also, for what it's worth, I've had this setting for as long as I've been using neovim and installing it with brew, but I only began to have this problem last Friday. So something relevant may have changed recently.)

I'm really not sure what would have changed here that caused it to be picked up all of a sudden. :-(

In any case, thanks for your patience and all your help.

You're welcome! And likewise: thanks for sticking to it and figuring it out. I'm sure it will help someone down the road at the very least.

[justinmk]

@telemachus Maybe try env=std (from #114 (comment) )

brew install neovim --env=std

[owais]

I'm facing the same issue. --env=std does not help. Installing manually by checking our neovim repo and building works but with homebrew does not.

~ $ brew install neovim
==> Installing neovim from neovim/neovim
==> Downloading https://github.com/neovim/neovim/archive/v0.1.7.tar.gz
Already downloaded: /Users/owais/Library/Caches/Homebrew/neovim-0.1.7.tar.gz
==> Downloading https://github.com/luvit/luv/archive/146f1ce4c08c3b67f604c9ee1e124b1cf5c15cf3.tar.gz
Already downloaded: /Users/owais/Library/Caches/Homebrew/neovim--luv-3.tar.gz
==> Downloading https://raw.githubusercontent.com/neovim/deps/master/opt/LuaJIT-2.0.4.tar.gz
Already downloaded: /Users/owais/Library/Caches/Homebrew/neovim--luajit-2.0.4.tar.gz
==> Downloading https://github.com/keplerproject/luarocks/archive/5d8a16526573b36d5b22aa74866120c998466697.tar.gz
Already downloaded: /Users/owais/Library/Caches/Homebrew/neovim--luarocks-998466697.tar.gz
==> Building third-party dependencies.
==> cmake ../third-party -DUSE_BUNDLED_BUSTED=OFF -DUSE_BUNDLED_GPERF=OFF -DUSE_BUNDLED_LIBUV=OFF -DUSE_BUNDLED_MSGPACK=OFF -DUSE_BUNDLED_UNIBILIUM=OFF -
==> make VERBOSE=1
Last 15 lines from /Users/owais/Library/Logs/Homebrew/neovim/02.make:
make[3]: *** [all] Error 2
make[2]: *** [build/src/luv-static-stamp/luv-static-build] Error 2
make[1]: *** [CMakeFiles/luv-static.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
No existing manifest. Attempting to rebuild...
Updating manifest for /tmp/neovim-20170103-37840-162fxtv/neovim-0.1.7/deps-build/usr/lib/luarocks/rocks
luarocks scm-1 is now built and installed in /tmp/neovim-20170103-37840-162fxtv/neovim-0.1.7/deps-build/usr (license: MIT/X11)

cd /tmp/neovim-20170103-37840-162fxtv/neovim-0.1.7/deps-build/build/src/luarocks && /usr/local/Cellar/cmake/3.7.1/bin/cmake -E touch /tmp/neovim-20170103-37840-162fxtv/neovim-0.1.7/deps-build/build/src/luarocks-stamp/luarocks-install
[ 81%] Completed 'luarocks'
/usr/local/Cellar/cmake/3.7.1/bin/cmake -E make_directory /tmp/neovim-20170103-37840-162fxtv/neovim-0.1.7/deps-build/CMakeFiles
/usr/local/Cellar/cmake/3.7.1/bin/cmake -E touch /tmp/neovim-20170103-37840-162fxtv/neovim-0.1.7/deps-build/CMakeFiles/luarocks-complete
/usr/local/Cellar/cmake/3.7.1/bin/cmake -E touch /tmp/neovim-20170103-37840-162fxtv/neovim-0.1.7/deps-build/build/src/luarocks-stamp/luarocks-done
[ 81%] Built target luarocks
make: *** [all] Error 2

If reporting this issue please do so at (not Homebrew/brew or Homebrew/core):
https://github.com/neovim/homebrew-neovim/issues

These open issues may also help:
Can't install via homebrew-neovim if luarocks path is set explicitly https://github.com/neovim/homebrew-neovim/issues/149
~ $

[owais]

Turning up verbosity revealed that luv build was failing because of missing uv.h header file. Re-installing libuv fixed the problem.

brew uninstall libuv
brew install libuv

[RichardJohnn]

for what it's worth i unset LUA_CPATH LUA_PATH before brew installing neovim

[amckinlay]

I had to unset LUA_CPATH LUA_PATH, too. --env=std did not work.

[telemachus]

@RichardJohnn @amckinlay That was my experience too. --env=std does not work, but not setting custom LUA_ paths does.

[justinmk]

Perhaps if the lua interpreter that we pick is our deps version, maybe we could clear the associated environment variable?

Certainly we should do that in the brew formula at least. Even in core cmake it should be fine to do that , just print some info message mentioning it.