install native and cross lua executables & add LuaRocks plugin #1017

starius · 2015-11-29T21:44:43Z

No description provided.

tonytheodore · 2015-11-30T02:34:38Z

plugins/luarocks/luarocks.mk

+        --with-lua-bin='$(PREFIX)/bin' \
+        --with-downloader='wget' \
+        --with-md5-checker='openssl'
+    $(MAKE) -C '$(1)' build install \


install doesn't depend on build so (on OSX) the symlink is created first and doesn't have execute permissions. These steps should be separated.

tonytheodore · 2015-11-30T03:01:16Z

Looks good!

Plugins currently require some boilerplate:

$(PKG)_MAKEFILE := $(lastword $(MAKEFILE_LIST))

for updates to work, I might see if there's some sort of include hook so it isn't necessary.

Additional metadata like:

$(PKG)_WEBSITE  := http://www.gnu.org/software/automake
$(PKG)_OWNER    := https://github.com/starius

may also be useful, but the REAMDE is an excellent approach.

tonytheodore · 2015-11-30T03:13:08Z

plugins/luarocks/lpeg.mk

+    $(WGET) -q -O- 'http://www.inf.puc-rio.br/~roberto/lpeg/' | \
+    $(SED) -n 's,.*lpeg-\([0-9][^>]*\)\.tar.*,\1,p' | \
+    sort -h | tail -1
+endef


tonytheodore · 2015-11-30T05:58:05Z

Plugins currently require some boilerplate:

See 2944ccd, $(PKG)_MAKEFILE is no longer needed.

How to build all packages from a particular plugin? from #935 (comment)

There's also now an automatic all-<plugin> target which is the last segment of the plugin path. all-rocks.mk can be removed and make all-luarocks MXE_PLUGIN_DIRS=plugins/luarocks ... will work.

> install doesn't depend on build so (on OSX) the symlink is > created first and doesn't have execute permissions. These > steps should be separated. See mxe#1017

See mxe#1017

... instead of $(PREFIX)/bin, not to clobber the user's normal tools. See mxe#1017

> There's also now an automatic all-<plugin> target which is the > last segment of the plugin path. all-rocks.mk can be removed > and make all-luarocks MXE_PLUGIN_DIRS=plugins/luarocks ... > will work. See mxe#1017

... instead of $(PREFIX)/bin, not to clobber the user's normal tools. See mxe#1017

> There's also now an automatic all-<plugin> target which is the > last segment of the plugin path. all-rocks.mk can be removed > and make all-luarocks MXE_PLUGIN_DIRS=plugins/luarocks ... > will work. See mxe#1017

starius · 2015-12-01T22:21:04Z

I have taken into account all the comments.

Build on Wheezy produces broken lua.exe / lua53.dll. It can't find libgcc_s_sjlj-1.dll.

$ make all-luarocks MXE_PLUGIN_DIRS=plugins/luarocks MXE_TARGETS='i686-w64-mingw32.shared x86_64-w64-mingw32.shared'

$ ./usr/bin/i686-w64-mingw32.shared-lua 
err:module:import_dll Library libgcc_s_sjlj-1.dll (which is needed by L"Z:\\home\\mxe\\mxe-luarocks-review\\usr\\i686-w64-mingw32.shared\\bin\\lua53.dll") not found
err:module:import_dll Library lua53.dll (which is needed by L"Z:\\home\\mxe\\mxe-luarocks-review\\usr\\i686-w64-mingw32.shared\\bin\\lua.exe") not found
err:module:LdrInitializeThunk Main exe initialization for L"Z:\\home\\mxe\\mxe-luarocks-review\\usr\\i686-w64-mingw32.shared\\bin\\lua.exe" failed, status c0000135

tonytheodore · 2015-12-02T01:59:56Z

Build on Wheezy produces broken lua.exe / lua53.dll. It can't find libgcc_s_sjlj-1.dll

Is it only Wheezy and/or lua.exe? The gcc *.dlls are in a version-specific subdir in case we were going to do side-by-side gcc versions (which now seems unlikely).

Does moving the *.dlls the the parent dir work?

starius · 2015-12-02T09:59:15Z

It doesn't depend on Wheezy/Jessie.

Clean installation:

$ make all-luarocks MXE_PLUGIN_DIRS=plugins/luarocks MXE_TARGETS='i686-w64-mingw32.shared x86_64-w64-mingw32.shared'
$ find usr/ -name libgcc_s_sjlj-1.dll
usr/i686-w64-mingw32.shared/bin/gcc-4.9.3/libgcc_s_sjlj-1.dll

Old installation (gcc rebuilt):

$ find usr/ -name libgcc_s_sjlj-1.dll
usr/i686-w64-mingw32.shared/bin/libgcc_s_sjlj-1.dll
usr/i686-w64-mingw32.shared/bin/gcc-4.9.3/libgcc_s_sjlj-1.dll

tonytheodore · 2015-12-02T10:26:06Z

I'm not sure what just happened. I tried creating a pull request using the web interface and it created the branch but seems to have merged it without a pull request ae8f6de.

tonytheodore · 2015-12-02T11:41:30Z

Reverted now 38a9ead, I'll do a proper fix tomorrow.

starius · 2015-12-02T20:46:18Z

For the record, the problem seems to be introduced by 9020e9e, #965.

tonytheodore · 2015-12-03T08:04:59Z

See #1032

tonytheodore · 2015-12-03T09:03:11Z

llthreads2, lpeg, and lua-rapidjson depend on luasocket:

$ make all-luarocks MXE_PLUGIN_DIRS=plugins/luarocks -j4
[build]     lua                    i686-w64-mingw32.shared
[done]      lua                    i686-w64-mingw32.shared                                2380 KiB       0m6.850s
[build]     luarocks               i686-w64-mingw32.shared
[done]      luarocks               i686-w64-mingw32.shared                                732 KiB        0m6.548s
[build]     llthreads2             i686-w64-mingw32.shared
[build]     lpeg                   i686-w64-mingw32.shared
[build]     lua-rapidjson          i686-w64-mingw32.shared
[build]     luasocket              i686-w64-mingw32.shared

Failed to build package llthreads2 for target i686-w64-mingw32.shared!
------------------------------------------------------------
/Users/tonyt/dev/mxe/usr/bin/i686-w64-mingw32.shared-gcc -shared -o llthreads2.dll -L/Users/tonyt/dev/mxe/usr/i686-w64-mingw32.shared/lib src/l52util.obj src/llthread.obj -lkernel32 -llua
No existing manifest. Attempting to rebuild...
Updating manifest for /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.shared/lib/luarocks/rocks
Error: rock_manifest file not found for luasocket scm-0 - not a LuaRocks 2 tree?
make[1]: *** [build-only-llthreads2_i686-w64-mingw32.shared] Error 1
real    0m6.847s
user    0m6.154s
sys 0m0.606s
------------------------------------------------------------
[log]      /Users/tonyt/dev/mxe/log/llthreads2_i686-w64-mingw32.shared

make: *** [/Users/tonyt/dev/mxe/usr/i686-w64-mingw32.shared/installed/llthreads2] Error 1
make: *** Waiting for unfinished jobs....

Failed to build package lpeg for target i686-w64-mingw32.shared!
------------------------------------------------------------
/Users/tonyt/dev/mxe/usr/bin/i686-w64-mingw32.shared-gcc -shared -o lpeg.dll -L/Users/tonyt/dev/mxe/usr/i686-w64-mingw32.shared/lib lpcap.obj lpcode.obj lpprint.obj lptree.obj lpvm.obj -llua
No existing manifest. Attempting to rebuild...
Updating manifest for /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.shared/lib/luarocks/rocks
Error: rock_manifest file not found for luasocket scm-0 - not a LuaRocks 2 tree?
make[1]: *** [build-only-lpeg_i686-w64-mingw32.shared] Error 1
real    0m7.079s
user    0m6.417s
sys 0m0.575s
------------------------------------------------------------
[log]      /Users/tonyt/dev/mxe/log/lpeg_i686-w64-mingw32.shared

make: *** [/Users/tonyt/dev/mxe/usr/i686-w64-mingw32.shared/installed/lpeg] Error 1

Failed to build package lua-rapidjson for target i686-w64-mingw32.shared!
------------------------------------------------------------
-- Installing: /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.shared/lib/luarocks/rocks/rapidjson/0.2.2-1/lib/rapidjson.dll
No existing manifest. Attempting to rebuild...
Updating manifest for /Users/tonyt/dev/mxe/usr/i686-w64-mingw32.shared/lib/luarocks/rocks
Error: rock_manifest file not found for luasocket scm-0 - not a LuaRocks 2 tree?
make[1]: *** [build-only-lua-rapidjson_i686-w64-mingw32.shared] Error 1
real    0m11.512s
user    0m10.398s
sys 0m0.916s
------------------------------------------------------------
[log]      /Users/tonyt/dev/mxe/log/lua-rapidjson_i686-w64-mingw32.shared

make: *** [/Users/tonyt/dev/mxe/usr/i686-w64-mingw32.shared/installed/lua-rapidjson] Error 1
[done]      luasocket              i686-w64-mingw32.shared                                2340 KiB       0m12.423s

Easy fix tonytheodore@297f697

starius · 2015-12-03T22:51:52Z

llthreads2, lpeg, and lua-rapidjson do not depend on luasocket. None of them needs network. Can you show logs for llthreads2, lpeg, and lua-rapidjson, please? Probably the build was failed because of race condition (-j4). Luarocks updates information about rocks installed and it seems to do it without synchronization between multiple processes. If it is true, some kind of mutex should be introduced to prevent this.

Another possible reason of printing luasocket in error logs of other rocks is that the build from the local rockspec failed for some reason and luarocks tried to download rocks from luarocks.org. Luarocks uses wget or luasocket for downloading. This behaviour should be suppressed in MXE.

tonytheodore · 2015-12-04T03:11:49Z

Can you show logs for llthreads2, lpeg, and lua-rapidjson, please?

Only llthreads2 and lpeg this time: https://gist.github.com/tonytheodore/7765744f895cdad87035

Luarocks uses wget or luasocket for downloading. This behaviour should be suppressed in MXE.

The PRELOAD trick is working on OSX now, disabling it always succeeds:

make all-luarocks MXE_PLUGIN_DIRS=plugins/luarocks -j4 PRELOAD=

tonytheodore · 2015-12-04T03:42:40Z

Luarocks updates information about rocks installed and it seems to do it without synchronization between multiple processes. If it is true, some kind of mutex should be introduced to prevent this.

A workaround is to put a .NOTPARALLEL target in one of the files:

diff --git a/plugins/luarocks/luarocks.mk b/plugins/luarocks/luarocks.mk
index b7cbbae..51209bb 100644
--- a/plugins/luarocks/luarocks.mk
+++ b/plugins/luarocks/luarocks.mk
@@ -18,6 +18,10 @@ define $(PKG)_UPDATE
     $(SORT) -h | tail -1
 endef

+# Luarocks updates information about rocks installed and it seems
+# to do it without synchronization between multiple processes.
+.NOTPARALLEL:
+
 # shared-only because Lua loads modules in runtime

 define $(PKG)_BUILD_SHARED

See mxe#1017 (comment) See http://wiki.bash-hackers.org/howto/mutex

starius · 2015-12-05T21:48:26Z

I wrapped luarocks main tool with bash script which provides a mutex based on mkdir.
See http://wiki.bash-hackers.org/howto/mutex for the explanation
Please test if it works for you.

tonytheodore · 2015-12-06T00:54:51Z

Thanks, the lock dir needs to exist:

diff --git a/plugins/luarocks/luarocks.mk b/plugins/luarocks/luarocks.mk
index 10131ee..a852c37 100644
--- a/plugins/luarocks/luarocks.mk
+++ b/plugins/luarocks/luarocks.mk
@@ -39,6 +39,7 @@ define $(PKG)_BUILD_SHARED

     # create bash wrapper adding inter-process mutex
     # see https://github.com/mxe/mxe/pull/1017#issuecomment-161557440
+    $(INSTALL) -d '$(PREFIX)/$(TARGET)/lib/luarocks'
     (echo '#!/usr/bin/env bash'; \
      echo 'echo "== Using MXE wrapper: $(PREFIX)/$(TARGET)/bin/luarocks"'; \
      echo '# Creating a directory is an atomic operation, that is why'; \

and it all works well!

Source: mxe#1017 (comment)

starius · 2015-12-06T01:27:21Z

Fixed.

tonytheodore · 2015-12-06T01:34:26Z

Do you want to do any rebasing? Otherwise I think this is ready to merge.

LuaRocks plugin adds * package "luarocks" * several packages with rocks (lpeg, luasocket, llthreads2, lua-rapidjson) * plugins/luarocks/README.md * virtual rock "all-rocks" depending on all rocks * short test for rocks

> install doesn't depend on build so (on OSX) the symlink is > created first and doesn't have execute permissions. These > steps should be separated. See mxe#1017

See mxe#1017

... instead of $(PREFIX)/bin, not to clobber the user's normal tools. See mxe#1017

> There's also now an automatic all-<plugin> target which is the > last segment of the plugin path. all-rocks.mk can be removed > and make all-luarocks MXE_PLUGIN_DIRS=plugins/luarocks ... > will work. See mxe#1017

See mxe#1017 (comment) See http://wiki.bash-hackers.org/howto/mutex

Source: mxe#1017 (comment)

starius · 2015-12-06T02:39:31Z

Rebased

install native and cross lua executables & add LuaRocks plugin

tonytheodore · 2015-12-06T02:57:23Z

Well done!

starius mentioned this pull request Nov 29, 2015

MXE Plugins #935

Merged

tonytheodore reviewed Nov 30, 2015
View reviewed changes

starius force-pushed the luarocks2 branch from d30ed1e to db97ce1 Compare December 1, 2015 21:24

starius added a commit to LuaAndC/mxe that referenced this pull request Dec 1, 2015

luarocks: separate build and install

1528f29

> install doesn't depend on build so (on OSX) the symlink is > created first and doesn't have execute permissions. These > steps should be separated. See mxe#1017

starius added a commit to LuaAndC/mxe that referenced this pull request Dec 1, 2015

move lua_SHORTVER from luarocks.mk to lua.mk

a4cd6b5

See mxe#1017

starius added a commit to LuaAndC/mxe that referenced this pull request Dec 1, 2015

install lua symlinks to $(PREFIX)/$(BUILD)/bin

faa9d4f

... instead of $(PREFIX)/bin, not to clobber the user's normal tools. See mxe#1017

starius force-pushed the luarocks2 branch from db97ce1 to 4831e96 Compare December 1, 2015 21:48

starius added a commit to LuaAndC/mxe that referenced this pull request Dec 1, 2015

install lua symlinks to $(PREFIX)/$(BUILD)/bin

647b59e

... instead of $(PREFIX)/bin, not to clobber the user's normal tools. See mxe#1017

tonytheodore mentioned this pull request Dec 3, 2015

gcc: don't install *.dlls to version specific directory #1032

Merged

starius added a commit to LuaAndC/mxe that referenced this pull request Dec 5, 2015

luarocks wrapper prevents race condition

682153f

See mxe#1017 (comment) See http://wiki.bash-hackers.org/howto/mutex

starius pushed a commit to LuaAndC/mxe that referenced this pull request Dec 6, 2015

luarocks: create the lock dir's parent

f1064fe

Source: mxe#1017 (comment)

starius and others added 12 commits December 6, 2015 05:38

lua: install cross-compiled binary to <target>/bin

580be67

lua: enable native target

08488f9

add plugin luarocks

43cb319

LuaRocks plugin adds * package "luarocks" * several packages with rocks (lpeg, luasocket, llthreads2, lua-rapidjson) * plugins/luarocks/README.md * virtual rock "all-rocks" depending on all rocks * short test for rocks

luarocks: separate build and install

c788548

> install doesn't depend on build so (on OSX) the symlink is > created first and doesn't have execute permissions. These > steps should be separated. See mxe#1017

move lua_SHORTVER from luarocks.mk to lua.mk

5c58492

See mxe#1017

install lua symlinks to $(PREFIX)/$(BUILD)/bin

a79bd18

... instead of $(PREFIX)/bin, not to clobber the user's normal tools. See mxe#1017

add _WEBSITE and _OWNER to luarocks/*.mk

f37f78b

lpeg.mk, luarocks.mk: s/sort/$(SORT)/

f89bc08

switch to new argument for MXE_GET_GITHUB_TAGS

e8d8052

remove virtual package all-rocks

ab3635c

> There's also now an automatic all-<plugin> target which is the > last segment of the plugin path. all-rocks.mk can be removed > and make all-luarocks MXE_PLUGIN_DIRS=plugins/luarocks ... > will work. See mxe#1017

luarocks wrapper prevents race condition

51a6141

See mxe#1017 (comment) See http://wiki.bash-hackers.org/howto/mutex

luarocks: create the lock dir's parent

e6f1f26

Source: mxe#1017 (comment)

starius force-pushed the luarocks2 branch from f1064fe to e6f1f26 Compare December 6, 2015 02:38

starius added a commit that referenced this pull request Dec 6, 2015

Merge pull request #1017 from LuaAndC/luarocks2

99af3f7

install native and cross lua executables & add LuaRocks plugin

starius merged commit 99af3f7 into mxe:master Dec 6, 2015

starius deleted the luarocks2 branch December 6, 2015 02:44

starius mentioned this pull request Jan 1, 2016

add package libechonest #1116

Closed

starius mentioned this pull request Jan 30, 2016

add new platform: MXE luarocks/luarocks#491

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

install native and cross lua executables & add LuaRocks plugin #1017

install native and cross lua executables & add LuaRocks plugin #1017

starius commented Nov 29, 2015

tonytheodore Nov 30, 2015

tonytheodore commented Nov 30, 2015

tonytheodore Nov 30, 2015

tonytheodore commented Nov 30, 2015

starius commented Dec 1, 2015

tonytheodore commented Dec 2, 2015

starius commented Dec 2, 2015

tonytheodore commented Dec 2, 2015

tonytheodore commented Dec 2, 2015

starius commented Dec 2, 2015

tonytheodore commented Dec 3, 2015

tonytheodore commented Dec 3, 2015

starius commented Dec 3, 2015

tonytheodore commented Dec 4, 2015

tonytheodore commented Dec 4, 2015

starius commented Dec 5, 2015

tonytheodore commented Dec 6, 2015

starius commented Dec 6, 2015

tonytheodore commented Dec 6, 2015

starius commented Dec 6, 2015

tonytheodore commented Dec 6, 2015

install native and cross lua executables & add LuaRocks plugin #1017

install native and cross lua executables & add LuaRocks plugin #1017

Conversation

starius commented Nov 29, 2015

tonytheodore Nov 30, 2015

Choose a reason for hiding this comment

tonytheodore commented Nov 30, 2015

tonytheodore Nov 30, 2015

Choose a reason for hiding this comment

tonytheodore commented Nov 30, 2015

starius commented Dec 1, 2015

tonytheodore commented Dec 2, 2015

starius commented Dec 2, 2015

tonytheodore commented Dec 2, 2015

tonytheodore commented Dec 2, 2015

starius commented Dec 2, 2015

tonytheodore commented Dec 3, 2015

tonytheodore commented Dec 3, 2015

starius commented Dec 3, 2015

tonytheodore commented Dec 4, 2015

tonytheodore commented Dec 4, 2015

starius commented Dec 5, 2015

tonytheodore commented Dec 6, 2015

starius commented Dec 6, 2015

tonytheodore commented Dec 6, 2015

starius commented Dec 6, 2015

tonytheodore commented Dec 6, 2015