Lua object class support #7338

dotnwat · 2016-01-24T01:19:29Z

No description provided.

yuyuyu101 · 2016-01-24T04:22:43Z

COOL!

Not sure we need to contains lua source in ceph repo. Specify a external directory may better?

cxwshawn · 2016-01-24T06:22:14Z

cool! looking forward to being merged !

liewegas · 2016-01-25T22:59:40Z

Does it makes sense to use a submodule instead of an extracted tarball for lua?

dotnwat · 2016-01-26T00:12:44Z

@yuyuyu101 @liewegas I have no issue with Lua being in a submodule (what's the motivation?) The only important thing is that we control that submodule because we may want to customize the VM in the future, but I'm assuming I target a ceph/lua repo?

liewegas · 2016-01-26T02:19:20Z

yeah. the submodule makes it easier to move between lua versions, keeps lua and ceph code separate, and avoids munging up any source code stats.

dotnwat · 2016-01-26T06:14:03Z

sounds good. i'll need to have someone create ceph/lua repo.. looks like I don't have permission.

liewegas · 2016-01-26T13:46:25Z

Forked! https://github.com/ceph/lua

dotnwat · 2016-01-27T23:55:37Z

Thanks. I need push access to that repo. It isn't a vanilla Lua distribution, and its build configuration has been modified to assume cmake, so we'll need to hack it a bit to play nice with automake too.

batrick · 2016-01-28T16:01:43Z

Hi @noahdesu, this is an exciting feature! I'm still looking through the code and have yet to try this out but I was wondering why you include Roberto's Lua struct module when Lua 5.3 now has a pack function (which is loosely based on struct).

dotnwat · 2016-01-28T18:49:43Z

Hey @batrick thanks for pointing that out! I didn't realize that Lua 5.3 had this function. I wanted to include some useful serialization modules out-of-the-box so people could immediately start using this. It sounds like it might be OK to nuke the struct module for now. Let me know if you have any other questions or issues using cls_lua.

cbodley · 2016-01-28T19:02:22Z

src/CMakeLists.txt

+target_include_directories(cephlua
+    PUBLIC lua/lua-5.3.2/src/
+)
+


could we move the cmake bits into a lua/CMakeLists.txt please?

Absolutely. I'm in the process of stuffing Lua into a git submodule and I'll add this to the todo list.

dotnwat · 2016-01-30T22:13:05Z

@liewegas mind adding me to ceph/lua so I can push there?

yehudasa · 2016-01-30T23:36:47Z

@noahdesu I updated the permissions there

dotnwat · 2016-02-01T04:43:28Z

Thanks @yehudasa that worked. @liewegas ok got it all submodule-ified. @cbodley could you take a look at the new cmake files?

cbodley · 2016-02-01T14:35:43Z

@noahdesu the cmake looks good, thanks. i'll check it out and make sure it builds for me too

dotnwat · 2016-02-07T17:22:01Z

Rebased to remove merge recent merge conflicts.

batrick · 2016-02-14T00:02:51Z

src/cls/lua/cls_lua.cc

+  cls_method_context_t hctx = clslua_get_hctx(L);
+  const char *key = luaL_checkstring(L, 1);
+  bufferlist  *bl = clslua_pushbufferlist(L, NULL);
+  int ret = cls_cxx_map_get_val(hctx, key, bl);


I believe ret needs to be negated on error right? In my tests, it returns a negative value which gives us:

Unknown error -2

in the logs.

Do you mean negated to resolve to the correct errno string via strerror, or negated to change the logic?

To get the correct errno string I believe.

Thanks for catching this Patrick. I pushed a fix for it that seems to work (note that I also rebased to fix a merge conflict in master) 50739d7

athanatos · 2016-05-04T22:02:36Z

There was some discussion of this at one of the recent CDM's. I think one of the main concerns was that we need a mechanism to whitelist classes for users?

dotnwat · 2016-06-27T21:51:20Z

@athanatos yes. i just created PR #9972 for review, that adds the loading and caps whitelist changes for object classes.

ktdreyer · 2016-08-04T14:45:34Z

Noah, could you build against the system lua instead of bundling it into Ceph? (for example RHEL/CentOS 7 have lua-devel-5.1.4-14.el7)

When we bundle lua's source code into Ceph, the Ceph build time becomes longer, and the Ceph project becomes responsible for fixing CVEs in the bundled version, etc.

batrick · 2016-08-04T15:48:50Z

Does RHEL7 have Lua 5.3? Optionally building against the system Lua 5.3 sounds reasonable to me.

Also, Lua's source is pretty small. On my humble laptop a parallel make builds in 2.5 seconds.

dotnwat · 2016-08-04T16:13:20Z

I'm not opposed to linking against a system library, but it seems the precedent is to avoid this when embedding the VM mostly due to compatibility and customization issues. For instance Redis disables byte-code execution with a patch to the VM source, and other things are possible like switching out the native numeric data type.

From the Lua documentation: "Different versions are really different. The API is likely to be a little different (but with compatibility switches), and there is no ABI compatibility: applications that embed Lua and C libraries for Lua must be recompiled. The virtual machine is also very likely to be different in a new version: Lua programs that have been precompiled for one version will not load in a different version."

That might be manageable when creating releases, but strikes me as a headache waiting to happen especially for people mixing Lua versions. Thoughts?

ktdreyer · 2016-08-04T16:19:46Z

There are pros and cons to bundling, and I wanted to at least have the discussion so I understand why we're doing it in each case. As long as the team is ok maintaining it when security fixes come up, I'm ok with it.

dotnwat · 2016-08-04T16:25:25Z

Sounds good. I'm happy to do any tests or research on the issue for everyone to agree on an approach.

dotnwat · 2016-08-09T01:30:59Z

the libgmock_main.so problem was caused by some cmake stuff that the lua submodule was doing that was causing various other libraries to be built as shared libraries rather than the expected static variety.

@yuriw anyway, let's give this a shot again. i was able to install from packages on deb/rhel and things are working now.

tchaikov · 2016-08-09T02:32:43Z

the libgmock_main.so problem was caused by some cmake stuff that the lua submodule was doing that was causing various other libraries to be built as shared libraries rather than the expected static variety.

@noahdesu in case you didn't find it, see https://github.com/ceph/lua/blob/lua-5.1/cmake/dist.cmake#L77 . i suffered from this when working with gmock also. we need to reset this variable after including the lua subdirectory, or save and restore it before and after including the lua subdirectory.

dotnwat · 2016-08-09T03:08:51Z

@tchaikov nice thanks for the reference, that was a tough one to track down. is there any reason to patch the lua variable? when i was reading through dist.cmake i didn't see anything in there we care about. it was packaging and installation stuff.

tchaikov · 2016-08-09T03:15:58Z

Noah, see https://cmake.org/cmake/help/v3.0/variable/BUILD_SHARED_LIBS.html . this variable changes the default behavior of add_library(). and as we are including lua, not building it as a separated module, this variable actually impacts the whole project, including the gmock, which is also included by us.

dotnwat · 2016-08-09T03:17:21Z

@tchaikov cool thanks. i'll get that fixed up tomorrow.

dotnwat · 2016-08-09T20:42:10Z

@tchaikov I think I misread your comment. I understand the global impact of the BUILD_SHARED_LIBS variable. I was asking about patching it indirectly: that variable is set in lua/cmake/dist.cmake which contains stuff related to installation and packaging---cmake boilerplate that we do not depend on, including shared library building which doesn't matter since we build lua as a static library. So I disabled the entire inclusion of lua/cmake/dist.cmake, including the BUILD_SHARED_LIBS global variable. This file also sets numerous cmake variables, so we don't need to worry about them clobbering other settings either.

batrick · 2016-08-09T21:00:25Z

@noahdesu, have you seen the thread on ceph-devel concerning Lua packaging? I'd link to it but gmane appears down (maybe permanently).

In particular, I'm concerned about including Ceph libraries in the lua submodule. Can those be moved src/cls/lua?

dotnwat · 2016-08-09T21:08:02Z

@batrick thanks for the pointer, i'll chime in on the list. about your question here: you are referring to cmsgpack and the bufferlist bindings? absolutely, they can be moved to src/cls/lua. lua was originally moved to a submodule and the reason people wanted that was that it 1) screwed up LOC stats and 2) ostensibly was easier to manage. Since I didn't author cmsgpack it seemed more appropriate there, but its easy to move. The reasoning behind leaving bufferlist bindings there was because it was general purpose.

tchaikov · 2016-08-10T03:00:25Z

I was asking about patching it indirectly: that variable is set in lua/cmake/dist.cmake which contains stuff related to installation and packaging---cmake boilerplate that we do not depend on, including shared library building which doesn't matter since we build lua as a static library. So I disabled the entire inclusion of lua/cmake/dist.cmake, including the BUILD_SHARED_LIBS global variable.

@noahdesu sorry, i failed to understand your question. ic. i thought a less intrusive way would be easier for maintenance. but when i am looking at https://github.com/ceph/lua/commits/lua-5.3-ceph: well, we have already added a handful of patches on top of lua-5.3 branch, yet another one won't hurt. =)

michaelsevilla · 2016-09-02T18:08:03Z

What is the status on this?

dotnwat · 2016-09-02T22:15:18Z

I need to take another look at the thread on ceph-devel to make sure nothing is missing, but I think we just need to slap needs-qa back on this.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Introduces cls_lua that allows object classes to be created dynamically using the Lua language. Each request is processed in an empty Lua VM instance, and scripts can be submitted using bufferlist or JSON encoding. Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

yuriw · 2016-09-22T16:18:19Z

http://pulpito.ceph.com/yuriw-2016-09-19_19:42:27-rados-wip-yuri-testing2_2016_09_19-distro-basic-smithi/

rerun http://pulpito.ceph.com/yuriw-2016-09-20_14:50:21-rados-wip-yuri-testing2_2016_09_19-distro-basic-smithi/

yuyuyu101 added feature core labels Jan 24, 2016

cbodley reviewed Jan 28, 2016
View reviewed changes

dotnwat force-pushed the cls-lua branch from f4a9527 to 0ee49be Compare February 1, 2016 03:04

dotnwat force-pushed the cls-lua branch from 0ee49be to a375f00 Compare February 7, 2016 17:21

dotnwat force-pushed the cls-lua branch from a375f00 to 628ffda Compare February 12, 2016 15:45

batrick reviewed Feb 14, 2016
View reviewed changes

dotnwat force-pushed the cls-lua branch from 628ffda to 50739d7 Compare February 15, 2016 20:20

dotnwat force-pushed the cls-lua branch from 50739d7 to 3f14c5f Compare March 18, 2016 16:42

athanatos added the pending-discussion label May 4, 2016

dotnwat force-pushed the cls-lua branch from 3f14c5f to 705a474 Compare May 28, 2016 15:39

dotnwat force-pushed the cls-lua branch 2 times, most recently from 223aa95 to 91c7c43 Compare June 27, 2016 05:01

dotnwat force-pushed the cls-lua branch from abc439f to 7e3458c Compare August 8, 2016 21:25

dotnwat added the needs-qa label Aug 9, 2016

dotnwat removed the needs-qa label Aug 9, 2016

dotnwat added the needs-qa label Aug 9, 2016

dotnwat removed the needs-qa label Aug 9, 2016

michaelsevilla mentioned this pull request Aug 26, 2016

fs: Mantle: A Programmable Metadata Load Balancer #10887

Merged

dotnwat added 2 commits September 13, 2016 15:34

liblua: build Lua convenience library

e638336

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

dotnwat force-pushed the cls-lua branch from 8aa7d90 to cb33c5c Compare September 13, 2016 22:53

dotnwat added the needs-qa label Sep 14, 2016

yuriw added the wip-yuri2-testing label Sep 15, 2016

yuriw merged commit ab42f60 into master Sep 22, 2016

batrick deleted the cls-lua branch September 23, 2016 16:49

Lua object class support #7338

Lua object class support #7338

Conversation

dotnwat commented Jan 24, 2016

yuyuyu101 commented Jan 24, 2016

cxwshawn commented Jan 24, 2016

liewegas commented Jan 25, 2016

dotnwat commented Jan 26, 2016

liewegas commented Jan 26, 2016 via email

dotnwat commented Jan 26, 2016

liewegas commented Jan 26, 2016 via email

dotnwat commented Jan 27, 2016

batrick commented Jan 28, 2016

dotnwat commented Jan 28, 2016

cbodley Jan 28, 2016

Choose a reason for hiding this comment

dotnwat Jan 28, 2016

Choose a reason for hiding this comment

dotnwat commented Jan 30, 2016

yehudasa commented Jan 30, 2016

dotnwat commented Feb 1, 2016

cbodley commented Feb 1, 2016

dotnwat commented Feb 7, 2016

batrick Feb 14, 2016

Choose a reason for hiding this comment

dotnwat Feb 14, 2016

Choose a reason for hiding this comment

batrick Feb 14, 2016

Choose a reason for hiding this comment

dotnwat Feb 15, 2016

Choose a reason for hiding this comment

athanatos commented May 4, 2016

dotnwat commented Jun 27, 2016

ktdreyer commented Aug 4, 2016

batrick commented Aug 4, 2016

dotnwat commented Aug 4, 2016

ktdreyer commented Aug 4, 2016

dotnwat commented Aug 4, 2016

dotnwat commented Aug 9, 2016

tchaikov commented Aug 9, 2016 • edited

dotnwat commented Aug 9, 2016

tchaikov commented Aug 9, 2016 • edited

dotnwat commented Aug 9, 2016

dotnwat commented Aug 9, 2016 • edited

batrick commented Aug 9, 2016

dotnwat commented Aug 9, 2016 • edited

tchaikov commented Aug 10, 2016

michaelsevilla commented Sep 2, 2016

dotnwat commented Sep 2, 2016

yuriw commented Sep 22, 2016

tchaikov commented Aug 9, 2016 •

edited

tchaikov commented Aug 9, 2016 •

edited

dotnwat commented Aug 9, 2016 •

edited

dotnwat commented Aug 9, 2016 •

edited