Pull the lua_sandbox out of the Heka build process and packaging

[trink]

The lua_sandbox package should just be a dependency.

[trink]

I have received a -1 on pulling it out and having a Heka package dependency on the sandbox. So for now it will remain in. The lua_sandbox package and the heka package will not be able to co-exist on a box. We discussed statically linking the sandbox into Heka but it will not happen at this time.

[bbinet]

I'm interested in this issue, because I want to use Hindsight, which requires the lua_sandbox package to be installed, but I also would like Heka to be installed so that I can use tools like heka-cat.
For now I think that Heka current v0.9.2 release use an outdated version of the lua_sandbox which is not compatible with Hindsight, but when v0.10 will be released, do you think Hindsight will be able to use the lua_sandbox bundled into Heka?

[bbinet]

I don't really know why this issue was closed, but I think it would make sense to pull the lua_sandbox out of Heka itself as it is now a required dependency for both Heka and Hinsight.
And we can imagine the lua_sandbox could have its own life cycle: it could have more frequent releases than Heka and Hindsight when lua modules are being added or updated.
What do you think?

[trink]

Yes I still think pulling it out is the way to go @rafrombrc will need some more convincing. As for HS working with the sandbox installed with Heka 0.10 the answer is not at the moment (the lua_sandbox SHA in Heka has to be updated). BTW I was experimenting with an hs-cat it was about 6x faster on my box if that is all you want Heka for.

[bbinet]

Yes, hs-cat would be really useful to me, thanks.

If possible, I still would like to be able to install both Heka and Hindsight on the same machine, so that I can easily revert back to Heka just in case.
I think the best option would be to pull the lua_sandbox out of Heka, but as a temporary solution if it is just a matter of updating the lua_sandbox SHA to make it compatible with Hindsight, it would be great to do it before the upcoming 0.10 version is released.

[rafrombrc]

I'm +1 in principle to having a separate, shared lua_sandbox package. I'm definitely a strong +1 to having both Hindsight and Heka living on a machine together and not interfering with each other. What I want to avoid is adding yet more friction to the already somewhat burdensome process of getting started with a Heka build.

In 0.9 (IIRC) we switched from a monolithic Heka binary to using a separate lua_sandbox shared library. We gained a lot from that, but we also saw a considerable increase in installation support requests. My understanding is that if we move lua_sandbox to a separate package, folks building Heka for the first time will have to now complete two separate builds, with probably a bit of hand wiring to get them to work together.

An alternative would be to update the build to work with the new setup, grabbing both repos, building everything correctly, generating multiple binary packages with dependencies set up, etc. This is not a small amount of work, however.

Any ideas re: how to best resolve this, or (even better) help updating the Heka build to smoothly handle a separate sandbox build is welcome. :)

[trixpan]

@rafrombrc:

I've gone through that pain myself and being an user more than a developer it should be reasonably easy to flag some of the pains around building / installing the packages on Debian/Ubuntu and RHEL/Centos systems. I am happy to update the documentation to address some of the build/install things that explode right on you face when you build/install heka into a vanilla system.

I am biased (I like the idea of chrooting hekad), but meanwhile, perhaps static linking is the way to move forward(backwards?) until an updated build is figured out?

As it is now we already missing some of the advantages of dynamic linking (i.e. if people rely on source build.sh they already end with an old version of the sandbox anyhow, the RPM/DEB packages may end up with library conflicts)

[bbinet]

When we are in the process to build Heka, there is already some dependencies to install like cmake, so in my opinion this is not a problem to require the users to also install the lua_sandbox as an external package before building. I also think that people who wants to build Heka could be considered as advanced users. People will manually build the lua_sandbox only if they actually need a custom build, otherwise they can use an official RPM/DEB package.

[rafrombrc]

@bbinet I wish I could agree with you, but in my experience the amount of friction in the build experience has a direct impact on whether or not folks decide to use Heka, and also on the amount of support requests that we get.

Besides, using the official packages doesn't even solve everything. What we're saying is that the Heka package would depend on a separate lua sandbox package, which means that the sandbox package should be a package dependency. Getting CPack to handle all of this correctly is a non-trivial amount of work. I'm in support of that work being done, but I'm not in support of moving the sandbox to a separate package until that work has been done.

[sathieu]

@bbinet I find the current Debian packaging great and easy (I don't know for RPM).

I think the way forward ould be proper integration in the distribution, I may work on this on the Debian side, but days are only 24hours long ;-)

[bbinet]

@rafrombrc I'd be happy to help, but I don't have any experience with CPack, Cmake.
In the mean time, I've seen that #1733 has been merged, which means that when Heka 0.10 is released, we should be able to install Hindsight side by side with Heka and configure it to use the lua_sandbox coming from the Heka package.

[bbinet]

I've realized that #1733 has been merged in the dev branch, so it won't be available in the 0.10 version: is it possible to cherry-pick the commit to the versions/0.10 branch?