Heroku environment #88
Comments
|
That's odd. It's possible that the way the build system is checking for shared libraries is no longer valid. What does the following give you in a heroku console? require 'rbconfig'
RbConfig::CONFIG["ENABLE_SHARED"] |
|
irb(main):001:0> require 'rbconfig' |
|
That's strange. That error shouldn't be hitting then if you get |
|
Hey, I maintain the Ruby buildpack. We use the same image for building rubies that we use for running in production. I've got limited experience with c-extensions, but maybe I can help. What exactly is Rice trying to do when it's getting this exception. Is the error coming from trying to include a header or something? Maybe one of the path's is different/missing ? Can you think of anything we can do to trouble shoot or dig deeper into this error? |
|
@schneems Hey! Rice doesn't work against statically linked Ruby builds so in one of the first things we do in @MikaelMollberg what version of Ruby are you using in your app? |
|
I am using ruby 2.2.1p85 |
|
@MikaelMollberg Thanks. That then doesn't match up with the build flags Heroku gave you, as some of those flags reference files and libraries used for ruby-1.9. |
|
Wait, has anyone actually been able to use rice on Heroku before? As far as I know it's never worked on heroku. Basically to use a native gem on Heroku you need to either have a pre-compiled native available on rubygems or you need to use a Cedar 14 compatible environment and gem-compiler to compile the gem, then unpack it and vendorize it ALONG WITH librub.so/libruby.so.2.2/libruby.so.2.2.0 and any shared libraries your're linking against. BUT! There is a reason I am here now and that reason is there seems to be some sort of issue with any [static] ruby past 2.2.0. From 2.2.1 on gems using rice, both pre-compiled gems on rubygems and vendorized versions don't work. They throw "LoadError: incompatible library version" because they can't link to libruby.so. I don't think this is a Heroku issue so much as it is a flaw of statically compiled ruby in 2.2.1 on. @schneems If we could get dynamic ruby on Heroku this would be fixed. I tested this on a cedar:14 Docker instance and know this to be true. If we can't get dynamic ruby then the only solution I can think of is to find some way to get rice to work with static ruby - but I don't see that being any sort of easy solution ( @jasonroelofs can you confirm getting rice to work on static ruby is likely non-viable? ).
Heroku is only using static ruby, returning "no" is accurate. Previously (2.2.0 and below) simply vendorizing and including the location of libruby.so[.*] in the LD_LIBRARY_PATH overrode this but from 2.2.1 this seems to not be the case. As a side note, both 2.2.0 and 2.2.3 return "no" when checking ENABLE_SHARED but 2.2.0 still loads shared libs if you have them available. EDIT:
I think for 2.2.1+ this is the issue. |
|
@schneems Straight question: what's the viability of getting a dynamic ruby buildpack? Is that doable or is there some issue? |
|
Sorry for dropping off the issue. All my github replies were going to my trash. I'm not directly in charge of building binaries (in the past). This is what we've been using for our build process https://github.com/hone/docker-heroku-ruby-builder. There was a concerted effort to statically compile the Ruby libraries, it solved highly problematic issues, however I wasn't building rubies for the platform at that time so I don't know what the issues were. |
|
@schneems I'm actually building an isolated dynamic ruby 2.2.3 on the cedar:14 Docker image and I was going to kludge that into a test app to play with it. I'd imagine if you tried really heard you could also just use the multi buildpack and grab the brightbox rubies. I don't know if either of these would work but if either of them would then I'd imagine it'd light up your dyno manager with whatever issue you were trying to avoid. Do you think you can track down some info on why dynamic was a no-go? I wonder if whatever issues existed have been resolved. |
|
@hone I'm totally dragging you into this because you seem to be a major player in the MRI build for Heroku and because you are awesome. Do you think a dynamic ruby build would be viable? Are you aware of what the issues are with dynamic ruby on Heroku? |
|
He's on vacation right now. |
|
@Kagetsuki The reason Rice won't work with a static build of Ruby is because Rice itself builds a linkable object (librice.a) that the final extension will itself link against to build the final Rice extension. Then you need to be able to link against the Ruby library to complete the process. That said I'm not an expert at library compilation capabilities so it may be possible to have a build of Rice that does all compilation in one shot and get it working under static Ruby. I put that warning text in place a long time ago and don't exactly remember what I was running into at the time. |
|
Aaah. Well you're awesome too @schneems . If you get any info please give us a heads up. I still owe you for helping me out before - you continue to rock. |
|
@jasonroelofs I've spent way too much of my life messing with gcc/binutils to not give this a shot. I'm guessing if I wanted to try some things I should be looking at ruby/lib/mkmf-rice.rb.in first? Any pointers on what to start poking and prodding? I'm guessing if I did get static compilation to work the biggest downside would be size since you're basically going to be smashing libruby into everything. That's an extra 13MB and change for every native. |
|
@Kagetsuki Yeah that sounds about right. The |
|
@jasonroelofs Thanks, I'll give it a shot! |
|
Quick question: IF we can link static, SHOULD we link static by default? I ask because when you release precompiled gems on rubygems you'd need to have them static to be compatible with rubies not compiled with dynamic support (like Heroku and I think defaults for rbenv). At the same time if you had a dynamic ruby it'd end up being more bulk and I guess there is also the possibility there could be some issues if you have something else running a dynamic link to a slightly different libruby. |
|
@Kagetsuki That makes sense, and providing a binary gem would be nice, as it does take quite a long time to |
|
@jasonroelofs Just for clarification I'm talking about gems complied with rice, not rice itself. I'm pretty sure providing a precompiled version of rice could amplify problems if there were any issues like libruby version incompatibilities etc. I mentioned above. I'm not super sure any of these issues would actually arise but seeing as to how rubygems/bundler have such rough granularity on specifying binary versions I'm not liking the idea. I was more talking about the actual compilation done by rice, EG gems like my rapngasm. |
|
So I played around a bit and ended up directly modifying the Makefile to see if I could get a good combination. The key lines here ended up being: which gives me 100% passing on unittest AND to confirm with ldd: BAM! No libruby! A non static build ldd for comparison: But.... Don't get excited yet. vm_unittest gives me: Was this the issue you encountered? Any ideas? |
|
@Kagetsuki About the binary builds: right, that makes sense. And yes, regarding the vm_unittest errors, this suddenly makes a lot of sense. The details I ran into are hazy but I believe they were more to do with the static global variables that I have to be honest I have no idea who or if anyone actually uses the VM side of things, and maybe this is something that could be pulled out into a separate piece? |
|
@jasonroelofs I'm going to be straight an honest here: I have no idea what the "VM side of things" even means so I'll leave such decisions up to you. Would you like me to do anything to proceed? |
|
Sorry, the I need to carve out some time to go through this and a few other Issues and give some thought to where this should go. If leaving the VM class out of Rice means we can work with static-build Ruby, that would be a worthwhile trade-off IMO. Thanks for your help looking into this! |
|
@jasonroelofs That sounds like an entirely different type of project. I vote breaking this VM out if there's no need of it to perform the core rice task of building Ruby interfaces to C++. Give me a heads up if there is something I can help with. I'm not good for time this month (and actually quite behind after blowing half the week messing with native builds on Heroku) but I'd love to see this move along. If necessary I think I can loose a few more hours of sleep here and there 😁 |
|
Ha! I'll let you know if I run into something that you can help with, but I'd hate to be responsible for someone losing sleep! |
|
@jasonroelofs I've got a 9 month old and two kids in elementary school, I run a small startup still in the red with a product [emojidex] that always needs more work done and I'm a member of an armature motor-sports team. You wouldn't be the first reason for me to loose sleep by a long-shot. |
|
@Kagetsuki I've got a branch up that clears out |
|
Rock on @jasonroelofs ! I'm just finishing up some tasks right now but I'll give it a test on Ubuntu 15.04, 14.04, Debian and for the hell of it I'll give it a shot on my Windows VM because I actually had to test a build of something else on it anyway. Edit: I'll test with a system ruby on 14.04, 15.04 and Debian I'm using RVM. |
|
@jasonroelofs OK, not trying to put any pressure on you though. Just tell me if there's anything I can do to help. |
|
As far as I can tell this is going back to building-against-libruby-static that probably isn't fully solved yet. Also it looks like I'm wrong about RVM on OS X, I'm getting static and shared libraries built for any Ruby I install. Now that I'm trying to force actual static-only build for 2.2.3 under RVM, it's hitting failures trying to build the It's starting to look like Ruby 2.x is not a big fan of static linking, and I'm having a hard time tracking down how C-extensions even work on Heroku with a static-built Ruby. If you've got any insight or ideas @Kagetsuki I'd love to hear them. Btw, compiling static builds on rvm is |
|
@jasonroelofs On both my desktop and notebook I got libssl linking errors when I tried to install a static ruby with RVM (using the command you mention). Heroku basically has a super minimalist environment with a static ruby. Basically there is this stripped Ubuntu 14.04 image and on top of that a build-pack sets up a static Ruby of the version specified (usually in your Gemfile) on deployment. The problem here is that we have little solid information about how that Ruby is specifically composed. I can confirm there is a libruby.a, but anything over 2.2.0 seems to be unworkable (incompatible library errors). @schneems , just now when I wrote the above sentence I realized something. libruby for 2.2.x is always versioned 2.2. This is going to sound stupid, but is it possible the 2.2.0 libruby, which has the same 2.2 version code as the libruby for 2.2.1, 2.2.2, and 2.2.3, is being used on every ruby deploy from 2.2.0 to 2.2.3? If so that would explain why we get no issues on 2.2.0 and issues on 2.2.1~2.2.3. Otherwise there are several entries in http://svn.ruby-lang.org/repos/ruby/tags/v2_2_3/ChangeLog which catch my eye. EG: I still suspect something changed somewhere around 2.2.1 and it's messed everything up since |
|
@Kagetsuki if I understand what you're saying, no the libruby for 2.2.0 is not shared with 2.2.1/2.2.2/2.2.3/etc. |
|
@hone Good, I should hope so! I hate to take up your time but can you give us some insight as to weather or not a dynamic Ruby would be possible on Heroku? Is there a specific reason that you know of all Rubies on Heroku are static? |
|
Sorry to bump, but we've actually got an update for our site running on heroku and if we can't build we can't deploy. Anything I can do to help move this along? EDIT: I'm grinding away at this now in a cedar-14 container. Hopefully I'll have a PR for you soon. |
|
@hone @schneems So I tweaked some things and got a build working on the cedar-14 docker image. I grabbed an instance on heroku to double check. It's completely different. The ruby on an actual cedar-14 instance is absolutely different than the one on a cedar-14 container. Any ideas as to why this is or any suggestions as to how I can get the same ruby on a cedar-14 instance on a cedar-14 container? |
|
@jasonroelofs I've spent the entire day banging away at this and I've made basically no progress. Part of the problem is I can't get a build of a static ruby that behaves the same way as the ruby on heroku. I can confirm though that the build flags @MikaelMollberg states in the top post are probably inaccurate (perhaps @hone or @schneems can comment on this?). Can anyone give me a heads up if a build on a static ruby OTHER than the one on cedar-14 is giving the error? If not I have a strong suspicion this is specifically an issue with how ruby on heroku is built and NOT an inherent issue with static rubies. |
|
@Kagetsuki Sorry, I really don't have anything else to offer right now. I haven't been able to build a static Ruby on OS X that works and links with Rice. I get the same relocation errors that you are getting. I wish I had more to offer but I'm also not sure where to go from here. |
Thanks for the confirmation! What's confusing to me is that it appears everything is being built with -fPIC - can you confirm that everything in the chain should be being built with PIC? It also occurs to me that PIC works differently in static and dynamic libraries. Maybe we should be building librice as dynamic? The thing is I'm still not completely sure how rice works so I'm not sure it would even be worth trying this. If you can think of anything else give me a heads up and I'll give it a try. |
|
I have not been able to get a build running on a static ruby BUT I did kludge a build and got it to run on a dyno using the following process:
So, basically the static ruby on heroku WILL run dynamically linked things against a dynamic libruby - but it's an absolute mess to do and not anything maintanable. I don't find this to be an acceptable solution so I'm going to either keep trying to get static builds to work properly or see if I can get a dynamic ruby on heroku. @schneems or @hone , I hate to keep bothering you, but can I please get a response as to why dynamic rubies are not available on heroku? I don't really understand how build packs work and I couldn't seem to find where the build args were in the ruby buildpack, so I'd really appreciate some information or some help. |
|
@jasonroelofs Sorry for filling this thread with crap, but I'll give you a status update: Clarification: If what I'm being told is in fact correct, automake/libtool will actively disable PIC for static compilation. In this case we're being told to add -fPIC likely because some automake magic is disabling it at some key point in the build chain. Furthermore, when using libtool, you really need to tell it to disable static building completely to get it to really build portable PIC code as libtool will default to try and optimize to the architecture/configuration it's being run on. Update: I have the .la building but past that I can't seem to coax it into actually building an .so to link against or to get the .la linked or whatever libtool is supposed to do. I'm reminded of why I started using CMake instead of automake. |
|
Yeah I wouldn't be opposed to moving to CMake, but the big problem there is that it then requires people to have CMake installed to be able to build Rice and build gems that are using Rice. We use autotools as we can make a decent assumption that a system with a C++ compiler will also have the autotools installed. |
|
The problems you ran into using our docker image may have been to using the default installed Ruby version. When you deploy with the buildpack we download and unpack a new version for example 2.2.3 https://heroku-buildpack-ruby.s3.amazonaws.com/cedar-14/ruby-2.2.3.tgz. |
|
@jasonroelofs Aah I didn't mean to imply we should move to CMake, just that I'm used to CMake because I made the decision to use it a long time ago after comparing generative make systems and found CMake to be the one I liked the most. Moving to CMake now would be a massive undertaking which I would only recommend if for some reason automake just won't work. Status update: We do have librice generating on static environments. Instead of continuing to prod at the build chain I'm going to try and manually write a build line to see if I can get anything to work. |
|
@schneems Any idea where I can find the build flags that are used to generate that ruby? And I hate to ask again, but any chance we can get some idea as to why we can't have a dynamic ruby on heroku? |
|
I don't think dynamic built rubies is a possibility. Using static built libs solves a host of otherwise hard to debug and solve problems. Here's flags https://github.com/hone/docker-heroku-ruby-builder/blob/48d17c8a02e0b779dec19947015a8b83bdad508c/build.rb#L140-L142 |
|
@schneems I really need some clarification on why and I need it now. I have an app on heroku that I currently can't update because of this and I NEED to update it. This is years of work and the future of my business coming to a halt because of one "small" issue. On top of that, as I mentioned above, I HAVE successfully build dynamic ruby on a dyno; so I at least know it can be built and run in a tethered shell - so I'm not buying "it can't be done because of reasons". |
|
Regardless of what you will and won't "buy" we won't be changing the way future rubies are compiled or recompiling any. I recommend you either fork the buildpack and compile your own ruby, or deploy somewhere that does not provide pre-compiled executables. For some perspective rubygems.org has 109,873 gems with over 6,511,481,79 downloads. Of all of those gems, this is the only one that i've seen in memory that hasn't been able to install on Heroku due to how we build Ruby. I understand you're frustrated and you're on a deadline. I'm interested in providing what help I can. Recompiling all the rubies on the platform are not on that list. |
We're talking about building, not installing - and that gave me an idea for a kludge that looks like it will let me deploy and update but it will be absolutely specific to heroku - so not a final solution. I'll report if it worked; but from what I'm seeing passenger actually did a similar hack a little while ago to overcome some temporary build issues almost a year ago. |
|
@jasonroelofs I'm putting @lev-k on this from monday morning. He's a savant. This particular issue has gotten pretty junked up (mostly due to me freaking out). We've also determined this issue is not Heroku specific but rather an issue with libruby-static.a specifically. What do you say to maybe closing this and starting up a separate issue like "Static build". Also, there are a few changes I'd like to propose:
|
|
This ticket has definitely gone in a direction I'm not particularly happy about. Lets start a new discussion for sure, but to answer your points here:
|
|
|
@jasonroelofs I'm afraid this may be a final status update from me.
If anyone has any ideas it would be nice to hear them. Otherwise @jasonroelofs I think this issue is spent and should be closed. @schneems if you're going to be at Ruby Kaigi give me a heads up - I owe you a good meal or a few drinks over this one. |
|
In the end were were forced to migrate to a different method without rice and doing so required I actually do make a buildpack and learn how buildpacks work; which was actually easier than I thought. Basically you can make a dynamic ruby buildpack, it will run, and rice will build on it. According to @schneems this should not be done so I'm basically going to stop there. Essentially, as it stands now, your options are:
I'm still hoping someone fixes Rice for static rubies (and maybe for Windows too at some point?) so we can use it again in the future. It is hands down the most flexible wrapper/interface framework for C++ to Ruby. With that said I'm officially dropping out of this discussion. Best of luck and thanks to everyone for their time and effort. |
|
@Kagetsuki Understood, sorry we couldn't figure out a good solution for you, and thanks for the info and work you put into this. I'll put what you've figured out into documentation so we a good starting point if someone else wants to pick this up and try to continue on. |
|
@jasonroelofs I'm sorry I couldn't get it to work. I was certain I could for a while there but that relocation issue is just an impassible roadblock. I honestly think it has something to do with Ruby core but for the life of me I can't figure out what it is, where it originates from, or when the patch was applied that lead to the issue. As for my findings I'm not sure how public they should be made. I don't want to discourage anyone away from Rice and I'm worried that my comment may do so - but at the same time I don't want anyone to go through the month and a half of trying out countless different semi-solutions only to come to the conclusion that I did 😞 |
After a long discussion and a lot of work in #88, we discovered that there are two issues at play here that are out of our control. 1) Heroku runs it's own custom static-link Ruby which we can't get working with Rice. 2) Ruby 2.2.2 and on has some serious issues with static linking in general and there doesn't seem to be any pressure to fix this. With that, I have to re-enable the static Ruby check so people know when they are trying to run Rice on a platform that won't support it.
Hi, I tried installing rice on heroku but received this error.
remote: extconf.rb:23:in
': Unfortunately Rice does not work against a Ruby without any shared libraries. (RuntimeError)remote: You'll need to rebuild Ruby with --enable-shared to use this library.
I contacted their support and they said that ruby is built with --enable-shared and told me to contact the maintainer. Any idea what could be wrong?
According to them Ruby is compiled with the following flags.
'--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/usr/lib/ruby1.9.1' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--enable-pthread' '--enable-shared' '--disable-rpath' '--disable-install-doc' '--with-vendordir=/usr/lib/ruby/vendor_ruby' '--with-sitedir=/usr/local/lib/site_ruby' '--program-suffix=1.9.1' '--with-soname=ruby-1.9.1' '--enable-ipv6' '--with-dbm-type=gdbm_compat' '--with-tklib=tk8.5' '--with-tcllib=tcl8.5' '--with-tcl-include=/usr/include/tcl8.5' '--with-tk-include=/usr/include/tcl8.5' '--with-tcl-lib=/usr/lib/x86_64-linux-gnu' '--with-tk-lib=/usr/lib/x86_64-linux-gnu' '--with-bundled-sha1' '--with-bundled-md5' '--with-bundled-rmd160' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall -fno-strict-aliasing' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -L/build/buildd/ruby1.9.1-1.9.3.484/debian/lib' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall -fno-strict-aliasing'
The text was updated successfully, but these errors were encountered: