Shipping libevent.a and libpcre.a with the linux binary packages

[lh3]

It seems that the crystal compiler requires libevent and pcre. If either library is absent, the linker will throw an error like

/usr/bin/ld: cannot find -levent (this usually means you need to install the development package for libevent)

Unfortunately, libevent and pcre are often missing from a system and installing/compiling them without root/sudo is non-trivial for inexperienced users.

A simple solution to this problem is to compile the two libraries as static libevent.a and libpcre.a on older Linux and put them in crystal-0.34-1/lib/crystal/lib along with libgc.a. This way, we can compile without system libevent and pcre. Furthermore, the resulting binary is not dynamically linked to libevent.so or libpcre.so – it is more portable.

PCRE and libevent don't have additional dependencies. They can be easily compiled as static libraries with ./configure --disable-shared. On my system, the compiled libraries can be found in .libs/lib{event,pcre}.a. I have tried this approach. It is working well for me.

[refi64]

This would potentially be quite a mess once you try to move it across distros...

…

On Wed, May 13, 2020, 12:10 AM Heng Li ***@***.***> wrote: It seems that the crystal compiler requires libevent and pcre. If either library is absent, the linker will throw an error like /usr/bin/ld: cannot find -levent (this usually means you need to install the development package for libevent) Unfortunately, libevent and pcre are often missing from a system and installing/compiling them without root/sudo is non-trivial for inexperienced users. A simple solution to this problem is to compile the two libraries as static libevent.a and libpcre.a on older Linux and put them in crystal-0.34-1/lib/crystal/lib along with libgc.a. This way, we can compile without system libevent and pcre. Furthermore, the resulting binary is not dynamically linked to libevent.so or libpcre.so – it is more portable. PCRE and libevent don't have additional dependencies. They can be easily compiled as static libraries with ./configure --disable-shared. On my system, the compiled libraries can be found in .libs/lib{event,pcre}.a. I have tried this approach. It is working well for me. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#9285>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAM4YSJPSRRHDF6L5Z5YMVDRRITVVANCNFSM4M7NKHGQ> .

[asterite]

@lh3 how are you installing Crystal?

[jhass]

This seems like an issue for https://github.com/crystal-lang/distribution-scripts

I hope with 1.0 more distros will accept us into their repositories and this will become less of an issue.

I'm happy to be on a distro where there's an official package already, so I don't have to rely on the big workaround that the above linked scripts essentially are :D

[RX14]

Installing many things without root access is tricky. I don't think crystal is special in this regard - or even has particularly unusual dependencies.

I'd rather use system libraries where at all appropriate, since it vastly reduces incompatibilities on systems with versioned glibc symbols.

[lh3]

@asterite I am using a distributed binary. More exactly: file crystal-0.34.0-1-linux-x86_64.tar.gz from the release page.

@jhass I am not familiar with distribution-scripts. My end goal is to be able to download a self-contained binary package such that my users and myself don't have to install additional libraries to use crystal. >90% of my users don't have root privilege on the machines they run tools and they don't know how to install libevent or pcre by themselves.

@RX14 and @refi64 I understand glibc is tricky. A common practice is to compile binaries on CentOS6, which is one of the oldest but still maintained distros. Then the compiled binary will work on recent systems. I haven't seen exceptions so far, though maybe I haven't used enough distros. Building on CentOS6 is essentially how anaconda ships portable binaries.

You can find precompiled static libraries for libevent and pcre on my FTP site:
ftp://ftp.dfci.harvard.edu/pub/hli/crystal-libs/. You can download the two files, copy to crystal-0.34.0-1/lib/crystal/lib and see if they work on your system. If there are no compiling errors, you may use ldd (on Linux). You should see that the resulting binary doesn't dynamically linked to libevent and libpcre any more.

[asterite]

@lh3 the releases page is just there for package managers and others to be able to build distributions. To install crystal you should use your OS distribution system. For example in mac if you use homebrew everything works out of the box.

[lh3]

@asterite On mac, everything is simple. The problem is linux. Most of my users don't use linuxbrew. I can't force them to install linuxbrew just for crystal.

[Blacksmoke16]

He means based on their actual OS. See https://crystal-lang.org/install/, which should cover the majority.

[lh3]

@Blacksmoke16 Could you elaborate? I said my users don't have root privilege (PS: I don't have sudo permission, either) – this is common among computing environments in academia. They are unable to install packages by themselves.

[Blacksmoke16]

To install crystal you should use your OS distribution system. For example in mac if you use homebrew everything works out of the box.

More so meant this part. I.e. if on Mac you would use homebrew. Arch or Manjaro would use pacman, etc.

this is common among computing environments in academia

I imagine they would need an admin to install it then? or maybe build from source (although would prob run into the same problem when trying to install llvm for example :/).

[lh3]

@Blacksmoke16 In an academia setting, admin is often non-responsive. That is why conda is becoming popular. I have already given a solution above which conda uses to some extent. I have experiences in dealing with the issue. I am reasonably certain it will work. You can try that.

although would prob run into the same problem when trying to install llvm for example

I can install crystal using the binary package from the release page. If someone can add the two *.a files to the binary package, my problem is solved, and I think it will work for others.

[waj]

We used to release crystal binaries using omnibus. Those packages contains libevent, libpcre and other libraries that are linked by some part of the standard library. @lh3 maybe you need just those two because are required by default, but libyaml must be added if YAML api is used for example.

Maybe someone can enlighten the conversation by providing references to where the decision to change was made, so the reasons are not exposed here all over again. I agree with that change anyway but probably there is still space for omnibus generated packages for some users. The package didn't contain actual linker and system libraries, so a build environment is still required.

[lh3]

@waj Thank you for the information. I appreciate. I will add one minor note. The binary package on the release page ships libgc.a because it is essential to crystal. libevent.a and libpcre.a are actually also essential in that we can't compile a "hello world" program without them.

[jhass]

Maybe educating your users to use a non-root package manager/distribution form such as linuxbrew, nix, stow, toast, junest, 0install, pkgbrew, zpkg, AppImage, Flatpak etc. could prove useful outside this single instance, solving this problem more generically?

Rather than reintroducing Omnibus (it was quite the pain to maintain I think), I would look into making Flatpak and/or AppImage official distribution methods.

[waj]

Julia? 🤔🙂

libgc.a is shipped because we added some patches for MT and are not yet released. Also some distributions ship older versions we couldn't use with Crystal.

[lh3]

@jhass I understand your concerns. If it is indeed tricky to add these two files, I will probably provide my users with my own binary crystal package with the files. Another less-ideal (to me)
solution is to provide a crystal package via conda. It is much more popular than linuxbrew and other package managers in my field.

@waj Oh, sorry. It is a typo. I am playing around several newer languages to see what to recommend to non-CS researchers (I can't tolerate the performance of python). So far crystal is the top on my list. I think it has a better and cleaner design than Nim and Julia. Libevent.a/libpcre.a is the last bit that has been stopping me.

[jhass]

Ah, I didn't realize Anaconda is essentially a package manager, it's frontpage tells me nothing about what it really does :D

To me that sounds like a good solution, we should just have Crystal packages for everything :) But of course that's impossible to do by the core team and we need the help of respective communities to maintain such packages. I see our responsibility as making that possible and easy. Maintaining a Crystal package in the AUR was essentially one of my first contributions to Crystal and it lead to Archlinux to be one of the first (if not the first) distribution shipping Crystal in its repositories :)

[lh3]

Ok. I will explore how to add a crystal package to conda. I need to seek help from the conda community as I am not that familiar with the conda build system, either. This will take time.

Meanwhile, I still hope you may consider to add necessary static libraries to the x64-linux tar.gz. Doing this will benefit more users. I really appreciate that you build a statically linked crystal binary, which makes things much easier. Shipping a few more static libraries would be perfect. For the time being, I will provide my own x64-linux crystal package by adding libevent.a and libpcre.a.

[bcardiff]

@RX14 do you recall the rational behind dropping those libraries in distribution-scripts with respect what was done in the omnibus originally?

[RX14]

Because gc isn't packaged - or new enough - in every distro. We got actual errors from users when they were using the distro libgc package. The same was not true with pcre or libevent - people installed the distro package and it just worked. Now, the reason is because we ship a patch libgc for MT.

I agree the solution is conda or similar. If people don't have root, they need a user-mode package manager. conda, nix and linuxbrew fill that role.

[lh3]

@RX14 Thanks for the explanation. I respect your decision. Just curious: have libevent.a and libpcre.a caused troubles when they were shipped with the binary tar.gz?

[straight-shoota]

I don't think the reason they were dropped is that they caused trouble, but they're simply non-essential. It's better not to drag that baggage in the core project when it's not necessary.

[asterite]

But they are not baggage. They are essential to every crystal program. That's the reason why we distributed them statically. I didn't know that changed.

Sure, libyaml is required if you use yaml, same goes for libgmp for BigInt and such, and maybe we should include those.

[RX14]

They are baggage because they're more things we need to maintain.

There's a fairly rare usecase (not having root access) where this is a problem, so I wouldn't rush to change this.

[straight-shoota]

@asterite I don't mean the libraries themselves are baggage but maintining their distribution with crystal requires a lot of additional work. Of course, if we can get a full batteries-included distribution package without much effort, I'm all in.

Even not having root access is not really a problem. You can get the libraries without root. There are specific non-root package managers. Or you can just download it.

[asterite]

But aren't these libraries already available for download? We would just need one step in the build process which is to download them and put them in the tar.gz

[lh3]

There's a fairly rare usecase (not having root access)

Probably >90% users in the field of bioinformatics don't have root access. I am reasonably certain that quite a few other fields in academia have the same problem.

maintaining their distribution with crystal requires a lot of additional work.

You can compile the two libraries once and copy them to future binary packages. You may only upgrade them occasionally.

Or you can just download it.

The purpose of providing statically linked crystal binary is to make it run on every linux. Not shipping essential libraries with crystal has defeated the goal.

[RX14]

You may only upgrade them occasionally.

And we have to track them for security issues and make sure we keep them fairly well up to date. libpcre especially is a CVE quagmire as it JITs code.

If someone else in the core team wants to maintain this then it's on them, but I think the risk is fairly high and the benefits are low. So I'll duck out of this conversation.

[lh3]

And we have to track them for security issues and make sure we keep them fairly well up to date.

The crystal compiler is mostly written in crystal. When you compile the static crystal binary, you need to keep track of the security issues in crystal dependencies anyway.

I am trying to promote crystal in my field. I just think having the suggested change will make me much easier to recommend crystal. Anyway, thank you for your time. I appreciate.

[lh3]

Due to the way crystal is distributed, I don't think I have a chance to persuade devs in my field to use crystal, so I am providing portable binaries through my repository for now.

I will talk to people who are familiar with conda. It will take time. Also, distribution through conda is not ideal, either, because all compiled executables will be dynamically linked to libevent and libpcre. This will complicate binary distributions of developers' tools. I can't tell users: "you have to install crystal, libevent and pcre to use my tool". This will push 90% of my users to other tools.

Again, for users, the best solution is to include static libraries in tar.gz. You need to keep these libraries updated anyway to compile the statically linked crystal binary. I don't see why this adds much burden. Anyway, I am closing the issue as you seem not interested.

[straight-shoota]

That would be terrific @winni2k. As someone working in the field of this use case, you probably have a better angle than most other contributors.
Let me know if you need any assistance, knowledge or act. The community chat rooms are also a good ressource to get help on really anything.

[lh3]

The conda solution is not ideal for two reasons. First, not everyone is using conda. Second, binaries compiled with dynamic libevent and libpcre require endusers to have the two libraries installed. This hurts binary distribution.

The best solution is still on the crystal end: put libevent.a and libpcre.a in the binary distribution package. You already have the two libraries compiled as you need them to link the crystal executable statically. You only need to copy the two files to the crystal-1.0.0-1/lib/crystal/lib.

[winni2k]

For what it's worth, I tried building a conda package from both the 1.0.0 and 0.32.0 tar files on a CentOS/7 VM that does not have pcre installed. In both cases, I got segmentation faults when trying to compile a hello world example file through the test suite of conda build .. Here is a gist of the configuration files that I used. I'm not sure how to troubleshoot further, but perhaps someone else reading this issue might find it helpful.

Update
However, there is a hacky workaround that I can use to install the crystal compiler into an arbitrary conda environment by manually moving the libpcre.a and libevent.a libraries into the right place as @lh3 suggested, just within the conda environment. I have created a bash script to automate setting up the environment. I think this means that building a conda env for crystal should be possible in principle.

[lh3]

Just notice that crystal is providing pcre and libevent as static libraries on mac. If you can do the same to the Linux binary package, it would be great. You already have the mechanism.

[straight-shoota]

The distribution workflows for macOS and linux are completely different, so there's not really something to re-use. And the macOS ecosystem is simpler.

Basically we already have the static libraries in the linux build process because they're necessary for linking the compiler.
But we're already shipping libgc.a and libcrystal.a and they're built separately on Debian, while the compiler itself is built and linked on Alpine.
So this relates to the question whether the generic linux distribution package should or could be truely portable across different linux systems.

[lh3]

Thanks for the explanation. This makes sense.

So this relates to the question whether the generic linux distribution package should or could be truely portable across different linux systems.

I am not an expert, but I believe the answer is yes. libevent.a and libpcre.a are as essential as libgc.a to crystal. Lacking the two libraries make the linux binary package non-functional.

[RX14]

To provide insight: The historical reason the packages have included libgc.a and not libevent and pcre, is that the latter two are a lot more common on linux systems - especially pcre, and are always available on package managers. For OSX this hasn't been as easy without homebrew.

One reason I have been reluctant to support packaging pcre and libevent is that they have a lot of CVEs (pcre 48, libevent 5 to bdwgc's 1). Dynamically linking to distribution versions of pcre especially would make a lot of sense because keeping PCRE up to date after CVEs were published would be as simple as keeping up to date with security updates on your distro, instead of downloading the latest crystal version, and rebuilding your binaries.

I understand that in data science, untrusted inputs aren't as much of a problem, and I do want to work to find a solution, but we have to be careful not to degrade security of those using the linux tarball to compile other types of applications.

[straight-shoota]

IIRC the only reason we distribute libgc.a is because there hasn't been a release of bdwgc in two years and the patch we need for multithreading support is not available in system packages.

[straight-shoota]

A good solution might be to provide two different packages. A minimal one with bare minimum of libraries (libgc and libcrystal as long as we need them), and a batteries-included package with all necessary libraries at least for core lib.
I would expect that the big package could easily build on the small, just adding in a few more lib files.

[lh3]

the latter two are a lot more common on linux systems

On CentOS, libevent and pcre are essential packages and widely available. However, compiling crystal programs require libevent-devel and pcre-devel, the development packages. Looking at the rpm content of pcre-devel, I see it just adds a symbolic link libpcre.so to the right version, but we need the development packages anyway.

A good solution might be to provide two different packages.

This sounds good to me.

[RX14]

I think two packages is the best solution, as long as they're documented and have the necessary warnings

[HertzDevil]

Just hit this recently because I was trying to get Crystal to work on Compiler Explorer.

PCRE reached end-of-life about a week ago, so if we distribute Crystal with PCRE 8.45 then it is always "up-to-date". Obviously it's a different story if we migrate to PCRE2 though.

If we build PCRE ourselves we also always get JIT support (#3852) regardless of the system library version.

[RX14]

development doesn't mean no security bugfixes i think. it is a positive though.

[orangeSi]

The libevent and pcre had packaged into crystal-xxx-linux-x86_64-bundled.tar.gz since crystal-1.2.0-1 in the https://github.com/crystal-lang/crystal/releases

$ls crystal-1.4.1-1/lib/crystal/lib
libevent.a  libevent_pthreads.a  libpcre.a

[noraj]

As the documentations says

Building fully statically linked executables is currently only supported on Alpine Linux.

For that purpose, the documentation recommends using musl-libc.

Official Docker Images based on Alpine Linux are available on Docker Hub at crystallang/crystal. The latest release is tagged as crystallang/crystal:latest-alpine.

As an example replace the local build:

$ crystal build src/miniss.cr --release --static -o bin/miniss
$ shards build --release --static

with

$ docker run --rm -it -v $PWD:/workspace -w /workspace crystallang/crystal:1.7.2-alpine \
    crystal build src/miniss.cr --release --static -o bin/miniss
$ docker run --rm -it -v $PWD:/workspace -w /workspace crystallang/crystal:1.7.2-alpine \
    shards build --release --static

Since you are cross-compiling from a docker container, it would be wiser to ensure to provide the correct cross compilation flags.

Find your platform with llvm-config --host-target on your host.

So change from

$ crystal build src/miniss.cr --release --static -o bin/miniss --cross-compile --target x86_64-pc-linux-gnu
$ cc bin/miniss.o -o bin/miniss  -rdynamic -static -L/home/noraj/.asdf/installs/crystal/1.7.2/bin/../lib/crystal -lpcre -lm -lgc -lpthread -levent  -lrt -lpthread -ldl

to

$ docker run --rm -it -v $PWD:/workspace -w /workspace crystallang/crystal:1.7.2-alpine \
    crystal build src/miniss.cr --release --static -o bin/miniss --cross-compile --target x86_64-pc-linux-gnu
$ docker run --rm -it -v $PWD:/workspace -w /workspace crystallang/crystal:1.7.2-alpine \
    cc bin/miniss.o -o bin/miniss  -rdynamic -static -L/usr/bin/../lib/crystal -lpcre -lm -lgc -lpthread -levent -lrt -lpthread -ldl

Finally, check your binary is fully static:

$ ldd bin/miniss
        not a dynamic executable

[Blacksmoke16]

@noraj Pretty sure this issue was related to using Crystal itself and not statically building some library/application. I.e. the case where you're installing Crystal via https://crystal-lang.org/install/from_targz/ and are unable to use a package manager or the ability to build the necessary libs from source.

[Sija]

@Blacksmoke16 doesn't seem like it... what makes you think that?

[Blacksmoke16]

@Sija Because the OP mentions Unfortunately, libevent and pcre are often missing from a system and installing/compiling them without root/sudo is non-trivial for inexperienced users. and the name of the issue is Shipping libevent.a and libpcre.a with the linux binary packages and it was solved via crystal-lang/distribution-scripts#102 which added the static files to the *-bundled.tar.gz release assets.

[Sija]

@Blacksmoke16 Ah, it seems I misread the last comments. Somehow I thought they're referring to the documentation and not OP's issue, thank you for correcting me 🙏

[noraj]

@Blacksmoke16 Hum 🤔 because I encountered the same issue as the author only when statically building (libevent and libpcre) while I had no issue when dynamically building and at runtime (with crystal both from archlinux system and asdf). Those libraries are also missing from archlinux and even when you are root and have pcre and pcre2 installed you still don't have libpcre.a (there is no package providing it). Unlike the author I had no issue for dynamic building and runtime even if my code is using some regexp. And since internet leads me here when looking for those errors I guess that it will help others.

[Blacksmoke16]

@noraj Right, this issue is NOT related to building your application statically, but instead using Crystal itself from the pre-built Crystal binaries provided via the *.tar.gz release assets on a system that doesn't have pcre installed. I.e. when your only way to use Crystal is to download/extract the archive on a user local level.

In your case, you probably installed Crystal via your package manager which installed all the runtime deps, such as pcre, along with Crystal itself. This explains why you are able to run your applications just fine. However, your comment is correct when you want to go and build a static binary for something you wrote when Crystal is already installed.

[noraj]

In your case, you probably installed Crystal via your package manager which installed all the runtime deps, such as pcre, along with Crystal itself. This explains why you are able to run your applications just fine. However, your comment is correct when you want to go and build a static binary for something you wrote when Crystal is already installed.

I said that I tried with ASDF as well (https://crystal-lang.org/install/from_asdf/) which which is on a user local level too. This is because asdf-crystal doesn't use the bundled tarball.

[Blacksmoke16]

Okay, but is the issue that when you do like crystal build --static app.cr it fails, or when you try to do crystal run app.cr it fails? This issue relates to the latter in the case Crystal was installed via the *.tar.gz archive.

I'm not so sure ASDF can use the bundled tarball since the pre-built static libs included in it would prob only work on linux?

EDIT: If you're running into an issue, the forums or one of that chat rooms might be a good place to get help.

[noraj]

Okay, but is the issue that when you do like crystal build --static app.cr it fails, or when you try to do crystal run app.cr it fails? This issue relates to the latter in the case Crystal was installed via the *.tar.gz archive.

In my case crystal build --release app.cr and crystal run app.cr doesn't fails but crystal build --release --static app.cr fails both with Archlinux Crystal and ASDF Crystal (and also with the bundled tarball Crystall because the static libraries are not in the proper path or something).

I'm not so sure ASDF can use the bundled tarball since the pre-built static libs included in it would prob only work on linux?

It's possible to make an if only for linux.

EDIT: If you're running into an issue, the forums or one of that chat rooms might be a good place to get help.

I don't need help, I found the solution with the Alpine docker container, since Crystal supports static build only on Alpine. But as it's not trivial to find the answer in the official document on the exact way to do it I thought it would be nice from me to share the solution.

Update: But yeah I understand the original issue is a bit different and it's maybe not the perfect place to share this solution but it's not off topic either as it has the same cause and the docker solution could be helpful for user local level too.

[Blacksmoke16]

In my case crystal build --release app.cr and crystal run app.cr doesn't fails but crystal build --release --static app.cr fails both with Archlinux Crystal and ASDF Crystal (and also with the bundled tarball Crystall because the static libraries are not in the proper path or something).

Okay yea, that's a separate issue than what this issue is about and is somewhat expected as, as you already pointed out, Alpine docker container is the easiest way to create a static binary of an application.

EDIT: Worth pointing out, it's not only supported on Alpine linux. It just so happens to be the easiest. My understanding is you can do it on any* system where you have access to that static *.a libs and pointing the CRYSTAL_LIBRARY_PATH env var at them.

*At least for Linux static builds as I know Mac is a bit diff/not supported at all.

But as it's not trivial to find the answer in the official document on the exact way to do it

https://crystal-lang.org/reference/1.7/guides/static_linking.html spells this all out quite well. If there's something there that you found missing or confusing, opening an issue on the book's repo https://github.com/crystal-lang/crystal-book would be more helpful.