Fix loading of libffi on OpenBSD

[zmyrgel]

Use short library name 'libffi.so' for OpenBSD. Libffi package might change the version number but the library should work without the full "libffi.so.0.0" suffix.

In addition, add compiler flag to include /usr/local/include to header search path. These are not added by default in OpenBSD. Otherwise libffi fails to find /usr/local/include/ffi.h.

[luismbo]

@liamh Any downside to adding libffi.so as a fallback on every unix? (Not just OpenBSD, as @zmyrgel suggests.)

[sionescu]

It should be safe, although for the future we should add support for pkg-config and use that on all *nix systems.

[liamh]

Read this:

http://lispcaveats.tumblr.com/post/13259176455/ffi-linking-against-shared-libraries

This might also apply to a recent change for OSX which hardwired a path
into the library spec.

On Thu, Feb 20, 2014 at 5:21 PM, Luís Oliveira notifications@github.comwrote:

@liamh https://github.com/liamh Any downside to adding libffi.so as a
fallback on every unix? (Not just OpenBSD, as @zmyrgelhttps://github.com/zmyrgelsuggests.)

Reply to this email directly or view it on GitHubhttps://github.com//pull/37#issuecomment-35676558
.

[zmyrgel]

You can read OpenBSD developers side of shared libraries from following links:
https://bugzilla.mozilla.org/show_bug.cgi?id=650772
http://www.openbsd.org/faq/ports/specialtopics.html#SharedLibs

[luismbo]

OK, so, as I understand it, given a libfoo.X.Y.so library, OpenBSD only creates the symlink libfoo.so. It doesn't symlink to libfoo.X.so like Linux distributions do.

@liamh Is your point is that a future version of libffi might not be compatible with cffi-libffi, thus we should specify the major version?

@sionescu using pkg-config seems like a good idea? If you have time/inclination, please open a launchpad wishlist item and expand on the advantages of doing that.

@zmyrgel Can you squash your commits and make it clear in the commit message (and/or a comment) that we're working around OpenBSD's decision not to create a libffi.so.X symlink?

[zmyrgel]

@luismbo OpenBSD doesn't use symlinks at all for shared libraries.

I think its best to put (:openbsd "libffi.so") with comment stating how openbsd linker handles loading it instead of keeping it at the end of :unix list.

Btw, how to squash my commits? I'm not that well versed in github use.

[luismbo]

On Fri, Feb 21, 2014 at 11:06 AM, Timo Myyrä notifications@github.com wrote:

@luismbo OpenBSD doesn't use symlinks at all for shared libraries.

Oh. So there isn't a "libffi.so" file? Is it the dynamic linker who's
mapping that to some libffi.X.Y.so?

I think its best to put (:openbsd "libffi.so") with comment stating how openbsd linker handles loading it instead of keeping it at the end of :unix list.

Agreed.

Btw, how to squash my commits? I'm not that well versed in github use.

git rebase -i <hash of the last commit that's not yours>

Then, when the interactive editor pops up, leave the pick instruction
on the first commit and use squash for the remaining commits.

(See for instance
http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)

[liamh]

Reading the OpenBSD links, it seems like we can't just specify the major
version as in Linux. However we can specify ".0" and it will pick
the highest minor version library with the same major version. This is the
right thing to do because of the possibility of a new incompatible version
of the libffi API, which would mean a new major version. I interpret
Anton's blog as saying that UNIXoids should follow this practice. While the
details differ between Linux and OpenBSD, the same effect is achievable in
a slightly different way.

On Fri, Feb 21, 2014 at 6:16 AM, Luís Oliveira notifications@github.comwrote:

On Fri, Feb 21, 2014 at 11:06 AM, Timo Myyrä notifications@github.com
wrote:

@luismbo OpenBSD doesn't use symlinks at all for shared libraries.

Oh. So there isn't a "libffi.so" file? Is it the dynamic linker who's
mapping that to some libffi.X.Y.so?

I think its best to put (:openbsd "libffi.so") with comment stating how
openbsd linker handles loading it instead of keeping it at the end of :unix
list.

Agreed.

Btw, how to squash my commits? I'm not that well versed in github use.

git rebase -i <hash of the last commit that's not yours>

Then, when the interactive editor pops up, leave the pick instruction
on the first commit and use squash for the remaining commits.

(See for instance
<
http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html>)

Reply to this email directly or view it on GitHubhttps://github.com//pull/37#issuecomment-35722156
.

[zmyrgel]

Actually you can specify libffi.so.0 and it will then find the latest minor for it.
You can also specify libffi.so.0.0 and then it will find library with that version without seeking anything. For details:
http://www.openbsd.org/cgi-bin/man.cgi?query=dlopen&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html

Using the versionless library loading would ease maintenance. If the library gets bumped it will most likely work by recompiling the cffi-libffi. Otherwise someone needs to keep an eye on the library and patch the cffi-libffi on each bump. And how many bumps should be listed in the cffi-libffi's list? The OpenBSD's version number might be bumped several times within single release or might keep stable for few releases.

[liamh]

If we have

(:unix (:or "libffi.so.6" "libffi32.so.6" "libffi.so.5"
"libffi32.so.5" "libffi.so"))

and no :openbsd clause, does this work for OpenBSD? (I'm assuming here that
absent an :openbsd clause, OpenBSD will follow the :unix clause.)

I think it desirable to have as few OS conditionalizations as possible in
order to enhance maintainability. This clause provides major versions as
the first choices, as good practice suggests, but provides the non-specific
version as a fallback. This is how our :windows line works. If this works
for OpenBSD and Linux, it is the start of consolidating all the Unix family
OSes into one clause.

[zmyrgel]

Above works but it would be better to have distict :openbsd line there.
Lets say in the near future the libffi package in OpenBSD gets updated few times and the main package which Linux uses stays the same. Like OpenBSD ports people add local patches to the port so it works on some new archs but haven't yet pushed changes to upstream.
Then, when I update the libffi package I get library like libffi.so.7.0 but I still have the old library present in the system with libffi.so.6.0. With the above line no matter what it would load the libffi.so.6.0 and not the newer libffi.so.7.0. If the old library is linked against other old stuff which might be removed at this point all compilation of cffi-libffi will fail.
With (:openbsd "libffi.so") line the compilation of cffi-libffi would always use the latest version and link against it.

[liamh]

On Sat, Feb 22, 2014 at 10:41 AM, Timo Myyrä notifications@github.comwrote:

Above works but it would be better to have distict :openbsd line there.
Lets say in the near future the libffi package in OpenBSD gets updated few
times and the main package which Linux uses stays the same. Like OpenBSD
ports people add local patches to the port so it works on some new archs
but haven't yet pushed changes to upstream.
Then, when I update the libffi package I get library like libffi.so.7.0
but I still have the old library present in the system with libffi.so.6.0.
With the above line no matter what it would load the libffi.so.6.0 and not
the newer libffi.so.7.0. If the old library is linked against other old
stuff which might be removed at this point all compilation of cffi-libffi
will fail.
With (:openbsd "libffi.so") line the compilation of cffi-libffi would
always use the latest version and link against it.

In that case the right thing to do is to add "libffi.so.7" at the head of
the :unix clause, everything continues to work as desired; wherever 7
doesn't exist yet, regardless of OS, it is just skipped and 6 is chosen.
Much like what happened recently with the 5->6 transition; just among Linux
distros, the various distros got 6 at different times. The OS involved is a
red herring, and it is a significant advantage to have as few OS clauses as
possible.

Reply to this email directly or view it on GitHubhttps://github.com//pull/37#issuecomment-35805500
.

[zmyrgel]

Point I was trying to make is that OpenBSD lib version can conflict with the libffi package's version.
For example currently the version is at 0.0 but the actual library version is 5.9 I think. If you try to put both version numbers to same list I believe at some point they will cause problems.
Like are the libraries before 5 supported? Once the OpenBSD lib is updated it will go to 1.0, 2.0, 3.0 etc and Linux is already at 5 or 6.

[luismbo]

I wasn't aware that the X in libfoo.so.X doesn't match the library's actual major version. That's unfortunate.

Well, if we want to be strict about which versions of libffi we can load, then we have to maintain separate version lists for each OS. But if we don't, then using "libffi.so" should work for both Linux and OpenBSD.

Usually we have to specify major versions on Linux since only dev packages install the versionless libfoo.so symlink, but we need to grovel libffi's headers anyway.

So, @liamh why don't we just simplify all this to a simple (:unix "libffi.so")?

[liamh]

Because major version numbers are an inherent part of the library
specification in the Unix family and are considered an important advantage
over the Windows style in which API changes are a major hassle, but
specifically, Linux does not define a library without a major version
number except for "development" packages. See Anton's blog I referenced at
the beginning. As you say, we need the development version because we have
to parse the header file, but I think it would be smart to adhere to the
version standard and then provide libffi.so at the end of the list for OSes
like OpenBSD which do not adhere to the correct version numbers.

Don't overlook the fact that cffi-libffi provides an example for users of
CFFI, because this define-foreign-library form is actually part of CFFI.
People will try to duplicate the behavior on Linux with libraries in
distribution packages that are not -dev, and fail, and wonder why. Then
they'll soft link the .so.n to .so, which is not a good idea. I speak with
authority on this, because that's what I used to do.

On Sat, Feb 22, 2014 at 5:22 PM, Luís Oliveira notifications@github.comwrote:

I wasn't aware that the X in libfoo.so.X doesn't match the library's
actual major version. That's unfortunate.

Well, if we want to be strict about which versions of libffi we can load,
then we have to maintain separate version lists for each OS. But if we
don't, then using "libffi.so" should work for both Linux and OpenBSD.

Usually we have to specify major versions on Linux since only dev packages
install the versionless libfoo.so symlink, but we need to grovel libffi's
headers anyway.

So, @liamh https://github.com/liamh why don't we just simplify all this
to a simple (:unix "libffi.so")?

Reply to this email directly or view it on GitHubhttps://github.com//pull/37#issuecomment-35816918
.

[luismbo]

@liamh adding "libffi.so" at the end obliviates the whole point of specifying library versions for the sake of ensuring binary compatibility. I'm sympathetic to your point of setting a good example, but I don't think that'd be a good example.

So, either we maintain two different sets of versions for OpenBSD and Linux (and possibly others?) or we just use "libffi.so".

I'm currently inclined towards the latter option.

[zmyrgel]

I'd say best to just add '(:openbsd "libffi.so")' to the list as OpenBSD library versions are different from the more commonly used versioning. Keeping them both in same list might cause issues at some point.

To be exact, OpenBSD doesn't use any "development" packages like Linux but a single package. And OpenBSD doesn't use any symlinks in shared libraries. There is no symlink or file named "libffi.so" in there. Thats just shorthand for the linker to load the shared library with highest major.minor version.

[liamh]

Cherry-picked onto v0.14.0 as cc07246.