Failure when loaded in Alpine Linux with musl-dev #106

ronaldtse · 2022-03-22T13:15:51Z

In Alpine, the musl-dev library is used instead of glibc. Somehow, the platform gem "x86_64-linux" is installed for expressir.

$ docker run --platform=linux/amd64 -it ruby:3.1-alpine sh
/ # apk add gcc g++ cmake make libc-dev
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
(1/25) Installing libacl (2.2.53-r0)
(2/25) Installing libbz2 (1.0.8-r1)
(3/25) Installing expat (2.4.7-r0)
(4/25) Installing lz4-libs (1.9.3-r1)
(5/25) Installing xz-libs (5.2.5-r0)
(6/25) Installing zstd-libs (1.5.0-r0)
(7/25) Installing libarchive (3.6.0-r0)
(8/25) Installing brotli-libs (1.0.9-r5)
(9/25) Installing nghttp2-libs (1.46.0-r0)
(10/25) Installing libcurl (7.80.0-r0)
(11/25) Installing rhash-libs (1.4.2-r2)
(12/25) Installing libuv (1.42.0-r0)
(13/25) Installing cmake (3.21.3-r0)
(14/25) Installing binutils (2.37-r3)
(15/25) Installing libgomp (10.3.1_git20211027-r0)
(16/25) Installing libatomic (10.3.1_git20211027-r0)
(17/25) Installing libgphobos (10.3.1_git20211027-r0)
(18/25) Installing isl22 (0.22-r0)
(19/25) Installing mpfr4 (4.1.0-r0)
(20/25) Installing mpc1 (1.2.1-r0)
(21/25) Installing gcc (10.3.1_git20211027-r0)
(22/25) Installing musl-dev (1.2.2-r7)
(23/25) Installing libc-dev (0.7.2-r3)
(24/25) Installing g++ (10.3.1_git20211027-r0)
(25/25) Installing make (4.3-r0)
Executing busybox-1.34.1-r4.trigger
OK: 256 MiB in 60 packages
/ # gem install expressir
Fetching expressir-1.2.1-x86_64-linux.gem
Fetching thor-1.2.1.gem
Successfully installed thor-1.2.1
Successfully installed expressir-1.2.1-x86_64-linux
2 gems installed
/ # irb
irb(main):001:0> require 'expressir'
=> true
irb(main):002:0> require 'expressir/express/parser'
/usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/parser.rb:5:in `require_relative': cannot load such file -- /usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/express_parser (LoadError)
        from /usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/parser.rb:5:in `rescue in <top (required)>'
        from /usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/parser.rb:1:in `<top (required)>'
        from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from (irb):3:in `<main>'                                            
        from /usr/local/lib/ruby/gems/3.1.0/gems/irb-1.4.1/exe/irb:11:in `<top (required)>'
        from /usr/local/bin/irb:25:in `load'                                
        from /usr/local/bin/irb:25:in `<main>'                              
/usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/parser.rb:3:in `require_relative': Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/3.1/express_parser.so) - /usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/3.1/express_parser.so (LoadError)
	from /usr/local/bundle/gems/expressir-1.2.1-x86_64-linux/lib/expressir/express/parser.rb:3:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from (irb):3:in `<main>'
	from /usr/local/lib/ruby/gems/3.1.0/gems/irb-1.4.1/exe/irb:11:in `<top (required)>'
	from /usr/local/bin/irb:25:in `load'
	from /usr/local/bin/irb:25:in `<main>'

When I install on aarch64, it correctly detects that I am on a different platform and therefore does not install the platform gem and compiles properly.

$ docker run --platform=linux/arm64/v8 -it ruby:3.1-alpine sh
/ # apk add gcc g++ cmake make libc-dev
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/aarch64/APKINDEX.tar.gz
(1/25) Installing libacl (2.2.53-r0)
(2/25) Installing libbz2 (1.0.8-r1)
(3/25) Installing expat (2.4.7-r0)
(4/25) Installing lz4-libs (1.9.3-r1)
(5/25) Installing xz-libs (5.2.5-r0)
(6/25) Installing zstd-libs (1.5.0-r0)
(7/25) Installing libarchive (3.6.0-r0)
(8/25) Installing brotli-libs (1.0.9-r5)
(9/25) Installing nghttp2-libs (1.46.0-r0)
(10/25) Installing libcurl (7.80.0-r0)
(11/25) Installing rhash-libs (1.4.2-r2)
(12/25) Installing libuv (1.42.0-r0)
(13/25) Installing cmake (3.21.3-r0)
(14/25) Installing binutils (2.37-r3)
(15/25) Installing libgomp (10.3.1_git20211027-r0)
(16/25) Installing libatomic (10.3.1_git20211027-r0)
(17/25) Installing libgphobos (10.3.1_git20211027-r0)
(18/25) Installing isl22 (0.22-r0)
(19/25) Installing mpfr4 (4.1.0-r0)
(20/25) Installing mpc1 (1.2.1-r0)
(21/25) Installing gcc (10.3.1_git20211027-r0)
(22/25) Installing musl-dev (1.2.2-r7)
(23/25) Installing libc-dev (0.7.2-r3)
(24/25) Installing g++ (10.3.1_git20211027-r0)
(25/25) Installing make (4.3-r0)
Executing busybox-1.34.1-r4.trigger
OK: 250 MiB in 60 packages
/ # gem install expressir
Fetching rice-4.0.3.gem
Fetching expressir-1.2.1.gem
Fetching thor-1.2.1.gem
Successfully installed thor-1.2.1
Successfully installed rice-4.0.3
Building native extensions. This could take a while...
Successfully installed expressir-1.2.1
3 gems installed
/ # irb
irb(main):001:0> require 'expressir'
=> true
irb(main):002:0> require 'expressir/express/parser'
=> true
irb(main):003:0>

ronaldtse · 2022-03-22T13:28:00Z

The problem is also related to this: metanorma/emf2svg-ruby#17

alexeymorozov · 2022-03-25T18:18:24Z

$ gem environment | grep -A 2 "RUBYGEMS PLATFORMS"
  - RUBYGEMS PLATFORMS:
     - ruby
     - x86_64-linux-musl

Platform is detected as x86_64-linux-musl.

$ gem help platforms
RubyGems matches platforms as follows:

  * The CPU must match exactly unless one of the platforms has
    "universal" as the CPU or the local CPU starts with "arm" and the gem's
    CPU is exactly "arm" (for gems that support generic ARM architecture).
  * The OS must match exactly.
  * The versions must match exactly unless one of the versions is nil.

So it matches x86_64-linux because the version part is nil.

The next step is to create a build for x86_64-linux-musl, or rename x86_64-linux to a more specific build.

ronaldtse · 2022-03-26T02:59:52Z

The mismatch between glibc and musl platforms has been subject of work at rubygems:

And that it is a known issue on Alpine that Rubygems can mistakenly install glibc platform gems on Alpine, and therefore they maintain such a patch to only install platform=ruby gems:

I reviewed the code in rubygems/rubygems#4082 and apparently there is this rather undocumented differentiation of platforms:

https://github.com/lloeki/rubygems/blob/2743af107e1d542a3cece421ead752e1b8141f93/test/rubygems/test_gem_platform.rb#L136-L139

      'x86_64-linux'           => ['x86_64',    'linux',     nil],
      'x86_64-linux-gnu'       => ['x86_64',    'linux',     nil],
      'x86_64-linux-musl'      => ['x86_64',    'linux',     'musl'],
      'x86_64-linux-uclibc'    => ['x86_64',    'linux',     'uclibc'],

According to this PR, technically, a platform gem that specifies x86_64-linux should not be installed on x86_64-linux-musl

However, in the latest master branch of Rubygems, this differentiation has been removed (this PR was reverted in rubygems/rubygems#4434).

https://github.com/lloeki/rubygems/blob/8a712a0dd52e661ea973845b70681da09ce75edd/test/rubygems/test_gem_platform.rb#L136-L137

      'x86_64-linux'           => ['x86_64',    'linux',     nil],
      'x86_64-linux-musl'      => ['x86_64',    'linux',     'musl'],

And the behavior at rubygems is now just broken:

Backport non gnu libc linux support from RubyGems rubygems/rubygems#4488

The way we have to solve this is either:

The x86_64-linux platform gem works natively on musl. I don't know if our source code can be fixed that way (i.e. avoiding GNU extensions)
We publish platform gems for x86_64-linux-musl so that on Alpine the x86_64-linux-musl is always used.

maxirmx · 2022-03-26T08:18:47Z

"/lib/ld-linux-x86-64.so.2" -- is a dynamic loader.
IMHO it is worth trying compatibility package: https://pkgs.alpinelinux.org/contents?file=ld-linux-x86-64*

expressir is using rake-compiler-dock that supports:
- platform: x86-mingw32
- platform: x64-mingw-ucrt
- platform: x64-mingw32
- platform: x86-linux
- platform: x86_64-linux
- platform: x86_64-darwin
- platform: arm64-darwin
- platform: arm-linux
- platform: aarch64-linux
- platform: jruby

Alpine will need to be added to this list

ronaldtse · 2022-03-26T09:14:11Z

The compatibility package gcompat worked:

/ # apk add gcc g++ cmake make libc-dev
...
/ # apk add gcompat
(1/3) Installing musl-obstack (1.2.2-r0)
(2/3) Installing libucontext (1.1-r0)
(3/3) Installing gcompat (1.0.0-r4)
OK: 256 MiB in 64 packages
/ # irb
irb(main):001:0> require 'expressir'
=> true
irb(main):002:0> require 'expressir/express/parser'
=> true

@maxirmx Maybe we should make a PR to rake-compiler-block?

maxirmx · 2022-03-26T09:43:43Z

@maxirmx Maybe we should make a PR to rake-compiler-block?

@ronaldtse, we can if ruby really support differentiation but I am afraid that it won't be straightforwards either.

maxirmx · 2022-04-04T21:35:25Z

This (or closely related) issue is a blocker for:
metanorma/packed-mn#158

Whe packaged with tebako in linux-musl environment expressir loads x86_64-linux extension which requires /lib/ld-linux-x86-64.so
It is worth to note that this issue occurs only for expressir and libemf2svg (that inherits platform logic from expressir) All other gems with native extensions work

ronaldtse · 2022-04-05T01:42:41Z

It is worth to note that this issue occurs only for expressir and libemf2svg (that inherits platform logic from expressir) All other gems with native extensions work

Then if we update the platform logic, so it does not treat it as x86_64-linux, it should work?

maxirmx · 2022-04-05T07:38:16Z

Then if we update the platform logic, so it does not treat it as x86_64-linux, it should work?

I do not understand why other gems work. Probably it shall be possible to implement platform logic as others do it

maxirmx · 2022-05-30T18:45:12Z

Well, so far it looks like a mistery

nokogiri gem taht uses the same build system installs and runs on Alpine using linux prebuild native gem.
The authour and maintainer of nokogiri is sure that gcompat is a prerequisite, but it is not true.

maxirmx · 2022-05-30T18:47:09Z

Expressir looks different for ldd. The reason is yet unknown

maxirmx · 2022-06-03T14:38:09Z

Summary of research re Alpine (musl) support

Since the effort to support x86_64-linux-musl platform looks stalled (Backport non gnu libc linux support from RubyGems rubygems/rubygems#4488) I was looking at options to make the x86_64-linux platform gems (expressir and others) work natively on musl
Initial view was to identify gnu extensions in express_parser and either replace them with std equivalents or neutralize by linking glibc statically. However, I found that the show-stopper is related to functionality defined by standard. Any C/C++ static variable results in the usage of thread local storage (tls); tls implementation is provided by dynamic linker; dynamic linkers are not compatible between gnu and musl.
Rice implementation is heavily dependant on static variables that it creates. I do not see any way to change it.

The bottom line. For gems that use Rice there is no way to x86_64-linux platform gems to work on musl unless gcompat package is installed

maxirmx · 2022-06-03T14:46:23Z

What can be done

We make take over rubygems platform effort (Backport non gnu libc linux support from RubyGems rubygems/rubygems#4488) and attempt to drive it to completion. This is the 'right' approach but it may last forever
For expressir and similar gems we can deploy musl binaries inside linux gem and implement a hack that will use these libaries if on Alpine. This is the risky approach, result cannot be guaranteed.
We can make gcompat or force_ruby_platform a requirement for Alpine installations. I can probably implement reasonable error message if this requirement is not met. This is 'fast' approach, though packed_mn may require additional effort.

ronaldtse · 2022-06-04T01:28:20Z

@maxirmx thank you for the extensive investigation!

We can make gcompat or force_ruby_platform a requirement for Alpine installations. I can probably implement reasonable error message if this requirement is not met. This is 'fast' approach, though packed_mn may require additional effort.

This is reasonable for packed-mn and our gems. We should take this approach.

We make take over rubygems platform effort (Support non gnu libc linux (part 2) rubygems/rubygems#4488) and attempt to drive it to completion. This is the 'right' approach but it may last forever

This is risky for us because we do not know when the Ruby team is interested enough in musl. This PR has implications in the Rubygems system because other there is a backwards compatibility concern too, i.e. all currently available platform gems are assigned to 'glibc' to differentiate from 'musl'.

For expressir and similar gems we can deploy musl binaries inside linux gem and implement a hack that will use these libaries if on Alpine. This is the risky approach, result cannot be guaranteed.

I think this is also a reasonable approach, and it should be doable, but it will bloat the gems, and there are not that many people who use Alpine anyway. Prefer using the gcompat requirement.

maxirmx · 2022-06-04T19:12:41Z

I will do more tests with gcompat and/or packed_mn to be sure that gcompat is solid approach

maxirmx · 2022-06-09T19:59:56Z

Update
Alpine test workflow: https://github.com/lutaml/expressir/actions/workflows/alpine.yml

force_ruby_platform works, of course
gcompat does not work. It is good enough to let extension load; however, the tests do not pass with blind exception

RuntimeError:
        No error information
      # ./lib/expressir/express/parser.rb:35:in `initialize'
      # ./lib/expressir/express/parser.rb:35:in `new'
      # ./lib/expressir/express/parser.rb:35:in `from_file'
      # ./spec/expressir/model/model_element_spec.rb:174:in `block (3 levels) in <top (required)>'
      # ./spec/spec_helper.rb:19:in `block (2 levels) in <top (required)>'

I found that this exception is related to std::call_once, i.e.: thread local memory again.

force_ruby_platform helped to create packed_mn Alpine package but it will be painful for any user to take this approach. This setting cannot be applied to a single gem so we actually force to build everything in the bundle even if there are precompiled extensions out there.

gcompat issue can be debugged further but neither time nor effort can be predicted.

Attn @ronaldtse

ronaldtse · 2022-06-13T17:18:43Z

@maxirmx what is the best course ahead for now? Is Alpine a lost cause?

maxirmx · 2022-06-13T20:04:03Z

@ronaldtse
Considering Expressir native extension gor musl I can imagine three options:

fix bundler (Backport non gnu libc linux support from RubyGems rubygems/rubygems#4488)
hack pandler. For example, develop bundler plugin that will patch bundler on the go. Not sure if it is possible though
debug run-time exception with gcompat. In order to do it some minimal example is required. Expressir extension is really big and virtually impossible to use it for debugging. This effor may end up with something simple, like compilation flag, but can also lead to gcompat patch.

If you ask about packed-mn, the first version is ready as far as compilation and packaging is considered (metanorma/packed-mn#158 (comment)) There is run-time issue with fontist gem that needs to be fixed

So the real question is if and where you are ready to invest

ronaldtse · 2022-08-23T00:33:34Z

Rubygems now supports differentiation of platform between glibc and musl: Support non gnu libc linux platforms rubygems/rubygems#5852 , which works from 3.1.0.pre.1 onwards
We wait for the merging of Backport non gnu libc linux support from RubyGems rubygems/rubygems#4488 for bundler to work
Then we can re-do this task of publishing the gem on the musl platforms

- MSys packaging for gems without native extensions (partial implementation #27) - Ruby version upgrade to 2.7.7 - Bundler version upgrade to 2.3.22 -- we can address lutaml/expressir#106 now - Packaging optimization - Ruby extensions are not reconfigured on each tabako execution anymore (adresses #91)

ronaldtse · 2023-09-01T03:10:57Z

@maxirmx is this issue to be closed? Thanks!

maxirmx · 2023-09-01T06:01:12Z

@ronaldtse

Actually there are (there were) two related issues:

There was an issue with bundler which ignored specialization and installed x86_64-linux extension on Alpine even when x86_64-linux-musl was present. This issue was fixed in bundler.
There are some gems which provide gnu version of extension and name it x86_64-linux In this case it gets installed on Alpine and fails for some gems.

As a solution for both problems we do not use precompiled native extensions when Alpine version of tebako package is created. If we consider metanorma the issue was nokogiri. The maintainers claimed that its x64-linux extension worked for Alpine but in fact it did not. At least not for me.

So the answers is

We can publish x64-linux-musl native extension (now we do not have it published). It will close this issue.
We can revisit metanorma packaging on alpine to see if others also resolved there issues on Alpine. If they did it we can use prebuilt native extensions. will help to reduce Alpine package size and packaging time.

ronaldtse · 2023-09-02T11:58:08Z

@maxirmx in this case, could you please help schedule 1 and 2 after the recent tebako work? Thanks!

maxirmx · 2023-09-05T20:26:01Z

Packaging of expressir gem is implemented with rack-compiler-dock
I suggest we wait until rake-compiler/rake-compiler-dock#75 is completed

ronaldtse changed the title ~~Failure when loaded in Alpine Linux with musc-dev~~ Failure when loaded in Alpine Linux with musl-dev Mar 22, 2022

ronaldtse assigned alexeymorozov Mar 22, 2022

ronaldtse added the bug Something isn't working label Mar 22, 2022

ronaldtse mentioned this issue Mar 22, 2022

Add Alpine version of container (linux/amd64) metanorma/metanorma-docker#143

Merged

alexeymorozov mentioned this issue Mar 25, 2022

WIP: aarch64-linux fontist/ffi-libarchive-binary#13

Merged

ronaldtse mentioned this issue Mar 26, 2022

Backport non gnu libc linux support from RubyGems rubygems/rubygems#4488

Merged

4 tasks

maxirmx mentioned this issue Apr 4, 2022

Binary distribution does not work for packed-mn metanorma/emf2svg-ruby#24

Closed

maxirmx self-assigned this May 29, 2022

maxirmx mentioned this issue Jun 2, 2022

Investigate size of express_parser.so for Ruby 3.0, 3.1 #122

Closed

maxirmx mentioned this issue Jun 10, 2022

Add Alpine linux build for packed-mn metanorma/packed-mn#158

Closed

maxirmx mentioned this issue Jun 23, 2022

Alpine workflow backlog metanorma/packed-mn#169

Open

ronaldtse unassigned alexeymorozov Aug 23, 2022

ronaldtse added the enhancement New feature or request label Aug 23, 2022

maxirmx mentioned this issue Feb 12, 2023

Partial MSys port and other enhancements tamatebako/tebako#83

Merged

maxirmx mentioned this issue Sep 3, 2023

Revisit Alpine packaging metanorma/packed-mn#212

Closed

maxirmx linked a pull request Dec 16, 2023 that will close this issue

Use (x86_64/aarch64)-linux-gnu target platform for native extensions instead of (x86_64/aarch64)-linux #138

Merged

ronaldtse closed this as completed in #138 Dec 17, 2023

maxirmx mentioned this issue Dec 21, 2023

bundle install is crashing on Ubuntu with Ruby 2.7 #141

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failure when loaded in Alpine Linux with musl-dev #106

Failure when loaded in Alpine Linux with musl-dev #106

ronaldtse commented Mar 22, 2022 •

edited

Loading

ronaldtse commented Mar 22, 2022

alexeymorozov commented Mar 25, 2022

ronaldtse commented Mar 26, 2022

maxirmx commented Mar 26, 2022

ronaldtse commented Mar 26, 2022

maxirmx commented Mar 26, 2022

maxirmx commented Apr 4, 2022

ronaldtse commented Apr 5, 2022

maxirmx commented Apr 5, 2022 •

edited

Loading

maxirmx commented May 30, 2022

maxirmx commented May 30, 2022

maxirmx commented Jun 3, 2022 •

edited

Loading

maxirmx commented Jun 3, 2022

ronaldtse commented Jun 4, 2022

maxirmx commented Jun 4, 2022

maxirmx commented Jun 9, 2022 •

edited

Loading

ronaldtse commented Jun 13, 2022

maxirmx commented Jun 13, 2022

ronaldtse commented Aug 23, 2022

ronaldtse commented Sep 1, 2023

maxirmx commented Sep 1, 2023

ronaldtse commented Sep 2, 2023

maxirmx commented Sep 5, 2023

Failure when loaded in Alpine Linux with musl-dev #106

Failure when loaded in Alpine Linux with musl-dev #106

Comments

ronaldtse commented Mar 22, 2022 • edited Loading

ronaldtse commented Mar 22, 2022

alexeymorozov commented Mar 25, 2022

ronaldtse commented Mar 26, 2022

maxirmx commented Mar 26, 2022

ronaldtse commented Mar 26, 2022

maxirmx commented Mar 26, 2022

maxirmx commented Apr 4, 2022

ronaldtse commented Apr 5, 2022

maxirmx commented Apr 5, 2022 • edited Loading

maxirmx commented May 30, 2022

maxirmx commented May 30, 2022

maxirmx commented Jun 3, 2022 • edited Loading

maxirmx commented Jun 3, 2022

ronaldtse commented Jun 4, 2022

maxirmx commented Jun 4, 2022

maxirmx commented Jun 9, 2022 • edited Loading

ronaldtse commented Jun 13, 2022

maxirmx commented Jun 13, 2022

ronaldtse commented Aug 23, 2022

ronaldtse commented Sep 1, 2023

maxirmx commented Sep 1, 2023

ronaldtse commented Sep 2, 2023

maxirmx commented Sep 5, 2023

ronaldtse commented Mar 22, 2022 •

edited

Loading

maxirmx commented Apr 5, 2022 •

edited

Loading

maxirmx commented Jun 3, 2022 •

edited

Loading

maxirmx commented Jun 9, 2022 •

edited

Loading