[Proposal] Improved support for binary gems

[segiddins]

I published a post outlining some goals for improved precompiled gem support that has received some feedback.

The high-level goal is to introduce support for something like python's "wheel", where gems are able to target a very specific "platform" such that they can contain a single precompiled binary that runs on that platform (as opposed to the current state, where e.g. nokogiri ships a arm64-darwin gem version that contains separate binaries for ruby 3.1, 3.2, 3.3, 3.4).

I wanted to create this discussion as a place where folks can leave their feedback, with an emphasis on proposing alternatives to the different parts.

[kou]

Thanks for creating this.

Let's use one thread per topic for easy to discuss.

[kou]

@segiddins It seems that there are no more topics to be discussed for now. Could you share your opinions for existing 3 topics?

[segiddins]

Thank you for the feedback everyone!
I appreciate all the concerns that have been surfaced here. Overall, my feeling is that the "wheels" proposal does not really create new concerns here that don't already exist in the current situation for precompiled gems, while bringing several benefits that I personally think are worthwhile.

The overarching goal of this project is to improve the general state of things surrounding gems with native extensions. That means security, ergonomics, developer experience, performance, etc., with the understanding that a system that we all like working with is better for everyone and provides us more leverage to have "secure by default" options where possible.

A big part of this work, in addition to supporting the extended identifiers for platform selection, will be improving the tooling around native extensions. By having a workflow that "just works" that folks can adopt, we can make it easy for gem maintainers to ship gems that are easy to install for users, have source provenance, come from trusted (aka SLSA compliant) workflows, and support the most popular platforms that gem consumers are using. Additionally, we want to lower the barrier to shipping precompiled gems, as they have proven wildly popular with users overall (a huge thanks to @flavorjones for everything he's done in this space).

Finally, this will be an opportunity for the rubygems team to help build out parts of this toolchain, and continue maintaining it as a critical part of the ruby packaging ecosystem moving forwards.

[kou]

Thanks.
I created a developer experience topic from your comment: https://github.com/orgs/rubygems/discussions/8645#discussioncomment-13243371

[mghaught]

@kou I appreciate that you brought this feedback to us.

Since I've stepped into the director role, I've gone deeper into security than previously. One aspect is that I'm participating in several working groups across language ecosystems. A common theme is learning what has worked for another ecosystem and thinking about how that could apply to the Ruby ecosystem.

In particular, advancing supply chain security is one of our top priorities. We're leveraging the experiences from these other ecosystems and we're receiving funding to implement them. In this particular case, the precompiled binary project is our big focus this year. We appreciate the work Samuel is doing here to move our community forward.

We are grateful for the feedback and want this to proceed with involvement from our community. Many of you are deeper in one aspect or another in how you develop gems so it's a gift when you bring our awareness to concerns there.

It's easy to miss that we haven't shared a specific detail. Heck, I've thought that we should refine our 'how to work in the open' approach with significant changes like what we're exploring here.

Kudos to Sam for putting his work out there so early on. This will be a big step forward for RubyGems. Much gratitude to all of you for being willing to bring your concerns to us. I’d like to know how else I can assist with these efforts. Please feel free to contact me directly if you’d like us to adjust how we roll projects like this out.

[kou]

@mghaught Thanks for sharing your thoughts.

Currently, I think that existing threads in this discussion are stopped by waiting for Samuel's comments. How to proceed from this?

[kou]

Security about binaries built by developers.

https://github.com/orgs/rubygems/discussions/8575

the gems are build on developer machines, I have zero trust in the delivered binaries.

[voxik]

The developers might not be evil, they system might be just compromised. I have bigger trust in RubyGems infra, OTOH if there is some issue, it might have even bigger impact. But we all have some level of trust to RubyGems infra anyway, so this would not be new concern per-se

[segiddins]

I've discussed the possibility of using RubyGems-owned infrastructure to build arbitrary gems, but this is not something that will be feasible for us to implement for the time being. That being said, the status quo is that gems with precompiled binaries are already built on developer machines, so this is no worse, and we plan to release tooling that helps developers to build & publish their gem matrix with trusted provenance, ideally with an SLSA level 3 compliant builder workflow

[simi]

@voxik do you have trust in for example GitHub Actions environment (or any other CD) with shared build provenance? IMHO that's best and independent solution.

[kou]

@segiddins Can we use GitHub Actions for the "RubyGems-owned infrastructure"? (It seems that the "builder workflow" is a GitHub Actions workflow.) If we can, I can help for preparing it.

BTW, can we clarify the security part of this proposal?

That being said, the status quo is that gems with precompiled binaries are already built on developer machines, so this is no worse

It means that precompiled binaries aren't safer than normal gems, right?

we plan to release tooling that helps developers to build & publish their gem matrix with trusted provenance, ideally with an SLSA level 3 compliant builder workflow

This is the important part for security in this proposal, right?

[voxik]

If there was some standard GH action to build binary gems, that could work 👍

[kou]

Binary gem maintainers need to synchronize with CRuby releases.

NOTE: This is not a new problem with the proposal. The current binary gem mechanism also has this problem.

CRuby will be released every Christmas. If binary gem maintainers don't release binary gems that support the new CRuby release soon, users can't use the new CRuby release.

It's (high?) maintenance cost for binary gem maintainer.

It's inconvenient for CRuby users.

[kou]

As you mention, this is an existing painpoint. I believe this proposal improves the situation, as it becomes possible to publish a new artifact that's targeted specifically for the new ruby release.

As a gem developer perspective, this proposal doesn't improve the situation... If precompiled gem mechanism is improved (as a user perspective) without solving existing painpoints, users request precompiled gem to gem developers more. Gem developers will need more maintenance costs. I'm afraid of decreasing gem developers by more maintenance costs. Decreasing gem developers isn't a good situation in Ruby ecosystem. So this proposal without solving existing painpoint doesn't improve the situation.

In addition, users would automatically pick up the source distribution when running a newer ruby than has precompiled binaries published for it.

As a gem user perspective, I agree with this.

But I already mentioned in https://github.com/orgs/rubygems/discussions/8645#discussioncomment-12917093 , building the picked source distribution will be failed because a user environment doesn't have enough tools to build the source distribution.

For what its worth, in talking with friends in the python community, it seems to be very rare that people go back and publish old versions of their packages with a new wheel when a new python is released.

Can we clarify this?

Pre-conditions:

• Python package A exists
• A has 1.0.0 and 1.0.1 releases
• 1.0.0 has Python 3.9-3.12 wheels
• 1.0.1 has Python 3.9-3.12 wheels too

If Python 3.13 is released, A releases 1.0.2 that includes Python 3.9-3.13 wheels and Python 3.13 wheels for 1.0.0 and 1.0.1 aren't released, right?

Or A releases only Python 3.13 wheel for 1.0.1 because 1.0.1 is the latest release and 1.0.0 isn't the latest release?

(I think that the former is correct.)

I think that we can do better than wheels in Python :-)

This is exactly a painpoint focused in this thread. It's inconvenient for gem developers. If RubyGems builds and publishes precompiled gems for new Ruby from normal gems, gem developers don't need to release a precompiled gem for new Ruby by themselves. It's my proposal for this painpoint and I think that this proposal (adding a Ruby version to tag) matches my proposal.

[kou]

I was thinking about like self-efficient CI build action. Like GitHub Action being notified new Ruby was released (probably not simply possible, but could be possible with help of some RubyGems.org publicly available automation), run tests, build and publish new version.

Interesting. Is the CI build action executed in repositories of each gem or RubyGems-owned repository?

I think that we can implement notification on new Ruby release. We can collaborate with Ruby core team. I can help it as a member of Ruby core team.

[simi]

@kou currently it could be done by using API access to repo and workflow_dispatch type of workflow. That way anyone connecting gem repo to RubyGems.org small (non-existing currently) app can get build started when new Ruby is released.

[kou]

Thanks. I understand that the CI build action executed in a repository of each gem.

[kou]

But I already mentioned in https://github.com/orgs/rubygems/discussions/8645#discussioncomment-12917093 , building the picked source distribution will be failed because a user environment doesn't have enough tools to build the source distribution.

FYI: There is a pandas 2.3.0 example: pandas-dev/pandas#61564

pandas 2.3.0 didn't provide Python 3.9 wheels (FYI: they are provided now) and pip install pandas picked the source distribution automatically as expected. But it's failed because a user didn't have build environment (Visual Studio).

[kou]

Binary bindings gem maintainers need to update bundled external libraries when any vulnerabilities are found in external libraries. (e.g. libxml2 in Nokogiri case.)

NOTE: This is not a new problem with the proposal. The current binary gem mechanism also has this problem.

Because, in general, binary bindings gem includes depended libraries.

It's high maintenance cost for binary bindings gem maintainers because nobody knows when a vulnerability is found.

If updated binary bindings gems aren't released immediately, users will be at risk.

[kou]

This idea will not provide benefits without packaging system integration idea...

Or... if another libxml2 bindings including bindings for other languages such as lxml for Python have similar patches, this idea may provide benefits without packaging system integration idea.

[segiddins]

I appreciate this concern, but I don't think it's exactly what we're trying to address with this work. C libraries bundled into a gem in reality are no different from the source of the gem, in terms of the risks & responsibilities they entail. Part of depending on nokogiri is trusting that there will be security updates as needed, and that goes for whether the vulnerable code in question comes from libxml or the C code in nokogiri, or the ruby code in nokogiri for that matter.

[kou]

Ah, sorry. I should have mentioned that my base concern explicitly. As I mentioned in https://github.com/orgs/rubygems/discussions/8645#discussioncomment-13243151 , my concern is that implement this proposal without solving existing painpoints may decrease gem developers by increased gem maintenance cost.

Can we implement this proposal after we solve existing precompiled gem painpoints?

[segiddins]

Can you please start a new thread in this discussion listing the pain points you'd like to see solved, and explain why they need to be solved first before this proposal moves forward?

[kou]

OK. I've started a new thread: https://github.com/orgs/rubygems/discussions/8645#discussioncomment-13293536

[kou]

Developer experience may not be improved by this proposal.

Let's use a separated topic for developer experience. Because this proposal focus on developer experience too.

https://github.com/orgs/rubygems/discussions/8645#discussioncomment-13238676

The overarching goal of this project is to improve the general state of things surrounding gems with native extensions. That means security, ergonomics, developer experience, performance, etc., with the understanding that a system that we all like working with is better for everyone and provides us more leverage to have "secure by default" options where possible.

[kou]

This proposal improves developer experience by providing a workflow to build precompiled gems, right?

https://github.com/orgs/rubygems/discussions/8645#discussioncomment-13238676

A big part of this work, in addition to supporting the extended identifiers for platform selection, will be improving the tooling around native extensions. By having a workflow that "just works" that folks can adopt, we can make it easy for gem maintainers to ship gems that are easy to install for users, have source provenance, come from trusted (aka SLSA compliant) workflows, and support the most popular platforms that gem consumers are using. Additionally, we want to lower the barrier to shipping precompiled gems, as they have proven wildly popular with users overall (a huge thanks to @flavorjones for everything he's done in this space).

Do you have a plan to use existing rake-compiler-dock approach?

As a rake-compiler maintainer, the cross-compiling approach used by rake-compiler is another painpoint of building precompiled gems. If we depend on it, I think that developer experience will be worse. Because cross-compiling is more difficult than normal build.

[flavorjones]

@kou I agree that cross-compilation is difficult and it would be better to provide tools that compile natively on a variety of hardware and operating systems in cloud environments.

In the python community there are a few tools that provide a framework for native compilation (not cross), linting, and testing. Two that are mentioned on https://pythonwheels.com/ are:

• https://github.com/pypa/cibuildwheel
• https://github.com/multi-build/multibuild

both of which are built using lower-level projects:

• PEPs that specify glibc/musl compatibility for Linux distros
  • see https://github.com/mayeut/pep600_compliance?tab=readme-ov-file#distro-compatibility for an overview
• https://github.com/pypa/manylinux - a set of Linux OCI images providing PEP-compliant build environments
• https://github.com/pypa/auditwheel - lints/standardize built Linux binaries
• https://github.com/matthew-brett/delocate - lints/standardizes built MacOS binaries

In particular, I like https://github.com/pypa/cibuildwheel and I have a weak opinion that we should use it as a basis for solving this problem in Ruby.

There is also existing code that we can start with when piecing together some of this functionality. For example, the code in Nokogiri that lints the dynamic libraries across Linux, Windows, and MacOS (similar in purpose to auditwheel and delocate above).

And to be clear, I would be very interested in helping to build any tools that improve the developer experience in this area.

[kou]

Thanks for sharing tools used for wheels.

I agree that https://github.com/pypa/cibuildwheel like tool is useful for us even if we build binary gems by developers or by RubyGems.org. Based on https://cibuildwheel.pypa.io/en/stable/#usage and https://cibuildwheel.pypa.io/en/stable/#how-it-works , https://github.com/pypa/cibuildwheel uses native (not cross) compilation as much as possible.

I also agree that https://github.com/pypa/auditwheel / https://github.com/matthew-brett/delocate like linters are useful for us. Nokogiri's code will help us for this area.

And to be clear, I would be very interested in helping to build any tools that improve the developer experience in this area.

Me too.

[kou]

Which existing pain points of binary gems should we solved before this proposal moves forward?

This proposal may improve usability for the following case:

• Ruby 3.5 is released
• A released binary gem doesn't support Ruby 3.5 yet
• bundle install with Ruby 3.5 uses source gem not a binary gem

Because this proposal adds Ruby version to binary gem version.

(FYI: I doubt that this improves usability. See https://github.com/orgs/rubygems/discussions/8645#discussioncomment-12917093 for details.)

If this proposal improves usability, users want to use more binary gems and users will request binary gems to gem developers. If we don't solve existing pain points of binary gems before this proposal moves forward, gem developers' maintenance costs will be increased. It may reduce gem developers and it's worse for Ruby ecosystem.

Here are existing pain points (including proposals that solve them):

• Binary gem maintainers need to synchronize with CRuby releases: https://github.com/orgs/rubygems/discussions/8645#discussioncomment-12904868
• Binary bindings gem maintainers need to update bundled external libraries when any vulnerabilities are found in external libraries: https://github.com/orgs/rubygems/discussions/8645#discussioncomment-12904902
• Cross-compiling based binary gem build system is difficult: https://github.com/orgs/rubygems/discussions/8645#discussioncomment-13243371