Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add doc/distribution.md to help reduce some of the confusion. #6856

Merged
merged 7 commits into from Jan 21, 2023
Merged

Add doc/distribution.md to help reduce some of the confusion. #6856

Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
06908e9
Initial guess work.
ioquatix Dec 4, 2022
0e72cb6
Updated documentation based on initial feedback.
ioquatix Dec 4, 2022
e15057d
Update doc/distribution.md
ioquatix Dec 4, 2022
104aa1d
Update doc/distribution.md
ioquatix Dec 4, 2022
7d41ffe
Update doc/distribution.md
ioquatix Dec 4, 2022
541a27e
Update doc/distribution.md
ioquatix Dec 4, 2022
9f528d6
Update doc/distribution.md
ioquatix Dec 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
50 changes: 50 additions & 0 deletions doc/distribution.md
View file
Open in desktop
@@ -0,0 +1,50 @@
# Distribution

This document outlines the expected way to distribute Ruby, with a specific focus on building Ruby packages for operating system distributions.

## Building a Ruby Tarball

The standard way to build a package for distribution is to build a tarball. This tarball includes all the related files need to build and install Ruby correctly. This includes the Ruby source code, the Ruby standard library, and the Ruby documentation.

```bash
ioquatix marked this conversation as resolved.
Show resolved Hide resolved
$ ./autogen.sh
$ ./configure -C
$ make
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Building ruby from the source itself is not necessary, if ruby is available already.

Simply ruby ./tool/make-snapshot tmp can work.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But most distributions build ruby in an environment where ruby is not yet available.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why needs to make tarballs in such environment, instead of installing?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, working ruby (currently >= 2.2) is required to build ruby from a git-cloned source tree, and tool/make-snapshot works only on a git-cloned source tree.
Although this sounds like chicken-and-egg, make dist is not intended for all Ruby developers, but for only the Ruby release managers.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use https://cache.ruby-lang.org/pub/ruby/3.0/ruby-3.0.4.tar.xz to build ruby from and it doesn't like like we need ruby for that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, this section is useful for building snapshot packages. And I am using tool/make-snapshot for that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ioquatix I agree that it is a good idea, but it probably is not "for operating system distributions".

@voxik You're creating tarballs by yourself and then building packages?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we don't. For official Fedora releases, we always use official tarballs. I don't remember any exception for past decade.

However, I am using self prepared tarballs for test builds, especially closer to the major upstream releases. I could use a nightly snapshots, but if there is some fast turnaround with fixes, the nightly snapshots are not flexible enough. And since I already have script which prepares the tarball as well as adjusts patches we are carrying around and updates part of the .spec file, I am using them always.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was very confusing, the first section of this document was basically something package managers should rarely if ever do (they should use release tarballs for packaging). I fixed it in 826067b.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

$ make dist
```

This will create several tarball in the `dist` directory. The tarball will be named e.g. `ruby-<version>.tar.gz` (several different compression formats will be generated).
ioquatix marked this conversation as resolved.
Show resolved Hide resolved

### Official Releases

The tarball for official releases is created by the release manager. The release manager will run the above commands to create the tarball. The release manager will then upload the tarball to the [Ruby website](https://www.ruby-lang.org/en/downloads/).

Downstream distributors should use the official release tarballs as part of their build process. This ensures that the tarball is created in a consistent way, and that the tarball is crytographically verified.

## Building a Ruby Package

Most distributions have a tool to build packages from a tarball. For example, Debian has `dpkg-buildpackage` and Fedora has `rpmbuild`. These tools will take the tarball and build a package for the distribution.

```bash
$ pkgver=3.1.3
$ curl https://cache.ruby-lang.org/pub/ruby/${pkgver:0:3}/ruby-${pkgver}.tar.xz --output ruby-${pkgver}.tar.xz
ioquatix marked this conversation as resolved.
Show resolved Hide resolved
$ tar xpvf ruby-${pkgver}.tar.xz
$ cd ruby-${pkgver}
$ ./configure
$ make
$ make install
```

## Updating the Ruby Standard Library

The Ruby standard library is a collection of Ruby files that are included with Ruby. These files are used to provide the basic functionality of Ruby. The standard library is located in the `lib` directory and is distributed as part of the Ruby tarball.

Occasionally, the standard library needs to be updated, for example a security issue might be found in a default gem or standard gem. There are two main ways that Ruby would update this code.

### Releasing an Updated Ruby Gem

Normally, the Ruby gem maintainer will release an updated gem. This gem can be installed alongside the default gem. This allows the user to update the gem without having to update Ruby.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, this suggestion would not work in practice. If distribution wanted to fix some issue, then it would be fixed by applying patch. Because as it is stated above "The standard way to build a package for distribution is to build a tarball." If there was a way to modify the tarball or override parts of the expanded tarball, then this could be different discussion.

Copy link
Member Author

@ioquatix ioquatix Dec 4, 2022 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should try to get clarity on this because IIUC, if a vulnerability is found in a gem, this is the current process. If the process does not work well, let's propose a better way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From distro POV, the vulnerable code needs to be fixed. And the fix does not mean shipping the fixed gem but keeping the vulnerable version around. Distro also promises that security fixes won't break the user code. This is hard to achieve with dependencies locked in Gemfile (or possibly not locked, because who would specify e.g. dependency on cgi gem, when it is part of StdLib and this causes various other issues). All in all, patching is the way to go. The vulnerability is fixed, the vulnerable code is not shipped anymore, the Gemfile.lock files are still valid.

Now, of course I could imagine some improvements. E.g. one problem is, that the default gems are not kept as a gems in the StdLib. This basically disallows to remove the vulnerable code. Quite some time ago, I have proposed this change:

https://bugs.ruby-lang.org/issues/14737

### Releasing a New Ruby Version

If the update is critical, then the Ruby maintainers may decide to release a new version of Ruby. This new version will include the updated standard library.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who is the target audience of this paragraph? The Ruby maintainers od the distribution package maintainers? This kind of speaks to the former, while it should provide explanations to latter.

Copy link
Contributor

@voxik voxik Dec 4, 2022 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the question about the target audience should be generalized to the whole document, because it is not clear anymore, after re-reading this document

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The target audience is downstream maintainers, but it's explaining the process by which they would receive or need to make changes.