Building from source fails due to missing private key

[Apteryks]

Hello,

Attempting to build ttfunk 1.7.0 from source results in:

starting phase `build'
ERROR:  While executing gem ... (Errno::ENOENT)
    No such file or directory @ rb_sysopen - /homeless-shelter/.gem/gem-private_key.pem
error: in phase 'build': uncaught exception:
%exception #<&invoke-error program: "gem" arguments: ("build" "./ttfunk.gemspec") exit-status: 1 term-signal: #f stop-signal: #f> 
phase `build' failed after 0.5 seconds

This can be worked around by removing the .*spec.signing_key.* line from the ttfunk.gemspec file.

[Apteryks]

Same problem with resolution in another project: https://github.com/styx/jquery-ui-rails-cdn/pull/12/files

[gettalong]

I think that if you need to build the gem yourself, you should just change what you need to do so. I don't think that there is a guarantee that everybody is supposed to be able to build the gem.

[pointlessone]

I guess the linked solution is acceptable. The cert is useful only for pushing to Rubygems anyway. However, I'd argue that building a gem yourself in this day and age is probably not the default. Pretty much every end-user application uses bundler for dependency management. Bundler supports git checkouts and that's probably what you want to use over building gems yourself.

[Apteryks]

I think that if you need to build the gem yourself, you should just change what you need to do so. I don't think that there is a guarantee that everybody is supposed to be able to build the gem.

Hi! GNU/Linux distributions such as Guix System, Nix OS or Debian are downstream users too; which is the context I'm working in (updating the ruby-ttfunk Guix package). Distributions package things themselves rather than use tools such as bundler (because that makes it possible to reuse their individual package components, forming an actual software distribution).

[pointlessone]

Is there a reason you need to repackage the gems?

[Apteryks]

Guix build packages in a reproducible fashion inside a containerized environment with just what's needed -- there's no git metadata nor git command nor network. git is often invoked in .gemspec files to get the list of files to build; that wouldn't work (the git metadata is not deterministic so cannot be part of the source input).

Currently the ruby build system in Guix does something like: unpack the gem, extract its gemspec file, adjust it as mentioned above, build it (using gem build, run the test suite, then install it (using gem install).

Is there's another way to go around it?

[pointlessone]

You're up for a bad time with that approach and not only with Prawn. Since a long time ago a gem is a distribution package only. It means it may not contain everything needed for development. It doesn't have to contain tests, for example. Likewise, it doesn't have to contain everything needed for repackaging. For example, there's no cert in it I use to sign the gems. You repackaging Prawn gems effectively strip the signature and end users don't have a way to verify if the version they have is the same I released. If you're not doing anything non-standard like bypassing rubygems during installation and using a packaged gem anyway, I suggest you use the officially released gems. I'd be glad helping with those. Otherwise, repackaging is not a supported use case.

[Apteryks]

The current strategy works rather painlessly for the ~500 Ruby packages offered in Guix, but I'm open to improvements. Guix users have a way to trace back the source quite easily, as it either comes from the original rubygems gem, which gets verified with a sha256 checksum, or from git sometimes when the test suite is not in the gem, as you've mentioned (which also has a sha256 sum and is traceable to a commit or tag).

What do you mean by repackaging? Are rubygems source distributions, binary distributions or both? As you can see, I'm no expert but eager to learn.

[pointlessone]

Are rubygems source distributions, binary distributions or both?

It's hard to clearly classify gems as any of the options you provided. Ruby is a scripting language so most of the gems package clear unobfuscated source code. But that's just a coincidence.

I'm not familiar with Guix but I think you'd be familiar with the concept of build-time dependencies and run-time dependencies. Build-time dependencies are the packages you need to build your code but they're not needed during the runtime. GCC, for instance is needed to build Ruby interpreter but you usually don't need GCC to run the interpreter. On the other hand, you need libyaml installed to be able to parse YAML from Ruby scripts. That is a run-time dependency.

Gems are run-time dependencies. And they're packaged to fulfil that role first of all. This means that a valid gem can contain only the files providing the functional parts of the codebase. Basically, whatever can be required ("imported") by the end user application. It doesn't need to contain tests, documentation, development infrastructure, etc.

However, there are gems that provide binary extensions that need to be compiled on installation. These do not fit as neatly into runtime dependency box as they package C (or Rust, or whatever) source files that are getting compiled during installation. There are pre-compiled gems (e.g. nokogiri) but packaged C source is still the default arrangement. That said, these gems still only have to provide the minimum to fulfil their runtime dependency goal. They don't need to package anything extra.

Rubygems is not quite a young system. It was born before GitHub or a public repo being a basic requirement for an open source project. In those times gems indeed were full distributions, with tests, docs, and such. Gemspecs did specify docs and tests but since then these requirements were relaxed and even removed. For example, over 2 years ago rubygems/rubygems#3758 (Rubygems 3.2) was merged making it clearer that tests should not be a part of a packaged gem. It's still possible to specify test files in the gem specification but that's mostly for backwards compatibility and not a reflection of the current gem packaging recommendation.

So to summarise: you should expect only what's required to run the code in the gem, everything else is a bonus and may disappear at any moment.

[Apteryks]

Hi @pointlessone, and thanks a lot for detailing what gems are and what purpose they serve. Perhaps the Guix gem importer (guix import gem ttfunk) should default to fetch the source from git instead of from the gem file; in some cases tests are still included by that seems to be becoming more and more of a minority of projects, as you've hinted at. This would resolve the issue I had here.

With regard to the issue at hand, there's still something that escapes me; I'm now building ruby-ttfunk from git (I had to for the tests anyway), so the certs directory is present, but removing my previous patching out of the cert directive:

(define-public ruby-ttfunk
  (package
    (name "ruby-ttfunk")
    (version "1.7.0")
    (source
     (origin
       (method git-fetch)
       ;; Fetch from github as the gem does not contain testing code.
       (uri (git-reference
             (url "https://github.com/prawnpdf/ttfunk")
             (commit version)))
       (file-name (git-file-name name version))
       (sha256
        (base32
         "1jyxn928mpyb1sikjd93s3v8fmh33232pq41ziaph513j7am6fi5"))))
    (build-system ruby-build-system)
    (arguments
     (list #:test-target "spec"         ;avoid the rubocop target
           ;; #:phases
           ;; #~(modify-phases %standard-phases
           ;;     (add-after 'unpack 'remove-missing-key-directive
           ;;       ;; This seem to be a common problem in Ruby projects (see:
           ;;       ;; https://github.com/prawnpdf/ttfunk/issues/99).
           ;;       (lambda _
           ;;         (substitute* "ttfunk.gemspec"
           ;;           ((".*spec.signing_key.*") "")))))
           ))
    (native-inputs (list ruby-prawn-dev))
    (synopsis "Font metrics parser for the Prawn PDF generator")
    (description
     "TTFunk is a TrueType font parser written in pure Ruby.  It is used as
part of the Prawn PDF generator.")
    (home-page "https://github.com/prawnpdf/ttfunk")
    ;; From the README: "Matz's terms for Ruby, GPLv2, or GPLv3. See LICENSE
    ;; for details."
    (license %prawn-project-licenses)))

Leads to:

phase `replace-git-ls-files' succeeded after 0.0 seconds
starting phase `build'                                                                                    ".
ERROR:  While executing gem ... (Errno::ENOENT)                                                           ..
    No such file or directory @ rb_sysopen - /homeless-shelter/.gem/gem-private_key.pem
error: in phase 'build': uncaught exception:
%exception #<&invoke-error program: "gem" arguments: ("build" "./ttfunk.gemspec") exit-status: 1 term-signal: #f stop-signal: #f> 
phase `build' failed after 0.1 seconds
command "gem" "build" "./ttfunk.gemspec" failed with status 

It seems to be caused by https://github.com/prawnpdf/ttfunk/blob/master/ttfunk.gemspec#L15; this file doesn't exist in my environment (HOME is set to /homeless-shelter, which doesn't exist -- another trick for reproducibility inherited from Nix; see: https://edolstra.github.io/pubs/phd-thesis.pdf).

Thanks again for taking the time to explain how things work in the land of Ruby!

[pointlessone]

That's sort of expected. The private key has to be, well, private. I use it to sign the final gem. And only I have to have it in order for everyone to know that if the gem is properly signed it must've been by me. There's a certificate ("public" counterpart) to let users verify that signature is correct.

I will turn this into a warning to allow unsigned builds for people who want to maintain their own fork. But I still want to be sure we don't forget to sign the gems.

Meanwhile, you can use a patch similar to one you linked earlier if you don't want to sign the gem.

[Apteryks]

Hi! Thanks, I'm sure that'll be appreciate by distributions who package this and sometimes need to work from the git tree instead of from the gem bundle to make minor, distribution-specific adjustments to the source.

[pointlessone]

This should be resolved now for Prawn, TTFunk, and pdf-core.