opam2{rpm,deb}

Török Edwin edited this page May 14, 2015 · 1 revision

Table of Contents

Wishlist for packaging tool

Goals

  1. (local) opam repository as input, .spec/Debian as output
  2. lint the output (both opam and possibly manually edited spec/Debian), there are already external tools for this
  3. test that the build requirements are complete, i.e. it builds in a minimal environment. there are external tools mock/pbuilder for a clean/official solution, or docker for an insecure and somewhat faster solution
  4. allow to manually modify the generated .spec and debian/ files and still keep track of changes in opam metadata
  5. allow incremental refinement of the spec/debian rules, don't require everything to be 100% compliant/correct syntax
  6. allow to override opam metadata (could just git clone the opam repo and use that)
  7. opam2rpm and opam2deb should be similar to use (i.e. maybe it should be opam2pkg rpm and opam2pkg deb), and I don't necesarely want to limit this to rpm and deb, someone is welcome to create an opam2openbsd or opam2brew, etc.
  8. developing the automation tool shouldn't take (significanlty) longer than manually writing the packaging for those 9 packages
  9. the goal would also be to make it easier for upstream reviewers to review all these packages, at some point we should consider involving them in the discussion once we have some draft tool to make sure we generate good quality packages

Alternative formulation of goals

  1. assist in distribution packaging by generating (quality) package templates from the available metadata
  2. save package QA time by incorporating fixes that other sources of testing have already identified:
  • upstream via _oasis or opam
  • the official opam repo
  • other distributions
  • if upstream has 'make check/make test' equivalent propagate to opam/distro package
  • not sure about 'make doc', what is the preferred form to provide html docs?
  1. save package review time by running some checks/comparisons on the distribution package and other sources of metadata (opam/oasis), and allow easy use together with distribution provided QA tools (lintian, rpmlint,opam lint)
  2. provide easy way to test for completeness of version constraints (#3, #9 on old list) and that all required files are available in a (sub)package (compare with ocamlfind META where available). Integrate with distribution provided QA tools (pbuilder, mock/fedora-review)
  3. assist upstream and other distribution packagers
  • by upstreaming the metadata changes
  • by sending metadata changes back to opam
  • but don't put additional burden on upstream maintainers to deal with downstream distributions, unless they actually want to (i.e. allow the changes to be kept as patches, or separate metadata)
  1. avoid duplication of work by tracking already existing packages or WiP packaging in distributions (the package lifecycle tool)

Note: it is probably a good idea to start with something small that is already useful (e.g. extracting version constraint and C dependencies), and gradually add more tools rather than write one monolithic tool.

Automation

  • mapping of opam to Fedora/Debian naming conventions (some opam packages have ocaml- already in them, so take care not to duplicate that)
  • generate .spec and debian/ file templates from opam metadata + some overrides. If this was just a one-way/one-time conversion I'd go with something simple (perhaps like ocaml-mustache), however see below
  • check license name compliance (Fedora and Debian have their standard name lists), and make sure the package actually has a license file
  • don't need full roundtrip capability between opam and spec/debian, the tool should serve three purposes only:
  • generate initial templates
  • show a smart diff
  • lint the opam metadata / spec file
  • shouldn't require to update the conversion tool just to support an obscure corner-case that can be fixed more easily by hand in the spec file
  • doesn't have to be perfect or support the full features of spec/debian/opam files. lets start with the minimum requirements and add fields/features as we need them
  • its not a substitute for human review, the final output always has to be tested and reviewed by a human

Metadata

  • findlib file in opam
  • OASIS can store additional metadata (fields starting with X)
  • opam metadata: depexts, conf-* opam packages, build rules, constraints, etc.
  • opam-repository, where some packages implicitly assume a "modern" Linux environment (has native compiler, dynlink, certain tools and C headers, etc.). For distribution packages all these dependencies will have to be made explicit

There can be multiple sources of metadata, each one containing fixes as required in its respective distribution medium, and just because a constraint had to be updated to fix a FTBFS in opam, doesn't mean that the debian/fedora constraints are wrong or have to be updated. Its not easy to choose one canonical source for this, because a constraint that works in opam may be insufficient in deb and viceversa. Still it'd be useful to be able to synchronize all these constraints for the next upstream release, or for the next opam/deb/rpm package version.

Version constraints (opam vs distro)

constraints on maximum versions

  • added in opam when a new version of a library is released and it is found that some of the packages that depend on it no longer build with the newer version (e.g. ocsigenserver and lwt). Upstream couldn't have foreseen that at the time it released its library/application, although it can make a new release that supports the new library version and adds the required maximum constraints to the old versions. In some cases such version constraints are added by the upstream or maintainer of the library itself, as it was done for the recent ctypes update
  • in deb/rpm build rules the max version is usually expressed as a conflicts or breaks, but it is only relevant when newer upstream versions/patches that fix the build don't exist yet (you don't have to - and even can't - go and patch old deb/rpm packages to update their dependencies whenever newer conflicting upstream versions of libraries are released)

constraints on minimum versions

  • similar to max version: added when it is found out that the build actually breaks due to an older version, although opam may lack very old versions of packages that distributions might have. just because opam doesn't have a minimum version constraint, it doesn't mean that the distro package doesn't need one
  • in distributions minimum constraints are needed if you want to make it clear that a backport will fail unless it has the updated package available, or if you want to reuse the same build rules for different distributions (debian and ubuntu, or fedora and epel/centos).
  • exact version constraints: these are less useful, unless you want to build a framework perhaps

Core packages takes an interesting approach here, it preemptively adds maximum constraints even if such newer versions don't exist yet: fieldslib >= 109.20.00 & < 109.21.00. I take it that this means that library version X.Y.00 to X.Y.99 are guaranteed to be backwards compatible at the API (and semantic) level, but X.Y+1 might break API compatibility. They do have to release updated versions of all such dependant packages (or fix their constraints) when releasing X.Y+1 though. Not all OCaml packages offer such guarantees though.

Existing tools and libraries

OCaml

opam2debian

  • Creates a Debian package using OPAM to build the code and Debian tools to build the package
  • opam package

ocaml-debian-formats

  • Can parse debian/ metadata
  • opam package (pinnable, but not in opam-repository yet)

oasis2debian

  • Can parse _oasis
  • Can generate debian/ customizable via cmdline flags
  • opam package: ocaml/oasis2debian#1

Dose3

  • Has code for parsing rpm .spec files (partially depends on librpm)
  • Has code for parsing debian/ metadata
  • opam package with rpm library support: ocaml/opam-repository#4075

Other

Might be useful as a starting point:

pkgwat

Both a Python library and a command-line tool for querying information about Fedora packages: bugs, koji builds, changelog, release versions, bodhi updates.

whohas

Query multiple distributions for package names

pybugz

Query bugzilla from Python

Implementation thoughts for opampkg

for generating the template:

  • oasis2deb if package is using oasis, with a wrapper that reads opam metadata (for C dependencies), script in opampkg deb
  • opam-lib, debian-formats and opampkg deb
  • opam-lib, spec-formats(doesn't exist yet), and opampkg rpm
  • make sure to run 'make check/make test' or equivalent if available in upstream package

for parsing the packages:

  • dose(?) for parsing debian and spec files, at least for version constraints.
  • opam-lib for opam
  • oasis lib for _oasis
  • compute version intervals for all dependencies and present their intersection (mapping rpm/deb package names as needed), opampkg constraints

opampkg lint

look for missing fields or intervals that are wider than necessary

opampkg review

  • for version constraints: integrate with pbuilder/mock/docker for easy testing in a clean/minimal environment, this should probably be just a shell script that runs all the external tools with appropriate flags
  • for META file: use findlib and dose(?) or external tools (rpm2cpio/cpio --list, dpkg-deb --contents) to load the list of files in each package and make sure all files are packaged, or do a test install a fakechroot/docker container and make sure that ocamlfind ocamlc/ocamlfind ocamlopt and ocaml #require works if the package is a library. I don't know if this could be done at package build time, perhaps after cleaning the buildroot?

opampkg patches

  • patches in opam for their packages
  • modified version constraints in opam (compared to upstream opam/oasis)
  • patches in distributions for their packages
  • modified version constraints in distributions
  • additional build-time/run-time non-OCaml dependencies in distributions (or opam depext/conf-*)

opampkg lifecycle

  • could be based on pkgwat-python-api and python-bugzilla/pybugz for fedora. Is there a Debian equivalent for querying bugs.debian.org and tracker.debian.org?
  • could be based on ocaml-github for querying presence/status of PR for opam package

Automation for tracking a package through its lifecycle (separate tool)

Since packaging an OCaml application can involve packaging/tracking a large number of dependant libraries some tool to track the status of each package (lifecycle) would be useful. This is not specific to OCaml, I'm surprised that upstream distros don't have this already, but anyway using Fedora terminology I want a single CLI tool that can:

  • query yum repo for stable packages
  • query pkgdb for existence of package in other releases
  • query bodhi for pending/testing packages
  • query koji for builds submitted/completed
  • query for existence of scm (git) repo
  • query bugzilla for package review and find out its state (submitted, under review, approved, waiting for scm repo, scm repo created)

pkgwat (see above) does some of this, and perhaps bugz could be used to query for package reviews.

For Debian a similar tool that:

  • query bugs.debian.org for RFP (request for packaging), ITP (intent to package) and its status
  • query NEW queue for uploaded source package
  • query package tracker for buildd status
  • this can also avoid duplicating work, i.e. finding out that someone else is already packaging my package or its dependencies
  • might also be useful for other packages
  • can be done manually by visiting a bunch of webpages in turn and searching on each, but there ought to be some CLI tools to help or at least some (web)API
  • doesn't have to be necesarely written in Ocaml, if we can reuse existing tools and have this just as a shell/python script the better

Documentation on packaging

Fedora

Debian

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.