Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

when embedding opam package information (app_info), do not use --rec argument to opam #189

Merged
merged 1 commit into from Dec 16, 2019

Conversation

@hannesm
Copy link
Member

hannesm commented Dec 14, 2019

While testing reproducible builds of my MirageOS unikernels with orb in a modified version, which does the following steps:

  • create a switch yyy to build a unikernel (== opam package)
  • this switch yyy knows about a list of opam repositories provided on command-line
  • the opam solver selects packages as dependencies and installs the unikernel, resulting in a single ELF artifact
  • the build is done, some information is collected:
    • an (slightly modified, see ocaml/opam#4040) opam export
    • hashes of the installed artifacts by the unikernel package
    • environment data (SOURCE_DATE_EPOCH, BUILD_PATH, OS/OS_FAMILY/...)

Now, the goal is to reproduce the very same (bitwise equal) binary. This is achieved by orb rebuild:

  • take environment and export file from above
  • create an empty switch yyy (using the same BUILD_PATH as before)
  • opam import the exported switch (this installs the unikernel as well)
  • compare the hashes

That works reliable, the issue (fixed in this PR) is that opam list --installed --rec <pkgs> returns different results in the first and second build (I suspect since the --rec leads to a code path in opam that uses the cudf solver (which seems to not know about packages from other repositories than the default repository in the second build). This PR avoids using --rec, but does the recursion/fixpoint in functoria. When --rec is not used, only the opam file is consulted, no cudf involved --> same result and reproducibility.

As further information (//cc @AltGr @rjbou to take a brief look whether this is an opam issue or not -- my intuition is that list --required-by yy should always be a subset of list --rec --required-by yy) -- this is the imported switch above, i.e. it has all the packages installed, but only knows about the "default" repository:

$ opam list --installed --required-by conduit-mirage
# Packages matching: installed & required-by(conduit-mirage)
# Name                  # Installed # Synopsis
awa-mirage              0.0.1~dev   SSH implementation in OCaml
conduit-lwt             2.0.99~dev  A portable network connection establishment library using L
cstruct                 5.1.1       Access C-like structures directly from OCaml
dns-client              4.2.0       Pure DNS resolver API
dune                    2.0.0       Fast, portable, and opinionated build system
ipaddr                  4.0.0       A library for manipulation of IP (and MAC) address represen
ipaddr-sexp             4.0.0       A library for manipulation of IP address representations us
mirage-clock            3.0.1       Libraries and module types for portable clocks
mirage-flow             2.0.1       Flow implementations and combinators for MirageOS
mirage-flow-combinators 2.0.1       Flow implementations and combinators for MirageOS specializ
mirage-kv               3.0.1       MirageOS signatures for key/value devices
mirage-random           2.0.0       Random-related devices for MirageOS
mirage-stack            2.0.1       MirageOS signatures for network stacks
ocaml                   4.09.0      The OCaml compiler (virtual package)
ppx_sexp_conv           v0.12.0     [@@deriving] plugin to generate S-expression conversion fun
ptime                   0.8.5       POSIX time for OCaml
sexplib                 v0.12.0     Library for serializing OCaml values to and from S-expressi
tls                     0.11.0~dev  Transport Layer Security purely in OCaml
vchan                   5.0.0       Xen Vchan implementation
xenstore                2.1.1       Xenstore protocol in pure OCaml

With --rec, there are more packages, but some are missing (conduit-lwt / awa-mirage):

$ opam list --installed --rec --required-by conduit-mirage
# Packages matching: installed & rec-required-by(conduit-mirage)
# Name                    # Installed # Synopsis
asn1-combinators          0.2.0-1     Embed typed ASN.1 grammars in OCaml
astring                   0.8.3       Alternative String module for OCaml
base                      v0.12.2     Full standard library replacement for OCaml
base-bytes                base        Bytes library distributed with the OCaml compiler
base-threads              base
base-unix                 base
bigarray-compat           1.0.0       Compatibility library to use Stdlib.Bigarray when possibl
cmdliner                  1.0.4       Declarative definition of command line interfaces for OCa
conduit-mirage            2.0.99~dev  A network connection establishment library for MirageOS
conf-gmp                  1           Virtual package relying on a GMP lib system installation
conf-m4                   1           Virtual package relying on m4
conf-perl                 1           Virtual package relying on perl
conf-pkg-config           1.1         Virtual package relying on pkg-config installation
cppo                      1.6.6       Code preprocessor like cpp for OCaml
cpuid                     0.1.2       Detect CPU features
cstruct                   5.1.1       Access C-like structures directly from OCaml
cstruct-lwt               5.1.1       Access C-like structures directly from OCaml
cstruct-sexp              5.1.1       S-expression serialisers for C-like structures
dns                       4.2.0       An opinionated Domain Name System (DNS) library
dns-client                4.2.0       Pure DNS resolver API
domain-name               0.3.0       RFC 1035 Internet domain names
dune                      2.0.0       Fast, portable, and opinionated build system
dune-configurator         2.0.0       Helper library for gathering system configuration
dune-private-libs         2.0.0       Private libraries of Dune
duration                  0.1.3       Conversions to various time units
fmt                       0.8.8       OCaml Format pretty-printer combinators
functoria-runtime         3.0.2       Runtime support library for functoria-generated code
gmap                      0.3.0       Heterogenous maps over a GADT
gmp-freestanding          6.1.2-2     The GNU Multiple Precision Arithmetic Library
io-page                   2.3.0       Support for efficient handling of I/O memory pages
ipaddr                    4.0.0       A library for manipulation of IP (and MAC) address repres
ipaddr-sexp               4.0.0       A library for manipulation of IP address representations 
logs                      0.7.0       Logging infrastructure for OCaml
lru                       0.3.0       Scalable LRU caches
lwt                       4.4.0       Promises and event-driven I/O
lwt-dllist                1.0.0       Mutable doubly-linked list with Lwt iterators
macaddr                   4.0.0       A library for manipulation of MAC address representations
mirage-clock              3.0.1       Libraries and module types for portable clocks
mirage-clock-freestanding 3.0.1       Paravirtual implementation of the MirageOS Clock interfac
mirage-device             2.0.0       Abstract devices for MirageOS
mirage-entropy            0.5.0       Entropy source for MirageOS unikernels
mirage-flow               2.0.1       Flow implementations and combinators for MirageOS
mirage-flow-combinators   2.0.1       Flow implementations and combinators for MirageOS special
mirage-kv                 3.0.1       MirageOS signatures for key/value devices
mirage-no-xen             1           Virtual package conflicting with mirage-xen
mirage-profile            0.9.1       Collect runtime profiling information in CTF format
mirage-protocols          4.0.1       MirageOS signatures for network protocols
mirage-random             2.0.0       Random-related devices for MirageOS
mirage-runtime            3.7.1       The base MirageOS runtime library, part of every MirageOS
mirage-stack              2.0.1       MirageOS signatures for network stacks
mmap                      1.1.0       File mapping functionality
mtime                     1.2.0       Monotonic wall-clock time for OCaml
num                       1.3         The legacy Num library for arbitrary-precision integer an
obuild                    0.1.10      simple package build system for OCaml
ocaml                     4.09.0      The OCaml compiler (virtual package)
ocaml-base-compiler       4.09.0      Official release 4.09.0
ocaml-compiler-libs       v0.12.1     OCaml compiler libraries repackaged
ocaml-config              1           OCaml Switch Configuration
ocaml-freestanding        0.4.7       Freestanding OCaml runtime
ocaml-migrate-parsetree   1.5.0       Convert OCaml parsetrees between different versions
ocaml-src                 4.09.0      Compiler sources
ocamlbuild                0.14.0      OCamlbuild is a build system with builtin rules to easily
ocamlfind                 1.8.1       A library manager for OCaml
ocb-stubblr               0.1.1-1     OCamlbuild plugin for C stubs
ocplib-endian             1.0         Optimised functions to read and write int16/32/64 from st
parsexp                   v0.12.0     S-expression parsing library
ppx_cstruct               5.1.1       Access C-like structures directly from OCaml
ppx_derivers              1.2.1       Shared [@@deriving] plugin registry
ppx_deriving              4.4         Type-driven code generation for OCaml >=4.02.2
ppx_sexp_conv             v0.12.0     [@@deriving] plugin to generate S-expression conversion f
ppx_tools                 5.3+4.08.0  Tools for authors of ppx rewriters and other syntactic to
ppx_tools_versioned       5.2.3       A variant of ppx_tools based on ocaml-migrate-parsetree
ppxfind                   1.3         Tool combining ocamlfind and ppx
ppxlib                    0.8.1       Base library and tools for ppx rewriters
psq                       0.2.0       Functional Priority Search Queues
ptime                     0.8.5       POSIX time for OCaml
randomconv                0.1.2       Convert from random byte vectors (Cstruct.t) to random na
re                        1.9.0       RE is a regular expression library for OCaml
result                    1.4         Compatibility Result module
rresult                   0.6.0       Result value combinators for OCaml
seq                       base        Compatibility package for OCaml's standard iterator type 
sexplib                   v0.12.0     Library for serializing OCaml values to and from S-expres
sexplib0                  v0.12.0     Library containing the definition of S-expressions and so
solo5-bindings-hvt        0.6.4       Solo5 sandboxed execution environment (hvt target)
stdio                     v0.12.0     Standard IO library for OCaml
stdlib-shims              0.1.0       Backport some of the new stdlib features to older compile
stringext                 1.6.0       Extra string functions for OCaml
topkg                     1.0.1       The transitory OCaml software packager
uchar                     0.0.2       Compatibility library for OCaml's Uchar module
uri                       3.1.0       An RFC3986 URI/URL parsing library
vchan                     5.0.0       Xen Vchan implementation
x509                      0.8.1       Public Key Infrastructure (RFC 5280, PKCS) purely in OCam
xenstore                  2.1.1       Xenstore protocol in pure OCaml
xenstore_transport        1.1.0       Low-level libraries for connecting to a xenstore service 
zarith                    1.7         Implements arithmetic and logical operations over arbitra
zarith-freestanding       1.7-2       Implements arithmetic and logical operations over arbitra
…ositories

being properly configured. instead, use 'opam list --required-by ..' and do the
recursion in functoria manually.

the latter uses code path in functoria which only depends on opam files, and not
on the solver.
@hannesm hannesm force-pushed the hannesm:opam-no-rec branch from 7e33995 to d4071a7 Dec 14, 2019
@samoht

This comment has been minimized.

Copy link
Member

samoht commented Dec 16, 2019

Looks great. I guess multiple opam calls is a bit slow, but it's better to be precise.

@samoht samoht merged commit b09e9c5 into mirage:master Dec 16, 2019
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
ocaml-ci Passed
Details
@hannesm

This comment has been minimized.

Copy link
Member Author

hannesm commented Dec 16, 2019

it actually turns out that multiple calls without --rec are "just file reads", while a call with --rec involves the CUDF solver (which takes quite some time). (all AFAIU, I don't know the internals too well)

hannesm added a commit to hannesm/opam-repository that referenced this pull request Dec 16, 2019
CHANGES:

* Fix equality for `'a impl` values, which caused issue in `mirage configure`
  when multiple keys share the same name (issue mirage/functoria#187, fix mirage/functoria#188 by @samoht)
* App_info: avoid `opam list --rec` which uses the CUDF solver, instead do
  fixpoint manually. Fixes reproducibility with `orb` (mirage/functoria#189 @hannesm)
@rjbou

This comment has been minimized.

Copy link

rjbou commented Dec 17, 2019

@hannesm
As you noted, packages are not computed the same way: --required-by is map/list manipulation while with --rec it is using the cudf graph (not the solver). These two commands should return a common subset, unless there is choices (|) in dependencies.

On the loaded repositories, if --no-switch is not given, switch state is loaded with all its recorded repositories. And the packages selection is done on repositories packages and installed ones (with or without depopts).

Feel free to open an issue if there is incoherent/incorrect behaviors :) It can also help other that have the same questions

@samoht

This comment has been minimized.

Copy link
Member

samoht commented Jan 7, 2020

Unfortunately this still doesn't seem to work properly.

$ opam list --installed -s --required-by fmt.0.8.8
ocaml
ocamlbuild
ocamlfind
seq
stdlib-shims
topkg

but

$ opam install fmt.0.8.8
$ opam list --installed -s --required-by fmt
ocaml
ocamlbuild
ocamlfind
result
seq
stdlib-shims
topkg
uchar
@hannesm

This comment has been minimized.

Copy link
Member Author

hannesm commented Jan 7, 2020

@samoht maybe dune-build-info will solve the job for us soon? :)

@hannesm

This comment has been minimized.

Copy link
Member Author

hannesm commented Jan 7, 2020

since we're querying only the package name (without version) and have it installed, it may be an overapproximation of the included packages -- which is fine with me since build dependencies are in there as well.

@samoht

This comment has been minimized.

Copy link
Member

samoht commented Jan 7, 2020

It's just not very reproducible, the result is different in the CI and on my machine (which is annoying). I'll check dune-build-info but in the meantime I've pushed mirage/mirage@c41e02f

@rjbou

This comment has been minimized.

Copy link

rjbou commented Jan 8, 2020

@samoht --installed is a filter on the resulting package list, not a selector on the given package. Then the first command gives the list of all dependencies of all versions of fmt that are installed in the switch, while the second one restrain it to that 0.8.8 version.

It is more visible on the not shortened output

$ opam list  --installed --required fmt
# Packages matching: installed & required-by(fmt)
# Name       # Installed # Synopsis
ocaml        4.07.1      The OCaml compiler (virtual package)
ocamlbuild   0.14.0      OCamlbuild is a build system with builtin rules to easily build most OCaml projects.
ocamlfind    1.8.1       A library manager for OCaml
result       1.4         Compatibility Result module
seq          base        Compatibility package for OCaml's standard iterator type starting from 4.07.
stdlib-shims 0.1.0       Backport some of the new stdlib features to older compiler
topkg        1.0.1       The transitory OCaml software packager
uchar        0.0.2       Compatibility library for OCaml's Uchar module

$ opam list  --installed --required fmt.0.8.8
# Packages matching: installed & required-by(fmt.0.8.8)
# Name       # Installed # Synopsis
ocaml        4.07.1      The OCaml compiler (virtual package)
ocamlbuild   0.14.0      OCamlbuild is a build system with builtin rules to easily build most OCaml projects.
ocamlfind    1.8.1       A library manager for OCaml
seq          base        Compatibility package for OCaml's standard iterator type starting from 4.07.
stdlib-shims 0.1.0       Backport some of the new stdlib features to older compiler
topkg        1.0.1       The transitory OCaml software packager
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.