builds of openmpi are not yet reproducible

[bmwiedemann]

Background information

What version of Open MPI are you using?

1.10.7 on openSUSE Tumbleweed (aka Factory)

Details of the problem

For https://reproducible-builds.org/ I tried to build packages of openmpi twice and found various diffs in the binaries.

One source of such diffs should be removed by
#3756

But there are still invocations of

`date`
`hostname`

in
openmpi-1.10.7/autogen.pl
openmpi-1.10.7/config/opal_functions.m4
openmpi-1.10.7/config/opal_get_version.m4
openmpi-1.10.7/ompi/tools/ompi_info/Makefile.am
openmpi-1.10.7/orte/tools/orte-info/Makefile.am
openmpi-1.10.7/oshmem/tools/oshmem_info/Makefile.am

master also has
ompi/tools/mpisync/Makefile.am
opal/mca/pmix/pmix2x/pmix/config/pmix_functions.m4
opal/mca/pmix/pmix2x/pmix/config/pmix_get_version.sh

Would be nice if those date usages could be either dropped or use the SOURCE_DATE_EPOCH environment var to override the current time (see examples in https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal#Bash_.2F_POSIX_shell )

[jsquyres]

I think we'd be amenable to $SOURCE_DATE_EPOCH in the way it is described in the page you sent (i.e., if it's set, use it, otherwise, use the same date mechanisms we've been using). Could you post another PR?

(the PMIX stuff will be submitted upstream to PMIX -- but they copy a lot of their build system / methodology from us, so they'll likely be amenable, too)

FWIW: the hostname usage (and date usage, for that matter) are mostly an issue of logging -- e.g., ompi_info shows the build date and hostname. This can be helpful in debugging, for example, if a user builds on one machine but then tries to run on an different machine with a different / incompatible set of userspace libraries, etc. I understand why a distro, for example, would want to have the same build date for all of its software; using $SOURCE_DATE_EPOCH seems like a fine solution.

Is there a $SOURCE_DATE_EPOCH analog for hostname? I.e., does a reproducible build require identical builds on different machines?

[rhc54]

Yeah, we'll take it upstream to PMIx. If it doesn't already exist, perhaps something like $SOURCE_HOSTNAME?

[ggouaillardet]

i already started implementing this, and i will likely PR today

[bmwiedemann]

On 2017-06-27 16:37, Jeff Squyres wrote: Is there a |$SOURCE_DATE_EPOCH| analog for |hostname|? I.e., does a reproducible build require identical builds on different machines?

one common approach I have seen is to re-use $SOURCE_DATE_EPOCH as an indicator for 'someone wants the build to be reproducible' e.g. ```bash if [ -n "$SOURCE_DATE_EPOCH" ] ; then hostname=someconstant fi ```

[jsquyres]

@bmwiedemann Ralph suggested $SOURCE_HOSTNAME -- does that work for you? Otherwise, we wouldn't know what to pick for someconstant.

[bmwiedemann]

The disadvantage of an extra variable is that a dozen distributions would have to discover and use it then, while $SOURCE_DATE_EPOCH is already well-established.

I usually use hostname=reproducible to indicate that it is not the real build hostname

[bmwiedemann]

https://www.zq1.de/~bernhard/temp/openmpi-reproducible-patch.txt is my test-patch that made the build fully reproducible, but I considered too unclean to upstream in this form.

[jsquyres]

@bmwiedemann Just to be sure I understand: you're suggesting using $hostname and not $SOURCE_HOSTNAME, right?

Be sure to see #3779, too (but it doesn't [yet] include anything for hostname).

[bmwiedemann]

I meant: do not use an extra variable to signify that the build should override the hostname, but look at the existance of $SOURCE_DATE_EPOCH for that.

[jsquyres]

@bmwiedemann I have to admit that that's a bit weird to me. I can see the argument of using $SOURCE_DATE_EPOCH because that's a value that's at least rooted in some reality / some user choice. But if we arbitrarily pick a non-existent hostname, that seems counter to the point, doesn't it? It would also defeat the point of having a hostname at all.

Do reproducible builds have to produce exactly the same output when run on multiple different servers? Or just at different times on the same server?

[rhc54]

That seems a little ugly as it requires one infer some intent of the user that may not exist. However, I don't feel strongly about it. I would, though, suggest that somebody in the distro world fix the ambiguity by creating a separate variable.

[bmwiedemann]

@jsquyres we have a build farm for openSUSE and the idea is to be able to (re)create identical binaries on any of the build hosts. (otherwise we have to re-publish build results to the mirrors and users wasting their bandwidth when nothing in the code actually changed)
At the moment, rpm metadata still contain the actual build hostname and a specialized compare script ignores that. But in a few years, we might be at a point when we do not need that anymore, because rpms themselves will be identical bit-by-bit no matter where and when they were built.
And the funny thing is that then you do not need to store time and place of the build anymore.

[infinity0]

To give some background, SOURCE_DATE_EPOCH was not meant to be the "go-to" solution for achieving reproducible builds. Ideally we'd like software projects to be reproducible "by default". SOURCE_DATE_EPOCH was a compromise solution specifically for timestamps because (a) so many build tools were taking inputs from the system clock (b) it would've been hard to change this int to a string "unknown" in many pieces of software, and (c) it does give some useful information about the software (the source code).

If the build is reproducible, then many aspects of the build process including the machine's host name, the date, etc, become irrelevant for debugging because the binary output only depends on the source code plus the build dependencies. The hostname is only going to be of interest to a small group of individuals who know something about that host. In some contexts, it could even be an undesired privacy leak.

It is true that not all builds are going to be reproducible, so emitting the hostname somewhere, might be good for debugging purposes, until the build becomes reproducible. However, to aid the push towards reproducibility, this should not form part of the distributed build products, but some auxiliary side products. For example in Debian we're putting this stuff in a buildinfo file. But if you really need this sort of information to be part of a distributed build product, it would be better for distributors, if this were not the default behaviour.

For hostnames, I'd suggest that a default value of unknown would be sufficient, unless someone specifically sets OPENMPI_BUILD_HOSTNAME or other specific variable. (e.g. we already have "unknown" on GNU architecture triplets.) You could set this on your buildbots or other equivalent infrastructure for debugging, but for other people doing "release builds" they would get reproducibility by default.

[bmwiedemann]

sounds good.

[ggouaillardet]

from a vendor/packager point of view, i understand the value of having reproducible builds.
from a developer point of view, i also understand timestamps/hostnames can be valuable information too.

so i'd rather give everyone the option to choose what fits best.
SOURCE_DATE_EPOCH looks like a de facto standard, so let's just use it.
there is no such thing (yet) for hostname, so i think it is just simpler we just add yet an other configure option and give everyone the choice to force hostname (or not) and how.
an other option is to use SOURCE_HOSTNAME as previously suggested and hope it will become part of the de facto standard.

the rationale is i'd rather have ompi_info --all tell me Open MPI was built on the "put your vendor name here" build farm, than on an unknown host or nodeXYZ from a given vendor build farm.

when a consensus is reached, i will update #3779 accordingly.

[rhc54]

Yeah, Jeff and I discussed this a bit earlier today and feel that we need to keep that hostname in there. I think using SOURCE_HOSTNAME is a good compromise - it gives the packagers what they want and retains the identification desired by others.

A "standard" has to start somewhere 😄

[infinity0]

I think there was some misunderstanding of what I wrote. My comments to do with SOURCE_DATE_EPOCH were not to discourage using that, but more to explain why a hostname envvar isn't a good idea. I certainly would recommend against using it, for any project that would like to achieve reproducible builds.

Having too many environment variables defeats the point of having reproducible builds by default. At the very least, pick a different name, SOURCE_HOSTNAME doesn't make any sense - the build machine's hostname isn't a characteristic of the source code.

Defaulting the value to insert-vendor-here would also be preferable to defaulting it to the actual build host. Then you might as well call it "vendor" rather than "hostname".

[ggouaillardet]

@infinity0 any better idea for the environment variable instead of SOURCE_HOSTNAME ?
i do not think we want to make builds reproducible by default, but we definitely want to give the option to make builds reproducible.

[infinity0]

Would you be happy with adding a --release flag for ./configure or something similar? When this is set, then the build could fallback to default values like 'unknown' for things like the hostname. This would make reproducible-builds easier to achieve: Suppose in the future you wanted to add extra information to the build like the kernel version, ABI number of bits, etc. Users then could tell the build to omit all of these by giving --release, rather than having to set all of them individually. (This distinction between "debug" and "release" builds has precedence in some toolchains.)

I could suggest one of HOSTNAME (already set by the shell) BUILD_HOSTNAME or MPI_BUILD_HOSTNAME, or you could take some inspiration off existing examples. But I think this is a secondary concern to my previous paragraph, i.e. to offer a single "release" flag that does provide reproducible builds without requiring distro maintainers to set an envvar for every new piece of build-time-specific information that might be added.

[rhc54]

I think we are falling towards not making the build reproducible per your definition by default, but I don't think we have any particular position regarding the switch to use when indicating that desired behavior. I agree with your concern about having a bunch of flags, but I also wonder if we won't encounter situations where one distro may feel that certain fields should be set a particular way, while another distro wants something else.

However, that said, nobody has asked for this before and I don't see some of the usual parties commenting here, so perhaps we should just deal with the immediate request. If @bmwiedemann is comfortable with the --release configure option, then I'd suggest we just go that direction, replacing the date and hostname with fixed values.

[jsquyres]

A few points in no particular order...

Use cases

There's a few use cases that we're concerned with:

1. Developer builds
2. Packager builds (usually OS's and distros)
3. End-user builds

I think that most software packages probably fall into using pre-built packager builds. But Open MPI is a bit different -- in the environments where it is used, it is quite common for the sysadmins to install a specific OS/distro, and then go manually install the latest compilers, network stacks, and MPI implementation (because they're usually more recent than are what are available in the OS/distro).

Don't get me wrong -- I'm not making any judgements about OS/distro release speeds -- there's very good reasons that they go slow (stability === good!). I'm just saying that we do have a fair number of users who actually download and build Open MPI manually -- perhaps more than most other open source packages.

Meaning: the 3rd case is still fairly important to us, which is exactly why we have elements like the hostname and timestamp in some of our build product. I.e., that hostname and timestamp are meaningful to the sysadmins who do end-user builds.

Reproducible builds

As @rhc54 and @ggouaillardet have said, I think we're quite amenable to reproducible builds. But it may still not be our default (because of reasons cited above).

Keep in mind, however, that Open MPI is highly configurable. There are a few less than 16 gazillion configure options, and Open MPI's configure script aggressively looks for all kinds of dependent software packages that we can utilize. In most cases, these the presence or absence of these dependent software packages decide whether individual plugins get built -- not whether the entire Open MPI package can be built.

As such, it is extremely common for different sites -- and even different installations of Open MPI across different resources at the same site -- to have different configurations of Open MPI.

This at least part of the reason that the idea of "reproducible by default" is kinda weird to us. We are strong believers that if you build it twice the same way in identical environment it should be totally deterministic, of course (e.g., #3755 was a great fix). But getting exactly the same binary output when building on different resources is unlikely -- but that is by design.

Specifically: we're not trying to hide the differences between different resources and environments -- we actively utilize those differences.

CLI options

All the above being said, note that we can't quite do --release as a CLI option to configure -- we can do AC_ARG_WITH or AC_ARG_ENABLE (i.e., --with-release or --enable-release). But the GNU Autotools won't let us add plain ol' --release.

That being said, the presence of $SOURCE_DATE_EPOCH is still probably fine; its presence implies --release-like behavior. So --release (or something like it) may not be necessary.

What to do?

My remaining question, however, is still about the hostname. I guess there's a few options here -- if $SOURCE_DATE_EPOCH is present:

1. Use unknown for the hostname
2. Use reproducible-builds.org for the hostname.
3. Use $HOSTNAME for the hostname
4. Use $SOURCE_HOSTNAME (or something like it) for the hostname
   • I'm not in favor of doing something like $MPI_HOSTNAME or something that would never have a chance of being used by others; we might as well take a stab at something that might be useful to others who may end up in the same situation as us
5. Not include the hostname in ompi_info (and friends) output altogether

I kinda like using reproducible-builds.org, but $HOSTNAME has its appeal, too. I don't like using unknown. But I think we're open to suggestions here.

Sidenote: we could also add a field in ompi_info (and friends) "Reproducible build: yes|no". That would give some context to the hostname/timestamp fields in ompi_info.

[ggouaillardet]

as far as i am concerned, i'd rather go with $SOURCE_DATE_EPOCH and $BUILD_HOSTNAME.
this provides the required flexibility to generate reproducible builds.
last but not least, we do not need do decide what the "best" default date and/or hostname are.
for example

• unknown is best for privacy
• build.<distro>.com makes it clear via ompi_info that the library is provided by the distro
• something else could be misleading

if we decide to go with --enable-release, then we could use the date entry from the VERSION file (i assume distros will build Open MPI based on the official tarballs)