Build not reproducible

[utkarsh2102]

Whilst working on the Reproducible Builds front, I came across that libgit2 doesn't build reproducibly.

Here are the CI logs: https://salsa.debian.org/debian/libgit2/-/jobs/661147

It'd be nice if we can have a fix for this? :)

[lhchavez]

Are there tools / tutorials to reproduce this locally? I see that there's https://reproducible-builds.org/docs/ that somewhat documents that, but I was wondering if there's an easy way for a random person interested in contributing to this (maybe me) to get up to speed and finally make sure the build is deterministic. And add a test to the CI to make sure this does not regress.

[utkarsh2102]

Hi @lhchavez,

Are there tools / tutorials to reproduces this locally?

There's reprotest available in the Debian repositories. But that is to be run on debs I think.
I personally rely on CI while checking this :)

I can give you push access to my fork of https://salsa.debian.org/debian/libgit2 so you can check via CI, if you want to?

[lhchavez]

Hi @lhchavez,

Are there tools / tutorials to reproduces this locally?

There's reprotest available in the Debian repositories. But that is to be run on debs I think.
I personally rely on CI while checking this :)

I can give you push access to my fork of https://salsa.debian.org/debian/libgit2 so you can check via CI, if you want to?

oh i was hoping to avoid relying on CI. that's slow (well, slower vs. doing it locally) and more difficult to debug when things go wrong :( if that's the case, maybe I won't be able to help.

[pks-t]

On Thu, May 14, 2020 at 03:53:08PM -0700, Utkarsh Gupta wrote: Whilst working on the [Reproducible Builds](https://reproducible-builds.org/) front, I came across that libgit2 doesn't build reproducibly. Here are the CI logs: https://salsa.debian.org/debian/libgit2/-/jobs/661147 It'd be nice if we can have a fix for this? :)

Did you try building with `-DENABLE_REPRODUCIBLE_BUILDS=ON`? I've done some work on this a few years ago, and with this option it builds reproducible for me.

[ethomson]

Did you try building with -DENABLE_REPRODUCIBLE_BUILDS=ON? I've done some work on this a few years ago, and with this option it builds reproducible for me.

Curious: what's the downside of making this the default?

[pks-t]

On Fri, May 15, 2020 at 08:28:14AM -0700, Edward Thomson wrote: > Did you try building with `-DENABLE_REPRODUCIBLE_BUILDS=ON`? I've done some work on this a few years ago, and with this option it builds reproducible for me. Curious: what's the downside of making this the default?

We're fiddling with some internal of CMake to make it word in order to get reproducible behaviour of both ar and ranlib which is not portable across platforms: ```CMake IF (ENABLE_REPRODUCIBLE_BUILDS) SET(CMAKE_C_ARCHIVE_CREATE "<CMAKE_AR> Dqc <TARGET> <LINK_FLAGS> <OBJECTS>") SET(CMAKE_C_ARCHIVE_APPEND "<CMAKE_AR> Dq <TARGET> <LINK_FLAGS> <OBJECTS>") SET(CMAKE_C_ARCHIVE_FINISH "<CMAKE_RANLIB> -D <TARGET>") ENDIF() ``` As this rather looks like magic to me and we'd probably open a can of worms where we have to implement stuff for each platform, I'm not too thrilled to make it the default.

[utkarsh2102]

Hi Patrick,

Did you try building with -DENABLE_REPRODUCIBLE_BUILDS=ON? I've done some work on this a few years ago, and with this option it builds reproducible for me.

I did now, but unfortunately, it doesn't seem to work.
CI logs: https://salsa.debian.org/debian/libgit2/-/jobs/742301

I am not really good at intercepting them so can't make out much out of this^ :/

[pks-t]

Huh, I just tried it and the resulting shared object has the same hash on both runs. Different in your build setup is that you're building the library in different paths, though, and indeed: if I use two different build directories, then the resulting hashes are different. Interestingly, this doesn't happen if building with -DCMAKE_BUILD_TYPE=Release.

Even though this doesn't seem to be true, either, cause I can't reproduce anymore. :/ So this is an intermittent error. One thing I've found that's different between CMake runs is the order of files because we use FILE(GLOB ...) everywhere, and its order is not deterministic. So it could be the linker is producing different code depending on the order of objects.

So I did a test and added list(sort ...) after every every globbing so we have deterministic ordering of files, and now each of the 60 test compilations I did produced the exact same hash. I'll create a pull request for this.

[pks-t]

Please give #5523 a go

[utkarsh2102]

Hi @pks-t @ethomson,

Thanks for your work, I just uploaded 1.0.1 to the Debian archive.
However, the build is still not reproducible. Here is the log file: reprotest.log

Please let me know if you can make out anything from it or if there's anything I can help to make the build reproducible.

[ethomson]

🤔 I see a lot of file paths in there. If you set -ffile-prefix-map when enabling reproducible builds does this fare better?

eg,

SET(CMAKE_C_FLAGS "-ffile-prefix-map=${CMAKE_SOURCE_DIR}=. ${CMAKE_C_FLAGS}")

[utkarsh2102]

On a side note, can we have reproducible builds enabled by default?
I mean, it's definitely a good thing to have anyway! What do you think? :)

[ethomson]

Well, we don't have reproducible builds yet. I think it probably makes sense to enable this by default. But I'd like to see what's involved in making a truly reproducible build before we make that decision. If we turn reproducible builds on by default, and actually making them reproducible has some onerous settings that aren't good defaults, then we'll regret it.

[utkarsh2102]

Well, we don't have reproducible builds yet. I think it probably makes sense to enable this by default. But I'd like to see what's involved in making a truly reproducible build before we make that decision. If we turn reproducible builds on by default, and actually making them reproducible has some onerous settings that aren't good defaults, then we'll regret it.

Ah, okay, makes sense. Let me try to enable repro builds in Debian and in case I get that working, I'll send a patch maybe :)

I see a lot of file paths in there. If you set -ffile-prefix-map when enabling reproducible builds does this fare better?
eg, SET(CMAKE_C_FLAGS "-ffile-prefix-map=${CMAKE_SOURCE_DIR}=. ${CMAKE_C_FLAGS}")

With this, the new log file is here: reprotest.log
Not sure how better has it gotten, if at all :)