Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make libmvndnative.* builds reproducible #628

Closed
hboutemy opened this issue May 1, 2022 · 5 comments
Closed

make libmvndnative.* builds reproducible #628

hboutemy opened this issue May 1, 2022 · 5 comments
Milestone

Comments

@hboutemy
Copy link
Member

hboutemy commented May 1, 2022

currently, when doing cd native ; make native-all, we see that on the 14 native libs built and stored in src/main/resources/org/mvndaemon/mvnd/nativ/, 11 are not reproducible = each run gives a different output, that will lead to a Git commit during a release even if there was no code change

ideally, such rebuilds should give reproducible output https://reproducible-builds.org/ , even if they use dockcross https://github.com/dockcross/dockcross

the non-reproducible targets are:

  • Linux for all arch (arm, arm64, armv6, armv7, ppc64, x86, x86_64)
  • Mac for x86 and x86_64 arch (arm64 arch is ok...)
  • Windows for all arch (x86, x86_64)

FreeBSD is already ok for all arch (x86, x86_64)

@gnodet
Copy link
Contributor

gnodet commented May 2, 2022

#631 fixes the problem for me on windows, but on all other platforms, the builds are reproducible for me.
This must be a docker environment issue similar to #627

@hboutemy
Copy link
Member Author

hboutemy commented May 3, 2022

FYI, I just rebuilt the libs and even with #628 merged, I get a different dll than the one stored in Git

I'm curious to see how the GH release script in GH Actions rebuilding the libs will behave: get the same output or something different?

@gnodet
Copy link
Contributor

gnodet commented May 3, 2022

@hboutemy this should be fixed if you want to have another look ....

@hboutemy
Copy link
Member Author

hboutemy commented May 4, 2022

wow, really nice: I now have only 1 difference = src/main/resources/org/mvndaemon/mvnd/nativ/Linux/x86/libmvndnative.so
I suppose this is due to the fact that it is my platform (I'm running Ubuntu), then my local build did not use the containerized build
using diffoscope, I see

│ -  [     0]  GCC: (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
│ +  [     0]  GCC: (Debian 10.2.1-6) 10.2.1 20210110

last remaining issue: I still have 1 Mac arch and the 2 FreeBSD archs created as root in target, with the FreeBSD ones copied to src as root also:

$ ll target/
total 80
drwxrwxr-x 20 herve herve 4096 mai    4 09:02 ./
drwxrwxr-x  6 herve herve 4096 mai    4 09:01 ../
drwxr-xr-x  3 herve herve 4096 mai    4 09:01 classes/
drwxrwxr-x  6 herve herve 4096 mai    4 09:01 crossbuild/
drwxr-xr-x  7 herve herve 4096 mai    4 09:02 ducible/
drwxrwxr-x  4 herve herve 4096 mai    4 09:01 inc/
drwxr-xr-x  2 root  root  4096 mai    4 09:02 native-FreeBSD-x86/
drwxr-xr-x  2 root  root  4096 mai    4 09:02 native-FreeBSD-x86_64/
drwxr-xr-x  2 herve herve 4096 mai    4 09:01 native-Linux-arm/
drwxr-xr-x  2 herve herve 4096 mai    4 09:01 native-Linux-arm64/
drwxr-xr-x  2 herve herve 4096 mai    4 09:01 native-Linux-armv6/
drwxr-xr-x  2 herve herve 4096 mai    4 09:01 native-Linux-armv7/
drwxr-xr-x  2 herve herve 4096 mai    4 09:01 native-Linux-ppc64/
drwxr-xr-x  2 herve herve 4096 mai    4 09:01 native-Linux-x86/
drwxr-xr-x  2 herve herve 4096 mai    4 09:01 native-Linux-x86_64/
drwxr-xr-x  2 root  root  4096 mai    4 09:02 native-Mac-arm64/
drwxr-xr-x  2 herve herve 4096 mai    4 09:02 native-Mac-x86/
drwxr-xr-x  2 herve herve 4096 mai    4 09:02 native-Mac-x86_64/
drwxr-xr-x  2 herve herve 4096 mai    4 09:02 native-Windows-x86/
drwxr-xr-x  2 herve herve 4096 mai    4 09:02 native-Windows-x86_64/

$ ll src/main/resources/org/mvndaemon/mvnd/nativ/*/*
src/main/resources/org/mvndaemon/mvnd/nativ/FreeBSD/x86:
total 16
drwxrwxr-x 2 herve herve 4096 mai    4 09:02 ./
drwxrwxr-x 4 herve herve 4096 mai    4 09:01 ../
-rwxr-xr-x 1 root  root  4803 mai    4 09:02 libmvndnative.so*

src/main/resources/org/mvndaemon/mvnd/nativ/FreeBSD/x86_64:
total 16
drwxrwxr-x 2 herve herve 4096 mai    4 09:02 ./
drwxrwxr-x 4 herve herve 4096 mai    4 09:01 ../
-rwxr-xr-x 1 root  root  6755 mai    4 09:02 libmvndnative.so*

src/main/resources/org/mvndaemon/mvnd/nativ/Linux/arm:
total 16
drwxr-xr-x 2 herve herve 4096 mai    4 09:01 ./
drwxrwxr-x 9 herve herve 4096 mai    4 08:58 ../
-rwxrwxr-x 1 herve herve 7804 mai    4 09:01 libmvndnative.so*

src/main/resources/org/mvndaemon/mvnd/nativ/Linux/arm64:
total 20
drwxr-xr-x 2 herve herve 4096 mai    4 09:01 ./
drwxrwxr-x 9 herve herve 4096 mai    4 08:58 ../
-rwxrwxr-x 1 herve herve 8864 mai    4 09:01 libmvndnative.so*

src/main/resources/org/mvndaemon/mvnd/nativ/Linux/armv6:
total 20
drwxr-xr-x 2 herve herve 4096 mai    4 09:01 ./
drwxrwxr-x 9 herve herve 4096 mai    4 08:58 ../
-rwxrwxr-x 1 herve herve 8564 mai    4 09:01 libmvndnative.so*

src/main/resources/org/mvndaemon/mvnd/nativ/Linux/armv7:
total 16
drwxr-xr-x 2 herve herve 4096 mai    4 09:01 ./
drwxrwxr-x 9 herve herve 4096 mai    4 08:58 ../
-rwxrwxr-x 1 herve herve 7824 mai    4 09:01 libmvndnative.so*

src/main/resources/org/mvndaemon/mvnd/nativ/Linux/ppc64:
total 24
drwxr-xr-x 2 herve herve  4096 mai    4 09:01 ./
drwxrwxr-x 9 herve herve  4096 mai    4 08:58 ../
-rwxrwxr-x 1 herve herve 70304 mai    4 09:01 libmvndnative.so*

src/main/resources/org/mvndaemon/mvnd/nativ/Linux/x86:
total 24
drwxr-xr-x 2 herve herve  4096 mai    4 09:01 ./
drwxrwxr-x 9 herve herve  4096 mai    4 08:58 ../
-rwxrwxr-x 1 herve herve 15188 mai    4 09:01 libmvndnative.so*

src/main/resources/org/mvndaemon/mvnd/nativ/Linux/x86_64:
total 20
drwxr-xr-x 2 herve herve 4096 mai    4 09:01 ./
drwxrwxr-x 9 herve herve 4096 mai    4 08:58 ../
-rwxrwxr-x 1 herve herve 8384 mai    4 09:01 libmvndnative.so*

src/main/resources/org/mvndaemon/mvnd/nativ/Mac/arm64:
total 60
drwxrwxr-x 2 herve herve  4096 mai    4 09:01 ./
drwxrwxr-x 5 herve herve  4096 mai    4 09:01 ../
-rwxrwxr-x 1 herve herve 50801 mai    4 09:02 libmvndnative.jnilib*

src/main/resources/org/mvndaemon/mvnd/nativ/Mac/x86:
total 20
drwxr-xr-x 2 herve herve 4096 mai    4 09:01 ./
drwxrwxr-x 5 herve herve 4096 mai    4 09:01 ../
-rwxrwxr-x 1 herve herve 9044 mai    4 09:02 libmvndnative.jnilib*

src/main/resources/org/mvndaemon/mvnd/nativ/Mac/x86_64:
total 20
drwxr-xr-x 2 herve herve 4096 mai    4 09:01 ./
drwxrwxr-x 5 herve herve 4096 mai    4 09:01 ../
-rwxrwxr-x 1 herve herve 9260 mai    4 09:02 libmvndnative.jnilib*

src/main/resources/org/mvndaemon/mvnd/nativ/Windows/x86:
total 96
drwxr-xr-x 2 herve herve  4096 mai    4 09:01 ./
drwxrwxr-x 4 herve herve  4096 mai    4 08:58 ../
-rwxrwxr-x 1 herve herve 87265 mai    4 09:02 mvndnative.dll*

src/main/resources/org/mvndaemon/mvnd/nativ/Windows/x86_64:
total 112
drwxr-xr-x 2 herve herve   4096 mai    4 09:01 ./
drwxrwxr-x 4 herve herve   4096 mai    4 08:58 ../
-rwxrwxr-x 1 herve herve 103015 mai    4 09:02 mvndnative.dll*

really nice work done, very good improvements

@gnodet
Copy link
Contributor

gnodet commented Sep 8, 2022

Closing this one, I think the work has been done for 0.8.0.
@hboutemy please reopen if needed.

@gnodet gnodet closed this as completed Sep 8, 2022
@gnodet gnodet added this to the 0.8.0 milestone Sep 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants