Fail when make install and deterministic build is enabled

[joaohf]

Hi,

I'm trying to build yaws 2.1.1 with deterministic build enabled like that:

export YAWS_DETERMINISTIC_BUILD = "true"
export SOURCE_DATE_EPOCH=1683409604

Looks like configuring and make are OK. However when installing I got some errors like the following:

| NOTE: make WARNINGS_AS_ERRORS= DESTDIR=/build/tmp/work/core2-64-poky-linux/yaws/2.1.1+gitAUTOINC+4ef3fa2a1d-r0/image install
| Making install in src
| make[1]: Entering directory '/build/tmp/work/core2-64-poky-linux/yaws/2.1.1+gitAUTOINC+4ef3fa2a1d-r0/git/src'
| make[1]: Leaving directory '/build/tmp/work/core2-64-poky-linux/yaws/2.1.1+gitAUTOINC+4ef3fa2a1d-r0/git/src'
| make[1]: *** No rule to make target 'yaws.hrl', needed by '../ebin/yaws.beam'.  Stop.
| make: *** [Makefile:560: install-recursive] Error 1

When disabling YAWS_DETERMINISTIC_BUILD flag the make install pass. But that is not an option for my case. Deterministic building is necessary.

I didn't find an explanation for: *** No rule to make target 'yaws.hrl', needed by '../ebin/yaws.beam'. Stop.

I'm attaching three log files for my build environment.

do_compile.txt
do_configure.txt
do_install.txt

[vinoski]

That error comes from make, and it's basically telling you that yaws.hrl is not found and make has no rules for how to create it. Please make sure the file include/yaws.hrl exists relative to the top of your Yaws source tree. If it doesn't, then something might be removing it.

Thanks for the logs, but can you also show the full command lines you're using for each step? Also, are the two environment variables you mention the only two you're setting that are Yaws-related? It would be nice to see if we can reproduce this issue with a setup that's as simple as possible, and to do that we need to know the environment and commands you're using.

[joaohf]

Thanks for the reply.

Alright,

The file 'include/yaws.hrl' exists in the top Yaws source tree (with deterministic flags enabled):

root@c04e16613bbd:/build/tmp/work/core2-64-poky-linux/yaws/2.1.1+gitAUTOINC+4ef3fa2a1d-r0/git# ls include/yaws.hrl 
include/yaws.hrl
root@c04e16613bbd:/build/tmp/work/core2-64-poky-linux/yaws/2.1.1+gitAUTOINC+4ef3fa2a1d-r0/git# mkdir /tmp/p
root@c04e16613bbd:/build/tmp/work/core2-64-poky-linux/yaws/2.1.1+gitAUTOINC+4ef3fa2a1d-r0/git# make install DESTDIR=/tmp/p
Making install in src
make[1]: Entering directory '/build/tmp/work/core2-64-poky-linux/yaws/2.1.1+gitAUTOINC+4ef3fa2a1d-r0/git/src'
make[1]: *** No rule to make target 'yaws.hrl', needed by '../ebin/yaws.beam'.  Stop.
make[1]: Leaving directory '/build/tmp/work/core2-64-poky-linux/yaws/2.1.1+gitAUTOINC+4ef3fa2a1d-r0/git/src'
make: *** [Makefile:560: install-recursive] Error 1

So, for Yaws-related there is only variables to set deterministic build. Nothing else.

I'm attaching some shell scripts that the YP environment has created in order to execute all steps for configure, compile and install. They are pure shell script and have all environment variables used. It could looks too YP specific but they are just shell script (start reading from the end of the file which calls the respective function to do the action for instance do_compile).

• In additional to the previous output log, here is the shell script commands when building with deterministic flags:

run_do_configure.txt
run_do_compile.txt
run_do_install.txt

• Build using non-deterministic flags, shell script commands and output log

run_do_compile_non_deterministic.txt
run_do_configure_non_deterministic.txt
run_do_install_non_deterministic.txt
log_do_install_non_deterministic.txt
log_do_configure_non_deterministic.txt
log_do_compile_non_deterministic.txt

When running 'make install DESTDIR=/tmp/p' without deterministic build, the make works as expected.

[avtobiff]

The issue is that the dependency Makefiles generated by erlc -M -MF <file>.Pbeam
are different when using +deterministic and not.

Without +deterministic:

yaws/src$ erlc -M -MF .deps/yaws.Pbeam -MT ../ebin/yaws.beam -I ../.. yaws.erl
yaws/src$ cat .deps/yaws.Pbeam
../ebin/yaws.beam: yaws.erl ../include/yaws.hrl ../include/yaws_api.hrl \
  yaws_appdeps.hrl yaws_debug.hrl \
  /usr/lib/erlang/lib/kernel-8.5.3/include/file.hrl

With +deterministic:

yaws/src$ erlc -M -MF .deps/yaws.Pbeam -MT ../ebin/yaws.beam -I ../.. +deterministic yaws.erl
yaws/src$ cat .deps/yaws.Pbeam
../ebin/yaws.beam: yaws.erl yaws.hrl yaws_api.hrl yaws_appdeps.hrl \
  yaws_debug.hrl file.hrl

This can be solved somewhat ugly by adding phony targets to the depencency
Makefiles when building deterministically, i.e. adding the -MP erlc option.

With -MP +deterministic:

yaws/src$ erlc -M -MP -MF .deps/yaws.Pbeam -MT ../ebin/yaws.beam -I ../.. +deterministic yaws.erl
yaws/src$ cat .deps/yaws.Pbeam
../ebin/yaws.beam: yaws.erl yaws.hrl yaws_api.hrl yaws_appdeps.hrl \
  yaws_debug.hrl file.hrl

yaws.hrl:

yaws_api.hrl:

yaws_appdeps.hrl:

yaws_debug.hrl:

file.hrl:

This could perhaps be regarded as a bug in OTP, but it is tricky with paths
when building deterministically. If the include path is absolute, then it
will be impossible to generate deterministic builds from two different
directories.