bins change on rebuild, even though there's no change in source code?

[emiltin]

i'm working on smarter caching of test files, so that extracting and preparing will not be run if only the osrm-routed binary changed.

but it seems that relinking will change the bins even though no source code has changed. somehow linking must involve a timestamp or random number? after running 'rake' from the root folder, all the bins have new a new sha1sum, which makes it impossible to skip the extraction/preparing step.

running osrm-extract -v shows the same version string; the git revision is unchanged, and there's no data info. so it must be something else.

[DennisOSRM]

Strange, indeed. Will investigate.

[DennisOSRM]

Confirmed, it does indeed include a compile timestamp in the binary.

[emiltin]

hm, is that part of the cmake machinery? or can it be removed?

[DennisOSRM]

There's code that picking up the compile time from the compiler through a macro:

        SimpleLogger().Write() << "starting up engines, " << g_GIT_DESCRIPTION << ", "
                               << "compiled at " << __DATE__ << ", " __TIME__;

[emiltin]

is something similar used in extract and prepare?

[emiltin]

i guess. could we remove it?

[DennisOSRM]

I would have to check, but I think yes. Checking right now if removing fixes things.

[emiltin]

ok

[DennisOSRM]

Still some difference. will continue digging.

[DennisOSRM]

Should we remove this from all binaries?

[emiltin]

yes if possible. if we can avoid change in bins, it can speed up tests a lot. for example, if you only change/relink osrm-routed, you don't need to extract and prepare data when rerunning tests, which will probably speed up tests 5-10 times.

[DennisOSRM]

Progress:

$ md5sum osrm-routed 
25ec47bae4c9659d00f7889512654934  osrm-routed
$ make -j3 osrm-routed 
[  5%] Built target LOGGER
[ 11%] Built target COORDINATE
[ 17%] Configuring FingerPrint.cpp
[ 23%] Building CXX object CMakeFiles/OSRM.dir/Util/FingerPrint.cpp.o
Linking CXX static library libOSRM.a
[ 76%] Built target OSRM
Linking CXX executable osrm-routed
[100%] Built target osrm-routed
$ md5sum osrm-routed 
25ec47bae4c9659d00f7889512654934  osrm-routed

[emiltin]

cool

[DennisOSRM]

fix upcoming in the next minutes once local test run succeeds.

[emiltin]

nice

[emiltin]

if i rebuild without code change, then bins now stay the same. so far so good.
but if i change source code used only in one of the bins, it seems all the bins change, which defeats the purpose. for exampe, if i change prepare.cpp and rebuild, all the bins change.

~/code/osrm-backend$ (smartcache) sha1sum build/osrm-extract
e6c1feb4a9d07cce222c0e2578bd97d6d83be070  build/osrm-extract

~/code/osrm-backend$ (smartcache) sha1sum build/osrm-prepare
c84a0973f59c367da0ae73aba462a41c8aabedb2  build/osrm-prepare

~/code/osrm-backend$ (smartcache) sha1sum build/osrm-routed
4df97cbb6c54f73e0d4fc707fcfcf259150dde44  build/osrm-routed

~/code/osrm-backend$ (smartcache) echo " " >> prepare.cpp

~/code/osrm-backend$ (smartcache *) rake
[  2%] Built target COORDINATE
[  5%] Configuring FingerPrint.cpp
[  5%] Built target FingerPrintConfigure
[  7%] Configuring FingerPrint.cpp
[ 10%] Building CXX object CMakeFiles/FINGERPRINT.dir/Util/FingerPrint.cpp.o
[ 10%] Built target FINGERPRINT
[ 12%] Built target GITDESCRIPTION
[ 17%] Built target IMPORT
[ 20%] Built target LOGGER
[ 23%] Configuring FingerPrint.cpp
[ 25%] Building CXX object CMakeFiles/OSRM.dir/Util/FingerPrint.cpp.o
Linking CXX static library libOSRM.a
[ 48%] Built target OSRM
Linking CXX executable osrm-datastore
[ 51%] Built target osrm-datastore
Linking CXX executable osrm-extract
[ 71%] Built target osrm-extract
Scanning dependencies of target osrm-prepare
[ 74%] Building CXX object CMakeFiles/osrm-prepare.dir/prepare.cpp.o
Linking CXX executable osrm-prepare
[ 89%] Built target osrm-prepare
Linking CXX executable osrm-routed
[100%] Built target osrm-routed

~/code/osrm-backend$ (smartcache *) sha1sum build/osrm-extract
71d94fbe7934a0f7420bb312d02f903643c52454  build/osrm-extract

~/code/osrm-backend$ (smartcache *) sha1sum build/osrm-prepare
6511289a5b36fc9ccd307fdbdcd81b8604128cad  build/osrm-prepare

~/code/osrm-backend$ (smartcache *) sha1sum build/osrm-routed
1ca445e13e1c3eb016c66f989972d64cbc8eb9ca  build/osrm-routed

~/code/osrm-backend$ (smartcache *) rake
[  2%] Built target COORDINATE
[  5%] Configuring FingerPrint.cpp
[  5%] Built target FingerPrintConfigure
[  7%] Configuring FingerPrint.cpp
[ 10%] Building CXX object CMakeFiles/FINGERPRINT.dir/Util/FingerPrint.cpp.o
[ 10%] Built target FINGERPRINT
[ 12%] Built target GITDESCRIPTION
[ 17%] Built target IMPORT
[ 20%] Built target LOGGER
[ 23%] Configuring FingerPrint.cpp
[ 25%] Building CXX object CMakeFiles/OSRM.dir/Util/FingerPrint.cpp.o
Linking CXX static library libOSRM.a
[ 48%] Built target OSRM
Linking CXX executable osrm-datastore
[ 51%] Built target osrm-datastore
Linking CXX executable osrm-extract
[ 71%] Built target osrm-extract
Linking CXX executable osrm-prepare
[ 89%] Built target osrm-prepare
Linking CXX executable osrm-routed
[100%] Built target osrm-routed

[emiltin]

also worth noting is that a git commit will cause all bins to change because they include the git rev as part of the version info, all thus all caching will be invalidated except the basic osm file.
if we want to avoid this, then we need remove the git revision info from the bins as well, but i think that would be a shame.

[DennisOSRM]

The change in source must change the generated binary. For example, added or removed white spaces dont change the resulting binary.

[emiltin]

fair point, but the thing is that changing prepare.cpp causes all bins to change. why is this? is it due to the cmake fingerprint? it seems it combines md5s from multiple files:

set(OLDFILE ${SOURCE_DIR}/Util/FingerPrint.cpp)
if (EXISTS ${OLDFILE})
file(REMOVE_RECURSE ${OLDFILE})
endif()
file(MD5 ${SOURCE_DIR}/prepare.cpp MD5PREPARE)
file(MD5 ${SOURCE_DIR}/DataStructures/StaticRTree.h MD5RTREE)
file(MD5 ${SOURCE_DIR}/Util/GraphLoader.h MD5GRAPH)
file(MD5 ${SOURCE_DIR}/Server/DataStructures/InternalDataFacade.h MD5OBJECTS)

CONFIGURE_FILE( ${SOURCE_DIR}/Util/FingerPrint.cpp.in ${SOURCE_DIR}/Util/FingerPrint.cpp )

[emiltin]

actually changing whitespace does result in a recompile and a change in alls bin, as can be seen from the log above.

[DennisOSRM]

Yes, it takes the md5sum of prepare.cpp thus changing the resulting checksums.

actually changing whitespace does result in a recompile and a change in alls bin, as can be seen from the log above.

right, but this is only true for our build. We want the checksum to change when any file that contains I/O code changes.

[DennisOSRM]

I meant to say:

We want the fingerprint to change when any file that contains I/O code changes.

[emiltin]

i'm not sure i understand the cmake fingerprinting.

[emiltin]

is the fingerprint used when saving files?

[DennisOSRM]

It is used to detect when data files have been prepared with a different build, i.e. git revision. Then a warning is issued.

[emiltin]

for releases, we want to make sure all the bins use the same fingerprint, or we would get warnings about mismatched fingerprints. this is what happens now; a change in prepare.cpp causes all bins to be rebuild, with the same new fingerprint.

during development we don't want a change in fingerprint to force a rebuild of all bins, because we want to be able to change source code for one bin without causing all bins to be rebuild (and cache files to be invalidated). this implies that the different bins might use different fingerprints during tests, and that we would have to handle that somehow.

[emiltin]

the fingerprint is not linked to git revision, but rather to md5 hashes of key source code files, right?

[DennisOSRM]

You are right that the two are not directly connected. But changes in either will change the checksum of resulting binaries.

Am 17.10.2014 um 13:08 schrieb Emil Tin notifications@github.com:

the fingerprint is not linked to git revision, but rather to md5 hashes of key source code files, right?

—
Reply to this email directly or view it on GitHub.

[DennisOSRM]

for releases, we want to make sure all the bins use the same fingerprint, or we would get warnings about mismatched fingerprints. this is what happens now; a change in prepare.cpp causes all bins to be rebuild, with the same new fingerprint.

Yep, that's an inconvenience. Once the I/O code has been refactored into a single file, this will change.

during development we don't want a change in fingerprint to force a rebuild of all bins, because we want to be able to change source code for one bin without causing all bins to be rebuild (and cache files to be invalidated). this implies that the different bins might use different fingerprints during tests, and that we would have to handle that somehow.

The Fingerprint object that is linked to all binaries is automatically generated and compiled during each build. Even with no change in the source, the build script automatically takes care of this:

[  2%] Built target COORDINATE
[  5%] Configuring FingerPrint.cpp
[  5%] Built target FingerPrintConfigure
[  8%] Configuring FingerPrint.cpp
[ 10%] Building CXX object CMakeFiles/FINGERPRINT.dir/Util/FingerPrint.cpp.o
[ 10%] Built target FINGERPRINT
[ 13%] Built target GITDESCRIPTION
[ 18%] Built target IMPORT
[ 21%] Built target LOGGER
Linking CXX static library libOSRM.a
[ 45%] Built target OSRM
Linking CXX executable osrm-datastore
[ 48%] Built target osrm-datastore
Linking CXX executable osrm-extract
[ 70%] Built target osrm-extract
Linking CXX executable osrm-prepare
[ 89%] Built target osrm-prepare
Linking CXX executable osrm-routed
[100%] Built target osrm-routed

As soon as the interface between files changes, i.e a file that loads or stores data from/to disk, all binaries are relinked with an updated fingerprint.