[question] maintain patchset without requiring rebuilding.

[avih]

I'm maintaining a small patchset at my local MXE repo and typically rebasing it on top on MXE master.

Till recently it was mostly for ffmpeg master, but I then added also mingw 5.0.

It turns out that after I pull+rebase now, it thinks mingw is out of date and rebuilds everything from scratch. I didn't try it enough times yet to be able to say that it always happens though.

If this is sort of by design (because mingw-w64.mk got touched from the rebase), is there any way to somehow work around it? I'd think that as long as the content of the file didn't change since the last make, it shouldn't be considered outdated, but I don't know if/how that could be achieved.

Help?

[starius]

touch usr/installed/* usr/*/installed/* helps normally, but it is not reliable, as it can touch a dependency after touching a follower.

Another approach I use is make build-only-foo_i686-w64-mingw32.static.

If it doesn't work, I rebuild everything. It takes less then hour to rebuild gcc and it consumes most time for me.

[starius]

By the way, can you isolate your patches to a plugin to avoid rebasing?

[avih]

Thanks will try both suggestions (build and plugins). Will have to check out how the plugins thingy works.

Isn't there some option/hack for make to identify a file as "out of date" based on a checksum of that file rather than its date?

[starius]

Isn't there some option/hack for make to identify a file as "out of date" based on a checksum of that file rather than its date?

I don't know about such an option. It would require to store checksums of all files. One more problem: MXE creates empty files in installed/, so their timestamp is the only data which can be used to determine if they are out-of-date.

By the way, take a looks at build-pkg. It creates git repo in usr and updates it after each package built. Prior to building new package, it creates a new git branch on top of branches representing dependencies. And it touches all files needed to avoid rebuilding.

[avih]

It would require to store checksums of all files. One more problem: MXE creates empty files in installed/, so their timestamp is the only data which can be used to determine if they are out-of-date.

Well, it's likely that such local patches would mostly touch make files, and so the files at the installed dir could contain the checksum of the makefile to which it corresponds. I'll try to play with that and also look at build-pkg. Thanks.

[tonytheodore]

touch usr/installed/*

make -t [...] is much more reliable.

By the way, can you isolate your patches to a plugin to avoid rebasing?

+1, unless the changes are small, in which case an include path/to/mychanges.mk at the bottom would play nicely with both make and git.

@avih, I still need to document plugins but browse through the existing ones to get a feel for what they do - note they can be entirely local.

Isn't there some option/hack for make to identify a file as "out of date" based on a checksum of that file rather than its date?

It would require to store checksums of all files.

Indeed, see https://www.cmcrossroads.com/article/rebuilding-when-files-checksum-changes for an implementation from the author of the GNU Make Standard Library. It gets very complicated very quickly - it's one of the rare cases where I don't think make is well suited, even if it can be done.

[avih]

Indeed, see https://www.cmcrossroads.com/article/rebuilding-when-files-checksum-changes

Interesting. So I'm not the first to want checksum instead of date as a change indication ;) A bit weird that no proper solution exists so far for this.

[avih]

+1, unless the changes are small, in which case an include path/to/mychanges.mk at the bottom would play nicely with both make and git.

Yeah, for a long while I was only modifying ffmpeg.mk, and recently also mingw-w64.mk (for newer version of it, which also meant removing the mingw patch since it's not needed anymore).

Where should I put this include statement?

[tonytheodore]

Where should I put this include statement?

I was thinking at the bottom of the package you're modifying, but on reflection, it isn't such a good idea. There's no easy way to toggle it on/off and it's a layer of unnecessary indirection.

[avih]

FWIW, I've solved it with the following script, and invoking it unconditionally at the beginning of the Makefile on src/* like so (after TOP_DIR is defined):

HASHSTAMP_DUMMY := $(shell (>&2 echo "[check timestamps]"))
HASHSTAMP_DUMMY := $(shell (>&2 $(TOP_DIR)/tools/hashstamp.sh $(TOP_DIR)/src/*))
HASHSTAMP_DUMMY := $(shell (>&2 echo "[done timestamps]"))

It seems to be working well for me so far.

In a nutshell: for every file it's invoked on, it creates another file with the hash of the original and the date of the original, but if such file already exist and contains the same hash as the original, then it copies (restores) its timestamp (if modified). Basically, make would only see a newer timestamp if the hash has changed since the last invocation of make. This seems like a usefully generic solution to this problem.

FWIW, there are about 800 files at src/* and it takes about 2s to complete on every make. I can definitely live with that.

I put it as tools/hashstamp.sh:

#!/usr/bin/env bash

STAMPSSUM="md5sum"
STAMPSDIR=".hashstamps"
v=$HASHSTAMPS_VERBOSE  # 0 - quiet, 1 (def) - updates only, 2 - verbose
[ -z $v ] && v=1

updatestamp() {
  while ! [ -z "$1" ]; do
    if ! [ -d $1 ]; then  # skip directories quietly
      if ! [ -e $1 ]; then
        >&2 echo "Cannot find '$1'. Skipping."
        shift
        continue
      fi
      local sum="$($STAMPSSUM $1 | cut -d ' ' -f1)"
      local stampsdir="$(dirname $1)/$STAMPSDIR"
      local stampfile="$stampsdir/$(basename $1).hashstamp"
      local stamp=$( [ -e $stampfile ] && cat $stampfile || echo '' )

      if [ "$sum" == "$stamp" ]; then
        if [ $(stat -c "%Y" $1) != $(stat -c "%Y" $stampfile) ]; then
          [ $v -ge 1 ] && echo "'$1' - Restoring timestamp from '$stampfile'."
          touch -r $stampfile $1
        else
          [ $v -ge 2 ] && echo "'$1' - Not touching."
        fi
      else
        if ! ( [ -d $stampsdir ] || mkdir $stampsdir >/dev/null 2>&1 ); then
          >&2 echo "Cannot create dir '$stampsdir' for '$1'. Skipping."
          shift
          continue
        fi
        [ $v -ge 1 ] && echo "'$1' - Creating/updating stamp at '$stampfile'."
        echo "$sum" > $stampfile
        touch -r $1 $stampfile
      fi

    fi

    shift
  done
}

updatestamp "$@"