Delta Update mechanism

[mavrothal]

As I mentioned elsewhere I think that deltas could be used to update running puppies without any increase of the savefile/folder.
So below are 3 little scripts (2 & 3 could be fused) to generate SFSs, initrd and vmlinuz deltas from ISOs (create_deltas_from_iso.sh), check for and download tarballs with the appropriate deltas (delta_update.sh) and finally update and patch your files with the downloaded deltas (apply_delta_to_puppy). Repeat till no more updates.
I know that people are not very keen on updates, thus no pull request. On the other hand "discussions" in the forum are really frustrating specially for such a thing that is only for puppy builders.
So here they are for keeps, modifications and maybe discussion.

#!/bin/sh
# create_deltas_from_iso.sh

UPDNAME="$1" # MUST use DISTRO_FILE_PREFIX
UPDVER="$2" # version number

# Or use config file if present
[ -f root/.config/delta_updates] && . /root/.config/delta_updates

[ "$UPDNAME" = "" -o "$UPDVER" = '' ] && echo provide update name and version && exit 1

[ ! "$BASE" ] && BASE=/mnt/sr0 # Can be changed to anything but BASE in the ISO of puppy that we 
[ ! "$TARGET" ] && TARGET=/mnt/sr1 # want to update and TARGET the ISO of the update

[ -z "$(ls -a $BASE)" -o -z "$(ls -A $TARGET)" ] && echo load the ISOs && exit 1

rm -rf "$UPDNAME"-"$UPDVER"
mkdir -p "$UPDNAME"-"$UPDVER"

for F in $(ls "$BASE"/{vmlinuz,initrd.gz,*.sfs})
	do
	F=$(basename $F)
	MD1="$(md5sum $BASE/$F | cut -f 1,4 -d '/')"
	MD2="$(md5sum $TARGET/$F | cut -f 1,4 -d '/')"
	echo "$MD1" >> "$UPDNAME"-"$UPDVER"/"$UPDNAME-$UPDVER"-target.md5.txt
	echo "$MD2" >> "$UPDNAME"-"$UPDVER"/"$UPDNAME"-update-"$UPDVER".md5.txt
	if [ "$MD1" != "$MD2" ];then
		xdelta3 -D -R -e -s "$BASE"/"$F" "$TARGET"/"$F" "$UPDNAME-$UPDVER"/"$F"_delta
	fi
done

sync

[ $? -ne 0 ] && rm -rf "$UPDNAME"-"$UPDVER" && echo deltas faied

tar czf "$UPDNAME"-"$UPDVER".tar.gz "$UPDNAME"-"$UPDVER"/

#!/bin/sh
# delta_update.sh 

[ -f root/.config/delta_updates] && . /root/.config/delta_updates
. /root/.packages/DISTRO_PET_REPOS
. /initrd/DISTRO_SPECS
. /etc/rc.d/PUPSTATE

# Check for frugal in HD or USB
[ "$PUPMODE" != "12" -a "$PUPMODE" != "13" -a "$PUPMODE" != "6" -a "$PUPMODE" != "7" ] \
	&& echo only for frugal installs && exit 1

# Check for delta updates
if [ -f /etc/"$DISTRO_FILE_PREFIX"-update-*.md5.txt ]; then
	VERNUMB=$(echo "$(basename /etc/upupbb-update-*.md5.txt)"| cut -f1 -d '.' | cut -f3 -d '-')
	NVER=$(expr $VERNUMB + 1)
	LOOKFOR="$DISTRO_FILE_PREFIX"-"$NVER".tar.gz
else
	LOOKFOR="$DISTRO_FILE_PREFIX"-1.tar.gz
fi

[ ! "$REPO" ] && REPO=$DISTRO_FILE_PREFIX
[ ! "$URL" ] && URL="http://distro.ibiblio.org/puppylinux"
if [ "$(wget -4 -t 2 -T 20 --waitretry=20 --spider -S "$URL"/pet_packages-"$REPO"/"$LOOKFOR" \
	2>&1 | grep 'Remote file exists')" != "" ];then
	wget -c "$URL"/pet_packages-"$REPO"/"$LOOKFOR" -P /tmp
	[ $? -eq 0 ] && sync || exit 1
	/apply_delta_to_puppy.sh /tmp/"$LOOKFOR"
	[ $? -eq 0 ] && sync || exit 1
else
	exit 0
fi

# Rerun to get the next update if available
nohup delta_update.sh &
exit 0

#!/bin/sh
#apply_delta_to_puppy

[ -f root/.config/delta_updates] && . /root/.config/delta_updates
. /initrd/DISTRO_SPECS
. /etc/rc.d/PUPSTATE

# Check for frugal in HD or USB
[ "$PUPMODE" != "12" -a "$PUPMODE" != "13" -a "$PUPMODE" != "6" -a "$PUPMODE" != "7" ] \
	&& echo only for frugal installs && exit 1


[ ! "$UPDPATH" ] && UPDPATH="$1"
[ "$UPDPATH" = "" ] && echo provide an update tarball && exit 1
UPDFILE=$(basename "$1")
UPDNAME=${UPDFILE%%-*}
UPDVER=$(echo $UPDFILE | tr -dc '0-9')
INSTP="$PUPSFS"
INSTP=${INSTP##*,}
INSTP=${INSTP%/*}
INSTALPATH=/mnt/home"$INSTP"

# Check puppy
[ "$DISTRO_FILE_PREFIX" != "$UPDNAME" ] && echo not appropriate puppy && exit 1

# Check versio
if [ -f /etc/upupbb-update-*.md5.txt ]; then
	INSTVER=$(echo "$(basename /etc/upupbb-update-*.md5.txt)"| cut -f1 -d '.' | cut -f3 -d '-')
	[ "$INSTVER" -ge "$UPDVER" ] && echo update too old && exit 1
fi

tar xzf "$1" -C /tmp/

# final check
for L in $(cat /tmp/"$UPDNAME"-"$UPDVER"/"$UPDNAME"-"$UPDVER"-target.md5.txt | cut -f3 -d ' ')
do
	MD5=$(md5sum "$INSTALPATH"/"$L" | cut -f1 -d ' ')
	if [ "$(grep "$MD5" /tmp/"$UPDNAME"-"$UPDVER"/"$UPDNAME"-"$UPDVER"-target.md5.txt)" = "" ];then
		echo not appropriate puppy && exit 1
	fi
done

# Apply deltas
rm -rf "$INSTALPATH"/old_"$UPDNAME"_files
mkdir -p "$INSTALPATH"/old_"$UPDNAME"_files
for F in $(ls /tmp/"$UPDNAME"-"$UPDVER"/*_delta)
do
	FL=$(basename $F)
	FL=${FL%_*}
	mv "$INSTALPATH"/"$FL" "$INSTALPATH"/old_"$UPDNAME"_files/
	xdelta3 -D -R -d -s "$INSTALPATH"/old_"$UPDNAME"_files/"$FL" "$F" "$INSTALPATH"/"$FL"
done

if [ $? -ne 0 ]; then
	cp -a --remove-destination "$INSTALPATH"/old_"$UPDNAME"_files/* "$INSTALPATH"/
	echo upate failed
	exit 1
fi

# Check once more that we are OK
for L in $(cat /tmp/"$UPDNAME"-"$UPDVER"/"$UPDNAME"-update-"$UPDVER".md5.txt | cut -f3 -d ' ')
do
	MD5=$(md5sum "$INSTALPATH"/"$L" | cut -f1 -d ' ')
	if [ "$(grep "$MD5" /tmp/"$UPDNAME"-"$UPDVER"/"$UPDNAME"-update-"$UPDVER".md5.txt)" = "" ];then
		cp -a --remove-destination "$INSTALPATH"/old_"$UPDNAME"_files/* "$INSTALPATH"/
		echo upate failed
		exit 1
	fi
done

if [ $? -eq 0 ];then
	rm -f /etc/"$UPDNAME"-update-*.md5.txt 2>/dev/null
	cp /tmp/"$UPDNAME"-"$UPDVER"/"$UPDNAME"-update-"$UPDVER".md5.txt /etc/
fi
echo Update done!
exit 0

They work fine as far as I could test (download was tested against "bogus" files).
Obviously could be polished if ever integrated to puppy.

[mavrothal]

could be polished

Here is a more "user friendly" delta_update.sh (script 2 above).
And I should leave it to that.

#!/bin/sh
# delta_update.sh

[ -f root/.config/delta_updates] && . /root/.config/delta_updates
. /root/.packages/DISTRO_PET_REPOS
. /initrd/DISTRO_SPECS
. /etc/rc.d/PUPSTATE

# Check for frugal in HD or USB
[ "$PUPMODE" != "12" -a "$PUPMODE" != "13" -a "$PUPMODE" != "6" -a "$PUPMODE" != "7" ] \
	&& /usr/lib/gtkdialog/box_ok "$(gettext 'Wrong pupmode')" error "$(gettext 'Can only update frugal installs on HD or USB') ($(gettext 'for now...'))" && exit 1

# Check for internet connection
[ "$(ifconfig | grep -iE 'bcast|broadcat')" = "" ] && /usr/lib/gtkdialog/box_ok "$(gettext 'No Internet')" error "$(gettext 'It would appear that you are not connected to the internet.
Please make sure that there is an active internet connection')" && exit 1

if [ "$(grep delta_update /etc/anacrontab 2>/dev/null)" = "" -a \
	! -f /root/.config/autostart/delta_check.sh ]; then
	#Some info if no auto setup yet
	/usr/lib/gtkdialog/box_ok "$(gettext 'Delta Upadte')" info "$(gettext 'This program will check the puppylinux distribution site for updates of your ') $DISTRO_NAME $DISTRO_VERSION $(gettext ' puppy.
Please make sure you have an internet connection and do not close your computer till the program finishes.')"

	/usr/lib/gtkdialog/box_yesno --yes-first  "$(gettext 'Background Update')" "$(gettext 'Do you want to set up a background update check once a week?')"
	if [ $? -eq 0 ];then
		# Check if we have anacron (not in puppies). Not extensivelly tested. 
		# Relies on user to setup anacron (for now)
		ANACRON=$(which anacron)
		if [ "$ANACRON" -a "$(grep delta_update /etc/anacrontab)" = "" ];then
			# set to check every week
			echo -e '7\t15\ttest.daily\tdelta_update.sh' >>  /etc/anacrontab
			yaf-splash -placement top -bg orange -timeout 20 -text "$(gettext 'You can stop background updates deletting the ') delta_update.sh $(gettext ' containing line of the') /etc/anacrontab $(gettext ' text file.')" &
		else
			cat << EOF >> /root/.config/autostart/delta_check.sh
#!/bin/sh
sleep 900 # run 15 min after boot
if [ -f /var/local/upgrade_check ]; then
   DAY=\$(expr \$(cat /var/local/upgrade_check) + 7)
   [ "\$DAY" -lt "\$((\$(date +%s)/86400))" ] && delta_update.sh
fi
EOF
			chmod 755 /root/.config/autostart/delta_check.sh
			yaf-splash -placement top -bg orange -timeout 20 -text "$(gettext 'You can stop background updates deletting or inactivating the ') /root/.config/autostart/delta_check.sh $(gettext ' script.')" &
		fi
	fi	
fi

yaf-splash -placement bottom -bg green -timeout 5 -text "$(gettext 'Checking for') $DISTRO_NAME  $DISTRO_VERSION $(gettext ' updates...')" &

# Check for delta updates
if [ -f /etc/"$DISTRO_FILE_PREFIX"-update-*.md5.txt ]; then
	VERNUMB=$(echo "$(basename /etc/upupbb-update-*.md5.txt)"| cut -f1 -d '.' | cut -f3 -d '-')
	NVER=$(expr $VERNUMB + 1)
	LOOKFOR="$DISTRO_FILE_PREFIX"-"$NVER".tar.gz
else
	LOOKFOR="$DISTRO_FILE_PREFIX"-1.tar.gz
fi

[ ! "$REPO" ] && REPO=$DISTRO_FILE_PREFIX
[ ! "$URL" ] && URL="http://distro.ibiblio.org/puppylinux"
if [ "$(wget -4 -t 2 -T 20 --waitretry=20 --spider -S "$URL"/pet_packages-"$REPO"/"$LOOKFOR" \
	2>&1 | grep 'Remote file exists')" != "" ];then
	wget -c "$URL"/pet_packages-"$REPO"/"$LOOKFOR" -P /tmp
	if [ $? -ne 0 ]; then
		/usr/lib/gtkdialog/box_ok "$(gettext 'Failed download')" error "$(gettext 'Failed to download the ') $LOOKFOR  $(gettext ' update.
The file is present, so try again latter.')"
		sync
		exit 1
	fi
	/apply_delta_to_puppy.sh /tmp/"$LOOKFOR"
	if [ $? -eq 0 ];then
		sync
		/usr/lib/gtkdialog/box_ok "$(gettext 'Updated')" info "$DISTRO_NAME  $(gettext ' was updated.')"
	else
		/usr/lib/gtkdialog/box_ok "$(gettext 'Failed to update')" error "$(gettext 'Failed to update your ') $DISTRO_NAME  $(gettext ' installation.
Please report the problem to the puppy builder.')"
		exit 1
	fi
else
	/usr/lib/gtkdialog/box_ok "$(gettext 'Up-to-date')" info "$(gettext 'Your ') $DISTRO_NAME  $(gettext ' is uo to date.')"
	# Stamp the date of last check
	[ ! "$ANACRON" ] && echo $(($(date +%s)/86400)) > /var/local/upgrade_check
	exit 0
fi

# Rerun to get the next update if available
nohup delta_update.sh &
exit 0

[wdlkmpx]

This is a good idea. I haven't tested this, but this certainly should be the standard way to produce incremental updates

There are also issues with big changes that will cause some damage to current installations, like this one #1224

[mavrothal]

I haven't tested this

I tested both the delta generation and tarball installation with peebee's upupbb +N series and is fine.
The server part is not fully tested as there are no relevant packages in any server, but is ok with other packages.

If this is to ever go through, it may need to also generate an md5sum file for the tarball ( md5sum "$UPDNAME"-"$UPDVER".tar.gz > "$UPDNAME"-"$UPDVER".md5.txt ) and place it in the server, so the download can be checked against the md5 file ( LOOKFORMD5=${LOOKFOR%%.*}.md5.txt; wget -c "$URL"/pet_packages-"$REPO"/"$LOOKFORMD5" -P /tmp; [ "$(grep "$(md5sum /tmp/"$LOOKFOR")" /tmp/"$LOOKFORMD5" )" = "" ] && rm -f /tmp/"$LOOKFOR" && echo download failed ) before installation.

[mavrothal]

Here is a pet with bit more verbal and tested version of the 3 scripts that can generate and install delta-based updates
(Remove fake zip extension)

delta_update-1.pet.zip