Merge pull request #1 from pljones/feature/initial-versions

README.md

@@ -1,5 +1,89 @@
 # Jamulus Jam Exporter
 
+_General note:_
+
+The scripts here are derived from what I use on my own server - they have been altered to be less dependent
+on the way I run my server and hence are not scripts I actually run.  Which means my changes are subject to bugs.
+
 This comprises two scripts:
 * A bash script to monitor the Jamulus recording base directory for new recordings
 * A bash script to apply some judicious rules and compression before uploading the recordings offsite
+
+## Systemd Service
+Also supplied is a systemd service file to start the monitor script:
+* `src/systemd/inotify-publisher.service`
+
+This contains the path to the monitor script in the `ExecStart=` line and the `User=` and `Group=` lines need to
+match how you usually run Jamulus.
+
+## inotify-publisher.sh monitoring script
+The configuration section at the top has the following:
+* `JAMULUS_ROOT=/opt/Jamulus`
+* `JAMULUS_RECORDING_DIR=${JAMULUS_ROOT}/run/recording`
+* `JAMULUS_STATUSPAGE=${JAMULUS_ROOT}/run/status.html`
+* `PUBLISH_SCRIPT=${JAMULUS_ROOT}/bin/publish-recordings.sh`
+* `NO_CLIENT_CONNECTED="No client connected"`
+
+You may need to edit more than just `JAMULUS_ROOT` - adjust to suit.
+I'm not sure if the status file entry `NO_CLIENT_CONNECT` gets translated - if so, the local value is needed here.
+
+The script uses one program that you might not have installed by default, `inotifywait`.
+* http://inotify-tools.sourceforge.net/
+
+I would expect your distribution makes this available.
+
+
+## publish-recordings.sh prepare and upload script
+**NOTE** PLEASE read and understand, at least basically, what this does _before_ using it.  It makes _destructive edits_
+to recordings that you might not want.  It was written to do what I needed and is provided for people to have a base to
+work from, _not_ as a working solution to your needs.
+
+### What it does
+Given the right `RECORDING_DIR`, this iterates over all subdirectories, looking for Reaper RPP files.
+(Currently, the Audacity LOF files are ignored and become wrong.)
+
+The logical processing is as follows.
+
+For each RPP file, the WAV files are examined to determine their audio length and (EBU) volume.  Where the file
+is considered "too short" or "too quiet", it is removed (deleted on disk and edited out of the RPP file).
+Retained files then have audio compression applied, updating the RPP file with the new name (i.e. WAV -> OPUS).
+Any _track_ that now has no entries is also removed.  If the project has no tracks, the recording directory is deleted.
+
+After the above processing, any remaining recording directory gets zipped (without the broken LOF)
+and uploaded to `RECORDING_HOST_DIR`.
+
+### Configuration
+
+There is one main dependency here: the FFMpeg suite - both `ffprobe` and `ffmpeg` itself are used.
+* https://ffmpeg.org/
+
+It also uses `zip`.
+* http://infozip.sourceforge.net/
+
+Most distributions provide versions that will be adequate.
+
+The configuration section here is simpler:
+* `RECORDING_DIR=/opt/Jamulus/run/recording`
+* `RECORDING_HOST_DIR=drealm.info:html/jamulus/`
+
+The script off-sites the recordings - `RECORDING_HOST_DIR` is the target.  It uses `scp` as the user running the script.
+If run from `inotify-publisher.sh` under the systemd service, that will be the `User=` user.  Make sure you have installed
+that user's public key in your hosting provider's `authorized_keys` (using the expected key type).
+
+
+## recover-recording.sh
+Also included is a "recovery mode" script.  If the RPP file isn't generated when a recording is in progress, this generates
+the file.  It's essential to tell it whether you have 64 or 128 sample frames or the timing will go horribly wrong.
+The server name is just for the namespace UUID creation.
+* `JAMULUS_SERVERNAME=jamulus.drealm.info`
+* `JAMULUS_OPTS=("-F")`
+
+Note the format of `JAMULUS_OPTS` is a bash array, as it's taken from my server run configuration.
+You need to use `-F` here, if you use 64 sample frames -- `--fastupdate` isn't supported.
+
+The script does have a `--help` option.  Run it from the directory containing the failed recording.
+It emits the RPP to stdout - you can redirect it to a file of your choosing.
+
+Note that `inotify-publisher.sh` will have "skipped" this directory in its monitoring, as no RPP existed.
+If no recording is currently in progress and you want to make inotify-publisher.sh notice the repair, create and then
+delete a directory in the base recording directory.

src/bash/inotify-publisher.sh

@@ -0,0 +1,61 @@
+#!/bin/bash -e
+
+#    inotify-publisher.sh Trigger jam publishing automatically
+#    Copyright (C) 2020 Peter L Jones <peter@drealm.info>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+#
+#    See LICENCE.txt for the full text.
+
+# Configuration
+JAMULUS_ROOT=/opt/Jamulus
+JAMULUS_RECORDING_DIR=${JAMULUS_ROOT}/run/recording
+JAMULUS_STATUSPAGE=${JAMULUS_ROOT}/run/status.html
+PUBLISH_SCRIPT=${JAMULUS_ROOT}/bin/publish-recordings.sh
+NO_CLIENT_CONNECTED="No client connected"
+
+# Most recent processing check
+MOST_RECENT=0
+
+# Do not return until a new jamdir exists in the recording dir
+wait_for_new_jamdir () {
+	while [[ ${MOST_RECENT} -ge $(date -r "${JAMULUS_RECORDING_DIR}" "+%s")
+		|| -z $(find "${JAMULUS_RECORDING_DIR}" -mindepth 1 -type d -prune) ]]
+	do
+		inotifywait -q -e create -e close_write "${JAMULUS_RECORDING_DIR}"
+	done
+	true
+}
+
+# Do not return until the server has no connections
+wait_for_quiet () {
+	# wait until the status page exists
+	while ! test -f "${JAMULUS_STATUSPAGE}"
+	do
+		inotifywait -q -e create -e close_write "${JAMULUS_STATUSPAGE}"
+	done
+
+	# wait until no one connected
+	while ! grep -q "${NO_CLIENT_CONNECTED}" "${JAMULUS_STATUSPAGE}"
+	do
+		inotifywait -q -e close_write "${JAMULUS_STATUSPAGE}"
+	done
+	true
+}
+
+while wait_for_new_jamdir && wait_for_quiet
+do
+	MOST_RECENT=$(date -r "${JAMULUS_RECORDING_DIR}" "+%s")
+	"${PUBLISH_SCRIPT}" || true
+done

src/bash/publish-recordings.sh

@@ -0,0 +1,113 @@
+#!/bin/bash -e
+
+#    publish-recordings.sh Prepare and upload recordings off-site
+#    Copyright (C) 2020 Peter L Jones <peter@drealm.info>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+#
+#    See LICENCE.txt for the full text.
+
+RECORDING_DIR=/opt/Jamulus/run/recording
+RECORDING_HOST_DIR=drealm.info:html/jamulus/
+
+cd "${RECORDING_DIR}"
+
+find -maxdepth 1 -type d -name 'Jam-*' | sort | \
+while read jamDir
+do
+	rppFile="${jamDir#./}.rpp"
+	[[ -f "${jamDir}/${rppFile}" ]] || continue
+	(
+		cd "$jamDir"
+
+		find -maxdepth 1 -type f -name '*.wav' | sort | while read wavFile
+		do
+			lra=0
+			integrated=0
+			removeWaveFromRpp=false
+
+			duration=$(ffprobe -v 0 -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "$wavFile")
+			if [[ ${duration%.*} -lt 60 ]]
+			then
+				removeWaveFromRpp=true
+				echo -n ''
+			else
+				opusFile="${wavFile%.wav}.opus"
+				declare -a stats=($(
+					nice -n 19 ffmpeg -y -hide_banner -nostats -nostdin -i "${wavFile}" -af ebur128 -b:a 160k "${opusFile}" 2>&1 | \
+						grep '^ *\(I:\|LRA:\)' | { read x y x; read x z x; echo $y $z; }
+				))
+[[ ${#stats[@]} -eq 2 ]]
+				integrated=${stats[0]}
+				lra=${stats[1]}
+				echo "$duration $lra $integrated" | awk '{ if ( $1 >= 60 && ( $2 > 6 || $3 > -48 ) ) exit 0; exit 1 }' || {
+					rm "${opusFile}"
+					removeWaveFromRpp=true
+				}
+			fi
+
+			if $removeWaveFromRpp
+			then
+				echo "Removed ${wavFile} - duration $duration, lra $lra, integrated $integrated"
+				# Magic sed command to remove an item from a track with a particular source wave file
+				sed -e '/^    <ITEM */{N;N;N;N;N;N;N;N;N;N;N;N}' \
+					-e "\%^ *<SOURCE WAVE\n *FILE *[^>]*${wavFile}\"\n *>\n%Md" \
+					"${rppFile}" > "${rppFile}.tmp" && \
+					mv "${rppFile}.tmp" "${rppFile}"
+			else
+				echo "Kept ${opusFile} - duration $duration, lra $lra, integrated $integrated"
+			fi
+
+			rm "$wavFile"
+		done
+
+		# Magic sed command to remove empty tracks
+		sed -e '/^  <TRACK {/{N;N;N}' -e '/^ *<TRACK\([^>]\|\n\)*>$/d' \
+			"${rppFile}" > "${rppFile}.tmp" && \
+			mv "${rppFile}.tmp" "${rppFile}"
+
+		if grep -q 'ITEM' "${rppFile}"
+		then
+			# Replace any remaining references to WAV files with OPUS compressed versions
+			sed -e 's/\.wav/.opus/' -e 's/WAVE/OPUS/' \
+				"${rppFile}" > "${rppFile}.tmp" && \
+				mv "${rppFile}.tmp" "${rppFile}"
+			# Note, Audacity won't like the OPUS files...
+		else
+			# As no items were left, remove the project
+			echo "Removed ${rppFile}"
+			rm "${rppFile}"
+			echo "Removed ${rppFile/rpp/lof}"
+			rm "${rppFile/rpp/lof}"
+		fi
+
+	)
+
+	if [[ "$(cd "${jamDir}"; echo *)" == "*" ]]
+	then
+		rmdir "${jamDir}"
+	else
+		zip -r "${jamDir}.zip" "${jamDir}" -i '*.opus' '*.rpp' && {
+			rm -r "${jamDir}"
+			i=10
+			while [[ $i -gt 0 ]] && ! scp -o ConnectionAttempts=6 "${jamDir}.zip" ${RECORDING_HOST_DIR}
+			do
+				(( i-- ))
+				sleep $(( 11 - i ))
+			done
+			[[ $i -gt 0 ]]
+		}
+	fi
+
+done

src/bash/recover-recording.sh

@@ -0,0 +1,120 @@
+#!/bin/bash -e
+if [[ "$1" == "--help" ]]
+then
+cat <<-HELPMSG
+	Recreate a Reaper RPP project file from a Jamulus recording directory.
+
+	$(basename "$0") [--help]
+	  --help  display this help message
+
+	The intent of this script is to take an existing collection of Jamulus recorded
+	WAVE files and generate a Reaper RPP project file that matches the one the server
+	should have created.  It may sometimes be needed, for example when the server fails
+	to terminate the recording correctly (as that is when the RPP file gets written).
+
+	The script should be run from the directory containing the "failed" recording.
+	It outputs the RPP file on stdout - redirect this to your chosen project filename.
+
+	It requires  /etc/init.d/Jamulus  to be the provided start up script, so it can
+	read the configuration settings.
+
+HELPMSG
+	exit 0
+fi
+
+# Set the variables
+JAMULUS_SERVERNAME=jamulus.drealm.info
+JAMULUS_OPTS=("-F")
+
+siteNamespace=$(uuidgen -n @url -N jamulus:${JAMULUS_SERVERNAME} --sha1)
+projectName=$(basename "$(pwd)")
+
+if [[ "${projectName:0:4}" != "Jam-" ]]
+then
+	echo "Run this script from the failed recording directory." >&2
+	exit 1
+fi
+
+if [[ "$(echo *.wav)" == "*.wav" ]]
+then
+	echo "There are no WAVE files in this directory from which to create a project." >&2
+	exit 1
+fi
+
+projectNamespace=$(uuidgen -n $siteNamespace -N "${projectName}" --sha1)
+projectDate=$( p=${projectName#Jam-}; echo ${p:0:4}-${p:4:2}-${p:6:2} ${p:9:2}:${p:11:2}:${p:13:2}.${p:15} )
+frameRate=$([[ ${JAMULUS_OPTS[@]} =~ " -F" ]] && echo 64 || echo 128)
+
+secondsAt48K () {
+	echo $1 | awk '{ printf "%.14f\n", $1 / 48000; }'
+}
+
+echo '<REAPER_PROJECT 0.1 "5.0"' $(date -d "$projectDate" -u '+%s')
+echo ' RECORD_PATH "" ""'
+echo ' SAMPLERATE 48000 0 0'
+echo ' TEMPO 120 4 4'
+
+iid=0
+prevIP=''
+for x in $(ls -1 *.wav | sort -t- -k2)
+do
+
+	# Some initial cleaning up
+	if [[ -z "$x" ]]
+	then
+		rm "$x"
+		continue
+	fi
+	# If ffmpeg/ffprobe and sox cannot cope then give up
+	if ! ffprobe -v error -i "$x" >/dev/null 2>&1
+	then
+		if sox --ignore-length "$x" "FIXED-$x" >/dev/null 2>&1
+		then
+			mv "FIXED-$x" "$x"
+		else
+			rm -f "$x" "FIXED-$x"
+			continue
+		fi
+	fi
+
+	IP=${x#*-}
+	IP=${IP%%-*}
+	if [[ "$prevIP" != "$IP" ]]
+	then
+		iidt=0
+		if [[ "$prevIP" != "" ]]
+		then
+			echo '  NAME '$trackName
+			echo ' >'
+		fi
+		prevIP="$IP"
+		echo ' <TRACK {'$(uuidgen -n $projectNamespace -N $IP --sha1)'}'
+		echo '  TRACKID {'$(uuidgen -n $projectNamespace -N $IP --sha1)'}'
+		trackName="${x%-*-*.wav}"
+	else
+		[[ "${trackName:0:5}" == "____-" ]] && trackName="${x%-*-*.wav}"
+	fi
+	(( iid++ )) || true
+	(( iidt++ )) || true
+	echo '  <ITEM'
+	echo '   FADEIN 0 0 0 0 0 0'
+	echo '   FADEOUT 0 0 0 0 0 0'
+	filePos="${x#*-*-}"
+	filePos="${filePos%-*.wav}"
+	echo '   POSITION '$(secondsAt48K $(( $filePos * $frameRate )) )
+	echo '   LENGTH '$(secondsAt48K $(soxi -s "$x"))
+	echo '   IGUID {'$(uuidgen -n $projectNamespace -N $IP --sha1)'}'
+	echo '   IID '$iid
+	echo '   NAME '$IP' ('$iidt')'
+	echo '   GUID {'$(uuidgen -n $projectNamespace -N "$IP-$iidt" --sha1)'}'
+	echo '   <SOURCE WAVE'
+	echo '    FILE "'$x'"'
+	echo '   >'
+	echo '  >'
+done
+if [[ "$prevIP" != "" ]]
+then
+	echo '  NAME '$trackName
+	echo ' >'
+fi
+echo '>'

src/systemd/inotify-publisher.service

@@ -0,0 +1,12 @@
+[Unit]
+Description=inotifywait-based Jamulus jam publisher
+; Needs local file systems mounted but that's about it - covered by default basic.target
+
+[Service]
+Type=simple
+ExecStart=/opt/Jamulus/bin/inotify-publisher.sh
+User=Jamulus
+Group=Jamulus
+
+[Install]
+WantedBy=multi-user.target