Auto backup of past long-term graphs

[Quaxo76]

I've written three short scripts that save the small graphs to three folders (the 1-day at the end of the day, the 1-week at the end of the week, and the 1-month at the end of the month) to keep them for future reference. They are activated using cron. Maybe others could be interested in this, and it could be made an option in the software?

My scripts are really crude, as I'm not at all an expert at this. The 24-hour graph is backed up at 23:59 each day, the 7-day at 23:58 on Sundays, and the monthly at 23:57 on the 30th day of the month. Not the 31st because of course not all months have 31 days, and those months would be skipped. Now that I think of it, with my script February probably gets skipped too, so this should be looked into.

It would have been better to save the files at 0:00 of the first day of the month (and the same for the other graphs) but then it would already be the following month, and since I use the "date" command to time-code the files, then the file name would not match the period of time it refers to.

Anyway, I created three folders under .temperature:

cd ~/.temperature
mkdir daily
mkdir weekly
mkdir monthly

Then I created the three scripts, named daily_copy.sh, weekly_copy.sh, monthly_copy.sh. They just copy the relevant file to the correct folder (date-coding it in the filename).

daily_copy.sh:

#!/bin/bash
cp ~/.temperature/temperature-1-days.png ~/.temperature/daily/`date +%Y_%m_%d_`1_day.png

weekly_copy.sh:

#!/bin/bash
cp ~/.temperature/temperature-7-days.png ~/.temperature/weekly/`date +%Y_%m_%d_`7_days.png

monthly_copy.sh:

#!/bin/bash
cp ~/.temperature/temperature-30-days.png ~/.temperature/monthly/`date +%Y_%m_%d_`30_days.png

I then made them executable:

sudo chmod +x daily_copy.sh
sudo chmod +x weekly_copy.sh
sudo chmod +x monthly_copy.sh

Then I edited the file /etc/crontab and added the following lines:

59 23   * * *   pi      /home/pi/.temperature/daily_copy.sh
58 23   * * 7   pi      /home/pi/.temperature/weekly_copy.sh
57 23   30 * *  pi      /home/pi/.temperature/monthly_copy.sh

and, without the need to reboot, I stopped and restarted cron:

sudo /etc/init.d/cron stop
sudo /etc/init.d/cron start

and that's it. Very crude, but it works, with the exceptions described above...

Cristian

[tobyweston]

Should prob move this to the docs and then close this ticket

[Quaxo76]

Toby, the problem is that as they are now, the monthly script (as described above) does not back up February, and skips the last day for months with 31 days. Someone with programming skills (which I don't have) should probably have a look at the scripts before adding this to the documents...

[ultraqwertz]

Hi Toby and Quaxo76,

I have written a bash script to archive the graphs for a permanent graphical overview of the past records.

The archive folder is placed in /home/pi/.temperature and contains subfolders for the years, weeks and months. The daily graphs will be placed in a folder named by month below the year-folder and the 7- and 30-day-graphs will be placed in folders "weekly" and "monthly" below the year folder.

The script should be self-explaining:

#!/bin/bash

# This script archives all graphs on a regular basis and is intended to be run
# daily at 23:59 as a cron job (add using "crontab -e"):
#
# 59 23 * * * /bin/bash /home/pi/.temperature/archive_graphs.sh
#
# The graph for the past day (showing 00:00 to 24:00) is archived every day.
#
# On Sundays, the graph for the past 7 days is archived
# (showing Monday 00:00 to Sunday 24:00).
#
# On the last day of each month, the graph for the past 30 days
# is archived. Depending on the number of days in the current month
# this will be slightly more or less than a month. Be aware that you 
# will miss some days this way and have overlapping data for some other days.

# Some variables:
# Today's date as a two-digit number
day=$(date +"%d")
# Current month as a two-digit number
month=$(date +"%m")
# Current year as a two-digit number
year=$(date +"%Y")
# Today's weekday as a one-digit number (1=Monday)
day_of_week=$(date "+%u")
# Tomorrow's date as a two-digit number
tomorrow=$(date "+%d" -d tomorrow)

# Locations
graph_folder=/home/pi/.temperature
archive_folder="$graph_folder"/archive
log="$graph_folder"/archive.log

# Logging the start date
echo "Start: "$(date) >> "$log"

# Daily
# Keep today's graph in an archive subfolder for the current month
mkdir -p "$archive_folder"/"$year"/"$month"
cp -v "$graph_folder"/temperature-1-days.png "$archive_folder"/"$year"/"$month"/"$year"-"$month"-"$day".png >> "$log" 2>&1


# Weekly
# Late each Sunday night copy the last weekly graph in an archive subfolder for the current year
if [ "$day_of_week" = "7" ] ; then
    mkdir -p "$archive_folder"/"$year"/weekly
    date_a_week_ago=$(date "+%d" -d "-10079 minutes")
    month_a_week_ago=$(date "+%m" -d "-10079 minutes")
    year_a_week_ago=$(date "+%Y" -d "-10079 minutes")
    cp -v "$graph_folder"/temperature-7-days.png "$archive_folder"/"$year"/weekly/"$year_a_week_ago"-"$month_a_week_ago"-"$date_a_week_ago"_to_"$year"-"$month"-"$day".png >> "$log" 2>&1
fi

# Monthly
# Keep last month's graph in an archive subfolder for the current year

    if [ "$tomorrow" = "01" ] ; then
	mkdir -p "$archive_folder"/"$year"/monthly
	cp -v "$graph_folder"/temperature-30-days.png "$archive_folder"/"$year"/monthly/"$year"-"$month".png >> "$log" 2>&1
    fi

echo "---------------------------------------------" >> "$log"

exit 0

Copy the contents into a file named archive_graphs.sh and put this in /home/pi/.temperature. Do a chmod +x /home/pi/.temperature/archive_graphs.sh and don't forget the cron job.

I will try to show the contents if this archive as an image gallery (I hope PiGallery will work) but have not tried yet. Thank you, Toby, for a great project!

[tobyweston]

Awesome work, thanks all 😄

[ultraqwertz]

You're welcome :) There is no "proper" webserver included in temperature-machine, right? Otherwise I would try to use that to display the archive.

[tobyweston]

It has a webserver embedded, if you send backup files to the assets folder (from memory), they'll be available via a browser (just with no UI if that makes sense)

[Quaxo76]

This looks like a good moment to mention again a wish... To add to the graph screen, maybe in the title or somewhere else, the "time frame" (date and time) of that graph. That would also help in case of a crash... A few months ago my system crashed and stopped updating the graphs, and I didn't notice for days... :) Il giorno Ven 1 Mar 2019, 15:23 Toby <notifications@github.com> ha scritto:

…

It has a webserver embedded, if you send backup files to the assets folder (from memory), they'll be available via a browser (just with no UI if that makes sense) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#44 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ah5kiAUa1PfZXptyejsKpKA9hk5b1VYqks5vSTfrgaJpZM4Rjjbe> .

[tobyweston]

Yep, good call. Did we capture that as a specific issue on Github? If not, can you add one pls?

[ultraqwertz]

Here is a quick workaround, at least for the graphs that are archived with above script: install ImageMagick (sudo apt install imagemagick) and add the following lines to the script after the lines starting with "cp -v":

# Add date as text to daily graph
convert "$archive_folder"/"$year"/"$month"/"$year"-"$month"-"$day".png -gravity NorthEast -pointsize 14 -annotate +35+10 "$year"-"$month"-"$day" "$archive_folder"/"$year"/"$month"/"$year"-"$month"-"$day".png

# Add start and end date as text to weekly graph
convert "$archive_folder"/"$year"/weekly/"$year_a_week_ago"-"$month_a_week_ago"-"$date_a_week_ago"_to_"$year"-"$month"-"$day".png -gravity NorthEast -pointsize 14 -annotate +35+10 "$year_a_week_ago"-"$month_a_week_ago"-"$date_a_week_ago"\ to\ "$year"-"$month"-"$day" "$archive_folder"/"$year"/weekly/"$year_a_week_ago"-"$month_a_week_ago"-"$date_a_week_ago"_to_"$year"-"$month"-"$day".png

# Add month as text to monthly graph
convert "$archive_folder"/"$year"/monthly/"$year"-"$month".png -gravity NorthEast -pointsize 14 -annotate +35+10 "$year"-"$month" "$archive_folder"/"$year"/monthly/"$year"-"$month".png

It's not very pretty because font and font size are different from the rest of the text on the graphs. Maybe you can give a hint about the font(s) used, Toby? In any case there are lots of options to change the look or positioning of the text: ImageMagick v6 Examples -- Text to Image Handling. And of course it would be best if temperature-machine put the text on the graphs natively...

[ultraqwertz]

I couldn't get PiGallery to work but Single File PHP Gallery is very easy to setup and works very well. You just have to install a webserver, e.g. apache2, php and php-gd beforehand.