Scheduled Export: launchd invokes shell script with macOS permission granted (in place of crontab) #1080
Replies: 9 comments 11 replies
-
Thank you for the excellent write-up! I wonder if it might be useful to have an osxphotos command that would create the plist, the shell script, etc for the user automatically? |
Beta Was this translation helpful? Give feedback.
-
I've gone the crontab approach ;)
$ crontab -l
44 3 * * sat,sun,mon /Users/MSP/Apps/runOSxPhotos.cron I do have to setup under Settings -> Energy Saver -> a power up schedule 3:43 am and then my #!/bin/bash
# Look for entries named **SET** to adjust to your configuration.
if [ -x /usr/libexec/path_helper ]; then
eval `/usr/libexec/path_helper -s`
fi
# set verbose level
__VERBOSE=5
declare -a LOG_LEVELS
# https://en.wikipedia.org/wiki/Syslog#Severity_level
LOG_LEVELS=([0]="emerg" [1]="alert" [2]="crit" [3]="err" [4]="warning" [5]="notice" [6]="info" [7]="debug")
function .log () {
local LEVEL=${1}
shift
if [ ${__VERBOSE} -ge ${LEVEL} ]; then
echo "[${LOG_LEVELS[$LEVEL]}]" "$@"
fi
}
# Source .bash_login
.log 7 Before source .bash_login PATH=$PATH
# **SET** To get all settings.
source ~/.bash_login
.log 7 After source .bash_login PATH=$PATH
# Activate Python Virtual Environment for osxphotos
# **SET** Set the location of your Python environment
source /Users/MSP/Documents/Apps/osxphotos/venv/bin/activate
# Set locale to en_GB.UTF-8
export LANG=en_GB.UTF-8
# Initial settings
PRG=`basename $0`
# **SET** Set location of this script
WORKFOLDER=~/Apps
# **SET** Set location of log file
FOLDER=~/Logs
LOGFILE=${FOLDER}/${PRG}.err
# **SET** Set Destination folder
DSTFOLDER=/Volumes/photo/Family.Photos
albumfile="/Users/Shared/Pictures/iPhoto Shared Library.photoslibrary/database/Photos.sqlite"
cd ${WORKFOLDER}
.log 7 CWD= `pwd`
.log 5 --- Start `date`| tee -a ${LOGFILE}
.log 5 Check if Filesystem "${DSTFOLDER}" is available | tee -a ${LOGFILE}
if [ -d "${DSTFOLDER}" ]; then
.log 5 Filesystem available "${DSTFOLDER}"! Continuing... | tee -a ${LOGFILE}
if [ -f "${albumfile}" ]
then
.log 5 "Continuing..."
# **SET** Set location of TOML file
caffeinate -s osxphotos export "${DSTFOLDER}" --exportdb ${FOLDER}/.osxphotos_export.db --load-config ${WORKFOLDER}/colibri.toml >> ${LOGFILE} 2>&1
.log 5 Result: $? | tee -a ${LOGFILE}
else
.log 5 photos.db file not found. Not running.
.log 5 photos.db file not found. Not running. | tee -a ${LOGFILE}
fi
else
.log 5 Folder "${DSTFOLDER}" not available! | tee -a ${LOGFILE}
fi
.log 5 --- End `date`| tee -a ${LOGFILE}
.log 5 --------- Tail ERR file... link to file: ${LOGFILE}
tail -50 $LOGFILE
.log 5 --------- |
Beta Was this translation helpful? Give feedback.
-
@rajscode this has been incredibly helpful after a number of hours of banging my head against the wall trying to achieve the exact same. Thank you for sharing!!! @RhetTbull I strongly support you adding whatever functionality would help people achieve this in osxphotos - I am sure I wont be the last person to come along trying to do incremental unattended backups! |
Beta Was this translation helpful? Give feedback.
-
Glad to read you got it going! Key part is you are setting it up as type 3 above and that is under the local user control that was logged in when it was set up. I personally run two different accounts on my Mac and when I need to modify my LaunchAgent (plist) I log in as that user account (2nd account in my case) and make the changes. Reasons as to why I run two different accounts on the Mac are detailed at: https://forums.macrumors.com/threads/backup-solution-icloud-full-photo-library-on-external-drive-whilst-optimized-reduced-version-on-internal-drive.2369845/ |
Beta Was this translation helpful? Give feedback.
-
@rajscode thank you for your kind response, and for the link to your (very detailed) post. It seems that the method detailed in your post would function in lieu of @RhetTbull fine piece of software? I personally have a second mac mini running, that has an optimised library on it. I then use OSXPhotos to export on a daily basis to the internal zfs mirrored drives I have setup inside a linux box i built a while back. These are shared on my local network via SMB. I then use Duplicati in a Docker container to create an encrypted backup the photos (and all the other files I store there) to Backblaze B2. Total overkill probably but its 3-2-1 backup and makes me sleep better at night :-) I am now really happy with my setup thanks to both you and @RhetTbull for getting the missing piece of my particular puzzle (exporting the originals to a flat folder structure). All of this makes me strangely happy! Regardless, thanks again - I am always very appreciative of folk like you who take the time and effort to post their solutions online for others to use/modify! |
Beta Was this translation helpful? Give feedback.
-
After a homebrew upgrade/update and/or OSXPHOTOS version upgrade, where an updated version is installed two types of errors may occur on a regular basis (1) OSXPHOTOS fails to run with python errors (2) Permission errors for MacOS Privacy and Security occur when executing OSXPHOTOS via a launchd job Commands typically used to update OSXPHOTOS version and/or fix OSXPHOTOS when a python update has occurred via homebrew pipx reinstall osxphotos
OR
pipx reinstall-all If you want to avoid re-doing (2) Privacy and Security settings for each python 3.x release that Home Brew automatically updates you could specify which python3 version OSXPHOTOS continues to use. Issue the following commands in place of the one above when updating OSXPHOTOS version (can specify any major installed version listed as links in: ls -la /opt/homebrew/bin/python* ) pipx uninstall osxphotos
pipx install --python python3.11 osxphotos To verify pipx installed for osxphotos properly and which version of python is being used check via the following: pipx list To check which python3 version pipx uses by default do: pipx environment Your machine probably has multiple python3 versions installed if using homebrew. Each time you run a homebrew update / hombrew upgrade cycle python3 and/or pipx will get updgraded to the latest version. NB: If you do the above, recommend every so often for security purposes and/or OSXPHOTOS requirement purposes you update pipx to specify a newer version of Python3.
|
Beta Was this translation helpful? Give feedback.
-
UPDATE to the posts above and the method I am using now as it is the easiest way I've found in order not to have to do any future interventations after 1st time setup. In place of granting each updated version of Python3 access via Privacy & Security in MacOS, grant those priveleges to an actual launcher application that you create via Automator. That app initiates the osxphotos export shell script. By priveledging a simple App you won't need to re-privelege every time python is updated as the Launcher app is the one who has the overall permissions now. Tip This launcher app also works for those running their osxphotos export command shell script manually from terminal as the same Privacy & Security TCC priveleges need to be granted repeatedly. So instead run the launcher app. You can make all the changes you want to your shell script as often as you want as long as you call it using the launcher app you won't have to grant updated P&S rights. General Steps:
(B) Test script_runner App and permission it with Privacy & Security one time instead of having to permission each version of Python3 when it gets updated by Homebrew for example:
open -a $HOME/bin/script_runner.app $HOME/bin/photos_library_export_to_disk.sh
(C) If everything above in the manually initiated run has worked fine and your export finished properly you can now decide if you want to create a scheduled re-occuring LaunchAgent. If yes, steps outlined in 1st post above BUT utilize the example XML below instead to craete a LaunchD job. NOTE: Please modify a few string values below appropriately:
Create a plist text file that contains the XML contents below (refer to details in the 1st post on how and where to create this file) <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Disabled</key>
<false/>
<key>Label</key>
<string>com.myphotobackup.PhotoLibraryExportToDisk.ScriptRunner</string>
<key>ProgramArguments</key>
<array>
<string>open</string>
<string>-a</string>
<string>/Users/userphotobackup/bin/script_runner.app</string>
<string>/Users/userphotobackup/bin/photos_library_export_to_disk_launchdjob.sh</string>
</array>
<key>StartCalendarInterval</key>
<array>
<dict>
<key>Hour</key>
<integer>2</integer>
<key>Minute</key>
<integer>5</integer>
<key>Weekday</key>
<integer>6</integer>
</dict>
</array>
</dict>
</plist> Note To clean up no longer relevant Privacy & Security Settings Items such as older python3 versions referenced but not removeable via the UI, don't use tccutil but instead look into carefully deleting entries from the TCC DB directly via SQL since most of the tcc entries for this purpose are in the user space. Step A: Go to System Settings / Privacy & Security / Photos
Caution This worked for me on MacOS Sonomoa 14.3.1. Step C: Removing the old and no longer useful or needed python3 entries that we turned off above in steps 1 and 2:
sqlite3 ~/Library/Application\ Support/com.apple.TCC/TCC.db "select client, auth_value, service from access where auth_value='0' and client like '%python%' order by client"
sqlite3 ~/Library/Application\ Support/com.apple.TCC/TCC.db "delete from access where auth_value='0' and client='/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/bin/python3.11'" NOTE: Don't execute command above as is. You must modify the client='' portion. Make sure you maintain the proper quotes symbols. Also please remember prior comment re: service values |
Beta Was this translation helpful? Give feedback.
-
Thank you @rajscode for this idea! |
Beta Was this translation helpful? Give feedback.
-
Finally figured it out. Based on @rajscode guide above, I created an App called ScriptRunner.app.
The second part is the log file - for the life of me I couldn't get the console output to show up when running the scipt through an automator app, so I basically had all lines printed with The third thing I had to do was give ScriptRunner.app Full Disk Access. Otherwise, I would get a popup similar to this one microsoft/vscode#201087 on every run. Ofcourse I also need to give ScriptRunner access to Apple Photos but that is preserved between runs and I don't get that annoying popup on every single run. All in all - it works great! :D EDIT: Here's a better implementation that prints to console output in real time and not just at the end of the execution (thanks ChatGPT):
|
Beta Was this translation helpful? Give feedback.
-
Hopefully this might be helpful to others who want schedule a job for an osxphotos export.
Been using osxphotos export --download-missing --use-photokit for a while with a shell script. Has worked great when kicking off manually. Not so great when trying with user crontab under macOS Ventura. Would get:
could not get authorizaton to use Photos: auth_status = 2
Decided to instead try using launchd and launchctl as I figured I'd be able to load and unload the job on demand as the user and I'd see any additional permission popups potentially.
There are three locations with launchd where you can put a job plist config file:
I used the 3rd option. First I created a shell script and saved it to:
$HOME/bin/photos_library_export_to_disk.sh
I tested out the shell script by running it within my Terminal and granting permission for any popups that the OS generated. Once I knew the script was working properly and exporting photos I moved on to creating a launchd plist job file.
Check to see you have a $HOME/Library/LaunchAgents directory. If it doesn't exist for your user account, create the LaunchAgents directory.
Create a text file that contains the XML contents below and save it to:
$HOME/Library/LaunchAgents/com.userphotobackup.PhotosLibraryExportToDisk.plist
The plist above, once loaded, will invoke the shell script when it is first loaded and run.
From your terminal issue the following to have the launchd daemon load above plist config file:
launchctl load $HOME/Library/LaunchAgents/com.userphotobackup.PhotosLibraryExportToDisk.plist
A couple things to keep an eye on:
/tmp/com.userphotobackup.PhotosLibraryExportToDisk.stderr
/tmp/com.userphotobackup.PhotosLibraryExportToDisk.stdout
You can tail -F the files in another terminal window to keep an eye.
Once you know everything is working as expected we can STOP the job from running (or if it's gone haywire and you need to make changes, grant priveleges, etc..). From the terminal issue the following command:
launchctl unload $HOME/Library/LaunchAgents/com.userphotobackup.PhotosLibraryExportToDisk.plist
NOTE: When you run the job under launchd control via the launchctl load command above you will see multiple popups occur for permission authorization. I ended up having the grant the following priveleges at various stages from start to end of everything.
Privacy & Security:
UPDATED INFO: After you do everything above and below and it's all working and a month later it stops working and you see: Permission Error 1 or something similiar you've most likely updated your Python3 install (i.e. brew update / brew upgrade). In that situation you are going to need to manually repermission for the newer python3 install (i.e python@3.9 upgraded to python@3.11) in two areas: (1) Files & Folders - Removable Volumes (2) Photos for the Privacy & Security. You'll end up having two instance of python3 showing up there.
UPDATED INFO No. 2 - See this comment further below in making some decisions: #1080 (comment)
![Screenshot 2023-06-17 at 2 39 08 AM](https://private-user-images.githubusercontent.com/99123253/246592020-7b7b5944-846f-4611-9255-008ac450e4d7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU3NDYyMTgsIm5iZiI6MTcyNTc0NTkxOCwicGF0aCI6Ii85OTEyMzI1My8yNDY1OTIwMjAtN2I3YjU5NDQtODQ2Zi00NjExLTkyNTUtMDA4YWM0NTBlNGQ3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA3VDIxNTE1OFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTM2NDUzZWE3MTY4NjFjNzRjZjE1NmFlOTU4NjZiMDc2YzdmY2Q2OTE1NzYyYjZlZTgyNzYzNGIzOGI5MzFmZWEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.D2cCq2YszqU_Hti4y61p_I5xXTxvwOraDTRKnslLIlk)
![Screenshot 2023-06-17 at 2 39 56 AM](https://private-user-images.githubusercontent.com/99123253/246592036-b297fc05-765d-40c6-8e64-9c5ddb0ffcca.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU3NDYyMTgsIm5iZiI6MTcyNTc0NTkxOCwicGF0aCI6Ii85OTEyMzI1My8yNDY1OTIwMzYtYjI5N2ZjMDUtNzY1ZC00MGM2LThlNjQtOWM1ZGRiMGZmY2NhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA3VDIxNTE1OFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBmODU2ZjllNzRhZTRlYjRhYjEwYWE1ZGM2YmNmZmUxMDhhYmE3ODk1MjRjMGY4YjAxODg5NjlhMTgyYTM1OWYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ppozpWpyH8Q50XJy89E9O4P-8JZwJkS6V0aDwZsdS-c)
Once the osxphotos export is working properly via launchd plist job you can now modify that plist job file so it doesn't run on load but instead ONLY runs at a particular time / day. i.e. every Friday at 12:05AM. The updated plist file would look like:
ps: There is a GUI based tool to manage launchd and plists called LaunchControl..
(Not affiliated with them. Just an FYI. It costs 21 dollars.)
Beta Was this translation helpful? Give feedback.
All reactions