Custom sidecar mako templates

[neilpa]

Is your feature request related to a problem? Please describe.

I have a large variety of semi-structured data that I currently embed in the Photos caption field as a catch-all. Some of these are tag-like and could be keywords. However, the bulk is either hierarchical or key-value in nature. I'd like to be able to parse and clean this data so that I can map it to more appropriate structure in the XMP export.

Describe the solution you'd like

I've been thinking about this for a while and custom sidecar templates via mako (like the existing XMP ones) seem like a reasonable approach. This would definitely be an advanced feature and possible footgun. However, I suspect it would be relatively simply to implement, aligns with existing templating functionality, and provides all the flexibility one could need. I could even imagine a repository user-contributed templates to handle specific quirks of all the various photo apps out there.

Describe alternatives you've considered

I've considered re-writing the exported XMP/JSON sidecar files post-export. However, I worry about that not playing nice with --update.

There's also properties like the photo UUID which aren't available. I currently work-around this by including the UUID in the file name. However, I'd like to avoid this because it makes the names annoyingly long.

[RhetTbull]

I like this idea.

There's also properties like the photo UUID which aren't available.

I don't understand this -- not available from where?

Assuming you don't also want also the current XMP sidecar, you could rig this now by editing the file xmp_sidecar_beta.mako in your site-packages/osxphotos/templates/ directory of wherever osxphotos is installed (~/.local/pipx if using pipx) and then passing the --beta flag which causes osxphotos to use this template instead of the normal one. This is how I test changes to the XMP file. Of course, the file will be named .xmp and it may not be a valid XMP but it would let you play around with the template.

I'll add this to the queue -- should be fairly straightforward to implement. Will need to add the files generated to the update list as you noted and provide a template for the filename so you can customize the output sidecar name.

[neilpa]

I like this idea.

Awesome 😄

There's also properties like the photo UUID which aren't available.

I don't understand this -- not available from where?

I meant there's no way to get the UUID of the photo post-export unless you explicitly include it in the filename template. Somehow I had overlooked --keyword-template and --description-template so that's a misunderstanding on my part.

[RhetTbull]

I'm close to having this implemented. Here's the interface so far. Does this meet your use case? Basically, you can provide a mako template, a custom osxphotos template which will be used to render the filename of the sidecar, and flags to either strip excess whitespace and/or excess new lines (which happens with Mako sometimes). The --sidecar-template option can be repeated so you can use this to write multiple different sidecars.

Edit: I also had to add a flag that tells osxphotos what to do when the photo is skipped. With regular sidecars, the sidecar file is still written (or updated or skipped) even if photo is skipped as while the photo may not have changed (causing it to be skipped), the content of the sidecar (which includes metadata) may have changed. For user sidecars, osxphotos won't know if the sidecar should be written or not as it has no knowledge of the contents of the sidecar thus the user must explicitly specify behavior. The options are always write a sidecar for skipped files (WRITE_SKIPED = 1) or never write a sidecar for skipped files (WRITE_SKIPPED = 0). This is definitely an advanced feature but it may be very useful for certain use cases.

--sidecar-template MAKO_TEMPLATE_FILE SIDECAR_FILENAME_TEMPLATE WRITE_SKIPPED STRIP_WHITESPACE STRIP_LINES
                                Create a custom sidecar file for each photo exported
                                with user provided Mako template (MAKO_TEMPLATE_FILE). MAKO_TEMPLATE_FILE
                                must be a valid Mako template (see https://www.makotemplates.org/). The
                                template will passed the following variables: photo (PhotoInfo object for
                                the photo being exported), sidecar_path (pathlib.Path
                                object for the path to the sidecar being written), and photo_path
                                (pathlib.Path object for the path to the exported photo.
                                SIDECAR_TEMPLATE_FILENAME must be a valid template string (see Templating
                                System in help) which will be rendered to generate the filename of the
                                sidecar file. The `{filepath}` template variable may be used in the
                                SIDECAR_TEMPLATE_FILENAME to refer to the filename of the photo being
                                exported. WRITE_SKIPPED is a boolean value (true/false, yes/no, 1/0 are all
                                valid values) and indicates whether or not write the sidecar file even if
                                the photo is skipped during export. If WRITE_SKIPPED is false, the sidecar
                                file will not be written if the photo is skipped during export. If
                                WRITE_SKIPPED is true, the sidecar file will be written even if the photo is
                                skipped during export. STRIP_WHITESPACE and STRIP_LINES are boolean values
                                (true/false, yes/no, 1/0 are all valid values) and indicate whether or not
                                to strip whitespace and blank lines from the resulting sidecar file. For
                                example, to create a sidecar file with extension .xmp using a template file
                                named 'sidecar.mako' and write a sidecar for skipped photos and strip blank
                                lines but not whitespace: `--sidecar-template sidecar.mako '{filepath}.xmp'
                                yes no yes`. To do the same but to drop the photo extension from the sidecar
                                filename: `--sidecar-template sidecar.mako
                                '{filepath.parent}/{filepath.stem}.xmp' yes no yes --sidecar-drop-ext`. For
                                an example Mako file see https://raw.githubusercontent.com/RhetTbull/osxphot
                                os/main/examples/custom_sidecar.mako

[RhetTbull]

I believe this is now working. I also updated report writer to indicate which files are user generated sidecars in the report. Works with --update and --cleanup. I need to add tests then will get this pushed out soon.

[neilpa]

Looks awesome! I'll be porting my hacks that were patching the beta template file to use this instead.

Happened to notice you have a typo in the docs, you reference both SIDECAR_FILENAME_TEMPLATE and SIDECAR_TEMPLATE_FILENAME.

[neilpa]

I also left a comment about potentially caching the compiled template.

[RhetTbull]

This is implemented in v0.61.0. I added caching of the template as you suggested and fixed the typos. Please let me know what you think and if this meets your use cases. The Mako sidecar is passed three variables:

• photo: a PhotoInfo object for the photo being exported
• photo_path: a pathlib.Path object for the photo file being exported
• sidecar_path: a pathlib.Path object for the sidecar file being written

There are three example Mako sidecars in the examples directory:

• custom_sidecar.mako: creates a simple text file with a few details.
• custom_sidecar_xmp.mako: creates an XMP sidecar
• custom_sidecar_json.mako: creates a JSON sidecar with the full JSON dump of the PhotoInfo object for the photo (everything osxphotos knows about the. photo).

Because the PhotoInfo object is available to the sidecar template, this last example is as simple as:

${photo.json(shallow=False, indent=4)}

This is definitely a niche option but as you suggested, it might be very useful for creating and sharing sidecars for specific apps.

[neilpa]

Please let me know what you think and if this meets your use cases.

This is great and covers everything I need so far.

Because the PhotoInfo object is available to the sidecar template, this last example is as simple as:

${photo.json(shallow=False, indent=4)}

The trivial JSON output that I have full control over is super nice. In my case I'm joining photos to some external CSV files that I can lookup records via keys (mostly URLs) that I've added as part of description field.

This makes for a great plugin system since you get the full power of python for every photo being processed. I bet you could even do some crazy stuff like convert videos to GIF and emit that. (Assuming there's some way to force mako to render binary data).

The other thing I'm wondering, should there be a way to conditionally write the sidecar, even for files that are not skipped. E.g. a flag that says, if your template renders to an empty buffer, skip writing the empty file. That would have to be another boolean option to --sidecar-template. At that point it might be better to collapse WRITE_SKIPPED STRIP_WHITESPACE STRIP_LINES into a final option of comma separated string flags, e.g. --sidecar-template my-template.mako '{filename}.ext' write_skipped,strip_lines. That would make it easier to add new flags and more obvious at the command line what's happening than the trailing true false true. Empty files are fine in my use case but food for thought.

[RhetTbull]

I like the idea of comma separated flags. I'd thought about doing a bit mask but figured that might not be intuitive for some users. The option would still need a fixed number of arguments so if no flags used there would still need to be something like none for the flags.

[RhetTbull]

Click, which osxphotos uses for option parsing, does not support variable number of arguments to an option. Wondering which of these is better:

--sidecar-template my-template.mako '{filename}.ext' write_skipped,strip_lines
--sidecar-template my-template.mako '{filename}.ext' none

or
--sidecar-template my-template.mako '{filename}.ext' --sidecar-options write_skipped,strip_lines
--sidecar-template my-template.mako '{filename}.ext'

As much as I hate adding yet another option, am leaning towards moving these to --sidecar-options which makes simple use case easier and allows options to be changed in the future without changes to the basic --sidecar-template option.

[neilpa]

I'd slightly lean to the latter option as well for simplicity. The one minor downside to that approach is that if you use multiple custom sidecars in a single export you get the same custom flags for each. Unless there's a way to say each --sidecar-options ... applies only to the previous --sidecar-template ....

[RhetTbull]

Unfortunately you'd be stuck with a single set of options and click doesn't provide a way to group options. That's a good point that keeping the options with the sidecar is important (and why I originally included those with --sidecar-template

[neilpa]

In that case I think an explicit none is still better than a series of extra bools.

[RhetTbull]

Any error created by the template should be logged. Add user_sidecar_error field and catch errors to log them. There could be an option added for catch_error or raise_error to define behavior -- catch/log the error and keep writing sidecars or abort the export.

[RhetTbull]

Completed in v0.62.0. --sidecar-template now accepts one or more named options