Background Jobs

Background Jobs

PhotoPress runs work over many images in the background, in small batches, so no request times out. Settings tabs that start a job show its progress and a Cancel button. The batches run through Action Scheduler, which PhotoPress bundles.

Requirements

• WordPress 6.6 or later.
• Nothing to install. Action Scheduler is bundled with PhotoPress as the Composer package woocommerce/action-scheduler (composer.json).
• Action Scheduler runs its queue from WP-Cron and from its own async requests. A site with WP-Cron off still moves a job on while a settings tab watches it. See The progress panel.
• search.index needs Relevanssi active.

Using it

Jobs start from these places:

Tab	Control	Job
Image Sizes	Save and regenerate N affected images	images.regenerate
Metadata, Image Taxonomies	Reprocess all images, or Save and reprocess on a taxonomy or parent keyword	metadata.reprocess
Metadata, alt text, description, licensing	Save and reprocess all images	metadata.alt_text, metadata.description, metadata.license
Pages & Search	Re-index all images, or turning on Images in search results	search.index
none	Renaming or deleting an image taxonomy term with more than 25 images, while Relevanssi indexes images	search.terms

One job of each type runs at a time.

The progress panel

The panel shows a progress bar, "N of M", the number failed, and a status:

Status	Meaning
Starting…	The start request is on its way.
Waiting to start…	Queued. No batch has run yet.
Running…	A batch is running or due.
Running (resting between batches)…	A paced job resting before its next batch.
Waiting: the server is busy…	A paced job waiting for memory or load to drop.
Finished date	Done.
Canceled date	Canceled, or stopped because its type is no longer available.

Errors lists the items that failed, with the reason. Cancel stops the job.

While the panel is open it asks for the job's progress every 2 seconds. If the job has not been updated for more than 5 seconds, that request also runs a batch (nudge). You can leave the page. The job keeps running through Action Scheduler.

The panel shows a job that ended only if you watched it run. A new visit to the tab starts without it.

How it works

Job types

Type	Label	What one item does	Batch items	Batch seconds	Paced
images.regenerate	Regenerate image sizes	Makes the image's missing or outdated sizes. See Image Sizes.	5	10	yes
metadata.reprocess	Re-read image metadata	Reads the file's metadata into taxonomy terms, alt text and description.	10	20	yes
metadata.alt_text	Reprocess alt text	Sets the alt text from the file and template.	10	20	yes
metadata.description	Reprocess descriptions	Sets the description from the file and template.	10	20	yes
metadata.license	Reprocess licensing metadata	Writes the license into the XMP of each of the image's files.	5	10	yes
search.index	Add images to the search index	Indexes the image in Relevanssi. Registered only while Relevanssi is active.	25	10	yes
search.terms	Refresh the images of renamed and deleted terms	Announces that the image's terms changed, for Relevanssi's index and, with Proxy Cache Purge, the page cache.	25	10	yes
gallery.upgrade	Update galleries saved by earlier versions	Converts a post's legacy Child Pages blocks and records its Taxonomy Galleries. Started once per site after the update (Child Pages Gallery, Taxonomy Gallery). Items are post IDs.	20	20	no

Registrations: images.php#L61, metadata.php#L69, ImageSearch.php#L88, TermChanges.php#L62, GalleryUpgrade.php#L31. A type that sets no batch size gets 50 items and 20 seconds, unpaced (code).

Items are image IDs, or post IDs for gallery.upgrade, in ascending order. The job keeps a cursor, the last ID done.

A metadata.reprocess started while one runs is queued in the option photopress_reprocess_queue and starts when the running one finishes. search.terms work that arrives while one runs waits in photopress_search_term_changes.

Batches

runBatch:

1. Takes the job's lock, the option photopress_job_<id>_lock, with add_option(). Of two requests only one gets it. A lock older than 120 seconds is taken over.
2. A paced job checks whether the server is busy. If so, it waits 60 seconds and the batch ends.
3. Fires photopress_job_batch_started.
4. Asks the type for the next items after the cursor, and processes them until the batch's item count or time is used, or PHP has used more than 80% of memory_limit.
5. Fewer items than asked for, ending at the cursor, means the job is done.
6. A paced job then rests as long as the batch took, so it works at most half the time.
7. Releases the lock, fires photopress_job_batch_ended, and queues the next batch, or, when done, runs the type's finish step and fires photopress_job_ended.

Pacing

A paced job counts the server as busy (busy) when:

Check	Threshold	Filter
MemAvailable in /proc/meminfo	below 300 MB	photopress_job_min_memory (bytes)
One-minute load average (sys_getloadavg())	above 0.7 × the processors in /proc/cpuinfo	photopress_job_max_load

Where either value cannot be read, as on Windows, that check counts as not busy.

The item in hand

Before processing an item, the batch saves it in the job as current (code). If the batch dies on it (out of memory or time, which no catch sees), the next batch finds it there, records it as failed with "This item stopped its batch (out of memory or time); skipped.", and moves past it (code). An exception from an item is caught and recorded as a failure. The job keeps the 50 most recent errors.

Starting

start:

• A start lock, the option photopress_job_start_<md5 of type>, stops two starts of a type at once, as from a double click. A second start within 30 seconds gets 409 photopress_job_running.
• A queued or running job of the type also gives 409 photopress_job_running.
• The job's total is counted, the job is saved as the option photopress_job_<id> (12 hex characters), and its ID goes first in photopress_jobs. The last 20 jobs are kept. Older ones are deleted with their locks.
• The first batch is queued as an async Action Scheduler action.

Canceling

Cancel (DELETE /photopress/v1/jobs/<id>) sets the job to cancelled, unschedules its pending batch and fires photopress_job_ended (code). A batch already running checks the job before each item and stops. The item in progress finishes.

Recovery of stalled jobs

recover runs after each run of Action Scheduler's queue, and before each start:

• A queued or running job with no lock held and no pending batch gets a batch queued. This covers a batch that died and a lost action.
• A job whose type is no longer registered, as when Relevanssi is switched off during search.index, is canceled with the error "Stopped: this kind of job is not available now, as when a plugin it needs is switched off." It can be started again once the type is back.

Running again

Type	Effect of running it again over the same images
images.regenerate	Each image gets only the work it still needs. Up-to-date images are skipped. A remake that died left the image as it was.
metadata.reprocess, metadata.alt_text, metadata.description	Values are read again from the file and template. The same file and settings give the same values.
metadata.license	A file that already has the license is not written. The image's metadata is saved again only when a file changed.
search.index	Each image is indexed again.
search.terms	Each image's terms are announced again.

Action Scheduler

Scheduler.php loads Action Scheduler only on the requests that need it:

• From the start of the request (plugins_loaded at priority -1, before Action Scheduler registers its copies at 0): when PhotoPress has work scheduled, or another plugin has already loaded Action Scheduler (code).
• When PhotoPress schedules work (ensure): loaded after init, Action Scheduler initializes itself, as it does in themes.

The option photopress_scheduled holds pending or idle, and is autoloaded. A site without it, as after an update or deactivation, counts as pending.

After each queue run, settle recovers stalled jobs. When no job is unfinished and no PhotoPress action is pending or in progress, it sets idle. If no other plugin had loaded Action Scheduler, it also clears Action Scheduler's every-minute WP-Cron event (action_scheduler_run_queue with WP Cron). Action Scheduler schedules that event again when it next loads. On WP-Cron requests where Action Scheduler is not loaded, PhotoPress clears a leftover event at init (code).

Other plugins that bundle Action Scheduler. When another plugin's copy is loaded by plugins_loaded, PhotoPress loads its own copy from the start of every request too. Each copy registers, and Action Scheduler uses the newest. PhotoPress never clears the WP-Cron event in that case.

All PhotoPress actions use the group photopress:

Action	Scheduled by
photopress_job_batch	Jobs (argument: job ID)
photopress_cdn_invalidate	CDN invalidation. See Offload Media and CDN.
photopress_offload_delete_objects	Bucket deletion. See Offload Media and CDN.
photopress_replace_settle	File replacement. See Replacing Image Files.
photopress_gallery_remove_retry	Removing a deleted image from a post that was being edited. See Gallery Images Added and Removed.

WordPress integration

Hook	Type	Code	Purpose
plugins_loaded	action	Scheduler.php#L47	Priority -1: loads Action Scheduler from the start when needed.
action_scheduler_after_process_queue	action (Action Scheduler)	Scheduler.php#L48	Recovers stalled jobs. Marks PhotoPress idle when nothing is left.
init	action	Scheduler.php#L67	WP-Cron requests only: clears a leftover WP-Cron event.
photopress_job_batch	Action Scheduler action	Jobs.php#L81	Runs a batch.
rest_api_init	action	Jobs.php#L82	Registers the routes.

Action Scheduler functions used: as_enqueue_async_action(), as_schedule_single_action(), as_get_scheduled_actions(), as_has_scheduled_action(), as_unschedule_all_actions(). Core: wp_clear_scheduled_hook(), wp_next_scheduled(), add_option() for the locks.

Data stored

Name	Kind	Purpose
photopress_jobs	option	IDs of the last 20 jobs, newest first.
photopress_job_<id>	option	A job: type, args, total, done, failed, errors, cursor, current, status, times, user, next, waiting.
photopress_job_<id>_lock	option	The batch lock: when taken.
photopress_job_start_<md5>	option	The start lock of a type.
photopress_scheduled	option, autoloaded	pending or idle.
photopress_reprocess_queue	option	metadata.reprocess runs waiting for the running one.
photopress_search_term_changes	option	Terms and images waiting for the running search.terms job.

Actions and their logs are in Action Scheduler's own tables.

REST routes

All require manage_options (JobsRest.php).

Route	Method	Purpose
/photopress/v1/jobs	GET	Recent jobs, newest first. type filters by type.
/photopress/v1/jobs	POST	Starts a job: type (required), args (object). 409 when one of the type runs.
/photopress/v1/jobs/types	GET	Registered types: type, label, description.
/photopress/v1/jobs/<id>	GET	The job. Runs a batch if one is due (see the nudge above). 404 if unknown.
/photopress/v1/jobs/<id>	DELETE	Cancels the job.

A job in a response has id, type, label, total, done, failed, errors, status (queued, running, done, cancelled), created, updated, finished, next (when a paced job's next batch may run, Unix time) and waiting (busy while it waits for the server, else empty) (code).

Third-party integrations

Action Scheduler (bundled, also shipped by WooCommerce and other plugins). PhotoPress uses its public as_* functions and its action_scheduler_after_process_queue action. When another plugin bundles it, both copies register and Action Scheduler picks the newest.

Relevanssi. search.index is registered only while Relevanssi is active. A search.index job left unfinished when Relevanssi is switched off is canceled by recovery.

Proxy Cache Purge. During a batch PhotoPress holds back the purges its items would cause. When the job ends, the cache is emptied once if any batch held one back. If the job is canceled during a batch, the cache is emptied at once.

Hooks PhotoPress provides

Hook	Type	Arguments	When	Code
photopress_job_batch_started	action	array $job	A batch is about to process items, in this request.	Jobs.php#L311
photopress_job_batch_ended	action	array $job (status running, done, or cancelled from elsewhere)	A batch ended, in this request.	Jobs.php#L376
photopress_job_ended	action	array $job	A job finished, was canceled, or was stopped because its type is gone.	Jobs.php#L391
photopress_job_min_memory	filter	int $bytes (default 300 MB)	Each busy check of a paced job.	Jobs.php#L419
photopress_job_max_load	filter	float $load (default 0.7 × processors)	Each busy check of a paced job.	Jobs.php#L429

A module registers a job type with \PhotoPress\jobs\Jobs::register( $type, $definition ): label, description, count( $args ), items( $after, $limit, $args ), process( $item, $args ) returning true or a WP_Error, and optionally batch_seconds, batch_items, pace, finish( $job ) (code).

Limits, quirks and known issues

• One job per type at a time. Different types can run at once.
• A paced job on a server that stays busy waits indefinitely, checking every 60 seconds.
• An item that kills its batch is skipped and counted as failed. It is not retried.
• Only the last 20 jobs and the last 50 errors of each job are kept.
• The total is counted at start. When the job finishes, the total is raised to the number processed if that is higher.

Related

• Image Sizes
• Offload Media and CDN
• Replacing Image Files
• Lifecycle and Data
• Image Taxonomies
• Alt Text and Descriptions
• Licensing
• Image Search
• Page Cache
• Hooks Reference