Limit pending image transforms in GeneratePendingTransforms job

[SHoogland]

Description

When adding webp transforms to an existing site, it will trigger an enormous amount of pending image transforms (2000+)
When this job tries to transform all of them at once the job will most likely hit its timeout (which leaves imagemagick tmp files around)

Is it an idea to limit the query in this job to a maximum? or an option to disable generating images in the background as a temporary solution?

cms/src/queue/jobs/GeneratePendingTransforms.php

Line 24 in 35eafe4

public function execute($queue)

For example:

cms/src/services/AssetTransforms.php

Lines 953 to 959 in 35eafe4

	public function getPendingTransformIndexIds(): array
	{
	return $this->_createTransformIndexQuery()
	->select(['id'])
	->where(['fileExists' => false, 'inProgress' => false])
	->column();
	}

with a ->limit(50)

Steps to reproduce

1. Change your code to use new image sizes or formats (webp)
2. publish that code to an existing site with a lot of images and using srcsets
3. this will trigger 100s of pending image transforms
4. which the generate pending transforms jobs tries to generate all at once.

Additional info

• Craft version: Craft Pro 3.5.17.1
• PHP version: 7.4.3
• Database driver & version: MySQL 8.0.22
• Image driver & version Imagick 3.4.4 (ImageMagick 6.9.10-23)
• Plugins & versions:
  CP Field Inspect 1.2.3
  Entry Instructions 1.0.7
  Environment protection 1.1.1
  Field Manager 2.2.2
  Position Fieldtype 1.0.16
  Redactor 2.8.5
  SEO 3.6.7
  Super Table 2.6.5

[brandonkelly]

Where are you seeing those ImageMagick temp files?

[SHoogland]

In the /tmp folder on our server

[SHoogland]

But that is kind of besides the point... do we expect the transformJob to resize 2000+ images in (the default) 300 seconds.

[brandonkelly]

Well typically there wouldn’t be… transforms only get added to that queue when their URL has been requested by a template. Do you know what is triggering these? Just a bunch of front-end web requests?

[SHoogland]

Well, we do graphql queries like these to generate images for all the different screensizes including a webp copy:

Which generate about 24 assettransformindex rows for each image on a particular page.
So that can add up quite quickly if we change a lot of the requested sizes at the same time (with an update to a site).

Context: we place all those images in a picture element with srcsets

[brandonkelly]

That background job shouldn’t be getting queued up for GraphQL requests in the first place:

cms/src/controllers/GraphqlController.php

Lines 156 to 157 in 48dffa2

	// Generate all transforms immediately
	Craft::$app->getConfig()->getGeneral()->generateTransformsBeforePageLoad = true;

Are you doing something to override that?

In any case, this might be a good case for using either ImageOptimize or Imager X, both of which can offload transform generation to imgix, which is a better transform solution for larger loads.

[SHoogland]

I don't think the job itself is getting queued by the graphql call.
But all the pending transforms are created at that point.

And the page that calls the graphql query will try to render one of the images, which triggers the job.

I guess the immediately: false overrides the "generateTransfromsBeforePageLoad" setting?

Anyway, can't we just disable the GeneratePendingTransforms job? so images only get transformed when requested?

[brandonkelly]

Ah gotcha. And sorry, I missed the immediately: false arguments in your query.

I just added a new property that you can override to prevent these jobs from getting added, for the next 3.6 release.

To get the change early, change your craftcms/cms requirement in composer.json to "3.6.x-dev as 3.6.0-RC3" and run composer update.

Then open config/app.php and add this:

return [
    'components' => [
        'assets' => [
            'generatePendingTransformsViaQueue' => false,
        ],
    ],
];

[martijnvanonz]

Hi Brandon,

We found out when you use this setting it can give problems with the cache tag in twig.
I think this would be a nice solution to the problem. What do you think?

[brandonkelly]

Craft 3.6 has now been officially released ✨

[brandonkelly]

@martijnvanonz Sorry, missed your question. The {% cache %} tag will only cache templates that don’t include any ungenerated image transform URLs, and changing this setting will increase the chances of that, yes.