webp content-type not returned when CDN enabled

[kwyc]

Version of Dianoga

6.0.0

Environment description

Production Sitecore 9.3

What configs you have enabled

Production CD:

• Dianoga.DisableForSites.config
• Dianoga.Jpeg.config
• Dianoga.Log.config
• Dianoga.NextGenFormats.CDN.config
• Dianoga.Strategy.GetMediaStreamSync.config
• z.01.Dianoga.NextGenFormats.WebP.config

Production CM:

• Dianoga.DisableForSites.config
• Dianoga.Jpeg.config
• Dianoga.Log.config
• Dianoga.Strategy.GetMediaStreamSync.config
• z.01.Dianoga.NextGenFormats.WebP.config

Reproducible steps (1... 2... 3...) that cause the issue

Load CDN site vs non-CDN site and check content-type on .webp images. Expect image/webp instead of image/jpeg.

What you expected to see, versus what you actually saw

On CD, when CDN is enabled,

Expected to see image/webp instead of image/jpeg.

On CM, when CDN is disabled,

image/webp is returned which is correct behaviour.

GenerateCacheKey orders seems correct.

Relevant logs

Following up from closed issue,
#110

[Antonytm]

I have prepared a branch, where it could be easily tested on Docker containers.

Quick steps to reproduce:

1. Get changes from this branch https://github.com/antonytm/Dianoga/tree/feature/113-webp-cdn
2. cd \integration-tests
3. Run init.ps1
4. Run docker-compose -f docker-compose.yml -f docker-compose.override.113.yml up -d
5. Open https://cd.dockerexamples.localhost/jpeg# page

Expected result: img src should be /-/media/Project/Dianoga/Test/jpeg/jpg00.jpg?h=4032&iar=0&w=3024&extension=webp
Actual result: img src is /-/media/Project/Dianoga/Test/jpeg/jpg00.jpg?h=4032&iar=0&w=3024

The issue should be fixed in this PR:
#115

[markgibbons25]

Fixed in 6.0.1

[kwyc]

@markgibbons25 Thanks for looking into this issue! I tried the new 6.0.1 installation and replaced all files and configs; however, I still don't see the content-type being reflected on CD with CDN. As usual, CM without CDN is fine. Please advise.

Also, is there a way to force ".webp" extension. Currently if a jpg file is deemed "doesn't need to be optimized," original jpg file format is returned which is not ideal in our case.

In addition, looking at the above change in the "5ccab10" pull request, we had to comment off "mediaProvider
type="Dianoga.NextGenFormats.MediaProvider, Dianoga"
patch:instead="mediaProvider[@type='Sitecore.Resources.Media.MediaProvider, Sitecore.Kernel']"/ " in "Dianoga.NextGenFormats.CDN.config" as it is returning images files with "=?extension" which is not ideal.

/site/wwwroot/App_Config/Include/Dianoga
/site/wwwroot/App_Data/Dianoga Tools
/site/wwwroot/bin/Dianoga.dll

2864 20:39:07 INFO Dianoga: optimized /Images/Homepage/ic_volunteer_program.jpg [original size: 114925 bytes] [final size: 54913 bytes] [saved 60012 bytes / 52.22 %] [Optimized in 304ms] [Extension jpg]
11212 20:39:07 INFO Dianoga: optimized /Images/Homepage/ic_generic_multi-modal.jpg [original size: 138281 bytes] [final size: 63654 bytes] [saved 74627 bytes / 53.97 %] [Optimized in 493ms] [Extension jpg]
7628 20:39:07 INFO Dianoga: optimized /Images/Homepage/ic_promo_newsletter_signup.jpg [original size: 59000 bytes] [final size: 31599 bytes] [saved 27401 bytes / 46.44 %] [Optimized in 548ms] [Extension jpg]

[Antonytm]

Hi @kwyc

Unfortunately, there is no way to force Dianoga to know about the support of browser of wepb other than adding parameter ?extension=webp.

Let me explain why

The request comes to Sitecore in this order.
Browser => CDN => Sitecore. But that is the only first request. All subsequent requests will not come to Sitecore.

How it is possible to define if browser supports webp ?

The difference in requests from the browser that supports webp and does not support is in one small thing:
accept header parameter will contain image/webp value.

What happens if we do requests to CDN?

Browser => CDN => CDN doesn't care about header parameters and return from the cache
CDN will return the same cached item for browses with different accept. And we can't do anything as we are behind the CDN.

What it means for Dianoga and webp support?

As everything happens before the Sitecore server, we can't do anything "silently". We need to force the presence of some trigger in requests to force CDN to use different cache files for browsers with webp support and browsers without webp support. And the only way that was found for now: adding ?extension=webp query parameter. It will force CDN to have 2 caches for file image.jpeg:

1. image.jpeg
2. image.jpeg?extension=webp

Conclusion

1. Please uncomment <mediaProvider type="Dianoga.NextGenFormats.MediaProvider, Dianoga" patch:instead="mediaProvider[@type='Sitecore.Resources.Media.MediaProvider, Sitecore.Kernel']"/>
2. Clean Media_Cache folder on server
3. Purge CDN cache

And check if it works.

[kwyc]

@Antonytm Thanks for the detail explanation! I have tried 6.0.1 again and noticed that "?extension=webp" is not being appended to image urls even when the pipeline is enabled. Which tells me "extensions" variable is empty. Any thoughts on why that value is empty?

Also I've also tried to manually call image.jpeg?extension=webp on the CDN server and can confirm that content-type does return "content-type: image/webp" if the url is manually called.

Perhaps it is related to this issue,
https://www.rickvandenbosch.net/blog/front-door-incomplete-first-request/

[Antonytm]

@kwyc

You need to figure out what headers are available on the server. Probably CDN cleans them up.
I can suggest you to write custom processor for httpRequestBegin pipeline, iterate thought all available headers in request and log them. Then check logs, especially accept header.

Expected: the presence in accept header webp.
If it is not then the next step is to figure out, why it is absent.

[kwyc]

@Antonytm Thanks for your feedback, seems like it is an on-going issue with Azure FrontDoor.
MicrosoftDocs/azure-docs#86858

We will look into having FD to rewrite image urls from image.jpeg to image.jpeg?extension=webp

[kwyc]

@Antonytm Thanks for your assistance we finally got image/webp content-type returned on CDN! For anyone that's curious, we added a FD rewrite rule to append "?extension=webp".

[markgibbons25]

Issue with adding a FD rewrite rule to append "?extension=webp" on every request is you are essentially forcing webp even on browsers that can't use it. The good news is that all evergreen browsers now support webp https://caniuse.com/webp except for Safari on MacOs <10