fix #82 - improve performance of OptimizingMediaCache

[kennyschreurs]

this change allows limiting the maximum concurrent optimization tasks and reduces the overall memory footprint

[markgibbons25]

Can you test that in the case where the optimisation failed for some reason the original stream is returned and original image is rendered? As a test you could use the corrupted image from here #68

[kennyschreurs]

Isn't that already done in ShouldNotSquishCorruptedJpegLossy in JpegOptimOptimizerTests? If the OptimizingMediaCache.cs needs to be tested we need to add fakes for the Sitecore objects as well.
If still needed I'll try to have a look into it.

[markgibbons25]

Yeah good point. Would be nice to have tests for OptimizingMediaCache if you are volunteering :). I'll give this PR a test soon.

[markgibbons25]

So looking at this https://www.nuget.org/packages/System.Threading.Tasks.Dataflow/5.0.0 and this https://stackoverflow.com/questions/23245713/is-tpl-dataflow-included-with-either-net-4-5-or-net-4-5-1/23245803 looks like we will need to add this dependency that will flow up. Seems like this will be fine on .NET 4.5+ but I would love it to be tested on Sitecore 8 before I push this out, so will tag this as pre-release when I push the package.

[markgibbons25]

Found a concurrency issue testing locally with MaxConcurrentThreads = -1

32192 12:46:22 INFO  "C:\inetpub\wwwroot\sxa100sc.dev.local\App_Data\Dianoga Tools\pngquant\pngquant.exe --force --ext .png "C:\WINDOWS\TEMP\tmp71D6.png""
23736 12:46:22 INFO  "C:\inetpub\wwwroot\sxa100sc.dev.local\App_Data\Dianoga Tools\pngquant\pngquant.exe --force --ext .png "C:\WINDOWS\TEMP\tmp71D7.png""
14348 12:46:22 INFO  "C:\inetpub\wwwroot\sxa100sc.dev.local\App_Data\Dianoga Tools\pngquant\pngquant.exe --force --ext .png "C:\WINDOWS\TEMP\tmp71D8.png""
10380 12:46:22 INFO  "C:\inetpub\wwwroot\sxa100sc.dev.local\App_Data\Dianoga Tools\pngquant\pngquant.exe --force --ext .png "C:\WINDOWS\TEMP\tmp71D9.png""
19272 12:46:22 INFO  "C:\inetpub\wwwroot\sxa100sc.dev.local\App_Data\Dianoga Tools\pngquant\pngquant.exe --force --ext .png "C:\WINDOWS\TEMP\tmp71DA.png""
22572 12:46:22 INFO  "C:\inetpub\wwwroot\sxa100sc.dev.local\App_Data\Dianoga Tools\pngquant\pngquant.exe --force --ext .png "C:\WINDOWS\TEMP\tmp71DB.png""
30420 12:46:22 INFO  "C:\inetpub\wwwroot\sxa100sc.dev.local\App_Data\Dianoga Tools\pngquant\pngquant.exe --force --ext .png "C:\WINDOWS\TEMP\tmp71EB.png""
26760 12:46:22 INFO  "C:\inetpub\wwwroot\sxa100sc.dev.local\App_Data\Dianoga Tools\pngquant\pngquant.exe --force --ext .png "C:\WINDOWS\TEMP\tmp71FC.png""
33724 12:46:22 INFO  Dianoga: /Base Themes/Components Theme/Scripts/optimized-min cannot be optimized due to media type or path exclusion
18360 12:46:23 INFO  Dianoga: messages occurred while optimizing /Project/xx/Corporate/ Blog/Global/Social Icons/twitter_active: PngQuantCliOptimizer: the optimized image resulted in a larger file size (721 vs 506). Using the original instead.
33232 12:46:23 INFO  Dianoga: messages occurred while optimizing /Project/xx/Corporate/ Blog/Global/Social Icons/linkedin_active: PngQuantCliOptimizer: the optimized image resulted in a larger file size (1263 vs 895). Using the original instead.
31452 12:46:23 INFO  Dianoga: messages occurred while optimizing /Project/xx/Corporate/ Blog/Global/Social Icons/twitter_active: PngQuantCliOptimizer: the optimized image resulted in a larger file size (721 vs 506). Using the original instead.
26056 12:46:23 INFO  Dianoga: messages occurred while optimizing /Project/xx/Corporate/ Blog/Global/Social Icons/twitter_active: PngQuantCliOptimizer: the optimized image resulted in a larger file size (721 vs 506). Using the original instead.
4012 12:46:23 INFO  Dianoga: messages occurred while optimizing /Project/xx/Corporate/ Blog/Global/Social Icons/youtube_active: PngQuantCliOptimizer: the optimized image resulted in a larger file size (1113 vs 715). Using the original instead.
28184 12:46:23 INFO  Dianoga: messages occurred while optimizing /Project/xx/Corporate/ Blog/Global/Social Icons/linkedin_active: PngQuantCliOptimizer: the optimized image resulted in a larger file size (1263 vs 895). Using the original instead.
24220 12:46:23 INFO  Dianoga: messages occurred while optimizing /Project/xx/Corporate/ Blog/Global/Social Icons/twitter_active: PngQuantCliOptimizer: the optimized image resulted in a larger file size (721 vs 506). Using the original instead.
12204 12:46:23 INFO  Dianoga: messages occurred while optimizing /Project/xx/Corporate/ Blog/Global/Social Icons/youtube_active: PngQuantCliOptimizer: the optimized image resulted in a larger file size (1113 vs 715). Using the original instead.
32376 12:46:23 INFO  Dianoga: messages occurred while optimizing /Project/xx/Corporate/ Blog/Global/Social Icons/youtube_active: PngQuantCliOptimizer: the optimized image resulted in a larger file size (1113 vs 715). Using the original instead.
30208 12:46:23 INFO  Dianoga: messages occurred while optimizing /Project/xx/Corporate/ Blog/Global/Social Icons/facebook_active: PngQuantCliOptimizer: the optimized image resulted in a larger file size (577 vs 471). Using the original instead.
12204 12:46:23 ERROR OpenFileStream failed: C:\inetpub\wwwroot\sxa100sc.dev.local\App_Data\MediaCache\blogcm\202\373ab08f5f944ae98624fff990bcc8fc.png
Exception: System.IO.IOException
Message: The process cannot access the file 'C:\inetpub\wwwroot\sxa100sc.dev.local\App_Data\MediaCache\blogcm\202\373ab08f5f944ae98624fff990bcc8fc.png' because it is being used by another process.
Source: mscorlib
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at Sitecore.Abstractions.IO.FileWrapper.Open(String path, FileMode mode, FileAccess access, FileShare share)
   at Sitecore.IO.FileUtil.OpenFileStream(String filePath, FileMode mode, FileAccess access, FileShare share, Boolean createFolder, Boolean logErrors)
   
33012 12:46:23 INFO  "C:\inetpub\wwwroot\sxa100sc.dev.local\App_Data\Dianoga Tools\pngquant\pngquant.exe --force --ext .png "C:\WINDOWS\TEMP\tmp7377.png""
20344 12:46:23 INFO  "C:\inetpub\wwwroot\sxa100sc.dev.local\App_Data\Dianoga Tools\pngquant\pngquant.exe --force --ext .png "C:\WINDOWS\TEMP\tmp7378.png""
10948 12:46:23 INFO  Dianoga: messages occurred while optimizing /Project/xx/Corporate/ Blog/Global/Social Icons/facebook_active: PngQuantCliOptimizer: the optimized image resulted in a larger file size (577 vs 471). Using the original instead.
12504 12:46:23 INFO  Dianoga: messages occurred while optimizing /Project/xx/Corporate/ Blog/Global/Social Icons/facebook_active: PngQuantCliOptimizer: the optimized image resulted in a larger file size (577 vs 471). Using the original instead.
12504 12:46:23 ERROR OpenFileStream failed: C:\inetpub\wwwroot\sxa100sc.dev.local\App_Data\MediaCache\blogcm\224\9d05e8f7eb8c4f24a36d11b0da0f6036.png
Exception: System.IO.IOException
Message: The process cannot access the file 'C:\inetpub\wwwroot\sxa100sc.dev.local\App_Data\MediaCache\blogcm\224\9d05e8f7eb8c4f24a36d11b0da0f6036.png' because it is being used by another process.
Source: mscorlib
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at Sitecore.Abstractions.IO.FileWrapper.Open(String path, FileMode mode, FileAccess access, FileShare share)
   at Sitecore.IO.FileUtil.OpenFileStream(String filePath, FileMode mode, FileAccess access, FileShare share, Boolean createFolder, Boolean logErrors)

12504 12:46:23 ERROR Dianoga: Exception occurred on the background thread when optimizing: /Project/xx/Corporate/ Blog/Global/Social Icons/facebook_active
Exception: System.InvalidOperationException
Message: Could not create the buffer file needed for stream sharing.
Source: Sitecore.Kernel
   at Sitecore.IO.StreamSharer..ctor(Stream masterStream, String bufferFilePath)
   at Sitecore.Resources.Media.MediaCacheRecord.Initialize(Media sitecoreMedia, MediaOptions mediaOptions, MediaStream mediaStream)
   at Sitecore.Resources.Media.MediaCache.CreateCacheRecord(Media media, MediaOptions options, MediaStream stream)
   at Dianoga.Invokers.MediaCacheAsync.OptimizingMediaCache.Optimize(SiteContext currentSite, Media media, MediaOptions options) in C:\git\sitecore\Dianoga\src\Dianoga\Invokers\MediaCacheAsync\OptimizingMediaCache.cs:line 114

With the value set to 1 everything works as expected.

However with the value set to 4 I get the same errors. This is on a 6 core CPU. So should we hardcode the setting to 1 and not allow it to be changed?

[markgibbons25]

See separate comment about a concurrency issue