System.OutOfMemoryException when creating delta for (large) files

[themoritz]

The following happens with version 1.2.5 (via https://github.com/electron/windows-installer version 2.0.5) during diffing:

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.File.InternalReadAllBytes(String path, Boolean checkHost)
   at Squirrel.DeltaPackageBuilder.createDeltaForSingleFile(FileInfo targetFile, DirectoryInfo workingDirectory, Dictionary`2 baseFileListing)
   at Squirrel.DeltaPackageBuilder.CreateDeltaPackage(ReleasePackage basePackage, ReleasePackage newPackage, String outputFile)
   at Squirrel.Update.Program.Releasify(String package, String targetDir, String packagesDir, String bootstrapperExe, String backgroundGif, String signingOpts, String baseUrl, String setupIcon, Boolean generateMsi)
   at Squirrel.Update.Program.executeCommandLine(String[] args)
   at Squirrel.Update.Program.main(String[] args)
   at Squirrel.Update.Program.Main(String[] args)

Note that there is still enough physical memory when this happens (about 4gb). The files are about 160mb. Works without problems for files that are around 50mb.

[tomgolan]

Version 1.4.4 - I'm having the same issue. file size is around 130mb
Adding the error from SquirrelSetup.log:

DeltaPackageBuilder: We couldn't create a delta for sharedassets0.assets.resS, attempting to create bsdiff
DeltaPackageBuilder: We really couldn't create a delta for sharedassets0.assets.resS: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at Squirrel.Bsdiff.BinaryPatchUtility.Create(Byte[] oldData, Byte[] newData, Stream output)
at Squirrel.DeltaPackageBuilder.createDeltaForSingleFile(FileInfo targetFile, DirectoryInfo workingDirectory, Dictionary`2 baseFileListing)

[Thieum]

Instead of ReadAllByte here:

Squirrel.Windows/src/Squirrel/DeltaPackage.cs

Line 182 in de5baa9

var oldData = File.ReadAllBytes(baseFileListing[relativePath]);

We should compare files in streams like this:
https://stackoverflow.com/questions/968935/compare-binary-files-in-c-sharp

[shiftkey]

@Thieum I think this is worth exploring, so I've marked this as help wanted to see if we can make this more efficient.

[trodi]

I've reproduced this in latest, Squirrel.Windows v1.9.1 via electron-winstaller v3.0.4.

My environment was 8GB of RAM, package sizes ~780MB.

[anaisbetts]

Does this actually stop packaging? afaik even though the error message is unfortunate, the end result is correct - it's probably Unlikely that we'll be able to generate a usable diff for super gigantic files given our current algorithms (mspatch and bsdiff).

Proper support for giant files probably requires a block-based approach - i.e. diffing sections of a file instead of the whole thing en masse. While this is a Worthwhile Endeavor, it's also not a quick fix.

[trodi]

My call to electron-winstaller's createWindowsInstaller rejects its promise with the System.OutOfMemoryException. On disk, it has successfully created the full nupkg and updated RELEASES file with the additional new entry, just no delta.

Although not quick to fix, larger apps have the most to benefit from delta packages.

Surprisingly, when I went to reproduce this again, 2 of my tests actually produced a delta package. It was ~150MB instead of ~780MB. However, attempting the upgrade, squirrel was unable to apply the delta package and fell back to the full package.

DeltaPackageBuilder: Applying MSDiff to lib\net45\<app-name>_ExecutionStub.exe.diff
2019-10-29 15:13:05> Program: Failed to apply updates, falling back to full updates: System.Exception: Invalid release entry: 96695A3C969EB5669ABA99E03B99DE5AFECB78C5 <app-name>_ExecutionStub.exe.shasum 278528
   at Squirrel.ReleaseEntry.ParseReleaseEntry(String entry)
   at Squirrel.DeltaPackageBuilder.verifyPatchedFile(String relativeFilePath, String inputFile, String tempTargetFile)
   at Squirrel.DeltaPackageBuilder.applyDiffToFile(String deltaPath, String relativeFilePath, String workingDirectory)
   at Squirrel.DeltaPackageBuilder.<>c__DisplayClass3_1.<ApplyDeltaPackage>b__4(String file)
   at Squirrel.EnumerableExtensions.ForEach[TSource](IEnumerable`1 source, Action`1 onNext)
   at Squirrel.DeltaPackageBuilder.ApplyDeltaPackage(ReleasePackage basePackage, ReleasePackage deltaPackage, String outputFile)
   at Squirrel.UpdateManager.ApplyReleasesImpl.<>c__DisplayClass8_0.<createFullPackagesFromDeltas>b__3()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.UpdateManager.ApplyReleasesImpl.<createFullPackagesFromDeltas>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.UpdateManager.ApplyReleasesImpl.<ApplyReleases>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Squirrel.UpdateManager.<ApplyReleases>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Squirrel.Update.Program.<Update>d__5.MoveNext()

[goxiaoy]

I just fixed this by replacing the file compare function and updating bsdiff to vcdiff in my branch

https://github.com/Goxiaoy/Squirrel.Windows/tree/vcdiff

see the difference develop...Goxiaoy:vcdiff

[Obi-Dann]

@goxiaoy, thanks for that. I tried using your changes with vcdiff and deltas are generated much faster than with bsdiff and without memory exceptions or other issues, awesome work!
When I was testing it, I found that there's a boolean expression that should have && rather than ||
AurorNZ@9712eac#diff-b5814727b039661ec2313bdca28cd202b8a2e931133ed2f01f01b5ae12fc6844L132-R132

- !deltaPathRelativePaths.Contains(x.Replace(".diff", ".bsdiff")) || deltaPathRelativePaths.Contains(x.Replace(".diff", ".vcdiff"))
+ !deltaPathRelativePaths.Contains(x.Replace(".diff", ".bsdiff")) && deltaPathRelativePaths.Contains(x.Replace(".diff", ".vcdiff"))