Offline Package Drop may silently fail due to out of memory condition on the Octopus Server

[donnybell]

Team

• I've assigned a team label to this issue

Severity

Affecting at least one cloud instance

Version

Discovered in 2021.3.X

Latest Version

Not applicable

What happened?

It is possible for an artifact to not get created for an offline package drop where no error is logged in the task. This can happen when the Octopus Server server runs out of memory during the process. This creates confusion with the customer as they are unaware that something is wrong, other than the lack of an artifact.

Reproduction

Was not able to reproduce locally.

Error and Stacktrace

https://seq.octopushq.com/#/events/?filter=@Id%20%3D%20%27event-06cecc91bb0408d9cf24759b01000000%27&showExpanded&permalinkId=permalink-8500

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(ReadOnlySpan`1 buffer)
   at SharpCompress.Writers.Zip.ZipCentralDirectoryEntry.Write(Stream outputStream)
   at SharpCompress.Writers.Zip.ZipWriter.Dispose(Boolean isDisposing)
   at SharpCompress.Writers.AbstractWriter.Dispose()
   at SharpCompress.Archives.Zip.ZipArchive.SaveTo(Stream stream, WriterOptions options, IEnumerable`1 oldEntries, IEnumerable`1 newEntries)
   at SharpCompress.Archives.AbstractWritableArchive`2.SaveTo(Stream stream, WriterOptions options)
   at Octopus.Server.Orchestration.Targets.OfflineDrop.OfflineDropTarget.CollectArtifact(TargetManifest targetManifest, ITaskLog taskLog) in ./source/Octopus.Server/Orchestration/Targets/OfflineDrop/OfflineDropTarget.cs:line 45
   at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.CollectOfflineDropsAsArtifactsStepController.<>c__DisplayClass3_1`3.<<Execute>b__3>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/PlannedStepControllers/CollectOfflineDropsAsArtifactsStepController.cs:line 58
--- End of stack trace from previous location ---
   at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.<>c__DisplayClass4_0`2.<<Execute>b__0>d.MoveNext() in ./source/Octopus.Server/Infrastructure/Orchestration/UnitsOfWork/UnitOfWorkExecutor.cs:line 74
--- End of stack trace from previous location ---
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.DoAsync(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Core/Infrastructure/UnitsOfWork/UnitOfWorkExtensionMethods.cs:line 73
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.DoAsync(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Core/Infrastructure/UnitsOfWork/UnitOfWorkExtensionMethods.cs:line 73
   at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2](Func`4 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Server/Infrastructure/Orchestration/UnitsOfWork/UnitOfWorkExecutor.cs:line 75

More Information

Internal slack thread: https://octopusdeploy.slack.com/archives/CNHBHV2BX/p1639071833152700

Workaround

Increase the available memory for Octopus Server

[N-lson]

Release Note: Errors that occur in offline package drop steps no longer silently fail

[Octobob]

🎉 The fix for this issue has been released in:

Release stream	Release
2021.3	2021.3.10886
2022.1+	all releases