Add GZip compression for WorkflowInstance Data field on EF Core #4876
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This update adds a compression feature for workflow state data to reduce storage needs. A compression strategy resolver, None and GZip strategies have been implemented. Migration scripts were also updated, and a superfluous file(s) were removed.
This commit includes the creation of migration files for MySql, SqlServer, Sqlite, and PostgreSql contexts in the Elsa project. The files for version V3_1 were automatically generated and are to be implemented with the necessary changes in the Up and Down methods as per the requirements.
This commit introduces two new fields to the WorkflowInstances table: "DataCompressionAlgorithm" and "DataFormat". These changes allow storing additional contextual information about the payloads of workflow instances and enhance future handling and processing of this data.
sfmskywalker
requested review from
cristinamudura,
MariusVuscanNx and
raymonddenhaan
sfmskywalker
added
elsa 3
|
I want to apply this also to the ActivityState field of the ActivityExecutionRecord table - for some customers, these fields can reach many megabytes (10MB for some scenarios that I noticed). |
Deleted all migration files related to the Management module across MySql, SqlServer, Sqlite and PostgreSql data providers. These files included data compression algorithm and data format column additions in the WorkflowInstances table.
cristinamudura
approved these changes
Feb 4, 2024
src/modules/Elsa.EntityFrameworkCore/Modules/Runtime/ActivityExecutionLogStore.cs
Outdated
Show resolved
Hide resolved
sfmskywalker
added
schema change
|
RavenDB folks did come compression algorithm comparison recently against JSON data (zstd was a clear winner): ravendb/ravendb#17678 |
Removed ICompressionStrategyResolver interface and file, and added ICompressionCodec and ICompressionCodecResolver interface. Updated relevant classes for the new interface. Specifically, added Zstd class under Compression as a new compression method. Also modified WorkflowInstanceStore.cs, None.cs, EFCoreWorkflowInstanceStore.cs, GZip.cs, and ActivityExecutionLogStore.cs for uniform compression terminology.
Wow, ZSTD is not only significantly faster, but produces smaller output as well. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the ability to store workflow state in a compressed format for EF Core.
Future updates may include other persistence provider support such as Dapper and MongoDB.
Closes #4875