Moving message contracts between namespaces causes message bus deserialisation to fail

[uglybugger]

Team

• I've assigned a team label to this issue

Severity

This is likely to block many customers if it's released.

Version

Found in 2022.4.2848 because of namespace moves but the actual bug is from much earlier.

Latest Version

I could reproduce the problem in the latest build

What happened?

The message bus serialiser is configured to support serialisation of polymorphic types. This is what we want; however, it uses Newtonsoft.Json's default $type JSON property which includes the assembly-qualified full name of the type it's serialising. This is troublesome because:

• We expect engineers to be able to move types between namespaces freely.
• We have conventions in place which encourage engineers to move message contracts to discoverable namespaces.
• We intend to move message contracts between assemblies, which will also influence the type's assembly-qualified name.

Ultimately, this results in errors like this:

Reproduction

1. Run any version of Octopus Server which contains the message bus.
2. Perform an action which drops an event onto the bus. Any event - it doesn't matter, but I'd suggest an innocuous one which will be okay to be handled twice (e.g. a MachineCreatedEvent or similar).
3. Power down the server.
4. Move the event to another namespace.
5. Delete the message bus cursors for a handler of that event.
6. Power up the server again.
7. Observe that the message contract now fails to deserialise.

Error and Stacktrace

Nevermore.ReaderException: Error reading row 1, column 4. Error resolving type specified in JSON 'Octopus.Core.Features.Maintenance.MaintenanceModeChangedEvent, Octopus.Core'. Path 'Payload.$type', line 1, position 127..
Compiled reader expression:

(DbDataReader reader, DocumentReaderContext context) => 
{
    Type deserializeAsType = Octopus.Core.Infrastructure.MessageBus.MessageBusEvent
    context.Column = 0
    String temp0 = IIF(reader.IsDBNull(0), null, reader.GetString(0))
    context.Column = 1
    Int64 temp1 = reader.GetInt64(1)
    context.Column = 2
    String temp2 = IIF(reader.IsDBNull(2), null, reader.GetString(2))
    context.Column = 3
    DateTimeOffset temp3 = IIF(reader.IsDBNull(3), default(DateTimeOffset), Convert(reader.GetValue(3), DateTimeOffset))
    context.Column = 4
    MessageBusEvent deserializedFromJson = context.DeserializeText(reader, 4, deserializeAsType)
    MessageBusEvent result = deserializedFromJson
    if (result != null)
    {
        result.Id = temp0
        result.SequenceNumber = temp1
        result.PayloadType = temp2
        result.Created = temp3
    }

    result
}

 ---> Newtonsoft.Json.JsonSerializationException: Error resolving type specified in JSON 'Octopus.Core.Features.Maintenance.MaintenanceModeChangedEvent, Octopus.Core'. Path 'Payload.$type', line 1, position 127.
 ---> Newtonsoft.Json.JsonSerializationException: Could not find type 'Octopus.Core.Features.Maintenance.MaintenanceModeChangedEvent' in assembly 'Octopus.Core, Version=2022.4.1372.0, Culture=neutral, PublicKeyToken=null'.
   at Newtonsoft.Json.Serialization.DefaultSerializationBinder.GetTypeFromTypeNameKey(StructMultiKey`2 typeNameKey)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolveTypeName(JsonReader reader, Type& objectType, JsonContract& contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, String qualifiedTypeName)
   --- End of inner exception stack trace ---
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolveTypeName(JsonReader reader, Type& objectType, JsonContract& contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, String qualifiedTypeName)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadMetadataProperties(JsonReader reader, Type& objectType, JsonContract& contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue, Object& newValue, String& id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor`1 creator, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Nevermore.Advanced.Serialization.NewtonsoftDocumentSerializer.DeserializeSmallText(String text, Type type)
   at Nevermore.Advanced.ReaderStrategies.Documents.DocumentReaderContext.DeserializeText[TDocument](DbDataReader reader, Int32 index, Type concreteType)
   at lambda_method2493(Closure , DbDataReader , DocumentReaderContext )
   at Nevermore.Advanced.ReaderStrategies.Documents.DocumentReaderStrategy.<>c__DisplayClass4_1`1.<CreateReader>b__1(DbDataReader dbDataReader)
   --- End of inner exception stack trace ---
   at Nevermore.Advanced.ReaderStrategies.Documents.DocumentReaderStrategy.<>c__DisplayClass4_1`1.<CreateReader>b__1(DbDataReader dbDataReader)
   at Nevermore.Advanced.ReadTransaction.ProcessReaderAsync[TRecord](DbDataReader reader, PreparedCommand command, CancellationToken cancellationToken)+MoveNext()
   at Nevermore.Advanced.ReadTransaction.ProcessReaderAsync[TRecord](DbDataReader reader, PreparedCommand command, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Nevermore.Advanced.ReadTransaction.<>c__DisplayClass101_0`1.<<StreamAsync>g__Execute|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Nevermore.Advanced.ReadTransaction.<>c__DisplayClass101_0`1.<<StreamAsync>g__Execute|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Nevermore.Advanced.ReadTransaction.<>c__DisplayClass101_0`1.<<StreamAsync>g__Execute|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Nevermore.Advanced.ReadTransaction.<>c__DisplayClass101_0`1.<<StreamAsync>g__Execute|0>d.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at Nevermore.Advanced.ThreadSafeAsyncEnumerable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
   at Nevermore.Advanced.ThreadSafeAsyncEnumerable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
   at Nevermore.Advanced.ThreadSafeAsyncEnumerable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|424_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
   at System.Linq.AsyncEnumerable.<ToListAsync>g__Core|424_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/ToList.cs:line 36
   at Octopus.Core.Infrastructure.MessageBus.Transports.SqlServer.SqlMessageBusTransport.GetNextEventAfter(Int64 cursorSequenceNumber, IRawReadQueryExecutor tx, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/MessageBus/Transports/SqlServer/SqlMessageBusTransport.cs:line 224
   at Octopus.Core.Infrastructure.MessageBus.Transports.SqlServer.SqlMessageBusTransport.DispatchPendingEventsFor(String consumerGroupId, StartingPoint startingPoint, Func`3 dispatchAction, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/MessageBus/Transports/SqlServer/SqlMessageBusTransport.cs:line 148
   at Octopus.Core.Infrastructure.MessageBus.Transports.SqlServer.SqlMessageBusTransport.DispatchPendingEventsFor(String consumerGroupId, StartingPoint startingPoint, Func`3 dispatchAction, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/MessageBus/Transports/SqlServer/SqlMessageBusTransport.cs:line 161
   at Octopus.Core.Infrastructure.MessageBus.Transports.SqlServer.SqlTransportCrossClusterDecorator.DispatchPendingEventsFor(String consumerGroupId, StartingPoint startingPoint, Func`3 dispatchAction, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/MessageBus/Transports/SqlServer/SqlTransportCrossClusterDecorator.cs:line 59
   at Octopus.Core.Infrastructure.MessageBus.MessagePump.<PumpMessages>b__12_0(CancellationToken ct) in ./source/Octopus.Core/Infrastructure/MessageBus/MessagePump.cs:line 98
   at Polly.AsyncPolicy.<>c__DisplayClass40_0.<<ImplementationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)

More Information

No response

Workaround

As a bit of a brutal workaround, it's possible to just delete all of the events in the message bus's table.

1. Stop all Octopus Server instances.
2. UPDATE MessageBusCursor SET SequenceNumber = (SELECT MAX(SequenceNumber) FROM MessageBusEvent);
3. Start all Octopus Server instances.

This will ensure that the server can at least serialise and deserialise new events that it writes itself

[octoreleasebot]

Release Note: Update message bus deserialization to fix message bus event pump related failures

[gupta-kartik]

Fix released in: 2022.4.3438

[Octobob]

🎉 The fix for this issue has been released in:

Release stream	Release
2022.4	2022.4.3438
2023.1+	all releases