Could not find a datatype with identifier xxxx

[HalldorLyngmo]

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

Multiple v8 versions and v10.3.2

Bug summary

This is an issue I've only seen on Umbraco Cloud in connection with deployments. Thankfully it's not a frequent issue and it does seem to resolve itself after a couple of deployments in most cases. Nevertheless, it's can be quite frustrating for the customers to watch out for this, especially when it's a baseline/child setup.

It's been there since the beginning of v8 and even though I've not seen it reported on v9 and v10 I was able to reproduce the issue on a v10 project.

In short, there will be a deployment with changes to document and data types where instead of a successful extraction there will be a boot failed or an unsuccessful extraction.

[BootFailedException: Boot failed: Umbraco cannot run. See Umbraco's log file for more details. 

-> Umbraco.Core.Exceptions.BootFailedException: Boot failed. 

-> System.ArgumentException: Could not find a datatype with identifier 1163. 
Parameter name: id 

The data type can be found in the database and it's not always the same data type which will be the issue.
The workaround is simple. After you restart the site it will boot up just fine and you can extract the changes successfully.

My initial thought was that this was a Deploy issue with extracting changes before Umbraco had completed building the cache on boot-up. Following that, I asked Andy to take a look at a site where the issue could be reproduced consistently. Turns out the failure to boot-up was a red herring. This is what he found:

I don't think the issue sits within Deploy though. Here is Deploy is emitting an event saying "I've updated a data type", the CMS responds to that and the exception is thrown within there. Specifically within updating NuCache.

I tracked down a suspicious-looking TODO within the code path of the stack trace that looks relevant and suggests a form of race condition perhaps causing the issue (here in Umbraco 8, but it's still there in the latest code).

The TODO specifically mentions a potential race condition with data types and content types.

 // TODO: need to add a datatype lock
 // this is triggering datatypes reload in the factory, and right after we create some
 // content types by loading them ... there's a race condition here, which would require
 // some locking on datatypes

---

Here is a screenshot from a previous debug session where you can see how many datatypes are in publishedDataTypes when I hit the extraction error and after I restart the site and trigger extraction again.

Specifics

Exception has been thrown by the target of an invocation. ---> System.ArgumentException: Could not find a datatype with identifier 1134.
Parameter name: id
   at Umbraco.Core.Models.PublishedContent.PublishedContentTypeFactory.GetDataType(Int32 id)
   at Umbraco.Core.Models.PublishedContent.PublishedPropertyType..ctor(String propertyTypeAlias, Int32 dataTypeId, Boolean isUserProperty, ContentVariation variations, PropertyValueConverterCollection propertyValueConverters, IPublishedModelFactory publishedModelFactory, IPublishedContentTypeFactory factory)
   at Umbraco.Core.Models.PublishedContent.PublishedPropertyType..ctor(IPublishedContentType contentType, PropertyType propertyType, PropertyValueConverterCollection propertyValueConverters, IPublishedModelFactory publishedModelFactory, IPublishedContentTypeFactory factory)
   at Umbraco.Core.Models.PublishedContent.PublishedContentTypeFactory.CreatePropertyType(IPublishedContentType contentType, PropertyType propertyType)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Umbraco.Core.Models.PublishedContent.PublishedContentType..ctor(IContentTypeComposition contentType, IPublishedContentTypeFactory factory)
   at Umbraco.Core.Models.PublishedContent.PublishedContentTypeFactory.CreateContentType(IContentTypeComposition contentType)
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Umbraco.Web.PublishedCache.NuCache.ContentStore.UpdateDataTypesLocked(IEnumerable`1 dataTypeIds, Func`2 getContentType)
   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.Notify(JsonPayload[] payloads)
   at Umbraco.Core.PublishedModelFactoryExtensions.WithSafeLiveFactoryReset(IPublishedModelFactory factory, Action action)
   at Umbraco.Web.Cache.DataTypeCacheRefresher.Refresh(JsonPayload[] payloads)
   at Umbraco.Core.Sync.ServerMessengerBase.Deliver[TPayload](ICacheRefresher refresher, TPayload[] payload)
   at Umbraco.Web.Cache.DistributedCache.RefreshByPayload[TPayload](Guid refresherGuid, TPayload[] payload)
   at Umbraco.Web.Cache.DistributedCacheExtensions.RefreshDataTypeCache(DistributedCache dc, IDataType dataType)
   at Umbraco.Web.Cache.DistributedCacheBinder.DataTypeService_Saved(IDataTypeService sender, SaveEventArgs`1 e)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Umbraco.Web.Cache.DistributedCacheBinder.HandleEvents(IEnumerable`1 events)
   at Umbraco.Deploy.DeployEventDispatcher.ScopeExitCompleted()
   at Umbraco.Core.Events.QueuingEventDispatcherBase.ScopeExit(Boolean completed)
   at Umbraco.Core.Scoping.Scope.<>c__DisplayClass85_0.<RobustExit>b__1()
   at Umbraco.Core.Scoping.Scope.TryFinally(Int32 index, Action[] actions)
   at Umbraco.Core.Scoping.Scope.TryFinally(Int32 index, Action[] actions)
   at Umbraco.Core.Scoping.Scope.RobustExit(Boolean completed, Boolean onException)
   at Umbraco.Core.Scoping.Scope.Dispose()
   at Umbraco.Deploy.Work.WorkItems.DiskReadWorkItem.<ExecuteAsyncSub>d__16.MoveNext()

Steps to reproduce

I don't have a way to reproduce it without using Deploy and UDA files I got from a customer project.
The method would be to:

1. Create a v8 project (or v10) on Umbraco Cloud.
2. Extract one set of UDA files to the environment.
3. Copy modified UDA files to the environment.
4. Trigger an extraction and the exception should pop up in the logs.

Reach out to me and I'll ship you the two sets of UDA files.

Expected result / actual result

I expect a successful extraction and the Nucache to be updated without any issues.

---

This item has been added to our backlog AB#25490

[kjac]

Hi @HalldorLyngmo,

Thank you for reaching out 😄

We're going to dig into this as soon as we can - it's been added to our backlog.

[nikolajlauridsen]

I've spent some time looking into this, and I've found out what's happening, and how you can reproduce this issue manually, unfortunately. However, this is actually an issue in deploy, and not something we can fix in the CMS. This is gonna be a bit long, but this was a tricky one to nail down the cause of.

The issue stems from deploy suppressing notifications during deployment, and then later "playing them back" once the deployment is done.

The reproduction steps are key to understanding why this happens so here they are:

You will need a site with deploy that has:

• 1 Composition
• 1 Document type
• And we'll be creating multiple data types

On cloud:

1. Set up the document type so it has a property type, make sure you use the "create new" option instead of using the built-in ones, so you create a new data type
2. Set up the composition with a property type as well
3. Set the document type to use the composition
4. Now pull the changes to your local site and deploy them, but do not restart the site, this is important
5. Go back to cloud and update the configuration of the data type in the document type
6. Create a new data type and add it to the composition
7. Pull and deploy the changes

This will cause deploy to blow up, but only sometimes.

What's happening is that deploy will suppress notifications, and then do all the database updates/creates to the datatypes and then start publishing notifications for the cache afterwards. Occasionally deploy will then publish the cache update notification for the updated data type first, not the created one, and this makes everything explode.

This happens because when the notification is published for the updated data type, we'll first update that data type in the published data type cache, and as expected, we're all happy.

Next, we'll update the data type in the ContentStore, this involves finding any document types using the data type and updating those, this happens right here now since we're just updating a data type, our document type will be in the content types cache, and will therefore be picked up for an update.

Now updating the content type in the content type cache involves updating it from the database, which happens here, when getting this update content type from the database, we'll also get all the updated composition property types, which now includes the new data type we added in step 6...

And this is exactly why this blows up because after we've gotten the updated content type we need to make this into a PublishedContentType, which can be used as a published element. This entails creating all of the composition IPublishedPropertyTypes which happens here, now when creating a published property type, we need the data type for it, so we try to do this using the cache, but now we cannot find that data type in the cache (the newly created one), because deploy has not published the notification to add this data type to the cache yet, which makes then makes it explode.

So the TL;DR issue: Deploy attempts to update a data type (and therefore content types), before having added the necessary dependencies.

What can be done

Now I'm no deploy expert by any means, so ultimately this is up to the deploy team, but a possible way to solve this could be to ensure that all saved and deleted notifications are grouped together into a single notification instead of multiple and then published, this would mean that all data types would be updated in the data type cache first before we start looking at updating the content store, this would have the added benefit that it should be more performant.

One thing worth noting about this is that it will require some change in the CMS, is that our DistributedCacheBinder currently doesn't support this as it'll always iterate through the entities and refresh one at a time, as you can see here, so this would need to be updated, but that should be pretty trivial.

Since this is an issue in deploy, and the decision on how to fix it is in their hands, I'll go ahead and transfer this issue over to the deploy tracker.

[ronaldbarendse]

I've been debugging this issue and can confirm it is indeed triggered by Deploy saving multiple data types in a single scope (as part of a schema deployment), but only when one of those is a newly added data type. However, the exception is caused by the CMS not correctly updating an internal PublishedDataType lookup dictionary in PublishedContentTypeFactory:

• This lookup dictionary is populated with all data types on the first run (hence it works again after a restart), but once that's done, only the data types specified in the NotifyDataTypeChanges() call will be updated;
• After saving a data type, the published content cache (NuCache) is rebuild, which recreates the PublishedContentType that requires all data types used on that content type to exist in this internal lookup dictionary;
• If multiple data types are saved in a single scope, multiple rebuilds are done (the CMS already avoids this for content, media and member types by grouping the events);
• If the first saved data type requires updating a content/media/member type that also contains the other newly added data type, the internal lookup dictionary won't contain that new data type and will throw this exception...

TL;DR: The CMS currently doesn't correctly handle saving multiple data types when creating a PublishedContentType as part of rebuilding the published content cache (NuCache), resulting in the Could not find a datatype with identifier... exception.

---

There are some differences when data types are saved as part of a schema deployment in Deploy (compared to when saved in the back-office), making the exception more likely to occur:

• A schema deployment can easily contain multiple changes to data types in a single scope, while changes done in the back-office will create separate scopes for each operation/request;
• Deploy supresses all events by using a custom IEventDispatcher when creating a new scope, ensuring the handlers aren't raised/invoked after completing the scope, but the IDistributedCacheBinder does handle the events to ensure the cache refreshers are notified and updated correctly;
• All events are handled in this custom IEventDispatcher, compared to the CMS that only handles the 'last in' events in the QueuingEventDispatcher - I'm hesitant to update Deploy to do the same, but this could reduce the amount of (duplicate) events that are handled;
• Because all of this, multiple DataTypeCacheRefresher payloads are sent, triggering multiple rebuilds of the published content cache (NuCache).

We'll update Deploy v4+ to group the data type saved events/notifications as @nikolajlauridsen also suggested, so users can easily update to the latest Deploy version. I'll also create a PR for the CMS to do the same, as this can use most of the existing event grouping that's already implemented for content/media/member types 👍🏻

[AndyButland]

This fix will be released in the next patch releases for Deploy, due next Tuesday.