Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NuCache "panic" #4992

Closed
zpqrtbnk opened this issue Mar 14, 2019 · 10 comments

Comments

Projects
None yet
4 participants
@zpqrtbnk
Copy link
Contributor

commented Mar 14, 2019

Under heavy multithreaded loads, or on some yet unknown situations, NuCache can panic in ContentStore.CreateSnapshot(). It has been reported once by Shannon in september, 2018 - then again by Bjarke a few months later - so it's quite rare but happens nevertheless.

And then there is this Our thread so it definitively is a thing.

@crick3t

This comment has been minimized.

Copy link

commented Mar 14, 2019

For us it happens at every import, so we can replicate it.
Let me know if you need more information, code sample or something else.

(I am not sure how this issue tracing handled. Should I comment here or still on umbraco.com from now, or does not matter?)

Peter

@crick3t

This comment has been minimized.

Copy link

commented Mar 14, 2019

We have tried to replaced CreateAndSave with CreateContent, but we have the same issue...

While the import is running I get the following error in a pop up screen when I click on something in the back office:

For example - Settings - Log Viewer

An error occured

An error occurred when trying to create a controller of type 'LogViewerController'. Make sure that the controller has a parameterless public constructor.
Exception Details
System.InvalidOperationException: An error occurred when trying to create a controller of type 'LogViewerController'. Make sure that the controller has a parameterless public constructor.

Stacktrace
   at System.Web.Http.Dispatcher.DefaultHttpControllerActivator.Create(HttpRequestMessage request, HttpControllerDescriptor controllerDescriptor, Type controllerType)
   at System.Web.Http.Controllers.HttpControllerDescriptor.CreateController(HttpRequestMessage request)
   at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()

Inner Exception
System.Exception: panic
   at Umbraco.Web.PublishedCache.NuCache.ContentStore.CreateSnapshot()
   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.GetElements(Boolean previewDefault)
   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshot.get_Elements()
   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshot.get_Members()
   at Umbraco.Web.Runtime.WebRuntimeComposer.<>c.<Compose>b__0_2(IFactory factory)
   at Umbraco.Core.Composing.LightInject.LightInjectContainer.<>c__DisplayClass20_0`1.<Register>b__0(IServiceFactory f)
   at DynamicMethod(Object[] )
   at LightInject.ServiceContainer.<>c__DisplayClass150_0.<WrapAsFuncDelegate>b__0() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3798
   at LightInject.PerRequestLifeTime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6207
   at DynamicMethod(Object[] )
   at LightInject.ServiceContainer.GetInstance(Type serviceType) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3442
   at Umbraco.Core.Composing.LightInject.LightInjectContainer.GetInstance(Type type)
   at Umbraco.Core.FactoryExtensions.GetInstance[T](IFactory factory)
   at Umbraco.Web.Runtime.WebRuntimeComposer.<>c.<Compose>b__0_6(IFactory factory)
   at Umbraco.Core.Composing.LightInject.LightInjectContainer.<>c__DisplayClass20_0`1.<Register>b__0(IServiceFactory f)
   at DynamicMethod(Object[] )
   at LightInject.ServiceContainer.GetInstance(Type serviceType) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3442
   at Umbraco.Core.Composing.LightInject.LightInjectContainer.GetInstance(Type type)
   at Umbraco.Core.FactoryExtensions.GetInstance[T](IFactory factory)
   at Umbraco.Web.WebApi.UmbracoApiControllerBase..ctor()
   at DynamicMethod(Object[] )
   at LightInject.ServiceContainer.<>c__DisplayClass150_0.<WrapAsFuncDelegate>b__0() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3798
   at LightInject.PerRequestLifeTime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6207
   at DynamicMethod(Object[] )
   at LightInject.ServiceContainer.TryGetInstance(Type serviceType) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3498
   at LightInject.WebApi.LightInjectWebApiDependencyScope.GetService(Type serviceType) in C:\projects\lightinject-webapi\build\tmp\Net46\Binary\LightInject.WebApi\LightInject.WebApi.cs:line 187
   at System.Web.Http.Dispatcher.DefaultHttpControllerActivator.GetInstanceOrActivator(HttpRequestMessage request, Type controllerType, Func`1& activator)
   at System.Web.Http.Dispatcher.DefaultHttpControllerActivator.Create(HttpRequestMessage request, HttpControllerDescriptor controllerDescriptor, Type controllerType)

When I click on the Content tab at the left top corner:

Received an error from the server
An error occured
An error occurred when trying to create a controller of type 'ContentController'. Make sure that the controller has a parameterless public constructor.

Exception Details
System.InvalidOperationException: An error occurred when trying to create a controller of type 'ContentController'. Make sure that the controller has a parameterless public constructor.

Stacktrace
at System.Web.Http.Dispatcher.DefaultHttpControllerActivator.Create(HttpRequestMessage request, HttpControllerDescriptor controllerDescriptor, Type controllerType)
   at System.Web.Http.Controllers.HttpControllerDescriptor.CreateController(HttpRequestMessage request)
   at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()
Inner Exception
System.Exception: panic
at Umbraco.Web.PublishedCache.NuCache.ContentStore.CreateSnapshot()
   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.GetElements(Boolean previewDefault)
   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshot.get_Elements()
   at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshot.get_Members()
   at Umbraco.Web.Runtime.WebRuntimeComposer.<>c.<Compose>b__0_2(IFactory factory)
   at Umbraco.Core.Composing.LightInject.LightInjectContainer.<>c__DisplayClass20_0`1.<Register>b__0(IServiceFactory f)
   at DynamicMethod(Object[] )
   at LightInject.ServiceContainer.<>c__DisplayClass150_0.<WrapAsFuncDelegate>b__0() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3798
   at LightInject.PerRequestLifeTime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6207
   at DynamicMethod(Object[] )
   at LightInject.ServiceContainer.GetInstance(Type serviceType) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3442
   at Umbraco.Core.Composing.LightInject.LightInjectContainer.GetInstance(Type type)
   at Umbraco.Core.FactoryExtensions.GetInstance[T](IFactory factory)
   at Umbraco.Web.Runtime.WebRuntimeComposer.<>c.<Compose>b__0_6(IFactory factory)
   at Umbraco.Core.Composing.LightInject.LightInjectContainer.<>c__DisplayClass20_0`1.<Register>b__0(IServiceFactory f)
   at DynamicMethod(Object[] )
   at LightInject.ServiceContainer.<>c__DisplayClass150_0.<WrapAsFuncDelegate>b__0() in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3798
   at LightInject.PerRequestLifeTime.GetInstance(Func`1 createInstance, Scope scope) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 6207
   at DynamicMethod(Object[] )
   at LightInject.ServiceContainer.TryGetInstance(Type serviceType) in C:\projects\lightinject\src\LightInject\LightInject.cs:line 3498
   at LightInject.WebApi.LightInjectWebApiDependencyScope.GetService(Type serviceType) in C:\projects\lightinject-webapi\build\tmp\Net46\Binary\LightInject.WebApi\LightInject.WebApi.cs:line 187
   at System.Web.Http.Dispatcher.DefaultHttpControllerActivator.GetInstanceOrActivator(HttpRequestMessage request, Type controllerType, Func`1& activator)
   at System.Web.Http.Dispatcher.DefaultHttpControllerActivator.Create(HttpRequestMessage request, HttpControllerDescriptor controllerDescriptor, Type controllerType)
@zpqrtbnk

This comment has been minimized.

Copy link
Contributor Author

commented Mar 14, 2019

Please comment here, yes, now that it is a confirmed issue.

FWIW, I have been able to reproduce the panic in a unit test - now to fix it...

@zpqrtbnk

This comment has been minimized.

Copy link
Contributor Author

commented Mar 14, 2019

Assuming I can push a candidate fix for this issue in a branch, would you be up to build your own Umbraco and test? Are you running these tests with the stock 8.0.0 version, or something you have built?

@crick3t

This comment has been minimized.

Copy link

commented Mar 14, 2019

I have downloaded it from NuGet, nothing custom built.
I can try to figure it out and upgrade it, but I am from the UK, so I can only test it tomorrow earliest.

@zpqrtbnk

This comment has been minimized.

Copy link
Contributor Author

commented Mar 14, 2019

No problem, will ping you tomorrow.

@nul800sebastiaan

This comment has been minimized.

Copy link
Member

commented Mar 14, 2019

@zpqrtbnk Note: the build server will build it for you so you can link to the build output there, that way nobody needs to make a custom build.

@zpqrtbnk

This comment has been minimized.

Copy link
Contributor Author

commented Mar 15, 2019

A version with the fix is available here on our build server. Click on the top-right Artifacts link and download the NuGet packages from nupkg. If you can test, that would be super cool.

Note that the version is going to be 8.1.0-alpha.2673 - this is a temp build, use it for testing and for testing only.

@crick3t

This comment has been minimized.

Copy link

commented Mar 15, 2019

That's great, thanks.

I have updated it and rerun our import.

It was running fine for more than an hour and the contents were created correclty, so the fix seems to be working.
But then we got one error and that stopped the replication as it seems like Umbraco never returned with an error.
Weird, maybe that issue is on our side or the error is just not bulbing up and stuck on SaveAndPublish. I have to check that one.

It contains a NuCache error, but I am sure if this error is related to the original one, we talk about here if it is a new one.

Anyway, I am going to rerun the import and see if it happens again.

System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. 
Parameter name: length
 at CSharpTest.Net.IO.TransactedCompoundFile.Write(UInt32 handle, Byte[] bytes, Int32 offset, Int32 length)
 at CSharpTest.Net.Storage.BTreeFileStoreV2.Update[T](IStorageHandle handleIn, ISerializer`1 serializer, T node)
 at CSharpTest.Net.Collections.BPlusTree`2.StorageCache.OnItemRemoved(KeyValuePair`2 item)
 at CSharpTest.Net.Collections.LurchTable`2.TryDequeue(Predicate`1 predicate, KeyValuePair`2& value)
 at CSharpTest.Net.Collections.BPlusTree`2.StorageCache.Flush()
 at CSharpTest.Net.Collections.BPlusTree`2.StorageCache.Commit()
 at CSharpTest.Net.Collections.BPlusTree`2.CommitChanges(Boolean requiresLock)
 at Umbraco.Web.PublishedCache.NuCache.ContentStore.Release(WriteLockInfo lockInfo, Boolean commit)
 at Umbraco.Web.PublishedCache.NuCache.ContentStore.Set(ContentNodeKit kit)
 at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.NotifyLocked(IEnumerable`1 payloads, Boolean& draftChanged, Boolean& publishedChanged)
 at Umbraco.Web.PublishedCache.NuCache.PublishedSnapshotService.Notify(JsonPayload[] payloads, Boolean& draftChanged, Boolean& publishedChanged)
 at Umbraco.Web.Cache.ContentCacheRefresher.Refresh(JsonPayload[] payloads)
 at Umbraco.Core.Sync.ServerMessengerBase.DeliverLocal[TPayload](ICacheRefresher refresher, TPayload[] payload)
 at Umbraco.Core.Sync.ServerMessengerBase.Deliver[TPayload](ICacheRefresher refresher, TPayload[] payload)
 at Umbraco.Core.Sync.ServerMessengerBase.PerformRefresh[TPayload](ICacheRefresher refresher, TPayload[] payload)
 at Umbraco.Web.Cache.DistributedCache.RefreshByPayload[TPayload](Guid refresherGuid, IEnumerable`1 payloads)
 at Umbraco.Web.Cache.DistributedCacheExtensions.RefreshContentCache(DistributedCache dc, TreeChange`1[] changes)
 at Umbraco.Web.Cache.DistributedCacheBinder.ContentService_TreeChanged(IContentService sender, EventArgs args)
 at Umbraco.Core.Events.EventDefinition`2.RaiseEvent()
 at Umbraco.Core.Events.QueuingEventDispatcher.ScopeExitCompleted()
 at Umbraco.Core.Events.QueuingEventDispatcherBase.ScopeExit(Boolean completed)
 at Umbraco.Core.Scoping.Scope.<>c__DisplayClass72_0.<RobustExit>b__1()
 at Umbraco.Core.Scoping.Scope.TryFinally(Int32 index, Action[] actions)
 at Umbraco.Core.Scoping.Scope.TryFinally(Int32 index, Action[] actions)
 at Umbraco.Core.Scoping.Scope.RobustExit(Boolean completed, Boolean onException)
 at Umbraco.Core.Scoping.Scope.DisposeLastScope()
 at Umbraco.Core.Scoping.Scope.Dispose()
 at Umbraco.Core.Services.Implement.ContentService.SaveAndPublish(IContent content, String culture, Int32 userId, Boolean raiseEvents)
 at cmdportal.com.Controllers.ContentController.ImportArticle(Object objarticle) 
 in C:\CMDSVN\Websites\Controllers\ContentController.cs:line 1459
@zpqrtbnk

This comment has been minimized.

Copy link
Contributor Author

commented Mar 15, 2019

Thanks for the feedback. It seems that the original issue has been fixed, so I am moving this issue to review, so the fix can be merged and part of a next release.

Now about that new exception... I have created #4998 to track it. Let's discuss it there.

@zpqrtbnk zpqrtbnk removed their assignment Mar 19, 2019

@RobertCopilau RobertCopilau self-assigned this Mar 20, 2019

@ghost ghost removed the state/review label Mar 20, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.