fabric:/System/FaultAnalysisService unhealthy on a new DevCluster

[mkosieradzki]

When creating dev cluster (tried both powershell and VS2015, with rebooting machine, deleting C:\SFDevCluster etc) in Service Fabric Explorer FaultAnalysisService is unhealthy:

due to:
Error event: SourceId='System.FM', Property='State'.
Partition is below target replica or instance count.

Inside _Node3 (Primary) I have a following warning:
Unhealthy event: SourceId='System.RA', Property='ReplicaOpenStatus', HealthState='Warning', ConsiderWarningAsError=false.
Replica had multiple failures during open. Error =System.IO.FileNotFoundException (-2147024894)
The system cannot find the file specified. (Exception from HRESULT: 0x80070002)
at System.Fabric.Data.Log.Interop.NativeLog.IKPhysicalLogManager.EndOpenLogContainer(IFabricAsyncOperationContext Context, IKPhysicalLogContainer& Result)
at System.Fabric.Data.Log.Interop.PhysicalLogManager.b__5(IFabricAsyncOperationContext Context)
at System.Fabric.Interop.AsyncCallOutAdapter2`1.Finish(IFabricAsyncOperationContext context, Boolean expectedCompletedSynchronously)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Fabric.Data.Log.LogManager.d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.KtlLogManager.d__33.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.KtlLogManager.d__1a.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.LogManager.d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at Microsoft.ServiceFabric.Replicator.LoggingReplicator.d__54.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.DynamicStateManager.d__60.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.TransactionalReplicator.d__1d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.StatefulServiceReplica.d__a.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Services.Runtime.StatefulServiceReplicaAdapter.d__0.MoveNext()

[masnider]

This is usually just that you're out of diskspace, which results in the reliable log file not being able to be generated. The default installation requires around 10Gb of disk space today. Can you clean up the cluster (even just remove the cluster), delete the SFDevCluster folder, and check out your available disk space?

[mkosieradzki]

Thanks Matt. That's it. I have 7GB after cleaning (I still had 5GB free so I didn't suppose it was the reason). Thanks.

[masnider]

No problem. Let us know if you run into anything else.