Intermittent IIS App Pool Crash using EntityFramework 6.1.3

[AppSecSeanner]

We have an ongoing issue where our w3wp.exe process crashing intermittently on a ASP.NET MVC controller method that is executing a complicated linq query against EntityFramework 6.1.3.

We have enabled DebugDiag (2.2) on the server to capture dump files. The dump files show a Second Chance Exception of type AccessViolationException when loaded into WinDbg. With SOS extensions loaded !clrstack shows very little information, only 2 GCFrame lines. There is also an unmanaged stack trace when using !dumpstack that just appears to be raising the exception.

Recently DebugDiag was configured with a rule to produce a dump on First Chance Exceptions of type StackOverflow. Analyzing the dump with symbols loaded produces the attached from WinDbg when executing !dumpstack -EE.

There seems to be a problem in EntityFramework where it is calling ExpressionConverter+ConstantTranslator.TypedTranslate for a portion of our query. When examining the managed and unmanaged stack using !dumpstack there are many repeating calls to exception raising code in the unmanaged portion of the stack.

When we view the stack objects with !dso we seem to consume the stack with references to the same System.Linq.Enumerable+WhereSelectEnumerableIterator object.

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(b134.c04c): Stack overflow - code c00000fd (first/second chance not available)
clr!AdjustContextForWriteBarrier+0xb2:
000007fc'91344336 e815d3caff      call    clr!_security_check_cookie (000007fc'90ff1650)

!dso
OS Thread Id: 0xc04c (81)
RSP/REG          Object           Name
00000047037A75F0 00000045c8bc74b8 System.Linq.Enumerable+WhereSelectEnumerableIterator'2[[MyCompany.Entities.CurriculumGroupCourse, MyCompany.Entities],[System.Guid, mscorlib]]
00000047037A7670 00000045c8bc74b8 System.Linq.Enumerable+WhereSelectEnumerableIterator'2[[MyCompany.Entities.CurriculumGroupCourse, MyCompany.Entities],[System.Guid, mscorlib]]
00000047037A7688 00000045c8bc74b8 System.Linq.Enumerable+WhereSelectEnumerableIterator'2[[MyCompany.Entities.CurriculumGroupCourse, MyCompany.Entities],[System.Guid, mscorlib]]
00000047037A7690 00000045c83abc30 <Unloaded Type>

• This repeats many many times:

00000047037A7698 00000045c83abc30 <Unloaded Type>
00000047037A76B8 0000004682332d18 System.RuntimeType
00000047037A76C8 000000454275cb70 System.Reflection.RuntimeMethodInfo
00000047037A76D0 00000045c8c39530 System.Linq.Expressions.LogicalBinaryExpression
00000047037A7C50 00000045c8bc74b8 System.Linq.Enumerable+WhereSelectEnumerableIterator'2[[MyCompany.Entities.CurriculumGroupCourse, MyCompany.Entities],[System.Guid, mscorlib]]
00000047037A7CD0 00000045c8bc74b8 System.Linq.Enumerable+WhereSelectEnumerableIterator'2[[MyCompany.Entities.CurriculumGroupCourse, MyCompany.Entities],[System.Guid, mscorlib]]
00000047037A7CE8 00000045c8bc74b8 System.Linq.Enumerable+WhereSelectEnumerableIterator'2[[MyCompany.Entities.CurriculumGroupCourse, MyCompany.Entities],[System.Guid, mscorlib]]
00000047037A7CF0 00000045c83abc30 <Unloaded Type>

• We have verified we don't have any recursive calls.
• Calling the endpoint with the same URL does not reproduce the issue.
• The crash happens when making the same query but with different paramaters.
• We are not able to reproduce this in local environments, it only shows up on production.

It looks like there is a bug in EntityFramework which is causing the intermittent issue. Can the EF team provide any insight?

Attached is a dump of the managed stack.
WinDbgStack.txt

[divega]

@sglencross can you provide the LINQ query?

Also, can you try setting context.Configuration.UseDatabaseNullSemantics = true before that particular query? We have had previous reports that the default (UseDatabaseNullSemantics = false) can cause larger trees that in a very complicated query can cause stack overflow.

[AppSecSeanner]

Setting this before our query is problematic as we have abstracted our context from the linq operations on the DBSet. The query related to the stack overflow above is spread out across several classes and would be difficult to summarize without changing its behaviour.

We do have a simpler query not related to the stack dump above but has also intermittently crashed our IIS app pool. We suspect it is also having the same kind of issue.

//_dbTerms is a IDbSet<Term> exposed on the context
public IEnumerable<TermModel> Execute(int languageId, Guid clientId)
{
    var clientSpecificTerms = _dbTerms.Where(t => t.ClientId == clientId && t.LanguageId == languageId);

    var languageFallbackTerms = _dbTerms.Where(t => t.ClientId == null && t.LanguageId == languageId &&
                                                                    clientSpecificTerms.All(clientTerm => clientTerm.Key != t.Key));

    var clientDefaultTerms = _dbTerms.Where(t => t.ClientId == clientId && t.LanguageId == Languages.English.Id &&
                                                                clientSpecificTerms.All(clientTerm => clientTerm.Key != t.Key) &&
                                                                languageFallbackTerms.All(languageFallback => languageFallback.Key != t.Key));

    var defaultTerms = _dbTerms.Where(t => t.LanguageId == Languages.English.Id && t.ClientId == null &&
                                                            clientSpecificTerms.All(clientTerm => clientTerm.Key != t.Key) &&
                                                            languageFallbackTerms.All(languageFallback => languageFallback.Key != t.Key) &&
                                                            clientDefaultTerms.All(clientDefaultTerm => clientDefaultTerm.Key != t.Key));

    return clientSpecificTerms.Union(languageFallbackTerms)
                              .Union(clientDefaultTerms)
                              .Union(defaultTerms)
                              .OrderBy(t => t.Key)
                              .Select(t => new TermModel { Key = t.Key, Value = t.Value })
                              .ToList();
}

This is the generated SQL

SELECT 1 AS [C1], [Project2].[Key] AS[Key], [Limit1].[Value] AS[Value] FROM 
    (SELECT[Distinct1].[Key] AS[Key]FROM ( SELECT DISTINCT[Extent1].[Key] AS[Key] FROM[dbo].[Terms] AS[Extent1] 
            WHERE([Extent1].[LanguageId] IN(@languageId,1)) 
            AND([Extent1].[ClientId] = @clientId OR[Extent1].[ClientId] IS NULL) 
        ) AS[Distinct1] 
    ) AS [Project2] 
    OUTER APPLY (SELECT TOP (1) [Project3].[Value] AS [Value] FROM 
        (SELECT CASE WHEN([Extent2].[LanguageId] = @languageId) THEN cast(1 as bit) WHEN([Extent2].[LanguageId] <> @languageId) THEN cast(0 as bit) END AS[C1], 
                CASE WHEN([Extent2].[ClientId] = @clientId) THEN cast(1 as bit) WHEN(NOT(([Extent2].[ClientId] = @clientId) AND((CASE WHEN([Extent2].[ClientId] IS NULL) THEN cast(1 as bit) ELSE cast(0 as bit) END) = 0))) THEN cast(0 as bit) END AS[C2], [Extent2].[Value] AS[Value] 
            FROM[dbo].[Terms] AS[Extent2] 
            WHERE ([Extent2].[LanguageId] IN (@languageId,1)) 
            AND ([Extent2].[ClientId] = @clientId OR[Extent2].[ClientId] IS NULL) 
            AND (([Project2].[Key] = [Extent2].[Key]) OR(([Project2].[Key] IS NULL) 
            AND ([Extent2].[Key] IS NULL))) 
    ) AS [Project3] 
    ORDER BY [Project3].[C1] DESC, [Project3].[C2] DESC) AS[Limit1]

Would changing the UseDatabaseNullSemantics setting on this query prevent the intermittent crashes happening with this query?

[AppSecSeanner]

We are currently on .Net 4.5.2 and have been informed that there is a known issue that may be contributing to our crashes. Specifically the CLR issue addressed in this hotfix:
https://support.microsoft.com/en-us/kb/3139544#bookmark-issuesresolve
We are going to move to .Net 4.6.1 in the next week or two which will hopefully solve our issue.

[AppSecSeanner]

Upgrading the .Net version from 4.5.2 to 4.6.1 in our production environment solved our intermittent IIS app pool crashing issues.