Memory leak in Ubuntu 16.04

[Idanby]

I am running .net core 2.0 app on Ubuntu 16.04.
I have a small memory leak. The leak growing ~1 percent every one hour.
I was created a dump but I cant find the source of the leak. I see huge amount of Byte[],SQLClient and threads but still i can't find the source even I run the GCRoot on byte[] objects (after DumpHeap on byte[]).
The mostly used components are: EF Core,Quartz,AWS..
When I run the app on windows machine the memory looks good.
Please find below the main part on the dump (three hours up time):
Any help would be greatly appreciated.

0007f3f77d0f730      303        24240 System.Threading.Thread

00007f3f79b86d60      448        25088 Microsoft.Extensions.DependencyInjection.ServiceDescriptor

00007f3f78c2c1e0       33        25200 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Newtonsoft.Json.Linq.JToken, Newtonsoft.Json]][]

00007f3f77d13700      450        25200 System.RuntimeType+RuntimeTypeCache+MemberInfoCache`1[[System.Reflection.RuntimePropertyInfo, System.Private.CoreLib]]

00007f3f7d550d10       32        25568 System.Data.ProviderBase.DbConnectionInternal[]

00007f3f7b96bfb0      272        26400 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[StructureMap.Pipeline.Instance, StructureMap]][]

00007f3f7bd2a780      413        26432 System.Action`1[[StructureMap.Graph.PluginFamily, StructureMap]]

00007f3f77cc2948       26        26808 System.UInt32[]

00007f3f77d24500      849        27168 System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunnerWithContext

00007f3f7d7d6760      173        27680 System.Security.Cryptography.X509Certificates.X509Certificate2

00007f3f77cc69e8      522        28416 System.Reflection.Emit.__ExceptionInfo[]

00007f3f77d138b0      525        29400 System.RuntimeType+RuntimeTypeCache+MemberInfoCache`1[[System.Reflection.RuntimeMethodInfo, System.Private.CoreLib]]

00007f3f7d20d0c0      764        30560 Microsoft.EntityFrameworkCore.Metadata.Internal.PropertyIndexes

00007f3f78da4b88       81        30912 System.Collections.Hashtable+bucket[]

00007f3f77d13940      570        31920 System.RuntimeType+RuntimeTypeCache+MemberInfoCache`1[[System.Reflection.RuntimeConstructorInfo, System.Private.CoreLib]]

00007f3f7d3fc4d8      836        33440 Microsoft.EntityFrameworkCore.Query.Expressions.ColumnExpression

00007f3f7d19ac20      158        35392 Microsoft.EntityFrameworkCore.Metadata.Internal.EntityType

00007f3f7dccbb90      249        35424 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Net.Http.Headers.HttpHeaders+HeaderStoreItemInfo, System.Net.Http]][]

00007f3f7dccb1a0      768        36864 System.Net.Http.Headers.HttpHeaders+HeaderStoreItemInfo

00007f3f77cc2a88       10        37306 System.UInt16[]

00007f3f77d44360     1594        38256 System.Reflection.Emit.DynamicScope

00007f3f7d1a7040     1598        38352 System.Collections.Generic.SortedDictionary`2+KeyValuePairComparer[[System.String, System.Private.CoreLib],[Microsoft.EntityFrameworkCore.Infrastructure.Annotation, Microsoft.EntityFrameworkCore]]

00007f3f7b96c618      616        39424 System.Action`1[[StructureMap.Graph.PluginGraph, StructureMap]]

00007f3f77d1e0a8      518        41440 System.Threading.Tasks.Task+ContingentProperties

00007f3f77d0b3b8      867        41616 System.Text.StringBuilder

00007f3f77cc30f8      710        42192 System.Type[]

00007f3f77358f30      229        43712 System.Reflection.RuntimePropertyInfo[][]

00007f3f7e126f08      616        44352 System.Threading.Tasks.Task`1[[Amazon.SQS.Model.ReceiveMessageResponse, AWSSDK.SQS]]

00007f3f77cc6a48      238        45168 System.Reflection.Emit.__FixupData[]

00007f3f7d2065d0     1438        46016 Microsoft.EntityFrameworkCore.Metadata.Internal.InternalPropertyBuilder

00007f3f78c29dd8     1998        47952 Newtonsoft.Json.Linq.JToken+LineInfoAnnotation

00007f3f7bec1040     1508        48256 System.Runtime.CompilerServices.Closure

00007f3f77d43ec0     1594        51008 System.Reflection.Emit.LineNumberInfo

00007f3f7db731e0     1345        53800 System.Collections.Generic.List`1[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[ThirdParty.Json.LitJson.JsonData, AWSSDK.Core]], System.Private.CoreLib]]

00007f3f7db73610      461        55416 System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[ThirdParty.Json.LitJson.JsonData, AWSSDK.Core]][]

00007f3f77d43fd0      630        60480 System.Reflection.Emit.__ExceptionInfo

00007f3f77c9f510      956        61184 System.Action

00007f3f78c2ad00      978        62592 Newtonsoft.Json.Linq.JValue

00007f3f7d1a1710     1598        63920 System.Collections.Generic.SortedDictionary`2[[System.String, System.Private.CoreLib],[Microsoft.EntityFrameworkCore.Infrastructure.Annotation, Microsoft.EntityFrameworkCore]]

00007f3f7d494e60        2        65584 Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.LibuvThread+Work[]

00007f3f7e7f7de0     2743        65832 System.Collections.Generic.Dictionary`2+ValueCollection[[System.Int32, System.Private.CoreLib],[System.Data.SqlClient.SNI.SNIMarsHandle, System.Data.SqlClient]]

00007f3f7d557368     2763        66312 System.Data.SqlClient.LastIOTimer

00007f3f7d550ff8     2763        66312 System.Data.SqlClient.SqlInternalConnectionTds+SyncAsyncLock

00007f3f77d0d850     1698        67920 System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]]

00007f3f7d1a24d0     2136        68352 Microsoft.EntityFrameworkCore.Internal.LazyRef`1[[System.Collections.Generic.SortedDictionary`2[[System.String, System.Private.CoreLib],[Microsoft.EntityFrameworkCore.Infrastructure.Annotation, Microsoft.EntityFrameworkCore]], System.Collections]]

00007f3f77d4ff58     2180        69760 System.Threading.AsyncLocalValueMap+OneElementAsyncLocalValueMap

00007f3f7d1a6e78     1782        71280 Microsoft.EntityFrameworkCore.Metadata.Internal.ConventionalAnnotation

00007f3f7d848968        1        71680 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Data.SqlClient.SNI.SNIHandle, System.Data.SqlClient],[System.Data.SqlClient.SNI.SNIMarsConnection, System.Data.SqlClient]][]

00007f3f7db73bd0      460        72960 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[ThirdParty.Json.LitJson.JsonData, AWSSDK.Core]][]

00007f3f77d446e8     1594        76512 System.Reflection.Emit.DynamicMethod+RTDynamicMethod

00007f3f77d43e60     1594        76512 System.Reflection.Emit.ScopeTree

00007f3f77d43ab8     1594        76512 System.Reflection.Emit.SignatureHelper

00007f3f7dcf2d30      332        77024 Newtonsoft.Json.Serialization.JsonProperty

00007f3f7d534508     2473        79136 System.Data.SqlClient.SqlConnectionPoolKey

00007f3f7d2099e8     1438        80528 System.Collections.Generic.SortedSet`1+Node[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.EntityFrameworkCore.Metadata.Internal.Property, Microsoft.EntityFrameworkCore]], System.Private.CoreLib]]

00007f3f7b96b948      299        81328 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]][]

00007f3f77cc46e8     2022        81704 System.Reflection.RuntimePropertyInfo[]

00007f3f77d1b720      832        86528 System.Reflection.RuntimeConstructorInfo

00007f3f78c2b168      984        86592 Newtonsoft.Json.Linq.JProperty

00007f3f77d15fa0     2181        87240 System.Threading.ExecutionContext

00007f3f7d678118     2763        88416 System.Net.CallbackClosure

00007f3f77d3eb08     2763        88416 System.Threading.Tasks.SingleProducerSingleConsumerQueue`1[[System.Threading.Tasks.Task, System.Private.CoreLib]]

00007f3f7d485468     2764        88448 System.Net.IPEndPoint

00007f3f7d1a7740     1598        89488 System.Collections.Generic.TreeSet`1[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.EntityFrameworkCore.Infrastructure.Annotation, Microsoft.EntityFrameworkCore]], System.Private.CoreLib]]

00007f3f7d847320     3815        91560 System.Data.SqlClient.SqlCollation

00007f3f77d29668     2983        95456 Microsoft.Win32.SafeHandles.SafeWaitHandle

00007f3f7d537ff8     2473        98920 System.Data.ProviderBase.DbConnectionPoolGroupOptions

00007f3f7d1a8070     1782        99792 System.Collections.Generic.SortedSet`1+Node[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.EntityFrameworkCore.Infrastructure.Annotation, Microsoft.EntityFrameworkCore]], System.Private.CoreLib]]

00007f3f7d538c50     2526       101040 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Data.ProviderBase.DbConnectionPoolIdentity, System.Data.SqlClient],[System.Data.ProviderBase.DbConnectionPool, System.Data.SqlClient]]

00007f3f7d8af0d8     2805       103136 System.Data.SqlClient._SqlMetaData[]

00007f3f7db72af0     1345       107600 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[ThirdParty.Json.LitJson.JsonData, AWSSDK.Core]]

00007f3f7d551380     2735       109400 System.Data.SqlClient.SessionStateRecord

00007f3f7d8499a8     2763       110520 System.Net.Sockets.Socket+CachedTaskEventArgs

00007f3f7d8402f0     2763       110520 System.Collections.Generic.List`1[[System.Data.SqlClient.TdsParserStateObject, System.Data.SqlClient]]

00007f3f77d3d7f8     2764       110560 System.Threading.AutoResetEvent

00007f3f77d1ea98     2764       110560 System.Threading.Tasks.TaskFactory

00007f3f798fbd48     2770       110800 System.Net.IPAddress

00007f3f77d443c0     1594       114768 System.Reflection.Emit.DynamicResolver

00007f3f7d539258     2473       118704 System.Data.SqlClient.SqlConnectionPoolGroupProviderInfo

00007f3f77d44950     3757       120224 System.Reflection.Emit.GenericFieldInfo

00007f3f77d13670      839       127528 System.RuntimeType+RuntimeTypeCache

00007f3f77d10968     5401       129624 System.RuntimeTypeHandle

00007f3f7dac9e38     1633       130640 ThirdParty.Json.LitJson.JsonData

00007f3f7d8ac368     2763       132624 System.Data.SqlClient.SqlReferenceCollection

00007f3f7d8488c0     2763       132624 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Data.SqlClient.SNI.SNIHandle, System.Data.SqlClient],[System.Data.SqlClient.SNI.SNIMarsConnection, System.Data.SqlClient]]

00007f3f7d840260     2763       132624 System.Data.SqlClient.TdsParserSessionPool

00007f3f7d678008     2763       132624 System.Net.Sockets.Socket+CacheSet

00007f3f7d5519e0     2763       132624 System.Data.SqlClient.SqlConnectionTimeoutErrorInternal

00007f3f77d67e10     2763       132624 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Int32, System.Private.CoreLib],[System.Threading.Tasks.ConcurrentExclusiveSchedulerPair+ProcessingMode, System.Private.CoreLib]]

00007f3f77d3efb0     2763       132624 System.Threading.Tasks.MultiProducerMultiConsumerQueue`1[[System.Threading.Tasks.Task, System.Private.CoreLib]]

00007f3f77d3e840     2763       132624 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Int32, System.Private.CoreLib],[System.Threading.Tasks.ConcurrentExclusiveSchedulerPair+ProcessingMode, System.Private.CoreLib]]

00007f3f77cc0168     1707       136640 System.String[]

00007f3f77d350e8     1594       140272 System.RuntimeMethodInfoStub

00007f3f7d5382a8     2526       141456 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Data.ProviderBase.DbConnectionPoolIdentity, System.Data.SqlClient],[System.Data.ProviderBase.DbConnectionPool, System.Data.SqlClient]]

00007f3f77cc4b80      657       144808 System.Reflection.RuntimeMethodInfo[]

00007f3f77d1b910     2033       146376 System.Reflection.RtFieldInfo

00007f3f77d135e0     6204       148896 System.RuntimeMethodHandle

00007f3f77d444f0     1594       153024 System.Reflection.Emit.DynamicMethod

00007f3f7d553a78     2741       153496 System.Data.SqlClient.SNI.SNIError

00007f3f7d8aeac8     2805       157080 System.Data.SqlClient._SqlMetaDataSet

00007f3f7d840898     2795       157976 System.Data.SqlClient.TdsParserStateObject[]

00007f3f7d555f48     2929       164024 System.Data.SqlClient.SNI.SNIPacket

00007f3f7d84a148     2650       169600 System.Action`5[[System.Int32, System.Private.CoreLib],[System.Byte[], System.Private.CoreLib],[System.Int32, System.Private.CoreLib],[System.Net.Sockets.SocketFlags, System.Net.Sockets],[System.Net.Sockets.SocketError, System.Net.Primitives]]

00007f3f7d673a48     2763       176832 System.Net.Sockets.SafeCloseSocket

00007f3f7d534cb0     2473       178056 System.Data.ProviderBase.DbConnectionPoolGroup

00007f3f77d202a8     2787       178368 System.Threading.SemaphoreSlim

00007f3f77d3e7e0     2763       198936 System.Threading.Tasks.ConcurrentExclusiveSchedulerPair

00007f3f77d12000     2666       213280 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.String, System.Private.CoreLib]]

00007f3f7d849190     2763       221040 System.Collections.Generic.Dictionary`2[[System.Int32, System.Private.CoreLib],[System.Data.SqlClient.SNI.SNIMarsHandle, System.Data.SqlClient]]

00007f3f7d307708     2763       221040 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Tuple`2[[System.String, System.Private.CoreLib],[System.String, System.Private.CoreLib]], System.Private.CoreLib]]

00007f3f77d44180     1594       242288 System.Reflection.Emit.DynamicILGenerator

00007f3f7d8480a8     2763       243144 System.Data.SqlClient.SNI.SNIMarsConnection

00007f3f77cc4d10     8408       252208 System.Reflection.ParameterInfo[]

00007f3f77d143c8     2457       255528 System.Reflection.RuntimePropertyInfo

00007f3f7d55b050     2763       265248 System.Net.Sockets.Socket

00007f3f7d551b68     2763       265248 System.Data.SqlClient.SqlConnectionTimeoutPhaseDuration[]

00007f3f7d551288     2763       265248 System.Data.SqlClient.SessionData

00007f3f7d308170     2763       265248 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Tuple`2[[System.String, System.Private.CoreLib],[System.String, System.Private.CoreLib]], System.Private.CoreLib]][]

00007f3f77d50730     5526       265248 System.Threading.Tasks.ConcurrentExclusiveSchedulerPair+ConcurrentExclusiveTaskScheduler

00007f3f7d84a5d8     5538       265824 System.Collections.Generic.Queue`1[[System.Data.SqlClient.SNI.SNIMarsQueuedPacket, System.Data.SqlClient]]

00007f3f7d84a380     5538       265824 System.Collections.Generic.Queue`1[[System.Data.SqlClient.SNI.SNIPacket, System.Data.SqlClient]]

00007f3f7d556b20     8313       266016 System.Data.SqlClient.SNI.TdsParserStateObjectManaged+WritePacketCache

00007f3f7d84ab38     8325       266400 System.Data.SqlClient.SNI.SNISMUXHeader

00007f3f7d84ace8     2763       266592 System.Collections.Generic.Dictionary`2+Entry[[System.Int32, System.Private.CoreLib],[System.Data.SqlClient.SNI.SNIMarsHandle, System.Data.SqlClient]][]

00007f3f77d0e400     6853       274120 System.RuntimeType

00007f3f7d676570     2763       287352 System.Net.Sockets.SocketAsyncContext

00007f3f7d556f38     5522       309200 System.Data.SqlClient.SNI.SNIPacket[]

00007f3f7d5572b8     8313       332520 System.Data.SqlClient.SNI.SspiClientContextStatus

00007f3f7d556bd0     8313       332520 System.Collections.Generic.Stack`1[[System.Data.SqlClient.SNI.SNIPacket, System.Data.SqlClient]]

00007f3f77cc20f8    10619       341864 System.RuntimeType[]

00007f3f7d200b78     1438       345120 Microsoft.EntityFrameworkCore.Metadata.Internal.Property

00007f3f7d67bd48     5533       354112 System.Data.SqlClient.SNI.SNIAsyncCallback

00007f3f7d558118     2763       375768 System.Data.SqlClient.SNI.SNITCPHandle

00007f3f7d552f80     2763       397872 System.Data.SqlClient.TdsParser

00007f3f7d5379a8     2473       415464 System.Data.SqlClient.SqlConnectionString

00007f3f77d1a1d0     2728       436480 System.IO.IOException

00007f3f77cc3440      983       439804 System.Char[]

00007f3f77d1df58    11076       443040 System.Threading.ManualResetEventSlim

00007f3f79a0e038     2728       458304 System.Net.Sockets.SocketException

00007f3f7d551ad8    19341       464184 System.Data.SqlClient.SqlConnectionTimeoutPhaseDuration

00007f3f77d25470    22299       535176 System.WeakReference

00007f3f77359fd0     2614       562896 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.String, System.Private.CoreLib]][]

00007f3f77d1ad60     6231       598176 System.Reflection.RuntimeParameterInfo

00007f3f7d308800     2763       641016 System.Collections.Concurrent.ConcurrentQueue`1+Segment[[System.Threading.Tasks.Task, System.Private.CoreLib]]

00007f3f7d556078     8313       665040 System.Collections.Generic.Dictionary`2[[System.Data.SqlClient.SNI.SNIPacket, System.Data.SqlClient],[System.Data.SqlClient.SNI.SNIPacket, System.Data.SqlClient]]

00007f3f7d538ad8     2526       687072 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Data.ProviderBase.DbConnectionPoolIdentity, System.Data.SqlClient],[System.Data.ProviderBase.DbConnectionPool, System.Data.SqlClient]][]

00007f3f7d53f5c0     2763       707328 System.Data.SqlClient.SqlInternalConnectionTds

00007f3f7d849098     5538       708864 System.Data.SqlClient.SNI.SNIMarsHandle

00007f3f77d1ace0     8936       714880 System.Signature

00007f3f7d671d78     2651       721072 System.Net.Sockets.Socket+Int32TaskSocketAsyncEventArgs

00007f3f77cf7e98     2763       751536 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Int32, System.Private.CoreLib],[System.Threading.Tasks.ConcurrentExclusiveSchedulerPair+ProcessingMode, System.Private.CoreLib]][]

00007f3f77ccb428     2765       773720 System.Threading.Tasks.Task[]

00007f3f7d539ab0    19454       778160 System.Diagnostics.Stopwatch

00007f3f7d8af068     4477       823768 System.Data.SqlClient._SqlMetaData

00007f3f7d537c88    17363       833424 System.Data.Common.NameValuePair

00007f3f7d8ae630     2763       950472 System.Data.ProviderBase.DbReferenceCollection+CollectionEntry[]

00007f3f77cc08a8    22211      1036504 System.Int32[]

00007f3f77cbefd8     9102      1043760 System.Object[]

00007f3f7d308750     2763      1171512 System.Threading.Tasks.SingleProducerSingleConsumerQueue`1+Segment[[System.Threading.Tasks.Task, System.Private.CoreLib]]

00007f3f77d13b80    11725      1219400 System.Reflection.RuntimeMethodInfo

00007f3f7d3089c8     2763      1480968 System.Collections.Concurrent.ConcurrentQueue`1+Segment+Slot[[System.Threading.Tasks.Task, System.Private.CoreLib]][]

00007f3f77d0b890    68168      1636032 System.Object

00007f3f7d555bb0     8313      3391704 System.Data.SqlClient.SNI.TdsParserStateObjectManaged

00007f3f77d0b460    68673      5385818 System.String

00007f3f77cd3890     2784      5727816 System.Byte[][]

00007f3f7d551410     2777      5753944 System.Data.SqlClient.SessionStateRecord[]

000000000239fd20    69262     41368920      Free

00007f3f77cc1210    76267    161547850 System.Byte[]

[Idanby]

apparently, MARS in SqlClient caused the memory leak. In my connection string MultipleActiveResultSets was set to true.I still don't know why this is affected just in Ububntu and not Windows too but when I set it to false, memory back to be stabled.

[tiaoyu]

in dotnet-sdk-2.1.4 or 2.1.200
I meet the problem too，Memory Leak in Centos7.4，but it's very super stable in Windows!（T . T）
and it looks good use the Mono to run .exe

Could you please tell me how to get the dump like upstairs ? thanku!!!(๑•ᴗ•๑)

[Idanby]

How your connection string looks like? With MultipleActiveResultSets = True also?

[tiaoyu]

just like this, no MultipleActiveResultSets
return string.Format("server = {0}; port = {1}; user id = {2}; password = {3}; database = {4}; charset=utf8",Database.Ip, Database.Port, Database.User, Database.Pwd, Database.Name);

[Idanby]

OK, so no MultipleActiveResultSets enabled.
Do you want to create a dump file to investigate?

[tiaoyu]

yes, do you know how to get the dump file in centos? Any help would be greatly appreciated.

[Idanby]

Yes.
Can you connect to server via ssh right?
If so, you need to install lldb and execute few commands to get the dump.
Pay attention that while you connected to lldb your service will be freeze.
http://blogs.microsoft.co.il/sasha/2017/02/26/analyzing-a-net-core-core-dump-on-linux/)
https://blogs.msdn.microsoft.com/wushuai/2017/03/13/debug-asp-net-core-via-lldb-on-ubuntu/

ps -ejH //Find you process id
sudo lldb-3.6 open lldb
process attach -p xxx(your processId)
plugin load /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.5/libsosplugin.so
setclrpath /usr/share/dotnet/shared/Microsoft.NETCore.App/2.0.5

Now type any SOS command,

[tiaoyu]

（T . T）i donot know how to descript my gratitude ...it's very kind of you.
I'll try right now. thank u!!!!!!!!o(∩_∩)o

[Idanby]

My pleasure.