Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak in HttpWebRequest / HttpWebResponse when using a WebProxy #8689

Open
smr888 opened this issue May 12, 2018 · 35 comments
Assignees

Comments

@smr888
Copy link

@smr888 smr888 commented May 12, 2018

This is a follow-on of sorts to bug #7356 . The test is essentially the same thing except through a WebProxy.

Steps to Reproduce

  1. Use Ubuntu 16.04. Install mono-devel, mono-utils. mono-profiler, whatever is necessary to have the compiler and profiler. I doubt this is specific to the Linux version, however. You will also need a proxy server through which to make the web requests. Replace yourproxyip and yourproxyport withthe proper values for your proxy server.
  2. The program below simply loops over a set of popular websites and gets their homepages through a proxy server, using HttpWebRequest and HttpWebResponse. Compile and run it under the profiler, e.g.
mono --gc=sgen --profile=log:heapshot=50gc TestMemGrow.exe
  1. Let it run for several hours (3-4, or more). After that, to view the profiler output enter:
mprof-report output.mlpd > out.txt
grep System.Byte out.txt
(grep -F "System.String" | grep -Fv "System.String[]") < out.txt

You should see lists of allocations of System.Byte[] and System.String that grow increasingly larger. You can leave the program running and re-execute those commands, to see the continued growth of memory usage as the profiler adds more heap shots to its data file. In my testing the heapshots came about 10 minutes apart.

I left in the GC calls from my prior bug, and this example writes the web page data to a file. It also writes exceptions to a file exceptions.txt, because it is possible (I really don't know) that it is the exception handling leaking memory, so that information might be useful.

To execute this code you will need a proxy server through which you make the web requests. Fill in your proxy server's IP and port in the appropriate place in the code.

CODE:

using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Runtime;

namespace TestMemGrow
{
    class Program
    {
        static void Main()
        {
		List<string> websites = new List<string>() { "http://www.amazon.com", "http://www.google.com", "http://www.yahoo.com", "http://www.ebay.com", "http://www.overstock.com",
                "http://www.marketwatch.com", "http://www.bn.com", "http://www.newegg.com", "http://www.wsj.com", "http://www.arstechnica.com", "http://www.slashdot.org",
                "http://www.mediaite.com", "http://www.disqus.com", "http://www.twitter.com", "http://www.snap.com", "http://www.facebook.com", "http://www.usps.com",
                "http://www.ups.com", "http://www.techcrunch.com", "http://www.oracle.com", "http://www.java.com", "http://www.apple.com", "http://www.microsoft.com",
                "http://www.ibm.com", "http://www.dell.com", "http://www.asus.com", "http://www.gigabyte.com", "http://www.intel.com", "http://www.crucial.com",
                "http://www.westerndigital.com", "http://www.samsung.com", "http://www.sandisk.com", "http://www.brother.com", "http://www.hp.com",
                "http://www.msn.com", "http://www.disney.com", "http://www.nintendo.com", "http://www.twitter.com", "http://www.youtube.com",
                "http://www.instagram.com", "http://www.linkedin.com", "http://www.wordpress.org", "http://www.pinterest.com", "http://www.wikipedia.org",
                "http://www.blogspot.com", "http://www.adobe.com", "http://www.tumblr.com", "http://www.vimeo.com", "http://www.flickr.com", "http://www.godaddy.com",
                "http://www.buydomains.com", "http://www.reddit.com", "http://www.w3.org","http://www.nytimes.com", "http://www.statcounter.com",
                "http://www.weebly.com","http://www.blogger.com","http://www.github.com", "http://www.jimdo.com", "http://www.myspace.com",
                "http://www.mozilla.org", "http://www.gravatar.com", "http://www.theguardian.com", "http://www.bluehost.com", "http://www.cnn.com", "http://www.foxnews.com",
                "http://www.msnbc.com", "http://www.wix.com", "http://www.paypal.com","http://www.stumbleupon.com", "http://www.digg.com","http://www.huffingtonpost.com",
                "http://www.feedburner.com", "http://www.imdb.com","http://www.yelp.com","http://www.dropbox.com", "http://www.baidu.com","http://www.washingtonpost.com",
                "http://www.slideshare.net","http://www.etsy.com","http://www.telegraph.co.uk", "http://www.about.com", "http://www.bing.com", "http://www.latimes.com",
                "http://www.tripadvisor.com","http://www.opera.com", "http://www.live.com", "http://www.wired.com", "http://www.bandcamp.com"};
			StreamWriter sw = new StreamWriter("exceptions.txt");

            while (true)
            {
                foreach (string website in websites)
                {
                    try
                    {
                        GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
                        GC.Collect();
			Console.WriteLine(DateTime.Now.ToString("HH:mm:ss") + " Getting " + website);
                        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(website);
			WebProxy myProxy = new WebProxy("yourproxyip", yourproxyport);
			request.Proxy = myProxy;
                        HttpWebResponse response = request.GetResponse() as HttpWebResponse;
                        StreamReader reader = new StreamReader(response.GetResponseStream());
                        File.WriteAllText("florb.txt", reader.ReadToEnd());
                        reader.Close();
                        response.Close();
                    }
                    catch (Exception ex)
                    {
			string exString = ex.ToString();
			exString = exString.Substring(0, Math.Min(150, exString.Length));
                        Console.WriteLine("Caught exception . . . " + exString);
			sw.WriteLine("EXCEPTION\r\n" + ex.ToString() + "\r\n");
			sw.Flush();
                    }
                }
            }
        }
    }
}

Current Behavior

Memory allocations of System.Byte[] and System.String increase apparently without limit.

Expected Behavior

The garbage collector would clean up these allocations.

On which platforms did you notice this

[ ] macOS
[X] Linux
[ ] Windows

Version Used:

Mono JIT compiler version 5.12.0.226 (tarball Thu May  3 09:48:32 UTC 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	Interpreter:   yes
	LLVM:          supported, not enabled.
	GC:            sgen (concurrent by default)

Other Matters

I am not at all certain whether this leak is caused by the use of a WebProxy itself, or by the increased number of exceptions, particularly WebExceptions, that occur when using a WebProxy to make the requests.

@smr888

This comment has been minimized.

Copy link
Author

@smr888 smr888 commented May 15, 2018

I had oversimplified the code I originally submitted and left out important closings of new objects. I retested with the code below, which fixes those problems. The memory leak still shows up. Please use the code below for testing.

CODE:

using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Runtime;

namespace TestMemGrow
{
    class Program
    {
        static void Main()
        {
            List<string> websites = new List<string>() { "http://www.amazon.com", "http://www.google.com", "http://www.yahoo.com", "http://www.ebay.com", "http://www.overstock.com",
                "http://www.marketwatch.com", "http://www.bn.com", "http://www.newegg.com", "http://www.wsj.com", "http://www.arstechnica.com", "http://www.slashdot.org",
                "http://www.mediaite.com", "http://www.disqus.com", "http://www.twitter.com", "http://www.snap.com", "http://www.facebook.com", "http://www.usps.com",
                "http://www.ups.com", "http://www.techcrunch.com", "http://www.oracle.com", "http://www.java.com", "http://www.apple.com", "http://www.microsoft.com",
                "http://www.ibm.com", "http://www.dell.com", "http://www.asus.com", "http://www.gigabyte.com", "http://www.intel.com", "http://www.crucial.com",
                "http://www.westerndigital.com", "http://www.samsung.com", "http://www.sandisk.com", "http://www.brother.com", "http://www.hp.com",
                "http://www.msn.com", "http://www.disney.com", "http://www.nintendo.com", "http://www.twitter.com", "http://www.youtube.com",
                "http://www.instagram.com", "http://www.linkedin.com", "http://www.wordpress.org", "http://www.pinterest.com", "http://www.wikipedia.org",
                "http://www.blogspot.com", "http://www.adobe.com", "http://www.tumblr.com", "http://www.vimeo.com", "http://www.flickr.com", "http://www.godaddy.com",
                "http://www.buydomains.com", "http://www.reddit.com", "http://www.w3.org","http://www.nytimes.com", "http://www.statcounter.com",
                "http://www.weebly.com","http://www.blogger.com","http://www.github.com", "http://www.jimdo.com", "http://www.myspace.com",
                "http://www.mozilla.org", "http://www.gravatar.com", "http://www.theguardian.com", "http://www.bluehost.com", "http://www.cnn.com", "http://www.foxnews.com",
                "http://www.msnbc.com", "http://www.wix.com", "http://www.paypal.com","http://www.stumbleupon.com", "http://www.digg.com","http://www.huffingtonpost.com",
                "http://www.feedburner.com", "http://www.imdb.com","http://www.yelp.com","http://www.dropbox.com", "http://www.baidu.com","http://www.washingtonpost.com",
                "http://www.slideshare.net","http://www.etsy.com","http://www.telegraph.co.uk", "http://www.about.com", "http://www.bing.com", "http://www.latimes.com",
                "http://www.tripadvisor.com","http://www.opera.com", "http://www.live.com", "http://www.wired.com", "http://www.bandcamp.com"};
            StreamWriter sw = new StreamWriter("exceptions.txt");

            while (true)
            {
                foreach (string website in websites)
                {
                    HttpWebResponse response = null;
                    StreamReader reader = null;
                    try
                    {
                        GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
                        GC.Collect();
                        Console.WriteLine(DateTime.Now.ToString("HH:mm:ss") + " Getting " + website);
                        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(website);
                        WebProxy myProxy = new WebProxy("yourproxyip", yourproxyport);
                        request.Proxy = myProxy;
                        response = request.GetResponse() as HttpWebResponse;
                        reader = new StreamReader(response.GetResponseStream());
                        File.WriteAllText("florb.txt", reader.ReadToEnd());
                        reader.Close();
                        reader = null;
                        response.Close();
                        response = null;
                    }
                    catch (Exception ex)
                    {
                        string exString = ex.ToString();
                        exString = exString.Substring(0, Math.Min(150, exString.Length));
                        Console.WriteLine("Caught exception . . . " + exString);
                        sw.WriteLine("EXCEPTION\r\n" + ex.ToString() + "\r\n");
                        sw.Flush();
                    }
                    finally
                    {
                        if (reader != null) reader.Close();
                        if (response != null) response.Close();
                    }
                    //Thread.Sleep(1000);
                }
            }
        }
    }
}
@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Jul 25, 2018

It seems the more exceptions there are the more memory is consumed. We removed several of the URLs that were causing exceptions and noticed the memory pressures were not as severe.

@smr888

This comment has been minimized.

Copy link
Author

@smr888 smr888 commented Jul 25, 2018

Interesting. I tried creating examples that just threw exceptions, but they did not leak. Nonetheless I have had the sense that the problem might have something to do with the exceptions, so I guess it's specific to the WebProxy.

@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Jul 27, 2018

We modified the test to avoid the proxy and it is showing that behaviour. I tried using the --trace option on the mprof-report in an attempt to see if I could get an idea of what was responsible for all the System.Byte[] objects but nothing stands out. I would resort to Environment.Stacktrace in AllocVector but it's a ves_xxxx type call.

@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Jul 27, 2018

The MONO_GC_DEBUG heap-dump reveals that the Byte[] objects being created are either 16400 or 16850 bytes in size - and their are thousands and thousands of them. @baulig Do these sizes mean anything to you?

@smr888

This comment has been minimized.

Copy link
Author

@smr888 smr888 commented Jul 29, 2018

It would be great to see that fixed. When our last memory leak bug was fixed we were able to go from restarting every two days to every four . . . but I would vastly prefer to return to what I didn't know at the time were the golden days of yesteryear, where we didn't have to restart at all.

@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Jul 30, 2018

Matching the --trace output and the size of the objects I see:

[0xb33feb40: 20.59841 24] ENTER: Mono.Net.Security.BufferOffsetSize2:.ctor (int)(this:0xb6c06170[Mono.Net.Security.BufferOffsetSize2 8689-ec.exe], 16834, )
[0xb33feb40: 20.59843 25] ENTER: (wrapper alloc) object:AllocVector (intptr,intptr)(0x918bff8, 0x41c2, )

for the 16850 byte allocations (16834 + 16), and,

[0xb33feb40: 20.59862 24] ENTER: Mono.Net.Security.BufferOffsetSize2:.ctor (int)(this:0xb6c06190[Mono.Net.Security.BufferOffsetSize2 8689-ec.exe], 16384, )
[0xb33feb40: 20.59864 25] ENTER: (wrapper alloc) object:AllocVector (intptr,intptr)(0x918bff8, 0x4000, )

for the 16400 (16384 + 16) allocation.

These appear to result from:

[0xb33feb40: 20.59758 16] ENTER: Mono.Btls.MonoBtlsProvider:CreateSslStream (System.IO.Stream,bool,Mono.Security.Interface.MonoTlsSettings)(this:0xb72b8638[Mono.Btls.MonoBtlsProvider 8689-ec.exe], [System.Net.Sockets.NetworkStream:0xb6c05f60], 0, [Mono.Security.Interface.MonoTlsSettings:0xb6c05fe0], )
[0xb33feb40: 20.59763 17] ENTER: System.Net.Security.SslStream:CreateMonoSslStream (System.IO.Stream,bool,Mono.Security.Interface.MonoTlsProvider,Mono.Security.Interface.MonoTlsSettings)([System.Net.Sockets.NetworkStream:0xb6c05f60], 0, [Mono.Btls.MonoBtlsProvider:0xb72b8638], [Mono.Security.Interface.MonoTlsSettings:0xb6c05fe0], )
[0xb33feb40: 20.59766 18] ENTER: (wrapper managed-to-native) object:__icall_wrapper_ves_icall_object_new_specific (intptr)(0xb38152d0, )
[0xb33feb40: 20.59768 18] LEAVE: (wrapper managed-to-native) object:__icall_wrapper_ves_icall_object_new_specific (intptr)[System.Net.Security.SslStream:0xb6c06088]
[0xb33feb40: 20.59772 18] ENTER: (wrapper remoting-invoke-with-check) System.Net.Security.SslStream:.ctor (System.IO.Stream,bool,Mono.Security.Interface.MonoTlsProvider,Mono.Security.Interface.MonoTlsSettings)(this:0xb6c06088[System.Net.Security.SslStream 8689-ec.exe], [System.Net.Sockets.NetworkStream:0xb6c05f60], 0, [Mono.Btls.MonoBtlsProvider:0xb72b8638], [Mono.Security.Interface.MonoTlsSettings:0xb6c05fe0], )
[0xb33feb40: 20.59776 19] ENTER: System.Net.Security.SslStream:.ctor (System.IO.Stream,bool,Mono.Security.Interface.MonoTlsProvider,Mono.Security.Interface.MonoTlsSettings)(this:0xb6c06088[System.Net.Security.SslStream 8689-ec.exe], [System.Net.Sockets.NetworkStream:0xb6c05f60], 0, [Mono.Btls.MonoBtlsProvider:0xb72b8638], [Mono.Security.Interface.MonoTlsSettings:0xb6c05fe0], )
:
:
:
[0xb33feb40: 20.59802 20] ENTER: Mono.Btls.MonoBtlsProvider:CreateSslStreamInternal (System.Net.Security.SslStream,System.IO.Stream,bool,Mono.Security.Interface.MonoTlsSettings)(this:0xb72b8638[Mono.Btls.MonoBtlsProvider 8689-ec.exe], [System.Net.Security.SslStream:0xb6c06088], [System.Net.Sockets.NetworkStream:0xb6c05f60], 0, [Mono.Security.Interface.MonoTlsSettings:0xb6c05fe0], )
[0xb33feb40: 20.59805 21] ENTER: (wrapper managed-to-native) object:__icall_wrapper_ves_icall_object_new_specific (intptr)(0xb3815658, )
[0xb33feb40: 20.59807 21] LEAVE: (wrapper managed-to-native) object:__icall_wrapper_ves_icall_object_new_specific (intptr)[Mono.Btls.MonoBtlsStream:0xb6c06108]
[0xb33feb40: 20.59812 21] ENTER: (wrapper remoting-invoke-with-check) Mono.Btls.MonoBtlsStream:.ctor (System.IO.Stream,bool,System.Net.Security.SslStream,Mono.Security.Interface.MonoTlsSettings,Mono.Security.Interface.MonoTlsProvider)(this:0xb6c06108[Mono.Btls.MonoBtlsStream 8689-ec.exe], [System.Net.Sockets.NetworkStream:0xb6c05f60], 0, [System.Net.Security.SslStream:0xb6c06088], [Mono.Security.Interface.MonoTlsSettings:0xb6c05fe0], [Mono.Btls.MonoBtlsProvider:0xb72b8638], )
[0xb33feb40: 20.59815 22] ENTER: Mono.Btls.MonoBtlsStream:.ctor (System.IO.Stream,bool,System.Net.Security.SslStream,Mono.Security.Interface.MonoTlsSettings,Mono.Security.Interface.MonoTlsProvider)(this:0xb6c06108[Mono.Btls.MonoBtlsStream 8689-ec.exe], [System.Net.Sockets.NetworkStream:0xb6c05f60], 0, [System.Net.Security.SslStream:0xb6c06088], [Mono.Security.Interface.MonoTlsSettings:0xb6c05fe0], [Mono.Btls.MonoBtlsProvider:0xb72b8638], )

[0xb33feb40: 20.59818 23] ENTER: Mono.Net.Security.MobileAuthenticatedStream:.ctor (System.IO.Stream,bool,System.Net.Security.SslStream,Mono.Security.Interface.MonoTlsSettings,Mono.Security.Interface.MonoTlsProvider)(this:0xb6c06108[Mono.Btls.MonoBtlsStream 8689-ec.exe], [System.Net.Sockets.NetworkStream:0xb6c05f60], 0, [System.Net.Security.SslStream:0xb6c06088], [Mono.Security.Interface.MonoTlsSettings:0xb6c05fe0], [Mono.Btls.MonoBtlsProvider:0xb72b8638], )

Which now prompts the question why are these objects still hanging around? These Byte[] objects correspond to readBuffer (16834 bytes) and writeBuffer (16384 bytes) in MobileAuthenticatedStream.cs. (Also, is 16834 a typo in the source code and it should also be 16384?)

@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Jul 30, 2018

The attached graph shows memory availability for runs where (orange) there are URLs that result in exceptions and (blue) where there are no such URLs. It appears that when the connect fails we are not disposing or ending gracefully.

8689mem

@smr888

This comment has been minimized.

Copy link
Author

@smr888 smr888 commented Jul 30, 2018

Heh, "16834" sort of sounds like a typo in the code for 16384. Or it's an interesting coincidence. (And I now see that you said that also.)

@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Jul 31, 2018

The profiler shows things like:

Heap shot 61 at 1352.664 secs: size: 135159264, object count: 833082, class count: 446, roots: 653860
             Bytes      Count  Average Class name
          98540192      12920     7626 System.Byte[] (bytes: +1538608, count: +197)
                670 root references (0 pinning)
                5074 references from: Mono.Net.Security.BufferOffsetSize2
                2669 references from: System.Net.WebRequestStream
                2542 references from: System.IO.FileStream
                2541 references from: System.Net.BufferOffsetSize
                73 references from: System.Net.ServicePoint
                3 references from: System.IO.CStreamWriter
                3 references from: Mono.Globalization.Unicode.SimpleCollator
                2 references from: System.IO.CStreamReader
                2 references from: System.IO.StreamWriter
                1 references from: System.TermInfoReader
           6576896      80239       81 System.String (bytes: +92472, count: +1133)
                139066 root references (0 pinning)
                27695 references from: System.Object[]
                26783 references from: System.Collections.Hashtable.bucket[]
                26714 references from: System.Collections.Specialized.NameObjectCollectionBase.NameObjectEntry
                8114 references from: System.Uri.UriInfo
                7947 references from: System.Uri
                7614 references from: System.Net.HttpWebRequest
                5074 references from: Mono.Btls.MonoBtlsContext
                5074 references from: Mono.Btls.MonoBtlsException
@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Jul 31, 2018

@migueldeicaza Who would be the best to discuss the next steps to identify this problem? I think I am on the right track but the methods involved look rather complex: compilation of the async tasks and a connection finite state machine appear to be part of the mix.

@marek-safar

This comment has been minimized.

Copy link
Member

@marek-safar marek-safar commented Aug 1, 2018

@baulig could you have a look. This is potentially related to the async fix we did for 2018-02.

@smr888 could you test it with newer versions if there is any difference?

@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Aug 2, 2018

@marek-safar These latest tests were done with 2018-06 as of 14 days ago plus a couple of local fixes (not related to the problem).

@smr888

This comment has been minimized.

Copy link
Author

@smr888 smr888 commented Aug 2, 2018

@marek-safar I have re-run my test with the 5.12.301 release, and I'm (somewhat) sorry to have to report that it no longer seems to show the memory leak I had found earlier.

So I rebuilt our software, updated the Monos on our servers, and reissued our stuff so that it is now all running on 5.12.301, and as a whole I find after 17 hours that memory usage on our servers is still growing over time. If anything, the growth rate appears to be slightly worse than what I was seeing in terms of overall server memory usage growth when I reported the bug in May. . . .

which is somewhat frustrating, because developing a test case that shows the memory leak has been the largest part of this problem for me. On the other hand, if something has been fixed, that's good.

I'll have to suggest that you follow Neale's line of thought and testing from here on, as he seems to be onto something that my test program is not, or is no longer, catching. Neale, please make sure you are using 5.12.301 or greater when testing, and thanks so much for your efforts.

@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Aug 2, 2018

An important fact I left out is that when we run the test program the root certificates are not installed so that we can force an exception in the webrequest. We added some code to collect memory stats and run 1000 iterations:

using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Runtime;
using System.Runtime.InteropServices;

namespace TestMemGrow
{
    [StructLayout(LayoutKind.Sequential)]
    public struct MEMORYSTATUS
    {
        public ulong dwLength;
        public ulong dwMemoryLoad;
        public ulong dwTotalPhys;
        public ulong dwAvailPhys;
        public ulong dwTotalPageFile;
        public ulong dwAvailPageFile;
        public ulong dwTotalVirtual;
        public ulong dwAvailVirtual;
    }

    class Program
    {
        private static MEMORYSTATUS GetMemoryStatusLinux()
        {
            MEMORYSTATUS retVal = new MEMORYSTATUS();
            try
            {
                using (TextReader reader = File.OpenText("/proc/meminfo"))
                {
                    ulong virtualUsed = 0;

                    string line;
                    while ((line = reader.ReadLine()) != null)
                    {
                        if (line != null)
                        {
                            string key = "";
                            ulong val = 0;
                            string[] items = System.Text.RegularExpressions.Regex.Split(line, @"\s+");
                            try
                            {
                                key = items[0].ToLower();
                                val = ulong.Parse(items[1]);

                                if (items.Length > 2)
                                {
                                    if (items[2] == "kB")
                                        val *= 1024;
                                    else if (items[2] == "mB")
                                        val *= 1024 * 1024;
                                    else
                                        Console.WriteLine("Unknown memory size multiplier {0}", items[2]);
                                }
                            }
                            catch (FormatException e)
                            {
                                Console.WriteLine("Couldn't parse: {0}, message:{1}", line, e.Message);
                            }
                            catch (IndexOutOfRangeException e)
                            {
                                Console.WriteLine("Couldn't parse: {0}, message:{1}", line, e.Message);
                            }

                            switch (key)
                            {
                                case "memtotal:":
                                    retVal.dwTotalPhys = val;
                                    break;
                                case "memavailable:":
                                    retVal.dwAvailPhys = val;
                                    break;
                                case "swaptotal:":
                                    retVal.dwTotalPageFile = val;
                                    break;
                                case "swapfree:":
                                    retVal.dwAvailPageFile = val;
                                    break;
                                case "vmalloctotal:":
                                    retVal.dwTotalVirtual = val;
                                    break;
                                case "vmallocused:":
                                    virtualUsed = val;
                                    break;
                                default:
                                    break;
                            }
                        }
                    }

                    retVal.dwAvailVirtual = (retVal.dwTotalVirtual > virtualUsed) ? retVal.dwTotalVirtual - virtualUsed : 0;

                    if (retVal.dwTotalPhys > 0)
                        retVal.dwMemoryLoad = ((100 * (ulong)(retVal.dwTotalPhys - retVal.dwAvailPhys)) / retVal.dwTotalPhys);
                }
            }
            catch (System.IO.IOException e)
            {
                Console.WriteLine(e.Message);
            }
            return retVal;
        }

        static void Main()
        {
            string[] websites = {
                              @"http://search.spotxchange.com/exception?_a=224962&_p=345b6.1ef83.6c3d&_z=1&_m=eNpFjUtvgzAQhP1bOBey6%2FhFpJ6aqlKVOKpEesgFGdsBRIEoBCV9%2FfbWIZW6K%2B3hm9kZlgAQQpDPQUqaIicKnTDSFArQI3qluJDoHVjDEAAkoZSlgpIHTQQklMlEQYKMkup0Oixms7Fruv7czchnNA7%2BmJdj7aJFJJyVHr0FWZhrbApc4d6ZvWT8Ghvd3ez1kHf%2BHB4CsH3f1D53fWvqLqDBm6OtkuHQny62Ml3pE9u3wWhGV%2FvO%2BnyqQiHkvAAXS14U8bUsNkZgjDRQ4eZT3TehyBhn5Gcaorer1e3ghAimFC4ICgL4F%2F8cYb80fW3WH7rVy5fz%2Bum52WQl2y2375vsrdXZI%2B5aXa0zC3pZ3v8C%2BXJpPg%3D%3D&_l=eNptjttKw0AQhvdZ9lrDTvZALXhRSpBKN15Yqc1N2Gy22iZuQk2w3dp3dzQIFTsnmP%2F%2FBgYIEIkNBIsLWagI3HrEI2V5%2BV%2BYpoSR0pka6SPldHykxaakY8oixpikV7Q1h6bvfhVxg9LaOdwhYsn1N%2FHeOv%2F3pC%2Fqjc3d3r4a%2F%2BLygUdbMYa27Xemc2X%2BZnaV69raWHeBxaCnE0mf5vNh4IePbdM9DyDpfeWbDz94Sip5hv6MGISQ4kxgmKAE56PPbKEhu7vfPiwmQsezsAoV6DirdZiFdJlAuk0OepnsdZjcfgHf5Vrb&_t=eNpVUV1vGjEQvJ%2BC%2FIw4GwxckfpwaQqlzRUUCC30KsvYS87ovno2FJL2v3d9iSLVDzO7M15714aLgtqZqgyiPuszCjqiDBiDKBqOxgw0VZIzSuk4UJk0ZVA1j7I0KoC3wmeyB6mqUrhrDWRC3hzSJXUur84UXmZDzM9GQyXOMjdaHExjnTg0srXRlFrYq3VQkMkzadzeEx7gDlVTCDSxJpM2w810xN%2BxPlcjynXEh1TTw0jygQTJOfBo%2F9%2FNrEusyqC9ZROv1n10DwBanJocpcy5epKGaah02bN15S4%2BUFWRhr9hb42DNDSlg8dG%2BqGEA%2BvSsABtZBr2KRuLXywNN36wNFxh%2Ffd9fgI2FGVVWpv3LkXuB68Nmfwgm2U8v8W0ZfF5hWGy5IjTuw3it483CVIcf2illpLlwCOPERczj%2FeLLw9I6w3W%2F%2BwSlRsonbDYAZnQv12ipZM4WNvLGrvtfKpOFjqx7txgZwR3WJCNyoStQbWPLP0nOGiszzScjQIfwcV5ah9FvMhC1vXrR5enPMezikpD%2FpJ1SWVfdVxBwHqUBl8f7u4CNuKDQfQnOU6z3Wxqtk8J3xXb63Z9XyRP88v2uL3uimS4uFU0OSq%2BmM3f%2FwOBe9XO&_b=eNozZfAL9fFBI2r8QzxNolwiK%2FxDkkF0VZS7W5ZfladJZJavkV94VK6%2Fi6exX4hTpr97qC0AcfcToA%3D%3D&beacon_type=exception&exception%5Bid%5D=%24EXCEPTION_ID&exception%5Bdata%5D=%24EXCEPTION_DATA&exception%5Btrace%5D=%24EXCEPTION_TRACE&syn%5Btiming%5D=%24TIMING_DATA",
                              @"http://search.spotxchange.com/beacon?_a=224962&_p=345b6.1ef83.6c3d&_z=1&_m=eNpFjUtvgzAQhP1bOBey6%2FhFpJ6aqlKVOKpEesgFGdsBRIEoBCV9%2FfbWIZW6K%2B3hm9kZlgAQQpDPQUqaIicKnTDSFArQI3qluJDoHVjDEAAkoZSlgpIHTQQklMlEQYKMkup0Oixms7Fruv7czchnNA7%2BmJdj7aJFJJyVHr0FWZhrbApc4d6ZvWT8Ghvd3ez1kHf%2BHB4CsH3f1D53fWvqLqDBm6OtkuHQny62Ml3pE9u3wWhGV%2FvO%2BnyqQiHkvAAXS14U8bUsNkZgjDRQ4eZT3TehyBhn5Gcaorer1e3ghAimFC4ICgL4F%2F8cYb80fW3WH7rVy5fz%2Bum52WQl2y2375vsrdXZI%2B5aXa0zC3pZ3v8C%2BXJpPg%3D%3D&_l=eNptjttKw0AQhvdZ9lrDTvZALXhRSpBKN15Yqc1N2Gy22iZuQk2w3dp3dzQIFTsnmP%2F%2FBgYIEIkNBIsLWagI3HrEI2V5%2BV%2BYpoSR0pka6SPldHykxaakY8oixpikV7Q1h6bvfhVxg9LaOdwhYsn1N%2FHeOv%2F3pC%2Fqjc3d3r4a%2F%2BLygUdbMYa27Xemc2X%2BZnaV69raWHeBxaCnE0mf5vNh4IePbdM9DyDpfeWbDz94Sip5hv6MGISQ4kxgmKAE56PPbKEhu7vfPiwmQsezsAoV6DirdZiFdJlAuk0OepnsdZjcfgHf5Vrb&_t=eNpVUtty0zAQ9ad09JypJUdxHHd4cGkpBdK0NA0EzGhka5Oo9a2WEpKU%2FjsrG4bBD3s5u2dvcgYyrysvCljAKKiIMmAMomgUjhkomkvOKKVjL99IXXl1u5aVzr2sZ72Q3hD20ACJibGytWRAmkIerC4dxEbo77SCWuxkoZVY6dZYsWplF8agVMIcjIWSxC%2BktZlTWMCu6rYUGETORpoNJtOQT1jA85ByFfERVXQVSj6UIDkHHmX%2FdWYDYvINdF0Wyf08wOgKQIltWyC0sbaJUz%2F1c1Wdmqa2e2fkdZn6PyEz2kLq68rCupVWuw3B2NQvQWmZ%2BgFlY%2FHMUn%2FhFkv9e%2BR%2FzYotsJGo6sqY4nRfFm7xRpP4O1ncJtcX6HZafLhHc3rLUb77tED55fJ8iipJ3nZQp6a3Qyd5gnJ25eTn2ccHVPMF8n8MSF5oqKwwOAGJ6euAKGklLtbNMsdpT97XWwMniTo5x8kIZhiQbb4RpoG8O7J0j2ChNc5TsNM5OAv21qnuKKKHhWyaP4%2FMsFBZK8AjBgNSG4fg53nslFJPxuP4xcRhTJoa%2F4aCnOl4OIxYcGbiYUx0KTuEj8PJ5KzLWxd11udRBMYxyettZduDQ%2Fikp6meRntGC2t8kb%2BVGYIcWdoe%2FjV79VjI0fz1bf6wX86vj7OLZLScJ3x6vClnV3fH5ePdfhksHm%2BO6ml5fBrOLi7f%2FAb8JfKW&_b=eNozYPAL9fFBI2r8qiINfbNCDSOr%2FHJ8s7IrI41CTXyNIk0jQ5LLI7MCjf3cA6v8w8OyfXMDbQF9gxRO&beacon_type=start&aid=16673b0d-75bb-11e8-aa61-123b06d30007&syn%5Btiming%5D=%24TIMING_DATA",
                              @"http://cdn.spotxcdn.com/website/integration_test/beacons/tracking.html?vast_impression",
                              @"http://cdn.spotxcdn.com/website/integration_test/beacons/tracking.html?vast_start",
                              @"http://sync.search.spotxchange.com/audience_sync/9?spotx_push=1&spotx_audience_id=16673b0d-75bb-11e8-aa61-123b06d30007&redir=http%3A%2F%2Fbeacon.krxd.net%2Fusermatch.gif%3Fpartner%3Dspotxchange%26partner_uid%3D16673b0d-75bb-11e8-aa61-123b06d30007",
                              @"http://sync.search.spotxchange.com/audience_sync/8?spotx_push=1&spotx_audience_id=16673b0d-75bb-11e8-aa61-123b06d30007&redir=http%3A%2F%2Fidsync.rlcdn.com%2F397806.gif%3Fpartner_uid%3D16673b0d-75bb-11e8-aa61-123b06d30007",
                              @"http://sync.search.spotxchange.com/partner?source=spotx",
                                  @"https://vop.sundaysky.com/sync/dmp?redirect=https%3A//sync.search.spotxchange.com/partner%3Fadv_id%3D6434%26uid%3D%7Bssky_uuid%7D%26img%3D1",
                              @"https://ad.turn.com/r/cs?pid=16",
                              @"https://rp.gwallet.com/r1/cm/p115",
                              @"https://sxp.mxptint.net/sn.ashx",
                              @"http://us-u.openx.net/w/1.0/cm?id=1e113f92-30ec-486e-9d01-4f846938d316&r=http%3A%2F%2Fsync.search.spotxchange.com%2Fpartner?adv_id=6575&uid=",
                              @"https://sync.mathtag.com/sync/img?mt_exid=30&redir=https%3A%2F%2Fsync.search.spotxchange.com%2Fpartner%3Fadv_id%3D6653%26uid%3D%5BMM_UUID%5D",
                              @"http://gu.dyntrk.com/adx/sx/us.php?dynk=spt1ex",
                              @"http://track.eyeviewads.com/sync/spotx",
                              @"https://spotx-match.dotomi.com/match/bounce/current?networkId=80426&version=1",
                              @"http://pr-bh.ybp.yahoo.com/sync/spotx/16673b0d-75bb-11e8-aa61-123b06d30007",
                              @"http://pixel-a.sitescout.com/dmp/pixelSync?nid=44",
                              @"https://usersync.videoamp.com/usersync/spotx?USER_ID=16673b0d-75bb-11e8-aa61-123b06d30007",
                              @"http://dsp.adfarm1.adition.com/cookie/?ssp=14",
                              @"http://ib.adnxs.com/getuid?http://sync.search.spotxchange.com/partner?adv_id=7715&uid=$UID",
                              @"https://sync.1rx.io/usersync2/spotxchange",
                              @"https://ad.sxp.smartclip.net/sync?type=red&dsp=48&dspuuid=16673b0d-75bb-11e8-aa61-123b06d30007",
                              @"http://bh.contextweb.com/bh/rtset?pid=560212&ev=1&rurl=http%3A%2F%2Fsync.search.spotxchange.com%2Fpartner%3Fadv_id%3D8185%26uid%3D%25%25VGUID%25%25",
                              @"https://pixel.quantserve.com/pixel/p--AWABd2hShFFm.gif?idmatch=0",
                              @"http://sync.tidaltv.com/genericusersync.ashx?dpid=spotx",
                              @"https://match.prod.bidr.io/cookie-sync/sx",
                              @"https://rtb.gumgum.com/getuid/9b0d8b39?r=https%3A%2F%2Fsync.search.spotxchange.com%2Fpartner%3Fadv_id%3D8343%26uid%3D",
                              @"https://cm.adgrx.com/bridge?AG_PID=spotx&AG_SETCOOKIE",
                              @"http://event.spotxchange.com/vast/impression?_x=WzE1MzA3NzI5MTUsIjgxZDZhN2FiODAxZTExZTg4NTY3MWVkMGNhNDEwMDA3IiwiMjI0OTYyIiwiMzQ1YjYuMWVmODMuNmMzZCIsIjY1NjUiLCIzNDViNi4xZWY4My42YzNkIiwiMDY0OTEyNGM2MDRkODQ1MGQwZjZhNDNhZWE0NGU0OGIiLCIxNjY3M2IwZC03NWJiLTExZTgtYWE2MS0xMjNiMDZkMzAwMDciXSZzZXRlYz1ZbVkwTldZek5UTmtZalkwTlRZeVl6STBObUl4WTJKaU4yTTBNekExTkRBPQ~~",
                                  @"http://sync.search.spotxchange.com/audience_sync/7?spotx_push=1&spotx_audience_id=16673b0d-75bb-11e8-aa61-123b06d30007&redir=http%3A%2F%2Fbcp.crwdcntrl.net%2Fmap%2Fc%3D4914%2Ftp%3DSPXC%2Ftpid%3D16673b0d-75bb-11e8-aa61-123b06d30007",
                              @"http://sync.search.spotxchange.com/audience_sync/12?spotx_push=1&spotx_audience_id=16673b0d-75bb-11e8-aa61-123b06d30007&redir=http%3A%2F%2Fcm.g.doubleclick.net%2Fpixel%3Fgoogle_nid%3Dspotx%26google_hm%3DMTY2NzNiMGQtNzViYi0xMWU4LWFhNjEtMTIzYjA2ZDMwMDA3",
                              @"http://sync.search.spotxchange.com/audience_sync/13?spotx_push=1&spotx_audience_id=16673b0d-75bb-11e8-aa61-123b06d30007&redir=http%3A%2F%2Fdpm.demdex.net%2Fibs%3Adpid%3D178522%26dpuuid%3D16673b0d-75bb-11e8-aa61-123b06d30007",
                              @"http://spotx-sync.nuggad.net/syncuid?dpid=6597&uid=16673b0d-75bb-11e8-aa61-123b06d30007&url=http%3A%2F%2Fsync.search.spotxchange.com%2Faudience_sync%2F14%3Fmetadata_only%3D1%26spotx_audience_id%3D16673b0d-75bb-11e8-aa61-123b06d30007",
                              @"http://sync.search.spotxchange.com/audience_sync/3?spotx_push=1&spotx_audience_id=16673b0d-75bb-11e8-aa61-123b06d30007&redir=http%3A%2F%2Floadm.exelator.com%2Fload%2F%3Fp%3D204%26g%3D1135%26j%3D0%26buid%3D16673b0d-75bb-11e8-aa61-123b06d30007",
                              @"http://sync.search.spotxchange.com/audience_sync/2?spotx_push=1&spotx_audience_id=16673b0d-75bb-11e8-aa61-123b06d30007&redir=http%3A%2F%2Ftags.bluekai.com%2Fsite%2F53296%3Flimit%3D0%26id%3D16673b0d-75bb-11e8-aa61-123b06d30007",
                              @"http://b.scorecardresearch.com/b?c1=1&c2=6272977&c3=224962&cv=1.3&cj=1",
                              @"http://b.scorecardresearch.com/p?c1=2&c2=6272977&ns_ap_sv=2.1511.10&ns_type=hidden&ns_st_it=a&ns_st_sv=4.0.0&ns_st_ad=1&ns_st_sq=1&ns_st_id=1530772915441809&ns_st_ec=1&ns_st_cn=1&ns_st_ev=play&ns_st_ct=va&ns_st_cl=15000&ns_st_pt=0&c3=224962&c4=345b6.1ef83.6c3d&c6=6565&ns_ts=1530772915441809",
                              @"https://tps30.doubleverify.com/visit.jpg?ctx=484047&cmp=2228897&sid=593588&plc=22288971&num=&adid=&advid=484048&adsrv=15&tagtype=&dvtagver=6.1.img&DVP_CHANNELID=224962&DVP_DEALID=345b6.1ef83.6c3d&DVP_CMPID=345b6.1ef83.6c3d&turl=http%3A%2F%2Funknown%2F&DVPX_SX_UID=16673b0d-75bb-11e8-aa61-123b06d30007&DVPX_SX_MID=81d6a7ab801e11e885671ed0ca410007&DVPX_IP=60.247.80.142&DVPX_UA=Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%3B+rv%3A60.0%29+Gecko%2F20100101+Firefox%2F60.0&DVP_PUBLISHERID=214454&DVP_PUBNAME=Ericsson Inc&DVP_DEVICETYPE=2&DVP_MAKE=&DVP_MODEL=&DVP_OS=1",

                              @"http://search.spotxchange.com/beacon?_a=224962&_p=345b6.1ef83.6c3d&_z=1&_m=eNpFiz0LwjAYhN9fJO%2Bl%2BRwcxEKXJsGSDnWrNlQp7hX87WpQwTu44Z47uWEmIqiKjREOiiwmPZrxZBkZyNYqbZAnPo8SzGxICOm0oH0oPxKQUkl6fkShb9tvgF5FBCd4BVsuxR%2F%2BFsWPIe0QmgNCmlWs%2Byo2YfG37hrr7uLTco%2F1cRmSXwvfvgHyXy%2Fi&_l=eNptj01vwjAMhvNbcobKadKKIXHiS0Oku9BDe6nS1EC%2FWMVaAWL89xkQYkjYjiU%2F7xvJFkwwj55gVMCk8lLfEbgeSMe3MmPjgGiGpiL5zCUfnnmaZ3zIwQEAj%2Fd4Y07fXfsg6oPQGpFm4cC0f3X8NLh7%2FdKlVW4TPNqt2W0wuftJ9gFItt3etJgltdmX2DaVsfjGS8EvFxaEy%2BW9XW9gtO2T3JorlPLUPwCUwldSDn71fFEGxaeMi81JF5Eb1XEerWZV4IZuPNGHqNZHPQ8PX6ty9Afkn04p&_t=eNpVUdty2jAQ9acwemawBAZcZvpgSqBpIZBCnELd0cjSgkV9iyQINO2%2FVzJ5qR7O2T2r1V6UAuNV6YVd0iUYRIgJEAJh2B8MCQjMWUAwxkOPZ0yWXqUOrJTcS29Zb%2BhmUHOtAY2QAn5SSpYH1EZ1zq5GFk4mfeufpYCKnlkuBd1LpQ3dK9aEbZAJqq%2FaQIFGb0iZ1JF9wOwrVVAbtDkZ05m9jAfBB9IN%2BAAHIgz6WOD9gAU9BiwIIAjT%2FyqTNtI8g6ZKHK03XRvdAwh6UrmVMmPqUeInPhdlR9eVuTiDV0Xiv0KqpYHEl6WBg2JGuilBm8QvQEiW%2BF1MhvSFJH7sBkv8tc3%2FnuYnIH1aVqXWeedS5G7wWqLRDxSvovuJdRumX9bWXKwCi9N5bPH5brywFEWfGqmhxarnMIgsLmcOvy2%2FPlnaxDb%2FZxvxXEJpqLYdoBH%2B20aCGWYHa3rZ2G5bn6uThlYkWmPbGbI3NDDFM6pr4M2SmfsEA0o7T8BZcnAWXIyjZin0JlNW1%2B8fXZ7y3L5VVALym9dGlX7X7fE80sHYe3iazz0yCHq98M9uEv%2FaHh9%2Fb4%2Fj7GF2d1lOtpfdcZotjvevi%2BKxt3yOs91sKhdH%2FvEfX8DTQA%3D%3D&_b=eNozYfAL9fFBI2r8w12NI3N9TfyyHE39QzKyorIcDfxcko19qxxNfF2CsiOznDL9XPyyfatCbQFnrxOe&beacon_type=recurring&view_percent=75",
                              @"http://search.spotxchange.com/beacon?_a=224962&_p=345b6.1ef83.6c3d&_z=1&_m=eNpFiz0LwjAYhN9fJO%2Bl%2BRwcxEKXJsGSDnWrNlQp7hX87WpQwTu44Z47uWEmIqiKjREOiiwmPZrxZBkZyNYqbZAnPo8SzGxICOm0oH0oPxKQUkl6fkShb9tvgF5FBCd4BVsuxR%2F%2BFsWPIe0QmgNCmlWs%2Byo2YfG37hrr7uLTco%2F1cRmSXwvfvgHyXy%2Fi&_l=eNptj01vwjAMhvNbcobKadKKIXHiS0Oku9BDe6nS1EC%2FWMVaAWL89xkQYkjYjiU%2F7xvJFkwwj55gVMCk8lLfEbgeSMe3MmPjgGiGpiL5zCUfnnmaZ3zIwQEAj%2Fd4Y07fXfsg6oPQGpFm4cC0f3X8NLh7%2FdKlVW4TPNqt2W0wuftJ9gFItt3etJgltdmX2DaVsfjGS8EvFxaEy%2BW9XW9gtO2T3JorlPLUPwCUwldSDn71fFEGxaeMi81JF5Eb1XEerWZV4IZuPNGHqNZHPQ8PX6ty9Afkn04p&_t=eNpVUdty2jAQ9acwemawBAZcZvpgSqBpIZBCnELd0cjSgkV9iyQINO2%2FVzJ5qR7O2T2r1V6UAuNV6YVd0iUYRIgJEAJh2B8MCQjMWUAwxkOPZ0yWXqUOrJTcS29Zb%2BhmUHOtAY2QAn5SSpYH1EZ1zq5GFk4mfeufpYCKnlkuBd1LpQ3dK9aEbZAJqq%2FaQIFGb0iZ1JF9wOwrVVAbtDkZ05m9jAfBB9IN%2BAAHIgz6WOD9gAU9BiwIIAjT%2FyqTNtI8g6ZKHK03XRvdAwh6UrmVMmPqUeInPhdlR9eVuTiDV0Xiv0KqpYHEl6WBg2JGuilBm8QvQEiW%2BF1MhvSFJH7sBkv8tc3%2FnuYnIH1aVqXWeedS5G7wWqLRDxSvovuJdRumX9bWXKwCi9N5bPH5brywFEWfGqmhxarnMIgsLmcOvy2%2FPlnaxDb%2FZxvxXEJpqLYdoBH%2B20aCGWYHa3rZ2G5bn6uThlYkWmPbGbI3NDDFM6pr4M2SmfsEA0o7T8BZcnAWXIyjZin0JlNW1%2B8fXZ7y3L5VVALym9dGlX7X7fE80sHYe3iazz0yCHq98M9uEv%2FaHh9%2Fb4%2Fj7GF2d1lOtpfdcZotjvevi%2BKxt3yOs91sKhdH%2FvEfX8DTQA%3D%3D&_b=eNozZvAL9fFBI2r8srJN%2FENcjf1cnHIjQ6Jyo3J9q6LC%2FbJ9XVwrfKu8cn1dQg0ic%2F2AMNQWAHwnFGs%3D&beacon_type=recurring&view_percent=50",
                              @"http://search.spotxchange.com/beacon?_a=224962&_p=345b6.1ef83.6c3d&_z=1&_m=eNpFiz0LwjAYhN9fJO%2Bl%2BRwcxEKXJsGSDnWrNlQp7hX87WpQwTu44Z47uWEmIqiKjREOiiwmPZrxZBkZyNYqbZAnPo8SzGxICOm0oH0oPxKQUkl6fkShb9tvgF5FBCd4BVsuxR%2F%2BFsWPIe0QmgNCmlWs%2Byo2YfG37hrr7uLTco%2F1cRmSXwvfvgHyXy%2Fi&_l=eNptj01vwjAMhvNbcobKadKKIXHiS0Oku9BDe6nS1EC%2FWMVaAWL89xkQYkjYjiU%2F7xvJFkwwj55gVMCk8lLfEbgeSMe3MmPjgGiGpiL5zCUfnnmaZ3zIwQEAj%2Fd4Y07fXfsg6oPQGpFm4cC0f3X8NLh7%2FdKlVW4TPNqt2W0wuftJ9gFItt3etJgltdmX2DaVsfjGS8EvFxaEy%2BW9XW9gtO2T3JorlPLUPwCUwldSDn71fFEGxaeMi81JF5Eb1XEerWZV4IZuPNGHqNZHPQ8PX6ty9Afkn04p&_t=eNpVUdty2jAQ9acwemawBAZcZvpgSqBpIZBCnELd0cjSgkV9iyQINO2%2FVzJ5qR7O2T2r1V6UAuNV6YVd0iUYRIgJEAJh2B8MCQjMWUAwxkOPZ0yWXqUOrJTcS29Zb%2BhmUHOtAY2QAn5SSpYH1EZ1zq5GFk4mfeufpYCKnlkuBd1LpQ3dK9aEbZAJqq%2FaQIFGb0iZ1JF9wOwrVVAbtDkZ05m9jAfBB9IN%2BAAHIgz6WOD9gAU9BiwIIAjT%2FyqTNtI8g6ZKHK03XRvdAwh6UrmVMmPqUeInPhdlR9eVuTiDV0Xiv0KqpYHEl6WBg2JGuilBm8QvQEiW%2BF1MhvSFJH7sBkv8tc3%2FnuYnIH1aVqXWeedS5G7wWqLRDxSvovuJdRumX9bWXKwCi9N5bPH5brywFEWfGqmhxarnMIgsLmcOvy2%2FPlnaxDb%2FZxvxXEJpqLYdoBH%2B20aCGWYHa3rZ2G5bn6uThlYkWmPbGbI3NDDFM6pr4M2SmfsEA0o7T8BZcnAWXIyjZin0JlNW1%2B8fXZ7y3L5VVALym9dGlX7X7fE80sHYe3iazz0yCHq98M9uEv%2FaHh9%2Fb4%2Fj7GF2d1lOtpfdcZotjvevi%2BKxt3yOs91sKhdH%2FvEfX8DTQA%3D%3D&_b=eNozYvAL9fFBI2p8qxwNIrOyK6JCUnL8XUIr%2FEIyMiOz0iv83X0rfLOyTX2rko38XVwrIqsibQGT7hUz&beacon_type=recurring&view_percent=25",
                              @"http://search.spotxchange.com/beacon?_a=224962&_p=345b6.1ef83.6c3d&_z=1&_m=eNpFiz0LwjAYhN9fJO%2Bl%2BRwcxEKXJsGSDnWrNlQp7hX87WpQwTu44Z47uWEmIqiKjREOiiwmPZrxZBkZyNYqbZAnPo8SzGxICOm0oH0oPxKQUkl6fkShb9tvgF5FBCd4BVsuxR%2F%2BFsWPIe0QmgNCmlWs%2Byo2YfG37hrr7uLTco%2F1cRmSXwvfvgHyXy%2Fi&_l=eNptj01vwjAMhvNbcobKadKKIXHiS0Oku9BDe6nS1EC%2FWMVaAWL89xkQYkjYjiU%2F7xvJFkwwj55gVMCk8lLfEbgeSMe3MmPjgGiGpiL5zCUfnnmaZ3zIwQEAj%2Fd4Y07fXfsg6oPQGpFm4cC0f3X8NLh7%2FdKlVW4TPNqt2W0wuftJ9gFItt3etJgltdmX2DaVsfjGS8EvFxaEy%2BW9XW9gtO2T3JorlPLUPwCUwldSDn71fFEGxaeMi81JF5Eb1XEerWZV4IZuPNGHqNZHPQ8PX6ty9Afkn04p&_t=eNpVUV2X0jAQ7U%2Fh5JlDkxKgco4PXVl1FVhWoIjWkxOSgUb7tU1AYPW%2FOy37Yh7unbmTSeYmO5CqLLwwYAGjoEPKgDEIw8FwxEBTJTmjlI48lUpTeGV9kIVR3u7W9UJugXCXCsiYqDKvMnBAuqTK5MWZvFHZAPOT0VCKk8yMFntTWyf2tWzLWJRa2It1kJPxC6ndriE8wO3LOhdYxJ5U2hQ30yF%2FwwKuhpTrkA%2Bopvuh5H0JknPg4e6%2Fm1mXWJVCe0scLVcBVvcAWhzrDKXUuWqc%2BImvdNGzVenOTYAeEv837KxxkPimcHCopTONSbAu8XPQRiZ%2BQNlIPLPEjxtjib%2FE%2Fq%2B77AhsIIqysDbrnfOsMV4ZMv5O4kX0MMG0ZfFpieFswRHfT2PEzf3dDCmK3rVSS7NFv0EeIT5%2BaPDL4%2Bc10irG%2Fh9dojIDhRMWJyBj%2BrdLtHQSjbWzrHDazsfyaKET6c4dTkZwhwVZq1TYClT7yLL5BAe1bTINJ6OgieDsGmofRdxkIavq9Z%2BLY5bhWXmpIbtlXVLaVx2X57Eepd58PZ16bMj7%2FfDP%2FOcT3%2BYPdLs6XGebNZtf1Xk2SbP59T6YX2Mzm%2Bhf3zbbwTx%2FevsPK2rScA%3D%3D&_b=eNozZPAL9fFBI2ois8Iy%2FN2DcvyqPCsjq7KN%2FKqSjaNy3TKjXMIy%2FLKyTfyyXMsjszyNfLN8bQGSHxTj&beacon_type=complete",
                              @"http://cdn.spotxcdn.com/website/integration_test/beacons/tracking.html?vast_firstQuartile",
                              @"http://cdn.spotxcdn.com/website/integration_test/beacons/tracking.html?vast_midpoint",
                              @"http://cdn.spotxcdn.com/website/integration_test/beacons/tracking.html?vast_thirdQuartile",
                              @"http://cdn.spotxcdn.com/website/integration_test/beacons/tracking.html?vast_complete",
                              @"http://event.spotxchange.com/vast/complete?_x=WzE1MzA3NzI5MTUsIjgxZDZhN2FiODAxZTExZTg4NTY3MWVkMGNhNDEwMDA3IiwiMjI0OTYyIiwiMzQ1YjYuMWVmODMuNmMzZCIsIjY1NjUiLCIzNDViNi4xZWY4My42YzNkIiwiMDY0OTEyNGM2MDRkODQ1MGQwZjZhNDNhZWE0NGU0OGIiXSZzZXRlYz1NV0UyTnpkalpURTBaamcxWVRCbU56UTJPRFpoWkRkbVlXRTROR1UzTVRRPQ~~",
                              @"http://event.spotxchange.com/vast/25?_x=WzE1MzA3NzI5MTUsIjgxZDZhN2FiODAxZTExZTg4NTY3MWVkMGNhNDEwMDA3IiwiMjI0OTYyIiwiMzQ1YjYuMWVmODMuNmMzZCIsIjY1NjUiLCIzNDViNi4xZWY4My42YzNkIiwiMDY0OTEyNGM2MDRkODQ1MGQwZjZhNDNhZWE0NGU0OGIiXSZzZXRlYz1NV0UyTnpkalpURTBaamcxWVRCbU56UTJPRFpoWkRkbVlXRTROR1UzTVRRPQ~~",
                              @"http://event.spotxchange.com/vast/50?_x=WzE1MzA3NzI5MTUsIjgxZDZhN2FiODAxZTExZTg4NTY3MWVkMGNhNDEwMDA3IiwiMjI0OTYyIiwiMzQ1YjYuMWVmODMuNmMzZCIsIjY1NjUiLCIzNDViNi4xZWY4My42YzNkIiwiMDY0OTEyNGM2MDRkODQ1MGQwZjZhNDNhZWE0NGU0OGIiXSZzZXRlYz1NV0UyTnpkalpURTBaamcxWVRCbU56UTJPRFpoWkRkbVlXRTROR1UzTVRRPQ~~",
                              @"http://event.spotxchange.com/vast/75?_x=WzE1MzA3NzI5MTUsIjgxZDZhN2FiODAxZTExZTg4NTY3MWVkMGNhNDEwMDA3IiwiMjI0OTYyIiwiMzQ1YjYuMWVmODMuNmMzZCIsIjY1NjUiLCIzNDViNi4xZWY4My42YzNkIiwiMDY0OTEyNGM2MDRkODQ1MGQwZjZhNDNhZWE0NGU0OGIiXSZzZXRlYz1NV0UyTnpkalpURTBaamcxWVRCbU56UTJPRFpoWkRkbVlXRTROR1UzTVRRPQ~~",
                              @"http://b.scorecardresearch.com/p?c1=2&c2=6272977&ns_ap_sv=2.1511.10&ns_type=hidden&ns_st_it=a&ns_st_sv=4.0.0&ns_st_ad=1&ns_st_sq=1&ns_st_id=1530772915441809&ns_st_ec=2&ns_st_cn=1&ns_st_ev=end&ns_st_ct=va&ns_st_cl=15000&ns_st_pt=0&c3=224962&c4=345b6.1ef83.6c3d&c6=6565&ns_ts=1530772915441809",
                              };

            StreamWriter sw = new StreamWriter("exceptions.txt");
            for (int iterations = 0; iterations < 1000; iterations++)
            {
                var memStats = GetMemoryStatusLinux();
                double pctUsed = (double) memStats.dwAvailPhys * 100 / (double) memStats.dwTotalPhys;
                Console.WriteLine("MEM: {0}, {1}", iterations, pctUsed);		    

                foreach (string website in websites)
                {
                    HttpWebResponse response = null;
                    StreamReader reader = null;
                    try
                    {
                        // GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
                        GC.Collect();
                        Console.Error.WriteLine(DateTime.Now.ToString("HH:mm:ss") + " Getting " + website);
                        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(website);
                        //WebProxy myProxy = new WebProxy("yourproxyip", yourproxyport);
                        //request.Proxy = myProxy;
                        response = request.GetResponse() as HttpWebResponse;
                        reader = new StreamReader(response.GetResponseStream());
                        File.WriteAllText("florb.txt", reader.ReadToEnd());
                        reader.Close();
                        reader = null;
                        response.Close();
                        response = null;
                    }
                    catch (Exception ex)
                    {
                        string exString = ex.ToString();
                        exString = exString.Substring(0, Math.Min(150, exString.Length));
                        Console.Error.WriteLine("Caught exception . . . " + exString);
                        sw.WriteLine("EXCEPTION\r\n" + ex.ToString() + "\r\n");
                        sw.Flush();
                    }
                    finally
                    {
                        if (reader != null) reader.Close();
                        if (response != null) response.Close();
                    }
                }
            }
        }
    }
}

When the program is run on the non-root-certificate system we start with memory utilization at 68.5% but by the 418th iteration we are down to 39.6%

When this program is run with the root certificates installed memory usage is flat: 93.35 at the start and 93.25 by the 418th iteration.

@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Aug 3, 2018

Here is a graph of the memory availability for the two cases:
8689

@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Aug 7, 2018

The leak only appears to happen if the handshake fails. If the URL is changed so that the request returns 404 things are fine. I have obtained trace output of the working and non-working case and the paths diverge at, unsurprisingly,Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=<n> where n=0, for the non-leak case, and n=1 for the leak case:
Non-leaking -

[0x7f8864ffc700: 2.10472 30] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0x7f88649f9700: 2.14909 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0x7f8864ffc700: 2.15994 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0x7f8864ffc700: 2.16560 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0x7f8864dfb700: 2.22728 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0x7f8864bfa700: 2.23891 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0x7f884761a700: 2.24481 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0x7f884761a700: 2.25049 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0x7f884761a700: 2.25644 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0x7f8864dfb700: 2.28471 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0x7f8864bfa700: 2.29275 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0x7f884761a700: 2.29849 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0x7f884761a700: 2.30408 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0x7f884761a700: 2.30973 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0x7f8864bfa700: 2.31561 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=0
[0x7f8864bfa700: 2.31702 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=0

Leaking -

[0xb3affb40: 3.44245 30] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0xb40ffb40: 3.52153 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0xb3dffb40: 3.52850 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0xb40ffb40: 3.53632 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=2
[0xb3dffb40: 3.60544 16] LEAVE: Mono.Btls.MonoBtlsContext:DoProcessHandshake ()result=1

I am trying to understand why those objects created by BufferOffsetSize2 are still seen as alive by GC.

I found if I made this mod:

--- a/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs
+++ b/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs
@@ -392,8 +392,11 @@ namespace Mono.Net.Security
                                }
                        }
 
-                       if (result.Error != null)
+                       if (result.Error != null) {
+                               readBuffer = null;
+                               writeBuffer = null;
                                result.Error.Throw ();
+                       }
                }
 
                protected abstract MobileTlsContext CreateContext (MonoSslAuthenticationOptions options);

I no longer have thousands and thousands of 16850/16400 sized System.Byte[] objects and memory use is substantially lower but still very large.

8689-mod

The mprof report now shows that System.Byte[] is pretty stable but System.String continues to grow:

        Heap shot 30 at 152.001 secs: size: 4538968, object count: 93967, class count: 461, roots: 0
             Bytes      Count  Average Class name
            958176      19962       48 System.Diagnostics.StackFrame (bytes: +33312, count: +694)
            734160       5506      133 System.String (bytes: +17312, count: +99)
            169168       5278       32 System.Diagnostics.StackFrame[] (bytes: +5856, count: +182)
            159944       5278       30 System.Diagnostics.StackTrace[] (bytes: +5488, count: +182)
            133368        448      297 System.Byte[] (bytes: +336, count: +14)
            126672       5278       24 System.Diagnostics.StackTrace (bytes: +4368, count: +182)
            104176        407      255 System.Net.HttpWebRequest (bytes: +3584, count: +14)
:
:
        Heap shot 92 at 548.499 secs: size: 12106320, object count: 258578, class count: 461, roots: 0
             Bytes      Count  Average Class name
           2912112      60669       48 System.Diagnostics.StackFrame (bytes: +29136, count: +607)
           1768320      11440      154 System.String (bytes: +17848, count: +111)
            515696      16133       31 System.Diagnostics.StackFrame[] (bytes: +5288, count: +169)
            488464      16133       30 System.Diagnostics.StackTrace[] (bytes: +5168, count: +169)
            387192      16133       24 System.Diagnostics.StackTrace (bytes: +4056, count: +169)
            317936       1242      255 System.Net.HttpWebRequest (bytes: +3328, count: +13)
            287912       3723       77 System.IntPtr[] (bytes: +3016, count: +39)
            198640       2483       80 Mono.Security.Interface.MonoTlsSettings (bytes: +2080, count: +26)
            158968       1285      123 System.Byte[] (bytes: +5904, count: +15)

I assume the String/Diagnostics are all related to the messages being generated when we get the SSL exception.

@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Aug 7, 2018

Further information from mprof-report with the --traces option:

       Heap shot 92 at 548.499 secs: size: 12106320, object count: 258578, class count: 461, roots: 489945
             Bytes      Count  Average Class name
           2912112      60669       48 System.Diagnostics.StackFrame (bytes: +29136, count: +607)
                14 root references (0 pinning)
                60669 references from: System.Diagnostics.StackFrame[]
           1768320      11440      154 System.String (bytes: +17848, count: +111)
                169040 root references (0 pinning)
                3726 references from: System.Net.HttpWebRequest
                2482 references from: Mono.Btls.MonoBtlsException
                2482 references from: Mono.Btls.MonoBtlsContext
                2482 references from: System.Net.WebException
                2482 references from: System.Security.Authentication.AuthenticationException
:
            515696      16133       31 System.Diagnostics.StackFrame[] (bytes: +5288, count: +169)
                8 root references (0 pinning)
                16133 references from: System.Diagnostics.StackTrace
            488464      16133       30 System.Diagnostics.StackTrace[] (bytes: +5168, count: +169)
                12410 references from: System.Diagnostics.StackTrace
                7446 references from: System.Runtime.ExceptionServices.ExceptionDispatchInfo
                1241 references from: Mono.Btls.MonoBtlsException
                1241 references from: System.Net.WebException
                1241 references from: System.Security.Authentication.AuthenticationException
            387192      16133       24 System.Diagnostics.StackTrace (bytes: +4056, count: +169)
                46089 references from: System.Diagnostics.StackTrace[]
            317936       1242      255 System.Net.HttpWebRequest (bytes: +3328, count: +13)
                2 root references (0 pinning)
                2484 references from: System.Net.HttpWebRequest
                1243 references from: System.Net.WebOperation
                1241 references from: Mono.Net.Security.MonoTlsStream
                1241 references from: Mono.Net.Security.ChainValidationHelper
                2 references from: System.Net.WebRequestStream
                2 references from: System.Net.WebResponseStream
:
            158968       1285      123 System.Byte[] (bytes: +5904, count: +15)
                681 root references (0 pinning)
                1245 references from: System.IO.FileStream
                14 references from: System.Net.ServicePoint
                3 references from: Mono.Globalization.Unicode.SimpleCollator
                2 references from: System.IO.UnexceptionalStreamWriter
                2 references from: System.Net.BufferOffsetSize
                2 references from: System.IO.StreamWriter
                2 references from: System.Net.WebRequestStream
                2 references from: System.IO.UnexceptionalStreamReader
                1 references from: System.IO.StreamReader

So now it appears that we are not cleaning up after the diagnostics have been produced and that's where I am stuck at the moment.

@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Aug 7, 2018

Interestingly, if I run the program below and check its heap shots I see no System.Diagnostics.xxxx objects in the report. Which makes sense as it appears that:

                1241 references from: Mono.Btls.MonoBtlsException
                1241 references from: System.Net.WebException
                1241 references from: System.Security.Authentication.AuthenticationException

Are keeping them alive. So why are those objects still active?

using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Runtime;
using System.Runtime.InteropServices;

namespace TestMemGrow
{
    class Program
    {
        static void Main()
        {
            StreamWriter sw = new StreamWriter("exceptions.txt");
            for (int iterations = 0; iterations < 100; iterations++)
            {
                try
                {
                    GC.Collect();
		    throw new NullReferenceException();
                }
                catch (Exception ex)
                {
                    string exString = ex.ToString();
                    exString = exString.Substring(0, Math.Min(150, exString.Length));
                    Console.Error.WriteLine("Caught exception . . . " + exString);
                    sw.WriteLine("EXCEPTION\r\n" + ex.ToString() + "\r\n");
                    sw.Flush();
		    ex = null;
		    exString = null;
                }
                finally
                {
                }
            }
        }
    }
}
@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Aug 8, 2018

If I make the following change:

--- a/mcs/class/System/Mono.Btls/MonoBtlsContext.cs
+++ b/mcs/class/System/Mono.Btls/MonoBtlsContext.cs
@@ -157,7 +157,7 @@ namespace Mono.Btls
                        }
                }
 
-               static Exception GetException (MonoBtlsSslError status)
+               private Exception GetException (MonoBtlsSslError status)
                {
                        string file;
                        int line;
@@ -176,6 +176,7 @@ namespace Mono.Btls
                                message = string.Format ("{0} {1}\n  at {2}:{3}", status, text, file, line);
                        else
                                message = string.Format ("{0} {1}", status, text);
+                       Dispose (true);
                        return new MonoBtlsException (message);
                }
 

The mprof report now shows for Heap shot 92:

        Heap shot 92 at 419.139 secs: size: 877328, object count: 14151, class count: 434, roots: 304871
             Bytes      Count  Average Class name
            236864       2696       87 System.String (bytes: +1656, count: +17)

And no sign of the StackTrace/StackFrames etc.

To compare the change in memory available for the unmodified case:

Start - 68.50
End   - 10.66
Delta - 57.34

Modified MonoBtlsContext.cs:

Start - 78.84
End   - 78.50
Delta - 0.34
@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Aug 8, 2018

Graph of memory usage with the mod:
8689-mod

@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Aug 8, 2018

@marek-safar @baulig Would you review the two patches I've come up with? If they make sense I will generate a PR but I think they need a once over before we get to that point.

This one reduces System.Byte[] from thousands of 16850 and 16400 byte arrays (also should they both be 16400 - or 16384 + 16 bytes)

--- a/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs
+++ b/mcs/class/System/Mono.Net.Security/MobileAuthenticatedStream.cs
@@ -392,8 +392,11 @@ namespace Mono.Net.Security
                                }
                        }
 
-                       if (result.Error != null)
+                       if (result.Error != null) {
+                               readBuffer = null;
+                               writeBuffer = null;
                                result.Error.Throw ();
+                       }
                }
 
                protected abstract MobileTlsContext CreateContext (MonoSslAuthenticationOptions options);

This one gets rid of all the Diagnostic objects and most of the String objects after an Exception during the Btls exchange

--- a/mcs/class/System/Mono.Btls/MonoBtlsContext.cs
+++ b/mcs/class/System/Mono.Btls/MonoBtlsContext.cs
@@ -157,7 +157,7 @@ namespace Mono.Btls
                        }
                }
 
-               static Exception GetException (MonoBtlsSslError status)
+               private Exception GetException (MonoBtlsSslError status)
                {
                        string file;
                        int line;
@@ -176,6 +176,7 @@ namespace Mono.Btls
                                message = string.Format ("{0} {1}\n  at {2}:{3}", status, text, file, line);
                        else
                                message = string.Format ("{0} {1}", status, text);
+                       Dispose (true);
                        return new MonoBtlsException (message);
                }
 
@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Aug 9, 2018

Come to think of it, shouldn't all that stuff be cleaned up upon the successful completion of the GetResponse() as well - if the KeepAlive property is not true? Each time in the loop if our test program a new httpwebrequest for ssl connections will result in ctx, ssl, bio, errbio ... objects that become orphaned? For persistent connections the Close operation would be responsible for the cleanup.

@baulig

This comment has been minimized.

Copy link
Member

@baulig baulig commented Aug 22, 2018

Regardless of KeepAlive - it should be cleaned up if the handshake failed.

@baulig

This comment has been minimized.

Copy link
Member

@baulig baulig commented Aug 22, 2018

I'll have a look at this and see whether I can figure something out.

@baulig baulig self-assigned this Aug 22, 2018
@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Aug 22, 2018

Regardless of KeepAlive - it should be cleaned up if the handshake failed.

Even when it's successful and KeepAlive is not set shouldn't there be cleanup as well? Once a response is received the ssl/ctx/bio objects etc. for that request/response are no longer useful/usable and will gradually eat away memory until the app stops.

@baulig

This comment has been minimized.

Copy link
Member

@baulig baulig commented Aug 23, 2018

I think I found the problem, PR incoming.

baulig added a commit to baulig/mono that referenced this issue Aug 23, 2018
…Fixes mono#8689.

When the TLS Handshake fails in `MonoTlsStream.CreateStream()`, then we must dispose
the `SslStream` to avoid a memory leak.

In addition to this, `MonoTlsStream` now implements `IDisposable` and is disposes
in `WebConnection.CloseSocket()`.
@baulig

This comment has been minimized.

Copy link
Member

@baulig baulig commented Aug 23, 2018

I had it running for about 5 minutes and it's not growing anymore. Adding the sslStream?.Dispose() to MonoTlsStream.CreateStream() should already be enough to fix it, but I added the IDisposable just to make extra sure. PR is up: #10254.

@nealef

This comment has been minimized.

Copy link
Contributor

@nealef nealef commented Aug 23, 2018

I ran it through 1000 iterations with these changes applied (and my ones backed off) plus the GC.SuppressFinalize(this); as suggested in the PR. I am also seeing no growth:
8689-64

@marek-safar marek-safar added this to the 2018-08 (5.18.xx) milestone Aug 27, 2018
akoeplinger added a commit that referenced this issue Aug 28, 2018
#10254)

When the TLS Handshake fails in `MonoTlsStream.CreateStream()`, then we must dispose
the `SslStream` to avoid a memory leak.

In addition to this, `MonoTlsStream` now implements `IDisposable` and is disposes
in `WebConnection.CloseSocket()`.

Fixes #8689
monojenkins added a commit to monojenkins/mono that referenced this issue Aug 28, 2018
…ixes mono#8689.

When the TLS Handshake fails in `MonoTlsStream.CreateStream()`, then we must dispose
the `SslStream` to avoid a memory leak.

In addition to this, `MonoTlsStream` now implements `IDisposable` and is disposes
in `WebConnection.CloseSocket()`.
monojenkins added a commit to monojenkins/mono that referenced this issue Aug 28, 2018
…ixes mono#8689.

When the TLS Handshake fails in `MonoTlsStream.CreateStream()`, then we must dispose
the `SslStream` to avoid a memory leak.

In addition to this, `MonoTlsStream` now implements `IDisposable` and is disposes
in `WebConnection.CloseSocket()`.
EgorBo added a commit to EgorBo/mono that referenced this issue Sep 10, 2018
mono#10254)

When the TLS Handshake fails in `MonoTlsStream.CreateStream()`, then we must dispose
the `SslStream` to avoid a memory leak.

In addition to this, `MonoTlsStream` now implements `IDisposable` and is disposes
in `WebConnection.CloseSocket()`.

Fixes mono#8689
@smr888

This comment has been minimized.

Copy link
Author

@smr888 smr888 commented Jul 17, 2019

If anyone can shed any light on this I'd appreciate it . . . I reported this bug last May, and nealef picked up the ball and ran with it further than I could have. The work on the bug was evidently completed in August of last year, but its resolution was apparently not incorporated into the release code until Mono 5.20.0, which came out on April 11. I'm curious as to why it took so long for the fix to make it to the release code. Simply a matter of priorities / testing?

FWIW we froze our usage of Mono at 5.14.0.177 due to this bug, choosing to work around by having our processes just stop after a couple of days and get restarted by a watchdog program. It would be nice to be able to go back to just letting the programs run, and I'll do some testing with the new Mono to see if I can.

Thanks again to mbaulig and nealef for their work on this.

P.S. I now see that the bugfix is also noted as incorporated in 5.18.0, which came out in December. Hmmm. Perhaps the 5.20.0 notation is an error. December, if correct, would make more sense to me in terms of time frame, and render my question moot.

@mysl

This comment has been minimized.

Copy link

@mysl mysl commented Jul 18, 2019

If you compare the files changed, the fix is actually already in 5.16, but not documented in the release notes

@smr888

This comment has been minimized.

Copy link
Author

@smr888 smr888 commented Jul 18, 2019

Hmm, ok thanks for that info. I am testing 5.20.0 to see if I can confirm that my test no longer leaks memory. Unfortunately the mono profiler doesn't seem to work -- mprof-report wasn't installed along with 5.20.0, had to do that by hand, and having done so no output.mlpd files are being created when I try to produce debug info. So it's System Monitor, not my favorite way of checking for memory growth.

@baulig

This comment has been minimized.

Copy link
Member

@baulig baulig commented Jul 22, 2019

It looks like we go a new variant of this, this time with AutomaticDecompression.

@baulig baulig reopened this Jul 22, 2019
@smr888

This comment has been minimized.

Copy link
Author

@smr888 smr888 commented Jul 23, 2019

To see the issue, add:

request.AutomaticDecompression =  DecompressionMethods.GZip | DecompressionMethods.Deflate;

before the call to GetResponse, and remove the WebProxy stuff. The decompression methods seem to leak memory. I believe this happens when exceptions are hit.

You can see the test code I was using on bug 15780, linked above.

This problem can be seen within 10 minutes. It does not require the hours that the original one did.

@smr888

This comment has been minimized.

Copy link
Author

@smr888 smr888 commented Aug 6, 2019

For anyone else encountering this problem, as a workaround it is fairly easy to just turn off AutomaticDecompression, say you'll accept gzip, then take a look at the response.ContentEncoding when you get a web page. If it contains "gzip", you can decompress the response stream yourself using the .NET facilities for doing so, and if not, just read the uncompressed data straight in. That's what I've done so that I could stomp this problem out now pending a fix.

Oh -- I had to "say you'll accept gzip" like this:

            request.Headers.Remove("Accept-Encoding");
            request.Headers.Set("Accept-Encoding", "gzip");

because it didn't seem to work when I tried it other ways.

These memory leaks occur when an exception is thrown while getting a web page and AutomaticDecompression is on. Unfortunately frequent exceptions are a way of life when getting web pages, so the workaround is helpful in suppressing leaks in long-running programs.

UnityAlex added a commit to Unity-Technologies/mono that referenced this issue Sep 3, 2019
mono#10254)

When the TLS Handshake fails in `MonoTlsStream.CreateStream()`, then we must dispose
the `SslStream` to avoid a memory leak.

In addition to this, `MonoTlsStream` now implements `IDisposable` and is disposes
in `WebConnection.CloseSocket()`.

Fixes mono#8689
@akoeplinger akoeplinger removed this from the 2018-08 (5.18.xx) milestone Oct 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.