Sockets Left in Close_Wait State after API Calls on Google Calendar Causing Server Resource Issue

[GoogleCodeExporter]

What steps will reproduce the problem?

1. Create a new calendar service 

 credential = new ServiceAccountCredential(new ServiceAccountCredential.Initializer(serviceAccountEmail)
                  {
                      User = userName,
                      Scopes = new[] { CalendarService.Scope.Calendar 
                  }
                  }.FromCertificate(certificate));

2. Use the service to perform an action on the calendar - e.g. get events

 IList<Google.Apis.Calendar.v3.Data.Event> eventList = new List<Google.Apis.Calendar.v3.Data.Event>();

            for (int i = 0; i < _MaxRetries; i++)
            {
                eventList.Clear();

                try
                {
                   retries++;
                   Google.Apis.Calendar.v3.EventsResource.ListRequest request = _CalendarService.Events.List(calendarId);
                   request.TimeMax = endDate;
                   request.TimeMin = startDate;
                   request.MaxResults = 1000;

                        var events = request.Execute();

                        IList<Google.Apis.Calendar.v3.Data.Event> items = events.Items;
                        foreach (Google.Apis.Calendar.v3.Data.Event e in items)
                        {
                            eventList.Add(e);
                        }

                    isSuccess = true;
                    break;
                }
                catch
                {
                    delay += _DelaySeed;
                    Thread.Sleep(delay);
                }

Note: this query is being run in an application that displays a summary of 
calendar events for our users for different internal Websites. 

3. On the server, run Netstat -anob >c:\temp\filename.txt

What is the expected output? What do you see instead?

- When the API call is performed for a single user, 4 socket connections appear 
on the server. There are multiple users using our production application, so a 
large number of connections are created. 
- After the API call completes, users see the calendar data on a Web page
- On the server, the connections are still in ESTABLISHED state for a few 
minutes
- After a while, we are seeing a build up of socket connections in Close_Wait 
state on our production servers. 
- Under some circumstances, the Close_Wait connections do not close and 
eventually the server begins to experience a resource issue
- We were expecting the socket connections to clear up faster

What version of the product are you using?

We are using the version of the Calendar and .NET Client Library prior to the 
latest release in December. We used Nuget to get the libraries 

PM> Install-Package Google.Apis.Calendar.v3 

What is your operating system?

Windows Server 2008

What is your IDE?

VS 2010

What is the .NET framework version?

.NET 4.0

Please provide any additional information below.

We have been experiencing this issue for over a month now since we migrated to 
V3 of the Calendar API. We only recently determined that the problem was 
related to the Google APIs. We have doubled checked our application code and it 
seems fine, as it is based on examples provided by Google. There are no "Close" 
options that we can call to initiate closing of the sockets / connections, and 
we are assuming that is handled at a lower level by the .NET Client Library. We 
were wondering if there have been any reported bugs that might explain sockets 
being left open on the server after API calls. This is an urgent issue for us, 
so your assistance would be greatly appreciated.

The following details were provided to us by a Microsoft engineer analysing a 
system dump:

From Microsoft engineer:
The issue is: w3wp.exe is holding 16000+ Windows Socket handle. All blocked 
Socket handle are in CLOSE_WAIT status.
http://blogs.technet.com/b/janelewis/archive/2010/03/09/explaining-close-wait.as
px
“CLOSE_WAIT 
Indicates that the server has received the first FIN signal from the client and 
the connection is in the process of being closed 

So this essentially means that his is a state where socket is waiting for the 
application to execute close() 

A socket can be in CLOSE_WAIT state indefinitely until the application closes 
it. 
Faulty scenarios would be like file descriptor leak, server not being execute 
close() on socket leading to pile up of close_wait sockets”

I also debugged the memory dump file. And all of those 16000+ sockets are 
created through AW3_GoogleData.dll and Google_Apis.dll and 
Google.Apis.Auth.OAuth2.ServiceAccountCredential (Google Token service).

Here is the detailed information for how those 16000+ leaked sockets been used 
for, which are all staying CLOSE_WAIT state.
https://www.googleapis.com/calendar/v3/calendars
and
https://accounts.google.com/o/oauth2/token 

The connection is like this:
URL:       
https://www.googleapis.com/calendar/v3/calendars/tdsb.on.ca_o793na7jl9e3mc46g9a9
94rqrs%40group.calendar.google.com/events?timeMax=2015-01-18T00%3A00%3A00.000-05
%3A00&timeMin=2014-12-18T00%3A00%3A00.000-05%3A00 
Verb:     GET
HTTP Header:
GET 
/calendar/v3/calendars/tdsb.on.ca_o793na7jl9e3mc46g9a994rqrs%40group.calendar.go
ogle.com/events?timeMax=2015-01-18T00%3A00%3A00.000-05%3A00&timeMin=2014-12-18T0
0%3A00%3A00.000-05%3A00 HTTP/1.1
User-Agent: TDSB Google API google-api-dotnet-client/1.9.0.26010 (gzip)
Authorization: Bearer 
ya29.4ADhbOH-XaD1UL4AP6DC5DoKGUjNfCt84-L7dqo92-AHgsiv4Lo8f5iu9nDC0_yJEuGgA2qnvkU
c3EST3y5eBzUuQR6TV56Mu83HxGV8T939BWUenV_5k4mcaPdSwb__jeO4wnsVywhrgQ
Host: www.googleapis.com
Accept-Encoding: gzip, deflate

Async:   TRUE
Connection Aborted:     FALSE
ConnectionGroupName:              51490320
Timeout:              -1/Infinite

    LocalAddress                   = 404715c0
    LocalAddressLength             = 16
        SOCKADDR @ 404715c0
            Family Type = 2 (IP)
            Port Number   = 51301
            Address   = 0.0.0.0
    RemoteAddress                  = 404715d0
    RemoteAddressLength            = 16
        SOCKADDR @ 404715d0
            Family Type = 2 (IP)
            Port Number   = 443
            Address   = 74.125.69.95

Another example:
URL:       https://accounts.google.com/o/oauth2/token
Verb:     POST
HTTP Header:
POST /o/oauth2/token HTTP/1.1
User-Agent: google-api-dotnet-client/1.9.0.26010 (gzip)
Content-Type: application/x-www-form-urlencoded
Host: accounts.google.com
Content-Length: 634

Original issue reported on code.google.com by anthony....@tdsb.on.ca on 8 Jan 2015 at 7:20

[GoogleCodeExporter]

Several things:

1. Can you attach also the code that creates the server?

2. We have a built in support for retries. Take a look in:
https://code.google.com/p/google-api-dotnet-client/source/browse/Src/GoogleApis.
Core/Apis/Http/ConfigurableMessageHandler.cs#178

3. In order that we will be able to further investigate this problem, can you 
create a console application and check if the problem occurs there?

4. Did you try to call service.Dispose() ? After finishing to use the service?
https://code.google.com/p/google-api-dotnet-client/source/browse/Src/GoogleApis/
Apis/Services/BaseClientService.cs#317

Thanks for reporting! Looking forward to hear form you.
Eyal

Original comment by pele...@google.com on 15 Jan 2015 at 6:03

• Changed state: Accepted

[GoogleCodeExporter]

Original comment by pele...@google.com on 15 Jan 2015 at 6:08

• Added labels: Priority-High, Component-Http
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Original comment by pele...@google.com on 27 Feb 2015 at 2:25

• Added labels: Priority-Medium
• Removed labels: Priority-High