Limit HTTP Connections

[rsrini83]

Hi,

My application is a self host microservice and purpose of this service is to upload multiples files to S3 instance. Each webrequest provides accessKey, secretKey and bucket information and contains 100 files to upload. Each webrequest is associated with an AmazoneS3Client instance, using this instance all 100 files will be pushed to S3 bucket. Looks like AmazonS3Client creating a HTTPConnection pool of 16 i think (Total virtual processors are 16) even though i may use 1 or 2. After a while server seems slow and i could see lot of connections are in TIME_WAIT ( checked using netstat -a).

Now the question here is, How can i restrict connection pool to 2 or 3 instead of 16.

P.S: When i tested with one single request, AmazonS3Client created 2 connections which i'm able to see the in Resource Monitor. So i presumed it creates one HttpWebRequest object per core. I might be wrong. But this is what i observed.

I need to resolve soon. Any help is appreciated. Let me know if any info is required.

Thanks in advance.

[randall-peakey-com]

Probably the most important thing to remember is to dispose of your instances.
If you do not, connections will hang around after they are disposed (or the application terminates).
Eventually you will hit a limit and won't be able to send any more requests.

In regards to using parallel (system.threading.tasks?), look into ParallelOptions (MaxDegreeOfParallelism)

[rsrini83]

@randall-peakey-com,

Thanks for your quick response. Yes i invoked dispose method after all files upload. and Yes i'm using MaxDegreeOfParallelism to 4.

Any idea by default how many connections get created ? Is there a way to restrict pool of connections to 2 or 3. I understand this will impact on the large files upload.

[randall-peakey-com]

Clarification requested....
Do you invoke dispose after all the files are uploaded, or do you dispose each instance as the individual files are uploaded?

I do not know how many will get created for each instance. Once I started disposing the instances after each use I didn't see any connections hanging around.

[rsrini83]

I dispose the object after all files are uploaded in the request(100 in my case). Application gets around 50 to 60 requests at a time, creating a thread pool is heavy weight so my process become slow with in a few minutes and sometimes it doesn't respond.

[PavelSafronov]

Can you post your code?

[rsrini83]

Here is the code,

public async Task<TransporterResponse> Upload(BackupRequest request, AWSStorageConfig storageConfig)
{
    var uploadFiles = request.UploadFiles; // Gets all 100 Files
    try
    {
        if (request.Location != null && request.Location.Length > 0 && !request.Location.EndsWith("/"))
        {
            request.Location += "/";
        }
        AWSCredentials awsCredentials = new Amazon.Runtime.BasicAWSCredentials(storageConfig.AccessKey, storageConfig.SecretKey);
        AmazonS3Config s3Config = new AmazonS3Config
        {
            Timeout = TimeSpan.FromSeconds(1000),            // Default value is 100 seconds
            ReadWriteTimeout = TimeSpan.FromSeconds(3000),   // Default value is 300 seconds
            MaxErrorRetry = 2,                              // Default value is 4 retries
            RegionEndpoint = GetRegion(storageConfig.Region),
            UseHttp = true,
            MaxIdleTime = 5,
            ConnectionLimit = 5,
            ProxyHost = null
        };
        List<Exception> uploadExceptions = new List<Exception>();
        AmazonS3Client s3Client = new AmazonS3Client(awsCredentials, s3Config);
        // With Parallel
        await uploadFiles.ParallelForEachAsync(async file =>
        {
            try
            {
                await GetUploadTask(request, storageConfig, s3Client, s3Config, file);
            }
            catch (Exception e)
            {
                uploadExceptions.Add(e);
            }
        }, maxDegreeOfParalellism: 5);
        s3Client.Dispose();
        if (uploadExceptions.Count != 0)
        {
            throw new AggregateException(uploadExceptions);
        }
    }
    catch (AmazonS3Exception s3Exception)
    {
        Logger.Error("[backup] - Completed with error");
        Logger.Debug(s3Exception.Message, s3Exception);
        if (s3Exception.ErrorCode == "NoSuchBucket")
        {
            // Create Bucket and retry if it is cloudcontrols3
        }
        return new TransporterResponse()
        {
            Status = "ERROR",
            ErrorCode = "ERR-S3-UPLOAD-FAILED",
            ErrorMessage = s3Exception.Message
        };
    } catch(Exception e)
    {
        Logger.Error("[backup] - Completed with error");
        Logger.Debug(e.Message, e);
        if (e is AggregateException)
        {
            throw e;
        } else
        {
            return new TransporterResponse()
            {
                Status = "ERROR",
                ErrorCode = "ERR-UPLOAD-FAILED",
                ErrorMessage = e.Message
            };
        }
    }
    Logger.Debug("[backup] - Backedup to s3 bucket");
    return new TransporterResponse()
    {
        Status = "SUCCESS"
    };
}

private async Task GetUploadTask(BackupRequest request, AWSStorageConfig storageConfig, AmazonS3Client s3Client, AmazonS3Config  s3Config, HttpFile file)
{
    String fileName = file.Name.Split(new char[] { ';' })[0];
    String key = request.Location + fileName;
    S3FileInfo info = new S3FileInfo(s3Client, storageConfig.Bucket, key);
    if(info.Exists)
    {
        // Throws 303 - Resource exist already.
        throw new GeneralServiceErrorException(Infrastructure.Errors.Enums.ServiceErrorEnum.DuplicateItem, "AWS Key exist already - " + key);
    }
    TransferUtility fileTransferUtility = new TransferUtility(s3Client);
    Stream iStream = file.Value; // This is the Stream.
    TransferUtilityUploadRequest fileTransferUtilityRequest = new TransferUtilityUploadRequest
    {
        BucketName = storageConfig.Bucket,
        StorageClass = S3StorageClass.Standard,
        PartSize = 6291456, // 6 MB.
        Key = key,
        InputStream = iStream, //stream,
        CannedACL = S3CannedACL.Private,
    };
    Logger.Debug("[GetUploadTask] - Item Content Length- " + iStream.Length);
    await fileTransferUtility.UploadAsync(fileTransferUtilityRequest);
    Logger.Debug("[GetUploadTask] - Total Time s3 Upload - " + s3UploadWatch.Elapsed.TotalMilliseconds);
}

[randall-peakey-com]

Have you tried to create and dispose the AmazonS3Client object inside GetUploadTask?

[rsrini83]

Initially i was doing that, upon a google search found one of link(github issue link) that suggested to reuse the client object across multiple threads instead of creating. Updated code to reuse.

[randall-peakey-com]

I'd also suggest disposing the fileTransferUtility when finished.

[rsrini83]

Oh okay. Will try that now.

[rsrini83]

Modified the code and tried now, you can see lot of connections are in TIME_WAIT state. My current test is with 20 requests in parallel, each request contains 100 files.

[randall-peakey-com]

Did you also switch to creating and disposing the AmazonS3Client object inside GetUploadTask?
Also as a test, if you attempt only 100 files, how many connections do you see in the time_wait state when it is finished?

[rsrini83]

No. Will try that tomorrow and update you. I need to go to bed now as it became very late (3 AM)...
Thanks for your time.