The Remote Host Closed the Connection Exception

[mikedotmundy]

Please go through all the tasks below

• Check this box only after you have successfully completed both the above tasks

Please provide a brief description of the problem. Please do not forget to attach the relevant screenshots from your side.

This is possibly related to #4652, but the exception/stack trace is different, so I am opening a new bug. This seems to be some issue with the GetAvatar feature introduced in v15.0.

Since updating to v15.1, (possibly in v15.0 also) we have been regularly seeing the following exception:

Stack Trace:

at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
   at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
   at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
   at RockWeb.GetAvatar.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Expected Behavior

No exception.

Actual Behavior

Exception is logged.

Steps to Reproduce

See above.

I have been unable to figure out exactly what page(s) or action(s) are causing these exceptions to happen, however, it is happening for people who do not have access to our internal Rock pages, which means most likely, it is happening when users are logged into their external account and/or associated pages.

Rock Version

v15.1

Client Culture Setting

en-US

[dataCollegechurch]

This is occurring on our end as well. I am seeing it for user who do have access to our internal rock pages. Here is one such request:

[jonedmiston]

The 0x80070057 error code means that Windows ran into a storage issue. Can you look at your system volume and report back:

1. How large is your system volume.
2. Storage size of the ~\App_Data\Avatar\Cache directory.

Looking at the caching strategy there are some optimizations that could be made to reduce the number of files produced.

[dataCollegechurch]

2.
I don't see a Cache Folder for that path:

[jonedmiston]

Other recommendations from Google:

1. It might seem silly, but there are many issues that can be caused by your system's date and time not syncing up properly with the real world, and error code 0x80070057 is one of them.
2. Run Chkdsk. If the underlying storage hardware you're running has some corrupted data, a damaged partition, or defective hardware, the Windows error-checking tool or the Chkdsk command can help find and fix them.

From: https://www.lifewire.com/fix-error-code-0x80070057-4691502

[dataCollegechurch]

The time on my Azure VM currently looks correct.
I ran chkdsk and will report back tomorrow if the issues continues.

[mikedotmundy]

1. 52 out of 126 GB Free space
2. I also do not have a ~\App_Data\Avatar\Cache directory, however my ~\App_Data\Cache directory is 91 MB.
3. My Azure VM is displaying the correct time.
4. Windows error-checking tool did not return any issues on my HD.

[dataCollegechurch]

Quick update: Exceptions continue to occur after chkdsk was run

[zackdutra]

We're experiencing the same issue with GetAvatar.ashx, exactly as shown above

[robotman3000]

@dataCollegechurch @mikedotmundy What version of IIS are you using? Is it the same for both of you? I noticed that both your stack traces have IIS7WorkerRequest. "IIS7" seems to suggest a very old version of IIS

[jonedmiston]

Can you all check your IIS version? If it is IIS7 Microsoft ended support of that in Jan 2020.

It sounds like the cached images are not being written (hence the lack of the ~\App_Data\Avatar\Cache). If that's the case can you check that the AppPool's Identity is running as LocalSystem (https://community.rockrms.com/documentation/bookcontent/1#configuringiis) and that LocalSystem would have permissions to create the folder (~\App_Data\Avatar\Cache).

[zackdutra]

Our stack trace also shows the IIS7 message, but we're running version 10

[mikedotmundy]

1. We are running IIS10.
2. Our AppPool Identity is running as LocalSystem (and is configured the same as the rest of the settings in the documentation).
3. I'm not sure how to check the permissions for LocalSystem.

[dataCollegechurch]

@mikedotmundy - I think you can look at the Security Permissions for the folder in question.

We are also running IIS10 and our App Pool is running as LocalSystem.

I believe LocalSystem shows up as SYSTEM most places. It does have access to create new folders in our case.

[mikedotmundy]

Ok thank you @dataCollegechurch - I wondered if that was it - my permissions look the same.

[dataCollegechurch]

This is interesting. I manually added the ~\App_Data\Avatar\Cache folder and the system did start adding files into it. Even so, now I am getting this flavor of exception messages:

[JimMichael]

Coming here just to report that I'm seeing this on v16.0 alpha (1.16.0.10). The exceptions happen when I visit a person profile (and sometimes the internal homepage) and the URLs in the exceptions point at valid profile images or images with the initials of the person, but not every time, or for every person.

I'll note that our Person Image file type is stored on Cloudinary, not on local storage or the db... in case that could be part of the issue. Like others here, we have no /App_Data/Avatar/Cache directory. Only Fonts and Masks are directories there.

Running on Win 2022/IIS10 here.

[mikedotmundy]

We are also continuing to see this on Alpha v1.16.0.10.

[dataCollegechurch]

We are also using cloudinary, @mikedotmundy @zackdutra , where are you storing your people images?

[mikedotmundy]

Ours are database

[zackdutra]

We switched to Azure blob storage, but some are still in database

[nairdo]

Greetings everyone. We've done some extensive research on this one and the advisement is:

• A) That exception is normal if/when a client browser drops a connection to the server. That can happen for a number of reasons.
• B) It's not generally advised to always ignore that exception unless you've considered the specific instance/case.
• C) Therefore, we're going to surgically handle (ignore) this particular exception (for the GetAvatar) call.

That means it's still possible there could be other places in Rock where a "The Remote Host Closed the Connection" exception occurs in the future. So, if it does happen, please feel free to report it with the stack trace and we'll consider handling/ignoring that case. Ultimately we could decide to ignore those exceptions system wide, but we'll get there if we get there.

[mikedotmundy]

Thanks @nairdo. Will this be included in v16.0?

Also, what about #4652?

[dataCollegechurch]

Should we open a separate issue to research why the ~\App_Data\Avatar\Cache folder is missing on some instances?

[dataCollegechurch]

Anecdotally it appears that this issue is occurring for some churches but not others. If this exception is driven by the client browser dropping the connection, I would expect for this error to be occurring for every church. Personally I would not be comfortable ignoring this error until we could determine either that it is occurring for all churches or why it is only occurring for some.