fixes #1827 Windows a deadlock on nng_close()

[gdamore]

My current theory is that for some reason that I don't yet fully understand, we have code waiting in the condition that didn't set the closing. (Possibly the failure is a synchronization since s_closing is changed while not protected by the global lock.)

At any rate, the attempt to avoid the cost of a wake up here is silly, as pthread_cond_broadcast (and one assumes other variants like the Windows implementation to which I don't have source) are nearly free when there are no waiters. (Pthreads uses a relaxed order memory read to look for waiters, so no barrier is involved.)

So we can just do the wake unconditionally.

I'd appreciate it if folks who are encountering the problem can tell me if this change resolves for them.

[codecov]

Codecov Report

Attention: Patch coverage is 81.81818% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 79.38%. Comparing base (aac4dc3) to head (1ab81ce).

Files	Patch %	Lines
src/sp/transport/tcp/tcp.c	0.00%	1 Missing ⚠️
src/sp/transport/tls/tls.c	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1828      +/-   ##
==========================================
+ Coverage   79.32%   79.38%   +0.05%     
==========================================
  Files          95       95              
  Lines       21491    21484       -7     
==========================================
+ Hits        17048    17054       +6     
+ Misses       4443     4430      -13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[alzix]

the issue is still reproducible on this branch.

[gdamore]

Please have another go with this branch -- another commit made which I hope will help.

[gdamore]

Well, that didn't work as well as hoped. Seems that the read/write cbs are also here.

[gdamore]

Ah reaping is needed because we are in the callback when we fail. And its interesting that this happens consistently for IPC, so that suggests that I'm on the right path.

[gdamore]

(Another go, restoring the reaping..)

[alzix]

i was not able to reproduce the original issue anymore, but I cannot get a decent number of iterations as the server is crashing on nni_list_node_remove as was previously reported

[alzix]

win_ipcconn.c:229 in ipc_send_cb there is a check:

if ((aio = nni_list_first(&c->send_aios)) == NULL) {
	// Should indicate that it was closed.
	nni_mtx_unlock(&c->mtx);
	return;
}

I think it does not do what is expected as I see in the debugger that c->closed == true

[alzix]

there are two types of crashes here: one in ipc_send_cb and the other in ipc_recv_cb. Both occur on close.
Based on my observation in these cases the aio object is malformed, and later leads to a crash.

either memory was not properly initialized or some other thread overwrote it.

from: https://en.wikipedia.org/wiki/Magic_number_(programming)#Debug_values

0xDDDDDDDD pattern is used by Microsoft's C/C++ debug free() function to mark freed heap memory

so it seems the aio contains dangling pointers...

[alzix]

from my observations, the problem occurs when ipc_recv_cb and/or ipc_send_cb are executed after nni_sock_shutdown

[gdamore]

@alzix thanks for the analysis. I will try to get to the bottom of this soon ... I've just been completely swamped with $dayjob.

[gdamore]

Definitely a use-after-free.

[gdamore]

This is very definitely windows specific. It may impact TCP as well, but the callback structure here is used with overlapped IO (a Windows thing.)

[gdamore]

So I guess the send_cb is somehow still running. I'm still trying to get to the bottom of this, because I would not expect that there are any posted I/Os at that point.

[itayzafrir]

added some info in PR #1831 (comment)

[alzix]

So I guess the send_cb is somehow still running. I'm still trying to get to the bottom of this, because I would not expect that there are any posted I/Os at that point.

According to https://learn.microsoft.com/en-us/windows/win32/fileio/canceling-pending-i-o-operations

There is no guarantee that underlying drivers correctly support cancellation.

perhaps this is the case?

[gdamore]

So I guess the send_cb is somehow still running. I'm still trying to get to the bottom of this, because I would not expect that there are any posted I/Os at that point.

According to https://learn.microsoft.com/en-us/windows/win32/fileio/canceling-pending-i-o-operations

There is no guarantee that underlying drivers correctly support cancellation.

perhaps this is the case?

Then the driver should continue to completion which would be fine. But Windows named pipes and TCP both support cancellation. The problem is a defect in my logic, not missing Windows functionality. I'm still working to get to the bottom of it -- I thought I had understood it but clearly I was missing something.