multi_wait improvements

[icing]

In multi_wait(), which is called all the time, multi_getsock() was called twice.

The first invocation was used to count the number of sockets, a sufficiently large struct pollfd array was allocated, and then the second invocation was used to populate that array. This is not only an overhead, but it assumes that the second call will deliver the same number of sockets as the first. Which is not really guaranteed. Also, when parallel transfers on the same connection are active, the same socket is collected for every transfer.

• only call multi_getsock() once for all transfers. Grow the struct pollfd array when needed.
• fold repeated sockets. This is done cheaply by just comparing the current socket to the last one. It will not catch all duplicates on multiple connections.

[icing]

@jay: I'd like your review on this because it touches on the Window parts which I tried to preserve to the best of my understanding.

[jay]

@mback2k is WSAEventSelect supposed to be called even when mask is 0?

[icing]

I think mask == 0 is used to remove a socket again. See line 1475 after the poll.

The check I added is for the case where a socket is returned, but neither POLLIN, nor POLLOUT is set. Should not happen, but needs to be handled. Calling WSAEventSelect() with 0 would then remove a socket and that is not correct in that situation. The socket might have been added before by another transfer.

[icing]

Thinking about this, I am not even sure this works correctly as is for parallel transfers. Think about this case: transfers A and B on the same connection socket X.

• A wants to POLLIN on X
• B wants to POLLOUT on X

Does calling WSAEventSelect() on X not override the previous mask?

[jay]

i recall there was some issue with wsa events, where marc had added it to purposely reset the flags.

Does calling WSAEventSelect() on X not override the previous mask?

it does override so if you want to wait on read and write you have to do it together. i hope marc will have time to review this pr, he has been busy so give it a few days

fwiw i used a build of this pr yesterday with many parallel http/2 httpbin drip replies and didn't see any difference in behavior

[mback2k]

@jay you are right regarding the resetting of flags needing to happen, but this comes later in the existing code:

curl/lib/multi.c

Line 1428 in a41cd15

WSAEventSelect(s, multi->wsa_event, 0);

That is why a little bit above that line I left this comment:

curl/lib/multi.c

Lines 1405 to 1407 in a41cd15

	/* With WinSock, we have to run the following section unconditionally
	to call WSAEventSelect(fd, event, 0) on all the sockets */
	{

Does calling WSAEventSelect() on X not override the previous mask?

Yes, it will overwrite the previous mask for the same multi->wsa_event. Anyway, I think your new check referenced by this review thread is fine, as mask == 0 should not happen at this point.

[mback2k]

Thinking about this, I am not even sure this works correctly as is for parallel transfers.

I am not into the multi-handle details here, would each transfer have its own multi-handle and therefore own multi->wsa_event? If not, you are probably right and the WSA event will need to be changed to be per-transfer if multi_wait is called individually per-transfer.

[icing]

There can be many transfers on the same multihandle and many transfers on the same connection (HTTP/2, HTTP/3). These report the connection socket with their individual needs.

When transfer A and B are on the same connection, A might request POLLIN and B POLLOUT. Without coalescing the two, the masks will overwrite each other.

[mback2k]

The main question is if multi_wait will be executed in parallel for these transfers on one multi handle. Also, I am no sure what will happen if the WSA event is changed while already being waited on in another thread for example. Maybe that is undefined behavior or not supported after all?

[bagder]

if multi_wait will be executed in parallel for these transfers on one multi handle

It cannot be executed in parallel for the same handle, no.

[mback2k]

Sorry, I won't be able to look at this before the weekend.

[bagder]

@mback2k @jay what's your take on this now? It would be neat to merge, but I don't sense that we have your 👍 on it just yet?

[calvin2021y]

this weekend I try test this patch, find it cannot merge into master.