We need wait_any/wait_all mechanism on subset of channels and condvars

[ochaton]

There are applications when we have subset of sources of data which triggers own condvars/channels. In global application architecture we have to wait for any events from subsets of these condvars/channels to pass data to the client. Now we may implement own tree of the condvars and channels based on fiber.cond() and fiber.channel() in Lua to achive required interface of communication. Maybe Core-based implementation will be faster

[Gerold103]

There are at least 2 approaches of how to implement such a feature. Poll-like and epoll-like.

Poll-like

It is similar to Linux system call poll(). Poll() in Linux takes an array of struct pollfd objects with file descriptors and returns those of them who has events (readable/writable).

Applied to fiber.cond in Tarantool it could look like this: fiber.cond_poll({cond1, cond2, cond3}, timeout). It would return when any cond becomes signaled, and returns the signaled ones.

Maybe it work work fine if cond count is tiny. But problems arise when the cond count becomes tens, hundreds, thousands, or more.

cond_poll() will need to scan the whole array to 'attach' the current fiber to each cond in the array. So as it could be woken up by cond.signal() to any of the conds. It will happen on each cond_poll() call, even if the array does not change at all.

Moreover, cond_poll() will need to add the current fiber to each cond into its list of waiters. And here is a second problem - rlist is an intrusive list. And there is just one link in struct fiber. It can't be added to multiple rlists at the same time.

Therefore, I would recommend to consider it a dead end. cond_poll() is simple, but it is shit not suitable for any performant application.

Epoll-like

It is similar to Linux system call epoll(). Epoll() in Linux is an object, a queue. Represented as a file descriptor. You can add other file descriptors to epoll, and when via epoll_wait() get all the signaled ones.

Strong side here is that you don't need to pass all the descriptors to epoll_wait(). You add them before hand one time. And epoll_wait() will only return the signaled ones. You don't even need to re-add them back.

Applied to fiber.cond in Tarantool it could look like a new thing called fiber.condmesh/condunion.

condmesh stores a list of fiber.cond objects. Each fiber.cond has a pointer at a condmesh, which can be NULL if the cond is not attached to any mesh.

When you signal a fiber.cond, it wakes all the waiter fibers + wakes its condmesh if it is not NULL. Also probably adds itself to a list of ready conds in the mesh.

Wake of a condmesh works exactly like a wake of a cond - it has a list of waiting fibers, which are woken up, each one.

To use the condmesh you need to create it using mesh = fiber.condmesh(). Then add conds to there using mesh.add(cond1) mesh.add(cond2) .... Then you make mesh.wait(timeout) and it returns either a list of conds or a number of conds. Or we can make mesh.add() take not only a cond but also an object returned from wait() when that cond is signaled. Like void * in C, but in Lua.

To remove a cond from the mesh you do mesh.del(cond1) mesh.del(cond2) ....

In that way it costs just + some small constant overhead for each cond, but makes mesh.wait() not depend on the number of conds in it.

I think this is the way to go.

Also this can be designed so as we could make nested meshes. Like nested epoll descriptors. Or even make each cond like a mesh. A tree of conds. This can be evolved from the design above.

[R-omk]

I suppose it is worth considering a more general construction that will work with channels by analogy as it is done with select in golang.

local select =  fiber.select{ch1, cond1, cond2}
select.add(ch2)
select.remove(cond2)

-- other fiber

local data, ch = select.pick(5) -- wait for 5 second
f2(data) -- any case
if ch == ch2 then  f1(data) end
if ch == nil then log("timeout") end
if ch == cond2  then  return end

But in this case, it need to determine how this function should behave
channel_object:has_readers()

[Gerold103]

I could use it in vshard. While working on tarantool/vshard#147 I realized I need to signal a lot of events to catch in a lot of places. The only 2 choices I have now - add a single cond which would be signaled on anything important (leading to tons of spurious wakeups of storage internal fibers), or implement my own cond-tree purely on fiber.sleep() + fiber.wakeup(). Would be better to have something builtin on the board.