Matcens sometimes stop working too early in co-op

[ghost]

It was brought to my attention that this hadn't been reported (thanks afpilot).
There is a seemingly random occurrence seen frequently in co-op games where all robot makers/matcens in a level will stop working. When this happens, all currently active matcens immediately stop producing anything, and matcen triggers no longer cause robots to be produced, regardless of whether the matcen the trigger pointed to has been exhausted or not.
The break typically happens while a matcen is in its production cycle; it will seemingly try to spawn a robot and may light up, but no robot is produced.

I get the feeling a race condition may be involved, possibly related to network state sync; I've seen it frequently in co-op, but never in single-player. It's also a fairly old bug; I don't remember whether it was observed in 0.57 (it may have been) but I'm pretty sure it was in 0.58.

[vLKp]

This sounds rather like #132, which not only was reported, but was reported by AFPilot. ;) Even so, it's good to get a reminder report on it. That ticket has been idle since 2015 waiting for further information on how to reproduce the problem.

I don't recall seeing this in 0.57.3, but I might have missed it. I think I only ran the multiplayer cooperative campaigns once in 0.57.3.

You say it's frequent in co-op games. Do you have a sense for how frequent? Can I count on it happening every time, only every other time, etc.?

You speculated on a network-related race condition. Is it accurate to say then that hosting a cooperative game that no one joins is insufficient; that instead, the game must be cooperative and guests must join? If so, do you have any sense whether the guests need to do anything? This would be easier to investigate if I can just join some dummy guests and leave them sitting in the start area. If I need real people driving each of the guest Pyros around the level, this will be more challenging to debug.

[ghost]

It seems to happen in the ballpark of 50% of the time per level. That's just a rough guess though. It's not guaranteed to happen but happens often enough that going through a level set we generally will have seen it several times.

The reason I was guessing network-related is that it doesn't happen in single-player... though it's possible there's some other difference between that and co-op. Other co-op bugs we've seen in past (including in vanilla D2) have come from things like dropped packets or contention between clients (thiefbot jitter) which makes it somewhat suspicious. I'm unclear on whether the other players need to do anything, though. That would need experimentation, though I should be able to get volunteers for the purpose. I also haven't proven whether other people need to join the game, so that's worth looking at.

[vLKp]

Volunteer testers could be helpful, thanks. If possible, I'd like to have your test runs use a modified build with debug logging for matcen operation. I'll try to write up the logging patches today.

[vLKp]

I pushed Log various matcen logic to debug/matcen-386 with debugging patches. Please get as many of your volunteers as possible to use this branch, rather than master. After you see a problem, once you exit that game, grab the gamelog.txt and attach it. Anything you can record about the time of day when you saw the problem will help me match the log to the failure.

[vLKp]

Based on what I noticed while writing the logging patch, I did some investigation of how the accounting works. Currently, it looks like multiplayer games will be reliably broken. It won't fail for you 50% of the time. It will always fail eventually. If I am right, the intermittent results are related to whether you trigger the matcen enough times to hit the faulty cap. If you do not keep reactivating it, you may not build up enough robots for it to break. The test on line 393 is why the matcens stop producing, but that is not the bug.

dxx-rebirth/similar/main/fuelcen.cpp

Lines 392 to 396 in 53a1d5f

	auto &plr = get_local_player();
	if ((plr.num_robots_level - plr.num_kills_level) >= (Gamesave_num_org_robots + Num_extry_robots))
	{
	return;
	}

This test is meant to prevent generating too many robots if the player is not destroying them in a timely manner. However, player::num_kills_level is not updated in multiplayer. It is only incremented in apply_damage_to_robot. If in multiplayer, then the increment is skipped and instead multiplayer messages are sent. However, none of those messages results in incrementing this counter, so it is stuck at 0 in multiplayer. That means that eventually, the test for "excess" robots will decide that there are too many, even if you have destroyed every robot the matcen created. Once that happens, no more can be produced.

[ghost]

Ouch.

Do you still want the testing done? I've been a little preoccupied this weekend but if you'd still find the traces useful I can still get some folks together.

[vLKp]

Yes, but include in your test the commits I just pushed (Update num_kills_level/num_kills_total in multiplayer, Update num_kills_level/num_kills_total for remote kills). The first commit seems to fix the problem for me in the five minutes I spent on Descent: First Strike level 6 at Insane level running a "cooperative" game with no one else present. The second should address multiplayer, but is only compile-tested.

If I am right, this should fully resolve the problem.

[vLKp]

I believe this is resolved by the change above. No response in roughly two weeks. Closing.

@Sirius-TR, if the problem persists, please reopen.