Poor performance after harvesters use up available ore/gems

[beercanchicken]

DESCRIPTION
In certain situations an AI can create a higher demand for ore than what the map produces. This is apparent when using a map that has isolated base areas. When the AI has multiple harvesters and runs the available ore down to the immediately generated supply (and the AI cannot reach more ore/gems), momentary but impactful performance lag spikes occur.

STEPS TO REPRODUCE THE ISSUE
Use this map (Big A Lake) for a demonstration:
https://resource.openra.net/maps/36796/

• Begin a skirmish game on this map with multiple AIs
• Play until one of the AI players runs its gem supply down to the mines
• The game should begin to show spikes in performance lag
• Destroy the ore trucks and the lag goes away; allow more ore trucks to chase a desperate supply of ore and the problem gets worse

EXPECTED BEHAVIOR
AI ore harvesters act normally, however there is a performance impact when many units are chasing a sparse ore supply.

edit: tested on pchote's 20201207-devtest build

[dnqbob]

I guess it is the harvester bot module, and I got a similar bug in CA long time ago.
Inq8/CAmod#23

Although they used their own harvester bot module but it also happened in normal harvester bot module.

[Mailaender]

Yes, it is doing a costly pathfinder check every 50 ticks by default. See also #9367.

[beercanchicken]

The lag spikes seem to coincide with these clusters of Invalid resource search locations.

[deleted-user-1]

This is apparent when using a map that has isolated base areas.

Sounds to me like #12435, potentially happening when the harvesters try path to ore on the other isolated part of the map and then it stops when ore on the accessible part has regrown.

Potentially unrelated to harvester code.

[Mailaender]

The harvester has a domain index check so it won't waste too much effort on impossible paths.

[anvilvapre]

I believe it does 2 path findings. First one to find ore/gems around it, then once found it will do another path find to plan a path to it.

[Mailaender]

As it says

Ordering to invalid

I guess in that case it is only doing one

OpenRA/OpenRA.Mods.Common/Traits/BotModules/HarvesterBotModule.cs

Lines 147 to 157 in 6810469

	Func<CPos, bool> isValidResource = cell =>
	domainIndex.IsPassable(actor.Location, cell, harv.Locomotor) &&
	harv.Harvester.CanHarvestCell(actor, cell) &&
	claimLayer.CanClaimCell(actor, cell);
	var path = pathfinder.FindPath(
	PathSearch.Search(world, harv.Locomotor, actor, BlockedByActor.Stationary, isValidResource)
	.WithCustomCost(loc => world.FindActorsInCircle(world.Map.CenterOfCell(loc), Info.HarvesterEnemyAvoidanceRadius)
	.Where(u => !u.IsDead && actor.Owner.RelationshipWith(u.Owner) == PlayerRelationship.Enemy)
	.Sum(u => Math.Max(WDist.Zero.Length, Info.HarvesterEnemyAvoidanceRadius.Length - (world.Map.CenterOfCell(loc) - u.CenterPosition).Length)))
	.FromPoint(actor.Location));

which seems to be very costly despite the domain index check.

Maybe we should do a check if the resource layer in that specific domain index is empty and then stop searches.

[anvilvapre]

#19532 optimizes this very slightly.

[reaperrr]

What I learned is that the enemy avoidance check in the harvester bot module increases the cost by a factor of roughly about 5(!).

With a performance implication that massive, I'm not convinced the enemy avoidance logic is worth it, and I don't see an easy way to significantly reduce its cost (reducing the scan radius will also reduce its effectiveness).

[pchote]

That code calls world.FindActorsInCircle for every cell in the pathfind - oof!

This and several other AI related features could be improved/optimized by having a player-level "enemy danger heat map" which uses something like the support power targeting logic to calculate the danger in coarse bins across the map so that harvesters / engineers / MCVs can avoid them, or squads can attack them.