Logistical transporters cause massive lag

[mazunki]

Issue description

I have seen issues in the past concerning the lag of pipes, but they're all closed. I believe there still is a need for a refactor of how pipes are pathfinding, seeing as, from how i'm reading LogisticalTransporterBase.onUpdateServer, each pipe unit is trying to find a valid direction to walk in. This is awful for performance, especially when this is done per tick.

Valid paths between nodes on the network should be calculated upon a change of the whole network, and cached in the network. The network should be responsible for sending items between the nodes, since it knows what paths to combine. If any block is changed adjacent to the network, of course, the whole network must be recalculated: but this scales much better than attempting to do this per server tick.

Furthermore, something should be done to handle congestion on a network. You would think ultimate pipes are actually able to handle sending several stacks of items per operation. Pipes should not accept items if congestion is too high on the network: see https://dl.acm.org/doi/pdf/10.1145/52325.52356 for an example implementation to avoid congestion dynamically, or just hardcap it (this would be detrimental to parallel routes on the same network).

Steps to reproduce

Initially, I had a filled a large field of hopper botany pots, and used Ultimate Logistical Transporters to pipe their output into a drawer. The lag was unbearable. I then decided to streamline all the connectors so there would be no circular pathfinding issues, which didn't seem to help either. I then set the direction of each row of hoppers to only push into the main bus at the end of the row, which didn't help either (related: there's no visual cue to highlight direction of allowed traffic, and the configurator is annoying to use: ideally you'd be able to set its mode to "connect/disconnect pipes", "change colour", "change push/pull direction" to facilitate doing this at large).

I can see my items flickering back and forth on the transparency of the block, which isn't causing any clientside fps lag, but the server's TPS is drowned. Is there anything I can do to make these pipes actually usable?

The alternative now is to "just use another mod": LaserIO, Modular Routers, Ender Chests, EnderIO, PrettyPipes, Integrated Dynamics, Integrated Tunnels; but this doesn't fix the issues in this mod.

Versions

I am playing on ATM9 0.2.39.
Minecraft 1.20.1
Forge 1.20.1-47.2.19
Mekanism 1.20.1-10.4.5.19

[thiakil]

Please note that due to your set up,

• each insertion to the pipe network needs to check every other botany pot to see if it will accept the item.
• last we knew, the pots will try to export every tick

I would recommend setting the pipes connected to the pots to be Pull mode.

NB: setting push/pull mode on a connection to another pipe has no effect (hence why there's no visual indicator)

[mazunki]

last we knew, the pots will try to export every tick

I believe that to be true too, but the pipes should only accept items if there's actually room for the items on the network. I have a feeling that's the main cause for lag at the moment. This combined with the long time to find a destination isn't good.

NB: setting push/pull mode on a connection to another pipe has no effect (hence why there's no visual indicator)

I see, I didn't know that. I feel this should be changed, since now you need a storage buffer inbetween the bus and the rows to have some control flow. I can't think of a reason to argue for its direction being ignored. As you mention, this does make every botany pot search to push into all the other botany pots.

I would recommend setting the pipes connected to the pots to be Pull mode.

I thought about it, and perhaps it would help a bit. It would also be a lot of work to do when done at scale. I might test it and report back, but I don't think it would change too much.

[thiakil]

I believe that to be true too, but the pipes should only accept items if there's actually room for the items on the network

That's exactly what it's doing though. You just have a large number of botany pots it has to check too. They'll send to the closest destination - pretty sure they don't scan the whole network just to insert to the first one they found

[mazunki]

Huh, that seems weird. I saw a bunch of items stuck in traffic, flickering back because they couldn't reach the destination. Over time, the amount of items on the network grew, steadily increasing congestion. Could there be another cause for this?

[thiakil]

Yes, that can happen when a destination says it can accept the item at simulation time, but rejects it when it actually gets there

[mazunki]

That shouldn't be the case here, I was using a Functional Storage drawer with plenty of space in it. Can the pipes themselves reject items in transit? Looking at the direction of the items being moved, it seems they were all flowing towards the drawers, and not back towards any of the hopper pots on the network.

[thiakil]

The individual pipes can't reject items no, they don't have an inventory at all

[pupnewfster]

Yeah... While some of it might be alleviated by #7748 if/when I get around to finishing it, the main issue here is that transporters are not that great when something is force inserting (especially every tick like the pots do) compared to when they pull things themselves... Which is one of the reasons why for a long time we didn't expose an item handler to them. Setting them all to pull should help at least slightly as then it won't have to check the other pots as potential valid destinations