Optimize Pipeline.run()

[silvanocerza]

The current logic of Pipeline.run() was kinda rewritten in a haste so we cut some corners and didn't optimize it.

The code is also not well documented making it harder to understand and reason about for other devs.

Ideally this should be done after #7611 to prevent the introduction of any bugs.

This will also make it easier to add new functionalities in the future.