CI: Fix pathfinder CI infrastructure

[leofang]

Great question. I think you've discovered a flaw in the current CI design... we weren't prepared to always fetch the pathfinder wheel from the main branch. So right now with the backport branch we do have two different sources...

As a bandaid, I can manually backport all changes in the pathfinder from the main branch. Later we can improve the CI design to fetch pathfinder from the main branch in the 12.9.x CI (just like how we fetch 12.9.x wheels from the backport branch in the main CI, but in the reversed direction). WDYT?

Originally posted by @leofang in #806 (comment)

[leofang]

Closed by #904. Not sure why GH did not close it automatically.