Add missing kAllGatherStart/kAllGatherDone verification to HLO verifier

[kredd2506]

Summary

Fixes #40191

VerifyAsynchronousInstructionPairs checks that async Start/Done instructions
are properly connected for all async op types except kAllGatherStart/kAllGatherDone.
This adds the missing verification, following the same pattern used for kAllReduceStart/kAllReduceDone.

Changes

• hlo_verifier.cc: Add kAllGatherStart and kAllGatherDone cases to the async pair verification switch
• hlo_verifier_test.cc: Add tests for valid AllGather Start/Done pair and invalid multiple-Done case

Test plan

• CI verification
• Tests follow existing AllReduce verifier test patterns
• Follow Google C++ style

[akuegel]

I think here you can still use ParseAndReturnVerifiedModule?

[kredd2506]

Good point, updated to use ParseAndReturnVerifiedModule for the valid case. I followed the existing AllReduceStartAndDone test which uses ParseAndReturnUnverifiedModule, but the verified version is cleaner.

[akuegel]

Thank you :)

[akuegel]

Just FYI, your new check exposed a test that had invalid HLO. I am fixing that and then your PR will get merged.

[kredd2506]

Thanks for fixing that! When I first saw the failing test I was reviewing my own logic thinking I'd broken something — but your fix makes sense. The verifier was assuming shapes were arrays/tuples without actually checking, and the nested tuple structure for AllGatherStart adds extra complexity.

Nice catch extending the same fix to ReduceScatter too. Glad the async pair check surfaced these.

Quick question — were these crashes exposed by an existing test that had invalid HLO, or was it previously valid HLO that just wasn't going through this verification path before?

[akuegel]

Quick question — were these crashes exposed by an existing test that had invalid HLO, or was it previously valid HLO that just wasn't going through this verification path before?

The crashes were exposed when I tried to adjust support_test.cc but got it wrong initially (creating invalid HLO that crashed instead of being rejected as invalid). After I fixed it for AllGather/AllGatherStart, I asked the KI to check whether there are more such bugs in this file, and it found ReduceScatter and fixed it for me. Only the test it created did not immediately have valid HLO, so I had to adapt it manually.