It looks like there is finally a constant time big integer library for Go that does everything we need: https://github.com/cronokirby/saferith
It still comes with warnings about being in Beta, but even if it isn't perfect, it will be better than the library we are using now, which makes no attempt to defend against side channel attacks. From the paper describing saferith, it seems that the slowdown in the SRP computations would be on the order of 3x. That will be dwarfed by our use of PBKDF2. That is, using this slower computation than what an attacker would have to use gives the attacker only a really small advantage.