The idea of the to was indeed to prevent overflow during the conversion, but due to the way unsigned numbers need to be stored in BSON, I think a blend between how it was and how you changed it would be correct:
else static if (is(T == uint)) return cast(uint)m_inputData.get!int();
else static if (is(T : int)) return m_inputData.get!int().to!T;
So for smaller types short/ushort and the byte sized types it would still use to to prevent overflow, but for uint it would accept overflow on purpose. Anything that wouldn't be caught by that?