This change caused a ~24% performance decrease

Relates to pion/webrtc#1564

This reverts commit e839a2af7fa0dafc94fe3f5cf5bf6e3b56ad6cc9.