sync.Map is causing confusion around garbage collection so just removing
the complexity all together.

Relates to pion/webrtc#1622