Fix removeDelegateListener from throwing

This prevents removeDelegateListener from throwing when called without having previously called addDelegateListener.