Not setting one of them in a SetTimeouts requests should not remove it.
This fixes a panicking `unwrap()`.