I'm not going to cancel our vacation plans because my wife is pregnant
Is that safe? How do you think she'll react to that?
Her feelings might get hurt but that's too bad
If she still wants to go then you two should go on the vacation. Just because she's pregnant doesn't automatically mean you need to cancel your vacation plans. Go out and have fun!
We are not going on vacation then. I don't want to be left with these thoughts of what if.
You should talk with her about it. You don't need to be afraid of going on vacation you might just need to change activities a bit. You can get a lot of accommodation to make sure you both have a good time.