I want to go and visit family but leave my wife and kids home.
Do you now want to take your wife and kids with you?
Well my parents would just much rather be alone with me for right now.
It's okay to take occasional trips by yourself, and to expect your wife to take care of the kids while you are away. However, if you are traveling for pleasure, you should at least ask if your family want to come with you.
Its hard to do things with your family, they always slow me down.
If you feel it is in the best interest of your parents for you to travel alone than so be it. However, if you are going alone because you think it would be more annoying to go with your family than you are doing it for the wrong reason.