I don't want to be on a vacation with my family
Why do you want to be on a vacation?
I got separated from my family when I was 2 years old and we only recently started talking.
I'm sure it's hard for you given that you don't know them very well, but it would be nice for you to get to know your family. If spending that much time with them seems overwhelming, maybe suggest a shorter activity.
Nah, I've decided to live with my Dad and the vacations will make me think of the family I lost.
Maybe you should just go for a day or so. It seems rude to not want to spend time with your family, even for a little bit.