In the past, I have wished that my ex's son wasn't his.
Why do you think you wish that?
Because he doesn't look like his father at all.
You and your ex's child doesn't deserve this kind of drama and thought process, because no one chooses your parents. I think it's best to just love the child as is, and to be amicable to your ex for the sake of your child. I think they would be grateful for that.
Why do you even care why I am thinking that? I don't care what you think.
Well, you brought it up and it seems like it might be something that's bothering you. You can't change who fathered your son, it's something you just have to accept and live with. Ruminating about it will only cause problems in your family.