I don't want my nephew to wear a flower girl dress to my wedding
What do you want your nephew to wear at your wedding?
I want him to wear a suit and I don't want him to become like his mom
It's okay to stand up for what you feel is right at your wedding. You should let your nephew's parents know how you feel and what you expect. Speaking up for yourself will ensure that you are satisfied with your wedding.
I don't want to cause a scene. There are other ways to deal with the matter.
It is normal to have a good formal dress code at a wedding. Maybe he just likes dressing up and would feel more comfortable in the dress though.