I want to ask my sister to dress down a bit, and to look more homely at my wedding
I have a feeling that isn't going to end well. What do you mean by "more homely"?
I do not think she should dress up too much, and she should wear a simple dress at the wedding
It's okay to want to be the center of attention at your own wedding, and to not want guests or family members to catch others' eyes. However, you should not be too impolite to your sister. She puts a lot of thought and effort into planning how to dress, so you shouldn't make her feel bad by telling her that she is dressed inappropriately.
I had to tell her that since I'm the one paying for the wedding.
Maybe you should just let this one go. She may get pretty offended if you try to control what she wears. You don't want her acting out a grudge on your big day do you?