Is it okay if I tell my muslim classmate she looked better without her hijab on?
Have you ever said that before?
I've told her repeatedly on different occasions but she didn't take me seriously.
While I think it's friendly of you to pay someone a compliment on how they look, in this case I don't think that's such a good idea.  Muslim women usually aren't allowed to remove their hijab in public.  You're probably making her uncomfortable by saying she should.
I don't care about what she's comfortable with. If a guy told a girl she looked better with her hijab on she would have said thank you.
You should try to find a way to compliment her that she will find appealing. Since she is muslim, she takes pride in wearing her hijab, so you are inadvertently giving her odd feedback. Let her know what other features you find to be nice.