I think my husband fantasizes about my daughter
Why do you think your husband fantasizes about your daughter?
I know he fantasizes about my daughter
If you are aware that your husband fantasizes about your daughter, it is your responsibility to get him help to deal with these fantasies in order to protect your daughter. It is wrong to fantasize about children. They should feel safe with and be protected by their parents.
It's fine. My daughter might be missing out.
The only thing she's missing out on right now is a healthy upbringing!  I can't believe you'd consider actually allowing this.  I think the both of you should speak to a professional sooner rather than later to deal with these kinds of thoughts.