I'm having an affair with a married coworker and it's slowly making me insane.
Are you serious? Why are you having an affair?
I'm not sure if I'm in love with him or just in love with the idea of having an affair.
You should not get involved with a married man. Especially if it is driving you insane. You need to cut it off now.
I am not sure if I am able to do that.
You need to take steps and do it. If things backfire on you, a lot can go wrong. You could lose your job and the man could get divorced.