I'm annoyed that my bf would ask me about my ex so often, and that he would insist I still think of my ex.
Does he not like your ex boyfriend?
Not really, but I do think that my past should be left in the past
It is good to think your past is your past but your boyfriend seems to be having doubts. You should talk to him, and let him know that you are over your ex and any relationship problems you once had.
I'm not going to talk to him over such sensitive issues.
It can be hard to raise these subjects, but I think it will save you a lot of trouble if you do so.  Perhaps the next time that he brings it up, you can ask him what he is really concerned about.  Then you can have a discussion about how you have left that part of your life behind, and that you are focused on him now.