I take every single religious pamphlet I see in public places
Is there a reason you take religious pamphlets specifically? Are you interesting in learning about them?
Because religion is harmful to this world.
There are so many different religions nowadays that any one isn't going to cause harm to the world. You're just interfering with people's freedom at that point. You really should stop doing that if you aren't genuinely interested in what they have to say.
So what? I don't care.
If you do not care, show this by ignoring it.  In the same way that you would not want a religious person to try and convert you or force it down your throat, you should show religious people the same respect.