I don't want to throw away my neighbor's junk mail
Why would you not just give it to them?
I'm interested about what kind of life he leads
If your neighbor wants you to throw away his junk mail, you should. It's rude to keep mail that belongs to others. If you are taking his mail and he doesn't know, you shouldn't do that either, it's illegal.
I'll stop taking it, but I hope the police don't catch me.
I think that if you stop now, there should be no issue. Just respect your neighbors privacy just  a little more.