What do you think of me annoying my neighbor?
That's mean. Why would you annoy your neighbor?
He annoys me with constant music at late hours
Your neighbor might annoy you but it's wrong to retaliate and irritate him on purpose. You should report your neighbor's obnoxious behavior to the authorities.
No way I want to irritate him because I don't respect him. I'm going to get back at him instead.
Then he will annoy you back and it will never end. Irritating people on purpose won't solve anything. You should work things out instead.