I used my neighbor's debit card at the store
Did you ask first? What happened to cause you to do that?
He gave me the card awhile back to but something for him and never remembered to ask for it back.
Unless you were buying something for him and he knew you were using the card you should not have done that. You committed fraud by using his card and you could get into a lot of trouble.
I don't care he probably won't notice.
That is stealing and you could end up in prison for that. You need to give the card back and pay him back.