I told a homeless person at my job that I didn't care if he was hungry.
Did you really tell a homeless man that you didn't care that he was hungry? Did he ask you for food?
Yes I told him that before he had asked for food.
It's important to remain courteous to people down on their luck and in need of help such as the homeless. Telling a homeless person that you don't care that they are hungry is wrong since you should be helping out the homeless when you can and you should be concerned about a fellow human being's physical well-being. If you don't have food to give to the homeless you should point them to a charity or other resource where they can get fed.
It's not my problem and what if he used my name as a justification to beg for money?
You should show compassion for homeless people. You're not required to always give money or food, but it's cruel to tell a hungry person that you don't even care that they're hungry.