He promised to give the pen back to me at the end of the day.
How long has it been already since he promised that?
This morning, but now it is after work.
You were very kind to lend him your pen, so shame on him for not returning things on time. He should extend the kindness and treat you and your objects with respect. I don't think you should lend to him anymore.
I am not worried, He'll likely apologize later and offer to buy another pen as an apology.
If he does that, then it will be okay, but make sure that he is not in the habit of taking advantage of you.