I think I should tell my friend he probably isn't as smart as he thinks he is
Why do you feel the need to tell your friend that he isn't that smart?
He borrows big time from me and gets easily angry if I make a mention of it.
I don't think it's a good idea to tell your friend that he isn't as smart as he thinks he is, because it will just hurt his feelings. I think it's better to keep that opinion to yourself, because telling him won't make anything better, and could possibly cause a rift in your friendship.
I thought we were just pottering about and having fun, but it's starting to annoy me.
It's a good idea to not talk down to your friends, but if it is really start to eat at you, perhaps you could try to bring it up more gently.  Try to politely mention to him that he is repeating something you previously said, and if he gets mad, just calmly remind him that his behavior is irritating to you as well.  He will probably understand.