I slept with my work colleague's wife
Why did you cheat with a coworker's wife?
It was harmless. We flirted one evening and we did it in my office.
Cheating isn't harmless and you shouldn't do that to a coworker. You should be honest with your coworker about what happened. If they find out you lied it will only make things worse at work.
That's not true. The less they know the better and it will not destroy their family.
You may be right that it will destroy their family if your coworker finds out, but you should not have cheated in the first place. Now that you have, keeping it from them is something you will endure in the future and I think your conscience can make you suffer and you may end up getting insane about this.