I'm planning not to go to confession at church because I have no sins to confess other than my infidelity.
Don't you think infidelity is a sin ?
I'm also considering not going to communion since I've been committing all these sins for a while.
If your religious beliefs require you to admit to your sins, you should confess. If you have been unfaithful to your spouse not only should you confess but also ask your spouse for forgiveness.
My religion shouldn't be the one that judges my sins and that's the way it should be.
Confessions are helpful to take the load off of your chest. There is no harm in doing so as it's anonymous. You should give it a try.