You know, for some reason I find my husband's recounts of being spanked growing up arousing.
Wait, really? What about his stories of being spanked growing up makes you aroused?
He told me once that it turns him on when I rest my hand on his naked back, that didn't do anything for me but then he explained that he enjoyed being spanked. That really turned me on.
As long as it doesn't turn into something unhealthy, it's not wrong to feel how you do when he tells childhood stories.  You might want to keep it a secret if you think it might make your husband feel uncomfortable, some things are better kept secret.
Maybe I'll try spanking him next time we're together.
If that's where you want to take it then that should be fine. It's normal for him to share information about when he was a child so that's fine and having a fetish is fine too. You and your partner seem really close so I think everything will work out fine for you.