Seems like the package owners have internal confusion. The best information I could find is that pypi uses the package name sphinx-rtd-theme (https://pypi.org/project/sphinx-rtd-theme/) while the github repository uses sphinx_rtd_theme (but see readthedocs/sphinx_rtd_theme#1363) and they specifically call out that conf.py should use
html_theme = "sphinx_rtd_theme"
Interestingly it works as is and I noticed only because conda installs both names and I wondered why.
I will only install this package with pip.
pip only lists sphinx-rtd-theme but does not care if you install it as either name.