This reverts commit 9c34805d6157e8b2c567991c1c04bcbb3cdde4df.

Reverted https://github.com/pytorch/pytorch/pull/194772 on behalf of https://github.com/georgehong due to Sorry for the churn, need to revert this part of the stack due to revert conflicts in https://github.com/pytorch/pytorch/pull/187002. A rebase and remerge should be all that's needed ([comment](https://github.com/pytorch/pytorch/pull/194379#issuecomment-5446744804))