This package includes a function to optimize functions under orthonormal constraints (Stiefel manifold) using the algorithm from the paper Zaiwen Wen and Wotao Yin. A Feasible method for Optimization with Orthogonality Constraints, Optimization Online, 11/2010. Also as Rice CAAM Tech Report TR10-26. It is a port of their MATLAB code, which can be found here.
You can install the package directly from Github using the devtools package.
# install.packages("devtools")
devtools::install_github("andland/FOptM")