You must press Apply or OK to rotate the object, but pressing Enter doesn't do anything (yet closes the window). This is not the behaviour in Arbitrary Scale.
(removed 1.5 issues)
I cannot reproduce this on GtkRadiant 1.6, it is only an issue on 1.5. Make sure you are testing the right version, we are only reporting issues here regarding the 1.6 branch.
CONFIRMED: Enter doesn't seem to do anything on either "Arbitrary Rotation" or "Scale" menus. I suggest adding this functionality to both menus.
Cannot reproduce: Decimals seem to work fine on GtkRadiant 1.6. (see 1.)