I spent the last couple of hours debugging what slows down development mode so much:
In the end it was the footnotes gem that bumped each request from 10ms to 10s.
Please kick Footnotes out or warn people. I had no idea which of the many template options or other parts of the environment where causing this.
Thanks for the alert and the suggestion. I'll add a warning.