In order to execute the STOQS functional tests and to execute Jupyter Notebooks in a STOQS Virtual Machine you will need to install X Window System software on the host computer for your STOQS VM.
From OS X Mountain Lion (10.8) Apple dropped dedicated support for the X11.app. Users are now directed to the open source XQuartz project. Install XQuartx from https://www.xquartz.org/; you may need to reboot after installing XQuartz.
This post seems to have some good suggestions for installing X Window System software on Windows: http://superuser.com/questions/99303/what-are-my-x-client-options-for-ms-windows