Fix lower bounds nightly build

[nvaytet]

• Lower bounds for runtime dependencies (scipy was the important one)
• Lower bounds for test dependencies (pytest was causing failures)
• Unit tests: move to using fixtures in conftest.py instead of having a common.py module and an __init__.py file.