Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

As-installed tests support #4914

Closed
wRAR opened this issue Dec 5, 2020 · 1 comment · Fixed by #5949
Closed

As-installed tests support #4914

wRAR opened this issue Dec 5, 2020 · 1 comment · Fixed by #5949

Comments

@wRAR
Copy link
Member

wRAR commented Dec 5, 2020

Debian supports testing installed packages by running a test suite against them. Doing that for Scrapy is currently impossible, it contains explicit protection against this: scrapy.utils.test.get_pythonpath() ensures PYTHONPATH for external scripts (including mockserver) points to the same instance of Scrapy. So:

  • if you run pytest in the Scrapy repo, tests will import scrapy.utils from the current dir and run python3 -m tests.mockserver with PYTHONPATH set to the current dir, testing the local code
  • if you also install Scrapy, nothing will change
  • if you remove the scrapy dir, which is what is usually done to test the installed code, tests will import scrapy.utils from the system directory, set PYTHONPATH to the system directory and python3 -m tests.mockserver will fail because the tests package is not installed.

I know some Python modules also install the tests package but I don't think this is a good idea. Maybe get_pythonpath() should learn how to find the instance we want, not sure how to do that though.

@wRAR
Copy link
Member Author

wRAR commented Nov 20, 2021

The places where get_testenv() is used:

  1. tests/mockserver.py - this is the main use, it's used to run python3 -m tests.mockserver via Popen so that the interpreter finds the module.
  2. tests/test_crawler.py - used to run scripts from tests/Crawler* via Popen. Not actually needed as far as I can see, except for tests/CrawlerRunner/ip_address.py which imports tests.mockserver.
  3. tests/test_commands.py - used to run python3 -m scrapy.cmdline via Popen. Needed to find the correct scrapy module and is fine to keep (and also suggests that we can't drop the functions).
  4. tests/test_proxy_connect.py - used to run python3 -c <inline script> via Popen. I don't think it's needed, as the scripts just imports mitmproxy and it doesn't need the Scrapy dir for this.
  5. tests/test_cmdline/__init__.py - used to run python3 -m scrapy.cmdline via Popen, see 3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant