Add an easy way to deselect all production tests

[lahwaacz]

Hi, I would like to execute tests while packaging for Arch Linux. However, it seems that many tests connect to the production hub at huggingface.co and obviously fail without authentication. This leads to errors like

FAILED tests/test_cache_no_symlinks.py::TestCacheLayoutIfSymlinksNotSupported::test_download_no_symlink_existing_file - huggingface_hub.errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-679d17ad-01f3bb1b69a02de07cc1430b;bf122663-13d4-4203-b238-2af7d50f7aae)

Repository Not Found for url: https://huggingface.co/julien-c/dummy-unknown/resolve/main/config.json.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid credentials in Authorization header

It seems that this is coming from

huggingface_hub/tests/testing_utils.py

Lines 218 to 221 in a733933

	def with_production_testing(func):
	file_download = patch("huggingface_hub.file_download.HUGGINGFACE_CO_URL_TEMPLATE", ENDPOINT_PRODUCTION_URL_SCHEME)
	hf_api = patch("huggingface_hub.constants.ENDPOINT", ENDPOINT_PRODUCTION)
	return hf_api(file_download(func))

Can you add proper pytest markers such that I can easily deselect all production tests with something like pytest -m "not production"? 🙏

[Wauplin]

Hi @lahwaacz can you retry with the latest version on main? We've fixed some auth issues in the tests this week which should have solved your problem: #2811. If errors persist, please let me know :)

And glad to hear about Arch packaging! Thanks for taking care of it 🤗

[julien-c]

(yes we have some Arch users on the team:))

[lahwaacz]

I'm sorry for the delay and thanks for your responses 😄

Today I tried to parallelize the test suite with pytest-xdist, but many tests failed on our build server with 48 cores. The errors were typically

huggingface_hub.errors.HfHubHTTPError: 503 Server Error: Service Temporarily Unavailable for url: https://hub-ci.huggingface.co/api/repos/create

so I guess that I still did not --deselect all tests that send external requests. I really wish there was a proper pytest marker for disabling all tests that communicate with something outside the build server itself.

[Wauplin]

I really wish there was a proper pytest marker for disabling all tests that communicate with something outside the build server itself.

We could add something like this but to be honest maybe 90% of our tests are making calls to either huggingface.co (the prod) or hub-ci.huggingface.co (the staging env). In both cases, they are calls made outside of the build server itself. We decided to go this way as this is a client library which main use case is to interact with the Hub. We didn't want to implement mocked tests (or similar) as we prefer to actually test everything works on real use-cases with a remote server.

All of this to say, most tests are making HTTP calls because that's what the package is meant for. I don't think it makes much sense to run only offline tests.

[lahwaacz]

Right, but we still like to run some tests for packaging purposes. This ensures that the software we package can be actually used in some way, and especially that it is compatible with other system-wide packages in our distribution. So even if we run only like 10% of your tests, which may seem like a small number, it would still be very useful for us I think.