-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: Use vcrpy more widely, regenerate broken cassettes #2128
base: dev
Are you sure you want to change the base?
Conversation
ebdfe2a
to
916b8d5
Compare
Attempting to ignore the broken tests, I see the following are also not vcr'd:
Besides those, I'm getting There are a couple more errors, but they're in a different vein, and hopefully by the time we get these errors settled the others will resolve themselves. Logs: |
In view of the above, I've added On my system, this reduces the total number of failed tests to 14:
pytest-chroot.log |
One cassette down, two to go. Though for some reason I've picked up a test failure in |
Turns out Moreover, I'm puzzled by the fact we have this function in the first place. |
49893ce
to
13fa4e3
Compare
Yet another function found:
|
29cbc5b
to
72d5674
Compare
Same reasoning as for tests/test_matching.py::test_ytmusic_matching -- this makes the tests more robust to API rate-limiting and denoises them from network problems. After all -- we're testing *our* behaviour wrt the APIs, not the general reliability of these workloads. - tests/console/test_entry_point.py::test_download_song - tests/console/test_entry_point.py::test_preload_song - tests/test_init.py::test_download - tests/test_init.py::test_get_urls - tests/utils/test_ffmpeg.py::test_convert - tests/utils/test_m3u.py::test_create_m3u_content - tests/utils/test_m3u.py::test_create_m3u_file - tests/utils/test_metadata.py::test_embed_metadata - tests/utils/test_search.py::test_parse_query
This is as far as I'm able to take it for now -- work is ramping up and won't be able to keep pushing on this in the near future. If anyone picks this up, the remaining items that need to be dealt with for this PR are:
The remaining test failures in the logs I've posted are due to #2058 and should be addressed there. The other failures should be addressed by the (newly/re)generated cassettes -- they no longer cause failures either my machine or my chroot with |
This is necessary to allow testing with --block-network -- just pass -m 'not novcr' as well to disable these tests. Tests affected: - tests/utils/test_ffmpeg.py::test_download_ffmpeg However, I doubt whether this function is even necessary -- shouldn't we be providing binaries with ffmpeg bundled instead for those users who can't install it conveniently on their machines?
To smooth out cassette regeneration workflow -- am constantly getting HTTP 401/429 responses, which is masking the cause of test failures.
These either threw HTTP 429 errors or ContentDecodingErrors. My hypothesis for the latter, which seems validated by testing, is that these cached broken responses.
Note
The scope of this PR has creeped since the initial posting.
Additions are listed at #2128 (comment)
test_matching: Use vcrpy to avoid being rate-limited by Spotify
Description
Among the network-based tests,
test_matching
has not been updated to usevcrpy
. This makes it hit the Spotify API with every run, making the tests flaky. This patch series enablesvcrpy
on the test and uploads the cassettes that were generated with it.Also, update test expectations for some tracks that now have more matches.
Related Issue
Closes: #2127
Motivation and Context
See above.
How Has This Been Tested?
Tests are running on a clean chroot with the following config:
(Specifically, I'm using Arch Linux's
pkgctl
to create asystemd-nspawn
container with only the minimal dependencies needed to build the AUR package installed.)Types of Changes
Checklist
My code follows the code style of this project
My change requires a change to the documentation
I have updated the documentation accordingly
I have read the CONTRIBUTING document
I have read the CORE VALUES document
It doesn't exist?
I have added tests to cover my changes
All new and existing tests passed
I am still getting these four failures, but these seem to be a bug in the YTMusic provider code (I suspect the "lookup by ISRC" code is dropping some matches) rather than a problem with the cassettes
Moreover, when I run in a chroot, on the first run I get errors due to attempts to overwrite cassettes (even when running with
--recording-mode=none --block-network
) pytest-initial-chroot, pytestdebug-initial-chroot, and on subsequent runs I getYT-DLP download error
in some cases pytest-later-chroot, pytestdebug-later-chroot. Outside the chroot, I sometimes get cassette-overwriting attempts pytest-cassette-system, pytestdebug-cassette-system, and sometimes only the four YTMusic errors above pytest-ytmusic-system, pytestdebug-ytmusic-system.I don't know what's wrong with my setup that I'm getting these inconsistent results, but this is as far as my knowledge carries me.