Add packse scenarios as pip tests #12580
Conversation
| @pytest.fixture(scope="session", autouse=True) | ||
| def start_packse_server() -> Generator[None, None, None]: | ||
| """Starts the packse server before tests run and ensures it's terminated after.""" | ||
| proc = subprocess.Popen( | ||
| ["packse", "serve", "--host", "127.0.0.1", "--port", "3141"] | ||
| ) | ||
| time.sleep(1) | ||
| yield | ||
| proc.terminate() |
There was a problem hiding this comment.
I would recommend using the static index packse generates as mentioned in astral-sh/packse#165 (comment) instead — the serve command is intended for development purposes.
There was a problem hiding this comment.
My concern with that, and maybe it's not justified, is that tests could fail because the scenarios from packse fetch & packse inspect would not align perfectly with the scenarios on the index.
There was a problem hiding this comment.
fetch defaults to pulling the current packse version tag which should always match the corresponding index for that tag. You just need to do packse inspect --no-hash since the index is versioned we don't need to include a hash to uniquely identify scenarios.
There was a problem hiding this comment.
I had a long throught about this and my main concern is that if pip wants to add it's own scenarios, or deviate away from packse scenarios (e.g. because they seemingly don't follow the spec astral-sh/packse#161) then using the astra index won't work.
However, for now, I will attempt to implement using the index, and that problem can be worried about in the future.
|
Some unexpected failures on Mac, but passed on Windows and Linux, probably won't have time to look this weekend but I'm sure I'll be able to figure them out. |
|
I am going to close this for now. I think packse is a great tool, and integrating it into pip's testing would be very useful. But I don't think pip can use all of the scenarios packse provides out of the box or the ones astral provides on it's index. It will therefore require maintenance of either an "allow" list of packse scenarios, or maintaining a separate collection of scenarios, this would therefore introduce a hidden maintenance cost that the PR does not make clear. Some of the issues are (in no particular order):
IMO, to achieve "testable standards" (as Charlie mentioned) I think there needs to be a delineation of scenarios, either by a flag or living in a separate repo, of those that are derived from standards and ones that are testing non-standard installer features or design choices. And, somehow, more members of the interested parties (those writing and discussing standards) need to be involved. But I think such a discussion needs to move to https://discuss.python.org/c/packaging/14. |
Fixes #12526
Current strategy:
packse fetchwhich git pulls the packse scenariospackse servewhich builds and serves the scenarios on a local indextest_packse_scenarioas a dynamically generated parameter from the information given inpackse inspectI do not have a lot of experience with test frameworks, so I'm not at all claiming this is the best approach, it's just what I got working, happy to significantly change it.
There is still more work to do discussing issues on packse side and identifying if currently failing scenarios are bugs on pip side. Once
EXPECTED_TO_FAILis much smaller I will unmark as draft.