doc: add list of fixtures to start of fixture chapter

[kpinc]

No description provided.

[bluetech]

Hi @kpinc. This seems like something that'd be nice to have. For our lazy ones, would you mind showing a screenshot of how this looks?

I am not sure about it's location. It seems out of order to give the list of built-in fixtures before explaining what they are. On the other hand, the list should be near the top to be useful.

I am not an expert in the docs system, but I wonder if there is any way to auto-generate this list, or avoid repeating somehow? I see that configuration values have a special custom confval sphinx object type; maybe adding one for fixtures would help?

[kpinc]

Hi Ran,

On Fri, 14 Feb 2020 05:13:04 -0800 Ran Benita ***@***.***> wrote: Hi @kpinc. This seems like something that'd be nice to have. For our lazy ones, would you mind showing a screenshot of how this looks?

Screenshot attached: pytest_doc_fixture_list.png Better might be the docs themselves. Attached are all the docs in html, for reference with a "file://" url: pytest_doc_fixture_list.tgz

I am not sure about it's location. It seems out of order to give the list of built-in fixtures before explaining what they are. On the other hand, the list should be near the top to be useful.

The cat is out of the bag on showing fixtures before explaining. The record_property and record_testsuite_property fixtures are used in the docs before the fixture chapter appears to explain what fixtures are. I think the first 2 sentences of the chapter explain what fixtures are clearly enough. IMO the very first things that should be explained in the fixture chapter are the issues that fixtures solve. After that a list of fixtures should help clarify what tooling is available. IMO the first sentence of the Wikipedia page on fixtures does a lot better job of explaining the point of a fixture than the pytest docs. Wikipedia writes: "A software test fixture sets up a system for the software testing process by initializing it, thereby satisfying any preconditions the system may have. ... This allows for tests to be repeatable, which is one of the key features of an effective test framework." The pytest docs write: "The purpose of test fixtures is to provide a fixed baseline upon which tests can reliably and repeatedly execute." This is adequate. I would write: "Fixtures initialize software tests. They provide a fixed baseline so that tests execute reliably and produce consistent, repeatable, results. Initialization may setup services, state, or other operating environments. These are accessed by test functions though arguments; for each fixture used by a test function there is typically a parameter (named after the fixture) in the test function's definition." At that point I'd start the 2nd sentence of the existing docs, which, compares pytest with xUnit, as a new paragraph. If you like, I can make this change part of the patch. After that, what sort of initialization each fixture supports should be pretty clear from a short description. See also issue #6565, that lead to this patch.: #6565

I am not an expert in the docs system, but I wonder if there is any way to auto-generate this list, or avoid repeating somehow? I see that configuration values have a special custom `confval` sphinx object type; maybe adding one for fixtures would help?

I am not an expert either, but any way to auto-generate is going to need 2 things: a description and a link target. I've never seen anything in RST that lets a link target be specified for some "label text" where the label text can appear in some arbitrary location with respect to the link target. This might be an issue. The descriptions I've used do not typically appear elsewhere in the documentation. They are generally much abbreviated versions of the first sentence or one of the first sentences of the existing documentation. But my descriptions are not even sentences. They are more "bullet points", although if you put a period on the end they would qualify as sentences. Before thinking about auto-generation we'd have to consider whether my short descriptions are suitable components of a longer description of each fixture and whether they could be worked into the source text in a way that makes RST link-labeling happy. The point to auto-generation would have to be that the short description of each fixture would be written into the source document alongside the documentation of the rest of the fixture. But if the short description does not duplicate anything else already in the docs this does not help a whole lot. Even if there were markup that could be added to fixture docs for auto-generation, this would still have to remember to be done. All in all, I see the auto-generation appeal. But this seems like a big step to take that might require multiple changes in different parts of the documentation. And I don't really want to go there without RST support. (Docbook/xml makes this sort of thing doable, although nothing is ever easy with xml. :) Regards, Karl <kop@karlpinc.com> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein

[kpinc]

Github sure makes it hard to attach files. (It won't take them by email, and won't take tgz files, and just seems painful to work with.)

pytest_doc_fixture_list.zip

[bluetech]

Screenshot attached: pytest_doc_fixture_list.png

Thanks! It looks good. I would personally add the periods at the end of the sentences, though.

The cat is out of the bag on showing fixtures before explaining.

You convinced me. I would add some "weasel" words however, to make it clear that this is a reference and there is more below. My suggestion is to remove @pytest.fixture from the list, and change the sentence introducing the list to this:

Fixtures are defined using the :ref:`@pytest.fixture <fixtures-api>` decorator, described below. Pytest also defines several useful built-in fixtures, listed here for reference:

<here comes the list>

WDYT?

If you like, I can make this change part of the patch.

I think all improvements to the docs are welcome. Should probably do it in a separate PR though, so the discussion can be focused.

All in all, I see the auto-generation appeal. But this seems like
a big step to take that might require multiple changes in different
parts of the documentation. And I don't really want to go there
without RST support.

Yeah, that's fine. From #6565 I gather that @nicoddemus and @blueyed both agree that it should not block this improvement.

BTW, your sentences I think would make an excellent first-sentences for the fixtures docstrings. If the docstrings are written in the standard style

@pytest.fixture
def record_property():
    """Add extra properties to the test.

    More text here ...
    """

then a machine should be able to extract them. Not sure how that would play with translations though.

Anyway that's just a thought, as said you don't need to do it :)

[bluetech]

Would add some indication on how it's different from capsys. Also below for capfdbinary.

[bluetech]

Would change pytest -> pytest plugins, presumably that's the better audience to target.

[bluetech]

Would change logging -> log capturing

[blueyed]

1. If builtin fixtures would be in their own page (e.g. fixtures.rst or builtin-fixtures.rst), it could be included via toctree:

.. toctree::
   :caption: List of built-in fixtures
   :name: fixturetoc

   builtin-fixtures

This however would not include the caption you've added, but that could be OK in general (adding it to the builtin-fixtures page instead maybe), or have it in the title.

2. in any case we should add:

    app.add_crossref_type(
        "fixture",
        "fixture",
        objname="built-in fixture",
        indextemplate="pair: %s; fixture",
    )

(to setup in doc/en/conf.py)
And then use that for cross-referencing:

.. fixture:: record_testsuite_property

record_testsuite_property
~~~~~~~~~~~~~~~~~~~~~~~~~

I think moving it out of reference is a good idea in general, especially also with regard to structuring the docs better (which blocks #6460).

[blueyed]

I would also order them alphabetical in any case, not "in order of introduction", which appears to refer to the order in references, right? (and is not really important in the toc/overview / on another page)

[kpinc]

On Sat, 15 Feb 2020 04:34:26 -0800 Daniel Hahler ***@***.***> wrote: I would also order them alphabetical in any case, not "in order of introduction", which appears to refer to the order in references, right? (and is not really important in the toc/overview / on another page)

"In order of introduction" is the order in which the fixtures are introduced and explained throughout the narrative. If I kinda-sorta had my druthers I'd put the "important ones" first. ;-) For me that's monkeypatch and caplog and then maybe capsys. (You've got to monkeypatch to do real unit-testing and almost every well-crafted program should log or produce error messages.) And I'd probably re-arrange the entire part of the documentation revolving around my ordering choices. But I decided not to radically hack-around major documentation pieces. ;-) So that sort of left me with "order of introduction", presuming that there's some logic to the existing ordering . But alphabetic makes sense for an overview-y/reference-y part of the narrative. So next revision will be alphabetical. Regards, Karl <kop@karlpinc.com> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein

[kpinc]

On Sat, 15 Feb 2020 02:52:44 -0800 Ran Benita ***@***.***> wrote: > Screenshot attached: pytest_doc_fixture_list.png Thanks! It looks good. I would personally add the periods at the end of the sentences, though.

I don't mind adding periods, but if in future the text is going to be pulled in as a TOC there won't be periods. So for now I'm going to leave off the periods. Let me know if you want periods.

> The cat is out of the bag on showing fixtures before explaining. You convinced me. I would add some "weasel" words however, to make it clear that this is a reference and there is more below. My suggestion is to remove ***@***.***` from the list, and change the sentence introducing the list to this: ``` Fixtures are defined using the ***@***.*** <fixtures-api>` decorator, described below. Pytest also defines several useful built-in fixtures, listed here for reference: <here comes the list> ```

Done.

> If you like, I can make this change part of the patch. I think all improvements to the docs are welcome. Should probably do it in a separate PR though, so the discussion can be focused.

Submitted a separate pull request, #6742.

BTW, your sentences I think would make an excellent first-sentences for the fixtures docstrings.

Thanks. I'm not 100% certain of this. Some might be just too abbreviated. I do think some of the first sentences of the fixture descriptions could be improved but decided that this didn't fit into this pull request. Regards, Karl <kop@karlpinc.com> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein

[kpinc]

On Sat, 15 Feb 2020 02:57:11 -0800 Ran Benita ***@***.***> wrote: bluetech commented on this pull request. + :ref:`capsys-api` + Capture writes to ``sys.stdout`` and ``sys.stderr`` + + :ref:`capsysbinary-api` + Capture writes to ``sys.stdout`` and ``sys.stderr`` Would add some indication on how it's different from `capsys`. Also below for `capfdbinary`.

Done.

+ + :ref:`testdir` + Provide a temporary test directory to aid in running, and Would change pytest -> pytest plugins, presumably that's the better audience to target.

Done.

+ :ref:`caplog-api` + Access, and control, logging Would change logging -> log capturing

Reworded. See what you think. Regards, Karl <kop@karlpinc.com> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein

[kpinc]

For grins, attached are another screenshot and set of html docs

pytest_doc_fixture_list_v2.zip

[kpinc]

On Sat, 15 Feb 2020 07:39:10 -0800 Ran Benita ***@***.***> wrote: bluetech commented on this pull request. > @@ -34,25 +34,54 @@ both styles, moving incrementally from classic > to new style, as you prefer. You can also start out from existing :ref:`unittest.TestCase style <unittest.TestCase>` or :ref:`nose based <nosestyle>` projects. -The built-in pytest :ref:`fixtures <fixtures-api>` are, in order of -introduction: +:ref:`Fixtures <fixtures-api>` are defined using the ***@***.*** <pytest.fixture-api>` decorator, :ref:`described +below <funcargs>`. Pytest has useful built-in fixtures, listed here +for reference: + + :ref:`capfd-api` + Capture, as text, output to file descriptors ``1`` and ``2`` + + :ref:`capfdbinary-api` + Capture, as bytes, output to file descriptors ``1`` and ``2`` + + :ref:`caplog-api` + Control logging and access log entries This fixture doesn't control logging per se, just how it's captured. Being completely accurate is a little clumsy but that would be something like `Control log capturing and access captured log entries`

Yeah. It does _configure_ logging, by allowing log-levels to be set. At least it configures logging for the purposes of log-capturing/ for the purposes of the test run. I'm presently of the opinion that the word "control" is vague enough that people won't be misled. I'm not sure that "control" is an "official technical term", which frees me to wave it around. ;-) I like the brevity of what I have now. Regards, Karl <kop@karlpinc.com> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein

[blueyed]

What about using the fixture crossref? I think it is nicer than having normal refs with (and without) the "-api" suffix.
(#6696 (comment))

[blueyed]

We could add py to the intersphinx config, but maybe not worth it? (it's meant to be phased out)

[kpinc]

On Sat, 15 Feb 2020 09:40:50 -0800 Daniel Hahler <notifications@github.com> wrote:

blueyed commented on this pull request. What about using the `fixture` crossref?

I replied via email with what's at bottom. My email was delivered to in-10.smtp.github.com[140.82.112.32] at Sat 15 Feb 2020 07:09:33 AM UTC. I did not get a bounce. And it seems you did not get the message.

I think it is nicer than having normal refs with (and without) the "-api" suffix.

I don't think I'm understanding here. More at bottom.

> + testing, pytest plugins + + :ref:`tmp_path-api` + Provide a :class:`pathlib.Path` object to a temporary directory + which is unique to each test function + + :ref:`tmp_path_factory-api` + Make session-scoped temporary directories and return + :class:`pathlib.Path` objects + + :ref:`tmpdir-api` + Provide a :class:`py.path.local` object to a temporary + directory which is unique to each test function; + replaced by :doc:`tmp_path <tmpdir>` + + .. _`py.path.local`: https://py.readthedocs.io/en/latest/path.html We could add py to the intersphinx config, but maybe not worth it? (it's meant to be phased out)

It's not clear what you'd like me to do with this. (I'm not inclined to make a change that will later need to be un-made.)

----------------------<snip>--------------

Date: Sat, 15 Feb 2020 07:09:30 -0600 From: "Karl O. Pinc" ***@***.***> To: Daniel Hahler ***@***.***> Cc: pytest-dev/pytest ***@***.***>, pytest-dev/pytest ***@***.***>, Mention ***@***.***> Subject: Re: [pytest-dev/pytest] Add list of fixtures to start of fixture chapter (#6696) Message-ID: ***@***.***> In-Reply-To: ***@***.***> References: ***@***.***> ***@***.***> On Sat, 15 Feb 2020 12:23:29 +0000 (UTC) Daniel Hahler ***@***.***> wrote: > 1. If builtin fixtures would be in their own page (e.g. fixtures.rst > or builtin-fixtures.rst), it could be included via `toctree`: ``` > .. toctree:: > :caption: List of built-in fixtures > :name: fixturetoc > > fixtures > ``` > This however would not include the caption you've added, but that > could be OK in general (adding it to the builtin-fixtures page > instead maybe), or have it in the title. > > 2. in any case we should add: > ```python > app.add_crossref_type( > "fixture", > "fixture", > objname="built-in fixture", > indextemplate="pair: %s; fixture", > ) > ``` > (to `setup` in `doc/en/conf.py`) > And then use that for cross-referencing: > ``` > .. fixture:: record_testsuite_property > > record_testsuite_property > ~~~~~~~~~~~~~~~~~~~~~~~~~ > ``` > > I think moving it out of `reference` is a good idea in general, > especially also with regard to structuring the docs better (which > blocks #6460). > By "it" I'm presuming that you mean all the fixtures. And I'm also presuming that you don't expect me to add the the changes you mention in this email to this pull request. No need to reply if my thinking is right.

----------------------<snip>-------------- So I don't think I read your message correctly. You would like #2, above, done. I can do that, and change the hyperlinks to fixtures in my patch. Honestly, this would seem to make sense as a separate pull request because wouldn't we also want to change all the other hyperlinks to fixture targets throughout the docs? Please explain what you mean by "moving out of 'reference'". And what linking or not-linking with *-api, or perhaps not suffixing link targets with "-api" has to do with what you want. My preference would be to have the list of fixtures refer to the api docs. The presumption is that the api docs are concise, yet complete, and link back to relevant parts of the narrative/tutorial when more explanation is needed. Your example uses the "fixture" crossref to establish an anchor within the narrative docs. This makes sense. It is where you'd want most inter-document fixture hyperlinks to target. But if that's the case I'd _not_ want to use "fixture" to do my patch's hyperlinking. Before I make any changes I'll wait for your reply. Regards, Karl <kop@karlpinc.com> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein

[blueyed]

@kpinc
Using a crossref allows for ":fixture:`request`" instead of ":ref:`request-api`".
This makes it more semantic from what I can tell.

Honestly, this would seem to make sense as a separate pull request because
wouldn't we also want to change all the other hyperlinks to fixture targets
throughout the docs?

Since you are adding new "-api" references here I think it makes sense to add
it here already.

I do not think you need to change existing references, but should also add a
fixture xref to it then (e.g. for testdir), and use "``:fixture:`testdir```"
in your list, of course.

Please explain what you mean by "moving out of
'reference'".

That's out of scope probably, but currently all fixtures and a lot more is just
"dumped" into "references", whereas I think it would make sense to have a
separate "builtin-fixtures" page. Let's not discuss this further here.

Your example uses the "fixture" crossref to establish an anchor within the
narrative docs.

No. You would add them where you've added the -api targets.

I hope this is clear/clearer now.

[kpinc]

On Sun, 16 Feb 2020 01:59:26 -0800 Daniel Hahler ***@***.***> wrote: @kpinc Using a crossref allows for ``":fixture:`request`"`` instead of ``":ref:`request-api`"``. This makes it more semantic from what I can tell. I hope this is clear/clearer now.

Yes. Thanks. Done. Regards, Karl <kop@karlpinc.com> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein

[kpinc]

Hi, As an FYI, this should be ready for review. Some of the CI failed, with things like 503 HTTP status codes. As far as I can tell this has nothing to do with the patch. Regards, Karl <kop@karlpinc.com> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein

[bluetech]

Some of the CI failed, with things like 503 HTTP status codes. As far as I can tell this has nothing to do with the patch.

I restarted it and it's all green now.

[blueyed]

Looking good!
I think it could/should make use of more new fixture xrefs in the list, and would also add punctuation to the text there (dots at the end), but that might have been mentioned/rejected already.

[blueyed]

Refer to ``:fixture:`tmpdir``` here?
(Also with other references in this list.)

[kpinc]

On Thu, 20 Feb 2020 07:22:26 -0800 Daniel Hahler ***@***.***> wrote: I think it could/should make use of more new fixture xrefs in the list,

See below.

and would also add punctuation to the text there (dots at the end), but that might have been mentioned/rejected already.

Not really. It's been talked about. My inclination is to leave it as bullet points, without periods at the end. My thought is that this sorta-kinda makes it like it's pulling text out of headings, as with a TOC. But I've no strong feelings. Modifying to put periods on the end.

> + :fixture:`testdir` + Provide a temporary test directory to aid in running, and + testing, pytest plugins + + :fixture:`tmp_path` + Provide a :class:`pathlib.Path` object to a temporary directory + which is unique to each test function + + :fixture:`tmp_path_factory` + Make session-scoped temporary directories and return + :class:`pathlib.Path` objects + + :fixture:`tmpdir` + Provide a :class:`py.path.local` object to a temporary + directory which is unique to each test function; + replaced by :doc:`tmp_path <tmpdir>` Refer to ``:fixture:`tmpdir``` here?

Ok, done. The original idea was to link into the narrative text from within the narrative description, but I don't think it really matters. I did this with a separate patch so if you like you can mess around and see what link endpoints you prefer.

(Also with other references in this list.)

There is only one other. Regards, Karl <kop@karlpinc.com> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein

[blueyed]

Awesome, thanks!