DOC: stats: documentation is not distribution-specific

[ahmedshaaban1]

Note from @mdhaber: This is a deep issue that is slated to be addressed as part of this project. Before composing a PR related to this issue, please comment below or - even better - consider resolving another open issue!

I think that the docstring of the scipy.stat.mean.cdf "the cumulative distribution function of the mean distribution." doesnot match with the scipy documentation. The documentation page of the scipy.stats.norm.cdf indicates that the second argument is
location. Yet, the docstring saying that it is the 3rd argument.

import scipy
scipy.stats.norm.cdf?

The docstring of the scipt.stats.mean.cdf

Signature: scipy.stats.norm.cdf(x, *args, **kwds)
Docstring:
Cumulative distribution function of the given RV.

Parameters
x : array_like
    quantiles
arg1, arg2, arg3,... : array_like
    The shape parameter(s) for the distribution (see docstring of the
    instance object for more information)
loc : array_like, optional
    location parameter (default=0)
scale : array_like, optional
    scale parameter (default=1)

Returns
cdf : ndarray
    Cumulative distribution function evaluated at `x`
File:      ~/anaconda3/lib/python3.7/site-packages/scipy/stats/_distn_infrastructure.py
Type:      method

Scipy/Numpy/Python version information:

1.2.1 1.16.2 sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)

[pvanmulbregt]

I believe that the poster's Issue is referring to a mismatch docstring with SciPy documentation scipy.stats.norm.cdf not scipy.stats.mean.cdf. [Autocorrect really prefers doctoring over docstring.]

[pvanmulbregt]

The docstring for scipy.stat.norm.cdf is the generic docstring for all norm methods in stats classes inheriting from rv_continuous, and has to allow for an arbitrary optional number of shape parameters arg1, arg2, .... The doc page is using the form explicitly for scipy.stat.norm.cdf. The difference is that scipy.stat.norm doesn't take any shape parameters, so the arg1, ... can be dropped.

[WarrenWeckesser]

@pvanmulbregt, yes, norm doesn't have any shape parameters, so it is confusing that the docstring shows arg1, arg2, arg3,... : array_like.

It would be great if we could get rid of those generic docstrings, and have the docstrings of the method of each distribution actually match the available parameters of that distribution. I think the current style favors developer convenience too much, at the expense of user experience.

[ahmedshaaban1]

Yes. Also the docstring of the scipy.stat.t.cdf is a littlte bit confusing for the users. The second argument of the docstring is

arg1, arg2, arg3,... : array_like       
    The shape parameter(s) for the distribution (see docstring of the
    instance object for more information)

Yet the second argument, as in SciPy documentation, is defined as the degree of freedom.

[ev-br]

I looked at these arg1, arg2, arg3 a while ago. IIRC one technical issue was that docstrings of methods are (were in python 2?) immutable, unlike the class docstring. The shape args are only known at import time, and we can construct the class docstring. But not the method docstrings.

[mdhaber]

We closed gh-11578 because it is mostly a duplicate, but when we try to resolve this, we should check not only the rendered documentation, but also the hints provided by IDEs. Currently PyCharm is not showing any useful class-level documentation, whereas Spyder at least shows something generic. It could just be a PyCharm issue, but I suspect that there is a way we can address the issue that will work in all IDEs.

[mdhaber]

@Carreau Based on your gh-13458, I wondered if I could interest you in this issue and related issues (gh-11578, gh-12848, gh-9706).

The basic problem is that SciPy stats distribution-specific documentation is automatically generated (e.g. norm, chi2) in a way that is not seemless.

For instance, users can't click to get distribution-specific documentation for the scipy.stats.norm.rvs method.

Their best best is the rv_continuous.rvs documentation, but that is totally generic:

It would be nice if there were auto-generated, distribution-specific documentation for each method that works just as well as if we had actually written it manually. Any tips?

[Carreau]

Any tips?

That is part of what I have been working on my spare time for the past few month, among other problem related to documentation I want to fix, like having less discrepancies between docs in the terminal/IDE vs sphinx/html – i'm happy to talk about it on one-on-one to get your thoughts. I would have to dive into each of those explicitly.

For the signature you might be able to assign __signature__ and mess with it; One of the question is how much work is ok to do at import time...

[mdhaber]

One of the question is how much work is ok to do at import time...

Well, on my machine the first time I import any scipy subpackage (e.g. import scipy.stats) or any function from any subpackage (e.g. from scipy.stats import norm), it takes about 0.25 s.

Subsequent imports of any subpackage or any functions/classes take microseconds. I'll guess that few people would be bothered if the initial import time were to increase by 10%? Perhaps 50% would be tolerable for most if it were to fix a big issue like this? Please jump in to correct me, anyone : )

[mdhaber]

Let's let this issue cover multivariate distributions (gh-12848), too: the methods documentation of univariate and multivariate distributions does not render properly because each is an instance of a class rather than a class. Hopefully, this can be addressed for both univariate and multivariate distributions in one shot.

[mdhaber]

@tirthasheshpatel I think you were trying to make the documentation of distributions more specific, right @tirthasheshpatel? How did it go?

[tirthasheshpatel]

I think you were trying to make the documentation of distributions more specific, right @tirthasheshpatel? How did it go?

I was able to link each method's docs of each distribution to a (unique) distribution-specific page. Here it is: tirthasheshpatel#14. Thankfully, we don't need to write an extension from scratch; we can use the autosummary templating mechanism.

We need to write docs for each distribution's methods for the change to be useful, otherwise, compared to the previous documentation, we no longer document the shape arguments that the methods take:

Before:

After:

[mdhaber]

Awesome! Want me to look into that a bit? I'll submit a PR to your branch.

[tirthasheshpatel]

Awesome! Want me to look into that a bit? I'll submit a PR to your branch.

Sure!

[mdhaber]

Based on our discussion in tirthasheshpatel#14, shall we consider this a wontfix under the current infrastructure? If we have to choose, it is probably more important to show the signature of the methods correctly than to show *args, **kwds and link to rv_continuous.

[tirthasheshpatel]

Based on our discussion in tirthasheshpatel#14, shall we consider this a wontfix under the current infrastructure?

Yes, we can label this wontfix now and come back when we deprecate/remove frozen distribution infrastructure.

[mdhaber]

I'm going to reopen this so the link in gh-15928 shows up in green. It's important to see that this is unresolved and can't really be resolved with the existing infrastructure. Never mind. It is clear that it is unresolved when it's in gray.