ARROW-7874: [Python][Archery] Validate docstrings with numpydoc

[kszucs]

No description provided.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - How to contribute patches

[jorisvandenbossche]

Cool!

I wanted to try it out, but do you know if there is a way to only run this check with archery? Eg archery lint --with-numpydoc 1 is still running all other lint checks as well, so to just run numpydoc you would need to manually disable all others

[jorisvandenbossche]

I think we should maybe start with a subset of the checks (an overview is here: https://github.com/numpy/numpydoc/blob/5e9a629c5882a32b5539345f807cd8fc8f0dc624/numpydoc/validate.py#L35).

Maybe things like all parameters documented, no wrong section names, are most important to start with.

[kszucs]

@jorisvandenbossche SGTM

[kszucs]

Yeah, I'm running it via

archery lint --with-clang-format false --with-cpplint false --with-iwyu false --with-cmake-format false --with-rat false --with-r false --with-rust false --with-docker false --with-numpydoc true

[kszucs]

@jorisvandenbossche I've added an standalone archery command for running it, because it requires pyarrow to be avilable for import during runtime. You can also pass a list of rules to ignore or include.

archery numpydoc --whitelist ... --blacklist ...
archery numpydoc -w RULE1 -w RULE2  # enable two rules
archery numpydoc -b RULE1 -b RULE2  # desallow two rules

It will help with iterating over the high priority violations.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-7874

[jorisvandenbossche]

Looks good!

[kszucs]

@jorisvandenbossche ready for review

[jorisvandenbossche]

In the output, I see things like

pyarrow.lib.table
-> table(data, names=None, schema=None, metadata=None)
SS03: Summary does not end with a period
SS06: Summary should fit in a single line
PR02: Unknown parameters {'data', 'schema', 'metadata', 'names'}
SA04: Missing description for See Also "Table.from_arrays" reference
SA04: Missing description for See Also "Table.from_pandas" reference
SA04: Missing description for See Also "Table.from_pydict" reference

So it seems that for cython methods the parameters are not recognized? (I thought this worked in pandas, though, I might need to check)

[kszucs]

Numpydoc looks for the signature, but:

>>> import pyarrow as pa
>>> import inspect
>>> inspect.signature(pa.table)
ValueError: no signature found for builtin <built-in function table>

[jorisvandenbossche]

Pandas example:

In [28]: inspect.signature(pd.Timestamp)   
Out[28]: <Signature (ts_input=<object object at 0x7faa0506d3a0>, freq=None, tz=None, unit=None, year=None, month=None, day=None, hour=None, minute=None, second=None, microsecond=None, nanosecond=None, tzinfo=None)>

where Timestamp is also defined in cython.
Could this be some cython option?

[jorisvandenbossche]

Ah, but for a function, it also doesn't work in pandas (we don't have many functions from cython directly exposed publicly). And Timestamp is a class, not cdef class, which might be the difference

[kszucs]

Yeah, cython doesn't set __text_signature__ for the cdef'd classes and functions, despite that embedsignature directive is set to True, and inspect.signature looks for that property.

[jorisvandenbossche]

What I find surprising is that eg pa.table is actually not a cdef or cpdef, but just a def

[jorisvandenbossche]

Anyway, I think this can be merged? Or should we wait a bit more for someone else to review?

[kszucs]

@jorisvandenbossche I managed to add a workaround, so now we can parse the signature of pa.table as

<Signature (data, names=None, schema=None, metadata=None)>

It alseo required some additional gymnastics to get rid of the cython specific typehints, se we convert

array(obj, type=None, mask=None, size=None, from_pandas=None,
      bool safe=True, MemoryPool memory_pool=None)

To:

<Signature (obj, type=None, mask=None, size=None, from_pandas=None,
            safe=True, memory_pool=None)>

[kszucs]

You can test it with archery numpydoc -w PR02

[kszucs]

+1, merging on green