DOC: add_newdocs should happen before PyType_Ready

[mattip]

Change add_newdocs to e.g. generate a header file at compile time with the docs (as is done in scipy.special), rather than patching them at runtime after PyType_Ready. Requires some hunting for the correct places to specify the docstrings. The current way tp_doc is hacked (after PyTypeReady) not only leaks the char* string yanked out of the PyStringObject, it does not work on PyPy.

gleaned from #10157, plagarized @pv 's hints

[eric-wieser]

The problem with generating the header file at compile time is that fixing a typo in one docstring now causes that header file to change, and now every single C file which includes that header has to be recompiled.

For testing whether a sphinx construct renders correctly, this is a huge burden.

Perhaps we need to generate a .h file for each function? So /doc_header/concatenate.h would contain a single #define of PyArray_docstring_concatenate, or similar?

Or at least, generate a header per C file that we currently have.

[njsmith]

Would it be terrible to move the actual docstrings into .c files in the first place, like most C API functions work, and then we wouldn't need to generate anything at all?

IIUC the only reason they're in a .py file in the first place is that building numpy used to be much harder, and they were split out so contributors could edit the docstrings without having to go through a build cycle. This is much less of an issue these days.

[pv]

Or, you could autogen the C files. Writing it in python files avoids dealing with C quoting and the "\n" everywhere...

[njsmith]

I guess I don't care that much -- it's not like we have any shortage of autogen code already :-)

If we do this, then instead of generating .h files, I think we can generate a .c file, and link to it. So docstring.c would export a char* called Pyarray_concatenate_docstring or whatever. This would avoid the rebuild-the-world issue.

[mattip]

For one idea how to use a function to generate docstrings in header files, see the code in pnumpy