at::from_blob documentation is broken

[elbaro]

📚 Documentation

https://pytorch.org/cppdocs/api/function_namespaceat_1a42d0b0697c2ec3f50d30ee4d6805a372.html?highlight=from_blob

doxygenfunction: Unable to resolve multiple matches for function “at::from_blob” with ...

cc @yf225 @glaringlee @jlin27 @mruberry

[mruberry]

Thanks for reporting this issue, @elbaro. We would accept a PR fixing this.

[samestep]

This issue seems much broader than just at::from_blob: searching for "doxygen" on pytorch.org/cppdocs yields 780 pages, all of which seem to show the same error message. Is there an easy way to figure out when these messages were introduced? I'd try git bisect but I feel like that would take too long.

[glaringlee]

@samestep
This is probably since the beginning when we added c++ documents using doxygen + breathe + sphinx, and mainly happened in at namespace. We use breathe to translate doxygen comments, it doesn't work well on multiple function matching if you provide the function name only when it is an overloaded function (template specialization doesn't work well also).
I noticed that the higher version breathe might do this better, but I haven't tested yet. One possible fix is to add .rst and provide more specific doxygenfunction:: for ambiguous functions, but this is not a trivial fix, need to research on how to get all the function signatures, and whether need a codegen, etc.

Most people use C++ apis doc for NN Module and Optimizers which doesn't have multi function match issue (we take care of the doxygen comments within the code already). We got some similar issues before, we normally re-direct user to the cpp/h file to look at comments there at the moment.

[samestep]

@glaringlee Thanks for the clarification! That sounds reasonable, in which case I probably don't have the expertise at the moment to resolve the issue. One thing I'm curious about, though: @malfet found that in pytorch/cppdocs@6ed43dd, there were 579 pages with this error message (and the number has since increased to 780), whereas in its parent commit pytorch/cppdocs@e00f7e6, there were only 6 (one of which is at::from_blob). So while I'm guessing you're right about this particular function, the broader appearance of this error message seems to be more recent; perhaps caused by #41312?

[glaringlee]

@glaringlee Thanks for the clarification! That sounds reasonable, in which case I probably don't have the expertise at the moment to resolve the issue. One thing I'm curious about, though: @malfet found that in pytorch/cppdocs@6ed43dd, there were 579 pages with this error message (and the number has since increased to 780), whereas in its parent commit pytorch/cppdocs@e00f7e6, there were only 6 (one of which is at::from_blob). So while I'm guessing you're right about this particular function, the broader appearance of this error message seems to be more recent; perhaps caused by #41312?

@samestep Ah, I see what you mean now. I was not aware of this. @ezyang, can you advise? thx

[ezyang]

To unblock for the point release, let's just revert the exhale version update, I didn't actually need it for anything. But to resolve the situation fully we will need to figure out what changed in the upgrade and come up with a strategy to deal with it.

[malfet]

Tentatively grabbing for myself, let me have a look at what is going on with the version update (I don't think we can easily revert now, some of the packages gone stale)

[mattip]

It seems breathe is pinned at 4.19.2, the last release was 4.24. Maybe updating breathe would help?

[mattip]

Up to breathe 4.14 there were two "Unable to resolve multiple matches" warnings generated by breathe. In 4.15-4.16, there were 13 warnings. Versions 4.17-4.18.0 of breathe cause sphinx errors:

breathe/renderer/sphinxrenderer.py", line 1519, in visit_friendclass \
     assert ''.join(n.astext() for n in self.render(node.get_type())) == "friend class"

Version 4.19.0 has 576 warnings.

There are sphinx domain warnings about "Duplicate declarations" (that changed in a sphinx 3.3 to "Duplicate C++ declarations"). These went down from 1500 in pre-1.19-breathe to less than 10 in breathe v1.19.

Exhale (which is used to run doxygen to create the XML files, then to call breathe to parse those XML files into sphinx RST direcectives) also emits a number of long warnings about places it detects that generate the same sphinx directive for functions with different signatures. However it only finds 4 of these: c10::overflows, torch::autograd::to_output_type, torch::data::make_data_loader, and torch::python::bind_module.

It would seem we need a better parser in breathe and a way to distinguish between the overridden declarations in sphinx in order to better fix this.

[ezyang]

RIP. Maybe we should fund some improvements to breathe :/

[mattip]

I distilled the failure down to a test I could add to breathe in the issue linked above this comment, maybe the author will be able to suggest a fix.

[mattip]

I added a fix to the breathe PR breathe-doc/breathe#605. Using that cleans up all but 14 of the "Unable to resolve ..." warnings. The ones left are

at::from_blob(void *, IntArrayRef, IntArrayRef, const std::function<void(void *)>&, \
                       const TensorOptions&, const c10::optional<Device>)
at::from_blob(void *, IntArrayRef, IntArrayRef, const TensorOptions&)
at::from_blob(void *, IntArrayRef, const TensorOptions&)
at::from_blob(void *, IntArrayRef, const std::function<void(void *)>&, const TensorOptions&)
torch::from_blob(void *, at::IntArrayRef, const at::TensorOptions&)
torch::from_blob(void *, at::IntArrayRef, const Deleter&, const at::TensorOptions&)
torch::from_blob(void *, at::IntArrayRef, at::IntArrayRef, const at:
torch::from_blob(void *, at::IntArrayRef, at::IntArrayRef, const Deleter&, const at:
torch::schema(const char *, c10::AliasAnalysisKind)
torch::schema(const char *)
torch::init(Func&&)
torch::python::bind_module(py::module, const char *)
torch::python::bind_module(py::module, const char *)
torch::detail::constructSchemaOrName(const char *)

14 is better than 576.

[vermeeren]

@mattip and others, I've just released Breathe 4.25.0 which should solve the (majority of) the problem. Thanks again Matti for the initial patchset and tests!

Edit: breathe-doc/breathe#606 was the fix used in the end.