New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doxygen: Misc ideas/proofs-of-concept #79

Open
marzer opened this Issue Jan 2, 2019 · 10 comments

Comments

2 participants
@marzer
Copy link

marzer commented Jan 2, 2019

Heyo,

Over the last few months of using m.css I've been working on a 'post-process' script to do some further work on the generated html, and now it's polished enough that I figure you might appreciate it as a set of ideas or proofs-of-concept.

The example implementation (and output html) is here: achilles-doc.zip. The post-process script is rebuild-documentation.py. I don't claim to be any sort of python guru so apologies if there's anything in there that is a bit odd, heh. It uses BeatifulSoup for html parsing, so if you want to actually run it with your own input files, you'd need to install the packages beautifulsoup4 and html5lib. If by some miracle you do want to lift any code out of it, go for it- consider it public domain :)

In no particular order:

Tags for inline namespaces

I know that Doxygen doesn't (currently) emit any sort of metadata indicating that a namespace is inline, so my implementation uses a simple list of namespaces, which could easily be added as one of the additional M_ configuration parameters.

Examples:

image
image

Relevant parts of the proof-of-concept:

  • rebuild-documentation.py:
    • InlineNamespaceFix1()
    • InlineNamespaceFix2()
    • InlineNamespaceFix3()

Collapsing std::enable_if (and aliases of it) in SFINAE contexts

Mostly just because these don't add much value to a function's documentation in the first instance, and tend to be quite verbose- I've allowed them to be expandable on click using some very minimal javascript.

If you pour through my code you'll note that I have to do some pretty gnarly regex to find the relevant part between the relevant '<' and '>' characters, and comes with some pretty annoying limitations:

  • It only works if the constraint is the last member of the template arguments
  • It fails if the constraint contains any '<' or '>' characters (e.g. sizeof(Foo) >= 4)

These would likely be much be easier to get around when the html was first generated, but also could be more easily automated after-the-fact if the contents of the constraint were wrapped in a <span class="sfinae"> or the like.

Examples:

image
image

Relevant parts of the proof-of-concept:

  • rebuild-documentation.py:
    • EnableIfFix()
  • Achilles.js

Contextually linking to external documentation

Things like members of the std namespace, types from <windows.h>, etc. I've used a different colour for these to denote 'this is external documentation', though I certainly don't think that's a necessity.

Mine just uses a list of compiled regular expressions and the URI's they relate to, but I guess it might need to be user-customizable, which has the potential to be quite complex. Also this particular step is easily the slowest one since it needs to check that it's not injecting links into existing <a> tags- probably unavoidable?

Examples:

image
image

Relevant parts of the proof-of-concept:

  • rebuild-documentation.py:
    • ExtDocLinksFix()

Lifting calling conventions out into labels

This one is pretty windows-specific, I guess.

Also, in addition to lifting the calling conventions, the code responsible for this also takes care of some edge cases where existing specifiers like constexpr and noexcept don't get properly converted into lables. I haven't dug deeply enough to tell you if that's because doxygen parses them inconsistently, or if it's from somewhere within m.css, but would be happy to help dig into this further.

Example:

image

Relevant parts of the proof-of-concept:

  • rebuild-documentation.py:
    • ModifiersFix1()
    • ModifiersFix2()
  • html/namespace_achilles_1_1_math.html

Generate sensible defaults for defaulted/deleted rule-of-six

The 'only generating output for explicitly documented code' goal is an amiable one, but does tend to run afoul of some common boilerplate-avoidance practices. For example, if I have the following macro to make immovable type definitions simpler:

#define IMMOVABLE(TYPE)						\
	TYPE(TYPE&&) = delete;					\
	TYPE& operator=(TYPE&&) = delete;		\
	TYPE(const TYPE&) = default;			\
	TYPE& operator=(const TYPE&) = default

I can't then add any documentation to these operations to ensure that someone reading the documentation would see deleted or defaulted.

Some default behaviour for these would be great; maybe even with some default briefs? Something like:
image
(I haven't actually implemented this one; the example was just some browser DOM editing)

Automation hints/hooks

Providing more automation hints would be great, since some of the workarounds I've used are quite brittle. One good example is the parsing of Function Documentation; each function gets it's own <section>, but the <section> tags themselves don't have any sort of ID or class that marks them as a function definition. I fetch them by first looking for a <h2>Function Documentation</h2> then collecting all of it's <section> siblings, but that feels brittle as hell.

Relevant parts of the proof-of-concept:

  • rebuild-documentation.py:
    • ModifiersFix2()

That's about it. I hope this is useful?

@mosra mosra added this to TODO in Doxygen theme via automation Jan 2, 2019

@mosra

This comment has been minimized.

Copy link
Owner

mosra commented Jan 2, 2019

Jesus 😮 😍

This is a very fine collection of very useful additions. Coincidentally I'm just about to publish a larger set of new features and improvements and some of these might be already partially solving a bunch of what you got here (in particular, the noexcept-related stuff). I'll have a blog post with details soon.

Just give me some time to reply properly to all of the above :)

@marzer

This comment has been minimized.

Copy link

marzer commented Jan 2, 2019

No dramas. I've just realized that the script does some pretty hamfisted stuff with file paths that will only work on Windows (b.c. of case insensitivity), so if you want to test it on Unix you might have to do some hackery :P

@mosra

This comment has been minimized.

Copy link
Owner

mosra commented Jan 3, 2019

Argh! Of course I forgot to reply, sorry 😅

(Blog post here: https://blog.magnum.graphics/meta/improved-doxygen-documentation-and-search/)

Tags for inline namespaces

Yes, definitely, I think this could go as an always-enabled feature, so no need for an option. I didn't have any inline namespace myself yet, that's probably the main reason this wasn't done yet. I hope this is easy to extract from the Doxygen XML files? TODOs for myself:

  • show inline for inline namespaces in namespace docs
  • show inline for inline namespaces in namespace and class trees
  • ... and while at it, show also final for classes in class trees

Collapsing std::enable_if (and aliases of it) in SFINAE contexts

I think the proposal in #56 could help solving this problem as well, in a more general way. It's originally just about hiding these completely, but I like the idea with JavaScript collapsing too, it could be able to do both. What do you say?

Contextually linking to external documentation

Do you know about Doxygen tag files? Those provide a way to link to external documentation (and to make this completely great, cppreference.com provides such a tag file). I'm using this quite heavily in the Magnum documentation, which links to external Corrade docs (a utility library) and also to STL -- see the Animation::TrackView class, for example:

image

I'm using a non-bold font for to distinguish the external references, experimented also with a different color but that turned out to be way too distracting.

Lifting calling conventions out into labels

I hope I ironed out most of the corner cases with noexcept / constexpr in 2e2b57f, please tell me if you still come across cases that are broken. Doxygen just gives me the full function signature and I have to extract these out of there, so it's very possible I'm making some errors there.

For the additional keywords, I can think of also stuff like new C++ attributes such as [[noreturn]], [[nodiscard]] etc. Since the list is too large and it would both not cover everything for some and be overly verbose for others, I think the best way would be a config option where you could specify what keywords you'd want to have parsed.

  • TODO for myself: provide such a setting

Generate sensible defaults for defaulted/deleted rule-of-six

Hmm. Doxygen actually has some pretty powerful preprocessor and I think it could be convinced to expand the IMMOVABLE macro before parsing the comments. So you could have the macro as:

#define IMMOVABLE(TYPE)				 \
        /** @brief Moving is not allowed */      \
	TYPE(TYPE&&) = delete;			 \
        /** @brief Moving is not allowed */      \
	TYPE& operator=(TYPE&&) = delete;	 \
        /** @brief Cop constructor */            \
	TYPE(const TYPE&) = default;	         \
        /** @brief Copy assignment */            \
	TYPE& operator=(const TYPE&) = default;

(Disclaimer: I never tried this myself. Maybe it doesn't work at all.)

Another idea I'm thinking about is that it could be possible to detect what the function is (whether a copy/move constructor or a copy/move assignment) and then providing some implicit comment for it, in case it's defaulted/deleted. But knowing how bad the parser sometimes is (and thinking about the template-heavy cases), I'm not sure if it's at all possible to implement this robustly.

Automation hints/hooks

Hmm, I have to think about this a bit more. I have a similar case, need to integrate for example the following source code annotation, which adds colored rectangle next to custom RGB literals, but didn't find a way to do that cleanly yet without hardcoding stuff directly:

image

I think this should be done on the Python level, rather than on the resulting HTML (which is too late, in my opinion). There's a well-documented hierarchic structure coming out of the parser which is then fed to Jinja2 templates, so the hooks could be added there I think. Somehow. Don't have an idea how to do that yet.

@marzer

This comment has been minimized.

Copy link

marzer commented Jan 4, 2019

RE inline namespaces: I had a cursory look through the doxygen XML, did a CTRL+F for 'inline', and didn't find anything relating to the namespaces. That's the extent of my investigation, though; there may have been some other indication that I missed...

RE collapsing templates: Am I reading that correctly, that there be some MCSS-related macros in the C++ source itself? I'm not sure how I feel about that idea, to be honest- inline documentation can be pretty noisy already. I suppose doing it that way means it's opt-in, which is nice. I certainly think there's value in having it hidden by javascript and allowing it to be toggled by the user, since you could then apply some basic parsing to the constraint to make them more readable, like this:
image
(this was just DOM-editing, I haven't written this parsing logic, since it's pretty hard to do once the document is HTML)

RE external documentation links: I was aware of tag files, but thought the need to generate them was a bit of a non-starter. Using some provided by cppreference.com obviously solves that issue! One thing my approach does that I like is handles plurals etc correctly:

Does the tag file approach handle things like that? A minor detail, I suppose.

RE calling conventions and labels: Adding a configurable list of symbols sounds great. At some point over the weekend I'll regenerate our documentation with out the label-related post-processes and see if I can find any missing corner cases for you :)

RE defaults for rule-of-six: I had not even considered adding comments to the macros themselves, to be honest. I'll try that this weekend and see what happens.

@mosra

This comment has been minimized.

Copy link
Owner

mosra commented Jan 4, 2019

inline namespaces

Argh, that's what I feared. Could you try generating the stock HTML output and see if it's there at least? If yes, then it's just not propagated to XML (which would not surpise me at all). If not, then we would need to open a feature request and wait a year or so until it gets implemented, because no way in hell I'm touching their C++ parser. Nope. ☠️

collapsing templates

Yes, that's what I meant, a new macro that gets recognized by m.css. I am doing a similar thing now, see here, but the #ifdef also gets pretty noisy. Doing C++ parsing on the theme side (in python) is also a thing I don't ever want to do ☠️, because unless you have the full context at that point, you'll almost certainly get it wrong. Imagine parsing pathological cases like this:

tempate<std::enable_if<Foo<Bar<Fizz, Buzz>::A>>::value

... is it a type Bar<Fizz, Buzz> with a member ::A or is it template<bool, bool> struct Foo, taking two parameters Bar < Fizz and Buzz > ::A? You'll never know unless Clang has your back. This would probably need to wait until I have Doxygen replaced with Clang-based parser and I can gather all the context easily. Sorry.

external docs

what about a list of @ref std::string_view "std::string_views"? ;) Other that, no, it doesn't do any such thing, you have to be explicit.

labels

defaults for rule-of-6

thanks in advance! 👍

@marzer

This comment has been minimized.

Copy link

marzer commented Jan 5, 2019

inline namespaces

Guess I should have mentioned that I actually tried the stock HTML output first, didn't see any inline on namespaces, which is what led me to quickly sussing the XML... 😢

labels

Your recent version is almost perfect. The only edge cases my post-process catches now (ignoring the custom calling conventions) are functions that return decltype(auto), e.g.:
image
(source xml: namespace_achilles.xml.txt)

deleted rule-of-six

I tried adding comments to the macros as in your example, with SKIP_FUNCTION_MACROS = NO and EXPAND_ONLY_PREDEF = NO, and nada :'(

@mosra

This comment has been minimized.

Copy link
Owner

mosra commented Jan 5, 2019

inline namespaces

Feature request submitted: doxygen/doxygen#6741

labels

Oh what the ☠️ 💥 💣. It attempts to "fix" the trailing return by moving the whole function suffix in front, including the constexpr:

<type>decltype(auto) constexpr</type>
<definition>decltype(auto) constexpr Achilles::Coerce</definition>

I bet this will be the case for all other keywords when they go together with trailing return types.

EDIT: oh, actually, looking again, the constexpr leaks all the way to the type. So I think parsing both <definition> and <type> for extra trailing keywords should do the trick, hopefully.

  • So I have some fun to do.

rule-of-six

Oh, too bad. Then the only option would be a "is it a constructor" / "is it an operator=" detection... and that will be non-trivial I'm afraid.

@marzer

This comment has been minimized.

Copy link

marzer commented Jan 9, 2019

Well, good to hear that the labels thing seems a relatively straightforward thing to implemement.

As for the rest of it, I'm totally happy to keep building additional stuff on using a post-process, since m.css has already made that vastly easier. I might have a go at the rule-of-six thing myself by injecting a separate pre-process step on the generated xml before invoking m.css.

edit: come to think of it, that would be a better way of handling the inline namespaces thing too, really. If we assume that at some point between now and the heat death of the universe the doxygen team will inject inline="yes" as part of the namespace's <compounddef>, would you be willing to give m.css the ability to consume that?

@mosra

This comment has been minimized.

Copy link
Owner

mosra commented Jan 9, 2019

injecting a separate pre-process step on the generated xml

When I get back to this (later this week, right now I'm working on stuff that's not related to m.css), I'll have a look. The idea got a bit more stable in my head and I think the detection could be fairly doable. Ugly, but doable.

If we assume that at some point between now and the heat death of the universe the doxygen team will inject inline="yes" as part of the namespace's <compounddef>, would you be willing to give m.css the ability to consume that?

You mean doing that before they have the corresponding code integrated? I would have no way to write a test for this and things that are not tested tend to get broken.

@mosra

This comment has been minimized.

Copy link
Owner

mosra commented Jan 11, 2019

Um, just realized, for the rule-of-six: as far as my experience goes, Doxygen always warns about undocumented functions unless it's a parameterless constructor or a destructor. So even if I implement some copy/move constructor/assignment detection on the m.css side, Doxygen will still warn about an undocumented function. Since (at least for me and a bunch of other users) the goal is to have a completely quiet output (and the script can't prevent Doxygen from complaining), I'm not sure if such a detection is worth implementing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment