Support incremental changes to doc

[AtomicMaya]

Reasons

Generating docs on large scale projects (ie. real world Flutter apps) takes time.

Proposal

Would it be possible to have an incremental doc system ? (on a file level, not necessarily a method level)

It would perhaps only generate docs for a file if the checksum of the file has changed, and delete docs for a file that does not exist. These checksums could be stored in a JSON file at doc/api, possibly following a file tree structure.

This functionality could be flag dependent ie. --incremental for the incremental update and --invalidate for deletion.

[srawlins]

I like the idea. This is a good proposal for a shorter turnaround on big projects like Flutter.

[jcollins-g]

While this seems like it would do something, docs do not have a one-to-one correlation with files (in many cases, it isn't even close to a one-to-one correlation). A truly live-updating or incremental documentation system will have to break away from dartdoc's current design need to understand and compute everything even before writing a file.

[AtomicMaya]

Most doc is class-based in Dart, no ? If people structure their projects with regards to compile and doc optimization (ie. classes when necessary, and ideally in other distinct files), then it could work, but that would require core changes indeed...

But if it works more like drawing a graph first -- also perfectly valid, maybe it could memorize/store graph states and just maybe update the relevant subtrees ? I'm pretty sure the time intensive task isn't the file I/O for the classes but the preceding step (I/O takes an acceptable 20-40s, but doc compilation takes north of 5 minutes) (again, just throwing out ideas, I actually don't know how it currently works, but could it stick with what @jcollins-g explained of dartdoc's functionality ?)

[jcollins-g]

@AtomicNicos Dartdoc's general strategy for the problem has evolved over time and isn't optimal, but it is closer to the graph case you alluded to. Very briefly, dartdoc's general strategy for generating docs is:

1. Build a representation of the source code using the Dart Analyzer, including an element for every defined class, typedef, variable, method, library, etc.
2. Build a secondary representation within Dartdoc that expands every single possible context every element from the analysis model could possibly exist in. For example, a class member might be in multiple libraries if the class is reexported, or even in multiple classes via inheritance. This is the the first part of "drawing the graph" you alluded to.
3. Calculate (with a lot of heuristics) a subset of Responsive (mobile-first) layout #2 for dartdoc canonicalization, so that we only document an analyzer element in the "one right place", with all other duplicates of that element in the graph knowing how to link to the corresponding canonical element.

The way we do two and three is not optimal, because in many instances we're recalculating parts of the "graph" multiple times -- [ModelElement]s do not know about neighbor elements that are "more canonical" than they are, instead it calculates the direct route to the canonical element for each one. We also have to process a lot of data that is then discarded because of how macros and templates are defined, which is something we plan to change.

A proper refactor to accomodate incremental updates would more directly allow for analysis elements being updated, and record enough data to know what parts of the graph need to be recomputed. A big project, I think.