Documenation Generation

[patrick-rodgers]

Looking to automate the creation of documentation pages for the core API. We will still need to author samples and examples. Research using JSDoc or similar to convert code comments into markdown pages.

[pbjorklund]

After writing some docs today I'm fully in agreement with that they should be auto-generated and we can have separate samples and usage examples.

Documenting by hand will work as long as the repo is small, but maintaining the docs will take quite an effort.

Got any idea's in mind of how to document usage patterns? Personally I like having some snippets to look at to see how things are supposed to work to get a reminder after reading the tests.

[alex-randall]

+1 for autogenerated documentation if at all possible. I experience the same pain myself with customer-specific reusable libraries that are written by hand. So difficult to keep it up to date, especially with the new ease of refactoring and growth of the library itself, which is inevitable.

Also, we should consider having relevant examples in the api documentation as well. I personally find some aspects of jQuery and moment's online documentation (in html) extremely helpful because they all have code examples/scenarios/deeper explanation/linking to related apis right in the api documentation. MSDN used to be that way, but I see less and less code examples these days (just "all up ones"). Code examples in the api documentation are extremely helpful to find that thing you need quick and copy and paste it into your code. To contrast: Angularjs's online documentation in my opinion is "ok", but kinda hard figure out how to use a specific thing (they focus on less examples, but focus on tutorials instead).

Another idea as well: perhaps long term the autogenerated (and merged) documentation should cover BOTH TypeScript and JavaScript with the ability to switch between them as needed

Finally, what output format of the documentation itself? I assume .md for now?
Long term would be awesome if it were integrated into MSDN. :)

Does anyone know options for autodocumenting typescript code as well as merging hand crafted content into it as well at build time?

[alex-randall]

FYI-I've done some research on this, there are some generators out there. Even found some for gulp!
But found that one or many of these are true for them:

• the output is not that good (pretty lame actually)
• only works with JavaScript
• the project has been abandoned a long time ago

So probably creating our own is our best option to achieve really good autogenerated documentation (with samples, and extra explanation merged into it as needed) that people can actually use.

Good news is that I've "started" the documentation autogeneration work as I have some familiarity with gulp. Nothing to pull or look at yet. Will come in the future when I have something to show and time permits.

Thanks!

PS-If someone beats me to it, so be it, I will not be offended. :)

Alex

[pbjorklund]

It would be great if you could note in this issue what solutions you have evaluated so everyone else can contribute if they have some tips before going the "roll our own route" :)

Kudos for sharing that you are working on this!

[alex-randall]

Thanks Patrik, rightly so and agree. Will reply back with details.

Patrick - please assign this to me (alex-randall)

[alex-randall]

As promised, here's NPM and gulp libraries I've looked at for generating GitHub flavored markdown (.md) from TypeScript files.

Summary:

As I already stated, most libraries support JavaScript only as the source. Some support TypeScript as the source, but output was html (not markdown). Some have other issues. See below.

Details:

http://typedoc.io/, https://www.npmjs.com/package/gulp-typedoc/, https://www.npmjs.com/package/typedoc (all same library)
TypeScript support, but only outputs html, not md, will look into it's json generation capabilities...

https://www.npmjs.com/package/typescript-docs
TypeScript support, but appears to only output html, not md

https://www.npmjs.com/package/docscript
TypeScript support, but tailored to ReactJs, not much documentation

https://www.npmjs.com/package/ts-quick-docs
TypeScript support, but only outputs json, seems very limited (only looks at const and interfaces)

https://www.npmjs.com/package/typescript-doc
TypeScript support, but no documentation and over a year old

https://www.npmjs.com/package/gulp-jsdoc-to-markdown/
no Typescript support (source files can only be JavaScript), generates github flavored markdown

https://www.npmjs.com/package/gulp-mox/
no Typescript support (source files can only be JavaScript)

https://www.npmjs.com/package/gulp-groc/
no Typescript support (source files can only be JavaScript)

https://www.npmjs.com/package/gulp-markdox2/
no Typescript support (source files can only be JavaScript)

https://www.npmjs.com/package/gulp-markdox/
no Typescript support (source files can only be JavaScript)

https://www.npmjs.com/package/gulp-documentation/
no Typescript support (source files can only be JavaScript)

https://www.npmjs.com/package/gulp-jsdoc3/
no Typescript support (source files can only be JavaScript)

https://www.npmjs.com/package/gulp-yuidoc/
no Typescript support (source files can only be JavaScript), plugin 2 years old

https://www.npmjs.com/package/gulp-docco/
no Typescript support (source files can only be JavaScript), plugin 2 years old

https://www.npmjs.com/package/gulp-jsdoc/
no Typescript support (source files can only be JavaScript), plugin stopped being maintained

Other Considerations and possible directions:

• Considered using http://typedoc.io/, taking it's html output, and then attempting convert the outputted html to markdown (maybe there's a library for that, but haven't looked yet). I anticipate it's not a simple thing because of the complexity of the html that is generated. Would have to do a custom coded "clean" process first on the generated html before it's converted to md.... Still might open to this if http://typedoc.io/ actually understands modern TypeScript and I can find a library that does the html to md conversion in an acceptable manner.
• Considered trying to get an abstract syntax tree from TypeScript itself... But it seems it's not really exposed or easy to consume http://stackoverflow.com/questions/20187928/typescript-get-syntax-tree http://stackoverflow.com/questions/18714501/how-can-we-get-the-syntax-tree-of-typescript

My Probable Direction:

Subject to change after a good night's sleep, time away, other experimentation, so I have time to reflect more, plus any other suggestions from anyone else :)

1. see if http://typedoc.io/ can really understand our modern TypeScript (probably point it to our pnp.d.ts definition file as the source to make it easier)

2. then,

see if the outputted html can be converted easily to md through other plugins,
if not, see how the output json file looks

3. if Core.JavaScript: Util.ts - other external dependencies #2 fails:

if the json file looks ok, then write a custom task to parse the json and generate .md files
if the json file does not look ok, then write a custom "pnp.d.ts parser" task (to json) and then a custom task that takes the json, parse it, and output to .md files

Of course if someone finds something else or has other ideas, I'm happy to take a look at them. Otherwise, I'm proceeding ahead since we need generated api docs, scenarios, and samples, asap and I care hugely about those things :)

Alex

[patrick-rodgers]

I am marking this with the assigned label to you @alex-randall. Appears I can't actually assign unless I add everyone to the team. Looking at those options as well.

[alex-randall]

Yep sounds good and thanks!

[alex-randall]

Update:

• Need help from others to see if we can use https://www.npmjs.com/package/gulp-typedoc/ (right now errors out for me, details below)
• Because of that failure, I have started "Plan B" (write a custom "pnp.d.ts parser" task (to json) and then a custom task that takes the json, parse it, and output to .md files) and will continue on this route unless someone gets the above to work. Good news is that so far, I have a .json file being output with some parsed information.

Details on gulp-typedoc errors:

test 1 gulp-typedoc test against pnp.d.ts errors:

Cannot find namespace 'SP'.
Cannot find name 'Promise'.

test 2 gulp-typedoc test against all .ts files under src and it's sub folders errors:

Cannot find name 'Promise'.
Cannot find name 'jQuery'.
Property 'format' does not exist on type 'StringConstructor'.
Cannot find name 'SP'.
Cannot find name '_spPageContextInfo'.
Cannot find name 'Response'.
Block-scoped variable 't' used before its declaration.`

Suspect:

• gulp-typedoc may not be using correct version of typescript compiler?
• may not be setting enough config options for typescript compilation?
• is this a symptom of ISSUE: No $pnp or pnp global object when using in Visual Studio (not Visual Studio Code) #36 ?
• something else completely different?

Would love to see if someone could get this working, other wise, I'm making progress with the hand crafted parsing route.

test 1 source code (sorry for weird formatting):

var gulp = require("gulp"),
typedoc = require("gulp-typedoc");

gulp.task("doctest1", function() {
return gulp
.src(["dist/pnp.d.ts"])
.pipe(typedoc({
// TypeScript options (see typescript docs)
module: "umd",
target: "es5",
includeDeclarations: true,

      // Output options (see typedoc docs) 
      out: "doctest1", 
      json: "doctest1/output.json",

      // TypeDoc options (see typedoc docs) 
      name: "my-project", 
      ignoreCompilerErrors: false,
      version: true
  }))

;
});

test 2 source code (sorry for weird formatting):

var gulp = require("gulp"),
typedoc = require("gulp-typedoc");

gulp.task("doctest2", function() {
return gulp
.src(["src/*/.ts"])
.pipe(typedoc({
// TypeScript options (see typescript docs)
module: "umd",
target: "es5",
includeDeclarations: true,

      // Output options (see typedoc docs) 
      out: "doctest2", 
      json: "doctest2/output.json",

      // TypeDoc options (see typedoc docs) 
      name: "my-project", 
      ignoreCompilerErrors: false,
      version: true
  }))

;
});

[ghost]

yo @alex-randall, been thinking about this (not coding!)....if the issue is a lack of support for TypeScript, why don't we create a version of the build which transpiles the TypeScript with comments and use JSDoc from there? I'm happy to pick it up

[alex-randall]

Hi @tobiaswest83 , thanks for bringing that up and asking!

I considered that too (should have notated that above), but if we go that route, then you lose the TypeScript richness. I personally don't find that acceptable, even for v1. ;)

Good news is that the parsing approach is going well! :) I've found that because it's the TypeScript definition file I'm using as the source of the parsing, everything is in one place and it's much more predictable what things are going to be. Working out nicely!

Can't wait to share it early next week! ;)

[patrick-rodgers]

My thinking here has changed a little. For this, since we are moving the core docs to the wiki - would it be easier to support an additional gulp command so that folks who pull the library can then do say "gulp docs" and produce locally the latest documentation based on the code comments. That would keep everything up to date while not filling the repo or forcing us to try and maintain a bunch of .md pages. Thoughts?

[alex-randall]

Hi @patrick-rodgers,

Good news is that I already created a gulp "docs" task and am going that exact route. :)

Regarding people not having documentation available on the site, I think that is a slippery slope... (especially for those who don't use gulp like site owners who just copy and paste code in Script Editor Web Parts).

Here's an idea: what if we put that docs task as part of the package task? That way when you gulp package it for release, the docs are built then too (then you can merge to the master branch)? This way the master branch will always have the latest docs? I really think we need the autogenerated api docs visible in web form.

Update from me 5/12/2016: autogen documentation going well and making progress. Will create a pull request when ready. Focusing on parsing the pnp.d.ts file as the source has yielded great success because things are in a predictable structure. I can't wait to wrap it up and share with everyone!

[patrick-rodgers]

I envision the docs being part of the sample project as well as in the wiki with the wiki being the primary source for docs.

[patrick-rodgers]

Going to reassign this one to myself for now - if anyone wants to help out, absolutely welcome it!

[patrick-rodgers]

Closing this based on #144, will open a new issue as help wanted for folks wanting to help with the styling. Thanks to everyone who helped in the docs discussion!