Added JSDoc

[RunDevelopment]

This PR adds generated JSDoc documentation to Prism (#1777) using the Minami theme. The generated docs directory will be accessible from prismjs.com/docs.

This contains the doc of all highlight functions, insertBefore, extend, all hook functions, and Prism.Token. I used the docs from our extending page as a base and added a few sentences.
This includes complete type information for all documented function (except for hooks where the type of the env parameter depends on the hook).

Because JSDoc requires a README for the index.html, I used Prism's README. We can also make one specifically for the doc. (I just didn't know what to write.)

Interactive demo. (Will only be available during the time this PR is open.)

Right now, only prism-core is used to generate the doc. Other documented plugins and languages will need some adjustment before JSDoc can handle them. And there's also the question of whether these components should even be part of the doc.

Implementation

I used gulp-jsdoc3, so npx gulp will also re-generate the doc (this takes less time than minifying the language definitions, so the build process won't take too much longer).

Because of some ... features of JSDoc, I had to make some debatable adjustments to the doc generation process:

1. JSDoc only adds the newly generated doc. It doesn't remove outdated files, i.e. if the file documented was deleted or renamed. So to fix this, gulp will first delete the whole doc directory.
2. You cannot disable the generation time in the Minami theme, so I use a regex on the HTML files to remove the date.
3. JSDoc also includes a few font files which our theme doesn't use, so those get removed.

Another rather unfortunate thing is that JSDoc doesn't use Prism to highlight source code.
We could make it with a few regex replacements but that would only add another point on my list of debatable decisions.

---

This resolves #1777.
This resolves #1555.
This resolves #1774.

[RunDevelopment]

/cc @mAAdhaTTah @LeaVerou @Golmote

[ExE-Boss]

It works decently with TypeScript’s checkJs, although that doesn’t yet support @memberof.

[RunDevelopment]

I looked a little into this and there don't seem to be any workarounds for this, are there?

[ExE-Boss]

The only solution would be if TypeScript would add support for @memberof or if we would migrate portions of this project to TypeScript.

[RunDevelopment]

Well, there is an issue so the hope is there...

I'm actually quite a fan of TS but I think that:

1. Porting the project for the sake of JS-Check isn't worth it.
2. TS isn't necessary for Prism.

Also, TS is not as popular as JS, so some people might get held back from contributing.

[ExE-Boss]

I have a few improvements to the JSDoc:

[ExE-Boss]

I prefer using the simple string[] form instead of Array<string>.

[RunDevelopment]

That's a good idea. I'll change all similar occurrences.

[RunDevelopment]

Ok. It's done.

After this comment, I gave it another try. This time with the goal of just getting something that works even if it barely works.

The main problem with marrying TS and JSDoc was that TS couldn't do namespaces and wasn't smart enough to figure out some things about the structure of Prism instances. That was a year ago. TS has since become a bit smarter and we had a PR that made it easier for TS to figure out Prism. So TS could now understand Prism. One problem solved.
Now namespaces. Screw namespaces! I just didn't use them. Without using namespaces, the generated doc is somewhat odd because all classes and type definitions are part of the global namespace. This isn't really scalable documentation-wise but TS can understand it, so sure why not. We only have 5 types in global space right now, so it isn't a big deal.

After some tinkering and hacking, I finally had something that worked with both TS and JSDoc.
I also used a different theme that is a lot more configurable and just plain better than the previous one.

It finally works and it even looks kinda nice. Demo.

The hacking

I still have a small CSS file to change/fix the theme but it's better now.

The real hacking comes with JSDoc's parser. It can't deal with some TS syntax (e.g. intersection types and type imports) and will throw an error.
To fix that, I added a handler for JSDoc parser events that will just remove all type imports via a small regex and hopes that it works then. This is nothing more than a hack but it works and I don't plan to change anything about it because the JSDoc team doesn't intend to support this syntax, so this is the best I can do.

TS

One of the main intentions was to get good type checking, so did that turnout?

Amazing. It works really well. You have to kinda work around some issue but it's doable. My goal of moving Prism toward strict typing via doc comments is almost fulfilled. There are only 2 type problems left in Core. For that alone, I want to see this merged.

Other nice things

Documentation is opt-in. You can add doc comments however you like and for whatever you like. Only if you mark a doc comment as @public will it be part of the generated documentation site.

Thanks to the new theme, we now have a search bar and we can add your own links to the nav.

Not so nice things

The doc site includes a "jump to source" function. This means that every change to Core will result in changes in the doc site. To mitigate this a little, the gulp docs task is not part of the default build task. If you make changes to the doc, you have to rebuild the site manually. Not nice, but not too bad either.

Closing thoughts

It's not perfect but it works. You can write some doc and it will generate a nice site for you.

I criticize JSDoc and my solutions to all the problems I had to deal with a lot but I'm actually really happy with this result.

Let me know what you think.
Awaiting review.

[mAAdhaTTah]

In another project, I actually wrote everything in JS and managed to generate .d.ts files from them. If we're going to put all this work into writing & maintaining the documentation, is it possible to do that here? Not a requirement for this PR, by any means, just throwing it out there.

[RunDevelopment]

is it possible to do that here?

Should be possible. Let's make an issue of this after this PR is through.

[mAAdhaTTah]

I'm generally ok with punting off defining TokenStream to a future PR and not blocking this on it.

[RunDevelopment]

Well, if we really narrow down the definition of a Token after this, then that's a breaking change.

Actually, let's just take the current definition of Token and TokenStream as they are in here right now. This will resolve #2071.

[mAAdhaTTah]

I'm definitely on board with these API docs in general. I would love to eventually get to a world where we generate both types of docs into a single cohesive website, but this works for now.

[RunDevelopment]

I should probably point out that we now have API doc, but I haven't actually linked to it yet... Let just me do that real quick.

[RunDevelopment]

Minor change: I narrowed down the type definition of a Token (reason).
I'm gonna merge now.