[docs] Add new process to generate query help for help site

[jf205]

This PR adds a process for generating query help documentation to use in on the CodeQL docs site so that we can move away from the Wiki.

For this new process I've added:

• a Python script that renders the query help as markdown. It does the following:

  • runs codeql generate query-help for the queries in a selected query suite for all the languages we currently support to produce a markdown file for each query help article
  • inserts metadata, a link to the query source file and information about the query suites the query is selected by into each article
  • create toc-<lang>.rst files which list the query help files for each language. These toc files are part the Sphinx project to render the query help as HTML.

• files needed to add a new Sphinx project. This includes:

  • a configuration file. It's similar to the conf.py files used in our other Sphinx docs, but it specifies recommonmark as an extension to build HTML from markdown.
  • a landing page for the project
  • a landing page for the query help for each language. These landing pages use includes to insert the TOCs generated by the docs generation script.

• an actions workflow file which:

  • checks out the CodeQL repos
  • obtains and sets up the CodeQL CLI
  • runs the docs generation script
  • uses a Sphinx action to convert the markdown file to HTML.
  • uploads the HTML artifacts to the actions workspace. We can then download the artifact for use on the site.

You can see a preview of the HTML output here.

[jf205]

When I ran this workflow with just the "code-scanning" and "security-extended" query suites it took about 12 minutes. I added "security-and-quality" in 74d274c and now it takes 32 minutes 🤦

[jf205]

@sj, @esbena: the markdown files generated by the script now have filenames based on the id property. The short language code (e.g. cpp, cs etc) isn't included in the name because the files are already sorted into language-specific directories. See this preview for examples of the URLs for each query help page. Please let me know if this works for you 😸

[esbena]

Thanks. I think using id for file names is the right choice, but I think you should keep the short language code as well. There is nothing elsewhere that prevents queries from diverging from the convention of having a short language code as a prefix of the id.

[jf205]

Thanks. I think using id for file names is the right choice, but I think you should keep the short language code as well. There is nothing elsewhere that prevents queries from diverging from the convention of having a short language code as a prefix of the id.

Ok, done 👍

For each query help page, I've also added a line at the bottom of the query metadata section that notes which code scanning query suite the query appears in.

[jf205]

We are nearing the end of the content work for the new docs site, so I'd love to move this along if possible. It's not super important to get the processes merged, but the output does need to be finalized before we use it on the new site.

With that in mind I'd like to pull in a review from @sj to make sure that we are on track from a product point of view please.

[shati-patel]

Yay, this looks great @jf205! 🎉 I've added a couple of minor (mostly optional) suggestions and questions.

I haven't looked too closely at the script/workflow, but let me know if there's anything specific you'd like me to test or review!

[jf205]

Thanks for the docs review @shati-patel. Following your comments I've:

• reworked the main landing page so that the TOC doesn't look so odd
• updated the titles on the landing page for each language (example)
• changed readme.txt -> readme.md and excluded it from the build
• Unrelated: I've also changed the sphinx builder to dirhtml. This means that the URLs for the query help articles won't have the .html extension, which i think is nicer (we already use this for the other CodeQL docs).

[shati-patel]

Thanks for the tweaks! All LGTM from a docs point of view, but I'll let someone more qualified take a look at the scripts 👀

[adityasharad]

Thanks! I'll do a proper review this week. I think some of the functionality you've written should eventually make its way into codeql generate query-help, and then we can run it internally against all the relevant repos, but that doesn't need to hold up this PR.

[jf205]

Thanks! I'll do a proper review this week. I think some of the functionality you've written should eventually make its way into codeql generate query-help, and then we can run it internally against all the relevant repos, but that doesn't need to hold up this PR.

👍🏻 thanks @adityasharad

Merging this now 🎉

[adityasharad]

Quick follow-up PR: #4770