CodeQL extension for VS Code docs update #12321

felicitymay · 2023-02-27T12:43:41Z

This is ready for review now. The changes have had a general review by stakeholders and the initial feedback addressed.

For a link to a workflow run that generates the preview docs, see the associated content issue or ask me.

jf205

I've left a round of suggestions just about the controller repository. I thought that might make them slightly easier to go through. I'll take a look a the rest of the docs now.

docs/codeql/codeql-for-visual-studio-code/running-codeql-queries-at-scale-with-mrva.rst

jf205

Here are some comments for the rest of the docs. I hope they are useful.

docs/codeql/codeql-for-visual-studio-code/running-codeql-queries-at-scale-with-mrva.rst

jf205 · 2023-02-27T14:36:16Z

docs/codeql/codeql-for-visual-studio-code/running-codeql-queries-at-scale-with-mrva.rst

+- Visibility of the repository
+- Whether analysis is still running (black, moving circle) or finished (green checkmark)
+- Number of stars the repository has on GitHub
+- How long ago the CodeQL database that was analyzed was created


This is actually the time the repo was last updated (which can be various different events, not just a commit). We've had several goes at choosing the wording and there is always this confusion 😬

Suggested change

- How long ago the CodeQL database that was analyzed was created

- How long ago the repository was last updated

🤔 - so the repository could have been updated 5 minutes ago, but the CodeQL database might be a week old because analysis is broken? Or do we only analyze repositories where the CodeQL database is reasonably up to date for the codebase?

so the repository could have been updated 5 minutes ago, but the CodeQL database might be a week old because analysis is broken? Or do we only analyze repositories where the CodeQL database is reasonably up to date for the codebase?

That's right. We have no clever ways of filtering repos based on when they were last updated.

Those metrics (stars, last updated) give the researcher some insight into the repos they are targeting: high star count = popular so impactful if a bug is found. Not updated recently = outdated and possibly vulnerable code.

Thanks for clarifying 👍🏻

docs/codeql/codeql-for-visual-studio-code/running-codeql-queries-at-scale-with-mrva.rst

jf205 · 2023-02-27T14:43:53Z

docs/codeql/codeql-for-visual-studio-code/troubleshooting-variant-analysis.rst

+
+In this example, the user ran variant analysis on a custom list of two repositories. One of the repositories was a private repository and could not be analyzed because they had a public controller repository. Only the public repository was analyzed. To analyze both repositories, they need to edit their settings and update the controller repository to a private repository. For information on how to edit the controller repository, see ":ref:`Customizing settings <customizing-settings>`."
+
+CodeQL extension error: Bundling pack failed


This is an edge case that our internal users run into, but external users are very unlikely to. I think it belongs in our internal docs rather than here.

docs/codeql/codeql-for-visual-studio-code/customizing-settings.rst

Co-authored-by: James Fletcher <42464962+jf205@users.noreply.github.com>

aeisenberg · 2023-02-27T18:15:24Z

docs/codeql/codeql-for-visual-studio-code/running-codeql-queries-at-scale-with-mrva.rst

+
+The core functionality of the CodeQL extension helps you write queries and run them locally against a CodeQL database. In contrast, variant analysis allows you to send your CodeQL query to GitHub.com to be tested against a list of repositories.
+
+When you run variant analysis against a list of repositories, your query is run against each repository that has a CodeQL database available to analyze. GitHub creates and stores CodeQL databases for thousands of public repositories, including every repository that runs code scanning using CodeQL. If you want to run variant analysis on your repositories, you need to enable code scanning using CodeQL on GitHub.com before adding your repository to a list for analysis (either default setup, or advanced setup using the CodeQL action). For information about enabling code scanning using CodeQL, see "`Configuring code scanning automatically <https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning-for-a-repository#configuring-code-scanning-automatically>`__."


Is it important to say how often the database is created and what branch is being used?

My understanding is that a database is saved only for the default branch and only a single database is saved. So, every time a code scanning is run on main, the database that gets created is stored and the previous database is removed.

This feels like information I would want to know as a user.

Good question @aeisenberg. I think stating that we only store the most recently created CodeQL database from the default branch should be sufficient.

mchammer01

@felicitymay - this LGTM ✨
I've left a few comments and suggestions for your consideration. The only thing blocking is the link to provide feedback.

docs/codeql/codeql-for-visual-studio-code/customizing-settings.rst

mchammer01 · 2023-02-28T08:28:08Z

docs/codeql/codeql-for-visual-studio-code/customizing-settings.rst

+      "variantAnalysis": {
+        "repositoryLists": [
+          {
+            "name": "My favourite JavaScript repos",


Nit: US spelling so should be "favorite"

mchammer01 · 2023-02-28T08:28:52Z

docs/codeql/codeql-for-visual-studio-code/customizing-settings.rst

+
+For information on the purpose and requirements for a controller repository, see ":ref:`Setting up a controller repository for variant analysis <controller-repository>`."
+
+You can also edit the items shown in the Variant analysis repositories panel by editing a file in your Visual Studio Code workspace called ``databases.json``. This file contains a JSON representation of all the items displayed in the panel. To open your ``databases.json`` file in an editor window, click the **{ }** icon in the top right of the variant analysis repositories panel. You can then see a structured representation of the repos, orgs and lists in your panel. For example:


You're not consistent in the way that you refer to the panel here:
Variant analysis repositories panel vs variant analysis repositories panel

I thought that I'd fixed all of these, thanks for spotting this one, I'll have another find and replace 💖

mchammer01 · 2023-02-28T08:30:17Z

docs/codeql/codeql-for-visual-studio-code/exploring-data-flow-with-path-queries.rst

@@ -30,6 +30,8 @@ Running path queries in VS Code
 #. Click each step to jump to it in the source code and investigate the problem further.
 #. To navigate the results from your keyboard, you can bind shortcuts to the **CodeQL: Navigate Up/Down/Left/Right in Result Viewer** commands.

+When you are ready to run a path query at scale, you can use the Variant Analysis Repositories panel to run the query against up to 1000 repositories on GitHub.com. For information on running analysis at scale across many CodeQL databases, see ":ref:`Running CodeQL queries at scale with multi-repository variant analysis <running-codeql-queries-at-scale-with-mrva>`."


Again about consistency: Variant Analysis Repositories panel (would be great to consolidate the capitalization and syntax we use for this panel)

mchammer01 · 2023-02-28T08:36:08Z

docs/codeql/codeql-for-visual-studio-code/exploring-data-flow-with-path-queries.rst

@@ -30,6 +30,8 @@ Running path queries in VS Code
 #. Click each step to jump to it in the source code and investigate the problem further.
 #. To navigate the results from your keyboard, you can bind shortcuts to the **CodeQL: Navigate Up/Down/Left/Right in Result Viewer** commands.

+When you are ready to run a path query at scale, you can use the Variant Analysis Repositories panel to run the query against up to 1000 repositories on GitHub.com. For information on running analysis at scale across many CodeQL databases, see ":ref:`Running CodeQL queries at scale with multi-repository variant analysis <running-codeql-queries-at-scale-with-mrva>`."


Should it be 1,000 here?

mchammer01 · 2023-02-28T08:50:40Z

docs/codeql/codeql-for-visual-studio-code/troubleshooting-variant-analysis.rst

+
+.. include:: ../reusables/beta-note-mrva.rst
+
+This article explains how to debug problems with variant analysis. That is, analysis run using GitHub Actions


Suggested change

This article explains how to debug problems with variant analysis. That is, analysis run using GitHub Actions

This article explains how to debug problems with variant analysis, that is, analysis run using GitHub Actions

docs/codeql/codeql-for-visual-studio-code/troubleshooting-variant-analysis.rst

mchammer01 · 2023-02-28T08:52:51Z

docs/codeql/codeql-for-visual-studio-code/troubleshooting-variant-analysis.rst

+    :width: 600
+    :alt: Screenshot of the "Variant Analysis Results" view showing a warning banner with the text "warning: Problem with controller repository" and "Publicly visible controller repository can't be used to analyze private repositories. 1 private repository was not analyzed." The "Show logs" button is highlighted with a dark orange outline.
+
+In this example, the user ran variant analysis on a custom list of two repositories. One of the repositories was a private repository and could not be analyzed because they had a public controller repository. Only the public repository was analyzed. To analyze both repositories, they need to edit their settings and update the controller repository to a private repository. For information on how to edit the controller repository, see ":ref:`Customizing settings <customizing-settings>`."


Suggested change

In this example, the user ran variant analysis on a custom list of two repositories. One of the repositories was a private repository and could not be analyzed because they had a public controller repository. Only the public repository was analyzed. To analyze both repositories, they need to edit their settings and update the controller repository to a private repository. For information on how to edit the controller repository, see ":ref:`Customizing settings <customizing-settings>`."

In this example, the user ran variant analysis on a custom list of two repositories. One of the repositories was a private repository and could not be analyzed because they had a public controller repository. Only the public repository was analyzed. To analyze both repositories, this user needs to edit their settings and update the controller repository to a private repository. For information on how to edit the controller repository, see ":ref:`Customizing settings <customizing-settings>`."

mchammer01 · 2023-02-28T08:53:24Z

docs/codeql/reusables/beta-note-mrva.rst

+
+    Note
+
+    Multi-repository variant analysis is currently available as a beta release and is subject to change. To use this feature you must upgrade the CodeQL extension for Visual Studio Code to a minimum of version 1.8.0.


Suggested change

Multi-repository variant analysis is currently available as a beta release and is subject to change. To use this feature you must upgrade the CodeQL extension for Visual Studio Code to a minimum of version 1.8.0.

Multi-repository variant analysis is currently available as a beta release and is subject to change. To use this feature, you must upgrade the CodeQL extension for Visual Studio Code to a minimum of version 1.8.0.

docs/codeql/reusables/beta-note-mrva.rst

docs/codeql/codeql-for-visual-studio-code/running-codeql-queries-at-scale-with-mrva.rst

jf205 · 2023-02-28T09:42:38Z

docs/codeql/codeql-for-visual-studio-code/running-codeql-queries-at-scale-with-mrva.rst

+
+The core functionality of the CodeQL extension helps you write queries and run them locally against a CodeQL database. In contrast, variant analysis allows you to send your CodeQL query to GitHub.com to be tested against a list of repositories.
+
+When you run variant analysis against a list of repositories, your query is run against each repository that has a CodeQL database available to analyze. GitHub creates and stores CodeQL databases for thousands of public repositories, including every repository that runs code scanning using CodeQL. If you want to run variant analysis on your repositories, you need to enable code scanning using CodeQL on GitHub.com before adding your repository to a list for analysis (either default setup, or advanced setup using the CodeQL action). For information about enabling code scanning using CodeQL, see "`Configuring code scanning automatically <https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning-for-a-repository#configuring-code-scanning-automatically>`__."


Good question @aeisenberg. I think stating that we only store the most recently created CodeQL database from the default branch should be sufficient.

Co-authored-by: James Fletcher <42464962+jf205@users.noreply.github.com>

jf205

One final suggestion, but otherwise this looks great (pending the last few missing links). Thanks @felicitymay (and @mchammer01 for the review) ❤️

docs/codeql/codeql-for-visual-studio-code/running-codeql-queries-at-scale-with-mrva.rst

…es-at-scale-with-mrva.rst Co-authored-by: James Fletcher <42464962+jf205@users.noreply.github.com>

docs/codeql/codeql-for-visual-studio-code/running-codeql-queries-at-scale-with-mrva.rst

…es-at-scale-with-mrva.rst Co-authored-by: James Fletcher <42464962+jf205@users.noreply.github.com>

felicitymay added 3 commits February 24, 2023 17:14

Add basic framework of new articles

874c362

First draft with bare bones

8af1207

More updates

70ed029

github-actions bot added the documentation label Feb 27, 2023

jf205 reviewed Feb 27, 2023

View reviewed changes

felicitymay changed the base branch from main to codeql-cli-2.12.3 February 27, 2023 14:41

jf205 reviewed Feb 27, 2023

View reviewed changes

felicitymay and others added 5 commits February 27, 2023 14:58

Apply suggestions from code review

9bee9c0

Co-authored-by: James Fletcher <42464962+jf205@users.noreply.github.com>

Fix typo in heading

29802af

Updates following initial PM review

49ec539

Tweaks to mention variant analysis in related articles

8e4eb9a

Revise troubleshooting article

1d13811

felicitymay marked this pull request as ready for review February 27, 2023 16:55

felicitymay requested a review from a team as a code owner February 27, 2023 16:55

aeisenberg reviewed Feb 27, 2023

View reviewed changes

mchammer01 self-requested a review February 28, 2023 07:39

mchammer01 approved these changes Feb 28, 2023

View reviewed changes

jf205 reviewed Feb 28, 2023

View reviewed changes

felicitymay and others added 2 commits February 28, 2023 09:58

Update for review feedback

b212537

Apply suggestions from code review

0af529e

Co-authored-by: James Fletcher <42464962+jf205@users.noreply.github.com>

jf205 approved these changes Feb 28, 2023

View reviewed changes

jf205 reviewed Feb 28, 2023

View reviewed changes

docs/codeql/codeql-for-visual-studio-code/running-codeql-queries-at-scale-with-mrva.rst Outdated Show resolved Hide resolved

Update docs/codeql/codeql-for-visual-studio-code/running-codeql-queri…

832dc27

…es-at-scale-with-mrva.rst Co-authored-by: James Fletcher <42464962+jf205@users.noreply.github.com>

jf205 reviewed Feb 28, 2023

View reviewed changes

docs/codeql/codeql-for-visual-studio-code/running-codeql-queries-at-scale-with-mrva.rst Outdated Show resolved Hide resolved

felicitymay and others added 2 commits February 28, 2023 12:29

Update docs/codeql/codeql-for-visual-studio-code/running-codeql-queri…

aba7440

…es-at-scale-with-mrva.rst Co-authored-by: James Fletcher <42464962+jf205@users.noreply.github.com>

Update docs/codeql/reusables/beta-note-mrva.rst

a7d221c

felicitymay merged commit 770326e into codeql-cli-2.12.3 Feb 28, 2023

felicitymay deleted the fc-7775-docs-update branch February 28, 2023 14:59

felicitymay mentioned this pull request Feb 28, 2023

Mergeback: codeql-cli-2.12.3 into main #12342

Merged

	- How long ago the CodeQL database that was analyzed was created
	- How long ago the repository was last updated


		In this example, the user ran variant analysis on a custom list of two repositories. One of the repositories was a private repository and could not be analyzed because they had a public controller repository. Only the public repository was analyzed. To analyze both repositories, they need to edit their settings and update the controller repository to a private repository. For information on how to edit the controller repository, see ":ref:`Customizing settings <customizing-settings>`."

		CodeQL extension error: Bundling pack failed


		The core functionality of the CodeQL extension helps you write queries and run them locally against a CodeQL database. In contrast, variant analysis allows you to send your CodeQL query to GitHub.com to be tested against a list of repositories.

		When you run variant analysis against a list of repositories, your query is run against each repository that has a CodeQL database available to analyze. GitHub creates and stores CodeQL databases for thousands of public repositories, including every repository that runs code scanning using CodeQL. If you want to run variant analysis on your repositories, you need to enable code scanning using CodeQL on GitHub.com before adding your repository to a list for analysis (either default setup, or advanced setup using the CodeQL action). For information about enabling code scanning using CodeQL, see "`Configuring code scanning automatically <https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning-for-a-repository#configuring-code-scanning-automatically>`__."


		For information on the purpose and requirements for a controller repository, see ":ref:`Setting up a controller repository for variant analysis <controller-repository>`."

		You can also edit the items shown in the Variant analysis repositories panel by editing a file in your Visual Studio Code workspace called ``databases.json``. This file contains a JSON representation of all the items displayed in the panel. To open your ``databases.json`` file in an editor window, click the { } icon in the top right of the variant analysis repositories panel. You can then see a structured representation of the repos, orgs and lists in your panel. For example:


		.. include:: ../reusables/beta-note-mrva.rst

		This article explains how to debug problems with variant analysis. That is, analysis run using GitHub Actions


		Note

		Multi-repository variant analysis is currently available as a beta release and is subject to change. To use this feature you must upgrade the CodeQL extension for Visual Studio Code to a minimum of version 1.8.0.

CodeQL extension for VS Code docs update #12321

CodeQL extension for VS Code docs update #12321

Uh oh!

Conversation

felicitymay commented Feb 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jf205 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jf205 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mchammer01 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jf205 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

felicitymay commented Feb 27, 2023 •

edited

Loading