Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for downloading packs from GHES #1221

Merged
merged 14 commits into from Sep 8, 2022

Conversation

aeisenberg
Copy link
Contributor

@aeisenberg aeisenberg commented Aug 29, 2022

This change adds:

  • new registries input in init action. This input is similar to the registries block in the qlconfig.yml file.
  • Change the downloadPacks function so that it accepts new parameters: - registries block - api auth
  • Generate a qlconfig.yml file with the registries block if one is supplied. Use this file when downloading packs.
  • temporarily set the GITHUB_TOKEN and CODEQL_REGISTRIES_AUTH based on api auth

TODO:

  1. integration test
  2. handle pack downloads when the config is generated by the CLI

Merge / deployment checklist

  • Confirm this change is backwards compatible with existing workflows.
  • Confirm the readme has been updated if necessary.
  • Confirm the changelog has been updated if necessary.

@aeisenberg aeisenberg requested a review from a team as a code owner August 29, 2022 19:58
@aeisenberg aeisenberg marked this pull request as draft August 29, 2022 20:16
@aeisenberg aeisenberg force-pushed the aeisenberg/ghes-pack-download branch 3 times, most recently from df07a38 to 8eba848 Compare August 29, 2022 20:48
Base automatically changed from aeisenberg/move-pack-download-to-init to main August 30, 2022 16:58
This change adds:

- new `registries` block allowed in code scanning config file
- new `registries-auth-tokens` input in init action
- Change the downloadPacks function so that it accepts new parameters:
    - registries block
    - api auth
- Generate a qlconfig.yml file with the registries block if one is
  supplied. Use this file when downloading packs.
- temporarily set the `GITHUB_TOKEN` and `CODEQL_REGISTRIES_AUTH` based
  on api auth

TODO:

1. integration test
2. handle pack downloads when the config is generated by the CLI
@aeisenberg aeisenberg force-pushed the aeisenberg/ghes-pack-download branch 3 times, most recently from 0308a5d to 01c45bf Compare August 30, 2022 18:03
@aeisenberg
Copy link
Contributor Author

Close and re-open to kick the PR checks into starting.

@aeisenberg aeisenberg closed this Aug 30, 2022
@aeisenberg aeisenberg reopened this Aug 30, 2022
@aeisenberg aeisenberg marked this pull request as ready for review August 30, 2022 19:06
@aeisenberg aeisenberg force-pushed the aeisenberg/ghes-pack-download branch from 01c45bf to db627c9 Compare August 30, 2022 19:39
@aeisenberg aeisenberg marked this pull request as draft August 30, 2022 20:31
@aeisenberg aeisenberg force-pushed the aeisenberg/ghes-pack-download branch from 976e4e2 to 7fd7f72 Compare August 30, 2022 21:20
@aeisenberg aeisenberg force-pushed the aeisenberg/ghes-pack-download branch from 7fd7f72 to 1d92118 Compare August 30, 2022 22:56
@aeisenberg
Copy link
Contributor Author

OK...this should work now. The integration test is not ideal since it won't fail if both the registries-auth-tokens input and the registries block are ignored. In this case it will just fall back to using the token input.

@aeisenberg aeisenberg marked this pull request as ready for review August 30, 2022 23:04
registries: |
[
{
"url": "https://ghcr.io/v2/",
"packages": "*/*",
"token": "${{ secrets.GITHUB_TOKEN }}"
}
]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hopefully not too much of a bikeshed, but why JSON rather than YAML here? YAML seems more consistent with other configuration files like the registries property in qlconfig.yml files and codeql-workspace.yml files.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thinking was that if we use JSON here, it's more explicit that this is another thing and we can say "JSON encoded string". People know what that means. However, if we use yaml here, we have to say: "well...you need to add an extra | since inputs only accept strings, but want to parse yaml".

I do agree, though, that yaml would be easier to read. I am quite ambivalent about this and maybe we should get more feedback on this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jf205 thoughts?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think i prefer YAML. If we have with a nice clear example in the docs then i think this will be fine for users.

Copy link
Contributor

@henrymercer henrymercer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly minor comments — this generally looks in good shape!

init/action.yml Outdated
required: false
registries:
description: |
A YAML string that defines the list of GitHub container registries to use for downloading packs. The string is in the following forma (the | is required on the first line):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A YAML string that defines the list of GitHub container registries to use for downloading packs. The string is in the following forma (the | is required on the first line):
A YAML string that defines the list of GitHub container registries to use for downloading packs. The string is in the following form (the | is required on the first line):

init/action.yml Outdated
packages: */*
token: ${{ secrets.GHCR_TOKEN }}

The url property contains the url to the container registry you want to connect to.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The url property contains the url to the container registry you want to connect to.
The url property contains the URL to the container registry you want to connect to.

init/action.yml Outdated

The token property contains a connection token for this registry.

If this input is missing, the `token` input is used for all pack downloads.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm surprised to read this — shouldn't we ensure that the token input is only ever passed to the current GitHub instance?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure what you mean here. If the registries input is missing, then we only connect to the current GitHub instance and we always use the token input to connect.

Would it be sufficient to say something like:

If you only need to download packages from this GitHub instance, use the token input instead.

?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification. I originally interpreted "the token input is used for all pack downloads" to mean that we would pass the token input to the registry for each pack. Your suggestion sounds a lot clearer 👍

{
url: "https://containers.GHEHOSTNAME1/v2/",
packages: "semmle/*",
token: "still-a-token",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor

Suggested change
token: "still-a-token",
token: "still-not-a-token",

go: ["c", "d"],
python: ["e", "f"],
},
undefined,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: Could add a // registries comment here so it's easier to see which input is undefined.

@@ -61,6 +61,23 @@ export interface UserConfig {

export type QueryFilter = ExcludeQueryFilter | IncludeQueryFilter;

export type RegistryConfig = SafeRegistryConfig & {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the terminology of safe vs unsafe is helpful in general, but two comments here:

  • I think it might be more important to name the unsafe one unsafe vs the safe one safe, so we think more carefully whenever we see Unsafe in the code.
  • It might be clearer (though more verbose) to use RegistryConfigNoCredentials and RegistryConfigWithCredentials for instance.

return registriesInput ? yaml.l(registriesInput) : undefined;
} catch (e) {
throw new Error(
`Invalid registries input. Must be a JSON string, but got: ${
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`Invalid registries input. Must be a JSON string, but got: ${
`Invalid registries input. Must be a YAML string, but got: ${

} catch (e) {
throw new Error(
`Invalid registries input. Must be a JSON string, but got: ${
e instanceof Error ? e.message : String(e)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This error message might contain credentials. Actions will mask out any secrets, but perhaps it'd be safe practice to avoid printing the input in the first place.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll remove the original error message from new new error.


function createRegistriesBlock(registries: RegistryConfig[]) {
// be sure to remove the `token` field from the registry before writing it to disk.
const safeRegistries = registries.map((registry) => ({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we enforce the absence of token using the type system?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can add an explicit return type to the function, but there is no way to ensure the token is removed from the object using the type system alone.

src/config-utils.ts Show resolved Hide resolved
@aeisenberg aeisenberg force-pushed the aeisenberg/ghes-pack-download branch 2 times, most recently from 0f06c56 to 0573cda Compare September 6, 2022 20:59
init/action.yml Show resolved Hide resolved
Copy link
Contributor

@adityasharad adityasharad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable, and I think you've addressed @henrymercer's comments. Some minor suggestions, and one reminder to add a CodeQL version number gate.

CHANGELOG.md Outdated
@@ -7,6 +7,7 @@ No user facing changes.
## 2.1.22 - 01 Sep 2022

- Downloading CodeQL packs has been moved to the `init` step. Previously, CodeQL packs were downloaded during the `analyze` step. [#1218](https://github.com/github/codeql-action/pull/1218)
- Allow CodeQL packs to be downloaded from GitHub Enterprise Server instances. [#1221](https://github.com/github/codeql-action/pull/1221)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we mention the registries field here briefly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see this entry is in the wrong version. I'll change that, too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd also like to link to the documentation when it is available, but that won't be out for a while.

init/action.yml Outdated Show resolved Hide resolved
src/config-utils.ts Show resolved Hide resolved
src/config-utils.ts Show resolved Hide resolved
@aeisenberg
Copy link
Contributor Author

Thanks for the comments. I addressed them all.

Copy link
Contributor

@adityasharad adityasharad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Assuming @jf205 is happy with the user-facing behaviour, this is good to go.

!(await codeQlVersionAbove(codeQL, CODEQL_VERSION_GHES_PACK_DOWNLOAD))
) {
throw new Error(
`'registries' input is not supported on CodeQL versions less than ${CODEQL_VERSION_GHES_PACK_DOWNLOAD}.`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just occurred to me: we may want to gate against the target GHES version too (if we're on GHES). Can be a follow-up; not critical for this PR.

@aeisenberg
Copy link
Contributor Author

@adityasharad would you mind taking another look. I added some better error handling and logic that ensures the registry urls end in/ to avoid the bug in 2.10.4.

@adityasharad
Copy link
Contributor

Code changes looks good; linter checks are grumbling.

Avoids a bug in 2.10.4. Also, add some better handling for invalid
registries blocks.
@aeisenberg aeisenberg merged commit 9ba4d50 into main Sep 8, 2022
335 checks passed
@aeisenberg aeisenberg deleted the aeisenberg/ghes-pack-download branch September 8, 2022 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants