Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Make more workflows public #5480

Open
Marcono1234 opened this issue Mar 22, 2021 · 3 comments
Open

CI: Make more workflows public #5480

Marcono1234 opened this issue Mar 22, 2021 · 3 comments
Labels
question Further information is requested

Comments

@Marcono1234
Copy link
Contributor

Marcono1234 commented Mar 22, 2021

Currently, it appears the most relevant workflows are only run internally (and might even have to be manually triggered by you?).
This makes contributions for external, new contributors (and probably for you as well) time consuming and cumbersome because:

  1. The contributors have to wait for the internal workflow run
  2. You maintainers then have to tell the contributors what went wrong and how to fix it
  3. If they get something different wrong, this whole cycle repeats

It would therefore be good to make as much of the workflows public as possible. Currently this would probably already be possible for the following cases:

  • Check query format: Have a workflow which runs codeql query format --check-only on all modified .ql and .qll files.
    The workflow should then tell contributors to fix the problem by either runing codeql query format (possibly with non-destructive flags) or using the VSCode extension command "Format Document".
  • Check query help files: Have a workflow which runs codeql generate query-help on all modified .qhelp and .ql files. Possibly also checking that every query has a help file, and that every query help file which has no query must end with .inc.qhelp (see Use .inc.qhelp extension for included help files #5275). This could possibly done for every non-test directory in which files were modified. The output of the command can be discarded, the main point here it to check for syntax errors.
  • Compile queries: For all modified .ql and .qll files run codeql query compile --check-only --warnings=error.
  • Run query tests: For all modified .qlref, .expected and .ql files in test directories run codeql test run. If it is possible to detect, then it would also be good to run the tests when the corresponding tested query changes.
  • Run scripts in config folder.

It would probably be necessary to create for most of these separate workflows for the different languages to use the correct --search-path for the CodeQL CLI commands.

Maybe it would also be useful to have a workflow detecting changes to database schema files and verifying that files for the database upgrade exist. In the past there have been a few situations where this was overlooked or done incorrectly, see #6704 and #6803.

Note that I am not asking for all workflows to be made public. It is very reasonable to keep workflows which publish artifacts, e.g. publishing rendered query helps to codeql.github.com, internal. Similarly it will still be necessary to internally run all queries regardless of which files were changed, e.g. in case changes to a library affected dependent queries or their test output (this is probably also so time consuming that running it as GitHub workflow might exceed the quotas).
The proposal of this issue is only to enhance contributing and speed up the workflow for both contributors and maintainers.

@dbartol
Copy link
Contributor

dbartol commented Mar 23, 2021

TL;DR: We're working on it.

We've been steadily migrating our previous internal Jenkins workflows to Actions over the past 6-8 months, and plan to have nearly everything moved to Actions sometime this year. We'll still have some private Actions workflows, like the ones that build the closed-source portions of our toolchain, but our goal is definitely to make everything we can public. We don't like having to tell you what went wrong any more than you like having to wait for us to tell you what went wrong:)

Turnaround time for PRs is also something we're trying to improve as part of this process, since it helps us even more than it helps you. There are some challenges to compiling only changed QL code, especially since so many of our PRs change libraries that are imported by a large number of queries, but we're doing what we can.

@github-actions
Copy link
Contributor

This issue is stale because it has been open 14 days with no activity. Comment or remove the stale label in order to avoid having this issue closed in 7 days.

@github-actions github-actions bot added the Stale label Apr 14, 2021
@intrigus-lgtm
Copy link
Contributor

Would it make sense to leave this open as a tracking issue and remove the question label so the bot won't flag it again?

@github-actions github-actions bot removed the Stale label Apr 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants