diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..f68af8b --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,361 @@ +# CodeQL CLI changelog + +## Release 2.2.3 (2020-06-15) + +CodeQL CLI 2.2.3 is the same as version 2.2.2, but re-released with a new +version number because the `v2.2.2` folder on the download site +originally contained the 2.2.0 binaries instead of the correct 2.2.2 +ones. + +If you have downloaded release 2.2.2, and `codeql --version` correctly +identifies itself as being that version, you don't need to upgrade to +2.2.3. + +## Release 2.2.2 (2020-06-12) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.24) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.24 instance, you need to create them with release + 2.1.4. + +### Improvements + +- Query evaluations that time out due to a `--timeout` option are no + longer silently discarded. Instead `codeql` will terminate with exit + code 33. Commands that evaluate multiple queries will produce as + much output as they can even if one of the queries times out. + +## Release 2.2.1 + +There is no CodeQL CLI version 2.2.1. This version number was used +internally to work around restrictions in the CodeQL for VS Code +extension. + +## Release 2.2.0 (2020-05-29) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.24) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.24 instance, you need to create them with release + 2.1.4. +- Starting with this release, the CodeQL CLI can be downloaded either + as a single `codeql.zip` file containing the CLI for all supported + platforms, or as a `codeql-PLATFORM.zip` that contains the files for + just one platform. The single-platform zips are faster to download. + +### QL language improvement + +- QL now supports the definition of new types as type unions. This + feature currently allows unions of branches from an already existing + algebraic data type and unions of database types. + +## Release 2.1.4 (2020-05-26) + +This release corresponds to release 1.24.x of LGTM Enterprise, and +should be used when creating databases that will be uploaded to it. +Future CLI releases (numbered 2.2.x) may produce databases that are not +backwards compatible with this version of LGTM Enterprise. + +For all purposes other than creating databases for LGTM Enterprise we +recommend that you continue upgrading to newer CLI releases as they +become available. + +### Features added + +- A new `codeql query format` command exposes the QL autoformatter for + use on the command line. + +### Bugs fixed + +- `-J` command-line options that contain spaces now ought to work on + Windows. They still do not work reliably on Linux or MacOS, though. + +## Release 2.1.3 (2020-05-13) + +### Bugs fixed + +- Fixes a bug in `codeql execute cli-server` (a helper used by the VS + Code extension) which would sometimes cause query compilation to + fail until the extension was restarted. +- Fixes a bug in `codeql database upgrade` which could lead to + performance losses if the upgraded database was subsequently used + with LGTM or the legacy Semmle Core product. +- Fixes a bug in the QL evaluator that would sometimes lead to crashes + for queries that use the new `unique` aggregate added in release + 2.1.0. +- The value of the `--compilation-cache-size` option is now correctly + interpreted as a number of megabytes rather than a number of bytes. + +## Release 2.1.2 (2020-05-06) + +- Updated license terms to allow CI use with GitHub Actions for + open-source software. + +### Potentially breaking changes + +- In [query suite definitions](https://help.semmle.com/codeql/codeql-cli/procedures/query-suites.html), filter + instructions that filter on the `query path` pseudo-tag will now + always see the relative path to the query expressed with `/` as a + directory separator, independently of the platform. Previously they + erroneously used the platform's directory separator, meaning that + query suites developed on Windows would not work correctly on Unix + systems (and vice versa) if they used `query path`. Existing suite + definitions developed on Windows may need to be updated to match the + new behavior. + +### Features added + +- A new `codeql test accept` subcommand helps automate updating the + expected output for unit tests after a desired change in query + behavior. This can also be done by the new `--learn` option for + `codeql test run`. + +### Bugs fixed + +- `codeql database create` will now report an explicit error if given + a `--command` argument that specifies an empty string. Previously + this would be accepted initially, leading to confusing failures + later. + +## Release 2.1.1 (2020-04-20) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. + +### Features added + +- `codeql resolve queries` accepts a `--format=bylanguage` option. + This is used to help automated workflows determine which languages + to create databases for, from the queries that are available to run. +- It is now possible to attempt to execute `.ql` files that are not in + a QL pack. This is used by a few specialized internal workflows. + However, standalone queries cannot import any of the dependencies + that you would usually declare in a `qlpack.yml` file, so will not + be useful in most cases. + +## Release 2.1.0 (2020-03-27) + +- The bundled extractors are updated to match the versions currently + used on LGTM.com. These are newer than the last release (1.23) of + LGTM Enterprise. If you plan to upload databases to an LGTM + Enterprise 1.23 instance, you need to create them with release + 2.0.1. For more information, see [Preparing CodeQL databases to + upload to + LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html) + in the LGTM admin help. + +### Potentially breaking changes + +- If you pass a directory name as a command-line argument to + `codeql test run`, it will now consider all `.ql` or `.qlref` files + found under that directory to be test queries, even if they have no + accompanying `.expected` file. Tests that lack an `.expected` file + will fail, but will generate an `.actual` file that you can rename + to `.expected` if you want to use the results. + + The goal of this change is to support existing workflows of + experienced CodeQL users, and also to provide clear error + indications if an `.expected` file is accidentally lost, renamed, or + misspelled. + + However, if you invoke `codeql test run` on a directory tree that + contains both tests and non-test queries, you will now encounter + errors if any of the `.ql` files can't be processed as test queries. + If you're affected by this change, you can suppress these errors by: + + - Adding a `tests` property to this QL pack to define specify + which directories contain only test queries and associated test + code. For more information, see + [About QL packs](https://help.semmle.com/codeql/codeql-cli/reference/qlpack-overview.html). + - Running `codeql test run` with a new `--strict-test-discovery` + option. + + In the longer term, we recommend that you reorganize the queries so + that test queries are stored in a directory tree that's separate + from actual queries. + +- `codeql database create` and `codeql database finalize` will no + longer recognize a `--no-duplicate-code` option. This option has + never had any effect, and its positive variant `--duplicate-code` + previously led to a fatal error. + +### Features added + +- A new XML extractor is included. It is not intended to be used as a + stand-alone extractor, but rather to augment the data produced by + other extractors. In particular, the C\# and Java extractors invoke + it during database creation to include information relevant to the + analysis of those languages, much like LGTM.com does. +- Two new plumbing commands `codeql database index-files` and + `codeql resolve files` have been added for support of invoking the + XML extractor support. These commands are generally only of interest + for extractor authors. +- Two new plumbing commands have been added to `codeql dataset`. The + `measure` subcommand can be used to collect size information from a + dataset, and the `check` subcommand can scan a dataset for database + inconsistencies. These commands are useful when developing a new + CodeQL extractor. +- The QL evaluator contains a number of features in support of an + internal experiment with using machine-learning techniques to + identify functions in unknown codebases as sources or sinks of + taint. This includes new command-line options `--ml-model-path` and + `--native-library-path` to several subcommands. As the new features + are not yet ready for general use, these new options should be + ignored by external CodeQL users. + +### Bugs fixed + +- Fixes a bug that could result in empty databases for C/C++. + Previously, extraction would mistakenly be skipped for source files + compiled with the Clang compiler, if the `-fintegrated-cc1` option + was specified. +- `codeql database create` and `codeql database init` will now, as + they have always been documented, refuse to create a database whose + parent directory doesn't already exist. +- `codeql test run` will no longer leave `.actual` files from previous + runs in the file system after a test passes. + +### QL language improvements + +- QL now supports set literals, and the QL extractor can identify them + with the `SetLiteral` class. For more information, see [Set literal + expressions](https://help.semmle.com/QL/ql-handbook/expressions.html#set-literal-expressions) + in the QL language reference. +- QL now supports a uniqueness aggregate. This can express constraints + that there is precisely one value. The syntax is taken from previous + aggregates such as `min` and `max`. + + ``` {.sourceCode .ql} + unique(int x | x = 4 or x = 2 * 2 | x) + ``` + +## Release 2.0.6 (2020-03-16) + +### Bugs fixed + +- Fixes a problem preventing `codeql database create` from working + with Python 3 on macOS. +- Fixes a problem preventing `codeql database create` from finding + locally installed Python packages. + +## Release 2.0.5 (2020-03-13) + +- The bundled extractors (which are responsible for converting source + code to databases for each supported language) are updated to match + the versions currently used on LGTM.com. These are newer than the + last release of LGTM Enterprise, so this release should not be used + if you plan to upload databases to an LGTM Enterprise instance. For + more information, see [Preparing CodeQL databases to upload to + LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html) + in the LGTM admin help. + +### Features added + +- `codeql test run` has a new `--slice` option that can be used to + parallelize tests over more machines. + +## Release 2.0.4 (2020-02-21) + +- The bundled extractors (which are responsible for converting source + code to databases for each supported language) are updated to match + the versions currently used on LGTM.com. These are newer than the + last release of LGTM Enterprise, so this release should not be used + if you plan to upload databases to an LGTM Enterprise instance. For + more information, see [Preparing CodeQL databases to upload to + LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html) + in the LGTM admin help. + +### Features added + +- Subcommands that execute queries (such as `codeql database analyze`) + now have a `--timeout` option that can be used to set a timeout to + automatically cancel query evaluations that appear to diverge. +- A new plumbing command `codeql query decompile` can display the DIL + intermediate representations that is included in the output of + `codeql query compile --dump-qlo --include-dil-in-qlo`. This is + useful mainly for certain internal workflows; the information + produced is the same as what `codeql query compile --dump-dil` + already outputs. + +### Bugs fixed + +- The `--debug` and `--tuple-counting` options to + `codeql test run` erroneously had no effect. Now they ought to work. + +## Release 2.0.3 (2020-02-12) + +### Bugs fixed + +- Fixes a bug where `codeql test run` would fail with the + message + `CatastrophicError: There should be a --library-path option for com.semmle.cli2.LibraryPathOptions.libraryPath but we didn't find it` + when running tests against the `master` branch of the CodeQL + libraries for certain languages. +- Otherwise identical to release 2.0.2. + +## Release 2.0.2 (2020-02-05) + +- The bundled extractors (which are responsible for converting source + code to databases for each supported language) are updated to match + the versions currently used on LGTM.com. These are newer than the + last release of LGTM Enterprise, so this release should not be used + if you plan to upload databases to an LGTM Enterprise instance. For + more information, see [Preparing CodeQL databases to upload to + LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html) + in the LGTM admin help. +- The parent and sibling directories of the unpacked CLI are no longer + searched recursively for QL packs. QL packs will only be found if + there's a `qlpack.yml` or `.codeqlmanifest.json` directly in a + parent or sibling directory. This should eliminate the very long + disk-scanning delays experienced by users who unpacked earlier + versions of the CLI in their home directory. +- Parent and sibling directories of the unpacked CLI will now be + searched for QL packs as a last resort, even if you give an explicit + `--search-path` option. This means, for example, that you can define + a search path in the [per-user configuration file](https://help.semmle.com/codeql/codeql-cli/reference/configuration-overview.html) without it depending on + where the CLI is unpacked. In particular, the setting can now be + meaningfully used by users who let the CodeQL for VS Code extension + manage the downloading and unpacking of the CLI. + +### Security updates + +- The `codeql database create` command and its relatives will no + longer attempt to find extractors located in the parent and sibling + directories of the unpacked CLI. This closes a security risk for + users who unpacked the CodeQL CLI in their home directory. This + could've resulted in arbitrary code execution if the user unpacked a + file archive containing a malicious extractor anywhere in the home + directory. Extractors will now only be found within the unpacked CLI + itself, or in directories explicitly listed in the `--search-path`. + It is expected that users will only point `--search-path` to + locations they trust at least as much as the CLI download itself. + +### Features added + +- This release supports executing query regression tests using the + `codeql test` command. For further information, see + [Testing custom queries](https://help.semmle.com/codeql/codeql-cli/procedures/test-queries.html). +- The error message if you try executing a query against a database + that needs to be upgraded (which can happen routinely if you're + using a fresh `master` checkout of the CodeQL libraries with the + bundled extractors) will now explicitly suggest a + `codeql database update` command to run. The database is not + automatically upgraded, as this may make it irreversibly + incompatible with older versions of the CodeQL libraries. This + allows users who want to compare behavior of different versions of + the libraries against the same database to make a copy before they + upgrade it. + +## Release 2.0.1 (2019-12-17) + +- Corresponds to LGTM Enterprise release 1.23. +- The bundled extractors (which are responsible for converting source + code to databases for each supported language) are updated to match + the extractor versions used in LGTM Enterprise. +- No other changes to the core CLI. + +## Release 2.0.0 (2019-11-14) + +- First public release. +