Skip to content

Create CHANGELOG.md #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 23, 2020
Merged

Create CHANGELOG.md #37

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
361 changes: 361 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,361 @@
# CodeQL CLI changelog

## Release 2.2.3 (2020-06-15)

CodeQL CLI 2.2.3 is the same as version 2.2.2, but re-released with a new
version number because the `v2.2.2` folder on the download site
originally contained the 2.2.0 binaries instead of the correct 2.2.2
ones.

If you have downloaded release 2.2.2, and `codeql --version` correctly
identifies itself as being that version, you don't need to upgrade to
2.2.3.

## Release 2.2.2 (2020-06-12)

- The bundled extractors are updated to match the versions currently
used on LGTM.com. These are newer than the last release (1.24) of
LGTM Enterprise. If you plan to upload databases to an LGTM
Enterprise 1.24 instance, you need to create them with release
2.1.4.

### Improvements

- Query evaluations that time out due to a `--timeout` option are no
longer silently discarded. Instead `codeql` will terminate with exit
code 33. Commands that evaluate multiple queries will produce as
much output as they can even if one of the queries times out.

## Release 2.2.1

There is no CodeQL CLI version 2.2.1. This version number was used
internally to work around restrictions in the CodeQL for VS Code
extension.

## Release 2.2.0 (2020-05-29)

- The bundled extractors are updated to match the versions currently
used on LGTM.com. These are newer than the last release (1.24) of
LGTM Enterprise. If you plan to upload databases to an LGTM
Enterprise 1.24 instance, you need to create them with release
2.1.4.
- Starting with this release, the CodeQL CLI can be downloaded either
as a single `codeql.zip` file containing the CLI for all supported
platforms, or as a `codeql-PLATFORM.zip` that contains the files for
just one platform. The single-platform zips are faster to download.

### QL language improvement

- QL now supports the definition of new types as type unions. This
feature currently allows unions of branches from an already existing
algebraic data type and unions of database types.

## Release 2.1.4 (2020-05-26)

This release corresponds to release 1.24.x of LGTM Enterprise, and
should be used when creating databases that will be uploaded to it.
Future CLI releases (numbered 2.2.x) may produce databases that are not
backwards compatible with this version of LGTM Enterprise.

For all purposes other than creating databases for LGTM Enterprise we
recommend that you continue upgrading to newer CLI releases as they
become available.

### Features added

- A new `codeql query format` command exposes the QL autoformatter for
use on the command line.

### Bugs fixed

- `-J` command-line options that contain spaces now ought to work on
Windows. They still do not work reliably on Linux or MacOS, though.

## Release 2.1.3 (2020-05-13)

### Bugs fixed

- Fixes a bug in `codeql execute cli-server` (a helper used by the VS
Code extension) which would sometimes cause query compilation to
fail until the extension was restarted.
- Fixes a bug in `codeql database upgrade` which could lead to
performance losses if the upgraded database was subsequently used
with LGTM or the legacy Semmle Core product.
- Fixes a bug in the QL evaluator that would sometimes lead to crashes
for queries that use the new `unique` aggregate added in release
2.1.0.
- The value of the `--compilation-cache-size` option is now correctly
interpreted as a number of megabytes rather than a number of bytes.

## Release 2.1.2 (2020-05-06)

- Updated license terms to allow CI use with GitHub Actions for
open-source software.

### Potentially breaking changes

- In [query suite definitions](https://help.semmle.com/codeql/codeql-cli/procedures/query-suites.html), filter
instructions that filter on the `query path` pseudo-tag will now
always see the relative path to the query expressed with `/` as a
directory separator, independently of the platform. Previously they
erroneously used the platform's directory separator, meaning that
query suites developed on Windows would not work correctly on Unix
systems (and vice versa) if they used `query path`. Existing suite
definitions developed on Windows may need to be updated to match the
new behavior.

### Features added

- A new `codeql test accept` subcommand helps automate updating the
expected output for unit tests after a desired change in query
behavior. This can also be done by the new `--learn` option for
`codeql test run`.

### Bugs fixed

- `codeql database create` will now report an explicit error if given
a `--command` argument that specifies an empty string. Previously
this would be accepted initially, leading to confusing failures
later.

## Release 2.1.1 (2020-04-20)

- The bundled extractors are updated to match the versions currently
used on LGTM.com.

### Features added

- `codeql resolve queries` accepts a `--format=bylanguage` option.
This is used to help automated workflows determine which languages
to create databases for, from the queries that are available to run.
- It is now possible to attempt to execute `.ql` files that are not in
a QL pack. This is used by a few specialized internal workflows.
However, standalone queries cannot import any of the dependencies
that you would usually declare in a `qlpack.yml` file, so will not
be useful in most cases.

## Release 2.1.0 (2020-03-27)

- The bundled extractors are updated to match the versions currently
used on LGTM.com. These are newer than the last release (1.23) of
LGTM Enterprise. If you plan to upload databases to an LGTM
Enterprise 1.23 instance, you need to create them with release
2.0.1. For more information, see [Preparing CodeQL databases to
upload to
LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html)
in the LGTM admin help.

### Potentially breaking changes

- If you pass a directory name as a command-line argument to
`codeql test run`, it will now consider all `.ql` or `.qlref` files
found under that directory to be test queries, even if they have no
accompanying `.expected` file. Tests that lack an `.expected` file
will fail, but will generate an `.actual` file that you can rename
to `.expected` if you want to use the results.

The goal of this change is to support existing workflows of
experienced CodeQL users, and also to provide clear error
indications if an `.expected` file is accidentally lost, renamed, or
misspelled.

However, if you invoke `codeql test run` on a directory tree that
contains both tests and non-test queries, you will now encounter
errors if any of the `.ql` files can't be processed as test queries.
If you're affected by this change, you can suppress these errors by:

- Adding a `tests` property to this QL pack to define specify
which directories contain only test queries and associated test
code. For more information, see
[About QL packs](https://help.semmle.com/codeql/codeql-cli/reference/qlpack-overview.html).
- Running `codeql test run` with a new `--strict-test-discovery`
option.

In the longer term, we recommend that you reorganize the queries so
that test queries are stored in a directory tree that's separate
from actual queries.

- `codeql database create` and `codeql database finalize` will no
longer recognize a `--no-duplicate-code` option. This option has
never had any effect, and its positive variant `--duplicate-code`
previously led to a fatal error.

### Features added

- A new XML extractor is included. It is not intended to be used as a
stand-alone extractor, but rather to augment the data produced by
other extractors. In particular, the C\# and Java extractors invoke
it during database creation to include information relevant to the
analysis of those languages, much like LGTM.com does.
- Two new plumbing commands `codeql database index-files` and
`codeql resolve files` have been added for support of invoking the
XML extractor support. These commands are generally only of interest
for extractor authors.
- Two new plumbing commands have been added to `codeql dataset`. The
`measure` subcommand can be used to collect size information from a
dataset, and the `check` subcommand can scan a dataset for database
inconsistencies. These commands are useful when developing a new
CodeQL extractor.
- The QL evaluator contains a number of features in support of an
internal experiment with using machine-learning techniques to
identify functions in unknown codebases as sources or sinks of
taint. This includes new command-line options `--ml-model-path` and
`--native-library-path` to several subcommands. As the new features
are not yet ready for general use, these new options should be
ignored by external CodeQL users.

### Bugs fixed

- Fixes a bug that could result in empty databases for C/C++.
Previously, extraction would mistakenly be skipped for source files
compiled with the Clang compiler, if the `-fintegrated-cc1` option
was specified.
- `codeql database create` and `codeql database init` will now, as
they have always been documented, refuse to create a database whose
parent directory doesn't already exist.
- `codeql test run` will no longer leave `.actual` files from previous
runs in the file system after a test passes.

### QL language improvements

- QL now supports set literals, and the QL extractor can identify them
with the `SetLiteral` class. For more information, see [Set literal
expressions](https://help.semmle.com/QL/ql-handbook/expressions.html#set-literal-expressions)
in the QL language reference.
- QL now supports a uniqueness aggregate. This can express constraints
that there is precisely one value. The syntax is taken from previous
aggregates such as `min` and `max`.

``` {.sourceCode .ql}
unique(int x | x = 4 or x = 2 * 2 | x)
```

## Release 2.0.6 (2020-03-16)

### Bugs fixed

- Fixes a problem preventing `codeql database create` from working
with Python 3 on macOS.
- Fixes a problem preventing `codeql database create` from finding
locally installed Python packages.

## Release 2.0.5 (2020-03-13)

- The bundled extractors (which are responsible for converting source
code to databases for each supported language) are updated to match
the versions currently used on LGTM.com. These are newer than the
last release of LGTM Enterprise, so this release should not be used
if you plan to upload databases to an LGTM Enterprise instance. For
more information, see [Preparing CodeQL databases to upload to
LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html)
in the LGTM admin help.

### Features added

- `codeql test run` has a new `--slice` option that can be used to
parallelize tests over more machines.

## Release 2.0.4 (2020-02-21)

- The bundled extractors (which are responsible for converting source
code to databases for each supported language) are updated to match
the versions currently used on LGTM.com. These are newer than the
last release of LGTM Enterprise, so this release should not be used
if you plan to upload databases to an LGTM Enterprise instance. For
more information, see [Preparing CodeQL databases to upload to
LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html)
in the LGTM admin help.

### Features added

- Subcommands that execute queries (such as `codeql database analyze`)
now have a `--timeout` option that can be used to set a timeout to
automatically cancel query evaluations that appear to diverge.
- A new plumbing command `codeql query decompile` can display the DIL
intermediate representations that is included in the output of
`codeql query compile --dump-qlo --include-dil-in-qlo`. This is
useful mainly for certain internal workflows; the information
produced is the same as what `codeql query compile --dump-dil`
already outputs.

### Bugs fixed

- The `--debug` and `--tuple-counting` options to
`codeql test run` erroneously had no effect. Now they ought to work.

## Release 2.0.3 (2020-02-12)

### Bugs fixed

- Fixes a bug where `codeql test run` would fail with the
message
`CatastrophicError: There should be a --library-path option for com.semmle.cli2.LibraryPathOptions.libraryPath but we didn't find it`
when running tests against the `master` branch of the CodeQL
libraries for certain languages.
- Otherwise identical to release 2.0.2.

## Release 2.0.2 (2020-02-05)

- The bundled extractors (which are responsible for converting source
code to databases for each supported language) are updated to match
the versions currently used on LGTM.com. These are newer than the
last release of LGTM Enterprise, so this release should not be used
if you plan to upload databases to an LGTM Enterprise instance. For
more information, see [Preparing CodeQL databases to upload to
LGTM](https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html)
in the LGTM admin help.
- The parent and sibling directories of the unpacked CLI are no longer
searched recursively for QL packs. QL packs will only be found if
there's a `qlpack.yml` or `.codeqlmanifest.json` directly in a
parent or sibling directory. This should eliminate the very long
disk-scanning delays experienced by users who unpacked earlier
versions of the CLI in their home directory.
- Parent and sibling directories of the unpacked CLI will now be
searched for QL packs as a last resort, even if you give an explicit
`--search-path` option. This means, for example, that you can define
a search path in the [per-user configuration file](https://help.semmle.com/codeql/codeql-cli/reference/configuration-overview.html) without it depending on
where the CLI is unpacked. In particular, the setting can now be
meaningfully used by users who let the CodeQL for VS Code extension
manage the downloading and unpacking of the CLI.

### Security updates

- The `codeql database create` command and its relatives will no
longer attempt to find extractors located in the parent and sibling
directories of the unpacked CLI. This closes a security risk for
users who unpacked the CodeQL CLI in their home directory. This
could've resulted in arbitrary code execution if the user unpacked a
file archive containing a malicious extractor anywhere in the home
directory. Extractors will now only be found within the unpacked CLI
itself, or in directories explicitly listed in the `--search-path`.
It is expected that users will only point `--search-path` to
locations they trust at least as much as the CLI download itself.

### Features added

- This release supports executing query regression tests using the
`codeql test` command. For further information, see
[Testing custom queries](https://help.semmle.com/codeql/codeql-cli/procedures/test-queries.html).
- The error message if you try executing a query against a database
that needs to be upgraded (which can happen routinely if you're
using a fresh `master` checkout of the CodeQL libraries with the
bundled extractors) will now explicitly suggest a
`codeql database update` command to run. The database is not
automatically upgraded, as this may make it irreversibly
incompatible with older versions of the CodeQL libraries. This
allows users who want to compare behavior of different versions of
the libraries against the same database to make a copy before they
upgrade it.

## Release 2.0.1 (2019-12-17)

- Corresponds to LGTM Enterprise release 1.23.
- The bundled extractors (which are responsible for converting source
code to databases for each supported language) are updated to match
the extractor versions used in LGTM Enterprise.
- No other changes to the core CLI.

## Release 2.0.0 (2019-11-14)

- First public release.