From a9a0cb928eecae95be66f0dca74d1074d70d298c Mon Sep 17 00:00:00 2001 From: Paolo Tranquilli Date: Mon, 30 Jun 2025 16:25:55 +0200 Subject: [PATCH 1/6] Rust: add `rust-analyzer` update instructions --- rust/README.md | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/rust/README.md b/rust/README.md index 4f462f1b70f6..425d2367c608 100644 --- a/rust/README.md +++ b/rust/README.md @@ -59,3 +59,51 @@ Sometimes, especially if resolving conflicts on generated files, you might need bazel run @codeql//rust/codegen -- --force ``` for code generation to succeed. + +### Updating `rust-analyzer` + +Here's a rundown of the typical actions to perform to do a rust-analyzer (and other dependencies) update. A one-time setup consists in +installing [`cargo-edit`](https://crates.io/crates/cargo-edit) with `cargo install cargo-edit`. On Ubuntu that also requires +`sudo apt install libssl-dev pkg-config`. + +1. from the root of the `codeql` repo checkout, run an upgrade, and commit the changes (skipping `pre-commit` hooks if you have them enabled): + ``` + cargo upgrade --incompatible --pinned + ``` +2. Look at a diff of the `Cargo.toml` files: if all `ra_ap_` prefixed dependencies have been updated to the same number, go on to the next step. + Otherwise, it means the latest `rust-analyzer` update has not been fully rolled out to all its crates in `crates.io`. + _All `ra_ap_` versions must agree!_ + Downgrade by hand to the minimum one you see, and run a `cargo update` after that to fix the `Cargo.lock` file. +3. Commit the changes, skipping `pre-commit` hooks if you have them enabled: + ``` + git commit -am 'Cargo: upgrade dependencies' --no-verify + ``` +4. Regenerate vendored bazel files, commit the changes: + ``` + misc/bazel/3rdparty/update_tree_sitter_extractors_deps.sh + git add . + git commit -am 'Bazel: regenerate vendored cargo dependencies' --no-verify + ``` +5. Run codegen + ``` + bazel run //rust/codegen + ``` + Take note whether `rust/schema/ast.py` was changed. That might need tweaks, new tests and/or downgrade/upgrade scripts down the line +6. Try compiling + ``` + bazel run //rust:install + ``` + * if it succeeds: good! You can move on to the next step. + * if it fails while compiling rust-analyzer dependencies, you need to update the rust toolchain. Sometimes the error will tell you + so explcitly, but it may happen that the error is more obscure. To update the rust toolchain: + * you will need to open a PR on the internal repo updating `RUST_VERSION` in `MODULE.bazel`. In general you can have this merged + independently of the changes in `codeql`. + * in `codeql`, update both `RUST_VERSION` in `MODULE.bazel` _and_ `rust-toolchain.toml` files. You may want to also update the + nightly toolchain in `rust/extractor/src/nightly-toolchain/rust-toolchain.toml` to a more recent date while you're at it. + * if it fails while compiling rust extractor code, you will need to adapt it to the new library version. + + If you had to do any changes, commit them. If you updated the rust toolchain, running `rust/lint.py` might reformat or apply new + lints to the code. +7. If in step 5 the schema was updated, add upgrade/downgrade scripts and a change note +8. Check with CI if everything is in order. +9. Run DCA. Iterate on the code if needed. From 9a96372f5314be94423fd8d9f224c0becb9d5458 Mon Sep 17 00:00:00 2001 From: Geoffrey White <40627776+geoffw0@users.noreply.github.com> Date: Thu, 11 Sep 2025 09:40:36 +0100 Subject: [PATCH 2/6] Rust: Add some more details. --- rust/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/rust/README.md b/rust/README.md index 425d2367c608..12cfe8b192ad 100644 --- a/rust/README.md +++ b/rust/README.md @@ -78,7 +78,7 @@ installing [`cargo-edit`](https://crates.io/crates/cargo-edit) with `cargo insta ``` git commit -am 'Cargo: upgrade dependencies' --no-verify ``` -4. Regenerate vendored bazel files, commit the changes: +4. Regenerate vendored bazel files (these allow faster builds, particularly on CI where it has to start from scratch each time), commit the changes: ``` misc/bazel/3rdparty/update_tree_sitter_extractors_deps.sh git add . @@ -101,6 +101,7 @@ installing [`cargo-edit`](https://crates.io/crates/cargo-edit) with `cargo insta * in `codeql`, update both `RUST_VERSION` in `MODULE.bazel` _and_ `rust-toolchain.toml` files. You may want to also update the nightly toolchain in `rust/extractor/src/nightly-toolchain/rust-toolchain.toml` to a more recent date while you're at it. * if it fails while compiling rust extractor code, you will need to adapt it to the new library version. + * for example updating annotations in `annotations.py`, adding / removing generated tests. If you had to do any changes, commit them. If you updated the rust toolchain, running `rust/lint.py` might reformat or apply new lints to the code. From 8af8c6d95aa6e6d1cc3e43c508fabe1a98dde551 Mon Sep 17 00:00:00 2001 From: Geoffrey White <40627776+geoffw0@users.noreply.github.com> Date: Thu, 11 Sep 2025 09:44:10 +0100 Subject: [PATCH 3/6] Rust: Move upgrade script to the very end, add a few details. --- rust/README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/rust/README.md b/rust/README.md index 12cfe8b192ad..d3747032c75a 100644 --- a/rust/README.md +++ b/rust/README.md @@ -88,7 +88,7 @@ installing [`cargo-edit`](https://crates.io/crates/cargo-edit) with `cargo insta ``` bazel run //rust/codegen ``` - Take note whether `rust/schema/ast.py` was changed. That might need tweaks, new tests and/or downgrade/upgrade scripts down the line + Take note whether `rust/schema/ast.py` was changed. That might need tweaks, new tests and/or downgrade/upgrade scripts down the line. 6. Try compiling ``` bazel run //rust:install @@ -105,6 +105,8 @@ installing [`cargo-edit`](https://crates.io/crates/cargo-edit) with `cargo insta If you had to do any changes, commit them. If you updated the rust toolchain, running `rust/lint.py` might reformat or apply new lints to the code. -7. If in step 5 the schema was updated, add upgrade/downgrade scripts and a change note -8. Check with CI if everything is in order. -9. Run DCA. Iterate on the code if needed. +7. Check with CI if everything is in order. +8. Run DCA. Iterate on the code if needed. +9. If in step 5 the schema was updated, add upgrade/downgrade scripts and a change note. This is best done last to reduce the chance of +merge conflicts (none of the other testing depends on having upgrade and downgrade scripts in place). See +[Upgrading a language database schema](docs/prepare-db-upgrade.md). From 32131cf9aea4ca29145384044a47c8862f9717ab Mon Sep 17 00:00:00 2001 From: Geoffrey White <40627776+geoffw0@users.noreply.github.com> Date: Thu, 11 Sep 2025 10:36:04 +0100 Subject: [PATCH 4/6] Update rust/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- rust/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rust/README.md b/rust/README.md index d3747032c75a..dfd36093333b 100644 --- a/rust/README.md +++ b/rust/README.md @@ -95,7 +95,7 @@ installing [`cargo-edit`](https://crates.io/crates/cargo-edit) with `cargo insta ``` * if it succeeds: good! You can move on to the next step. * if it fails while compiling rust-analyzer dependencies, you need to update the rust toolchain. Sometimes the error will tell you - so explcitly, but it may happen that the error is more obscure. To update the rust toolchain: + so explicitly, but it may happen that the error is more obscure. To update the rust toolchain: * you will need to open a PR on the internal repo updating `RUST_VERSION` in `MODULE.bazel`. In general you can have this merged independently of the changes in `codeql`. * in `codeql`, update both `RUST_VERSION` in `MODULE.bazel` _and_ `rust-toolchain.toml` files. You may want to also update the From aaa67a2da96bc2d9972810875c2442f4e173bd3a Mon Sep 17 00:00:00 2001 From: Geoffrey White <40627776+geoffw0@users.noreply.github.com> Date: Thu, 11 Sep 2025 17:12:22 +0100 Subject: [PATCH 5/6] Update rust/README.md Co-authored-by: Simon Friis Vindum --- rust/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rust/README.md b/rust/README.md index dfd36093333b..e820d1c11141 100644 --- a/rust/README.md +++ b/rust/README.md @@ -66,7 +66,7 @@ Here's a rundown of the typical actions to perform to do a rust-analyzer (and ot installing [`cargo-edit`](https://crates.io/crates/cargo-edit) with `cargo install cargo-edit`. On Ubuntu that also requires `sudo apt install libssl-dev pkg-config`. -1. from the root of the `codeql` repo checkout, run an upgrade, and commit the changes (skipping `pre-commit` hooks if you have them enabled): +1. From the root of the `codeql` repo checkout, run a Cargo upgrade: ``` cargo upgrade --incompatible --pinned ``` From 92124a90330b38f6c6cfcde8620d7e0a49dfe008 Mon Sep 17 00:00:00 2001 From: Geoffrey White <40627776+geoffw0@users.noreply.github.com> Date: Fri, 12 Sep 2025 09:01:13 +0100 Subject: [PATCH 6/6] Update rust/README.md --- rust/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rust/README.md b/rust/README.md index e820d1c11141..4fdbb0aedbdd 100644 --- a/rust/README.md +++ b/rust/README.md @@ -106,7 +106,7 @@ installing [`cargo-edit`](https://crates.io/crates/cargo-edit) with `cargo insta If you had to do any changes, commit them. If you updated the rust toolchain, running `rust/lint.py` might reformat or apply new lints to the code. 7. Check with CI if everything is in order. -8. Run DCA. Iterate on the code if needed. +8. Run DCA with database caching disabled. Iterate on the code if needed. 9. If in step 5 the schema was updated, add upgrade/downgrade scripts and a change note. This is best done last to reduce the chance of merge conflicts (none of the other testing depends on having upgrade and downgrade scripts in place). See [Upgrading a language database schema](docs/prepare-db-upgrade.md).