Update Clang toolchain from 18.0.0 to 18.1.7 #12077
Conversation
|
@maflcko you mentioned in #11626 that before doing this all existing projects should be un-pinned from their Also, do you know of a way to more easily enumerate the projects which break as a result of this upate? I probably can't feasibly build everything locally. If CI takes care of this though I can also just wait for that. |
|
Also, for reference, I've confirmed that by layering #12075 on top of this Rust projects no longer have any warnings in coverage builds and coverage looks like it might work. |
|
Also, following up from your comment here you mentioned that pinned projects might break since they're using clang 15. I think though that the decoding of coverage data supports older versions, just not newer versions, so my assumption would be that LLVM 18 tooling would be able to decode clang 15-generated coverage information. I don't have data to back up this assumption, however. |
|
/gcbrun trial_build.py all |
Yes, because the coverage container uses the current llvm to parse the coverage profile (regardless of what the project uses), but if the profile was generated with llvm-15 (pinned projects) it will fail. It should be possible to observe this in the trial build. |
In theory, yes, earlier coverage profiles can be read. However, the raw profile version is a separate versioning, and a breaking change every time, as far as I understood it. |
| @@ -50,7 +50,7 @@ LLVM_DEP_PACKAGES="build-essential make ninja-build git python3 python3-distutil | |||
| apt-get update && apt-get install -y $LLVM_DEP_PACKAGES --no-install-recommends | |||
|
|
|||
| # For manual bumping. | |||
| OUR_LLVM_REVISION=llvmorg-18-init-4631-gd50b56d1 | |||
| OUR_LLVM_REVISION=llvmorg-18.1.7 | |||
There was a problem hiding this comment.
unrelated nit: Could add a comment to check if CMAKE_VERSION is up-to-date? Otherwise, a future bump could cause issues, because cmake may not properly detect the newer llvm version. Most recently this happened for 32-bit builds.
Generally, I'd say the two should be bumped at the same time to avoid build issues in projects down the line.
There was a problem hiding this comment.
Resolved as part of #12083
There was a problem hiding this comment.
Thanks! I'll admit I wasn't sure what CMAKE_VERSION was referring to here so assistance is appreciated!
|
Thanks for the links! I'll start reviewing those hopefully soon. Also I was having a tough time understanding what you were referring to before about LLVM 15 breaking. I thought that because the current coverage container was using LLVM 18 that updating it to LLVM 18.1 wouldn't be an issue because it would already be broken with LLVM 15 coverage information. Upon checking though it looks like the version in the LLVM source for profiling was "8" both with LLVM 15 and LLVM 18.0.0. The first change happened with LLVM 18.1 so that also makes more sense to me. I'll try to dig in further to the failure logs and see what the impact of this is. Also I'll note I'm happy to rebase around other changes, so please don't block on me for anything. |
|
The failures should all be related to the raw coverage profile version in some way or another. I don't see another way other than to atomically and globally bump the coverage version for all projects and all languages. But that requires the projects to be un-pinned, and a rust-nightly bump to be combined into this pull. My recommendation would be to change #12075 to nightly-2024-02-12 for now, then wait for it to land and then bump |
|
Ok I've gone through many of the failures and I'm sort of quite new to updating the toolchain here so I wanted to ask a few questions. I've tried to bucket all the various failure logs into a few categories:
My main question is how to handle most of these. Two action items for this PR are to update Rust in this PR and update Swift as well. Everything else though I'm less certain about. For example resolving new Clang errors will require source changes. I tested a few of the There's still other failures I don't fully understand which I'm not sure if y'all would recognize or not |
|
Oh and one final category of failures I forgot to mention are those that failed to build but also failed to build according to their latest status, so I ignored a few builds like that. |
They are expected, I think, and can be ignored for now, because the infra check does not spin up arm64 machines. You can use
You will have to rebase or merge with master before the trial build. Otherwise the changes (2c03690) aren't picked up. |
|
If more than one project is affected by a build warning, you can soften it. For example: diff --git a/infra/base-images/base-clang/Dockerfile b/infra/base-images/base-clang/Dockerfile
index f61b85443..c82ed1008 100644
--- a/infra/base-images/base-clang/Dockerfile
+++ b/infra/base-images/base-clang/Dockerfile
@@ -58,9 +58,9 @@ ENV CCC "clang++"
# The implicit-function-declaration and implicit-int errors are downgraded to a
# warning, to allow compiling legacy code.
# See https://releases.llvm.org/16.0.0/tools/clang/docs/ReleaseNotes.html#potentially-breaking-changes
-# Same for deprecated-declarations, int-conversion,
+# Same for vla-cxx-extension, deprecated-declarations, int-conversion,
# incompatible-function-pointer-types, enum-constexpr-conversion
-ENV CFLAGS "-O1 -fno-omit-frame-pointer -gline-tables-only -Wno-error=enum-constexpr-conversion -Wno-error=incompatible-function-pointer-types -Wno-error=int-conversion -Wno-error=deprecated-declarations -Wno-error=implicit-function-declaration -Wno-error=implicit-int -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION"
+ENV CFLAGS "-O1 -fno-omit-frame-pointer -gline-tables-only -Wno-error=enum-constexpr-conversion -Wno-error=incompatible-function-pointer-types -Wno-error=int-conversion -Wno-error=deprecated-declarations -Wno-error=vla-cxx-extension -Wno-error=implicit-function-declaration -Wno-error=implicit-int -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION"
ENV CXXFLAGS_EXTRA "-stdlib=libc++"
ENV CXXFLAGS "$CFLAGS $CXXFLAGS_EXTRA" |
Looks like Swift 5.8.1 is currently used which uses LLVM 13.0.0. The latest release of Swift is 5.10.1 which comes with LLVM 15.0.0. It looks like Swift 6 is in development but I wasn't able to find a binary to download and see if it's at the right version. Given that I don't think there's an easy fix for Swift for now. |
alexcrichton
force-pushed
the
update-clang
branch
from
c90949c
to
eb72f1d
Compare
| -Wno-error=format-truncation \ | ||
| -Wno-error=thread-safety-reference-return \ | ||
| -Wno-error=nan-infinity-disabled \ |
There was a problem hiding this comment.
| -Wno-error=format-truncation \ | |
| -Wno-error=thread-safety-reference-return \ | |
| -Wno-error=nan-infinity-disabled \ |
nit: I'd say, if there is only a single project needing the suppressions, it should go into the project file. Especially, those warnings may point to real errors (as opposed to just warn about compile extensions, deprecations, or implicit behavior).
There was a problem hiding this comment.
I think you forgot to add the three -Wno-error= to the three project's build.sh in the latest push?
There was a problem hiding this comment.
Oh sorry I misinterpreted this as leaving the flags out entirely. I'll try to work through some C++-specific failures and add per-image flags.
|
/gcbrun trial_build.py all |
|
From the trial build I've categorized the failures into: Miscellaneous failuresVarious failures in C++ projectsProjects that fail to compile with the latest Rust compilerTransient failures - OOM, out of disk, network issues, etcSwift failures - swift older LLVM causes coverage mismatchJava related failures having to do with maven and network issuesFailures due to `@sha256` pinning, all failed due to coverage version mismatchGiven this categorization the open questions I would have are:
|
I am working on unpinning them, but it will take some time. See https://github.com/google/oss-fuzz/pulls?q=is%3Aopen+is%3Apr+author%3Amaflcko+%22Use+latest+builder%22 for the current progress. |
|
But yeah, I'd say to remove the pin of all projects here. This will fix a few projects, like #12128 (comment). A few will remain broken, but those can be handled later/separately. Also, make sure to rebase again to pick up d63f82f. |
This is done in the interest of assisting google#12075 and google#11626. Currently the Rust toolchain cannot be updated because the latest nightly uses LLVM 18.1.7 and coverage information breaks. This breakage is because LLVM 18.1.7 records coverage information with version "9" but LLVM 18.0.0 recorded coverage information with version "8". This means that the recordings created by Rust binaries use version "9" which are unreadable by the processing that OSS-Fuzz does with the 18.0.0-based toolchain using version "8". This commit updates the Clang toolchain to the latest 18.x.x release to get the two in sync so the same coverage recording version is used.
Pull in LLVM 18.1.7 with Rust to match the previous Clang update
Fixes wolfssl, muduo
These builds will all be broken with the Clang update anyway so go ahead and remove the pins to keep everything on the latest.
alexcrichton
force-pushed
the
update-clang
branch
from
3d874e4
to
d335e91
Compare
|
Sounds good, I've rebased, removed the one-off warning flag allowances, and removed |
Still fails at runtime with:
ERROR: AddressSanitizer: use-after-poison on address 0x506000000050 at pc 0x55df3d07ee4d bp 0x7fffb5205a60 sp 0x7fffb5205a58
but it's at least further than compiling
|
Only envoy/samba needed new flags, looks like the other projects have updated in the meantime and no longer need a fix |
Should fix the presubmit CI in https://github.com/google/oss-fuzz/actions/runs/9696807444/job/26759529232?pr=12077 (#12077) Co-authored-by: MarcoFalke <6399679+MarcoFalke@users.noreply.github.com>
| @@ -54,7 +54,7 @@ apt-get update && apt-get install -y $LLVM_DEP_PACKAGES --no-install-recommends | |||
| # languages, projects, ...) is needed. | |||
| # Check CMAKE_VERSION infra/base-images/base-clang/Dockerfile was released | |||
| # recently enough to fully support this clang version. | |||
| OUR_LLVM_REVISION=llvmorg-18-init-4631-gd50b56d1 | |||
| OUR_LLVM_REVISION=llvmorg-18.1.7 | |||
There was a problem hiding this comment.
nit: there's only a couple commits difference (llvm/llvm-project@llvmorg-18.1.7...llvmorg-18.1.8), but this could be 18.1.8 (which should also be the last 18.x tag).
|
Do we need another trial build here? |
|
A trial build can't hurt, but I'd say that the outstanding fixes, like #12096 should be merged first, then this pull request should be merged or rebased with master to pick up all fixes, then a trial build should be done. |
OK. Let me know when this is ready for another please |
|
|
This is done in the interest of assisting #12075 and #11626. Currently the Rust toolchain cannot be updated because the latest nightly uses LLVM 18.1.7 and coverage information breaks. This breakage is because LLVM 18.1.7 records coverage information with version "9" but LLVM 18.0.0 recorded coverage information with version "8". This means that the recordings created by Rust binaries use version "9" which are unreadable by the processing that OSS-Fuzz does with the 18.0.0-based toolchain using version "8".
This commit updates the Clang toolchain to the latest 18.x.x release to get the two in sync so the same coverage recording version is used.