[CMake] Generate a usable compile_commands.json #16718

TingPing · 2023-08-15T20:14:48Z

`5c9dd47`

[CMake] Generate a usable compile_commands.json
https://bugs.webkit.org/show_bug.cgi?id=260218

Reviewed by Michael Catanzaro.

When using unified builds CMake generates a compile_commands.json
that isn't useful by many tools as it lacks much of the sources.

This adds a script to expand UnifiedSources into the full list
of sources. It outputs it to webkit_compile_commands.json.

* Source/cmake/WebKitCommon.cmake:
* Tools/Scripts/rewrite-compile-commands: Added.

Canonical link: https://commits.webkit.org/266947@main

9463525

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 wincairo
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🛠 gtk
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🧪 gtk-wk2
✅ 🛠 🧪 jsc	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🧪 api-gtk
✅ 🛠 🧪 jsc-arm64	✅ 🛠 tv	✅ 🧪 mac-AS-debug-wk2	✅ 🛠 jsc-armv7
	✅ 🛠 tv-sim		✅ 🧪 jsc-armv7-tests
✅ 🛠 🧪 merge	✅ 🛠 watch		✅ 🛠 jsc-mips
	✅ 🛠 watch-sim		✅ 🧪 jsc-mips-tests

webkit-early-warning-system · 2023-08-15T20:25:19Z

EWS run on previous version of this PR (hash d3d2671)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
~~🧪 style~~	~~🛠 ios~~	~~🛠 mac~~	~~🛠 wpe~~	⏳ 🛠 wincairo
~~🧪 bindings~~	~~🛠 ios-sim~~	~~🛠 mac-AS-debug~~	~~🧪 wpe-wk2~~
~~🧪 webkitperl~~	~~🧪 ios-wk2~~	~~🧪 api-mac~~	~~🛠 gtk~~
	~~🧪 ios-wk2-wpt~~	~~🧪 mac-wk1~~	~~🧪 gtk-wk2~~
⏳ 🛠 🧪 jsc	~~🧪 api-ios~~	~~🧪 mac-wk2~~	~~🧪 api-gtk~~
⏳ 🛠 🧪 jsc-arm64	⏳ 🛠 tv	~~🧪 mac-AS-debug-wk2~~	~~🛠 jsc-armv7~~
	⏳ 🛠 tv-sim	~~🧪 mac-wk2-stress~~	~~🧪 jsc-armv7-tests~~
	⏳ 🛠 watch		~~🛠 jsc-mips~~
	⏳ 🛠 watch-sim		~~🧪 jsc-mips-tests~~

webkit-early-warning-system · 2023-08-15T20:38:13Z

EWS run on previous version of this PR (hash 0f0aa58)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	❌ 🛠 wincairo
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	⏳ 🧪 wpe-wk2
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🛠 gtk
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	~~🧪 gtk-wk2~~
⏳ 🛠 🧪 jsc	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🧪 api-gtk
✅ 🛠 🧪 jsc-arm64	✅ 🛠 tv	~~🧪 mac-AS-debug-wk2~~	✅ 🛠 jsc-armv7
	✅ 🛠 tv-sim		✅ 🧪 jsc-armv7-tests
	✅ 🛠 watch		✅ 🛠 jsc-mips
	✅ 🛠 watch-sim		✅ 🧪 jsc-mips-tests

webkit-early-warning-system · 2023-08-15T22:22:49Z

EWS run on previous version of this PR (hash 6ba926a)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	❌ 🛠 wincairo
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	✅ 🧪 wpe-wk2
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🛠 gtk
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🧪 gtk-wk2
✅ 🛠 🧪 jsc	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🧪 api-gtk
✅ 🛠 🧪 jsc-arm64	✅ 🛠 tv	✅ 🧪 mac-AS-debug-wk2	✅ 🛠 jsc-armv7
	✅ 🛠 tv-sim		✅ 🧪 jsc-armv7-tests
	✅ 🛠 watch		✅ 🛠 jsc-mips
	✅ 🛠 watch-sim		✅ 🧪 jsc-mips-tests

mcatanzaro

I would add a --help argument, even though the script is probably not intended to be called manually. Somebody else will thank you for it a few years from now....

TingPing · 2023-08-16T04:04:54Z

help is automatically added by argparse.

I’ll fix the windows failure though.

webkit-early-warning-system · 2023-08-16T14:29:48Z

EWS run on current version of this PR (hash 9463525)

Misc	iOS, tvOS & watchOS	macOS	Linux	Windows
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 wincairo
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-AS-debug	~~🧪 wpe-wk2~~
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🛠 gtk
	✅ 🧪 ios-wk2-wpt	✅ 🧪 mac-wk1	✅ 🧪 gtk-wk2
✅ 🛠 🧪 jsc	✅ 🧪 api-ios	✅ 🧪 mac-wk2	✅ 🧪 api-gtk
✅ 🛠 🧪 jsc-arm64	✅ 🛠 tv	✅ 🧪 mac-AS-debug-wk2	✅ 🛠 jsc-armv7
	✅ 🛠 tv-sim		✅ 🧪 jsc-armv7-tests
✅ 🛠 🧪 merge	✅ 🛠 watch		✅ 🛠 jsc-mips
	✅ 🛠 watch-sim		✅ 🧪 jsc-mips-tests

https://bugs.webkit.org/show_bug.cgi?id=260218 Reviewed by Michael Catanzaro. When using unified builds CMake generates a compile_commands.json that isn't useful by many tools as it lacks much of the sources. This adds a script to expand UnifiedSources into the full list of sources. It outputs it to webkit_compile_commands.json. * Source/cmake/WebKitCommon.cmake: * Tools/Scripts/rewrite-compile-commands: Added. Canonical link: https://commits.webkit.org/266947@main

webkit-commit-queue · 2023-08-16T15:23:58Z

Committed 266947@main (5c9dd47): https://commits.webkit.org/266947@main

Reviewed commits have been landed. Closing PR #16718 and removing active labels.

philn · 2023-08-25T13:31:54Z

I can't find out how to specify a custom file to clangd, seems like it really expects a file named compile_commands.json... Could we store this custom webkit file in a sub-directory as a workaround?

TingPing · 2023-08-25T22:51:48Z

Could we store this custom webkit file in a sub-directory as a workaround?

Yes I think this would be a good idea as I've hit other tools being hardcoded as well.

philn · 2023-08-26T11:48:53Z

Tools/Scripts/rewrite-compile-commands

+                    })
+                    break
+            else:
+                print('Failed to find', include_path)


This triggers for many files here, *MessageReceiver.cpp that don't exist and files from WebCore/DerivedSources/ mostly.

Is this OK?

Hmm in my local builds it found everything. What was your specific build config (and which commit maybe)?

Also was it ran after everything was built? You can run ninja webkit_compile_commands.json after a build to run it last.

We are kinda in a weird situation where this script technically depends on every generated source, maybe we could somehow actually collect that list, but I don't know a simple solution... So maybe it runs out of order sometimes.

I didn't want to rely on checking for files existing but the rules of where a file ends up didn't have a clear pattern, but maybe some static mapping is possible.

Just doing a clean GTK Debug build triggers the issue here, with sccache enabled, (edit: with ASan enabled too) the script runs early on, I suppose before the MessageReceiver.cpp files have been generated.

Actually those MessageReceiver.cpp files are for features we don't enable, such as GPUProcess, so it's expected that they shouldn't exist.

Actually those MessageReceiver.cpp files are for features we don't enable, such as GPUProcess, so it's expected that they shouldn't exist.

The file has to exist in some include path, it is compiled, otherwise the build would fail.

TingPing self-assigned this Aug 15, 2023

TingPing requested a review from JonWBedard as a code owner August 15, 2023 20:14

TingPing added the CMake Bugzilla component for CMake build system changes label Aug 15, 2023

TingPing requested a review from a team August 15, 2023 20:15

TingPing force-pushed the eng/CMake-Generate-a-usable-compile_commands-json branch from d3d2671 to 0f0aa58 Compare August 15, 2023 20:19

webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Aug 15, 2023

TingPing removed the merging-blocked Applied to prevent a change from being merged label Aug 15, 2023

TingPing force-pushed the eng/CMake-Generate-a-usable-compile_commands-json branch from 0f0aa58 to 6ba926a Compare August 15, 2023 22:21

mcatanzaro approved these changes Aug 16, 2023

View reviewed changes

webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Aug 16, 2023

TingPing removed the merging-blocked Applied to prevent a change from being merged label Aug 16, 2023

TingPing force-pushed the eng/CMake-Generate-a-usable-compile_commands-json branch from 6ba926a to 9463525 Compare August 16, 2023 14:29

TingPing added the merge-queue Applied to send a pull request to merge-queue label Aug 16, 2023

webkit-commit-queue force-pushed the eng/CMake-Generate-a-usable-compile_commands-json branch from 9463525 to 5c9dd47 Compare August 16, 2023 15:23

webkit-commit-queue merged commit 5c9dd47 into WebKit:main Aug 16, 2023

webkit-commit-queue removed the merge-queue Applied to send a pull request to merge-queue label Aug 16, 2023

philn reviewed Aug 26, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CMake] Generate a usable compile_commands.json #16718

[CMake] Generate a usable compile_commands.json #16718

TingPing commented Aug 15, 2023 •

edited by webkit-early-warning-system

Loading

webkit-early-warning-system commented Aug 15, 2023 •

edited

Loading

webkit-early-warning-system commented Aug 15, 2023 •

edited

Loading

webkit-early-warning-system commented Aug 15, 2023 •

edited

Loading

mcatanzaro left a comment

TingPing commented Aug 16, 2023 •

edited

Loading

webkit-early-warning-system commented Aug 16, 2023 •

edited

Loading

webkit-commit-queue commented Aug 16, 2023

philn commented Aug 25, 2023

TingPing commented Aug 25, 2023

philn Aug 26, 2023

TingPing Aug 26, 2023 •

edited

Loading

TingPing Aug 26, 2023 •

edited

Loading

philn Aug 27, 2023 •

edited

Loading

philn Aug 27, 2023

TingPing Aug 27, 2023

[CMake] Generate a usable compile_commands.json #16718

[CMake] Generate a usable compile_commands.json #16718

Conversation

TingPing commented Aug 15, 2023 • edited by webkit-early-warning-system Loading

5c9dd47

webkit-early-warning-system commented Aug 15, 2023 • edited Loading

webkit-early-warning-system commented Aug 15, 2023 • edited Loading

webkit-early-warning-system commented Aug 15, 2023 • edited Loading

mcatanzaro left a comment

Choose a reason for hiding this comment

TingPing commented Aug 16, 2023 • edited Loading

webkit-early-warning-system commented Aug 16, 2023 • edited Loading

webkit-commit-queue commented Aug 16, 2023

philn commented Aug 25, 2023

TingPing commented Aug 25, 2023

philn Aug 26, 2023

Choose a reason for hiding this comment

TingPing Aug 26, 2023 • edited Loading

Choose a reason for hiding this comment

TingPing Aug 26, 2023 • edited Loading

Choose a reason for hiding this comment

philn Aug 27, 2023 • edited Loading

Choose a reason for hiding this comment

philn Aug 27, 2023

Choose a reason for hiding this comment

TingPing Aug 27, 2023

Choose a reason for hiding this comment

TingPing commented Aug 15, 2023 •

edited by webkit-early-warning-system

Loading

`5c9dd47`

webkit-early-warning-system commented Aug 15, 2023 •

edited

Loading

webkit-early-warning-system commented Aug 15, 2023 •

edited

Loading

webkit-early-warning-system commented Aug 15, 2023 •

edited

Loading

TingPing commented Aug 16, 2023 •

edited

Loading

webkit-early-warning-system commented Aug 16, 2023 •

edited

Loading

TingPing Aug 26, 2023 •

edited

Loading

TingPing Aug 26, 2023 •

edited

Loading

philn Aug 27, 2023 •

edited

Loading