Review wubkat scripts #180
Conversation
In WebKitGTK gamepad support if provided by the library 'libmanette-0.2-dev'. This dependency is provided through Flatpak SDK and JHBuild. However, Ubuntu 22.04 features the required version of this package so it should be safe to add it to 'gtk/dependencies/atk'.
By default, WebKitGTK uses libsoup-3.0. This dependency is provided through Flatpak SDK and JHBuild. The build flag 'USE_SOUP2=ON' was useful to build WebKitGTK using system libraries only in OSes which didn't feature yet libsoup-3.0 (i.e Ubuntu 18.04). Ubuntu 22.04 features 'libsoup-3.0' so 'USE_SOUP2=ON' is not needed if package 'libsoup-3.0-dev' is installed in the system (currently is not listed at 'gtk/dependencies/atk').
|
Thanks for getting the appIfExists fix in so quickly! The wubkat build got about 44% through today before it broke on: With any luck m_pipeline is conditional on an ifdef that your changes here will help with! I should be able to take a look at this PR a few hours later today. |
There was a problem hiding this comment.
Thank you so much for the in-depth context about the build mechanisms! As was probably obvious, what I did was a mash-up of https://trac.webkit.org/wiki/BuildingGtk and what I found in your scripts. It's great to have a better idea of how all of this works now!
One thing to note for anyone reading along is that currently we are leaving the flatpaks in the "wubkat" lz4 tarball. We could manually add an idempotent removal step here, but I think given the initial sync time for these and that it sounds like the flatpak might become an option in the future for improved stability, I think it makes sense to leave those around for at least a few months in terms of the marginal costs at play (S3 storage costs versus human time required to re-establish their presence in the face of our failsafe timeouts, etc.)
|
@dpino One other stability thing that you might want to consider is whether it's practical to add jobs to webkit's CI system that can be responsible for running clang with the mozsearch indexer plugin active and then publish a resulting compressed archive of the resulting analysis files. This is how Firefox's mozilla-central (m-c) indexing works for searchfox. This historically helped with searchfox's m-c stability because the decision of which revision to make was based on the "latest" linux64 searchfox build artifact published to taskcluster. If the build was broken, we'd just use the previous build that had valid data. We've changed this up a bit now to instead pick a revision based on available code coverage, but the idea is the same. This approach also makes it somewhat more straightforward for the m-c tree's support of having analysis data for all of linux64, win64, osx64, and arm(64?) since we just download their analysis files and run merge-analyses on them. I think in practice all of the jobs are cross-compiled so they may all run on our linux64 builders, so in theory you could have a single machine run all four builds, but by having CI atomically do each build independently things are much saner and the searchfox taskcluster jobs are almost verbatim derived from the normal builders so anyone changing the builders can also easily change the searchfox jobs too. |
This regression broke the WebKitGTK Ubuntu LTS bot yesterday. It's already fixed. The WebKitGTK Ubuntu LTS (Ubuntu 22.04) bot is the best bot to observe in case there's a build failure in WebKit mozsearch. It builds with the One potential fix to prevent building a commit that may fail is to query the last successful build from the WebKitGTK Ubuntu LTS bot and checkout the corresponding commit before starting building. I attempted to do that some time ago when I started with WebKitSearch but I didn't manage to get it working. I wrote a script on how to get the corresponding git commit of the last successful build in WebKitGTK Ubuntu LTS bot. The idea of deploying a build bot for WebKit mozsearch, which can also generate some of the reusable artifacts like analysis index, etc, sounds very interesting. Likely I will explore this idea further. |
1 similar comment
This regression broke the WebKitGTK Ubuntu LTS bot yesterday. It's already fixed. The WebKitGTK Ubuntu LTS (Ubuntu 22.04) bot is the best bot to observe in case there's a build failure in WebKit mozsearch. It builds with the One potential fix to prevent building a commit that may fail is to query the last successful build from the WebKitGTK Ubuntu LTS bot and checkout the corresponding commit before starting building. I attempted to do that some time ago when I started with WebKitSearch but I didn't manage to get it working. I wrote a script on how to get the corresponding git commit of the last successful build in WebKitGTK Ubuntu LTS bot. The idea of deploying a build bot for WebKit mozsearch, which can also generate some of the reusable artifacts like analysis index, etc, sounds very interesting. Likely I will explore this idea further. |
Following up on WebKit/WebKit#1323 (comment)
This PR is not neccesary meant to be merged.
First thing is flatpak is not actually needed. This is a wrong understanding on my side about how things work. Before digging into this let me explain what are the 3 different ways of building WebKitGTK/WPE:
Tools/Scripts/update-webkitgtk-libsscript, which by default uses the Flatpak SDK (notice env var WEBKIT_JHBUILD=0). The Flatpak SDK is a way of building WebKitGTK/WPE inside a container that features all needed dependencies. This container is prebuilt for x86-64. The dependencies are listed in.bstfiles like for instance Tools/buildstream/elements/sdk/libsoup3.bst. Flatpak SDK can even feature updated versions of GCC or clang not available in the host OS. The advantage of building using the Flatpak SDK is that everyone gets the same results regardless of their host system. In other words, building with Flatpak SDK is completely the opposite to building using system libraries.jhbuild.modulesare not so well maintained.Tools/Scripts/update-webkitgtk-libsand simply go ahead and build WebKit. This is actually what the Ubuntu LTS & Debian Stable build bots do, as it can be seen in this build. Notice there is notjhbuildstep (the step that installs library dependencies). How does it work? By default, these bots build with the flag --no-experimetal-features, which is equivalent to set all the features listed WebPreferencesExperimental.yaml to disable. Since no experimental features are used is likely everything will build OK using system libraries.I can explain now why flatpak is not actually needed. The current build script builds WebKitGTK using cmake directly with CC set to clang. My understanding is that this the right way to build if we want mozsearch's clang-plugin to step in and get the code indexed. When I used
Tools/Script/build-webkitto build instead, I didn't see the clang-plugin being used. And since the script builds using cmake directly and not the tooling provided in the WebKit code base, I think neither Flatpak SDK is used neither the dependencies provided through JHBuild would be used.Another conclusion is that building with the
--no-experimental-featuresflag would make the mozsearch builds more resilient and less likely to break. The main disadvantage is that less code would get indexed, specially features that are currently being developed.I've been discussing with some colleagues what would be the best solution to make mozsearch builds more resilient. We think adding mozsearch's clang-plugin in the Flatpak SDK would alllow to build using flatpak, which will make the builds very unlikely to fail since Flatpak SDK is what the build bots use by default.
About the
gi-docgenhack, an alternative way to solve it would have been to build using-DENABLE_DOCUMENTATION=OFF. I think Ubuntu 22.04 features the right version ofgi-docgenso this dependency can now be satisfied through system libraries (gi-docgenshould be added toTools/gtk/dependencies/apt). Something similar happens-DENABLE_GAMEPADand-DUSE_LIBSOUP2=ON. Gamepad support is provided through the librarylibmanette-0.2-devandlibsoup3support (default version now) is provided through librarylibsoup-3.0-dev. Both libraries are featured in Ubuntu 22.04 and should be added toTools/gtk/dependencies/apt.I'm going to open a PR to fix the error you pointed in WebKit/WebKit#1323 (comment) and also add the missing system dependencies in
Tools/Scripts/gtk/apt.