Build system module dependencies

[davidgiven]

Hello,

I'm looking at packaging Ghidra for a Linux distro. Unfortunately, that distro doesn't have gradle, which means I'm having to redo the build system entirely. It's quite complex!

I see that the source is divided into many different modules. For example, Ghidra/Framework/Docking and Ghidra/Framework/Help. However, I don't see anywhere in the gradle build system where the dependencies between modules are encoded. For example, Help depends on Docking, so the latter has to be built first.

I have found the per-module build.gradle files, but these are extremely confusing --- e.g. Docking's one has app project('Help') in its dependencies section, which seems to be completely wrong; Help depends on Docking, not the other way round.

Can anyone enlighten me as to how this all works?

[d-millar]

@davidgiven While I admire your courage, I feel compelled to point out that you’re signing up for a world of pain here. Even if you succeed, staying in sync with future releases will be something of a nightmare. Have you tried building gradle from source for your distribution?

[davidgiven]

Sadly, gradle has an issue where version n can't be built from source without version n-1 of gradle, meaning that it can't be built in isolation. That's why you don't get gradle in distros like Debian.

I do actually have all of ghidra building and running using a simple Makefile already. Unfortunately, the unique way in which ghidra loads resources are loaded means that it doesn't actually work, meaning that I'm going to have to do it without taking shortcuts.

BTW, I assume there's a reason why ghidra doesn't use the traditional get resource() mechanism --- what is it?

[DemiMarie]

It’s actually worse: Gradle cannot be built from source without a nightly version of Gradle, which is often the previous nightly (as opposed to a release version). This is very different than Rust, which guarantees that version N can be built from source with version N - 1.

The only distro (Arch) that I know builds an up-to-date Gradle downloads a binary bootstrapping blob, which Debian (reasonably) refuses to allow. IIUC Debian does allow a one-time bootstrap, but Gradle would require repeatedly downloading binary blobs and that is not allowed.

[sre4ever]

That's what the Gradle folks tell people to scare them but the real story is slightly more subtle.

I've published here the last 200 versions of Gradle used to build Gradle: it appears to be a combination of nightlies, nightlies from topic branches, milestone/rc and releases n-1. That alone won't tell you which ones are the most often used (though that could be somewhat deduced from commit logs and dates), but it does tell you that some nightlies are used for longer than one night ^ ^.

The lack of guarantee is real though. If a rebuild of Gradle succeeds with any other version than the one used upstream, that's just luck, and that situation seems unlikely to change.

[d-millar]

Will have to differ your last question to rest of the team - I don't know off the top of my head. May I ask which distribution specifically you are targeting? (I've never encountered the n-1 issue, so am curious.)

[davidgiven]

I'm aiming at Debian, but right now I just want to get things building.

There's a reasonable writeup of the gradle problems here: gradle/gradle#16512 (Also, small clarification: you can get Gradle in Debian, but it's the ancient gradle 4.4.1 which ghidra won't build with.)

[d-millar]

While we're waiting on answers to your questions, I did try the following:

(1) Installed Debian 11 (VMWare)
(2) Downloaded gradle 8.10 (https://gradle.org/install, "complete" under "Installing manually")
(3) Installed a new jdk (AMazon Corretto 21)
(4) Did update-alternatives to point to that jdk's java.
(5) sudo apt install pip (seemed to be missing in the default install)
(5) Installed Ghidra in the usual way and was good to go.

[ryanmkurtz]

Can't you just use the gradle wrapper that is at the top level of the ghidra repo to build?

[ryanmkurtz]

I have found the per-module build.gradle files, but these are extremely confusing --- e.g. Docking's one has app project('Help') in its dependencies section, which seems to be completely wrong; Help depends on Docking, not the other way round.

It's totally defined in the per-module build.gradle files. Docking indeed depends on Help. Why does that seem completely wrong?

[davidgiven]

Can't you just use the gradle wrapper that is at the top level of the ghidra repo to build?

No; one of the rules of repository packaging is that you have to be able to do offline builds using only artifacts which are already in the repository. So downloading a gradle binary is out of the question. (BTW, as @d-millar said, manually building ghidra on Debian already works fine. I'm wanting to produce an actual Debian package for it.)

It's totally defined in the per-module build.gradle files. Docking indeed depends on Help. Why does that seem completely wrong?

Oops --- my mistake! I was looking at the wrong file...

Okay, that's a relief. I should be able to pull the dependency information out of the gradle files. Let me have a play with this and see how it goes.

[sre4ever]

Hi,

You might be interested to learn that I'm currently working on upgrading Gradle in Debian. It's not there yet but it's making its way and you can follow the progress there.

If you want to know how I ended up on that discussion, it's that because was referencing gradle/gradle#16512 about packaging Gradle for Debian.

[davidgiven]

That's good news! How are you handling the bootstrapping problem?

For my progress, I have ghidra building with a replacement build system, but it doesn't actually run. I haven't got round to debugging why yet (probably I'm putting resource files in the wrong place).

[sre4ever]

My current plan is to implement a multi-stage build, I haven't worked out the details or a PoC yet but plan to do that soon. Basically that would be a multi-stage build:

• 1st stage on a developer system with a binary distribution; builds scripts are added to capture a "trace" of compilation and JAR assembly tasks that can be used to rebuild using a shell script or plain old Makefiles (my fallback would be to scan debug log); developer reviews/tests and commits the generated scripts
• 2nd ad 3rd stage on a CI: first build gradle using the generated scripts and installs it in a temporary location, then rebuild gradle using that temporary build
• eventually repeat and check that we get the same binary result and the same scripts for 1st stage
• run test suites to check what was broken in the process.

If I can get all that to work that would take care of the reproducibility issue and the need to use a recent version of Gradle to rebuild itself.