Ghidra Scripting API changes

[ReversingWithMe]

Ghidra's scripting interface changes slightly between versions. Sometimes finding how the scripting API has changed and documentation of what exactly changed and how is challenging. This causes some scripts to require updates from version to version, at unexpected times. For some updates this is simple refactors as was the case with Comments being moved. Other times it could be new parameters, or large changes such as the Dwarf update with Ghidra 11.

In the case of simple Ghidra refactors, Java scripting is more of an issue than Jython's as it is tough to do try-catch imports the way that python supports.

As far as I am aware the only true API/semver guarantees are around the FlatAPI. I'm wondering if more things could get pulled into FlatAPI or into services to encourage more stable APIs. I don't have good statistics or data around what has changed with sub-versions and patches, but wondering if there are thoughts around this.

Other than expanding what needs to be covered, as would be the case with expanding FlatAPI, marking things with deprecated tags for 1 major or like 2 minor releases would give some time to apply fixes. I think this would only help me if I stayed up to date and never jumped versions, so the longer the better, but knowing explicitly what is idiomatic/preferred would be awesome.

[dragonmacher]

Generally, for public interfaces, we try to deprecate changes. We also tend to keep the deprecated code around for multiple major releases, which ends up working out to be many years. We try to be especially careful with the GhidraScript class, since that is the primary interface to Ghidra for script writers.

I'm sure we sometimes make mistakes and do not correctly deprecate, but I am not aware of this happening in the GhidraScript API. Perhaps more subtle is how we refactor the codebase in general and how those changes affect the scripts that we ship. If user have scripts that call the scripts that we provide, we probably break those user scripts on every release.

[dragonmacher]

Look, im not saying that its an easy problem to solve, but I think a little discretion would go a long way. At the very least a commit message better than "changed."

I agree and believe that we do this now, for the most part. The exception is that we do not use commit messages for documentation. As I mentioned, our release notes will call out important API changes users should know about. I acknowledge that better commit messages would be helpful though.

I cant speak to your internal dev practices, but I suspect many free fixes would go a long way. Like, for instance, asking developers to think about what might break and balance that with the value they think their changes add.

Again, I think we do this now. The exception here is for 'internal' APIs, which we break often. Where we are deficient here is in not doing a good job letting the outside know what is public vs internal. We hope to change this.

Again, not saying ghidra devs are stupid, just that everyone might be happier if we respect eachothers time and effort.

"Stupid is a stupid does" -Forest Gump

We all make mistakes.

Maybe you guys prioritize your programming prerogatives over our opinions, and thats totally fair for you to do, but then that also comes with tradeoffs, and everyone should be aware of that.

Yes. This is my central point, that many of these decisions come down to tradeoffs. Again, how best to communicate our API expectations to the outside?

I might also add that scripting an application is pretty common functionality, but i cannot attest to how well any one application does it. However, many open source projects (gdb comes to mind first) have scriptable interfaces, so perhaps there are ideas out there that are implementable on your typical underfunded open source keystone software :)

I personally think our scripting is better than average. Admittedly, I do not write many scripts and I am certainly not objective.

---

We appreciate the feedback. It's how we make the tool better.

[metamaterialsuit]

Well, one thing that is commonly done is to make additive changes to an API, not destructive ones. That would simplify things for scripting consumers but make your own code more difficult to navigate, I suppose. I am not a java programmer, and do not know what mechanisms it has to either enforce or warn developers about using deprecated interfaces.

The reason I reacted to this comment so strongly yesterday even though I've been complaining about it for years is because the specific breaking change I encountered that seems to be a simple consolidation of function parameters into an aggregate parameter. While we can disagree, and im aware that this is one single case, but breaking all dependencies on that class in out of tree code for avoiding autocompleting a few parameter names makes it seem like there isn't much a bar to clear for a breaking api change.

If there was some truly good argument for it that I don't understand, I apologize, but even then, you could have just left the old load around and implemented it in terms of new load.

Anyway, I've said my piece. Thanks for working on Ghidra and putting up with us!

[ReversingWithMe]

For what it is worth, I had started throwing AI at this question to figure out whether I actually thought there was an issue. The Ghidra team had really good deprecated tags, and so my question turned into whether I thought those should just get exposed differently or could I just track those for when my scripts should change.

e.g., Should I just create a script that scans deprecated and flags me when I should change, or should I submit a ticket for api flagging those differently somehow

https://reversingwithme.github.io/GhidraMigration/?from=Ghidra+11.2.1&to=Ghidra+12.1.3&scope=changes&evidence=all&tab=script

Which doesn't help at all for the Jython -> Python conversion which for the short term is the bigger deal. But having AI go slurp up scripts with the appropriate license and fixing them up seems easy enough that I haven't spent time on this at all.

AI/codex pretty trivially fixes scripts, with even a recent-ish cutoff model, so I had started being less worried about this, especially with AI being everywhere. I haven't run into a recent case where gemma/GPT can't just fix the script with a few iterations with support docs and headless also.

I had interest in collecting statistics on how often scripts off github are breaking, or even what are the API I thought statistically should be protected as more of first class features (flatapi), but largely seemed to no longer be a headache for folks.

[ryanmkurtz]

The reason I reacted to this comment so strongly yesterday even though I've been complaining about it for years is because the specific breaking change I encountered that seems to be a simple consolidation of function parameters into an aggregate parameter.

@metamaterialsuit are you referring to ImporterSettings?

[metamaterialsuit]

yep!