How to create a non-standard function definition?

[sampletext32]

I'm trying to re a x86 game engine, and it has a ton of classes. Some member functions "return" a float in ST registers, which doesn't correspond to any known calling convention, but it is as it is.
I wasn't able to find any meaningful way to create a function definition to accommodate that behavior and kept relying on comments in the vtable to keep track.
This is very annoying and I'd like to have an ability to strongly type those member functions in the vtables.
If there is a known way - please tell me.
P.s. engine functions reside in a separate dll, so I've made a custom data archive to transfer structs and functions between parts of the project, that is why I want to strongly type those across boundaries.

[ryanmkurtz]

From the CodeBrowser, Edit -> Options for '<your program>' -> Specification Extensions

Here, you can import an XML file that you define that contains your new calling convention. Here are some examples:

• https://github.com/NationalSecurityAgency/ghidra/blob/master/Ghidra/Processors/AARCH64/data/extensions/objc/objc_msgSend_stub.xml
• https://github.com/NationalSecurityAgency/ghidra/blob/master/Ghidra/Processors/x86/data/extensions/rust/unix64/cc.xml

[ABratovic]

Some member functions "return" a float in ST registers, which doesn't correspond to any known calling convention

Could be 32bit GNU. Have a look at p22 in Agner's calling_conventions.pdf at https://www.agner.org/optimize/#manuals

[ZX41R]

One thing I would check before adding a new convention: for 32-bit x86, returning float/double in ST0 is already in the standard compiler specs. Both x86win.cspec and x86gcc.cspec have the float output pentry on ST0 for the normal prototypes, including __thiscall.

So if the only odd part is the x87 return, a normal typed prototype such as float __thiscall fn(...) should be enough. If Ghidra is not carrying that through your vtable/data archive boundary, I would first verify the program's selected compiler spec and the calling convention on the function/type itself. A custom XML spec is still the right escape hatch if the engine has unusual stack cleanup or this placement, but the ST0 return by itself should not require one.