Re-think Function Types and Callable "Things"

[thestr4ng3r]

There currently are something like function types in the types database, but they aren't really types, as you for example can't use them as a type for a variable.
Instead, the current "function types" are more like a database of signatures for imported functions, identified by a name, and used in things like type propagation, which dirtily operates directly on the sdb.
Having such info in general is necessary.

In addition to that, there are also real analyzed functions that define their signature depending on their RzAnalysisVars. Afaik there is some conversion available to take the info from vars and put it into the db explained above, but such a conversion creates redundant info, must always be in sync and is thus highly error-prone and inconvenient.

However we also need actual function types to be used anywhere in structs, variables, etc. for use-cases like analyzing code using this:

struct JNINativeInterface {
    void*       reserved0;
    void*       reserved1;
    void*       reserved2;
    void*       reserved3;
    jint        (*GetVersion)(JNIEnv *);
    jclass      (*DefineClass)(JNIEnv*, const char*, jobject, const jbyte*,
                        jsize);
    jclass      (*FindClass)(JNIEnv*, const char*);
    jmethodID   (*FromReflectedMethod)(JNIEnv*, jobject);
    jfieldID    (*FromReflectedField)(JNIEnv*, jobject);
    /* spec doesn't show jboolean parameter */
    jobject     (*ToReflectedMethod)(JNIEnv*, jclass, jmethodID, jboolean);
...

There should be some kind of unified notion of something that is callable. I could imagine this to be for example a struct RzCallable that contains the return type, argument types, optionally argument names and a calling convention.
This struct could then be created from any source imaginable like RzAnalysisFunction variables, a database of imports, a type, etc. and used in analysis algorithms without having to think about what the original source was.

Now there are some decisions to be made how to best approach this:

• Do we have a dedicated "function type" plus a "callable" and convert from one or the other or have only one and for example embed an RzCallable in our regular type structs?
• Related to the above question, is there something that has to be present in an arbitrary "callable" from analysis perspective that does not belong into a "function type" or vice-versa?
• For function types, are they RzTypes or RzBaseTypes? If they are base types, that would actually be a little bit similar to what is already present in the database, but it would be harder to handle in cases like the struct above where the types are define in-line.

Current plan for implementing this

Function type is an RzType. This requires a few more changes for the imported function db but it's more flexible and closer to our final goal.
We shouldn't overengineer it so let's try to specify RzCallable and reuse it directly in RzType, refactoring this will be simple if we find out later it doesn't suffice.

Structure draft:

typedef struct rz_callable_arg_t {
	RZ_NULLABLE char *name; // optional, function types in C can have names to their args but they don't have to
	RzType *type;
} RzCallableArg;

typedef struct rz_callable_t {
	RzType *ret;
	RzPVector /*<RzCallableArg>*/ args;
	RZ_NULLABLE const char *cc; // optional
} RzCallable;

typedef enum {
	...,
	RZ_TYPE_KIND_FUNCTION // or _CALLABLE
} RzTypeKind;

struct rz_type_t {
	RzTypeKind kind;
	union {
		...,
		RzCallable function; // or callable
	};
};

Steps:

• Add RzCallable describing anything that can be called as explained above. This should probably go into the new RzTypes module (so this issue is sort of blocked by Bundle all types functionality in a new module RzTypes #369 can be done now).
• Add a new RzTypeKind which just contains a single RzCallable and thus wraps it and makes it possible to use function types anywhere, including such inline definitions as above.
• Convert the old function databases. They should be considered a database of known library function callables, not actual function types. This refactored database should have an api that only works with RzCallable (not RzType) and no raw sdb. Concretely I could imagine a hashtable of name -> RzCallable in RzAnalysis. This step might be a bit bigger since there is probably a lot that accesses the database.

Further steps:

• Add an api to RzAnalysisFunction that constructs an RzCallable from its variables
• Add an api to RzAnalysis so you can query the callable at any given address, no matter whether it's a regular function or an import. Care has to be taken of indirect function calls. So the address of a function somewhere in memory is not the same as an actual call target. About this point I am not 100% sure yet how exactly to design this API, it depends on how the info is used in analysis.

[XVilka]

Why? I already work on that.