Adapt basic block analysis for the Hexagon architecture.

[Rot127]

Your checklist for this pull request

• I've read the guidelines for contributing to this repository
• I made sure to follow the project's coding style
• I've documented or updated the documentation of every function and struct this PR changes. If not so I've explained why.
• I've added tests that prove my fix is effective or that my feature works (if possible)
• I've updated the rizin book with the relevant information (if needed)

Detailed description

The run_basic_block_analysis function does not continue analysis if it encounters an register jump and analysis.jmp.after=true. This is not correct, since analysis.jmp.after=true explicitly states that the analysis should continue after a jump instruction.

Now to the big problem.

All additions I made to the code are only executed if the analysis runs on the Hexagon architecture (see usage of is_hexagon flag). Because if it changes its analysis behavior for all architectures a lot (>60) tests fail.

Test plan

Tests were added.
Only the analysis.ijmp missing branch bug failed. Could someone, who is more familiar than me with the x86_64 arch, please take a look at this one? I can't really tell whether it is good that it failed or not.

Closing issues

kinda closes #2047 (Since for all other architectures than Hexagon the problem persists.)
closes #1796
closes rizinorg/rz-hexagon#36

Tiny additional feature added
Added analysis.trap.after=false config. Set to true, it will continue the analysis after a trap instruction.

[Rot127]

Just a random idea which came to my mind.
It would be possible to allow the user to fine grade when the analysis should be continue after jumps.
Something like that:

analysis.jmp.after.outofmap     // Continue analysis if jump leaves mapped memory region.
analysis.jmp.after.reg          // Continue analysis after a register jump.
analysis.jmp.after.cond         // Continue analysis after a conditional jump.
analysis.jmp.after.outfcn       // Continue analysis if jump leads out of function.

It would allow the user to configure the analysis style best fitting to the platform being analysed.
Do you like the idea?

[XVilka]

Regarding:

analysis.jmp.after.outofmap     // Continue analysis if jump leaves mapped memory region.
analysis.jmp.after.reg          // Continue analysis after a register jump.
analysis.jmp.after.cond         // Continue analysis after a conditional jump.
analysis.jmp.after.outfcn       // Continue analysis if jump leads out of function.

The idea is good, but this should be thoroughly tested with various combinations for main architectures.

[Rot127]

The idea is good, but this should be thoroughly tested with various combinations for main architectures.

After spending more time to adapt run_basic_block_analysis() for Hexagon, I think it is better to write a function for each architecture.
run_basic_block_analysis() is already really complex and adding exceptions everywhere will make it only worse.
Besides, shouldn't there only one way to correctly analyze assembly of an architecture? Or are there exceptions?

[Rot127]

Please wait with merging. It needs more testing.
Recognized a few mistakes and meanwhile am confused at which point it is right to treat an instruction as branch and when to continue analysis etc. Need to draw it out on paper again.

[Rot127]

After doing some reversing with the current state of this PR I think it should be ready to merge. The result is fairly decent.
Sorry for not marking it as draft in the first place.

[wargio]

these checks are kinda weird, i'm not sure about this, but maybe i would change this to a if-else-if-else statement.