invalid instruction handling: same across plaforms, and cleanly presented to clients

[derekbruening]

From bruen...@google.com on December 03, 2012 17:25:27

on windows we don't pass an invalid instr to the app: we assume we know
that it's invalid and we forge the exception

on linux: we set up a bb w/ an invalid instr which we then execute, mainly
b/c we can't easily emulate a core dump from user mode.

short-term: hide the invalid instr from the client to match windows and
avoid issues w/ clients handling invalid instrs.

long-term: client may want to change the instr, so we should present the
bb. we then need to decide how the API should handle them:

1. instruct clients to always check for invalid instrs before doing any
   queries on them. change our samples to do so. could use an iterator
   that skips both meta and invalid (xref issue provide instrlist app instruction iterator  #998 ).

2. make all of our IR routines all
   gracefully handle them. for some cases, like iterating over operands,
   this is appealing: an invalid instr just has 0 operands. but this
   doesn't seem a feasible solution in general, b/c some routines, like
   instr_length(), need to return an error: and then the client has to check
   for errors anyway. but we should probably make some IR routines handle
   them rather than requiring a check in the caller.

3. Qin's proposal: have a separate bb event for invalid instrs
   (we separate to own bb).

Original issue: http://code.google.com/p/dynamorio/issues/detail?id=1000

[derekbruening]

From zhao...@google.com on December 04, 2012 11:16:14

An related issue:
for Linux, the current behavior of the handling invalid instr is not correctly.
On seeing the invalid instr, DR will pass the invalid instruction to client before execution, which triggers a SIGILL later.
Then on the basic block recreation, i.e. recreate_bb_ilist, DR won't append the instruction into the bb->ilist, and the client won't see it!

This is caused by different value of bb->app_interp set by init_build_bb.
On the first build_bb_ilist, which is called from build_basic_block_fragment, bb->app_interp is true, so we will append that on into ilist. On the later bb_build_ilist called from recreate_bb_ilist, bb->app_interp is false, and we will destroy the instr instead.
So client will see different view on the same code.

[derekbruening]

Xref #3212
Xref #57
Xref #431

[derekbruening]

For #3212 I am going to make Linux behave like Windows. That means the client won't see the culprit insruction at all: it will just see a sudden fault. That's the "short-term" above. This issue remains open for a "long-term" solution.