Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce control flow verification. #213

Merged
merged 2 commits into from May 25, 2018

Conversation

@dvander
Copy link
Member

dvander commented May 24, 2018

This is something I've wanted to do for years, and @peace-maker finally convinced me to do it. SourcePawn needs some internal concept of a control-flow graph. It is the gateway to many important optimizations and transformations, and it will also make our method verification much more rigorous. In fact, it is not possible to attempt many optimizations without veracity.

Overview

This patch introduces the ControlFlowGraph data structure, which is a collection of Blocks. A block (or "basic block") is a unit of control flow. It has exactly one entry-point and exactly one exit-point. Blocks form a directed, cyclic graph - each one may have any number of successors or predecessors. The entry-point to a method is a single block, and any number of blocks may exit the method.

Verification Changes

ControlFlowGraphs are created by GraphBuilder, which is explained below. The graph builder performs some very, very basic opcode verification - only what is needed to ascertain the block structure. However, this verification does introduce two new SMX requirements:

  1. Jump targets must target an instruction that can be decoded as if ignoring control-flow. Before, it was possible to have a "split" instruction, like push.c <payload> with a jump pointing to the payload. This would decode in the interpreter, but would greatly confuse the JIT. It is now illegal, for the simple reason that analysis is super difficult if this is allowed (the code stream has to be constantly reparsed).
  2. Local control flow in a method may not extend beyond the method. This sounds obvious, but a malformed code stream could be missing a retn instruction, and simply flow into the next method. There was no verification for this before, and it is now formally disallowed.

Testing

I used a corpus of ~6,000 .smx files from the forums and web compiler and confirmed that no new verification failures were introduced, which is a good sign for our scrappy compiler (so far). The time to verify these methods did not significantly change despite the new passes involved, but that may change in the future as we expand verification.

Implementation

GraphBuilder has three steps: prescan, scan, and cleanup.

Prescanning

The prescan step runs through the code stream and ensures it can actually be decoded. That is, the code stream does not terminate in the middle of an instruction. It also computes the actual bounds of the method, and verifies jump and switch instructions. Finally, along the way, it computes two important pieces of information:

  1. An instruction bitmap, that maps which cells contain an opcode (versus a payload/parameter).
  2. An jump target bitmap. Any instruction which is the target of a jump, gets marked in this map.

Scanning

The scan step is a fairly simple breadth-first algorithm to build a ControlFlowGraph. It creates an entry block, and walks the instruction stream looking for where to create new blocks. A few scenarios create a new basic block:

  1. If an instruction was marked as a jump target, this means the instruction has two entry-points: the preceding instruction, and a jump from somewhere else. A loop would cause this, for example. This situation will terminate the current basic block, adding an implicit jump to a new block at the current code position.
  2. If an instruction is an unconditional jump, the current block is terminated, and a new block is created at the jump target.
  3. If an instruction is a conditional jump, then two new blocks must be created: one for the next instruction (the jump-not-taken case), and one for the jump target. The current block is terminated.
  4. If an instruction is a switch, then a block must be created for each case (including the default case), and the current block is terminated.

Finally, a return statement terminates a block. The algorithm is a graph algorithm - all paths are searched, until either (1) all blocks are terminated, or (2) the code stream ends abruptly. It uses a work-queue to avoid recursion, and a hash table to ensure that blocks are not duplicated.

Cleanup

The cleanup step iterates all blocks looking for any that were not terminated. In theory, this could only happen on at most one block - and only if the method is malformed (for example, does not contain a retn instruction). If this is the case, verification fails.

In debug builds it also verifies that no disconnected blocks exist. Since it is a graph algorithm, all blocks should be connected.

@dvander dvander force-pushed the cfg branch 2 times, most recently from 76ab485 to beec699 May 24, 2018
dvander added 2 commits May 22, 2018
This introduces three new data structures. The first is ControlFlowGraph,
which represents a collection of basic blocks for a method. A basic block is
a unit of code that terminates in a control flow instruction, and for which
the only entry point is the start of the block. Basic blocks are represented by
the new Block class.

Finally, there is a new GraphBuilder class which can extract a ControlFlowGraph
for a method. It also performs basic verification on control-flow, namely:
 - Jump targets must be a valid decoded instruction in the method.
 - The method's local control flow does not escape the method bounds.
This removes checks from MethodVerifier that are now unnecessary. Readability
of the code stream is guaranteed by GraphBuilder, as well as the validity of
jump targets.
@dvander dvander merged commit 88ea50e into master May 25, 2018
2 checks passed
2 checks passed
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@dvander dvander deleted the cfg branch May 25, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.