Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Subroutines and Static Jumps for the EVM #615
#Current PR: https://github.com/ethereum/EIPs/blob/master/EIPS/eip-615.md
This EIP introduces new EVM opcodes and conditions on EVM code to support subroutines and static jumps, disallow dynamic jumps, and thereby make EVM code amenable to linear-time static analysis. This will provide for better compilation, interpretation, transpilation, and formal analysis of EVM code.
EVM code is difficult to analyze formally, hobbling a critical tool for preventing the many expensive bugs our blockchain has experienced. And all current implementations of the Ethereum Virtual Machine, including the just-in-time compilers, are much too slow. This proposal identifies a major reason for this and proposes changes to the EVM specification to address the problem.
In particular, it imposes restrictions on current EVM code and proposes new instructions to help provide for
These goals are achieved by
We also propose to validate—in linear time—that EVM contracts correctly use subroutines, avoid misuse of the stack, and meet other safety conditions before placing them on the blockchain. Validated code precludes most runtime exceptions and the need to test for them. And well-behaved control flow and use of the stack makes life easier for interpreters, compilers, formal analysis, and other tools.
Currently the EVM supports dynamic jumps, where the address to jump to is an argument on the stack. These dynamic jumps obscure the structure of the code and thus mostly inhibit control- and data-flow analysis. This puts the quality and speed of optimized compilation fundamentally at odds. Further, since every jump can potentially be to any jump destination in the code, the number of possible paths through the code goes up as the product of the number of jumps by the number of destinations, as does the time complexity of static analysis. But absent dynamic jumps code can be statically analyzed in linear time.
Static analysis includes validation, and much of optimization, compilation, transpilation, and formal analysis; every part of the tool chain benefits when linear-time analysis is available. In particular, faster control-flow analysis means faster compilation of EVM code to native code, and better data-flow analysis can help the compiler and the interpreter better track the size of the values on the stack and use native 64- and 32-bit operations when possible. Also, conditions which are checked at validation time don’t have to be checked repeatedly at runtime.
Note that analyses of a contract’s bytecode before execution—such as optimizations performed before interpretation, JIT compilation, and on-the-fly machine code generation—must be efficient and linear-time. Otherwise, specially crafted contracts can be used as attack vectors against clients that use static analysis of EVM code before the creation or execution of contracts.
We propose to deprecate two existing instructions—
name instructions with one, two, and two or more arguments, respectively. An instruction is represented in the bytecode as a single-byte opcode. Any arguments are laid out as immediate data bytes following the opcode inline, interpreted as RLP-encoded, values, sequences, or lists of twos-complement signed integers. (Negative values are reserved for extensions.)
The two most important uses of
Static jumps are provided by
To support subroutines,
These five simple instructions form the primitives of the proposal.
In order to validate subroutines, EVM bytecode must be sequentially scanned matching jumps to their destinations. Since creation code has to contain the runtime code as data, that code might not correctly validate in the creation context and also does not have to be validated prior to the execution of the creation code. Because of that, there needs to be a way to place data into the bytecode that will be skipped over and not validated. Such data will prove useful for other purposes as well.
The primitive operations provide for static jumps. Dynamic jumps are also used for O(1) indirection: an address to jump to is selected to push on the stack and be jumped to. So we also propose two more instructions to provide for constrained indirection. We support these with vectors of
Dynamic jumps to a
Dynamic jumps to a
These operations provide convenient access to subroutine parameters and other variables at fixed stack offsets within a subroutine.
Jumps to and returns from subroutines are described here in terms of
We will adopt the following conventions to describe the machine state:
Defining the frame pointer so as to include the arguments is unconventional, but better fits our stack semantics and simplifies the remainder of the proposal.
The frame pointer and return stacks are internal to the subroutine mechanism, and not directly accessible to the program. This is necessary to prevent the program from modifying its state in ways that could be invalid.
The first instruction of an array of EVM bytecode begins execution of a main routine with no arguments,
Execution of a subroutine begins with
Execution of a subroutine is suspended during and resumed after execution of nested subroutines, and ends upon encountering a
For example, after a
the stack looks like this
We would like to consider EVM code valid if and only if no execution of the program can lead to an exceptional halting state. But we must and will validate code in linear time. So our validation does not consider the code’s data and computations, only its control flow and stack use. This means we will reject programs with invalid code paths, even if those paths cannot be executed at runtime. Most conditions can be validated, and will not need to be checked at runtime; the exceptions are sufficient gas and sufficient stack. So some false negatives and runtime checks are the price we pay for linear time validation.
Execution is as defined in the Yellow Paper—a sequence of changes in the EVM state. The conditions on valid code are preserved by state changes. At runtime, if execution of an instruction would violate a condition the execution is in an exceptional halting state. The yellow paper defines five such states.
We propose to expand and extend the Yellow Paper conditions to handle the new instructions we propose.
To handle the return stack we expand the conditions on stack size:
Given our more detailed description of the data stack we restate condition 3—stack underflow—as
Since the various
To handle the new jump instructions and subroutine boundaries we expand the conditions on jumps and jump destinations.
We have two new conditions on execution to ensure consistent use of the stack by subroutines:
Finally, we have one condition that prevents pathological uses of the stack:
In practice, we must test at runtime for conditions 1 and 2—sufficient gas and sufficient stack. We don’t know how much gas there will be, we don’t know how deep a recursion may go, and analysis of stack depth even for non-recursive programs is non-trivial.
All of the remaining conditions we validate statically.
Validation comprises two tasks:
We sketch out these two validation functions in pseudo-C below. To simplify the presentation only the five primitives are handled (
Validating that jumps are to valid addresses takes two sequential passes over the bytecode—one to build sets of jump destinations and subroutine entry points, another to check that addresses jumped to are in the appropriate sets.
Note that code like this is already run by EVMs to check dynamic jumps, including building the jump destination set every time a contract is run, and doing a lookup in the jump destination set before every jump.
This function can be seen as a symbolic execution of a subroutine in the EVM code, where only the effect of the instructions on the state being validated is computed. Thus the structure of this function is very similar to an EVM interpreter. This function can also be seen as an acyclic traversal of the directed graph formed by taking instructions as vertexes and sequential and branching connections as edges, checking conditions along the way. The traversal is accomplished via recursion, and cycles are broken by returning when a vertex which has already been visited is reached. The time complexity of this traversal is O(n-vertices + n-edges)
The basic approach is to call
Costs & Codes
All of the instructions are O(1) with a small constant, requiring just a few machine operations each, whereas a
We suggest the following opcodes:
These changes would need to be implemented in phases at decent intervals:
If desired, the period of deprecation can be extended indefinitely by continuing to accept code not versioned as new—but without validation. That is, by delaying step 2. Since we must continue to run old code this is not technically difficult.
Implementation of this proposal need not be difficult, At the least, interpreters can simply be extended with the new opcodes and run unchanged otherwise. The new opcodes require only stacks for the frame pointers and return offsets and the few pushes, pops, and assignments described above. Compiled code can use native call instructions. Further optimizations include minimizing runtime checks for exceptions and otherwise taking advantage of validated code wherever possible. An untested prototype is available in the lead author's cpp-ethereum repository.
This is a very useful proposal, especially for anything that involves decompilation of bytecode.
I have a few questions regarding the variable access section.
What is the semantics of GETLOCAL n, when n is greater than the number of arguments specified in BEGINSUB? I'd suspect an exception is thrown in that case?
It seems that the number of return variables specified in BEGINSUB, isn't used anywhere.
Actually, the following is really confusing to me:
Shouldn't this be rather "set FP to SP" or "set FP to SP - n_args" because the arguments have already been placed on the stack at this point.
Perhaps, Seed. It confuses me too. It's been a long while since I wrote and implemented this, so it will take a little study. Getting near bedtime here, and I've got 100 miles of driving to get home tomorrow, so please be patient. It's on my stack to do a cleanup pass and new PR for this and 616, so I much appreciate your review. (If you read C++ your questions might be answered in the code at https://github.com/ethereum/cpp-ethereum/tree/develop/libevm)
We started specifying the semantics of new instructions in Lem.
One of the things we noticed is that it makes more sense to setup the stack-frame with the BEGINSUB instruction because we have the number of function arguments at hand. It also makes sense when compared native assembly like x86, where the stack-frame is created in the body of the function, not by the CALL instruction.
I'll be continuing the specification next week and will probably have more comments to make.
It pushes FP[-n] on the stack.
As for setting up the stack frame, the semantics is given as:
Execution of a subroutine begins with
I'm not sure it matters where or in what order anything in between encountering the JUMPSUB and executing the BEGINSUB is described as happening, so long as the program winds up executing the instruction after the BEGINSUB with the correct stack. So get the semantics right and we can get the English right.
@seed I've edited to make clear that SP is mu-sub-s in the yellow paper, and removed the reference in the text to SP being the stack size. The rest of the pointer arithmetic in the text is (I think) now consistent with the stack growing towards lower addresses, so that positive offsets off the stack pointer and negative offsets off the frame pointer both reach into the stack frame. The validation pseudocode I have not fixed.
Thanks for updating the proposal. It does make a lot more sense.
Note that μs growing towards lower addresses doesn't make much sense to me in the context of the yellow paper because the EVM doesn't expose stack addresses in any way.
Let me try to confirm my understanding of the proposal with an example:
where μs = local2, μs = local1, μs = arg2, ...
@seed The EVM doesn't expose stack addresses, but in 9.5 the Yellow Paper says, "Stack items are added or removed from the left-most, lower-indexed portion of the series." And throughout Appendix H we have notation like
the stack looks like
The proposal uses the minus sign (-) instead of the em-dash (—) symbols in a few places which can lead to confusion.
I believe the first - should be a —. Also the syntax introduced, FP[-n], doesn't seem to be used elsewhere.
Here again em-dashes should be used.
You are right on the punctuation. I was too lazy to crawl through layers of menus to find the em-dash. I took the array syntax from the yellow paper (e.g. µs). It would be nice if markdown wasn't too primitive (that is, more primitive than what we used 40 years ago) to support subscripts, but might we still do better to switch from SP and FP to µs and µf?
You are also right about the example diagram.
We finished the Lem implementation of JUMPIF, JUMPTO, JUMPSUB, RETURNSUB, BEGINSUB, GETLOCAL, PUTLOCAL.
Few questions/comments resulting from this:
To properly test both the Lem code and the cpp-Ethereum client at this point we need the Solidity compiler to output subroutine instructions.
@seed Are you talking about initializing the return stack in the cpp-ethereum code or in the validation pseudocode?
No JUMPDEST is involved. Just a JUMPSUB to a BEGINSUB, and a RETURNSUB back to the JUMPSUB. The x86 CALL instruction pushes the PC on the stack and RET pops it and jumps to it. Then the PC gets incremented in the usual way. That was the model I had in mind, it's how the C++ code is written for JUMP, and I think it comports with (136) in the Yellow Paper. (The C++ code is not written to match this model for RETURNSUB, although the behavior is the same.)
For formal analysis purposes, all of this is implementable right now as an intermediate language that's easily compilable to evm. PUTLOCAL and GETLOCAL are indeed very useful but there's no need for separate opcodes - stack could be mapped to negative memory and accessed with MLOAD/MSTORE (ie. the stack starts at (-32) and every push goes downwards). Space of one-byte opcodes is limited.
Why not add it as a precompiled contract? Ie. the very first instruction (in EVM) calls it with code as an argument. The gas cost of that particular call could be set to zero, in effect working only as a costless (except storage) marking for a different vm version. This would totally avoid problems due to deprecating opcodes and backwards compatibility, allowing for a much more fundamental changes.
In principle I very much dislike the idea of breaking backwards compatibility. What if someone has a compiled contract on some private blockchain? It would be impossible to deploy on the public one without recompilation. What if there's some private or just unknown language that compiles to evm? Its compiler would have to be rewritten. Even if one doesn't exist at the moment (most likely), potential ones that may exist may not be written at all, because the risk of compatibility-breaking changes would be judged as too high. For all these reasons modern processors still support old 16 bit code - even including undocumented accidental instructions (ie. ffreep)!