core/vm: improved EVM run loop & instruction calling #3378

obscuren · 2016-11-29T14:23:18Z

Review #3348 first

This PR implements a refactored EVM and EVM run loop.

Simplified run loop

The run loop, which previously contained custom opcode executes have been removed and has been simplified to a few checks; including:

memory expansion
gas calculation & metering
instruction execution

Each operation consists out of 4 elements:

Execution function
gas cost function
stack validation function
memory size function

The execution function implements the operation's runtime behaviour, the gas cost function implements the operation gas costs function and greatly depends on the memory and stack, the stack validation function validates the stack and makes sure that enough items can be popped off and pushed on and the memory size function calculates the memory required for the operation and returns it.

Optional gas metering

This PR also allows the EVM to go unmetered. This is helpful for offline operations such as contract call's.

mention-bot · 2016-11-29T14:23:20Z

@obscuren, thanks for your PR! By analyzing the history of the files in this pull request, we identified @Arachnid, @tgerring and @Gustav-Simonsson to be potential reviewers.

holiman · 2016-11-30T19:31:38Z

core/call_context.go

 	// Depth check execution. Fail if we're trying to execute above the
 	// limit.
-	if env.Depth() > int(params.CallCreateDepth.Int64()) {
-		caller.ReturnGas(gas, gasPrice)
+	if env.Depth > int(params.CallCreateDepth.Int64()) {


Is not this limit deprecated nowadays, with EIP150? So this check should only be for pre-150, if I'm reading it right. And the same comment for Call above and Create below.

If it's left here because it's practically unreachable anyway, perhaps a comment about that would be in order.

Wrong PR. This was changed in the previous PR (which this one is based on). You're right though, but unfortunately there seem to be a few tests that still test this clause so I cannot remove this yet.

holiman · 2016-11-30T21:01:47Z

core/vm/gas_table.go

+
+		if eip158 {
+			// if empty and transfers value
+			if env.StateDB.Empty(address) && env.StateDB.GetBalance(contract.Address()).BitLen() > 0 {


The uglyness of env.StateDB.GetBalance(contract.Address()).BitLen() > 0 could be hidden within env.StateDB.HasBalance(contract.Address) , instead of having it all over the place.

And if there is ever a 0-value-account-bloomfilter or whatever, you can swap it out in one place.

This would add an additional requirement on the already fat StateDB so I'd rather not do this yet.

Your 0-value-account is a good argument to add this, however, I'll think about this a little bit

holiman · 2016-11-30T21:27:44Z

core/vm/vm.go

-			// non 0 => non 0 (or 0 => 0)
-			g = params.SstoreResetGas
+		var memorySize *big.Int
+		// calucalte the new memory size and expand the memory to fit


s/calucalte/calculate

obscuren · 2016-12-02T15:59:09Z

Preliminary benchmarks have given the following results:

Old vs new (metered):

benchmark                         old ns/op     new ns/op     delta
BenchmarkVmFibonacci16Tests-4     37668307      21323060      -43.39%

Old vs new (unmetered):

benchmark                         old ns/op     new ns/op     delta
BenchmarkVmFibonacci16Tests-4     37668307      14204094      -62.29%

fjl · 2016-12-15T20:45:03Z

core/vm/jump_table.go

+var jumpTable = [256]operation{
+	ADD: operation{
+		execute:       opAdd,
+		gasCost:       makeGenericGasFunc(ADD),


It feels like an additional indirection to have global gasTable and then create a function that looks up
in the table. Why not have constGasFunc(GasFastestStep) here?

func constGasFunc(gas *big.Int) gasFunc { return func(gt params.GasTable, env *Environment, contract *Contract, stack *Stack, mem *Memory, memorySize *big.Int) *big.Int { return gas } }

Fair point. It was laziness, actually.

fjl · 2016-12-15T21:02:25Z

core/vm/environment.go

@@ -35,6 +37,7 @@ type (

 // Context provides the EVM with auxilary information. Once provided it shouldn't be modified.
 type Context struct {
+	context.Context


I don't like this. The context should be passed into the function as a separate parameter.
Making it part of a struct is usually wrong. See context documentation:

Programs that use Contexts should follow these rules to keep interfaces consistent across packages and enable static analysis tools to check context propagation:

Do not store Contexts inside a struct type; instead, pass a Context explicitly to each function that needs it. The Context should be the first parameter, typically named ctx:

It's also misplaced in the VM Context struct it is supposed to contain info about the "blockchain context" in which execution is taking place.

If you want to enable quitting via context.Context, please pass it explicitly.
For recursive calls, you'll probably need to make additional unexported versions
of call, create, etc. because they don't need to take the context when recursing.

Yes, I'm aware of that. But circumstances are different here as well since the context is fully optional. Please provide an alternative approach.

If it can run without context, I'd suggest adding a Cancel method on environment that flips the atomic stop value.

That's one way @karalabe and I thought about it. Will think about this some more.

One thing I don't like about the current EVM is that it may only be used once, this is completely opaque and only becomes apparent when one inspects the source code or reads the documentation. This becomes apparent whit something like a Cancel method is added which sets an atomic field as it it will also introduce weird race conditions.

The main problem is that when the EVM is cancelled and the abort flag is set you'll no longer we able to run any other program, which imo kind of sucks.

Hmm, I suggested it because you made EVM/Environment single use. Why is it a problem though?

You can always make the EVM struct itself smaller by moving more fields into Context and by making the Context field a pointer. That would resolve potential problems with allocation because Context can be reused.

Thinking about it more, it feels like we are running in circles here. The Interpreter contained in EVM is supposed to be reusable for some reason, so all runtime state (including depth, etc) must be held in locals or in the EVM struct. Now you say that you're not happy with EVM being single use. Maybe we should make Interpreter single use and EVM reusable/thread safe?

I know that it had been since forever, but I've never liked the idea that it is single use, imo it's poorly designed and I'd like to change it. The Interpreter currently can be re-used, the EVM can not. The only reason the EVM can not be reused is because of the abort field that may be set.

Perhaps it is Context that should be single use. Argument for this is that you'll never apply a transaction twice and a context is based on a transaction (given it has a field like Origin).

fjl · 2016-12-15T21:02:56Z

core/vm/environment.go

@@ -66,29 +69,55 @@ type Environment struct {
 	// Depth is the current call stack
 	Depth int


I wonder why Depth is exported now that I look at this again.

It doesn't need to be (anymore). Will unexport.

Previously we had a large "requirement" check loop which switched around every opcode and calculated the gas, checked stack, and resized the memory when needed. This has been refactored in to the EVM operation table witch acts as a large jump table for the EVM containing functions for stack and gas requirement and execution. In addition the EVM can now also be used unmetered by setting the `DisableGasMetering` flag on the EVM config. This will cause the EVM to run unmetered without gas. Preliminary benchmarks have given the following results: **Old vs new (metered):** benchmark old ns/op new ns/op delta BenchmarkVmFibonacci16Tests-4 37668307 21323060 -43.39% **Old vs new (unmetered):** benchmark old ns/op new ns/op delta BenchmarkVmFibonacci16Tests-4 37668307 14204094 -62.29%

Implemented RunWithContext which allows the execution of the EVM to be halted mid-flight. This is useful when using the EVM unmetered or when using the Light Client, which requires control over the EVM run loop.

Renamed Environment to EVM and EVM to Interpreter. The rational behind this that the Environment used to be an interface and had multiple implementations across the code base. All of the logic implemented in the Environment is part of the EVM. The old EVM is actually the interpreter, which interprets the byte code. The old EVM is part of the EVM as a whole.

Removed the context.Context cancellation approach from vm.Environment and added a Cancel method instead; this however makes the environment a one shot object due to the `evm.abort` being set to 1, disallowing any further operation to be done using that environment.

obscuren · 2016-12-23T10:15:44Z

Review scheduled on 30th dec 15.00 (@fjl, @karalabe)

karalabe

SGTM per review call

The run loop, which previously contained custom opcode executes have been removed and has been simplified to a few checks. Each operation consists of 4 elements: execution function, gas cost function, stack validation function and memory size function. The execution function implements the operation's runtime behaviour, the gas cost function implements the operation gas costs function and greatly depends on the memory and stack, the stack validation function validates the stack and makes sure that enough items can be popped off and pushed on and the memory size function calculates the memory required for the operation and returns it. This commit also allows the EVM to go unmetered. This is helpful for offline operations such as contract calls.

obscuren added core type:feature labels Nov 29, 2016

obscuren added this to the 1.5.5 milestone Nov 29, 2016

obscuren force-pushed the improved-evm branch from eaf48f9 to 6bf9c0c Compare November 30, 2016 15:50

holiman reviewed Nov 30, 2016

View reviewed changes

obscuren force-pushed the improved-evm branch 4 times, most recently from 08cfbfc to ffa5bcb Compare December 2, 2016 15:56

obscuren force-pushed the improved-evm branch 2 times, most recently from c2e20bc to c868275 Compare December 2, 2016 16:33

obscuren added review and removed in progress labels Dec 2, 2016

obscuren force-pushed the improved-evm branch 4 times, most recently from 09f8858 to 81a28a1 Compare December 6, 2016 16:55

obscuren modified the milestones: 1.5.6, 1.5.5 Dec 11, 2016

obscuren force-pushed the improved-evm branch 3 times, most recently from e59adce to e0b64a4 Compare December 15, 2016 13:01

obscuren requested review from karalabe and fjl December 15, 2016 20:25

fjl requested changes Dec 15, 2016

View reviewed changes

obscuren force-pushed the improved-evm branch 2 times, most recently from 3140ced to 248a93b Compare December 23, 2016 10:06

obscuren added 14 commits December 23, 2016 11:06

core/vm: removal of experimental EVM

3a37e1b

core/vm: added RunWithContext to the EVM for cancellation

243c668

Implemented RunWithContext which allows the execution of the EVM to be halted mid-flight. This is useful when using the EVM unmetered or when using the Light Client, which requires control over the EVM run loop.

core/vm: inline docs

b4fff69

core/vm, tests: refactored precompiled contracts

0abf68b

cmd/evm: allow no gas metering

0ad8aaa

core/vm/runtime: removed leftover ruleset

556ab6f

core, core/vm: review changed from ethereum#3348

514f733

core, miner: review changes ethereum#3348 - ApplyTransaction API change

7c2109b

core, core/vm, tests: improved array tables & env context handling

54ab362

core/vm: unexported Depth (review change)

4e777fb

core/vm: removed gasTable global (review change)

f02b401

obscuren force-pushed the improved-evm branch from 248a93b to f02b401 Compare December 23, 2016 10:07

obscuren added 2 commits January 4, 2017 15:03

core/vm: customisable jump table

007cae7

core/vm: remove EVMInterpreter

6ed1f53

obscuren force-pushed the improved-evm branch from 465d8d6 to 6ed1f53 Compare January 4, 2017 14:41

karalabe approved these changes Jan 5, 2017

View reviewed changes

fjl approved these changes Jan 5, 2017

View reviewed changes

fjl merged commit bbc4ea4 into ethereum:master Jan 5, 2017

fjl removed the review label Jan 5, 2017

obscuren deleted the improved-evm branch January 5, 2017 10:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core/vm: improved EVM run loop & instruction calling #3378

core/vm: improved EVM run loop & instruction calling #3378

obscuren commented Nov 29, 2016 •

edited

mention-bot commented Nov 29, 2016

holiman Nov 30, 2016

obscuren Dec 2, 2016

holiman Nov 30, 2016

obscuren Dec 2, 2016

holiman Nov 30, 2016

obscuren commented Dec 2, 2016 •

edited

fjl Dec 15, 2016

obscuren Dec 16, 2016

fjl Dec 15, 2016

obscuren Dec 16, 2016

fjl Dec 16, 2016 •

edited

obscuren Dec 16, 2016 •

edited

obscuren Dec 16, 2016

fjl Dec 17, 2016

fjl Dec 17, 2016

fjl Dec 17, 2016

obscuren Dec 17, 2016

fjl Dec 15, 2016

obscuren Dec 16, 2016

obscuren commented Dec 23, 2016

karalabe left a comment

		@@ -66,29 +69,55 @@ type Environment struct {
		// Depth is the current call stack
		Depth int

core/vm: improved EVM run loop & instruction calling #3378

core/vm: improved EVM run loop & instruction calling #3378

Conversation

obscuren commented Nov 29, 2016 • edited

Simplified run loop

Optional gas metering

mention-bot commented Nov 29, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

obscuren commented Dec 2, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fjl Dec 16, 2016 • edited

Choose a reason for hiding this comment

obscuren Dec 16, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

obscuren commented Dec 23, 2016

karalabe left a comment

Choose a reason for hiding this comment

obscuren commented Nov 29, 2016 •

edited

obscuren commented Dec 2, 2016 •

edited

fjl Dec 16, 2016 •

edited

obscuren Dec 16, 2016 •

edited