Skip to content

Commit

Permalink
feat(avm): implement serialization for all existing operations (#4338)
Browse files Browse the repository at this point in the history 
This PR implements serialization for all existing operations. It also
adds `indirect` and other missing parameters to the operations (but does
not use them).

It's a big PR but the most interesting changes are
* The `serialization/` folder.
* Things that are not tests.

What I like
* Easy to add types and modify existing wire formats.
* Very explicit tests that will tell you exactly what we expect (too
repetitive? see [DRY vs DAMP in
tests](https://abseil.io/resources/swe-book/html/ch12.html#tests_and_code_sharing_dampcomma_not_dr)).
* Well-defined interfaces.
* Instruction serialization from object is also supported! This made
creating bytecode a breeze.

What I don't like
* ~~Due to limitations of TypeScript, there is more boilerplate in each
operation than I would like. Time permitting I will try to simplify
things further.~~ Thanks to @Maddiaa0 for coming up with a
boilerplate-less version.
* Some typing is not as strict (uses of `any`). 

What's still missing
* External calls cannot be added to the instruction set, because they
create a dependency cycle. This stems from the context using
`decodeFromBytecode` or, to paraphrase, from an instruction using an
interpreter (this was not introduced in this PR). It might be fixable
but not in this PR.

Closes #4218.

---------

Co-authored-by: Maddiaa0 <47148561+Maddiaa0@users.noreply.github.com>
  • Loading branch information
@fcarreiro @Maddiaa0
fcarreiro and Maddiaa0 committed Feb 1, 2024
1 parent 5d3fce3 commit 13e0683
Show file tree
Hide file tree
Showing 36 changed files with 2,190 additions and 948 deletions.
8 changes: 5 additions & 3 deletions yarn-project/acir-simulator/src/avm/avm_context.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@ import { AvmMachineState } from './avm_machine_state.js';
import { AvmMessageCallResult } from './avm_message_call_result.js';
import { AvmInterpreterError, executeAvm } from './interpreter/index.js';
import { AvmJournal } from './journal/journal.js';
import { decodeBytecode } from './opcodes/decode_bytecode.js';
import { Instruction } from './opcodes/index.js';
import { Instruction } from './opcodes/instruction.js';
import { decodeFromBytecode } from './serialization/bytecode_serialization.js';

// FIXME: dependency cycle.

/**
* Avm Executor manages the execution of the AVM
Expand Down Expand Up @@ -47,7 +49,7 @@ export class AvmContext {
throw new NoBytecodeFoundInterpreterError(this.executionEnvironment.address);
}

const instructions: Instruction[] = decodeBytecode(bytecode);
const instructions: Instruction[] = decodeFromBytecode(bytecode);

const machineState = new AvmMachineState(this.executionEnvironment);
return executeAvm(machineState, this.journal, instructions);
Expand Down
26 changes: 12 additions & 14 deletions yarn-project/acir-simulator/src/avm/index.test.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,30 +4,27 @@ import { AvmTestContractArtifact } from '@aztec/noir-contracts';
import { mock } from 'jest-mock-extended';

import { AvmMachineState } from './avm_machine_state.js';
import { TypeTag } from './avm_memory_types.js';
import { initExecutionEnvironment } from './fixtures/index.js';
import { executeAvm } from './interpreter/interpreter.js';
import { AvmJournal } from './journal/journal.js';
import { decodeBytecode } from './opcodes/decode_bytecode.js';
import { encodeToBytecode } from './opcodes/encode_to_bytecode.js';
import { Opcode } from './opcodes/opcodes.js';
import { Add, CalldataCopy, Return } from './opcodes/index.js';
import { decodeFromBytecode, encodeToBytecode } from './serialization/bytecode_serialization.js';

describe('avm', () => {
it('Should execute bytecode that performs basic addition', async () => {
const calldata: Fr[] = [new Fr(1), new Fr(2)];
const journal = mock<AvmJournal>();

// Construct bytecode
const calldataCopyArgs = [0, 2, 0];
const addArgs = [0, 1, 2];
const returnArgs = [2, 1];

const calldataCopyBytecode = encodeToBytecode(Opcode.CALLDATACOPY, calldataCopyArgs);
const addBytecode = encodeToBytecode(Opcode.ADD, addArgs);
const returnBytecode = encodeToBytecode(Opcode.RETURN, returnArgs);
const fullBytecode = Buffer.concat([calldataCopyBytecode, addBytecode, returnBytecode]);
const bytecode = encodeToBytecode([
new CalldataCopy(/*indirect=*/ 0, /*cdOffset=*/ 0, /*copySize=*/ 2, /*dstOffset=*/ 0),
new Add(/*indirect=*/ 0, TypeTag.FIELD, /*aOffset=*/ 0, /*bOffset=*/ 1, /*dstOffset=*/ 2),
new Return(/*indirect=*/ 0, /*returnOffset=*/ 2, /*copySize=*/ 1),
]);

// Decode bytecode into instructions
const instructions = decodeBytecode(fullBytecode);
const instructions = decodeFromBytecode(bytecode);

// Execute instructions
const context = new AvmMachineState(initExecutionEnvironment({ calldata }));
Expand All @@ -41,15 +38,16 @@ describe('avm', () => {
});

describe('testing transpiled Noir contracts', () => {
it('Should execute contract function that performs addition', async () => {
// TODO(https://github.com/AztecProtocol/aztec-packages/issues/4361): sync wire format w/transpiler.
it.skip('Should execute contract function that performs addition', async () => {
const calldata: Fr[] = [new Fr(1), new Fr(2)];
const journal = mock<AvmJournal>();

// Get contract function artifact
const addArtifact = AvmTestContractArtifact.functions.find(f => f.name === 'avm_addArgsReturn')!;

// Decode bytecode into instructions
const instructions = decodeBytecode(Buffer.from(addArtifact.bytecode, 'base64'));
const instructions = decodeFromBytecode(Buffer.from(addArtifact.bytecode, 'base64'));

// Execute instructions
const context = new AvmMachineState(initExecutionEnvironment({ calldata }));
Expand Down
7 changes: 4 additions & 3 deletions yarn-project/acir-simulator/src/avm/interpreter/interpreter.test.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { Fr } from '@aztec/foundation/fields';
import { MockProxy, mock } from 'jest-mock-extended';

import { AvmMachineState } from '../avm_machine_state.js';
import { TypeTag } from '../avm_memory_types.js';
import { initExecutionEnvironment } from '../fixtures/index.js';
import { AvmJournal } from '../journal/journal.js';
import { Add } from '../opcodes/arithmetic.js';
Expand All @@ -22,9 +23,9 @@ describe('interpreter', () => {
const calldata: Fr[] = [new Fr(1), new Fr(2)];

const instructions: Instruction[] = [
new CalldataCopy(/*cdOffset=*/ 0, /*copySize=*/ 2, /*dstOffset=*/ 0),
new Add(/*aOffset=*/ 0, /*bOffset=*/ 1, /*dstOffset=*/ 2),
new Return(/*returnOffset=*/ 2, /*copySize=*/ 1),
new CalldataCopy(/*indirect=*/ 0, /*cdOffset=*/ 0, /*copySize=*/ 2, /*dstOffset=*/ 0),
new Add(/*indirect=*/ 0, TypeTag.FIELD, /*aOffset=*/ 0, /*bOffset=*/ 1, /*dstOffset=*/ 2),
new Return(/*indirect=*/ 0, /*returnOffset=*/ 2, /*copySize=*/ 1),
];

const machineState = new AvmMachineState(initExecutionEnvironment({ calldata }));
Expand Down
11 changes: 5 additions & 6 deletions yarn-project/acir-simulator/src/avm/interpreter/interpreter.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { strict as assert } from 'assert';
import { AvmMachineState } from '../avm_machine_state.js';
import { AvmMessageCallResult } from '../avm_message_call_result.js';
import { AvmJournal } from '../journal/index.js';
import { Instruction } from '../opcodes/index.js';
import { Instruction, InstructionExecutionError } from '../opcodes/instruction.js';

/**
* Run the avm
Expand Down Expand Up @@ -36,14 +36,13 @@ export async function executeAvm(
}

return AvmMessageCallResult.success(returnData);
} catch (_e) {
if (!(_e instanceof AvmInterpreterError)) {
throw _e;
} catch (e) {
if (!(e instanceof AvmInterpreterError || e instanceof InstructionExecutionError)) {
throw e;
}

const revertReason: AvmInterpreterError = _e;
const revertData = machineState.getReturnData();
return AvmMessageCallResult.revert(revertData, revertReason);
return AvmMessageCallResult.revert(revertData, /*revertReason=*/ e);
}
}

Expand Down
139 changes: 98 additions & 41 deletions yarn-project/acir-simulator/src/avm/opcodes/accrued_substate.test.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,72 +18,129 @@ describe('Accrued Substate', () => {
machineState = new AvmMachineState(initExecutionEnvironment());
});

it('Should append a new note hash correctly', async () => {
const value = new Field(69n);
machineState.memory.set(0, value);

await new EmitNoteHash(0).execute(machineState, journal);

const journalState = journal.flush();
const expected = [value.toFr()];
expect(journalState.newNoteHashes).toEqual(expected);
describe('EmitNoteHash', () => {
it('Should (de)serialize correctly', () => {
const buf = Buffer.from([
EmitNoteHash.opcode, // opcode
0x01, // indirect
...Buffer.from('12345678', 'hex'), // dstOffset
]);
const inst = new EmitNoteHash(/*indirect=*/ 0x01, /*dstOffset=*/ 0x12345678);

expect(EmitNoteHash.deserialize(buf)).toEqual(inst);
expect(inst.serialize()).toEqual(buf);
});

it('Should append a new note hash correctly', async () => {
const value = new Field(69n);
machineState.memory.set(0, value);

await new EmitNoteHash(/*indirect=*/ 0, /*offset=*/ 0).execute(machineState, journal);

const journalState = journal.flush();
const expected = [value.toFr()];
expect(journalState.newNoteHashes).toEqual(expected);
});
});

it('Should append a new nullifier correctly', async () => {
const value = new Field(69n);
machineState.memory.set(0, value);
describe('EmitNullifier', () => {
it('Should (de)serialize correctly', () => {
const buf = Buffer.from([
EmitNullifier.opcode, // opcode
0x01, // indirect
...Buffer.from('12345678', 'hex'), // dstOffset
]);
const inst = new EmitNullifier(/*indirect=*/ 0x01, /*dstOffset=*/ 0x12345678);

expect(EmitNullifier.deserialize(buf)).toEqual(inst);
expect(inst.serialize()).toEqual(buf);
});

it('Should append a new nullifier correctly', async () => {
const value = new Field(69n);
machineState.memory.set(0, value);

await new EmitNullifier(/*indirect=*/ 0, /*offset=*/ 0).execute(machineState, journal);

const journalState = journal.flush();
const expected = [value.toFr()];
expect(journalState.newNullifiers).toEqual(expected);
});
});

await new EmitNullifier(0).execute(machineState, journal);
describe('EmitUnencryptedLog', () => {
it('Should (de)serialize correctly', () => {
const buf = Buffer.from([
EmitUnencryptedLog.opcode, // opcode
0x01, // indirect
...Buffer.from('12345678', 'hex'), // offset
...Buffer.from('a2345678', 'hex'), // length
]);
const inst = new EmitUnencryptedLog(/*indirect=*/ 0x01, /*dstOffset=*/ 0x12345678, /*length=*/ 0xa2345678);

const journalState = journal.flush();
const expected = [value.toFr()];
expect(journalState.newNullifiers).toEqual(expected);
});
expect(EmitUnencryptedLog.deserialize(buf)).toEqual(inst);
expect(inst.serialize()).toEqual(buf);
});

it('Should append unencrypted logs correctly', async () => {
const startOffset = 0;
it('Should append unencrypted logs correctly', async () => {
const startOffset = 0;

const values = [new Field(69n), new Field(420n), new Field(Field.MODULUS - 1n)];
machineState.memory.setSlice(0, values);
const values = [new Field(69n), new Field(420n), new Field(Field.MODULUS - 1n)];
machineState.memory.setSlice(0, values);

const length = values.length;
const length = values.length;

await new EmitUnencryptedLog(startOffset, length).execute(machineState, journal);
await new EmitUnencryptedLog(/*indirect=*/ 0, /*offset=*/ startOffset, length).execute(machineState, journal);

const journalState = journal.flush();
const expected = values.map(v => v.toFr());
expect(journalState.newLogs).toEqual([expected]);
const journalState = journal.flush();
const expected = values.map(v => v.toFr());
expect(journalState.newLogs).toEqual([expected]);
});
});

it('Should append l1 to l2 messages correctly', async () => {
const startOffset = 0;
describe('SendL2ToL1Message', () => {
it('Should (de)serialize correctly', () => {
const buf = Buffer.from([
SendL2ToL1Message.opcode, // opcode
0x01, // indirect
...Buffer.from('12345678', 'hex'), // offset
...Buffer.from('a2345678', 'hex'), // length
]);
const inst = new SendL2ToL1Message(/*indirect=*/ 0x01, /*dstOffset=*/ 0x12345678, /*length=*/ 0xa2345678);

expect(SendL2ToL1Message.deserialize(buf)).toEqual(inst);
expect(inst.serialize()).toEqual(buf);
});

it('Should append l1 to l2 messages correctly', async () => {
const startOffset = 0;

const values = [new Field(69n), new Field(420n), new Field(Field.MODULUS - 1n)];
machineState.memory.setSlice(0, values);
const values = [new Field(69n), new Field(420n), new Field(Field.MODULUS - 1n)];
machineState.memory.setSlice(0, values);

const length = values.length;
const length = values.length;

await new SendL2ToL1Message(startOffset, length).execute(machineState, journal);
await new SendL2ToL1Message(/*indirect=*/ 0, /*offset=*/ startOffset, length).execute(machineState, journal);

const journalState = journal.flush();
const expected = values.map(v => v.toFr());
expect(journalState.newLogs).toEqual([expected]);
const journalState = journal.flush();
const expected = values.map(v => v.toFr());
expect(journalState.newLogs).toEqual([expected]);
});
});

it('All substate instructions should fail within a static call', async () => {
const executionEnvironment = initExecutionEnvironment({ isStaticCall: true });
machineState = new AvmMachineState(executionEnvironment);

const instructions = [
new EmitNoteHash(0),
new EmitNullifier(0),
new EmitUnencryptedLog(0, 1),
new SendL2ToL1Message(0, 1),
new EmitNoteHash(/*indirect=*/ 0, /*offset=*/ 0),
new EmitNullifier(/*indirect=*/ 0, /*offset=*/ 0),
new EmitUnencryptedLog(/*indirect=*/ 0, /*offset=*/ 0, 1),
new SendL2ToL1Message(/*indirect=*/ 0, /*offset=*/ 0, 1),
];

for (const instruction of instructions) {
const inst = () => instruction.execute(machineState, journal);
await expect(inst()).rejects.toThrowError(StaticCallStorageAlterError);
await expect(instruction.execute(machineState, journal)).rejects.toThrow(StaticCallStorageAlterError);
}
});
});
25 changes: 17 additions & 8 deletions yarn-project/acir-simulator/src/avm/opcodes/accrued_substate.ts
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,16 @@
import { AvmMachineState } from '../avm_machine_state.js';
import { AvmJournal } from '../journal/journal.js';
import { Opcode, OperandType } from '../serialization/instruction_serialization.js';
import { Instruction } from './instruction.js';
import { StaticCallStorageAlterError } from './storage.js';

export class EmitNoteHash extends Instruction {
static type: string = 'EMITNOTEHASH';
static numberOfOperands = 1;
static readonly opcode: Opcode = Opcode.EMITNOTEHASH;
// Informs (de)serialization. See Instruction.deserialize.
static readonly wireFormat = [OperandType.UINT8, OperandType.UINT8, OperandType.UINT32];

constructor(private noteHashOffset: number) {
constructor(private indirect: number, private noteHashOffset: number) {
super();
}

Expand All @@ -25,9 +28,11 @@ export class EmitNoteHash extends Instruction {

export class EmitNullifier extends Instruction {
static type: string = 'EMITNULLIFIER';
static numberOfOperands = 1;
static readonly opcode: Opcode = Opcode.EMITNULLIFIER;
// Informs (de)serialization. See Instruction.deserialize.
static readonly wireFormat = [OperandType.UINT8, OperandType.UINT8, OperandType.UINT32];

constructor(private nullifierOffset: number) {
constructor(private indirect: number, private nullifierOffset: number) {
super();
}

Expand All @@ -45,9 +50,11 @@ export class EmitNullifier extends Instruction {

export class EmitUnencryptedLog extends Instruction {
static type: string = 'EMITUNENCRYPTEDLOG';
static numberOfOperands = 2;
static readonly opcode: Opcode = Opcode.EMITUNENCRYPTEDLOG;
// Informs (de)serialization. See Instruction.deserialize.
static readonly wireFormat = [OperandType.UINT8, OperandType.UINT8, OperandType.UINT32, OperandType.UINT32];

constructor(private logOffset: number, private logSize: number) {
constructor(private indirect: number, private logOffset: number, private logSize: number) {
super();
}

Expand All @@ -65,9 +72,11 @@ export class EmitUnencryptedLog extends Instruction {

export class SendL2ToL1Message extends Instruction {
static type: string = 'EMITUNENCRYPTEDLOG';
static numberOfOperands = 2;
static readonly opcode: Opcode = Opcode.SENDL2TOL1MSG;
// Informs (de)serialization. See Instruction.deserialize.
static readonly wireFormat = [OperandType.UINT8, OperandType.UINT8, OperandType.UINT32, OperandType.UINT32];

constructor(private msgOffset: number, private msgSize: number) {
constructor(private indirect: number, private msgOffset: number, private msgSize: number) {
super();
}

Expand Down

0 comments on commit 13e0683

Please sign in to comment.