sequential allocation into memory (#387)

* disabling the pager in my pretty printing run script

* change the elaboration of `new` and non-local Invocations to match the flat yul object style (#367)

* making a change to double check work flow

* adding a stub for computing the size of a contract; that'll be its own PR entirely one day fairly soon, but for now, we assume that we just need 32 bytes

* using hex values for some boilerplate so that it matches the docs

* reworking constructors to just allocate memory

* adding some helper functions in util. one is a renaming for transactions from child contracts; the other is basically a type-checked comment pointing out uses of some pretty substantian technical debt

* temporarily removing the code to translate child contracts. rewriting the translation of Invocations to remap everything to local invocations instead, into the names that will be created in the flat yul object

* removing an assertion and adding translation of string literals

* whitespace, scala style

* utility function for undoing the transaction renaming

* generate flat yul object from child contracts (#369)

* rename, add boolean for main-ness in translate decl

* changing signature of translateDeclaration to remove a return that doesn't do anything

* whitespace, scala style

* comments

* scraps; adding an assert

* changing how child contracts are processed

* rewrite of translateTransaction

* chainging signature of translateStatement

* more threading through inMain

* whitespace, scala style

* adding some docs

* removing the nicer output for typed names in favor of a bland one, per solc's current limitations. adding a comment about why.

* adding a temporary fix to the top of the dispatch table pending talking with MC"

* removing a stale comment

* comments

* adding missing declarations to invocations

* cleaning up comments and some messy logic around temporary variables

* removing comment

* more hacks for specific tests in advance of type info

* changing hardcoded value in the mustache file for this to be the current contract address

* adding a couple of args to get this to typecheck

* use type annotations in Yul codegen (#377)

* removing a util function to rename functions in the dispatch table; that no longer makes sense with the solution for flattening that we've settled on for now

* adding comments; expanding getcontractname

* ripping out some code, adding comments

* adding a comment that may help to locate a bug later

* moving comment closer to the source and updating it

* comments

* scraps; this change seemed likely but breaks a bunch of tests

* quick function to traverse over an expression and check a given predicate on all the annotations

* fixing some import dependencies

* traversal of a program rather than just an expression; i sure hope i don't have to come back and make this the fully polymorphic map reduce but i bet i will

* adding updates to the expression returned in the checker from just the ReferenceIdentifier cases, after some debugging with @mcoblenz

* adding additional ParserUtil.updateExprType uses as seems right. bunch of tests fail, some of which i expected some i didn't, and i don't know how to fix a lot of them

* updating width calculation to actually inspect the type, now that we know it

* removing an assert that caused a bunch of false negatives.

* small changes per PR comments

* copying a slightly cut down version of linked lists, but this needs to be edited down even more to be the right goal for a demo.

* compute the sizes of contracts for allocation (#378)

* copying a slightly cut down version of linked lists, but this needs to be edited down even more to be the right goal for a demo.

* computing sizes of types recursively

* adding some simple hard coded unit tests

* copying over a helper from type checker tests to test on actual contracts in files

* adding more general tests; it doesn't work but i think i know why

* scraps and printlines from trying to debug sizes

* changing the type of the size computation function so it has enough information, per MC. reworking tests

* renaming size function; fixing the main test

* adding some comments and a variable for pointer sizes

* formatting, reworking base test

* removing the two simpler unit tests. it is hard and somewhat pointless to be the parser and create a contract table by hand, the existing test is informative enough

* fixing type checker errors (#380)

* debugging scraps

* copying a slightly cut down version of linked lists, but this needs to be edited down even more to be the right goal for a demo.

* commenting out spammy prints

* patch from MC

* updating override syntax to not introduce a new field

* updating expr type update to set the location and use the one field rather than build new ones

* removing printlines

* updating logged errors in a few places; this fixes some but not all of #379

* error in resourcesTest was in the test itself; it was incorrectly looking for m to be at int

* formatting, whitespace

* fixed permission passing test; also errors in the definition of the test, not the checker

* fixing all permowned error

* scala style, whitespace

* scraps

* adding a comment, removing a comment, removing an assert

* compute offsets, store fields in memory (#384)

* fixing a bug in computing sizes by translating from set to list; adding an assert; whitespace and style

* adding helper to compute offsets

* typoe and adding a test to demonstrate that the bug computing sizes is gone

* adding a sandbox test case file

* adding a traversal of the declarations sequence in the contract rather than relying on the toList of an unordered collection; using that helper function both in computing sizes and offsets

* refactoring tests for util functions to reuse the code to produce a symbol table; adding tests for computing offests

* style, whitespace

* a couple more tests for offsets

* a couple more tests for offsets

* updating tests, adding comments for myself

* helper to wrap call to field offsets, changing sload and sstore to m*

* updating test to show that fields get overwritten; adding json so it gets run because it passes

* messing with test files

* changing the ganache test script to use the test skipping checking script that i think works across platforms, per #361

* adding sort, per #631

* scala style, fussy fixes to doc strings etc

* scala style, fussy fixes to doc strings etc

* scala style, fussy fixes to doc strings etc

* scala style, fussy fixes to doc strings etc

* some house keeping to clean up a util function, removing a source of bugs related to computing sizes

* refactoring to change the definition of typed name to have an actual type in it, not just its string representation

* adding clause to type name helper for contracts

* whitespace in obsidian files

* housekeeping, adding docs and removing stale todos

* Adding another passing test

* updating travis test script to work with solc 0.8.10

* iterating on travis script

bin/skipped_tests.sh

@@ -3,4 +3,4 @@
 # when run from the top level of the repo, this prints out the names of the obsidian files
 # in the ganache tests that do not also have a json file paired with them -- that means that
 # they won't be run by travis. this also appears in the output of the travis script at the very top.
-comm -13 <(ls resources/tests/GanacheTests/*.json | xargs basename -s '.json') <(ls resources/tests/GanacheTests/*.obs | xargs basename -s '.obs')
+comm -13 <(ls resources/tests/GanacheTests/*.json | xargs basename -s '.json' | sort) <(ls resources/tests/GanacheTests/*.obs | xargs basename -s '.obs' | sort)

resources/tests/GanacheTests/SG.json

@@ -0,0 +1,8 @@
+{
+    "gas" : 30000000,
+    "gasprice" : "0x9184e72a000",
+    "startingeth" : 5000000,
+    "numaccts" : 1,
+    "testexp" : "main()",
+    "expected" : "1800"
+}

resources/tests/GanacheTests/SG.obs

@@ -0,0 +1,29 @@
+contract IntContainer{
+    int x;
+    int y;
+    int z;
+    int f;
+
+    transaction set() {
+        x = -1;
+        f = -1;
+        z = -1;
+        y = -1;
+        x = 5;
+        y = 10;
+        z = 4;
+        f = 9;
+    }
+
+    transaction get() returns int{
+        return x*y*z*f; // 1800
+    }
+}
+
+main contract SG{
+    transaction main() returns int{
+        IntContainer ic = new IntContainer();
+        ic.set();
+        return ic.get();
+    }
+}

resources/tests/GanacheTests/SGTwoContainers.json

@@ -0,0 +1,8 @@
+{
+    "gas" : 30000000,
+    "gasprice" : "0x9184e72a000",
+    "startingeth" : 5000000,
+    "numaccts" : 1,
+    "testexp" : "main()",
+    "expected" : "3600"
+}

resources/tests/GanacheTests/SGTwoContainers.obs

@@ -0,0 +1,31 @@
+contract IntContainer{
+    int x;
+    int y;
+    int z;
+    int f;
+
+    transaction set() {
+	x = -1;
+	f = -1;
+	z = -1;
+	y = -1;
+        x = 5;
+	y = 10;
+	z = 4;
+	f = 9;
+    }
+
+    transaction get() returns int{
+        return x*y*z*f; // 1800
+    }
+}
+
+main contract SGTwoContainers{
+    transaction main() returns int{
+        IntContainer ic1 = new IntContainer();
+        ic1.set();
+	IntContainer ic2 = new IntContainer();
+	ic2.set();
+        return ic1.get()+ic2.get();
+    }
+}

resources/tests/GanacheTests/SetGetNoArgs.obs

@@ -1,7 +1,7 @@
 contract IntContainer{
     int x;
 
-    IntContainer@Owned() {
+    IntContainer() {
         x = 0;
     }
 

resources/tests/GanacheTests/SetGetNoArgsNoConstruct.obs

@@ -6,7 +6,7 @@ contract IntContainer{
     }
 
     transaction set2() {
-	    x = 10;
+        x = 10;
     }
 
     transaction get() returns int{
@@ -18,7 +18,7 @@ main contract SetGetNoArgsNoConstruct{
     transaction main() returns int{
         IntContainer ic = new IntContainer();
         ic.set1();
-	    ic.set2();
+        ic.set2();
         return (ic.get());
     }
 }

src/main/scala/edu/cmu/cs/obsidian/codegen/CodeGenYul.scala

@@ -1,6 +1,7 @@
 package edu.cmu.cs.obsidian.codegen
 
 import edu.cmu.cs.obsidian.CompilerOptions
+import edu.cmu.cs.obsidian.typecheck.StringType
 
 import java.io.{File, FileWriter}
 import java.nio.file.{Files, Path, Paths}
@@ -114,10 +115,7 @@ object CodeGenYul extends CodeGenerator {
       * names that they will have in the flattened translation. e.g. `f()` remains `f()` but `ic.f()` becomes
       * `IntContainer___f(this)` if ic is an IntContainer.
       *
-      * todo: right now we do not actually have enough type information to do the above in a robust way, but we will
-      * add that soon
-      *
-      * @param contract the contract to be translated
+      * @param contract     the contract to be translated
       * @param checkedTable the symbol table for that contract
       * @return the yul
       */
@@ -126,7 +124,7 @@ object CodeGenYul extends CodeGenerator {
 
         // translate declarations
         for (d <- contract.declarations) {
-            decls = decls ++ translateDeclaration(d, contract.name, checkedTable, true)
+            decls = decls ++ translateDeclaration(d, contract.name, checkedTable, inMain = true)
         }
 
         // create runtime object from just the declarations and with the subobject name suffix
@@ -148,15 +146,15 @@ object CodeGenYul extends CodeGenerator {
       * ready to be inserted into the translation of a main yul object. this will rename the transactions according to the
       * contract name.
       *
-      * @param c the contract to be translated
+      * @param c            the contract to be translated
       * @param checkedTable the symbol table of the contract
       * @return the YulObject representing the translation. note that all the fields other than `code` will be the empty sequence.
       */
     def translateNonMainContract(c: ObsidianContractImpl, checkedTable: SymbolTable): YulObject = {
         var translation: Seq[YulStatement] = Seq()
 
         for (d <- c.declarations) {
-            val dTranslated = translateDeclaration(d, c.name, checkedTable, false)
+            val dTranslated = translateDeclaration(d, c.name, checkedTable, inMain = false)
             translation = translation ++ dTranslated
         }
 
@@ -193,11 +191,6 @@ object CodeGenYul extends CodeGenerator {
             case _: Constructor =>
                 assert(assertion = false, "constructors not supported in Yul translation")
                 Seq()
-            // todo: previously this returned a pair of sequences, and this was the only clause
-            //   in which the left element was not empty. the left sequence would go in the code
-            //   part of the output object rather than the runtime, but that's not where we
-            //   want constructors to go.
-            // (translateConstructor(c, contractName, checkedTable), Seq())
             case _: TypeDecl =>
                 assert(assertion = false, "TODO")
                 Seq()
@@ -236,9 +229,9 @@ object CodeGenYul extends CodeGenerator {
         Seq(ExpressionStatement(deployExpr),
             FunctionDefinition(
                 new_name, // TODO rename transaction name (by adding prefix/suffix) iev: this seems to be done already
-                constructor.args.map(v => TypedName(v.varName, obsTypeToYulTypeAndSize(v.typIn.toString)._1)),
+                constructor.args.map(v => TypedName(v.varName, v.typIn)),
                 Seq(), //todo/iev: why is this always empty?
-                Block(constructor.body.flatMap((s: Statement) => translateStatement(s, None, contractName, checkedTable, true))))) //todo iev flatmap may be a bug to hide something wrong; None means that constructors don't return. is that true?
+                Block(constructor.body.flatMap((s: Statement) => translateStatement(s, None, contractName, checkedTable, inMain = true))))) //todo iev flatmap may be a bug to hide something wrong; None means that constructors don't return. is that true?
     }
 
     def translateTransaction(transaction: Transaction, contractName: String, checkedTable: SymbolTable, inMain: Boolean): Seq[YulStatement] = {
@@ -257,7 +250,7 @@ object CodeGenYul extends CodeGenerator {
             transaction.retType match {
                 case Some(t) =>
                     id = Some(nextRet())
-                    Seq(TypedName(id.get.name, obsTypeToYulTypeAndSize(t.toString)._1))
+                    Seq(TypedName(id.get.name, t))
                 case None => Seq()
             }
         }
@@ -267,8 +260,8 @@ object CodeGenYul extends CodeGenerator {
             if (inMain) {
                 Seq() // add nothing
             } else {
-                Seq(TypedName("this", "string")) // todo "this" is emphatically not a string but i'm not sure what the type of it ought to be; addr?
-            } ++ transaction.args.map(v => TypedName(v.varName, obsTypeToYulTypeAndSize(v.typIn.toString)._1))
+                Seq(TypedName("this", StringType())) // todo "this" is emphatically not a string but i'm not sure what the type of it ought to be; addr?
+            } ++ transaction.args.map(v => TypedName(v.varName, v.typIn))
 
         // form the body of the transaction by translating each statement found
         val body: Seq[YulStatement] = transaction.body.flatMap((s: Statement) => translateStatement(s, id, contractName, checkedTable, inMain))
@@ -307,11 +300,11 @@ object CodeGenYul extends CodeGenerator {
                         //  it also likely does not work correctly with shadowing.
                         val id = nextTemp()
                         val e_yul = translateExpr(id, e, contractName, checkedTable, inMain)
+                        val ct = checkedTable.contractLookup(contractName)
                         decl_0exp(id) +:
                             e_yul :+
-                            (if (checkedTable.contractLookup(contractName).allFields.exists(f => f.name.equals(x))) {
-                                //todo: compute offsets
-                                ExpressionStatement(apply("sstore", hexlit(keccak256(contractName + x)), id))
+                            (if (ct.allFields.exists(f => f.name.equals(x))) {
+                                ExpressionStatement(apply("mstore", Util.fieldFromThis(ct, x), id))
                             } else {
                                 assign1(Identifier(x), id)
                             })
@@ -425,19 +418,6 @@ object CodeGenYul extends CodeGenerator {
             assign1(retvar, apply("or", apply(s, e1id, e2id), apply("eq", e1id, e2id)))
     }
 
-
-    /**
-      * Given the type of a contract and a table, compute the size that we need to allocate for it in memory.
-      * TODO: as a simplifying assumption, for now this always returns 256.
-      *
-      * @param t      the contract type of interest
-      * @param symTab the symbol table to look in
-      * @return the size of memory needed for the contract
-      */
-    def contractSize(t: ContractType, symTab: SymbolTable): Int = {
-        256
-    }
-
     def translateExpr(retvar: Identifier, e: Expression, contractName: String, checkedTable: SymbolTable, inMain: Boolean): Seq[YulStatement] = {
         e match {
             case e: AtomicExpression =>
@@ -449,10 +429,10 @@ object CodeGenYul extends CodeGenerator {
 
                         // todo: this also assumes that everything is a u256 and does no type-directed
                         //  cleaning in the way that solc does
-                        if (checkedTable.contractLookup(contractName).allFields.exists(f => f.name.equals(x))) {
+                        val ct = checkedTable.contractLookup(contractName)
+                        if (ct.allFields.exists(f => f.name.equals(x))) {
                             val store_id = nextTemp()
-                            //todo: compute offsets
-                            Seq(decl_1exp(store_id, apply("sload", hexlit(keccak256(contractName + x)))),
+                            Seq(decl_1exp(store_id, apply("mload", Util.fieldFromThis(ct, x))),
                                 assign1(retvar, store_id))
                         } else {
                             Seq(assign1(retvar, Identifier(x)))
@@ -503,13 +483,13 @@ object CodeGenYul extends CodeGenerator {
                     case LessThanOrEquals(e1, e2) => geq_leq("slt", retvar, e1, e2, contractName, checkedTable, inMain)
                     case NotEquals(e1, e2) => translateExpr(retvar, LogicalNegation(Equals(e1, e2)), contractName, checkedTable, inMain)
                 }
-            case e@LocalInvocation(name, genericParams, params, args, obstype) => // todo: why are the middle two args not used?
+            case e@LocalInvocation(name, genericParams, params, args, obstype) =>
                 // look up the name of the function in the table, get its return type, and then compute
                 // how wide of a tuple that return type is. right now that's either 1 (if the
                 // transaction returns) or 0 (because it's void)
                 val width = obstype match {
                     case Some(t) => obsTypeToWidth(t)
-                    case None => assert(false, s"width failed on transaction named ${name} from expression ${e.toString}")
+                    case None => assert(assertion = false, s"width failed on transaction named $name from expression ${e.toString}")
                 }
 
                 // todo: some of this logic may be repeated in the dispatch table
@@ -535,7 +515,7 @@ object CodeGenYul extends CodeGenerator {
                 // todo: this does not work with non-void functions that are called without binding
                 //  their results, ie "f()" if f returns an int
                 ids.map(id => decl_0exp(id)) ++
-                seqs.flatten ++ (width match {
+                    seqs.flatten ++ (width match {
                     case 0 => Seq(ExpressionStatement(FunctionCall(Identifier(name), ids)))
                     case 1 =>
                         val id: Identifier = nextTemp()
@@ -548,11 +528,6 @@ object CodeGenYul extends CodeGenerator {
                 // returns a variable containing a memory address for the implicit `this` argument
                 // added to the translation of the transactions into the flat Yul object
 
-
-                // todo: this ultimately needs to be type-directed. to translate an invocation,
-                //  we need to know the type of the recipient expression being invoked so that we
-                //  can call the appropriate translated transaction in the big Yul object.
-
                 // we get a variable storing the address of the instance from recursively translating
                 // the recipient. we also form a Parser identifier with this, so that we can translate
                 // the invocation with a tailcall to translateExpr
@@ -561,22 +536,28 @@ object CodeGenYul extends CodeGenerator {
 
                 val recipient_yul = translateExpr(id_recipient, recipient, contractName, checkedTable, inMain)
 
-                ((decl_0exp(id_recipient) +: recipient_yul) ++
+                (decl_0exp(id_recipient) +: recipient_yul) ++
                     translateExpr(retvar, LocalInvocation(transactionNameMapping(getContractName(recipient), name),
                         genericParams,
                         params,
-                        this_address +: args, obstype), contractName, checkedTable, inMain)) // todo test this, too
+                        this_address +: args, obstype), contractName, checkedTable, inMain) // todo test this, too
 
             case Construction(contractType, args, isFFIInvocation, obstype) =>
                 // todo: currently we ignore the arguments to the constructor
                 assert(args.isEmpty, "contracts that take arguments are not yet supported")
 
-                val ct: Option[ContractTable] = checkedTable.contract(contractType.contractName)
+                // compute the amount of space we need to store something of this type, or assert
+                //   if that amount can't be computed.
+                val size: Int = checkedTable.contract(contractType.contractName) match {
+                    case Some(value) => sizeOfContract(value)
+                    case None => assert(assertion = false, s"contract table didn't contain contract name: ${contractType.contractName}"); -1
+                }
 
                 val id_memaddr = nextTemp()
+
                 Seq(
                     // grab the appropriate amount of space of memory sequentially, off the free memory pointer
-                    decl_1exp(id_memaddr, apply("allocate_memory", intlit(sizeOfContract(ct.get)))),
+                    decl_1exp(id_memaddr, apply("allocate_memory", intlit(size))),
 
                     // return the address that the space starts at
                     assign1(retvar, id_memaddr)