From 9a8882c9fc2e296d32be31ff9754cb554133ffc6 Mon Sep 17 00:00:00 2001 From: Chris Ward Date: Wed, 16 Jan 2019 15:43:47 +0200 Subject: [PATCH 001/109] Move string manipulation FAQ items to type docs Update docs/types/value-types.rst Co-Authored-By: ChrisChinchilla Update docs/types/value-types.rst Co-Authored-By: ChrisChinchilla Fixed formatting Re-add example Clarify text Rearrange string manipulation --- docs/control-structures.rst | 8 +++--- docs/frequently-asked-questions.rst | 38 ----------------------------- docs/types/reference-types.rst | 14 ++++++++--- docs/types/value-types.rst | 2 +- 4 files changed, 16 insertions(+), 46 deletions(-) diff --git a/docs/control-structures.rst b/docs/control-structures.rst index 46b3a7f1d941..45ba190ec43f 100644 --- a/docs/control-structures.rst +++ b/docs/control-structures.rst @@ -264,13 +264,13 @@ Complications for Arrays and Structs The semantics of assignments are a bit more complicated for non-value types like arrays and structs. Assigning *to* a state variable always creates an independent copy. On the other hand, assigning to a local variable creates an independent copy only for elementary types, i.e. static types that fit into 32 bytes. If structs or arrays (including ``bytes`` and ``string``) are assigned from a state variable to a local variable, the local variable holds a reference to the original state variable. A second assignment to the local variable does not modify the state but only changes the reference. Assignments to members (or elements) of the local variable *do* change the state. -In the example below the call to ``g(x)`` has no effect on ``x`` because it needs -to create an independent copy of the storage value in memory. However ``h(x)`` modifies ``x`` because a reference and -not a copy is passed. +In the example below the call to ``g(x)`` has no effect on ``x`` because it creates +an independent copy of the storage value in memory. However, ``h(x)`` successfully modifies ``x`` +because only a reference and not a copy is passed. :: - pragma solidity >=0.4.16 <0.6.0; + pragma solidity >=0.4.16 <0.6.0; contract C { uint[20] x; diff --git a/docs/frequently-asked-questions.rst b/docs/frequently-asked-questions.rst index 645789ceed9c..f009d3af5c8c 100644 --- a/docs/frequently-asked-questions.rst +++ b/docs/frequently-asked-questions.rst @@ -16,44 +16,6 @@ Enums are not supported by the ABI, they are just supported by Solidity. You have to do the mapping yourself for now, we might provide some help later. -What are some examples of basic string manipulation (``substring``, ``indexOf``, ``charAt``, etc)? -================================================================================================== - -There are some string utility functions at `stringUtils.sol `_ -which will be extended in the future. In addition, Arachnid has written `solidity-stringutils `_. - -For now, if you want to modify a string (even when you only want to know its length), -you should always convert it to a ``bytes`` first:: - - pragma solidity >=0.4.0 <0.6.0; - - contract C { - string s; - - function append(byte c) public { - bytes(s).push(c); - } - - function set(uint i, byte c) public { - bytes(s)[i] = c; - } - } - - -Can I concatenate two strings? -============================== - -Yes, you can use ``abi.encodePacked``:: - - pragma solidity >=0.4.0 <0.6.0; - - library ConcatHelper { - function concat(bytes memory a, bytes memory b) - internal pure returns (bytes memory) { - return abi.encodePacked(a, b); - } - } - ****************** Advanced Questions ****************** diff --git a/docs/types/reference-types.rst b/docs/types/reference-types.rst index b9fd9194a2fa..99d977f6c155 100644 --- a/docs/types/reference-types.rst +++ b/docs/types/reference-types.rst @@ -106,13 +106,24 @@ Array elements can be of any type, including mapping or struct. The general restrictions for types apply, in that mappings can only be stored in the ``storage`` data location and publicly-visible functions need parameters that are :ref:`ABI types `. +It is possible to mark arrays ``public`` and have Solidity create a :ref:`getter `. +The numeric index becomes a required parameter for the getter. + Accessing an array past its end causes a failing assertion. You can use the ``.push()`` method to append a new element at the end or assign to the ``.length`` :ref:`member ` to change the size (see below for caveats). method or increase the ``.length`` :ref:`member ` to add elements. +``bytes`` and ``strings`` as Arrays +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + Variables of type ``bytes`` and ``string`` are special arrays. A ``bytes`` is similar to ``byte[]``, but it is packed tightly in calldata and memory. ``string`` is equal to ``bytes`` but does not allow length or index access. +While Solidity does not have string manipulation functions, you can use +this implicit conversion for equivalent functionality. For example to compare +two strings ``keccak256(abi.encode(s1)) == keccak256(abi.encode(s2))``, or to +concatenate two strings already encoded with ``abi.encodePacked(s1, s2);``. + You should use ``bytes`` over ``byte[]`` because it is cheaper, since ``byte[]`` adds 31 padding bytes between the elements. As a general rule, use ``bytes`` for arbitrary-length raw byte data and ``string`` for arbitrary-length string (UTF-8) data. If you can limit the length to a certain number of bytes, @@ -124,9 +135,6 @@ always use one of the value types ``bytes1`` to ``bytes32`` because they are muc that you are accessing the low-level bytes of the UTF-8 representation, and not the individual characters. -It is possible to mark arrays ``public`` and have Solidity create a :ref:`getter `. -The numeric index becomes a required parameter for the getter. - .. index:: ! array;allocating, new Allocating Memory Arrays diff --git a/docs/types/value-types.rst b/docs/types/value-types.rst index 09db1423c3ab..a8846eaf0bc8 100644 --- a/docs/types/value-types.rst +++ b/docs/types/value-types.rst @@ -708,4 +708,4 @@ Another example that uses external function types:: } .. note:: - Lambda or inline functions are planned but not yet supported. \ No newline at end of file + Lambda or inline functions are planned but not yet supported. From 01e87cfc0834a3e133f0cd3084ac76f5b2728773 Mon Sep 17 00:00:00 2001 From: chriseth Date: Tue, 22 Jan 2019 13:46:33 +0100 Subject: [PATCH 002/109] Refactor struct encoder. --- libsolidity/codegen/ABIFunctions.cpp | 50 +++++++++++++--------------- 1 file changed, 24 insertions(+), 26 deletions(-) diff --git a/libsolidity/codegen/ABIFunctions.cpp b/libsolidity/codegen/ABIFunctions.cpp index c0fa81ce18be..9f4e0679dbb7 100644 --- a/libsolidity/codegen/ABIFunctions.cpp +++ b/libsolidity/codegen/ABIFunctions.cpp @@ -904,6 +904,8 @@ string ABIFunctions::abiEncodingFunctionStruct( <#members> { // + + let memberValue := } @@ -932,20 +934,10 @@ string ABIFunctions::abiEncodingFunctionStruct( bool dynamicMember = memberTypeTo->isDynamicallyEncoded(); if (dynamicMember) solAssert(dynamic, ""); - Whiskers memberTempl(R"( - - let memberValue := - )" + ( - dynamicMember ? - string(R"( - mstore(add(pos, ), sub(tail, pos)) - tail := (memberValue, tail) - )") : - string(R"( - (memberValue, add(pos, )) - )") - ) - ); + + members.push_back({}); + members.back()["preprocess"] = ""; + if (fromStorage) { solAssert(memberTypeFrom->isValueType() == memberTypeTo->isValueType(), ""); @@ -956,36 +948,42 @@ string ABIFunctions::abiEncodingFunctionStruct( { if (storageSlotOffset != previousSlotOffset) { - memberTempl("preprocess", "slotValue := sload(add(value, " + toCompactHexWithPrefix(storageSlotOffset) + "))"); + members.back()["preprocess"] = "slotValue := sload(add(value, " + toCompactHexWithPrefix(storageSlotOffset) + "))"; previousSlotOffset = storageSlotOffset; } - else - memberTempl("preprocess", ""); - memberTempl("retrieveValue", shiftRightFunction(intraSlotOffset * 8) + "(slotValue)"); + members.back()["retrieveValue"] = shiftRightFunction(intraSlotOffset * 8) + "(slotValue)"; } else { solAssert(memberTypeFrom->dataStoredIn(DataLocation::Storage), ""); solAssert(intraSlotOffset == 0, ""); - memberTempl("preprocess", ""); - memberTempl("retrieveValue", "add(value, " + toCompactHexWithPrefix(storageSlotOffset) + ")"); + members.back()["retrieveValue"] = "add(value, " + toCompactHexWithPrefix(storageSlotOffset) + ")"; } } else { - memberTempl("preprocess", ""); string sourceOffset = toCompactHexWithPrefix(_from.memoryOffsetOfMember(member.name)); - memberTempl("retrieveValue", "mload(add(value, " + sourceOffset + "))"); + members.back()["retrieveValue"] = "mload(add(value, " + sourceOffset + "))"; } - memberTempl("encodingOffset", toCompactHexWithPrefix(encodingOffset)); + + Whiskers encodeTempl( + dynamicMember ? + string(R"( + mstore(add(pos, ), sub(tail, pos)) + tail := (memberValue, tail) + )") : + string(R"( + (memberValue, add(pos, )) + )") + ); + encodeTempl("encodingOffset", toCompactHexWithPrefix(encodingOffset)); encodingOffset += dynamicMember ? 0x20 : memberTypeTo->calldataEncodedSize(); EncodingOptions subOptions(_options); subOptions.encodeFunctionFromStack = false; - memberTempl("abiEncode", abiEncodingFunction(*memberTypeFrom, *memberTypeTo, subOptions)); + encodeTempl("abiEncode", abiEncodingFunction(*memberTypeFrom, *memberTypeTo, subOptions)); - members.push_back({}); - members.back()["encode"] = memberTempl.render(); + members.back()["encode"] = encodeTempl.render(); members.back()["memberName"] = member.name; } templ("members", members); From 3f2898ea47b8619d7560b1e0a97e98edcc10c3af Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 17 Jan 2019 23:41:13 +0000 Subject: [PATCH 003/109] Do not ignore revertOnFailure flag when validating Address/Contract --- libsolidity/codegen/ABIFunctions.cpp | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/libsolidity/codegen/ABIFunctions.cpp b/libsolidity/codegen/ABIFunctions.cpp index c0fa81ce18be..155f73c4a09e 100644 --- a/libsolidity/codegen/ABIFunctions.cpp +++ b/libsolidity/codegen/ABIFunctions.cpp @@ -216,7 +216,7 @@ string ABIFunctions::cleanupFunction(Type const& _type, bool _revertOnFailure) switch (_type.category()) { case Type::Category::Address: - templ("body", "cleaned := " + cleanupFunction(IntegerType(160)) + "(value)"); + templ("body", "cleaned := " + cleanupFunction(IntegerType(160), _revertOnFailure) + "(value)"); break; case Type::Category::Integer: { @@ -265,7 +265,7 @@ string ABIFunctions::cleanupFunction(Type const& _type, bool _revertOnFailure) StateMutability::Payable : StateMutability::NonPayable ); - templ("body", "cleaned := " + cleanupFunction(addressType) + "(value)"); + templ("body", "cleaned := " + cleanupFunction(addressType, _revertOnFailure) + "(value)"); break; } case Type::Category::Enum: @@ -341,7 +341,7 @@ string ABIFunctions::conversionFunction(Type const& _from, Type const& _to) solAssert(_from.mobileType(), ""); body = Whiskers("converted := ((value))") - ("cleanEnum", cleanupFunction(_to, false)) + ("cleanEnum", cleanupFunction(_to)) // "mobileType()" returns integer type for rational ("cleanInt", cleanupFunction(*_from.mobileType())) .render(); From 70896deb67e040a73872e1bdae8ef82e84b3c9b6 Mon Sep 17 00:00:00 2001 From: chriseth Date: Tue, 22 Jan 2019 16:46:05 +0100 Subject: [PATCH 004/109] Update version and changelog. --- CMakeLists.txt | 2 +- Changelog.md | 15 +++++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 1db1d0529023..05af3504578a 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -10,7 +10,7 @@ include(EthPolicy) eth_policy() # project name and version should be set after cmake_policy CMP0048 -set(PROJECT_VERSION "0.5.3") +set(PROJECT_VERSION "0.5.4") project(solidity VERSION ${PROJECT_VERSION} LANGUAGES CXX) option(LLL "Build LLL" OFF) diff --git a/Changelog.md b/Changelog.md index 1238d4b5700e..ceb1402a843b 100644 --- a/Changelog.md +++ b/Changelog.md @@ -1,3 +1,18 @@ +### 0.5.4 (unreleased) + + +Language Features: + + +Compiler Features: + + +Bugfixes: + + +Build System: + + ### 0.5.3 (2019-01-22) Language Features: From a52de117ea58f97b799321fb7ea7be7cc1294873 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 27 Sep 2018 02:00:36 +0100 Subject: [PATCH 005/109] Do not use fork of openzeppelin anymore --- test/externalTests.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/externalTests.sh b/test/externalTests.sh index 126b13b5dfe1..0f01ed26890d 100755 --- a/test/externalTests.sh +++ b/test/externalTests.sh @@ -85,8 +85,8 @@ function test_truffle rm -rf "$DIR" } -# Using our temporary fork here. Hopefully to be merged into upstream after the 0.5.0 release. -test_truffle Zeppelin https://github.com/axic/openzeppelin-solidity.git solidity-050 +# Since Zeppelin 2.1.1 it supports Solidity 0.5.0. +test_truffle Zeppelin https://github.com/OpenZeppelin/openzeppelin-solidity.git master # Disabled temporarily as it needs to be updated to latest Truffle first. #test_truffle Gnosis https://github.com/axic/pm-contracts.git solidity-050 From 028bc7d20aece93decad4989bc42a14e3671aeb2 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 22 Jan 2019 16:45:49 +0000 Subject: [PATCH 006/109] Only replace solc-js copies in directories present during externalTests --- test/externalTests.sh | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/test/externalTests.sh b/test/externalTests.sh index 0f01ed26890d..f87777e3e520 100755 --- a/test/externalTests.sh +++ b/test/externalTests.sh @@ -59,10 +59,13 @@ function test_truffle for d in node_modules node_modules/truffle/node_modules do ( - cd $d - rm -rf solc - git clone --depth 1 -b v0.5.0 https://github.com/ethereum/solc-js.git solc - cp "$SOLJSON" solc/ + if [ -d "$d" ] + then + cd $d + rm -rf solc + git clone --depth 1 -b v0.5.0 https://github.com/ethereum/solc-js.git solc + cp "$SOLJSON" solc/ + fi ) done if [ "$name" == "Zeppelin" -o "$name" == "Gnosis" ]; then From 1684c70f7d3902d0299944e845abf30edd49e225 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 22 Jan 2019 17:18:27 +0000 Subject: [PATCH 007/109] Document the libsolc API --- libsolc/libsolc.h | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/libsolc/libsolc.h b/libsolc/libsolc.h index b58ee80533d2..baa39fd441c1 100644 --- a/libsolc/libsolc.h +++ b/libsolc/libsolc.h @@ -34,12 +34,25 @@ extern "C" { #endif -/// Callback used to retrieve additional source files. "Returns" two pointers that should be -/// heap-allocated and are free'd by the caller. +/// Callback used to retrieve additional source files. +/// +/// "Returns" two pointers that should be heap-allocated and are free'd by the caller. typedef void (*CStyleReadFileCallback)(char const* _path, char** o_contents, char** o_error); +/// Returns the complete license document. +/// +/// The pointer returned must not be freed by the caller. char const* solidity_license() SOLC_NOEXCEPT; + +/// Returns the compiler version. +/// +/// The pointer returned must not be freed by the caller. char const* solidity_version() SOLC_NOEXCEPT; + +/// Takes a "Standard Input JSON" and an optional callback (can be set to null). Returns +/// a "Standard Output JSON". Both are to be UTF-8 encoded. +/// +/// The pointer returned must not be freed by the caller. char const* solidity_compile(char const* _input, CStyleReadFileCallback _readCallback) SOLC_NOEXCEPT; #ifdef __cplusplus From 0a3beb72f27f6fcf75ab3fe67f2cb0b1c00924b7 Mon Sep 17 00:00:00 2001 From: chriseth Date: Tue, 22 Jan 2019 18:22:54 +0100 Subject: [PATCH 008/109] Some improvements to the ppa release script. --- scripts/release_ppa.sh | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/scripts/release_ppa.sh b/scripts/release_ppa.sh index 6cba7dffaf48..353b5efa94b4 100755 --- a/scripts/release_ppa.sh +++ b/scripts/release_ppa.sh @@ -53,7 +53,14 @@ packagename=solc static_build_distribution=cosmic -for distribution in bionic cosmic STATIC +DISTRIBUTIONS="bionic cosmic" + +if [ branch != develop ] +then + DISTRIBUTIONS="$DISTRIBUTIONS STATIC" +fi + +for distribution in $DISTRIBUTIONS do cd /tmp/ rm -rf $distribution @@ -96,7 +103,7 @@ commitdate=$(git show --format=%ci HEAD | head -n 1 | cut - -b1-10 | sed -e 's/- echo "$commithash" > commit_hash.txt if [ $branch = develop ] then - debversion="$version-develop-$commitdate-$commithash" + debversion="$version~develop-$commitdate-$commithash" else debversion="$version" echo -n > prerelease.txt # proper release From aaf620621fbd10f2eb5e362f5bc17cb2534d9085 Mon Sep 17 00:00:00 2001 From: chriseth Date: Tue, 22 Jan 2019 18:36:26 +0100 Subject: [PATCH 009/109] [DOCS] Fix mention of commandline parameter. --- docs/using-the-compiler.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 4749ef1f19e5..80c06d500b79 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -19,8 +19,8 @@ If you only want to compile a single file, you run it as ``solc --bin sourceFile Before you deploy your contract, activate the optimizer when compiling using ``solc --optimize --bin sourceFile.sol``. By default, the optimizer will optimize the contract assuming it is called 200 times across its lifetime. If you want the initial contract deployment to be cheaper and the later function executions to be more expensive, -set it to ``--runs=1``. If you expect many transactions and do not care for higher deployment cost and -output size, set ``--runs`` to a high number. +set it to ``--optimize-runs=1``. If you expect many transactions and do not care for higher deployment cost and +output size, set ``--optimize-runs`` to a high number. The commandline compiler will automatically read imported files from the filesystem, but it is also possible to provide path redirects using ``prefix=path`` in the following way: From 8c4c581eedb300d0bbd5b2f70ad61583c4fb8a9d Mon Sep 17 00:00:00 2001 From: chriseth Date: Tue, 22 Jan 2019 22:26:38 +0100 Subject: [PATCH 010/109] Add disco. --- scripts/release_ppa.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/release_ppa.sh b/scripts/release_ppa.sh index 353b5efa94b4..fcdc8d640f7a 100755 --- a/scripts/release_ppa.sh +++ b/scripts/release_ppa.sh @@ -53,7 +53,7 @@ packagename=solc static_build_distribution=cosmic -DISTRIBUTIONS="bionic cosmic" +DISTRIBUTIONS="bionic cosmic disco" if [ branch != develop ] then From 2f91781a809eb85978a56bef6758a9a3cf2fa773 Mon Sep 17 00:00:00 2001 From: Mudit Gupta Date: Wed, 23 Jan 2019 10:22:27 +0530 Subject: [PATCH 011/109] useLiteralContent data validation Squashed commit of the following: commit bbceee6997c1b951eb6054f7d5b7560fe1773690 Author: Mudit Gupta Date: Tue Jan 22 23:27:49 2019 +0530 Removed extra check commit 4f7c3cc58e94836b1dddcc5f2938da14adad6252 Author: Mudit Gupta Date: Tue Jan 22 17:51:05 2019 +0530 Updated test error message commit 5da45f2a23840e7bb1978853e1c184faec65b3e0 Author: Mudit Gupta Date: Tue Jan 22 17:46:09 2019 +0530 Renamed test commit e661418deac25f31c49091be3c80244f566fc14c Author: Leonardo Date: Tue Jan 22 17:41:56 2019 +0530 Update libsolidity/interface/StandardCompiler.cpp Co-Authored-By: maxsam4 commit d289b4dc1956ab736aada613e810cf2e2d124d4d Author: Mudit Gupta Date: Tue Jan 22 17:29:40 2019 +0530 changed style commit 5a7cf08db3027d75081d2e698dd51b807bef5abb Author: Mudit Gupta Date: Tue Jan 22 17:20:37 2019 +0530 Added test case for missing useLiteralContent commit 6e866c46aa92d6a89f2b341bd717c3886946f3ff Author: Mudit Gupta Date: Tue Jan 22 17:09:30 2019 +0530 input check for useLiteralContent --- libsolidity/interface/StandardCompiler.cpp | 2 ++ .../exit | 1 + .../input.json | 19 +++++++++++++++++ .../output.json | 1 + .../exit | 1 + .../input.json | 21 +++++++++++++++++++ .../output.json | 1 + 7 files changed, 46 insertions(+) create mode 100644 test/cmdlineTests/standard_missing_key_useLiteralContent/exit create mode 100644 test/cmdlineTests/standard_missing_key_useLiteralContent/input.json create mode 100644 test/cmdlineTests/standard_missing_key_useLiteralContent/output.json create mode 100644 test/cmdlineTests/standard_wrong_type_useLiteralContent/exit create mode 100644 test/cmdlineTests/standard_wrong_type_useLiteralContent/input.json create mode 100644 test/cmdlineTests/standard_wrong_type_useLiteralContent/output.json diff --git a/libsolidity/interface/StandardCompiler.cpp b/libsolidity/interface/StandardCompiler.cpp index 137a4439a060..606e858c61d5 100644 --- a/libsolidity/interface/StandardCompiler.cpp +++ b/libsolidity/interface/StandardCompiler.cpp @@ -298,6 +298,8 @@ boost::optional checkOptimizerKeys(Json::Value const& _input) boost::optional checkMetadataKeys(Json::Value const& _input) { + if (_input.isObject() && _input.isMember("useLiteralContent") && !_input["useLiteralContent"].isBool()) + return formatFatalError("JSONError", "\"settings.metadata.useLiteralContent\" must be Boolean"); static set keys{"useLiteralContent"}; return checkKeys(_input, keys, "settings.metadata"); } diff --git a/test/cmdlineTests/standard_missing_key_useLiteralContent/exit b/test/cmdlineTests/standard_missing_key_useLiteralContent/exit new file mode 100644 index 000000000000..573541ac9702 --- /dev/null +++ b/test/cmdlineTests/standard_missing_key_useLiteralContent/exit @@ -0,0 +1 @@ +0 diff --git a/test/cmdlineTests/standard_missing_key_useLiteralContent/input.json b/test/cmdlineTests/standard_missing_key_useLiteralContent/input.json new file mode 100644 index 000000000000..8627a282a806 --- /dev/null +++ b/test/cmdlineTests/standard_missing_key_useLiteralContent/input.json @@ -0,0 +1,19 @@ +{ + "language": "Solidity", + "sources": + { + "A": + { + "content": "pragma solidity >=0.0; contract C { function f() public pure {} }" + } + }, + "settings": + { + "optimizer": { + "enabled": true, + "runs": 200 + }, + "evmVersion": "byzantium", + "metadata": {} + } +} diff --git a/test/cmdlineTests/standard_missing_key_useLiteralContent/output.json b/test/cmdlineTests/standard_missing_key_useLiteralContent/output.json new file mode 100644 index 000000000000..14c325725838 --- /dev/null +++ b/test/cmdlineTests/standard_missing_key_useLiteralContent/output.json @@ -0,0 +1 @@ +{"sources":{"A":{"id":0}}} \ No newline at end of file diff --git a/test/cmdlineTests/standard_wrong_type_useLiteralContent/exit b/test/cmdlineTests/standard_wrong_type_useLiteralContent/exit new file mode 100644 index 000000000000..573541ac9702 --- /dev/null +++ b/test/cmdlineTests/standard_wrong_type_useLiteralContent/exit @@ -0,0 +1 @@ +0 diff --git a/test/cmdlineTests/standard_wrong_type_useLiteralContent/input.json b/test/cmdlineTests/standard_wrong_type_useLiteralContent/input.json new file mode 100644 index 000000000000..be4272b6caf7 --- /dev/null +++ b/test/cmdlineTests/standard_wrong_type_useLiteralContent/input.json @@ -0,0 +1,21 @@ +{ + "language": "Solidity", + "sources": + { + "A": + { + "content": "pragma solidity >=0.0; contract C { function f() public pure {} }" + } + }, + "settings": + { + "optimizer": { + "enabled": true, + "runs": 200 + }, + "evmVersion": "byzantium", + "metadata": { + "useLiteralContent": "literalContent" + } + } +} diff --git a/test/cmdlineTests/standard_wrong_type_useLiteralContent/output.json b/test/cmdlineTests/standard_wrong_type_useLiteralContent/output.json new file mode 100644 index 000000000000..47eb32ec4533 --- /dev/null +++ b/test/cmdlineTests/standard_wrong_type_useLiteralContent/output.json @@ -0,0 +1 @@ +{"errors":[{"component":"general","formattedMessage":"\"settings.metadata.useLiteralContent\" must be Boolean","message":"\"settings.metadata.useLiteralContent\" must be Boolean","severity":"error","type":"JSONError"}]} From 24b1de7df00f065fba4e7979d815d8406fc7b13e Mon Sep 17 00:00:00 2001 From: Bhargava Shastry Date: Thu, 17 Jan 2019 11:19:54 +0100 Subject: [PATCH 012/109] This PR refactors and shares oss-fuzz specific test harness code with the afl fuzzer harness. ChangeLog updated. --- Changelog.md | 1 + cmake/EthOptions.cmake | 1 + libdevcore/CommonData.h | 7 + test/tools/CMakeLists.txt | 6 +- test/tools/afl_fuzzer.cpp | 101 ++++++++++ test/tools/fuzzer.cpp | 217 ---------------------- test/tools/fuzzer_common.cpp | 114 ++++++++++++ test/tools/fuzzer_common.h | 35 ++++ test/tools/ossfuzz/CMakeLists.txt | 12 ++ test/tools/ossfuzz/README.md | 20 ++ test/tools/ossfuzz/const_opt_ossfuzz.cpp | 27 +++ test/tools/ossfuzz/solc_noopt_ossfuzz.cpp | 27 +++ test/tools/ossfuzz/solc_opt_ossfuzz.cpp | 27 +++ 13 files changed, 377 insertions(+), 218 deletions(-) create mode 100644 test/tools/afl_fuzzer.cpp delete mode 100644 test/tools/fuzzer.cpp create mode 100644 test/tools/fuzzer_common.cpp create mode 100644 test/tools/fuzzer_common.h create mode 100644 test/tools/ossfuzz/CMakeLists.txt create mode 100644 test/tools/ossfuzz/README.md create mode 100644 test/tools/ossfuzz/const_opt_ossfuzz.cpp create mode 100644 test/tools/ossfuzz/solc_noopt_ossfuzz.cpp create mode 100644 test/tools/ossfuzz/solc_opt_ossfuzz.cpp diff --git a/Changelog.md b/Changelog.md index ceb1402a843b..42e417d4f1f7 100644 --- a/Changelog.md +++ b/Changelog.md @@ -11,6 +11,7 @@ Bugfixes: Build System: + * Add support for continuous fuzzing via Google oss-fuzz ### 0.5.3 (2019-01-22) diff --git a/cmake/EthOptions.cmake b/cmake/EthOptions.cmake index a79e5135a4a6..68d6cb04505f 100644 --- a/cmake/EthOptions.cmake +++ b/cmake/EthOptions.cmake @@ -3,6 +3,7 @@ macro(configure_project) # features eth_default_option(COVERAGE OFF) + eth_default_option(OSSFUZZ OFF) # components eth_default_option(TESTS ON) diff --git a/libdevcore/CommonData.h b/libdevcore/CommonData.h index 98331936c689..adf9d70ce0d9 100644 --- a/libdevcore/CommonData.h +++ b/libdevcore/CommonData.h @@ -334,4 +334,11 @@ bool containerEqual(Container const& _lhs, Container const& _rhs, Compare&& _com return std::equal(std::begin(_lhs), std::end(_lhs), std::begin(_rhs), std::end(_rhs), std::forward(_compare)); } +inline std::string findAnyOf(std::string const& _haystack, std::vector const& _needles) +{ + for (std::string const& needle: _needles) + if (_haystack.find(needle) != std::string::npos) + return needle; + return ""; +} } diff --git a/test/tools/CMakeLists.txt b/test/tools/CMakeLists.txt index da8e0b39c5ac..7e070ebb551b 100644 --- a/test/tools/CMakeLists.txt +++ b/test/tools/CMakeLists.txt @@ -1,4 +1,8 @@ -add_executable(solfuzzer fuzzer.cpp) +if (OSSFUZZ) + add_subdirectory(ossfuzz) +endif() + +add_executable(solfuzzer afl_fuzzer.cpp fuzzer_common.cpp) target_link_libraries(solfuzzer PRIVATE libsolc evmasm ${Boost_PROGRAM_OPTIONS_LIBRARIES} ${Boost_SYSTEM_LIBRARIES}) add_executable(yulopti yulopti.cpp) diff --git a/test/tools/afl_fuzzer.cpp b/test/tools/afl_fuzzer.cpp new file mode 100644 index 000000000000..9f4ac86cd39e --- /dev/null +++ b/test/tools/afl_fuzzer.cpp @@ -0,0 +1,101 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ +/** + * Executable for use with AFL . + */ + +#include +#include + +using namespace std; +using namespace dev; +using namespace dev::eth; + +namespace po = boost::program_options; + +int main(int argc, char** argv) +{ + po::options_description options( + R"(solfuzzer, fuzz-testing binary for use with AFL. +Usage: solfuzzer [Options] < input +Reads a single source from stdin, compiles it and signals a failure for internal errors. + +Allowed options)", + po::options_description::m_default_line_length, + po::options_description::m_default_line_length - 23); + options.add_options() + ("help", "Show this help screen.") + ("quiet", "Only output errors.") + ( + "standard-json", + "Test via the standard-json interface, i.e. " + "input is expected to be JSON-encoded instead of " + "plain source file." + ) + ( + "const-opt", + "Run the constant optimizer instead of compiling. " + "Expects a binary string of up to 32 bytes on stdin." + ) + ( + "input-file", + po::value(), + "input file" + ) + ( + "without-optimizer", + "Run without optimizations. Cannot be used together with standard-json." + ); + + // All positional options should be interpreted as input files + po::positional_options_description filesPositions; + filesPositions.add("input-file", 1); + bool quiet = false; + + po::variables_map arguments; + try + { + po::command_line_parser cmdLineParser(argc, argv); + cmdLineParser.options(options).positional(filesPositions); + po::store(cmdLineParser.run(), arguments); + } + catch (po::error const& _exception) + { + cerr << _exception.what() << endl; + return 1; + } + + string input; + if (arguments.count("input-file")) + input = readFileAsString(arguments["input-file"].as()); + else + input = readStandardInput(); + + if (arguments.count("quiet")) + quiet = true; + + if (arguments.count("help")) + cout << options; + else if (arguments.count("const-opt")) + FuzzerUtil::testConstantOptimizer(input, quiet); + else if (arguments.count("standard-json")) + FuzzerUtil::testStandardCompiler(input, quiet); + else + FuzzerUtil::testCompiler(input, !arguments.count("without-optimizer"), quiet); + + return 0; +} \ No newline at end of file diff --git a/test/tools/fuzzer.cpp b/test/tools/fuzzer.cpp deleted file mode 100644 index 8633454c5497..000000000000 --- a/test/tools/fuzzer.cpp +++ /dev/null @@ -1,217 +0,0 @@ -/* - This file is part of solidity. - - solidity is free software: you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation, either version 3 of the License, or - (at your option) any later version. - - solidity is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with solidity. If not, see . -*/ -/** - * Executable for use with AFL . - */ - -#include -#include -#include -#include - -#include - -#include - -#include -#include -#include - -using namespace std; -using namespace dev; -using namespace dev::eth; -namespace po = boost::program_options; - -namespace -{ - -bool quiet = false; - -string contains(string const& _haystack, vector const& _needles) -{ - for (string const& needle: _needles) - if (_haystack.find(needle) != string::npos) - return needle; - return ""; -} - -void testConstantOptimizer(string const& input) -{ - if (!quiet) - cout << "Testing constant optimizer" << endl; - vector numbers; - stringstream sin(input); - - while (!sin.eof()) - { - h256 data; - sin.read(reinterpret_cast(data.data()), 32); - numbers.push_back(u256(data)); - } - if (!quiet) - cout << "Got " << numbers.size() << " inputs:" << endl; - - Assembly assembly; - for (u256 const& n: numbers) - { - if (!quiet) - cout << n << endl; - assembly.append(n); - } - for (bool isCreation: {false, true}) - { - for (unsigned runs: {1, 2, 3, 20, 40, 100, 200, 400, 1000}) - { - ConstantOptimisationMethod::optimiseConstants( - isCreation, - runs, - EVMVersion{}, - assembly, - const_cast(assembly.items()) - ); - } - } -} - -void runCompiler(string input) -{ - string outputString(solidity_compile(input.c_str(), nullptr)); - Json::Value output; - if (!jsonParseStrict(outputString, output)) - { - cout << "Compiler produced invalid JSON output." << endl; - abort(); - } - if (output.isMember("errors")) - for (auto const& error: output["errors"]) - { - string invalid = contains(error["type"].asString(), vector{ - "Exception", - "InternalCompilerError" - }); - if (!invalid.empty()) - { - cout << "Invalid error: \"" << error["type"].asString() << "\"" << endl; - abort(); - } - } -} - -void testStandardCompiler(string const& input) -{ - if (!quiet) - cout << "Testing compiler via JSON interface." << endl; - - runCompiler(input); -} - -void testCompiler(string const& input, bool optimize) -{ - if (!quiet) - cout << "Testing compiler " << (optimize ? "with" : "without") << " optimizer." << endl; - - Json::Value config = Json::objectValue; - config["language"] = "Solidity"; - config["sources"] = Json::objectValue; - config["sources"][""] = Json::objectValue; - config["sources"][""]["content"] = input; - config["settings"] = Json::objectValue; - config["settings"]["optimizer"] = Json::objectValue; - config["settings"]["optimizer"]["enabled"] = optimize; - config["settings"]["optimizer"]["runs"] = 200; - - // Enable all SourceUnit-level outputs. - config["settings"]["outputSelection"]["*"][""][0] = "*"; - // Enable all Contract-level outputs. - config["settings"]["outputSelection"]["*"]["*"][0] = "*"; - - runCompiler(jsonCompactPrint(config)); -} - -} - -int main(int argc, char** argv) -{ - po::options_description options( - R"(solfuzzer, fuzz-testing binary for use with AFL. -Usage: solfuzzer [Options] < input -Reads a single source from stdin, compiles it and signals a failure for internal errors. - -Allowed options)", - po::options_description::m_default_line_length, - po::options_description::m_default_line_length - 23); - options.add_options() - ("help", "Show this help screen.") - ("quiet", "Only output errors.") - ( - "standard-json", - "Test via the standard-json interface, i.e. " - "input is expected to be JSON-encoded instead of " - "plain source file." - ) - ( - "const-opt", - "Run the constant optimizer instead of compiling. " - "Expects a binary string of up to 32 bytes on stdin." - ) - ( - "input-file", - po::value(), - "input file" - ) - ( - "without-optimizer", - "Run without optimizations. Cannot be used together with standard-json." - ); - - // All positional options should be interpreted as input files - po::positional_options_description filesPositions; - filesPositions.add("input-file", 1); - - po::variables_map arguments; - try - { - po::command_line_parser cmdLineParser(argc, argv); - cmdLineParser.options(options).positional(filesPositions); - po::store(cmdLineParser.run(), arguments); - } - catch (po::error const& _exception) - { - cerr << _exception.what() << endl; - return 1; - } - - string input; - if (arguments.count("input-file")) - input = readFileAsString(arguments["input-file"].as()); - else - input = readStandardInput(); - - if (arguments.count("quiet")) - quiet = true; - - if (arguments.count("help")) - cout << options; - else if (arguments.count("const-opt")) - testConstantOptimizer(input); - else if (arguments.count("standard-json")) - testStandardCompiler(input); - else - testCompiler(input, !arguments.count("without-optimizer")); - - return 0; -} diff --git a/test/tools/fuzzer_common.cpp b/test/tools/fuzzer_common.cpp new file mode 100644 index 000000000000..a700c557e4e2 --- /dev/null +++ b/test/tools/fuzzer_common.cpp @@ -0,0 +1,114 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ + +#include +#include + +using namespace std; +using namespace dev; +using namespace dev::eth; + +void FuzzerUtil::runCompiler(string _input) +{ + string outputString(solidity_compile(_input.c_str(), nullptr)); + Json::Value output; + if (!jsonParseStrict(outputString, output)) + { + cout << "Compiler produced invalid JSON output." << endl; + abort(); + } + if (output.isMember("errors")) + for (auto const& error: output["errors"]) + { + string invalid = findAnyOf(error["type"].asString(), vector{ + "Exception", + "InternalCompilerError" + }); + if (!invalid.empty()) + { + cout << "Invalid error: \"" << error["type"].asString() << "\"" << endl; + abort(); + } + } +} + +void FuzzerUtil::testCompiler(string const& _input, bool _optimize, bool _quiet) +{ + if (!_quiet) + cout << "Testing compiler " << (_optimize ? "with" : "without") << " optimizer." << endl; + + Json::Value config = Json::objectValue; + config["language"] = "Solidity"; + config["sources"] = Json::objectValue; + config["sources"][""] = Json::objectValue; + config["sources"][""]["content"] = _input; + config["settings"] = Json::objectValue; + config["settings"]["optimizer"] = Json::objectValue; + config["settings"]["optimizer"]["enabled"] = _optimize; + config["settings"]["optimizer"]["runs"] = 200; + + // Enable all SourceUnit-level outputs. + config["settings"]["outputSelection"]["*"][""][0] = "*"; + // Enable all Contract-level outputs. + config["settings"]["outputSelection"]["*"]["*"][0] = "*"; + + runCompiler(jsonCompactPrint(config)); +} + +void FuzzerUtil::testConstantOptimizer(string const& _input, bool _quiet) +{ + if (!_quiet) + cout << "Testing constant optimizer" << endl; + vector numbers; + stringstream sin(_input); + + while (!sin.eof()) + { + h256 data; + sin.read(reinterpret_cast(data.data()), 32); + numbers.push_back(u256(data)); + } + if (!_quiet) + cout << "Got " << numbers.size() << " inputs:" << endl; + + Assembly assembly; + for (u256 const& n: numbers) + { + if (!_quiet) + cout << n << endl; + assembly.append(n); + } + for (bool isCreation: {false, true}) + for (unsigned runs: {1, 2, 3, 20, 40, 100, 200, 400, 1000}) + { + ConstantOptimisationMethod::optimiseConstants( + isCreation, + runs, + EVMVersion{}, + assembly, + const_cast(assembly.items()) + ); + } +} + +void FuzzerUtil::testStandardCompiler(string const& _input, bool _quiet) +{ + if (!_quiet) + cout << "Testing compiler via JSON interface." << endl; + + runCompiler(_input); +} diff --git a/test/tools/fuzzer_common.h b/test/tools/fuzzer_common.h new file mode 100644 index 000000000000..109d5d992993 --- /dev/null +++ b/test/tools/fuzzer_common.h @@ -0,0 +1,35 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ + +#include +#include +#include +#include + +#include + +#include +#include +#include + +struct FuzzerUtil +{ + static void runCompiler(std::string _input); + static void testCompiler(std::string const& _input, bool _optimize, bool quiet); + static void testConstantOptimizer(std::string const& _input, bool _quiet); + static void testStandardCompiler(std::string const& _input, bool _quiet); +}; \ No newline at end of file diff --git a/test/tools/ossfuzz/CMakeLists.txt b/test/tools/ossfuzz/CMakeLists.txt new file mode 100644 index 000000000000..6a32c0eb5124 --- /dev/null +++ b/test/tools/ossfuzz/CMakeLists.txt @@ -0,0 +1,12 @@ +add_custom_target(ossfuzz) +add_dependencies(ossfuzz solc_opt_ossfuzz solc_noopt_ossfuzz const_opt_ossfuzz) + +#[[FuzzingEngine.a is provided by oss-fuzz's Dockerized build environment]] +add_executable(solc_opt_ossfuzz solc_opt_ossfuzz.cpp ../fuzzer_common.cpp) +target_link_libraries(solc_opt_ossfuzz PRIVATE libsolc evmasm FuzzingEngine.a) + +add_executable(solc_noopt_ossfuzz solc_noopt_ossfuzz.cpp ../fuzzer_common.cpp) +target_link_libraries(solc_noopt_ossfuzz PRIVATE libsolc evmasm FuzzingEngine.a) + +add_executable(const_opt_ossfuzz const_opt_ossfuzz.cpp ../fuzzer_common.cpp) +target_link_libraries(const_opt_ossfuzz PRIVATE libsolc evmasm FuzzingEngine.a) diff --git a/test/tools/ossfuzz/README.md b/test/tools/ossfuzz/README.md new file mode 100644 index 000000000000..eb75f822a5bd --- /dev/null +++ b/test/tools/ossfuzz/README.md @@ -0,0 +1,20 @@ +## Intro + +[oss-fuzz][1] is Google's fuzzing infrastructure that performs continuous fuzzing. What this means is that, each and every upstream commit is automatically fetched by the infrastructure and fuzzed. + +## What does this directory contain? + +To help oss-fuzz do this, we (as project maintainers) need to provide the following: + +- test harnesses: C/C++ tests that define the `LLVMFuzzerTestOneInput` API. This determines what is to be fuzz tested. +- build infrastructure: (c)make targets per fuzzing binary. Fuzzing requires coverage and memory instrumentation of the code to be fuzzed. + +## What is libFuzzingEngine.a? + +`libFuzzingEngine.a` is an oss-fuzz-related dependency. It is present in the Dockerized environment in which Solidity's oss-fuzz code will be built. + +## Is this directory relevant for routine Solidity CI builds? + +No. This is the reason why the `add_subdirectory(ossfuzz)` cmake directive is nested under the `if (OSSFUZZ)` predicate. `OSSFUZZ` is a solidity-wide cmake option that is invoked by the ossfuzz solidity-builder-bot in order to compile solidity fuzzer binaries. + +[1]: https://github.com/google/oss-fuzz diff --git a/test/tools/ossfuzz/const_opt_ossfuzz.cpp b/test/tools/ossfuzz/const_opt_ossfuzz.cpp new file mode 100644 index 000000000000..b394d7dafdef --- /dev/null +++ b/test/tools/ossfuzz/const_opt_ossfuzz.cpp @@ -0,0 +1,27 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ + +#include + +using namespace std; + +extern "C" int LLVMFuzzerTestOneInput(uint8_t const* _data, size_t _size) +{ + string input(reinterpret_cast(_data), _size); + FuzzerUtil::testConstantOptimizer(input, true); + return 0; +} \ No newline at end of file diff --git a/test/tools/ossfuzz/solc_noopt_ossfuzz.cpp b/test/tools/ossfuzz/solc_noopt_ossfuzz.cpp new file mode 100644 index 000000000000..7e28c3aca613 --- /dev/null +++ b/test/tools/ossfuzz/solc_noopt_ossfuzz.cpp @@ -0,0 +1,27 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ + +#include + +using namespace std; + +extern "C" int LLVMFuzzerTestOneInput(uint8_t const* _data, size_t _size) +{ + string input(reinterpret_cast(_data), _size); + FuzzerUtil::testCompiler(input, /*optimize=*/false, /*quiet=*/true); + return 0; +} diff --git a/test/tools/ossfuzz/solc_opt_ossfuzz.cpp b/test/tools/ossfuzz/solc_opt_ossfuzz.cpp new file mode 100644 index 000000000000..3ad8e5f74f3e --- /dev/null +++ b/test/tools/ossfuzz/solc_opt_ossfuzz.cpp @@ -0,0 +1,27 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ + +#include + +using namespace std; + +extern "C" int LLVMFuzzerTestOneInput(uint8_t const* _data, size_t _size) +{ + string input(reinterpret_cast(_data), _size); + FuzzerUtil::testCompiler(input, /*optimize=*/true, /*quiet=*/true); + return 0; +} From e3791d6dcff4fccd6f3b8e54abda953eb624799f Mon Sep 17 00:00:00 2001 From: chriseth Date: Tue, 22 Jan 2019 12:30:46 +0100 Subject: [PATCH 013/109] Fix reported packed encoded size for arrays and structs. --- Changelog.md | 4 ++++ libsolidity/ast/Types.cpp | 11 +++++++---- 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/Changelog.md b/Changelog.md index ceb1402a843b..9ab994bf4ea2 100644 --- a/Changelog.md +++ b/Changelog.md @@ -1,5 +1,9 @@ ### 0.5.4 (unreleased) +Bugfixes: + * Type system: Properly report packed encoded size for arrays and structs (mostly unused until now). + + Language Features: diff --git a/libsolidity/ast/Types.cpp b/libsolidity/ast/Types.cpp index 3a8c9878f85e..691579042015 100644 --- a/libsolidity/ast/Types.cpp +++ b/libsolidity/ast/Types.cpp @@ -1716,8 +1716,10 @@ bigint ArrayType::unlimitedCalldataEncodedSize(bool _padded) const { if (isDynamicallySized()) return 32; - bigint size = bigint(length()) * (isByteArray() ? 1 : baseType()->calldataEncodedSize(_padded)); - size = ((size + 31) / 32) * 32; + // Array elements are always padded. + bigint size = bigint(length()) * (isByteArray() ? 1 : baseType()->calldataEncodedSize(true)); + if (_padded) + size = ((size + 31) / 32) * 32; return size; } @@ -2034,7 +2036,7 @@ bool StructType::operator==(Type const& _other) const return ReferenceType::operator==(other) && other.m_struct == m_struct; } -unsigned StructType::calldataEncodedSize(bool _padded) const +unsigned StructType::calldataEncodedSize(bool) const { unsigned size = 0; for (auto const& member: members(nullptr)) @@ -2042,7 +2044,8 @@ unsigned StructType::calldataEncodedSize(bool _padded) const return 0; else { - unsigned memberSize = member.type->calldataEncodedSize(_padded); + // Struct members are always padded. + unsigned memberSize = member.type->calldataEncodedSize(true); if (memberSize == 0) return 0; size += memberSize; From 14b553897ebef832305984e2737400d787ac61c7 Mon Sep 17 00:00:00 2001 From: chriseth Date: Wed, 23 Jan 2019 11:12:49 +0100 Subject: [PATCH 014/109] Tests. --- test/libsolidity/SolidityTypes.cpp | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/test/libsolidity/SolidityTypes.cpp b/test/libsolidity/SolidityTypes.cpp index c839afd4ddb9..251835fe23f8 100644 --- a/test/libsolidity/SolidityTypes.cpp +++ b/test/libsolidity/SolidityTypes.cpp @@ -219,6 +219,30 @@ BOOST_AUTO_TEST_CASE(type_identifiers) BOOST_CHECK_EQUAL(InaccessibleDynamicType().identifier(), "t_inaccessible"); } +BOOST_AUTO_TEST_CASE(encoded_sizes) +{ + BOOST_CHECK_EQUAL(IntegerType(16).calldataEncodedSize(true), 32); + BOOST_CHECK_EQUAL(IntegerType(16).calldataEncodedSize(false), 2); + + BOOST_CHECK_EQUAL(FixedBytesType(16).calldataEncodedSize(true), 32); + BOOST_CHECK_EQUAL(FixedBytesType(16).calldataEncodedSize(false), 16); + + BOOST_CHECK_EQUAL(BoolType().calldataEncodedSize(true), 32); + BOOST_CHECK_EQUAL(BoolType().calldataEncodedSize(false), 1); + + shared_ptr uint24Array = make_shared( + DataLocation::Memory, + make_shared(24), + 9 + ); + BOOST_CHECK_EQUAL(uint24Array->calldataEncodedSize(true), 9 * 32); + BOOST_CHECK_EQUAL(uint24Array->calldataEncodedSize(false), 9 * 32); + + ArrayType twoDimArray(DataLocation::Memory, uint24Array, 3); + BOOST_CHECK_EQUAL(twoDimArray.calldataEncodedSize(true), 9 * 3 * 32); + BOOST_CHECK_EQUAL(twoDimArray.calldataEncodedSize(false), 9 * 3 * 32); +} + BOOST_AUTO_TEST_SUITE_END() } From 1fc98b81336b4bfdb47ab44895c124f6d45f2d20 Mon Sep 17 00:00:00 2001 From: chriseth Date: Thu, 24 Jan 2019 19:57:32 +0100 Subject: [PATCH 015/109] Fix optimizer suite. --- libyul/optimiser/Suite.cpp | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/libyul/optimiser/Suite.cpp b/libyul/optimiser/Suite.cpp index 8cf6e1048951..9a955c279352 100644 --- a/libyul/optimiser/Suite.cpp +++ b/libyul/optimiser/Suite.cpp @@ -97,7 +97,7 @@ void OptimiserSuite::run( ExpressionJoiner::run(ast); ExpressionJoiner::run(ast); ExpressionInliner(_dialect, ast).run(); - UnusedPruner::runUntilStabilised(_dialect, ast); + UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); ExpressionSplitter{_dialect, dispenser}(ast); SSATransform::run(ast, dispenser); @@ -124,11 +124,11 @@ void OptimiserSuite::run( } ExpressionJoiner::run(ast); Rematerialiser::run(_dialect, ast); - UnusedPruner::runUntilStabilised(_dialect, ast); + UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); ExpressionJoiner::run(ast); - UnusedPruner::runUntilStabilised(_dialect, ast); + UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); ExpressionJoiner::run(ast); - UnusedPruner::runUntilStabilised(_dialect, ast); + UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); SSAReverser::run(ast); CommonSubexpressionEliminator{_dialect}(ast); @@ -136,7 +136,7 @@ void OptimiserSuite::run( ExpressionJoiner::run(ast); Rematerialiser::run(_dialect, ast); - UnusedPruner::runUntilStabilised(_dialect, ast); + UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); _ast = std::move(ast); } From 12e2187661ecd4963d4f4b9d4ce5697c67ed3398 Mon Sep 17 00:00:00 2001 From: chriseth Date: Thu, 24 Jan 2019 20:36:51 +0100 Subject: [PATCH 016/109] Some tuning. --- libyul/optimiser/Suite.cpp | 3 +- .../yulOptimizerTests/fullSuite/abi2.yul | 34 +++++++++---------- .../fullSuite/abi_example1.yul | 34 +++++++++---------- 3 files changed, 36 insertions(+), 35 deletions(-) diff --git a/libyul/optimiser/Suite.cpp b/libyul/optimiser/Suite.cpp index 8cf6e1048951..b60e0a3ed85f 100644 --- a/libyul/optimiser/Suite.cpp +++ b/libyul/optimiser/Suite.cpp @@ -79,8 +79,8 @@ void OptimiserSuite::run( RedundantAssignEliminator::run(_dialect, ast); RedundantAssignEliminator::run(_dialect, ast); - CommonSubexpressionEliminator{_dialect}(ast); ExpressionSimplifier::run(_dialect, ast); + CommonSubexpressionEliminator{_dialect}(ast); StructuralSimplifier{_dialect}(ast); (BlockFlattener{})(ast); SSATransform::run(ast, dispenser); @@ -108,6 +108,7 @@ void OptimiserSuite::run( (FunctionGrouper{})(ast); EquivalentFunctionCombiner::run(ast); FullInliner{ast, dispenser}.run(); + (BlockFlattener{})(ast); SSATransform::run(ast, dispenser); RedundantAssignEliminator::run(_dialect, ast); diff --git a/test/libyul/yulOptimizerTests/fullSuite/abi2.yul b/test/libyul/yulOptimizerTests/fullSuite/abi2.yul index 41a52c672481..14dafdb19598 100644 --- a/test/libyul/yulOptimizerTests/fullSuite/abi2.yul +++ b/test/libyul/yulOptimizerTests/fullSuite/abi2.yul @@ -1090,43 +1090,43 @@ // { // revert(value4, value4) // } -// value0_57 := and(calldataload(add(headStart_55, value4)), 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF) +// value0_57 := and(calldataload(headStart_55), 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF) // value1_58 := calldataload(add(headStart_55, 32)) // let offset_62 := calldataload(add(headStart_55, 64)) -// let _201 := 0xffffffffffffffff -// if gt(offset_62, _201) +// let _200 := 0xffffffffffffffff +// if gt(offset_62, _200) // { // revert(value4, value4) // } -// let _203 := add(headStart_55, offset_62) -// if iszero(slt(add(_203, 0x1f), dataEnd_56)) +// let _202 := add(headStart_55, offset_62) +// if iszero(slt(add(_202, 0x1f), dataEnd_56)) // { // revert(value4, value4) // } -// let abi_decode_length_15_246 := calldataload(_203) -// if gt(abi_decode_length_15_246, _201) +// let abi_decode_length_15_244 := calldataload(_202) +// if gt(abi_decode_length_15_244, _200) // { // revert(value4, value4) // } -// if gt(add(add(_203, abi_decode_length_15_246), 32), dataEnd_56) +// if gt(add(add(_202, abi_decode_length_15_244), 32), dataEnd_56) // { // revert(value4, value4) // } -// value2_59 := add(_203, 32) -// value3 := abi_decode_length_15_246 -// let _206 := calldataload(add(headStart_55, 96)) -// if iszero(lt(_206, 3)) +// value2_59 := add(_202, 32) +// value3 := abi_decode_length_15_244 +// let _205 := calldataload(add(headStart_55, 96)) +// if iszero(lt(_205, 3)) // { // revert(value4, value4) // } -// value4 := _206 +// value4 := _205 // } // function abi_encode_tuple_t_bytes32_t_address_t_uint256_t_bytes32_t_enum$_Operation_$1949_t_uint256_t_uint256_t_uint256_t_address_t_address_t_uint256__to_t_bytes32_t_address_t_uint256_t_bytes32_t_uint8_t_uint256_t_uint256_t_uint256_t_address_t_address_t_uint256_(headStart_252, value10_253, value9_254, value8_255, value7_256, value6_257, value5_258, value4_259, value3_260, value2_261, value1_262, value0_263) -> tail_264 // { // tail_264 := add(headStart_252, 352) // mstore(headStart_252, value0_263) -// let _413 := 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -// mstore(add(headStart_252, 32), and(value1_262, _413)) +// let _409 := 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF +// mstore(add(headStart_252, 32), and(value1_262, _409)) // mstore(add(headStart_252, 64), value2_261) // mstore(add(headStart_252, 96), value3_260) // if iszero(lt(value4_259, 3)) @@ -1137,8 +1137,8 @@ // mstore(add(headStart_252, 160), value5_258) // mstore(add(headStart_252, 192), value6_257) // mstore(add(headStart_252, 224), value7_256) -// mstore(add(headStart_252, 256), and(value8_255, _413)) -// mstore(add(headStart_252, 288), and(value9_254, _413)) +// mstore(add(headStart_252, 256), and(value8_255, _409)) +// mstore(add(headStart_252, 288), and(value9_254, _409)) // mstore(add(headStart_252, 320), value10_253) // } // } diff --git a/test/libyul/yulOptimizerTests/fullSuite/abi_example1.yul b/test/libyul/yulOptimizerTests/fullSuite/abi_example1.yul index d38025ae99d8..03b9c2e4cf5d 100644 --- a/test/libyul/yulOptimizerTests/fullSuite/abi_example1.yul +++ b/test/libyul/yulOptimizerTests/fullSuite/abi_example1.yul @@ -474,20 +474,20 @@ // abi_encode_i_69 := add(abi_encode_i_69, 1) // } // { -// let _580 := mload(abi_encode_srcPtr) -// let abi_encode_pos_71_672 := abi_encode_pos -// let abi_encode_srcPtr_73_674 := _580 -// let abi_encode_i_74_675 := _2 +// let _579 := mload(abi_encode_srcPtr) +// let abi_encode_pos_71_671 := abi_encode_pos +// let abi_encode_srcPtr_73_673 := _579 +// let abi_encode_i_74_674 := _2 // for { // } -// lt(abi_encode_i_74_675, 0x3) +// lt(abi_encode_i_74_674, 0x3) // { -// abi_encode_i_74_675 := add(abi_encode_i_74_675, 1) +// abi_encode_i_74_674 := add(abi_encode_i_74_674, 1) // } // { -// mstore(abi_encode_pos_71_672, and(mload(abi_encode_srcPtr_73_674), 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF)) -// abi_encode_srcPtr_73_674 := add(abi_encode_srcPtr_73_674, _1) -// abi_encode_pos_71_672 := add(abi_encode_pos_71_672, _1) +// mstore(abi_encode_pos_71_671, and(mload(abi_encode_srcPtr_73_673), 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF)) +// abi_encode_srcPtr_73_673 := add(abi_encode_srcPtr_73_673, _1) +// abi_encode_pos_71_671 := add(abi_encode_pos_71_671, _1) // } // abi_encode_srcPtr := add(abi_encode_srcPtr, _1) // abi_encode_pos := add(abi_encode_pos, 0x60) @@ -504,15 +504,15 @@ // { // revert(_2, _2) // } -// let abi_decode_value2_317 := abi_decode_t_array$_t_uint256_$dyn_memory_ptr(add(_221, abi_decode_offset_64), _220) +// let abi_decode_value2_314 := abi_decode_t_array$_t_uint256_$dyn_memory_ptr(add(_221, abi_decode_offset_64), _220) // let abi_decode_offset_65 := calldataload(add(_221, 96)) // if gt(abi_decode_offset_65, abi_decode__74) // { // revert(_2, _2) // } -// let abi_decode_value3_318 := abi_decode_t_array$_t_array$_t_uint256_$2_memory_$dyn_memory_ptr(add(_221, abi_decode_offset_65), _220) +// let abi_decode_value3_315 := abi_decode_t_array$_t_array$_t_uint256_$2_memory_$dyn_memory_ptr(add(_221, abi_decode_offset_65), _220) // sstore(calldataload(_221), calldataload(add(_221, _1))) -// sstore(abi_decode_value2_317, abi_decode_value3_318) +// sstore(abi_decode_value2_314, abi_decode_value3_315) // sstore(_2, abi_encode_pos) // function abi_decode_t_array$_t_array$_t_uint256_$2_memory_$dyn_memory_ptr(offset_3, end_4) -> array_5 // { @@ -544,10 +544,10 @@ // revert(0, 0) // } // let abi_decode_dst_15 := allocateMemory(array_allocation_size_t_array$_t_uint256_$2_memory(0x2)) -// let abi_decode_dst_15_1155 := abi_decode_dst_15 +// let abi_decode_dst_15_1154 := abi_decode_dst_15 // let abi_decode_src_16 := src_8 -// let abi_decode__239 := add(src_8, 0x40) -// if gt(abi_decode__239, end_4) +// let abi_decode__238 := add(src_8, 0x40) +// if gt(abi_decode__238, end_4) // { // revert(0, 0) // } @@ -563,9 +563,9 @@ // abi_decode_dst_15 := add(abi_decode_dst_15, _16) // abi_decode_src_16 := add(abi_decode_src_16, _16) // } -// mstore(dst_7, abi_decode_dst_15_1155) +// mstore(dst_7, abi_decode_dst_15_1154) // dst_7 := add(dst_7, _16) -// src_8 := abi_decode__239 +// src_8 := abi_decode__238 // } // } // function abi_decode_t_array$_t_uint256_$dyn_memory_ptr(offset_27, end_28) -> array_29 From 3b5f446018f72bb67be5a5959a7fc9c734f48c51 Mon Sep 17 00:00:00 2001 From: Shelly Grossman Date: Sat, 26 Jan 2019 15:30:17 +0200 Subject: [PATCH 017/109] Ensuring UTF-8 encoding for MSVC builds --- cmake/EthCompilerSettings.cmake | 1 + 1 file changed, 1 insertion(+) diff --git a/cmake/EthCompilerSettings.cmake b/cmake/EthCompilerSettings.cmake index d05ccaffa66d..99c03af00411 100644 --- a/cmake/EthCompilerSettings.cmake +++ b/cmake/EthCompilerSettings.cmake @@ -124,6 +124,7 @@ elseif (DEFINED MSVC) add_compile_options(/wd4800) # disable forcing value to bool 'true' or 'false' (performance warning) (4800) add_compile_options(-D_WIN32_WINNT=0x0600) # declare Windows Vista API requirement add_compile_options(-DNOMINMAX) # undefine windows.h MAX && MIN macros cause it cause conflicts with std::min && std::max functions + add_compile_options(/utf-8) # enable utf-8 encoding (solves warning 4819) # disable empty object file warning set(CMAKE_STATIC_LINKER_FLAGS "${CMAKE_STATIC_LINKER_FLAGS} /ignore:4221") From 06998bc8e1d2a8c708f18240b56fecde6c63d3ac Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Mon, 28 Jan 2019 10:40:33 +0000 Subject: [PATCH 018/109] Run constant optimiser on fresh inputs in the fuzzer --- test/tools/fuzzer_common.cpp | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/test/tools/fuzzer_common.cpp b/test/tools/fuzzer_common.cpp index a700c557e4e2..899ebedb80e2 100644 --- a/test/tools/fuzzer_common.cpp +++ b/test/tools/fuzzer_common.cpp @@ -95,12 +95,14 @@ void FuzzerUtil::testConstantOptimizer(string const& _input, bool _quiet) for (bool isCreation: {false, true}) for (unsigned runs: {1, 2, 3, 20, 40, 100, 200, 400, 1000}) { + // Make a copy here so that each time we start with the original state. + Assembly tmp = assembly; ConstantOptimisationMethod::optimiseConstants( isCreation, runs, EVMVersion{}, assembly, - const_cast(assembly.items()) + const_cast(tmp.items()) ); } } From 9d52325b2981a091a54de43cd7839455645f7a86 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Mon, 28 Jan 2019 11:20:46 +0000 Subject: [PATCH 019/109] Move ConstantOptimiser constructor to header --- libevmasm/ConstantOptimiser.cpp | 5 ----- libevmasm/ConstantOptimiser.h | 3 ++- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/libevmasm/ConstantOptimiser.cpp b/libevmasm/ConstantOptimiser.cpp index ae1a564373d1..b56415a9502d 100644 --- a/libevmasm/ConstantOptimiser.cpp +++ b/libevmasm/ConstantOptimiser.cpp @@ -134,11 +134,6 @@ bigint LiteralMethod::gasNeeded() const ); } -CodeCopyMethod::CodeCopyMethod(Params const& _params, u256 const& _value): - ConstantOptimisationMethod(_params, _value) -{ -} - bigint CodeCopyMethod::gasNeeded() const { return combineGas( diff --git a/libevmasm/ConstantOptimiser.h b/libevmasm/ConstantOptimiser.h index 04c43c5d6fa2..9108e03976a2 100644 --- a/libevmasm/ConstantOptimiser.h +++ b/libevmasm/ConstantOptimiser.h @@ -119,7 +119,8 @@ class LiteralMethod: public ConstantOptimisationMethod class CodeCopyMethod: public ConstantOptimisationMethod { public: - explicit CodeCopyMethod(Params const& _params, u256 const& _value); + explicit CodeCopyMethod(Params const& _params, u256 const& _value): + ConstantOptimisationMethod(_params, _value) {} bigint gasNeeded() const override; AssemblyItems execute(Assembly& _assembly) const override; From c33fc2e114f5454a88e27a7c52ec0a08368aa9a7 Mon Sep 17 00:00:00 2001 From: Chris Ward Date: Mon, 28 Jan 2019 14:12:16 +0100 Subject: [PATCH 020/109] Remove web3 return FAQ item --- docs/frequently-asked-questions.rst | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/docs/frequently-asked-questions.rst b/docs/frequently-asked-questions.rst index 4bfb8708c68d..37ae9da4381b 100644 --- a/docs/frequently-asked-questions.rst +++ b/docs/frequently-asked-questions.rst @@ -4,18 +4,6 @@ Frequently Asked Questions This list was originally compiled by `fivedogit `_. - -*************** -Basic Questions -*************** - -If I return an ``enum``, I only get integer values in web3.js. How to get the named values? -=========================================================================================== - -Enums are not supported by the ABI, they are just supported by Solidity. -You have to do the mapping yourself for now, we might provide some help -later. - ****************** Advanced Questions ****************** From 64312584dcf93cee511b65b7a3c6b7250e86cb5b Mon Sep 17 00:00:00 2001 From: chriseth Date: Mon, 28 Jan 2019 14:41:33 +0100 Subject: [PATCH 021/109] Do not provide access to external assembly variables from within functions. --- libsolidity/codegen/CompilerContext.cpp | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/libsolidity/codegen/CompilerContext.cpp b/libsolidity/codegen/CompilerContext.cpp index 861b1c98af96..d7839a0a40b8 100644 --- a/libsolidity/codegen/CompilerContext.cpp +++ b/libsolidity/codegen/CompilerContext.cpp @@ -336,9 +336,11 @@ void CompilerContext::appendInlineAssembly( identifierAccess.resolve = [&]( yul::Identifier const& _identifier, yul::IdentifierContext, - bool - ) + bool _insideFunction + ) -> size_t { + if (_insideFunction) + return size_t(-1); auto it = std::find(_localVariables.begin(), _localVariables.end(), _identifier.name.str()); return it == _localVariables.end() ? size_t(-1) : 1; }; From 230e27fd62699a447cdd182f8c40ffea406fd33b Mon Sep 17 00:00:00 2001 From: Chris Ward Date: Mon, 28 Jan 2019 14:49:21 +0100 Subject: [PATCH 022/109] Remove FAQ Item --- docs/frequently-asked-questions.rst | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs/frequently-asked-questions.rst b/docs/frequently-asked-questions.rst index 37ae9da4381b..e8ee18494ae8 100644 --- a/docs/frequently-asked-questions.rst +++ b/docs/frequently-asked-questions.rst @@ -8,14 +8,6 @@ This list was originally compiled by `fivedogit `_. Advanced Questions ****************** -Get return value from non-constant function from another contract -================================================================= - -The key point is that the calling contract needs to know about the function it intends to call. - -See `ping.sol `_ -and `pong.sol `_. - How do I initialize a contract with only a specific amount of wei? ================================================================== From 9aafa32825f17f6e66c0fd6f6747c28b1402069f Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Fri, 25 Jan 2019 01:59:42 +0000 Subject: [PATCH 023/109] Disallow empty import statements --- Changelog.md | 2 +- libsolidity/interface/CompilerStack.cpp | 2 ++ libsolidity/parsing/Parser.cpp | 2 ++ test/libsolidity/syntaxTests/parsing/import_empty.sol | 3 +++ 4 files changed, 8 insertions(+), 1 deletion(-) create mode 100644 test/libsolidity/syntaxTests/parsing/import_empty.sol diff --git a/Changelog.md b/Changelog.md index bc41fb8103e2..ff635c2083a0 100644 --- a/Changelog.md +++ b/Changelog.md @@ -1,10 +1,10 @@ ### 0.5.4 (unreleased) Bugfixes: + * Parser: Disallow empty import statements. * Type system: Properly report packed encoded size for arrays and structs (mostly unused until now). - Language Features: diff --git a/libsolidity/interface/CompilerStack.cpp b/libsolidity/interface/CompilerStack.cpp index 9e4da62d84d5..0c140887ccfc 100644 --- a/libsolidity/interface/CompilerStack.cpp +++ b/libsolidity/interface/CompilerStack.cpp @@ -681,6 +681,8 @@ StringMap CompilerStack::loadMissingSources(SourceUnit const& _ast, std::string for (auto const& node: _ast.nodes()) if (ImportDirective const* import = dynamic_cast(node.get())) { + solAssert(!import->path().empty(), "Import path cannot be empty."); + string importPath = dev::absolutePath(import->path(), _sourcePath); // The current value of `path` is the absolute path as seen from this source file. // We first have to apply remappings before we can store the actual absolute path diff --git a/libsolidity/parsing/Parser.cpp b/libsolidity/parsing/Parser.cpp index 35476a76feb5..8cbac5a291d2 100644 --- a/libsolidity/parsing/Parser.cpp +++ b/libsolidity/parsing/Parser.cpp @@ -221,6 +221,8 @@ ASTPointer Parser::parseImportDirective() fatalParserError("Expected import path."); path = getLiteralAndAdvance(); } + if (path->empty()) + fatalParserError("Import path cannot be empty."); nodeFactory.markEndPosition(); expectToken(Token::Semicolon); return nodeFactory.createNode(path, unitAlias, move(symbolAliases)); diff --git a/test/libsolidity/syntaxTests/parsing/import_empty.sol b/test/libsolidity/syntaxTests/parsing/import_empty.sol new file mode 100644 index 000000000000..1853f6ec33a3 --- /dev/null +++ b/test/libsolidity/syntaxTests/parsing/import_empty.sol @@ -0,0 +1,3 @@ +import ""; +// ---- +// ParserError: (9-10): Import path cannot be empty. From 4aa2d965dcdb7f63323f13d913d9d0a4d99a6d40 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Mon, 28 Jan 2019 15:44:41 +0000 Subject: [PATCH 024/109] Remove once-off helper in ConstantOptimiser For clarity. Makes code easier to understand. --- libevmasm/ConstantOptimiser.cpp | 2 +- libevmasm/ConstantOptimiser.h | 2 -- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/libevmasm/ConstantOptimiser.cpp b/libevmasm/ConstantOptimiser.cpp index b56415a9502d..76e96aa5f92a 100644 --- a/libevmasm/ConstantOptimiser.cpp +++ b/libevmasm/ConstantOptimiser.cpp @@ -129,7 +129,7 @@ bigint LiteralMethod::gasNeeded() const return combineGas( simpleRunGas({Instruction::PUSH1}), // PUSHX plus data - (m_params.isCreation ? GasCosts::txDataNonZeroGas : GasCosts::createDataGas) + dataGas(), + (m_params.isCreation ? GasCosts::txDataNonZeroGas : GasCosts::createDataGas) + dataGas(toCompactBigEndian(m_value, 1)), 0 ); } diff --git a/libevmasm/ConstantOptimiser.h b/libevmasm/ConstantOptimiser.h index 9108e03976a2..ff14a916e8b1 100644 --- a/libevmasm/ConstantOptimiser.h +++ b/libevmasm/ConstantOptimiser.h @@ -79,8 +79,6 @@ class ConstantOptimisationMethod static bigint simpleRunGas(AssemblyItems const& _items); /// @returns the gas needed to store the given data literally bigint dataGas(bytes const& _data) const; - /// @returns the gas needed to store the value literally - bigint dataGas() const { return dataGas(toCompactBigEndian(m_value, 1)); } static size_t bytesRequired(AssemblyItems const& _items); /// @returns the combined estimated gas usage taking @a m_params into account. bigint combineGas( From c19afd5ad3cd7d4ce9ffc1288bf5f48ad374eae0 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Mon, 28 Jan 2019 16:18:49 +0000 Subject: [PATCH 025/109] Fix fuzzer_common to keep assembly intact --- test/tools/fuzzer_common.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/tools/fuzzer_common.cpp b/test/tools/fuzzer_common.cpp index 899ebedb80e2..347f5e3b3abf 100644 --- a/test/tools/fuzzer_common.cpp +++ b/test/tools/fuzzer_common.cpp @@ -101,7 +101,7 @@ void FuzzerUtil::testConstantOptimizer(string const& _input, bool _quiet) isCreation, runs, EVMVersion{}, - assembly, + tmp, const_cast(tmp.items()) ); } From 0757b46f5decf23193160ca6d902e623d580da35 Mon Sep 17 00:00:00 2001 From: Leo Arias Date: Mon, 28 Jan 2019 11:18:47 -0600 Subject: [PATCH 026/109] snap: update libicu to bionic Required to build the snap in bionic, as explained in #5889. --- snap/snapcraft.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml index ea0f2ef51c32..b2decd2fa568 100644 --- a/snap/snapcraft.yaml +++ b/snap/snapcraft.yaml @@ -27,7 +27,7 @@ parts: source-type: git plugin: cmake build-packages: [build-essential, libboost-all-dev, libz3-dev] - stage-packages: [libicu55] + stage-packages: [libicu60] prepare: | if git describe --exact-match --tags 2> /dev/null then From ee09d545252e50b36f865da8e70cb46e3e21b219 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 29 Jan 2019 00:35:23 +0000 Subject: [PATCH 027/109] Change FIXMEs into TODOs where appropriate --- libsolidity/ast/Types.cpp | 4 ++-- libsolidity/interface/CompilerStack.cpp | 4 ++-- libyul/optimiser/InlinableExpressionFunctionFinder.cpp | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/libsolidity/ast/Types.cpp b/libsolidity/ast/Types.cpp index 691579042015..308c0fe5edda 100644 --- a/libsolidity/ast/Types.cpp +++ b/libsolidity/ast/Types.cpp @@ -3027,8 +3027,8 @@ u256 FunctionType::externalIdentifier() const bool FunctionType::isPure() const { - // FIXME: replace this with m_stateMutability == StateMutability::Pure once - // the callgraph analyzer is in place + // TODO: replace this with m_stateMutability == StateMutability::Pure once + // the callgraph analyzer is in place return m_kind == Kind::KECCAK256 || m_kind == Kind::ECRecover || diff --git a/libsolidity/interface/CompilerStack.cpp b/libsolidity/interface/CompilerStack.cpp index 0c140887ccfc..b97e064e856f 100644 --- a/libsolidity/interface/CompilerStack.cpp +++ b/libsolidity/interface/CompilerStack.cpp @@ -473,7 +473,7 @@ eth::LinkerObject const& CompilerStack::runtimeObject(string const& _contractNam return contract(_contractName).runtimeObject; } -/// FIXME: cache this string +/// TODO: cache this string string CompilerStack::assemblyString(string const& _contractName, StringMap _sourceCodes) const { if (m_stackState != CompilationSuccessful) @@ -486,7 +486,7 @@ string CompilerStack::assemblyString(string const& _contractName, StringMap _sou return string(); } -/// FIXME: cache the JSON +/// TODO: cache the JSON Json::Value CompilerStack::assemblyJSON(string const& _contractName, StringMap _sourceCodes) const { if (m_stackState != CompilationSuccessful) diff --git a/libyul/optimiser/InlinableExpressionFunctionFinder.cpp b/libyul/optimiser/InlinableExpressionFunctionFinder.cpp index f57faa7c2b31..f72e5357ce74 100644 --- a/libyul/optimiser/InlinableExpressionFunctionFinder.cpp +++ b/libyul/optimiser/InlinableExpressionFunctionFinder.cpp @@ -50,7 +50,7 @@ void InlinableExpressionFunctionFinder::operator()(FunctionDefinition const& _fu Assignment const& assignment = boost::get(bodyStatement); if (assignment.variableNames.size() == 1 && assignment.variableNames.front().name == retVariable) { - // FIXME: use code size metric here + // TODO: use code size metric here // We cannot overwrite previous settings, because this function definition // would not be valid here if we were searching inside a functionally inlinable From 3790f5fe0ee3d7343adc6ba7716e5f475408f5a2 Mon Sep 17 00:00:00 2001 From: Leonardo Alt Date: Tue, 29 Jan 2019 13:10:01 +0100 Subject: [PATCH 028/109] Use CVC4 instead of Z3 for Ubuntu CI tests --- .circleci/config.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 40aa6268691a..0f6821617722 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -114,7 +114,7 @@ jobs: name: Install build dependencies command: | apt-get -qq update - apt-get -qy install cmake libboost-regex-dev libboost-filesystem-dev libboost-test-dev libboost-system-dev libboost-program-options-dev libz3-dev + apt-get -qy install cmake libboost-regex-dev libboost-filesystem-dev libboost-test-dev libboost-system-dev libboost-program-options-dev libcvc4-dev ./scripts/install_obsolete_jsoncpp_1_7_4.sh - run: *setup_prerelease_commit_hash - run: *run_build @@ -159,7 +159,7 @@ jobs: name: Install build dependencies command: | apt-get -qq update - apt-get -qy install clang-7 cmake libboost-regex-dev libboost-filesystem-dev libboost-test-dev libboost-system-dev libboost-program-options-dev libz3-dev + apt-get -qy install clang-7 cmake libboost-regex-dev libboost-filesystem-dev libboost-test-dev libboost-system-dev libboost-program-options-dev libcvc4-dev ./scripts/install_obsolete_jsoncpp_1_7_4.sh - run: *setup_prerelease_commit_hash - run: *run_build @@ -249,7 +249,7 @@ jobs: name: Install dependencies command: | apt-get -qq update - apt-get -qy install libz3-dev libleveldb1v5 python-pip + apt-get -qy install libcvc4-dev libleveldb1v5 python-pip pip install codecov - run: mkdir -p test_results - run: From 9a33367bc61e4ce74b201891a3718f3c6757e415 Mon Sep 17 00:00:00 2001 From: Leonardo Alt Date: Wed, 23 Jan 2019 11:22:38 +0100 Subject: [PATCH 029/109] [SMTChecker] Warn when no solver was found and there are unhandled queries. --- liblangutil/ErrorReporter.h | 7 +++++++ libsolidity/formal/SMTChecker.cpp | 22 +++++++++++++++++++++- libsolidity/formal/SMTChecker.h | 11 ++++++++++- libsolidity/formal/SMTPortfolio.cpp | 9 +++++++++ libsolidity/formal/SMTPortfolio.h | 3 ++- libsolidity/formal/SolverInterface.h | 3 +++ 6 files changed, 52 insertions(+), 3 deletions(-) diff --git a/liblangutil/ErrorReporter.h b/liblangutil/ErrorReporter.h index d90e652eaebb..fe4321ce5418 100644 --- a/liblangutil/ErrorReporter.h +++ b/liblangutil/ErrorReporter.h @@ -22,6 +22,8 @@ #pragma once +#include + #include #include @@ -40,6 +42,11 @@ class ErrorReporter ErrorReporter& operator=(ErrorReporter const& _errorReporter); + void append(ErrorList const& _errorList) + { + m_errorList += _errorList; + } + void warning(std::string const& _description); void warning(SourceLocation const& _location, std::string const& _description); diff --git a/libsolidity/formal/SMTChecker.cpp b/libsolidity/formal/SMTChecker.cpp index 500b610f6236..912e2a69be7c 100644 --- a/libsolidity/formal/SMTChecker.cpp +++ b/libsolidity/formal/SMTChecker.cpp @@ -34,7 +34,8 @@ using namespace dev::solidity; SMTChecker::SMTChecker(ErrorReporter& _errorReporter, map const& _smtlib2Responses): m_interface(make_shared(_smtlib2Responses)), - m_errorReporter(_errorReporter) + m_errorReporterReference(_errorReporter), + m_errorReporter(m_smtErrors) { #if defined (HAVE_Z3) || defined (HAVE_CVC4) if (!_smtlib2Responses.empty()) @@ -53,6 +54,25 @@ void SMTChecker::analyze(SourceUnit const& _source, shared_ptr const& _ m_scanner = _scanner; if (_source.annotation().experimentalFeatures.count(ExperimentalFeature::SMTChecker)) _source.accept(*this); + + solAssert(m_interface->solvers() > 0, ""); + // If this check is true, Z3 and CVC4 are not available + // and the query answers were not provided, since SMTPortfolio + // guarantees that SmtLib2Interface is the first solver. + if (!m_interface->unhandledQueries().empty() && m_interface->solvers() == 1) + { + if (!m_noSolverWarning) + { + m_noSolverWarning = true; + m_errorReporterReference.warning( + SourceLocation(), + "SMTChecker analysis was not possible since no integrated SMT solver (Z3 or CVC4) was found." + ); + } + } + else + m_errorReporterReference.append(m_errorReporter.errors()); + m_errorReporter.clear(); } bool SMTChecker::visit(ContractDefinition const& _contract) diff --git a/libsolidity/formal/SMTChecker.h b/libsolidity/formal/SMTChecker.h index a85933c83493..1521c613b0a1 100644 --- a/libsolidity/formal/SMTChecker.h +++ b/libsolidity/formal/SMTChecker.h @@ -23,6 +23,7 @@ #include #include +#include #include #include @@ -215,6 +216,8 @@ class SMTChecker: private ASTConstVisitor std::shared_ptr m_variableUsage; bool m_loopExecutionHappened = false; bool m_arrayAssignmentHappened = false; + // True if the "No SMT solver available" warning was already created. + bool m_noSolverWarning = false; /// An Expression may have multiple smt::Expression due to /// repeated calls to the same function. std::unordered_map> m_expressions; @@ -225,7 +228,13 @@ class SMTChecker: private ASTConstVisitor /// Used to retrieve models. std::set m_uninterpretedTerms; std::vector m_pathConditions; - langutil::ErrorReporter& m_errorReporter; + /// ErrorReporter that comes from CompilerStack. + langutil::ErrorReporter& m_errorReporterReference; + /// Local SMTChecker ErrorReporter. + /// This is necessary to show the "No SMT solver available" + /// warning before the others in case it's needed. + langutil::ErrorReporter m_errorReporter; + langutil::ErrorList m_smtErrors; std::shared_ptr m_scanner; /// Stores the current path of function calls. diff --git a/libsolidity/formal/SMTPortfolio.cpp b/libsolidity/formal/SMTPortfolio.cpp index 2a109b894c54..34255f905832 100644 --- a/libsolidity/formal/SMTPortfolio.cpp +++ b/libsolidity/formal/SMTPortfolio.cpp @@ -129,6 +129,15 @@ pair> SMTPortfolio::check(vector const& return make_pair(lastResult, finalValues); } +vector SMTPortfolio::unhandledQueries() +{ + // This code assumes that the constructor guarantees that + // SmtLib2Interface is in position 0. + solAssert(!m_solvers.empty(), ""); + solAssert(dynamic_cast(m_solvers.at(0).get()), ""); + return m_solvers.at(0)->unhandledQueries(); +} + bool SMTPortfolio::solverAnswered(CheckResult result) { return result == CheckResult::SATISFIABLE || result == CheckResult::UNSATISFIABLE; diff --git a/libsolidity/formal/SMTPortfolio.h b/libsolidity/formal/SMTPortfolio.h index 8c38bd2e31ec..54be1547e992 100644 --- a/libsolidity/formal/SMTPortfolio.h +++ b/libsolidity/formal/SMTPortfolio.h @@ -54,7 +54,8 @@ class SMTPortfolio: public SolverInterface, public boost::noncopyable void addAssertion(Expression const& _expr) override; std::pair> check(std::vector const& _expressionsToEvaluate) override; - std::vector unhandledQueries() override { return m_solvers.at(0)->unhandledQueries(); } + std::vector unhandledQueries() override; + unsigned solvers() override { return m_solvers.size(); } private: static bool solverAnswered(CheckResult result); diff --git a/libsolidity/formal/SolverInterface.h b/libsolidity/formal/SolverInterface.h index 6e0b17acac5e..76991a58f540 100644 --- a/libsolidity/formal/SolverInterface.h +++ b/libsolidity/formal/SolverInterface.h @@ -305,6 +305,9 @@ class SolverInterface /// @returns a list of queries that the system was not able to respond to. virtual std::vector unhandledQueries() { return {}; } + /// @returns how many SMT solvers this interface has. + virtual unsigned solvers() { return 1; } + protected: // SMT query timeout in milliseconds. static int const queryTimeout = 10000; From c537321309435edd98c7ad13c7dc0d7c448d2cc8 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 29 Jan 2019 16:08:25 +0000 Subject: [PATCH 030/109] Add more debuggin in the fuzzer --- test/tools/fuzzer_common.cpp | 10 +++++++--- test/tools/fuzzer_common.h | 4 ++-- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/test/tools/fuzzer_common.cpp b/test/tools/fuzzer_common.cpp index 347f5e3b3abf..d2777d2ff567 100644 --- a/test/tools/fuzzer_common.cpp +++ b/test/tools/fuzzer_common.cpp @@ -22,9 +22,13 @@ using namespace std; using namespace dev; using namespace dev::eth; -void FuzzerUtil::runCompiler(string _input) +void FuzzerUtil::runCompiler(string _input, bool _quiet) { + if (!_quiet) + cout << "Input JSON: " << _input << endl; string outputString(solidity_compile(_input.c_str(), nullptr)); + if (!_quiet) + cout << "Output JSON: " << outputString << endl; Json::Value output; if (!jsonParseStrict(outputString, output)) { @@ -66,7 +70,7 @@ void FuzzerUtil::testCompiler(string const& _input, bool _optimize, bool _quiet) // Enable all Contract-level outputs. config["settings"]["outputSelection"]["*"]["*"][0] = "*"; - runCompiler(jsonCompactPrint(config)); + runCompiler(jsonCompactPrint(config), _quiet); } void FuzzerUtil::testConstantOptimizer(string const& _input, bool _quiet) @@ -112,5 +116,5 @@ void FuzzerUtil::testStandardCompiler(string const& _input, bool _quiet) if (!_quiet) cout << "Testing compiler via JSON interface." << endl; - runCompiler(_input); + runCompiler(_input, _quiet); } diff --git a/test/tools/fuzzer_common.h b/test/tools/fuzzer_common.h index 109d5d992993..645bbfe264cd 100644 --- a/test/tools/fuzzer_common.h +++ b/test/tools/fuzzer_common.h @@ -28,8 +28,8 @@ struct FuzzerUtil { - static void runCompiler(std::string _input); - static void testCompiler(std::string const& _input, bool _optimize, bool quiet); + static void runCompiler(std::string _input, bool _quiet); + static void testCompiler(std::string const& _input, bool _optimize, bool _quiet); static void testConstantOptimizer(std::string const& _input, bool _quiet); static void testStandardCompiler(std::string const& _input, bool _quiet); }; \ No newline at end of file From 80e5e706716a25067d9b2dde2f056bb41f66ad8d Mon Sep 17 00:00:00 2001 From: Leonardo Alt Date: Tue, 29 Jan 2019 16:38:59 +0100 Subject: [PATCH 031/109] Remove Z3 from Travis builds --- .travis.yml | 4 ---- 1 file changed, 4 deletions(-) diff --git a/.travis.yml b/.travis.yml index 6d3d70e0ef84..c91daada14a8 100644 --- a/.travis.yml +++ b/.travis.yml @@ -58,13 +58,11 @@ matrix: before_install: - sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test - sudo add-apt-repository -y ppa:mhier/libboost-latest - - sudo add-apt-repository -y ppa:hvr/z3 - sudo apt-get update -qq install: - sudo apt-get install -qq g++-8 gcc-8 - sudo apt-get install -qq libboost1.67-dev - sudo apt-get install -qq libleveldb1 - - sudo apt-get install -qq libz3-dev - sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-8 90 - sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 90 @@ -78,13 +76,11 @@ matrix: before_install: - sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test - sudo add-apt-repository -y ppa:mhier/libboost-latest - - sudo add-apt-repository -y ppa:hvr/z3 - sudo apt-get update -qq install: - sudo apt-get install -qq g++-8 gcc-8 - sudo apt-get install -qq libboost1.67-dev - sudo apt-get install -qq libleveldb1 - - sudo apt-get install -qq libz3-dev - sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-8 90 - sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 90 From 0e2b43e141b5dcee40f7ed129a90b07b92ee2d72 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 29 Jan 2019 16:09:09 +0000 Subject: [PATCH 032/109] Pass by reference in fuzzer --- test/tools/fuzzer_common.cpp | 2 +- test/tools/fuzzer_common.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/test/tools/fuzzer_common.cpp b/test/tools/fuzzer_common.cpp index d2777d2ff567..12afca512180 100644 --- a/test/tools/fuzzer_common.cpp +++ b/test/tools/fuzzer_common.cpp @@ -22,7 +22,7 @@ using namespace std; using namespace dev; using namespace dev::eth; -void FuzzerUtil::runCompiler(string _input, bool _quiet) +void FuzzerUtil::runCompiler(string const& _input, bool _quiet) { if (!_quiet) cout << "Input JSON: " << _input << endl; diff --git a/test/tools/fuzzer_common.h b/test/tools/fuzzer_common.h index 645bbfe264cd..a163f1d3a310 100644 --- a/test/tools/fuzzer_common.h +++ b/test/tools/fuzzer_common.h @@ -28,7 +28,7 @@ struct FuzzerUtil { - static void runCompiler(std::string _input, bool _quiet); + static void runCompiler(std::string const& _input, bool _quiet); static void testCompiler(std::string const& _input, bool _optimize, bool _quiet); static void testConstantOptimizer(std::string const& _input, bool _quiet); static void testStandardCompiler(std::string const& _input, bool _quiet); From 1276f3c4f3f2d5ec933367bf13566601eea5a210 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 29 Jan 2019 16:14:28 +0000 Subject: [PATCH 033/109] Clean up includes in the fuzzer --- test/tools/afl_fuzzer.cpp | 9 +++++++-- test/tools/fuzzer_common.cpp | 8 +++++++- test/tools/fuzzer_common.h | 11 +---------- 3 files changed, 15 insertions(+), 13 deletions(-) diff --git a/test/tools/afl_fuzzer.cpp b/test/tools/afl_fuzzer.cpp index 9f4ac86cd39e..d74be1efe0f2 100644 --- a/test/tools/afl_fuzzer.cpp +++ b/test/tools/afl_fuzzer.cpp @@ -19,11 +19,16 @@ */ #include + +#include + #include +#include +#include + using namespace std; using namespace dev; -using namespace dev::eth; namespace po = boost::program_options; @@ -98,4 +103,4 @@ Allowed options)", FuzzerUtil::testCompiler(input, !arguments.count("without-optimizer"), quiet); return 0; -} \ No newline at end of file +} diff --git a/test/tools/fuzzer_common.cpp b/test/tools/fuzzer_common.cpp index 12afca512180..b661cb54fa0f 100644 --- a/test/tools/fuzzer_common.cpp +++ b/test/tools/fuzzer_common.cpp @@ -16,7 +16,13 @@ */ #include -#include + +#include +#include +#include +#include + +#include using namespace std; using namespace dev; diff --git a/test/tools/fuzzer_common.h b/test/tools/fuzzer_common.h index a163f1d3a310..ded0b6730907 100644 --- a/test/tools/fuzzer_common.h +++ b/test/tools/fuzzer_common.h @@ -15,16 +15,7 @@ along with solidity. If not, see . */ -#include -#include -#include -#include - -#include - #include -#include -#include struct FuzzerUtil { @@ -32,4 +23,4 @@ struct FuzzerUtil static void testCompiler(std::string const& _input, bool _optimize, bool _quiet); static void testConstantOptimizer(std::string const& _input, bool _quiet); static void testStandardCompiler(std::string const& _input, bool _quiet); -}; \ No newline at end of file +}; From 35483422f35054a21fc0833b5f270c3158a68b7a Mon Sep 17 00:00:00 2001 From: chriseth Date: Mon, 28 Jan 2019 11:59:18 +0100 Subject: [PATCH 034/109] Pad code to multiple of 32 bytes. --- Changelog.md | 1 + libsolidity/codegen/ExpressionCompiler.cpp | 2 +- test/libsolidity/SolidityEndToEndTest.cpp | 19 +++++++++++++++++++ 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/Changelog.md b/Changelog.md index ff635c2083a0..d3172cb06352 100644 --- a/Changelog.md +++ b/Changelog.md @@ -1,6 +1,7 @@ ### 0.5.4 (unreleased) Bugfixes: + * Code generator: Defensively pad allocation of creationCode and runtimeCode to multiples of 32 bytes. * Parser: Disallow empty import statements. * Type system: Properly report packed encoded size for arrays and structs (mostly unused until now). diff --git a/libsolidity/codegen/ExpressionCompiler.cpp b/libsolidity/codegen/ExpressionCompiler.cpp index e6bb163dd51f..c1079ed32b31 100644 --- a/libsolidity/codegen/ExpressionCompiler.cpp +++ b/libsolidity/codegen/ExpressionCompiler.cpp @@ -1348,7 +1348,7 @@ bool ExpressionCompiler::visit(MemberAccess const& _memberAccess) m_context.appendInlineAssembly( Whiskers(R"({ mstore(start, sub(end, add(start, 0x20))) - mstore(, end) + mstore(, and(add(end, 31), not(31))) })")("free", to_string(CompilerUtils::freeMemoryPointer)).render(), {"start", "end"} ); diff --git a/test/libsolidity/SolidityEndToEndTest.cpp b/test/libsolidity/SolidityEndToEndTest.cpp index 38be5ae7c747..e738eb04bc2c 100644 --- a/test/libsolidity/SolidityEndToEndTest.cpp +++ b/test/libsolidity/SolidityEndToEndTest.cpp @@ -14275,6 +14275,25 @@ BOOST_AUTO_TEST_CASE(code_access) ABI_CHECK(codeRuntime1, codeRuntime2); } +BOOST_AUTO_TEST_CASE(code_access_padding) +{ + char const* sourceCode = R"( + contract C { + function diff() public pure returns (uint remainder) { + bytes memory a = type(D).creationCode; + bytes memory b = type(D).runtimeCode; + assembly { remainder := mod(sub(b, a), 0x20) } + } + } + contract D { + function f() public pure returns (uint) { return 7; } + } + )"; + compileAndRun(sourceCode, 0, "C"); + // This checks that the allocation function pads to multiples of 32 bytes + ABI_CHECK(callContractFunction("diff()"), encodeArgs(0)); +} + BOOST_AUTO_TEST_CASE(code_access_create) { char const* sourceCode = R"( From e2642c4d9d28591d8c37227857f623e99cb356d2 Mon Sep 17 00:00:00 2001 From: Chris Ward Date: Mon, 28 Jan 2019 14:26:21 +0100 Subject: [PATCH 035/109] Remove initialisation of contract with wei FAQ item --- docs/frequently-asked-questions.rst | 34 +---------------------------- 1 file changed, 1 insertion(+), 33 deletions(-) diff --git a/docs/frequently-asked-questions.rst b/docs/frequently-asked-questions.rst index e8ee18494ae8..9dc368abadb6 100644 --- a/docs/frequently-asked-questions.rst +++ b/docs/frequently-asked-questions.rst @@ -2,37 +2,5 @@ Frequently Asked Questions ########################### -This list was originally compiled by `fivedogit `_. - -****************** -Advanced Questions -****************** - -How do I initialize a contract with only a specific amount of wei? -================================================================== - -Currently the approach is a little ugly, but there is little that can be done to improve it. -In the case of a ``contract A`` calling a new instance of ``contract B``, parentheses have to be used around -``new B`` because ``B.value`` would refer to a member of ``B`` called ``value``. -You will need to make sure that you have both contracts aware of each other's presence and that ``contract B`` has a ``payable`` constructor. -In this example:: - - pragma solidity ^0.5.0; - - contract B { - constructor() public payable {} - } - - contract A { - B child; - - function test() public { - child = (new B).value(10)(); //construct a new B with 10 wei - } - } - -More Questions? -=============== - -If you have more questions or your question is not answered here, please talk to us on +If your question is not answered here, please talk to us on `gitter `_ or file an `issue `_. From 17a1e7aed515254e59e20e6735a44a6180ae5606 Mon Sep 17 00:00:00 2001 From: chriseth Date: Mon, 28 Jan 2019 17:19:41 +0100 Subject: [PATCH 036/109] Better error message for out of stack in assembly. --- libsolidity/codegen/AsmCodeGen.cpp | 31 ++++++++++++------ libyul/backends/evm/EVMCodeTransform.cpp | 40 ++++++++++++++--------- libyul/backends/evm/EVMCodeTransform.h | 8 ++++- libyul/backends/evm/EVMObjectCompiler.cpp | 2 ++ 4 files changed, 55 insertions(+), 26 deletions(-) diff --git a/libsolidity/codegen/AsmCodeGen.cpp b/libsolidity/codegen/AsmCodeGen.cpp index c04c1c34f910..02ae6e661583 100644 --- a/libsolidity/codegen/AsmCodeGen.cpp +++ b/libsolidity/codegen/AsmCodeGen.cpp @@ -184,14 +184,25 @@ void CodeGenerator::assemble( ) { EthAssemblyAdapter assemblyAdapter(_assembly); - CodeTransform( - assemblyAdapter, - _analysisInfo, - _parsedData, - *EVMDialect::strictAssemblyForEVM(), - _optimize, - false, - _identifierAccess, - _useNamedLabelsForFunctions - )(_parsedData); + try + { + CodeTransform( + assemblyAdapter, + _analysisInfo, + _parsedData, + *EVMDialect::strictAssemblyForEVM(), + _optimize, + false, + _identifierAccess, + _useNamedLabelsForFunctions + )(_parsedData); + } + catch (StackTooDeepError const& _e) + { + BOOST_THROW_EXCEPTION( + InternalCompilerError() << errinfo_comment( + "Stack too deep when compiling inline assembly" + + (_e.comment() ? ": " + *_e.comment() : ".") + )); + } } diff --git a/libyul/backends/evm/EVMCodeTransform.cpp b/libyul/backends/evm/EVMCodeTransform.cpp index 04dc504001db..d36540d46299 100644 --- a/libyul/backends/evm/EVMCodeTransform.cpp +++ b/libyul/backends/evm/EVMCodeTransform.cpp @@ -192,7 +192,8 @@ void CodeTransform::operator()(VariableDeclaration const& _varDecl) bool atTopOfStack = true; for (int varIndex = numVariables - 1; varIndex >= 0; --varIndex) { - auto& var = boost::get(m_scope->identifiers.at(_varDecl.variables[varIndex].name)); + YulString varName = _varDecl.variables[varIndex].name; + auto& var = boost::get(m_scope->identifiers.at(varName)); m_context->variableStackHeights[&var] = height + varIndex; if (!m_allowStackOpt) continue; @@ -217,7 +218,7 @@ void CodeTransform::operator()(VariableDeclaration const& _varDecl) m_unusedStackSlots.erase(m_unusedStackSlots.begin()); m_context->variableStackHeights[&var] = slot; m_assembly.setSourceLocation(_varDecl.location); - if (int heightDiff = variableHeightDiff(var, true)) + if (int heightDiff = variableHeightDiff(var, varName, true)) m_assembly.appendInstruction(solidity::swapInstruction(heightDiff - 1)); m_assembly.appendInstruction(solidity::Instruction::POP); --m_stackAdjustment; @@ -353,7 +354,7 @@ void CodeTransform::operator()(Identifier const& _identifier) { // TODO: opportunity for optimization: Do not DUP if this is the last reference // to the top most element of the stack - if (int heightDiff = variableHeightDiff(_var, false)) + if (int heightDiff = variableHeightDiff(_var, _identifier.name, false)) m_assembly.appendInstruction(solidity::dupInstruction(heightDiff)); else // Store something to balance the stack @@ -542,7 +543,14 @@ void CodeTransform::operator()(FunctionDefinition const& _function) for (size_t i = 0; i < _function.returnVariables.size(); ++i) stackLayout.push_back(i); // Move return values down, but keep order. - solAssert(stackLayout.size() <= 17, "Stack too deep"); + if (stackLayout.size() > 17) + BOOST_THROW_EXCEPTION(StackTooDeepError() << errinfo_comment( + "The function " + + _function.name.str() + + " has " + + to_string(stackLayout.size() - 17) + + " parameters or return variables too many to fit the stack size." + )); while (!stackLayout.empty() && stackLayout.back() != int(stackLayout.size() - 1)) if (stackLayout.back() < 0) { @@ -711,7 +719,7 @@ void CodeTransform::generateAssignment(Identifier const& _variableName) if (auto var = m_scope->lookup(_variableName.name)) { Scope::Variable const& _var = boost::get(*var); - if (int heightDiff = variableHeightDiff(_var, true)) + if (int heightDiff = variableHeightDiff(_var, _variableName.name, true)) m_assembly.appendInstruction(solidity::swapInstruction(heightDiff - 1)); m_assembly.appendInstruction(solidity::Instruction::POP); decreaseReference(_variableName.name, _var); @@ -726,19 +734,21 @@ void CodeTransform::generateAssignment(Identifier const& _variableName) } } -int CodeTransform::variableHeightDiff(Scope::Variable const& _var, bool _forSwap) const +int CodeTransform::variableHeightDiff(Scope::Variable const& _var, YulString _varName, bool _forSwap) const { solAssert(m_context->variableStackHeights.count(&_var), ""); int heightDiff = m_assembly.stackHeight() - m_context->variableStackHeights[&_var]; - if (heightDiff <= (_forSwap ? 1 : 0) || heightDiff > (_forSwap ? 17 : 16)) - { - solUnimplemented( - "Variable inaccessible, too deep inside stack (" + to_string(heightDiff) + ")" - ); - return 0; - } - else - return heightDiff; + solAssert(heightDiff > (_forSwap ? 1 : 0), "Negative stack difference for variable."); + int limit = _forSwap ? 17 : 16; + if (heightDiff > limit) + BOOST_THROW_EXCEPTION(StackTooDeepError() << errinfo_comment( + "Variable " + + _varName.str() + + " is " + + to_string(heightDiff - limit) + + " slot(s) too deep inside the stack." + )); + return heightDiff; } void CodeTransform::expectDeposit(int _deposit, int _oldHeight) const diff --git a/libyul/backends/evm/EVMCodeTransform.h b/libyul/backends/evm/EVMCodeTransform.h index 7be6f892c8b5..511fa73d757f 100644 --- a/libyul/backends/evm/EVMCodeTransform.h +++ b/libyul/backends/evm/EVMCodeTransform.h @@ -40,6 +40,8 @@ namespace yul struct AsmAnalysisInfo; class EVMAssembly; +struct StackTooDeepError: virtual YulException {}; + struct CodeTransformContext { std::map labelIDs; @@ -85,6 +87,10 @@ class CodeTransform: public boost::static_visitor<> public: /// Create the code transformer. /// @param _identifierAccess used to resolve identifiers external to the inline assembly + /// As a side-effect of its construction, translates the Yul code and appends it to the + /// given assembly. + /// Throws StackTooDeepError if a variable is not accessible or if a function has too + /// many parameters. CodeTransform( AbstractAssembly& _assembly, AsmAnalysisInfo& _analysisInfo, @@ -172,7 +178,7 @@ class CodeTransform: public boost::static_visitor<> /// Determines the stack height difference to the given variables. Throws /// if it is not yet in scope or the height difference is too large. Returns /// the (positive) stack height difference otherwise. - int variableHeightDiff(Scope::Variable const& _var, bool _forSwap) const; + int variableHeightDiff(Scope::Variable const& _var, YulString _name, bool _forSwap) const; void expectDeposit(int _deposit, int _oldHeight) const; diff --git a/libyul/backends/evm/EVMObjectCompiler.cpp b/libyul/backends/evm/EVMObjectCompiler.cpp index 3f7634b234a3..dfeb7bb3ad12 100644 --- a/libyul/backends/evm/EVMObjectCompiler.cpp +++ b/libyul/backends/evm/EVMObjectCompiler.cpp @@ -60,5 +60,7 @@ void EVMObjectCompiler::run(Object& _object, bool _optimize) yulAssert(_object.analysisInfo, "No analysis info."); yulAssert(_object.code, "No code."); + // We do not catch and re-throw the stack too deep exception here because it is a YulException, + // which should be native to this part of the code. CodeTransform{m_assembly, *_object.analysisInfo, *_object.code, m_dialect, _optimize, m_evm15}(*_object.code); } From 61810def3b39bbe67a3964223798dcabe2f09d2f Mon Sep 17 00:00:00 2001 From: Leonardo Alt Date: Wed, 30 Jan 2019 12:54:05 +0100 Subject: [PATCH 037/109] Fix typo in creationCode docs --- docs/units-and-global-variables.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/units-and-global-variables.rst b/docs/units-and-global-variables.rst index ce7706c197e6..1345d8d837dc 100644 --- a/docs/units-and-global-variables.rst +++ b/docs/units-and-global-variables.rst @@ -255,7 +255,7 @@ Type Information The expression ``type(X)`` can be used to retrieve information about the type ``X``. Currently, there is limited support for this feature, but it might be expanded in the future. The following properties are -available for a conract type ``C``: +available for a contract type ``C``: ``type(C).creationCode``: Memory byte array that contains the creation bytecode of the contract. From e1780e3ae8f9e6f0f0bdb598a086b51be3e4fa4f Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Wed, 30 Jan 2019 12:57:25 +0000 Subject: [PATCH 038/109] Check message identifier in RPCSession --- test/RPCSession.cpp | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/test/RPCSession.cpp b/test/RPCSession.cpp index 608481189bfc..1d685bf1b727 100644 --- a/test/RPCSession.cpp +++ b/test/RPCSession.cpp @@ -326,6 +326,14 @@ Json::Value RPCSession::rpcCall(string const& _methodName, vector const& if (!jsonParseStrict(reply, result, &errorMsg)) BOOST_REQUIRE_MESSAGE(false, errorMsg); + if (!result.isMember("id") || !result["id"].isUInt()) + BOOST_FAIL("Badly formatted JSON-RPC response (missing or non-integer \"id\")"); + if (result["id"].asUInt() != (m_rpcSequence - 1)) + BOOST_FAIL( + "Response identifier mismatch. " + "Expected " + to_string(m_rpcSequence - 1) + " but got " + to_string(result["id"].asUInt()) + "." + ); + if (result.isMember("error")) { if (_canFail) From 8e2de02fc24e147c14e4d8603a15bcb05f2b8dbc Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Wed, 30 Jan 2019 13:34:40 +0000 Subject: [PATCH 039/109] Format RPCSession parsing error more nicely --- test/RPCSession.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/RPCSession.cpp b/test/RPCSession.cpp index 1d685bf1b727..dd276417b0a4 100644 --- a/test/RPCSession.cpp +++ b/test/RPCSession.cpp @@ -324,7 +324,7 @@ Json::Value RPCSession::rpcCall(string const& _methodName, vector const& Json::Value result; string errorMsg; if (!jsonParseStrict(reply, result, &errorMsg)) - BOOST_REQUIRE_MESSAGE(false, errorMsg); + BOOST_FAIL("Failed to parse JSON-RPC response: " + errorMsg); if (!result.isMember("id") || !result["id"].isUInt()) BOOST_FAIL("Badly formatted JSON-RPC response (missing or non-integer \"id\")"); From 021d30f0362723150a35d7267048790f881976bc Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 31 Jan 2019 13:14:24 +0000 Subject: [PATCH 040/109] Use specific boost include in DocStringParser This is needed for C++17 support (at least on certain Boost and OS combinations) --- libsolidity/parsing/DocStringParser.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libsolidity/parsing/DocStringParser.cpp b/libsolidity/parsing/DocStringParser.cpp index d1d45150f23d..0df2bae87184 100644 --- a/libsolidity/parsing/DocStringParser.cpp +++ b/libsolidity/parsing/DocStringParser.cpp @@ -20,7 +20,7 @@ #include #include -#include +#include #include using namespace std; From b1cb949a4b2c50da233eedd21bfb72df2aafecca Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Wed, 30 Jan 2019 13:57:19 +0000 Subject: [PATCH 041/109] Shutdown IPC socket gracefully in RPCSession --- test/RPCSession.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/test/RPCSession.h b/test/RPCSession.h index 92f9da4a5ba2..7529f23c3e4a 100644 --- a/test/RPCSession.h +++ b/test/RPCSession.h @@ -57,7 +57,10 @@ class IPCSocket: public boost::noncopyable public: explicit IPCSocket(std::string const& _path); std::string sendRequest(std::string const& _req); - ~IPCSocket() { close(m_socket); } + ~IPCSocket() { + shutdown(m_socket, SHUT_RDWR); + close(m_socket); + } std::string const& path() const { return m_path; } From 554511b68e69ac88b1535a8f87075e1fe87d4278 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 22 Jan 2019 11:41:21 +0000 Subject: [PATCH 042/109] Introduce solidity_free in libsolc --- Changelog.md | 1 + libsolc/libsolc.cpp | 4 ++++ libsolc/libsolc.h | 5 +++++ test/libsolidity/LibSolc.cpp | 3 +++ test/tools/fuzzer_common.cpp | 4 ++++ 5 files changed, 17 insertions(+) diff --git a/Changelog.md b/Changelog.md index d3172cb06352..86787337e0f8 100644 --- a/Changelog.md +++ b/Changelog.md @@ -10,6 +10,7 @@ Language Features: Compiler Features: + * C API (``libsolc`` / raw ``soljson.js``): Introduce ``solidity_free`` method which releases all internal buffers to save memory. Bugfixes: diff --git a/libsolc/libsolc.cpp b/libsolc/libsolc.cpp index 766e1c0cd6c2..f20b50d46047 100644 --- a/libsolc/libsolc.cpp +++ b/libsolc/libsolc.cpp @@ -98,4 +98,8 @@ extern char const* solidity_compile(char const* _input, CStyleReadFileCallback _ s_outputBuffer = compile(_input, _readCallback); return s_outputBuffer.c_str(); } +extern void solidity_free() noexcept +{ + s_outputBuffer.clear(); +} } diff --git a/libsolc/libsolc.h b/libsolc/libsolc.h index baa39fd441c1..2c55c2342391 100644 --- a/libsolc/libsolc.h +++ b/libsolc/libsolc.h @@ -55,6 +55,11 @@ char const* solidity_version() SOLC_NOEXCEPT; /// The pointer returned must not be freed by the caller. char const* solidity_compile(char const* _input, CStyleReadFileCallback _readCallback) SOLC_NOEXCEPT; +/// Frees up any allocated memory. +/// +/// NOTE: the pointer returned by solidity_compile is invalid after calling this! +void solidity_free() SOLC_NOEXCEPT; + #ifdef __cplusplus } #endif diff --git a/test/libsolidity/LibSolc.cpp b/test/libsolidity/LibSolc.cpp index ec97f22f084e..62ec32f76090 100644 --- a/test/libsolidity/LibSolc.cpp +++ b/test/libsolidity/LibSolc.cpp @@ -45,6 +45,7 @@ Json::Value compile(string const& _input) string output(solidity_compile(_input.c_str(), nullptr)); Json::Value ret; BOOST_REQUIRE(jsonParseStrict(output, ret)); + solidity_free(); return ret; } @@ -56,12 +57,14 @@ BOOST_AUTO_TEST_CASE(read_version) { string output(solidity_version()); BOOST_CHECK(output.find(VersionString) == 0); + solidity_free(); } BOOST_AUTO_TEST_CASE(read_license) { string output(solidity_license()); BOOST_CHECK(output.find("GNU GENERAL PUBLIC LICENSE") != string::npos); + solidity_free(); } BOOST_AUTO_TEST_CASE(standard_compilation) diff --git a/test/tools/fuzzer_common.cpp b/test/tools/fuzzer_common.cpp index b661cb54fa0f..0ad86e6eada7 100644 --- a/test/tools/fuzzer_common.cpp +++ b/test/tools/fuzzer_common.cpp @@ -35,6 +35,10 @@ void FuzzerUtil::runCompiler(string const& _input, bool _quiet) string outputString(solidity_compile(_input.c_str(), nullptr)); if (!_quiet) cout << "Output JSON: " << outputString << endl; + + // This should be safe given the above copies the output. + solidity_free(); + Json::Value output; if (!jsonParseStrict(outputString, output)) { From ae1cd252b0286cdee6b68cf15e0b5c0057923814 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Mon, 28 Jan 2019 16:29:10 +0000 Subject: [PATCH 043/109] Take only Assembly instance as an input to ConstantOptimiser --- libevmasm/Assembly.cpp | 3 +-- libevmasm/Assembly.h | 3 +++ libevmasm/ConstantOptimiser.cpp | 6 ++++-- libevmasm/ConstantOptimiser.h | 5 ++--- test/tools/fuzzer_common.cpp | 3 +-- 5 files changed, 11 insertions(+), 9 deletions(-) diff --git a/libevmasm/Assembly.cpp b/libevmasm/Assembly.cpp index 231eed93667b..7cf7137d2886 100644 --- a/libevmasm/Assembly.cpp +++ b/libevmasm/Assembly.cpp @@ -496,8 +496,7 @@ map Assembly::optimiseInternal( _settings.isCreation, _settings.isCreation ? 1 : _settings.expectedExecutionsPerDeployment, _settings.evmVersion, - *this, - m_items + *this ); return tagReplacements; diff --git a/libevmasm/Assembly.h b/libevmasm/Assembly.h index 5dc6ef0e01fa..1718fbaeff07 100644 --- a/libevmasm/Assembly.h +++ b/libevmasm/Assembly.h @@ -86,6 +86,9 @@ class Assembly /// Returns the assembly items. AssemblyItems const& items() const { return m_items; } + /// Returns the mutable assembly items. Use with care! + AssemblyItems& items() { return m_items; } + int deposit() const { return m_deposit; } void adjustDeposit(int _adjustment) { m_deposit += _adjustment; assertThrow(m_deposit >= 0, InvalidDeposit, ""); } void setDeposit(int _deposit) { m_deposit = _deposit; assertThrow(m_deposit >= 0, InvalidDeposit, ""); } diff --git a/libevmasm/ConstantOptimiser.cpp b/libevmasm/ConstantOptimiser.cpp index 76e96aa5f92a..7e961632114e 100644 --- a/libevmasm/ConstantOptimiser.cpp +++ b/libevmasm/ConstantOptimiser.cpp @@ -30,10 +30,12 @@ unsigned ConstantOptimisationMethod::optimiseConstants( bool _isCreation, size_t _runs, solidity::EVMVersion _evmVersion, - Assembly& _assembly, - AssemblyItems& _items + Assembly& _assembly ) { + // TODO: design the optimiser in a way this is not needed + AssemblyItems& _items = _assembly.items(); + unsigned optimisations = 0; map pushes; for (AssemblyItem const& item: _items) diff --git a/libevmasm/ConstantOptimiser.h b/libevmasm/ConstantOptimiser.h index ff14a916e8b1..181f9351e009 100644 --- a/libevmasm/ConstantOptimiser.h +++ b/libevmasm/ConstantOptimiser.h @@ -47,14 +47,13 @@ class ConstantOptimisationMethod { public: /// Tries to optimised how constants are represented in the source code and modifies - /// @a _assembly and its @a _items. + /// @a _assembly. /// @returns zero if no optimisations could be performed. static unsigned optimiseConstants( bool _isCreation, size_t _runs, solidity::EVMVersion _evmVersion, - Assembly& _assembly, - AssemblyItems& _items + Assembly& _assembly ); struct Params diff --git a/test/tools/fuzzer_common.cpp b/test/tools/fuzzer_common.cpp index b661cb54fa0f..7f70b61c679c 100644 --- a/test/tools/fuzzer_common.cpp +++ b/test/tools/fuzzer_common.cpp @@ -111,8 +111,7 @@ void FuzzerUtil::testConstantOptimizer(string const& _input, bool _quiet) isCreation, runs, EVMVersion{}, - tmp, - const_cast(tmp.items()) + tmp ); } } From 027e1781d36cfa61c6408176a227de0978dccfb3 Mon Sep 17 00:00:00 2001 From: Daniel Kirchner Date: Mon, 4 Feb 2019 12:47:51 +0100 Subject: [PATCH 044/109] Install static boost on Alpine linux. --- scripts/install_deps.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/install_deps.sh b/scripts/install_deps.sh index 0d1620c4225c..0ed13fdd624c 100755 --- a/scripts/install_deps.sh +++ b/scripts/install_deps.sh @@ -162,7 +162,7 @@ case $(uname -s) in # See https://pkgs.alpinelinux.org/ apk update - apk add boost-dev build-base cmake git + apk add boost-dev boost-static build-base cmake git ;; From 11969cd760ac25f108b2afa2ff72e7c83405d028 Mon Sep 17 00:00:00 2001 From: chriseth Date: Mon, 4 Feb 2019 15:50:20 +0100 Subject: [PATCH 045/109] Add some more explicit moves required in some compiler and boost version combinations. --- libyul/optimiser/ForLoopInitRewriter.cpp | 2 +- libyul/optimiser/SSATransform.cpp | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/libyul/optimiser/ForLoopInitRewriter.cpp b/libyul/optimiser/ForLoopInitRewriter.cpp index 36d5db681464..6650e49ddb18 100644 --- a/libyul/optimiser/ForLoopInitRewriter.cpp +++ b/libyul/optimiser/ForLoopInitRewriter.cpp @@ -38,7 +38,7 @@ void ForLoopInitRewriter::operator()(Block& _block) vector rewrite; swap(rewrite, forLoop.pre.statements); rewrite.emplace_back(move(forLoop)); - return rewrite; + return std::move(rewrite); } else { diff --git a/libyul/optimiser/SSATransform.cpp b/libyul/optimiser/SSATransform.cpp index 33c875b5db15..dd10336c617c 100644 --- a/libyul/optimiser/SSATransform.cpp +++ b/libyul/optimiser/SSATransform.cpp @@ -97,7 +97,7 @@ void SSATransform::operator()(Block& _block) varDecl.value )); v.emplace_back(move(varDecl)); - return v; + return std::move(v); } else if (_s.type() == typeid(Assignment)) { @@ -115,7 +115,7 @@ void SSATransform::operator()(Block& _block) assignment.value )); v.emplace_back(move(assignment)); - return v; + return std::move(v); } else visit(_s); From 4cac45dc4aa7561781015d37dadde3431b915cd3 Mon Sep 17 00:00:00 2001 From: Bhargava Shastry Date: Tue, 29 Jan 2019 12:54:21 +0100 Subject: [PATCH 046/109] Add fuzzer config files for oss-fuzz and a solidity fuzzing dictionary. Update ossfuzz README.md. --- test/tools/ossfuzz/README.md | 8 + .../ossfuzz/config/solc_noopt_ossfuzz.options | 2 + .../ossfuzz/config/solc_opt_ossfuzz.options | 2 + test/tools/ossfuzz/config/solidity.dict | 213 ++++++++++++++++++ 4 files changed, 225 insertions(+) create mode 100644 test/tools/ossfuzz/config/solc_noopt_ossfuzz.options create mode 100644 test/tools/ossfuzz/config/solc_opt_ossfuzz.options create mode 100644 test/tools/ossfuzz/config/solidity.dict diff --git a/test/tools/ossfuzz/README.md b/test/tools/ossfuzz/README.md index eb75f822a5bd..70469513c4e5 100644 --- a/test/tools/ossfuzz/README.md +++ b/test/tools/ossfuzz/README.md @@ -8,6 +8,14 @@ To help oss-fuzz do this, we (as project maintainers) need to provide the follow - test harnesses: C/C++ tests that define the `LLVMFuzzerTestOneInput` API. This determines what is to be fuzz tested. - build infrastructure: (c)make targets per fuzzing binary. Fuzzing requires coverage and memory instrumentation of the code to be fuzzed. +- configuration files: These are files with the `.options` extension that are parsed by oss-fuzz. The only option that we use currently is the `dictionary` option that asks the fuzzing engines behind oss-fuzz to use the specified dictionary. The specified dictionary happens to be `solidity.dict.` + +`solidity.dict` contains Solidity-specific syntactical tokens that are more likely to guide the fuzzer towards generating parseable and varied Solidity input. + +To be consistent and aid better evaluation of the utility of the fuzzing dictionary, we stick to the following rules-of-thumb: + - Full tokens such as `block.number` are preceded and followed by a whitespace + - Incomplete tokens including function calls such as `msg.sender.send()` are abbreviated `.send(` to provide some leeway to the fuzzer to sythesize variants such as `address(this).send()` + - Language keywords are suffixed by a whitespace with the exception of those that end a line of code such as `break;` and `continue;` ## What is libFuzzingEngine.a? diff --git a/test/tools/ossfuzz/config/solc_noopt_ossfuzz.options b/test/tools/ossfuzz/config/solc_noopt_ossfuzz.options new file mode 100644 index 000000000000..d596157f5b19 --- /dev/null +++ b/test/tools/ossfuzz/config/solc_noopt_ossfuzz.options @@ -0,0 +1,2 @@ +[libfuzzer] +dict = solidity.dict diff --git a/test/tools/ossfuzz/config/solc_opt_ossfuzz.options b/test/tools/ossfuzz/config/solc_opt_ossfuzz.options new file mode 100644 index 000000000000..d596157f5b19 --- /dev/null +++ b/test/tools/ossfuzz/config/solc_opt_ossfuzz.options @@ -0,0 +1,2 @@ +[libfuzzer] +dict = solidity.dict diff --git a/test/tools/ossfuzz/config/solidity.dict b/test/tools/ossfuzz/config/solidity.dict new file mode 100644 index 000000000000..5ff5318e42b5 --- /dev/null +++ b/test/tools/ossfuzz/config/solidity.dict @@ -0,0 +1,213 @@ +" address(this).balance " +" block.coinbase " +" block.difficulty " +" block.gaslimit " +" block.number " +" block.timestamp " +" days " +" ether " +" finney " +" gasleft() " +" hours " +" minutes " +" msg.data " +" msg.gas " +" msg.sender " +" msg.sig " +" msg.value " +" now " +" seconds " +" szabo " +" tx.gasprice " +" tx.origin " +" weeks " +" wei " +" years " +"!=" +"%" +"&" +"(" +")" +"*" +"**" +"+" +"++" +"-" +"--" +".balance" +".call(" +".callcode(" +".creationCode" +".delegatecall(" +".gas(" +".kill(" +".length" +".pop();" +".push(" +".runtimeCode" +".send(" +".staticcall(" +".transfer(" +".value" +"/" +"//" +"0**0" +"1.1" +"2e10" +":=" +";" +"<" +"<<" +"<=" +"==" +">" +">=" +">>" +"[a, b, c]" +"\\udead" +"\\xff" +"^" +"abi.encode(" +"abi.encodePacked(" +"abi.encodeWithSelector(" +"abi.encodeWithSignature(" +"add(" +"addmod(" +"address(this).call(" +"address(this).callcode(" +"address(this).delegatecall(" +"address(this).send(" +"address(this).transfer(" +"anonymous" +"assembly { " +"assert(" +"block.blockhash(" +"bool " +"break;" +"byte " +"bytes(" +"bytes1 " +"bytes10 " +"bytes11 " +"bytes12 " +"bytes13 " +"bytes14 " +"bytes15 " +"bytes16 " +"bytes17 " +"bytes18 " +"bytes19 " +"bytes2 " +"bytes20 " +"bytes21 " +"bytes22 " +"bytes23 " +"bytes24 " +"bytes25 " +"bytes26 " +"bytes27 " +"bytes28 " +"bytes29 " +"bytes3 " +"bytes30 " +"bytes32 " +"bytes4 " +"bytes5 " +"bytes6 " +"bytes7 " +"bytes8 " +"bytes9 " +"constant " +"constructor " +"continue;" +"contract " +"delete " +"do " +"ecrecover(" +"else " +"emit a(" +"enum B { " +"event e(" +"external " +"false " +"fixed " +"fixed128x128 " +"for (a=0;a<2;a++) " +"function bid() public payable { " +"hex\"001122FF\"" +"if " +"int " +"int x = -2**255;" +"int256 " +"int8 " +"interface i { " +"internal " +"is " +"keccak256(" +"keccak256.gas(" +"keccak256.value(" +"let x := " +"library l { " +"log0(" +"log1(" +"log2(" +"log3(" +"log4(" +"mapping(" +"memory m = " +"modifier onlySeller() { " +"mulmod(" +"new " +"payable " +"pragma experimental ABIEncoderV2;" +"pragma experimental SMTChecker;" +"pragma solidity >=0.4.0;" +"pragma solidity ^90.90.0" +"public " +"pure " +"require(" +"require(msg.sender == 0,\"\"" +"return " +"returns (" +"revert(" +"ripemd160(" +"ripemd160.gas(" +"ripemd160.value(" +"self" +"selfdestruct(" +"sha256(" +"sha256.gas(" +"sha256.value(" +"sha3(" +"storage sto = " +"string memory str = " +"string storage str = " +"struct V { " +"suicide(" +"super " +"switch " +"this" +"throw " +"true " +"try " +"type(" +"ufixed " +"ufixed128x128 " +"uint " +"uint256 " +"uint8 " +"uint[] " +"uint[][5] " +"using " +"var " +"view " +"while " +"x % y" +"x * 2**y" +"x / 2**y" +"x << y" +"{ uint x; }" +"{" +"|" +"}" +"~" From dd5b43741ce58bc4326a2cec1a68aa3f28e84766 Mon Sep 17 00:00:00 2001 From: chriseth Date: Mon, 28 Jan 2019 18:11:58 +0100 Subject: [PATCH 047/109] No-output Assembly. --- libyul/CMakeLists.txt | 2 + libyul/backends/evm/NoOutputAssembly.cpp | 143 +++++++++++++++++++++++ libyul/backends/evm/NoOutputAssembly.h | 75 ++++++++++++ 3 files changed, 220 insertions(+) create mode 100644 libyul/backends/evm/NoOutputAssembly.cpp create mode 100644 libyul/backends/evm/NoOutputAssembly.h diff --git a/libyul/CMakeLists.txt b/libyul/CMakeLists.txt index 259f43f85277..edae65552547 100644 --- a/libyul/CMakeLists.txt +++ b/libyul/CMakeLists.txt @@ -32,6 +32,8 @@ add_library(yul backends/evm/EVMDialect.h backends/evm/EVMObjectCompiler.cpp backends/evm/EVMObjectCompiler.h + backends/evm/NoOutputAssembly.h + backends/evm/NoOutputAssembly.cpp optimiser/ASTCopier.cpp optimiser/ASTCopier.h optimiser/ASTWalker.cpp diff --git a/libyul/backends/evm/NoOutputAssembly.cpp b/libyul/backends/evm/NoOutputAssembly.cpp new file mode 100644 index 000000000000..81c67a5e4ad3 --- /dev/null +++ b/libyul/backends/evm/NoOutputAssembly.cpp @@ -0,0 +1,143 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ +/** + * Assembly interface that ignores everything. Can be used as a backend for a compilation dry-run. + */ + +#include + +#include + +#include + +using namespace std; +using namespace dev; +using namespace langutil; +using namespace yul; + + +void NoOutputAssembly::appendInstruction(solidity::Instruction _instr) +{ + m_stackHeight += solidity::instructionInfo(_instr).ret - solidity::instructionInfo(_instr).args; +} + +void NoOutputAssembly::appendConstant(u256 const&) +{ + appendInstruction(solidity::pushInstruction(1)); +} + +void NoOutputAssembly::appendLabel(LabelID) +{ + appendInstruction(solidity::Instruction::JUMPDEST); +} + +void NoOutputAssembly::appendLabelReference(LabelID) +{ + solAssert(!m_evm15, "Cannot use plain label references in EMV1.5 mode."); + appendInstruction(solidity::pushInstruction(1)); +} + +NoOutputAssembly::LabelID NoOutputAssembly::newLabelId() +{ + return 1; +} + +AbstractAssembly::LabelID NoOutputAssembly::namedLabel(string const&) +{ + return 1; +} + +void NoOutputAssembly::appendLinkerSymbol(string const&) +{ + solAssert(false, "Linker symbols not yet implemented."); +} + +void NoOutputAssembly::appendJump(int _stackDiffAfter) +{ + solAssert(!m_evm15, "Plain JUMP used for EVM 1.5"); + appendInstruction(solidity::Instruction::JUMP); + m_stackHeight += _stackDiffAfter; +} + +void NoOutputAssembly::appendJumpTo(LabelID _labelId, int _stackDiffAfter) +{ + if (m_evm15) + m_stackHeight += _stackDiffAfter; + else + { + appendLabelReference(_labelId); + appendJump(_stackDiffAfter); + } +} + +void NoOutputAssembly::appendJumpToIf(LabelID _labelId) +{ + if (m_evm15) + m_stackHeight--; + else + { + appendLabelReference(_labelId); + appendInstruction(solidity::Instruction::JUMPI); + } +} + +void NoOutputAssembly::appendBeginsub(LabelID, int _arguments) +{ + solAssert(m_evm15, "BEGINSUB used for EVM 1.0"); + solAssert(_arguments >= 0, ""); + m_stackHeight += _arguments; +} + +void NoOutputAssembly::appendJumpsub(LabelID, int _arguments, int _returns) +{ + solAssert(m_evm15, "JUMPSUB used for EVM 1.0"); + solAssert(_arguments >= 0 && _returns >= 0, ""); + m_stackHeight += _returns - _arguments; +} + +void NoOutputAssembly::appendReturnsub(int _returns, int _stackDiffAfter) +{ + solAssert(m_evm15, "RETURNSUB used for EVM 1.0"); + solAssert(_returns >= 0, ""); + m_stackHeight += _stackDiffAfter - _returns; +} + +void NoOutputAssembly::appendAssemblySize() +{ + appendInstruction(solidity::Instruction::PUSH1); +} + +pair, AbstractAssembly::SubID> NoOutputAssembly::createSubAssembly() +{ + solAssert(false, "Sub assemblies not implemented."); + return {}; +} + +void NoOutputAssembly::appendDataOffset(AbstractAssembly::SubID) +{ + appendInstruction(solidity::Instruction::PUSH1); +} + +void NoOutputAssembly::appendDataSize(AbstractAssembly::SubID) +{ + appendInstruction(solidity::Instruction::PUSH1); +} + +AbstractAssembly::SubID NoOutputAssembly::appendData(bytes const&) +{ + return 1; +} diff --git a/libyul/backends/evm/NoOutputAssembly.h b/libyul/backends/evm/NoOutputAssembly.h new file mode 100644 index 000000000000..b07569439811 --- /dev/null +++ b/libyul/backends/evm/NoOutputAssembly.h @@ -0,0 +1,75 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ +/** + * Assembly interface that ignores everything. Can be used as a backend for a compilation dry-run. + */ + +#pragma once + +#include + +#include + +#include + +namespace langutil +{ +struct SourceLocation; +} + +namespace yul +{ + +/** + * Assembly class that just ignores everything and only performs stack counting. + * The purpose is to use this assembly for compilation dry-runs. + */ +class NoOutputAssembly: public AbstractAssembly +{ +public: + explicit NoOutputAssembly(bool _evm15 = false): m_evm15(_evm15) { } + virtual ~NoOutputAssembly() = default; + + void setSourceLocation(langutil::SourceLocation const&) override {} + int stackHeight() const override { return m_stackHeight; } + void appendInstruction(dev::solidity::Instruction _instruction) override; + void appendConstant(dev::u256 const& _constant) override; + void appendLabel(LabelID _labelId) override; + void appendLabelReference(LabelID _labelId) override; + LabelID newLabelId() override; + LabelID namedLabel(std::string const& _name) override; + void appendLinkerSymbol(std::string const& _name) override; + + void appendJump(int _stackDiffAfter) override; + void appendJumpTo(LabelID _labelId, int _stackDiffAfter) override; + void appendJumpToIf(LabelID _labelId) override; + void appendBeginsub(LabelID _labelId, int _arguments) override; + void appendJumpsub(LabelID _labelId, int _arguments, int _returns) override; + void appendReturnsub(int _returns, int _stackDiffAfter) override; + + void appendAssemblySize() override; + std::pair, SubID> createSubAssembly() override; + void appendDataOffset(SubID _sub) override; + void appendDataSize(SubID _sub) override; + SubID appendData(dev::bytes const& _data) override; + +private: + bool m_evm15 = false; ///< if true, switch to evm1.5 mode + int m_stackHeight = 0; +}; + +} From 4f641e3732ce1bfaa71778d1809d4b4dac8fe9a4 Mon Sep 17 00:00:00 2001 From: chriseth Date: Tue, 29 Jan 2019 15:01:30 +0100 Subject: [PATCH 048/109] Helper function for analysis. --- libyul/AsmAnalysis.cpp | 20 ++++++++++++++++++++ libyul/AsmAnalysis.h | 6 ++++++ 2 files changed, 26 insertions(+) diff --git a/libyul/AsmAnalysis.cpp b/libyul/AsmAnalysis.cpp index a5552c512314..62beb3d3fb5d 100644 --- a/libyul/AsmAnalysis.cpp +++ b/libyul/AsmAnalysis.cpp @@ -55,6 +55,26 @@ bool AsmAnalyzer::analyze(Block const& _block) return (*this)(_block); } +AsmAnalysisInfo AsmAnalyzer::analyzeStrictAssertCorrect( + shared_ptr _dialect, + EVMVersion _evmVersion, + Block const& _ast +) +{ + ErrorList errorList; + langutil::ErrorReporter errors(errorList); + yul::AsmAnalysisInfo analysisInfo; + bool success = yul::AsmAnalyzer( + analysisInfo, + errors, + _evmVersion, + Error::Type::SyntaxError, + _dialect + ).analyze(_ast); + solAssert(success && errorList.empty(), "Invalid assembly/yul code."); + return analysisInfo; +} + bool AsmAnalyzer::operator()(Label const& _label) { solAssert(!_label.name.empty(), ""); diff --git a/libyul/AsmAnalysis.h b/libyul/AsmAnalysis.h index 21cc114223a6..19b0558cc1ad 100644 --- a/libyul/AsmAnalysis.h +++ b/libyul/AsmAnalysis.h @@ -72,6 +72,12 @@ class AsmAnalyzer: public boost::static_visitor bool analyze(Block const& _block); + static AsmAnalysisInfo analyzeStrictAssertCorrect( + std::shared_ptr _dialect, + dev::solidity::EVMVersion _evmVersion, + Block const& _ast + ); + bool operator()(Instruction const&); bool operator()(Literal const& _literal); bool operator()(Identifier const&); From 77baf6caf7d615c78ec94c261bde802f6553847a Mon Sep 17 00:00:00 2001 From: chriseth Date: Tue, 29 Jan 2019 10:51:25 +0100 Subject: [PATCH 049/109] Compilability checker. --- libsolidity/codegen/AsmCodeGen.cpp | 23 +-- libyul/CMakeLists.txt | 2 + libyul/CompilabilityChecker.cpp | 64 +++++++ libyul/CompilabilityChecker.h | 45 +++++ libyul/backends/evm/EVMCodeTransform.cpp | 94 +++++++---- libyul/backends/evm/EVMCodeTransform.h | 19 ++- libyul/backends/evm/EVMObjectCompiler.cpp | 4 +- test/libyul/CompilabilityChecker.cpp | 196 ++++++++++++++++++++++ 8 files changed, 406 insertions(+), 41 deletions(-) create mode 100644 libyul/CompilabilityChecker.cpp create mode 100644 libyul/CompilabilityChecker.h create mode 100644 test/libyul/CompilabilityChecker.cpp diff --git a/libsolidity/codegen/AsmCodeGen.cpp b/libsolidity/codegen/AsmCodeGen.cpp index 02ae6e661583..fb39fea1ade1 100644 --- a/libsolidity/codegen/AsmCodeGen.cpp +++ b/libsolidity/codegen/AsmCodeGen.cpp @@ -184,18 +184,20 @@ void CodeGenerator::assemble( ) { EthAssemblyAdapter assemblyAdapter(_assembly); + shared_ptr dialect = EVMDialect::strictAssemblyForEVM(); + CodeTransform transform( + assemblyAdapter, + _analysisInfo, + _parsedData, + *dialect, + _optimize, + false, + _identifierAccess, + _useNamedLabelsForFunctions + ); try { - CodeTransform( - assemblyAdapter, - _analysisInfo, - _parsedData, - *EVMDialect::strictAssemblyForEVM(), - _optimize, - false, - _identifierAccess, - _useNamedLabelsForFunctions - )(_parsedData); + transform(_parsedData); } catch (StackTooDeepError const& _e) { @@ -205,4 +207,5 @@ void CodeGenerator::assemble( (_e.comment() ? ": " + *_e.comment() : ".") )); } + solAssert(transform.stackErrors().empty(), "Stack errors present but not thrown."); } diff --git a/libyul/CMakeLists.txt b/libyul/CMakeLists.txt index edae65552547..13cd0c37f957 100644 --- a/libyul/CMakeLists.txt +++ b/libyul/CMakeLists.txt @@ -13,6 +13,8 @@ add_library(yul AsmScope.h AsmScopeFiller.cpp AsmScopeFiller.h + CompilabilityChecker.cpp + CompilabilityChecker.h Dialect.cpp Dialect.h Exceptions.h diff --git a/libyul/CompilabilityChecker.cpp b/libyul/CompilabilityChecker.cpp new file mode 100644 index 000000000000..1a3977efd1da --- /dev/null +++ b/libyul/CompilabilityChecker.cpp @@ -0,0 +1,64 @@ +/*( + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ +/** + * Component that checks whether all variables are reachable on the stack. + */ + +#include + +#include +#include + +#include +#include + +#include + +using namespace std; +using namespace yul; +using namespace dev; +using namespace dev::solidity; + +std::map CompilabilityChecker::run(std::shared_ptr _dialect, Block const& _ast) +{ + if (_dialect->flavour == AsmFlavour::Yul) + return {}; + + solAssert(_dialect->flavour == AsmFlavour::Strict, ""); + + EVMDialect const& evmDialect = dynamic_cast(*_dialect); + + bool optimize = true; + yul::AsmAnalysisInfo analysisInfo = + yul::AsmAnalyzer::analyzeStrictAssertCorrect(_dialect, EVMVersion(), _ast); + NoOutputAssembly assembly; + CodeTransform transform(assembly, analysisInfo, _ast, evmDialect, optimize); + try + { + transform(_ast); + } + catch (StackTooDeepError const&) + { + solAssert(!transform.stackErrors().empty(), "Got stack too deep exception that was not stored."); + } + + std::map functions; + for (StackTooDeepError const& error: transform.stackErrors()) + functions[error.functionName] = max(error.depth, functions[error.functionName]); + + return functions; +} diff --git a/libyul/CompilabilityChecker.h b/libyul/CompilabilityChecker.h new file mode 100644 index 000000000000..80c91f73a454 --- /dev/null +++ b/libyul/CompilabilityChecker.h @@ -0,0 +1,45 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ +/** + * Component that checks whether all variables are reachable on the stack. + */ + +#pragma once + +#include +#include + +#include +#include + +namespace yul +{ + +/** + * Component that checks whether all variables are reachable on the stack and + * returns a mapping from function name to the largest stack difference found + * in that function (no entry present if that function is compilable). + * This only works properly if the outermost block is compilable and + * functions are not nested. Otherwise, it might miss reporting some functions. + */ +class CompilabilityChecker +{ +public: + static std::map run(std::shared_ptr _dialect, Block const& _ast); +}; + +} diff --git a/libyul/backends/evm/EVMCodeTransform.cpp b/libyul/backends/evm/EVMCodeTransform.cpp index d36540d46299..7b90912d26ba 100644 --- a/libyul/backends/evm/EVMCodeTransform.cpp +++ b/libyul/backends/evm/EVMCodeTransform.cpp @@ -227,6 +227,18 @@ void CodeTransform::operator()(VariableDeclaration const& _varDecl) checkStackHeight(&_varDecl); } +void CodeTransform::stackError(StackTooDeepError _error, int _targetStackHeight) +{ + m_assembly.appendInstruction(solidity::Instruction::INVALID); + // Correct the stack. + while (m_assembly.stackHeight() > _targetStackHeight) + m_assembly.appendInstruction(solidity::Instruction::POP); + while (m_assembly.stackHeight() < _targetStackHeight) + m_assembly.appendConstant(u256(0)); + // Store error. + m_stackErrors.emplace_back(std::move(_error)); +} + void CodeTransform::operator()(Assignment const& _assignment) { int height = m_assembly.stackHeight(); @@ -513,18 +525,32 @@ void CodeTransform::operator()(FunctionDefinition const& _function) m_assembly.appendConstant(u256(0)); } - CodeTransform( - m_assembly, - m_info, - _function.body, - m_allowStackOpt, - m_dialect, - m_evm15, - m_identifierAccess, - m_useNamedLabelsForFunctions, - localStackAdjustment, - m_context - )(_function.body); + try + { + CodeTransform( + m_assembly, + m_info, + _function.body, + m_allowStackOpt, + m_dialect, + m_evm15, + m_identifierAccess, + m_useNamedLabelsForFunctions, + localStackAdjustment, + m_context + )(_function.body); + } + catch (StackTooDeepError const& _error) + { + // This exception will be re-thrown after the end of the surrounding block. + // It enables us to see which functions compiled successfully and which did not. + // Even if we emit actual code, add an illegal instruction to make sure that tests + // will catch it. + StackTooDeepError error(_error); + if (error.functionName.empty()) + error.functionName = _function.name; + stackError(error, height); + } { // The stack layout here is: @@ -544,28 +570,34 @@ void CodeTransform::operator()(FunctionDefinition const& _function) stackLayout.push_back(i); // Move return values down, but keep order. if (stackLayout.size() > 17) - BOOST_THROW_EXCEPTION(StackTooDeepError() << errinfo_comment( + { + StackTooDeepError error(_function.name, YulString{}, stackLayout.size() - 17); + error << errinfo_comment( "The function " + _function.name.str() + " has " + to_string(stackLayout.size() - 17) + " parameters or return variables too many to fit the stack size." - )); - while (!stackLayout.empty() && stackLayout.back() != int(stackLayout.size() - 1)) - if (stackLayout.back() < 0) - { - m_assembly.appendInstruction(solidity::Instruction::POP); - stackLayout.pop_back(); - } - else - { - m_assembly.appendInstruction(swapInstruction(stackLayout.size() - stackLayout.back() - 1)); - swap(stackLayout[stackLayout.back()], stackLayout.back()); - } - for (int i = 0; size_t(i) < stackLayout.size(); ++i) - solAssert(i == stackLayout[i], "Error reshuffling stack."); + ); + stackError(error, m_assembly.stackHeight() - _function.parameters.size()); + } + else + { + while (!stackLayout.empty() && stackLayout.back() != int(stackLayout.size() - 1)) + if (stackLayout.back() < 0) + { + m_assembly.appendInstruction(solidity::Instruction::POP); + stackLayout.pop_back(); + } + else + { + m_assembly.appendInstruction(swapInstruction(stackLayout.size() - stackLayout.back() - 1)); + swap(stackLayout[stackLayout.back()], stackLayout.back()); + } + for (int i = 0; size_t(i) < stackLayout.size(); ++i) + solAssert(i == stackLayout[i], "Error reshuffling stack."); + } } - if (m_evm15) m_assembly.appendReturnsub(_function.returnVariables.size(), stackHeightBefore); else @@ -623,6 +655,9 @@ void CodeTransform::operator()(Block const& _block) finalizeBlock(_block, blockStartStackHeight); m_scope = originalScope; + + if (!m_stackErrors.empty()) + BOOST_THROW_EXCEPTION(m_stackErrors.front()); } AbstractAssembly::LabelID CodeTransform::labelFromIdentifier(Identifier const& _identifier) @@ -741,7 +776,8 @@ int CodeTransform::variableHeightDiff(Scope::Variable const& _var, YulString _va solAssert(heightDiff > (_forSwap ? 1 : 0), "Negative stack difference for variable."); int limit = _forSwap ? 17 : 16; if (heightDiff > limit) - BOOST_THROW_EXCEPTION(StackTooDeepError() << errinfo_comment( + // throw exception with variable name and height diff + BOOST_THROW_EXCEPTION(StackTooDeepError(_varName, heightDiff - limit) << errinfo_comment( "Variable " + _varName.str() + " is " + diff --git a/libyul/backends/evm/EVMCodeTransform.h b/libyul/backends/evm/EVMCodeTransform.h index 511fa73d757f..e5f083b8af08 100644 --- a/libyul/backends/evm/EVMCodeTransform.h +++ b/libyul/backends/evm/EVMCodeTransform.h @@ -40,7 +40,16 @@ namespace yul struct AsmAnalysisInfo; class EVMAssembly; -struct StackTooDeepError: virtual YulException {}; +struct StackTooDeepError: virtual YulException +{ + StackTooDeepError(YulString _variable, int _depth): variable(_variable), depth(_depth) {} + StackTooDeepError(YulString _functionName, YulString _variable, int _depth): + functionName(_functionName), variable(_variable), depth(_depth) + {} + YulString functionName; + YulString variable; + int depth; +}; struct CodeTransformContext { @@ -115,6 +124,8 @@ class CodeTransform: public boost::static_visitor<> { } + std::vector const& stackErrors() const { return m_stackErrors; } + protected: using Context = CodeTransformContext; @@ -184,6 +195,10 @@ class CodeTransform: public boost::static_visitor<> void checkStackHeight(void const* _astElement) const; + /// Stores the stack error in the list of errors, appends an invalid opcode + /// and corrects the stack height to the target stack height. + void stackError(StackTooDeepError _error, int _targetStackSize); + AbstractAssembly& m_assembly; AsmAnalysisInfo& m_info; Scope* m_scope = nullptr; @@ -204,6 +219,8 @@ class CodeTransform: public boost::static_visitor<> /// statement level in the scope where the variable was defined. std::set m_variablesScheduledForDeletion; std::set m_unusedStackSlots; + + std::vector m_stackErrors; }; } diff --git a/libyul/backends/evm/EVMObjectCompiler.cpp b/libyul/backends/evm/EVMObjectCompiler.cpp index dfeb7bb3ad12..20e822e47bc8 100644 --- a/libyul/backends/evm/EVMObjectCompiler.cpp +++ b/libyul/backends/evm/EVMObjectCompiler.cpp @@ -62,5 +62,7 @@ void EVMObjectCompiler::run(Object& _object, bool _optimize) yulAssert(_object.code, "No code."); // We do not catch and re-throw the stack too deep exception here because it is a YulException, // which should be native to this part of the code. - CodeTransform{m_assembly, *_object.analysisInfo, *_object.code, m_dialect, _optimize, m_evm15}(*_object.code); + CodeTransform transform{m_assembly, *_object.analysisInfo, *_object.code, m_dialect, _optimize, m_evm15}; + transform(*_object.code); + yulAssert(transform.stackErrors().empty(), "Stack errors present but not thrown."); } diff --git a/test/libyul/CompilabilityChecker.cpp b/test/libyul/CompilabilityChecker.cpp new file mode 100644 index 000000000000..1429762a6461 --- /dev/null +++ b/test/libyul/CompilabilityChecker.cpp @@ -0,0 +1,196 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ +/** + * Unit tests for the compilability checker. + */ + +#include + +#include +#include + +#include + + +using namespace std; + +namespace yul +{ +namespace test +{ + +namespace +{ +string check(string const& _input) +{ + shared_ptr ast = yul::test::parse(_input, false).first; + BOOST_REQUIRE(ast); + map functions = CompilabilityChecker::run(EVMDialect::strictAssemblyForEVM(), *ast); + string out; + for (auto const& function: functions) + out += function.first.str() + ": " + to_string(function.second) + " "; + return out; +} +} + +BOOST_AUTO_TEST_SUITE(CompilabilityChecker) + +BOOST_AUTO_TEST_CASE(smoke_test) +{ + string out = check("{}"); + BOOST_CHECK_EQUAL(out, ""); +} + +BOOST_AUTO_TEST_CASE(simple_function) +{ + string out = check("{ function f(a, b) -> x, y { x := a y := b } }"); + BOOST_CHECK_EQUAL(out, ""); +} + +BOOST_AUTO_TEST_CASE(many_variables_few_uses) +{ + string out = check(R"({ + function f(a, b) -> x, y { + let r1 := 0 + let r2 := 0 + let r3 := 0 + let r4 := 0 + let r5 := 0 + let r6 := 0 + let r7 := 0 + let r8 := 0 + let r9 := 0 + let r10 := 0 + let r11 := 0 + let r12 := 0 + let r13 := 0 + let r14 := 0 + let r15 := 0 + let r16 := 0 + let r17 := 0 + let r18 := 0 + x := add(add(add(add(add(add(add(add(add(x, r9), r8), r7), r6), r5), r4), r3), r2), r1) + } + })"); + BOOST_CHECK_EQUAL(out, "f: 4 "); +} + +BOOST_AUTO_TEST_CASE(many_variables_many_uses) +{ + string out = check(R"({ + function f(a, b) -> x, y { + let r1 := 0 + let r2 := 0 + let r3 := 0 + let r4 := 0 + let r5 := 0 + let r6 := 0 + let r7 := 0 + let r8 := 0 + let r9 := 0 + let r10 := 0 + let r11 := 0 + let r12 := 0 + let r13 := 0 + let r14 := 0 + let r15 := 0 + let r16 := 0 + let r17 := 0 + let r18 := 0 + x := add(add(add(add(add(add(add(add(add(add(add(add(x, r12), r11), r10), r9), r8), r7), r6), r5), r4), r3), r2), r1) + } + })"); + BOOST_CHECK_EQUAL(out, "f: 10 "); +} + +BOOST_AUTO_TEST_CASE(many_return_variables) +{ + string out = check(R"({ + function f(a, b) -> r1, r2, r3, r4, r5, r6, r7, r8, r9, r10, r11, r12, r13, r14, r15, r16, r17, r18, r19 { + } + })"); + BOOST_CHECK_EQUAL(out, "f: 5 "); +} + +BOOST_AUTO_TEST_CASE(multiple_functions) +{ + string out = check(R"({ + function f(a, b) -> r1, r2, r3, r4, r5, r6, r7, r8, r9, r10, r11, r12, r13, r14, r15, r16, r17, r18, r19 { + } + function g(r1, r2, r3, r4, r5, r6, r7, r8, r9, r10, r11, r12, r13, r14, r15, r16, r17, r18, r19) -> x, y { + } + function h(x) { + let r1 := 0 + let r2 := 0 + let r3 := 0 + let r4 := 0 + let r5 := 0 + let r6 := 0 + let r7 := 0 + let r8 := 0 + let r9 := 0 + let r10 := 0 + let r11 := 0 + let r12 := 0 + let r13 := 0 + let r14 := 0 + let r15 := 0 + let r16 := 0 + let r17 := 0 + let r18 := 0 + x := add(add(add(add(add(add(add(add(add(add(add(add(x, r12), r11), r10), r9), r8), r7), r6), r5), r4), r3), r2), r1) + } + })"); + BOOST_CHECK_EQUAL(out, "h: 9 g: 5 f: 5 "); +} + +BOOST_AUTO_TEST_CASE(nested) +{ + string out = check(R"({ + function h(x) { + let r1 := 0 + let r2 := 0 + let r3 := 0 + let r4 := 0 + let r5 := 0 + let r6 := 0 + let r7 := 0 + let r8 := 0 + let r9 := 0 + let r10 := 0 + let r11 := 0 + let r12 := 0 + let r13 := 0 + let r14 := 0 + let r15 := 0 + let r16 := 0 + let r17 := 0 + let r18 := 0 + function f(a, b) -> t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, t11, t12, t13, t14, t15, t16, t17, t18, t19 { + function g(s1, s2, s3, s4, s5, s6, s7, s8, s9, s10, s11, s12, s13, s14, s15, s16, s17, s18, s19) -> w, v { + } + } + x := add(add(add(add(add(add(add(add(add(add(add(add(x, r12), r11), r10), r9), r8), r7), r6), r5), r4), r3), r2), r1) + } + })"); + BOOST_CHECK_EQUAL(out, "h: 9 "); +} + +BOOST_AUTO_TEST_SUITE_END() + +} +} From 22c8d74a8a93f9c3a2e67d0eedd0cbf1f030d75f Mon Sep 17 00:00:00 2001 From: chriseth Date: Thu, 31 Jan 2019 15:06:20 +0100 Subject: [PATCH 050/109] Store all stack errors before they are thrown. --- libyul/backends/evm/EVMCodeTransform.cpp | 11 +++++---- libyul/backends/evm/EVMCodeTransform.h | 2 +- test/libyul/CompilabilityChecker.cpp | 29 ++++++++++++++++++++++++ 3 files changed, 37 insertions(+), 5 deletions(-) diff --git a/libyul/backends/evm/EVMCodeTransform.cpp b/libyul/backends/evm/EVMCodeTransform.cpp index 7b90912d26ba..354627d2b166 100644 --- a/libyul/backends/evm/EVMCodeTransform.cpp +++ b/libyul/backends/evm/EVMCodeTransform.cpp @@ -769,21 +769,24 @@ void CodeTransform::generateAssignment(Identifier const& _variableName) } } -int CodeTransform::variableHeightDiff(Scope::Variable const& _var, YulString _varName, bool _forSwap) const +int CodeTransform::variableHeightDiff(Scope::Variable const& _var, YulString _varName, bool _forSwap) { solAssert(m_context->variableStackHeights.count(&_var), ""); int heightDiff = m_assembly.stackHeight() - m_context->variableStackHeights[&_var]; solAssert(heightDiff > (_forSwap ? 1 : 0), "Negative stack difference for variable."); int limit = _forSwap ? 17 : 16; if (heightDiff > limit) - // throw exception with variable name and height diff - BOOST_THROW_EXCEPTION(StackTooDeepError(_varName, heightDiff - limit) << errinfo_comment( + { + m_stackErrors.emplace_back(_varName, heightDiff - limit); + m_stackErrors.back() << errinfo_comment( "Variable " + _varName.str() + " is " + to_string(heightDiff - limit) + " slot(s) too deep inside the stack." - )); + ); + BOOST_THROW_EXCEPTION(m_stackErrors.back()); + } return heightDiff; } diff --git a/libyul/backends/evm/EVMCodeTransform.h b/libyul/backends/evm/EVMCodeTransform.h index e5f083b8af08..3ebf0f490e8b 100644 --- a/libyul/backends/evm/EVMCodeTransform.h +++ b/libyul/backends/evm/EVMCodeTransform.h @@ -189,7 +189,7 @@ class CodeTransform: public boost::static_visitor<> /// Determines the stack height difference to the given variables. Throws /// if it is not yet in scope or the height difference is too large. Returns /// the (positive) stack height difference otherwise. - int variableHeightDiff(Scope::Variable const& _var, YulString _name, bool _forSwap) const; + int variableHeightDiff(Scope::Variable const& _var, YulString _name, bool _forSwap); void expectDeposit(int _deposit, int _oldHeight) const; diff --git a/test/libyul/CompilabilityChecker.cpp b/test/libyul/CompilabilityChecker.cpp index 1429762a6461..9bb31997e352 100644 --- a/test/libyul/CompilabilityChecker.cpp +++ b/test/libyul/CompilabilityChecker.cpp @@ -190,6 +190,35 @@ BOOST_AUTO_TEST_CASE(nested) BOOST_CHECK_EQUAL(out, "h: 9 "); } +BOOST_AUTO_TEST_CASE(also_in_outer_block) +{ + string out = check(R"({ + let x := 0 + let r1 := 0 + let r2 := 0 + let r3 := 0 + let r4 := 0 + let r5 := 0 + let r6 := 0 + let r7 := 0 + let r8 := 0 + let r9 := 0 + let r10 := 0 + let r11 := 0 + let r12 := 0 + let r13 := 0 + let r14 := 0 + let r15 := 0 + let r16 := 0 + let r17 := 0 + let r18 := 0 + x := add(add(add(add(add(add(add(add(add(add(add(add(x, r12), r11), r10), r9), r8), r7), r6), r5), r4), r3), r2), r1) + function g(s1, s2, s3, s4, s5, s6, s7, s8, s9, s10, s11, s12, s13, s14, s15, s16, s17, s18, s19) -> w, v { + } + })"); + BOOST_CHECK_EQUAL(out, ": 9 "); +} + BOOST_AUTO_TEST_SUITE_END() } From b145934fdb67078feeeb4cbecddea47d642aced6 Mon Sep 17 00:00:00 2001 From: Daniel Kirchner Date: Fri, 11 Jan 2019 17:47:06 +0100 Subject: [PATCH 051/109] Improve examples in standard json documentation. --- docs/using-the-compiler.rst | 128 ++++++++++++++++++------------------ 1 file changed, 63 insertions(+), 65 deletions(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index 80c06d500b79..d79df5c7eb98 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -140,9 +140,9 @@ Input Description { // Required: Source code language, such as "Solidity", "Vyper", "lll", "assembly", etc. - language: "Solidity", + "language": "Solidity", // Required - sources: + "sources": { // The keys here are the "global" names of the source files, // imports can use other files via remappings (see below). @@ -155,13 +155,16 @@ Input Description // URL(s) should be imported in this order and the result checked against the // keccak256 hash (if available). If the hash doesn't match or none of the // URL(s) result in success, an error should be raised. + // Using the commandline interface only filesystem paths are supported. + // With the JavaScript interface the URL will be passed to the user-supplied + // read callback, so any URL supported by the callback can be used. "urls": [ "bzzr://56ab...", "ipfs://Qma...", + "/tmp/path/to/file.sol" // If files are used, their directories should be added to the command line via // `--allow-paths `. - "file:///tmp/path/to/file.sol" ] }, "mortal": @@ -173,26 +176,26 @@ Input Description } }, // Optional - settings: + "settings": { // Optional: Sorted list of remappings - remappings: [ ":g/dir" ], + "remappings": [ ":g/dir" ], // Optional: Optimizer settings - optimizer: { + "optimizer": { // disabled by default - enabled: true, + "enabled": true, // Optimize for how many times you intend to run the code. // Lower values will optimize more for initial deployment cost, higher values will optimize more for high-frequency usage. - runs: 200 + "runs": 200 }, - evmVersion: "byzantium", // Version of the EVM to compile for. Affects type checking and code generation. Can be homestead, tangerineWhistle, spuriousDragon, byzantium or constantinople + "evmVersion": "byzantium", // Version of the EVM to compile for. Affects type checking and code generation. Can be homestead, tangerineWhistle, spuriousDragon, byzantium or constantinople // Metadata settings (optional) - metadata: { + "metadata": { // Use only literal content and not URLs (false by default) - useLiteralContent: true + "useLiteralContent": true }, // Addresses of the libraries. If not all libraries are given here, it can result in unlinked objects whose output data is different. - libraries: { + "libraries": { // The top level key is the the name of the source file where the library is used. // If remappings are used, this source file should match the global path after remappings were applied. // If this key is an empty string, that refers to a global level. @@ -240,22 +243,17 @@ Input Description // Note that using a using `evm`, `evm.bytecode`, `ewasm`, etc. will select every // target part of that output. Additionally, `*` can be used as a wildcard to request everything. // - outputSelection: { - // Enable the metadata and bytecode outputs of every single contract. + "outputSelection": { "*": { - "*": [ "metadata", "evm.bytecode" ] + "*": [ + "metadata", "evm.bytecode" // Enable the metadata and bytecode outputs of every single contract. + , "evm.bytecode.sourceMap" // Enable the source map output of every single contract. + , "ast" // Enable the AST output of every single file. + ] }, // Enable the abi and opcodes output of MyContract defined in file def. "def": { "MyContract": [ "abi", "evm.bytecode.opcodes" ] - }, - // Enable the source map output of every single contract. - "*": { - "*": [ "evm.bytecode.sourceMap" ] - }, - // Enable the legacy AST output of every single file. - "*": { - "": [ "legacyAST" ] } } } @@ -269,106 +267,106 @@ Output Description { // Optional: not present if no errors/warnings were encountered - errors: [ + "errors": [ { // Optional: Location within the source file. - sourceLocation: { - file: "sourceFile.sol", - start: 0, - end: 100 + "sourceLocation": { + "file": "sourceFile.sol", + "start": 0, + "end": 100 ], // Mandatory: Error type, such as "TypeError", "InternalCompilerError", "Exception", etc. // See below for complete list of types. - type: "TypeError", + "type": "TypeError", // Mandatory: Component where the error originated, such as "general", "ewasm", etc. - component: "general", + "component": "general", // Mandatory ("error" or "warning") - severity: "error", + "severity": "error", // Mandatory - message: "Invalid keyword" + "message": "Invalid keyword" // Optional: the message formatted with source location - formattedMessage: "sourceFile.sol:100: Invalid keyword" + "formattedMessage": "sourceFile.sol:100: Invalid keyword" } ], // This contains the file-level outputs. In can be limited/filtered by the outputSelection settings. - sources: { + "sources": { "sourceFile.sol": { // Identifier of the source (used in source maps) - id: 1, + "id": 1, // The AST object - ast: {}, + "ast": {}, // The legacy AST object - legacyAST: {} + "legacyAST": {} } }, // This contains the contract-level outputs. It can be limited/filtered by the outputSelection settings. - contracts: { + "contracts": { "sourceFile.sol": { // If the language used has no contract names, this field should equal to an empty string. "ContractName": { // The Ethereum Contract ABI. If empty, it is represented as an empty array. // See https://github.com/ethereum/wiki/wiki/Ethereum-Contract-ABI - abi: [], + "abi": [], // See the Metadata Output documentation (serialised JSON string) - metadata: "{...}", + "metadata": "{...}", // User documentation (natspec) - userdoc: {}, + "userdoc": {}, // Developer documentation (natspec) - devdoc: {}, + "devdoc": {}, // Intermediate representation (string) - ir: "", + "ir": "", // EVM-related outputs - evm: { + "evm": { // Assembly (string) - assembly: "", + "assembly": "", // Old-style assembly (object) - legacyAssembly: {}, + "legacyAssembly": {}, // Bytecode and related details. - bytecode: { + "bytecode": { // The bytecode as a hex string. - object: "00fe", + "object": "00fe", // Opcodes list (string) - opcodes: "", + "opcodes": "", // The source mapping as a string. See the source mapping definition. - sourceMap: "", + "sourceMap": "", // If given, this is an unlinked object. - linkReferences: { + "linkReferences": { "libraryFile.sol": { // Byte offsets into the bytecode. Linking replaces the 20 bytes located there. "Library1": [ - { start: 0, length: 20 }, - { start: 200, length: 20 } + { "start": 0, "length": 20 }, + { "start": 200, "length": 20 } ] } } }, // The same layout as above. - deployedBytecode: { }, + "deployedBytecode": { }, // The list of function hashes - methodIdentifiers: { + "methodIdentifiers": { "delegate(address)": "5c19a95c" }, // Function gas estimates - gasEstimates: { - creation: { - codeDepositCost: "420000", - executionCost: "infinite", - totalCost: "infinite" + "gasEstimates": { + "creation": { + "codeDepositCost": "420000", + "executionCost": "infinite", + "totalCost": "infinite" }, - external: { + "external": { "delegate(address)": "25000" }, - internal: { + "internal": { "heavyLifting()": "infinite" } } }, // eWASM related outputs - ewasm: { + "ewasm": { // S-expressions format - wast: "", + "wast": "", // Binary format (hex string) - wasm: "" + "wasm": "" } } } From 57463ca3a52877b0ba4a42040ee634a7e1324625 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 31 Jan 2019 13:19:51 +0000 Subject: [PATCH 052/109] Add C++17 toolchain --- cmake/toolchains/cxx17.cmake | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 cmake/toolchains/cxx17.cmake diff --git a/cmake/toolchains/cxx17.cmake b/cmake/toolchains/cxx17.cmake new file mode 100644 index 000000000000..04a865ebd6c5 --- /dev/null +++ b/cmake/toolchains/cxx17.cmake @@ -0,0 +1,4 @@ +# Require C++17. +set(CMAKE_CXX_STANDARD 17) +set(CMAKE_CXX_STANDARD_REQUIRED TRUE) +set(CMAKE_CXX_EXTENSIONS OFF) From 6adabf37d19705e26eed56289d742abfbaffa142 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Thu, 31 Jan 2019 13:22:02 +0000 Subject: [PATCH 053/109] Build using C++17 on CircleCI --- .circleci/config.yml | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/.circleci/config.yml b/.circleci/config.yml index 0f6821617722..6ff111c6ae63 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -66,6 +66,7 @@ jobs: paths: - soljson.js - version.txt + test_emscripten_solcjs: docker: - image: circleci/node:10 @@ -84,6 +85,7 @@ jobs: name: Test solcjs command: | test/solcjsTests.sh /tmp/workspace/soljson.js $(cat /tmp/workspace/version.txt) + test_emscripten_external: docker: - image: circleci/node:10 @@ -102,6 +104,7 @@ jobs: name: External tests command: | test/externalTests.sh /tmp/workspace/soljson.js || test/externalTests.sh /tmp/workspace/soljson.js + build_x86_linux: docker: - image: buildpack-deps:bionic @@ -124,6 +127,24 @@ jobs: paths: - "*" + build_x86_linux_cxx17: + docker: + - image: buildpack-deps:bionic + environment: + TERM: xterm + CMAKE_OPTIONS: -DCMAKE_TOOLCHAIN_FILE=cmake/toolchains/cxx17.cmake + steps: + - checkout + - run: + name: Install build dependencies + command: | + apt-get -qq update + # Note: do not include cvc4 here, it is not C++17 compatible. + apt-get -qy install cmake libboost-regex-dev libboost-filesystem-dev libboost-test-dev libboost-system-dev libboost-program-options-dev + ./scripts/install_obsolete_jsoncpp_1_7_4.sh + - run: *setup_prerelease_commit_hash + - run: *run_build + build_x86_archlinux: docker: - image: archlinux/base @@ -350,6 +371,7 @@ workflows: requires: - build_emscripten - build_x86_linux: *build_on_tags + - build_x86_linux_cxx17: *build_on_tags - build_x86_clang7: *build_on_tags - build_x86_mac: *build_on_tags - test_x86_linux: From f56ab1c2e76cc12dd00417219c1589d044c08ebf Mon Sep 17 00:00:00 2001 From: Daniel Kirchner Date: Mon, 4 Feb 2019 14:10:27 +0100 Subject: [PATCH 054/109] Enable cvc4 for C++17 build by switching to Ubuntu disco. --- .circleci/config.yml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 6ff111c6ae63..8b36a05eed51 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -129,7 +129,7 @@ jobs: build_x86_linux_cxx17: docker: - - image: buildpack-deps:bionic + - image: buildpack-deps:disco environment: TERM: xterm CMAKE_OPTIONS: -DCMAKE_TOOLCHAIN_FILE=cmake/toolchains/cxx17.cmake @@ -139,8 +139,7 @@ jobs: name: Install build dependencies command: | apt-get -qq update - # Note: do not include cvc4 here, it is not C++17 compatible. - apt-get -qy install cmake libboost-regex-dev libboost-filesystem-dev libboost-test-dev libboost-system-dev libboost-program-options-dev + apt-get -qy install cmake libboost-regex-dev libboost-filesystem-dev libboost-test-dev libboost-system-dev libboost-program-options-dev libcvc4-dev ./scripts/install_obsolete_jsoncpp_1_7_4.sh - run: *setup_prerelease_commit_hash - run: *run_build From 70748af981fabd74b893977ecb29618aa49c4b07 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 5 Feb 2019 14:23:59 +0000 Subject: [PATCH 055/109] Do not persist output for clang7 build step --- .circleci/config.yml | 5 ----- 1 file changed, 5 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 0f6821617722..2b6d99f5a23a 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -163,11 +163,6 @@ jobs: ./scripts/install_obsolete_jsoncpp_1_7_4.sh - run: *setup_prerelease_commit_hash - run: *run_build - - store_artifacts: *solc_artifact - - persist_to_workspace: - root: build - paths: - - "*" build_x86_mac: macos: From c4604c0e35ec914a9c81b2c85699e6235e13de33 Mon Sep 17 00:00:00 2001 From: Daniel Kirchner Date: Tue, 5 Feb 2019 16:33:55 +0100 Subject: [PATCH 056/109] Fix ast in output selection in standard json doc. --- docs/using-the-compiler.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/using-the-compiler.rst b/docs/using-the-compiler.rst index d79df5c7eb98..b5f79b66b8bf 100644 --- a/docs/using-the-compiler.rst +++ b/docs/using-the-compiler.rst @@ -248,7 +248,9 @@ Input Description "*": [ "metadata", "evm.bytecode" // Enable the metadata and bytecode outputs of every single contract. , "evm.bytecode.sourceMap" // Enable the source map output of every single contract. - , "ast" // Enable the AST output of every single file. + ], + "": [ + "ast" // Enable the AST output of every single file. ] }, // Enable the abi and opcodes output of MyContract defined in file def. From 4eb48dd6b7f0b358fc9dbf9c506a1f932d0ea554 Mon Sep 17 00:00:00 2001 From: chriseth Date: Thu, 24 Jan 2019 16:07:59 +0100 Subject: [PATCH 057/109] Specify packed encoding and add warning. --- docs/abi-spec.rst | 77 ++++++++++++++++++++++++----- docs/miscellaneous.rst | 2 +- docs/units-and-global-variables.rst | 2 +- 3 files changed, 68 insertions(+), 13 deletions(-) diff --git a/docs/abi-spec.rst b/docs/abi-spec.rst index 26293a1f047c..fff087b0f38a 100644 --- a/docs/abi-spec.rst +++ b/docs/abi-spec.rst @@ -418,10 +418,22 @@ In effect, a log entry using this ABI is described as: - ``address``: the address of the contract (intrinsically provided by Ethereum); - ``topics[0]``: ``keccak(EVENT_NAME+"("+EVENT_ARGS.map(canonical_type_of).join(",")+")")`` (``canonical_type_of`` is a function that simply returns the canonical type of a given argument, e.g. for ``uint indexed foo``, it would return ``uint256``). If the event is declared as ``anonymous`` the ``topics[0]`` is not generated; -- ``topics[n]``: ``EVENT_INDEXED_ARGS[n - 1]`` (``EVENT_INDEXED_ARGS`` is the series of ``EVENT_ARGS`` that are indexed); -- ``data``: ``abi_serialise(EVENT_NON_INDEXED_ARGS)`` (``EVENT_NON_INDEXED_ARGS`` is the series of ``EVENT_ARGS`` that are not indexed, ``abi_serialise`` is the ABI serialisation function used for returning a series of typed values from a function, as described above). - -For all fixed-length Solidity types, the ``EVENT_INDEXED_ARGS`` array contains the 32-byte encoded value directly. However, for *types of dynamic length*, which include ``string``, ``bytes``, and arrays, ``EVENT_INDEXED_ARGS`` will contain the *Keccak hash* of the packed encoded value (see :ref:`abi_packed_mode`), rather than the encoded value directly. This allows applications to efficiently query for values of dynamic-length types (by setting the hash of the encoded value as the topic), but leaves applications unable to decode indexed values they have not queried for. For dynamic-length types, application developers face a trade-off between fast search for predetermined values (if the argument is indexed) and legibility of arbitrary values (which requires that the arguments not be indexed). Developers may overcome this tradeoff and achieve both efficient search and arbitrary legibility by defining events with two arguments — one indexed, one not — intended to hold the same value. +- ``topics[n]``: ``abi_encode(EVENT_INDEXED_ARGS[n - 1])`` (``EVENT_INDEXED_ARGS`` is the series of ``EVENT_ARGS`` that are indexed); +- ``data``: ABI encoding of ``EVENT_NON_INDEXED_ARGS`` (``EVENT_NON_INDEXED_ARGS`` is the series of ``EVENT_ARGS`` that are not indexed, ``abi_encode`` is the ABI encoding function used for returning a series of typed values from a function, as described above). + +For all types of length at most 32 bytes, the ``EVENT_INDEXED_ARGS`` array contains +the value directly, padded or sign-extended (for signed integers) to 32 bytes, just as for regular ABI encoding. +However, for all "complex" types or types of dynamic length, including all arrays, ``string``, ``bytes`` and structs, +``EVENT_INDEXED_ARGS`` will contain the *Keccak hash* of a special in-place encoded value +(see :ref:`indexed_event_encoding`), rather than the encoded value directly. +This allows applications to efficiently query for values of dynamic-length types +(by setting the hash of the encoded value as the topic), but leaves applications unable +to decode indexed values they have not queried for. For dynamic-length types, +application developers face a trade-off between fast search for predetermined values +(if the argument is indexed) and legibility of arbitrary values (which requires that +the arguments not be indexed). Developers may overcome this tradeoff and achieve both +efficient search and arbitrary legibility by defining events with two arguments — one +indexed, one not — intended to hold the same value. .. _abi_json: @@ -608,8 +620,9 @@ Through ``abi.encodePacked()``, Solidity supports a non-standard packed mode whe - types shorter than 32 bytes are neither zero padded nor sign extended and - dynamic types are encoded in-place and without the length. +- array elements are padded, but still encoded in-place -This packed mode is mainly used for indexed event parameters. +Furthermore, structs as well as nested arrays are not supported. As an example, the encoding of ``int16(-1), bytes1(0x42), uint16(0x03), string("Hello, world!")`` results in: @@ -622,12 +635,18 @@ As an example, the encoding of ``int16(-1), bytes1(0x42), uint16(0x03), string(" ^^^^^^^^^^^^^^^^^^^^^^^^^^ string("Hello, world!") without a length field More specifically: - - Each value type takes as many bytes as its range has. - - The encoding of a struct or fixed-size array is the concatenation of the - encoding of its members/elements without any separator or padding. - - Mapping members of structs are ignored as usual. - - Dynamically-sized types like ``string``, ``bytes`` or ``uint[]`` are encoded without - their length field. + - During the encoding, everything is encoded in-place. This means that there is + no distinction between head and tail, as in the ABI encoding, and the length + of an array is not encoded. + - The direct arguments of ``abi.encodePacked`` are encoded without padding, + as long as they are not arrays (or ``string`` or ``bytes``). + - The encoding of an array is the concatenation of the + encoding of its elements **with** padding. + - Dynamically-sized types like ``string``, ``bytes`` or ``uint[]`` are encoded + without their length field. + - The encoding of ``string`` or ``bytes`` does not apply padding at the end + unless it is part of an array or struct (then it is padded to a multiple of + 32 bytes). In general, the encoding is ambiguous as soon as there are two dynamically-sized elements, because of the missing length field. @@ -636,3 +655,39 @@ If padding is needed, explicit type conversions can be used: ``abi.encodePacked( Since packed encoding is not used when calling functions, there is no special support for prepending a function selector. Since the encoding is ambiguous, there is no decoding function. + +.. warning:: + + If you use ``keccak256(abi.encodePacked(a, b))`` and both ``a`` and ``b`` are dynamic types, + it is easy to craft collisions in the hash value by moving parts of ``a`` into ``b`` and + vice-versa. More specifically, ``abi.encodePacked("a", "bc") == abi.encodePacked("ab", "c")``. + If you use ``abi.encodePacked`` for signatures, authentication or data integrity, make + sure to always use the same types and check that at most one of them is dynamic. + Unless there is a compelling reason, ``abi.encode`` should be preferred. + + +.. _indexed_event_encoding: + +Encoding of Indexed Event Parameters +==================================== + +Indexed event parameters that are not value types, i.e. arrays and structs are not +stored directly but instead a keccak256-hash of an encoding is stored. This encoding +is defined as follows: + + - the encoding of a ``bytes`` and ``string`` value is just the string contents + without any padding or length prefix. + - the encoding of a struct is the concatenation of the encoding of its members, + always padded to a multiple of 32 bytes (even ``bytes`` and ``string``). + - the encoding of an array (both dynamically- and statically-sized) is + the concatenation of the encoding of its elements, always padded to a multiple + of 32 bytes (even ``bytes`` and ``string``) and without any length prefix + +In the above, as usual, a negative number is padded by sign extension and not zero padded. +``bytesNN`` types are padded on the right while ``uintNN`` / ``intNN`` are padded on the left. + +.. warning:: + + The encoding of a struct is ambiguous if it contains more than one dynamically-sized + array. Because of that, always re-check the event data and do not rely on the search result + based on the indexed parameters alone. diff --git a/docs/miscellaneous.rst b/docs/miscellaneous.rst index 69124c777ece..55409678dda9 100644 --- a/docs/miscellaneous.rst +++ b/docs/miscellaneous.rst @@ -351,7 +351,7 @@ Global Variables - ``abi.decode(bytes memory encodedData, (...)) returns (...)``: :ref:`ABI `-decodes the provided data. The types are given in parentheses as second argument. Example: ``(uint a, uint[2] memory b, bytes memory c) = abi.decode(data, (uint, uint[2], bytes))`` - ``abi.encode(...) returns (bytes memory)``: :ref:`ABI `-encodes the given arguments -- ``abi.encodePacked(...) returns (bytes memory)``: Performs :ref:`packed encoding ` of the given arguments +- ``abi.encodePacked(...) returns (bytes memory)``: Performs :ref:`packed encoding ` of the given arguments. Note that this encoding can be ambiguous! - ``abi.encodeWithSelector(bytes4 selector, ...) returns (bytes memory)``: :ref:`ABI `-encodes the given arguments starting from the second and prepends the given four-byte selector - ``abi.encodeWithSignature(string memory signature, ...) returns (bytes memory)``: Equivalent to ``abi.encodeWithSelector(bytes4(keccak256(bytes(signature)), ...)``` diff --git a/docs/units-and-global-variables.rst b/docs/units-and-global-variables.rst index ce7706c197e6..690fd1b65d1e 100644 --- a/docs/units-and-global-variables.rst +++ b/docs/units-and-global-variables.rst @@ -117,7 +117,7 @@ ABI Encoding and Decoding Functions - ``abi.decode(bytes memory encodedData, (...)) returns (...)``: ABI-decodes the given data, while the types are given in parentheses as second argument. Example: ``(uint a, uint[2] memory b, bytes memory c) = abi.decode(data, (uint, uint[2], bytes))`` - ``abi.encode(...) returns (bytes memory)``: ABI-encodes the given arguments -- ``abi.encodePacked(...) returns (bytes memory)``: Performs :ref:`packed encoding ` of the given arguments +- ``abi.encodePacked(...) returns (bytes memory)``: Performs :ref:`packed encoding ` of the given arguments. Note that packed encoding can be ambiguous! - ``abi.encodeWithSelector(bytes4 selector, ...) returns (bytes memory)``: ABI-encodes the given arguments starting from the second and prepends the given four-byte selector - ``abi.encodeWithSignature(string memory signature, ...) returns (bytes memory)``: Equivalent to ``abi.encodeWithSelector(bytes4(keccak256(bytes(signature))), ...)``` From 779b9986ee2d0b2b7ad23f050ec587fbeae08fa9 Mon Sep 17 00:00:00 2001 From: Chris Ward Date: Wed, 23 Jan 2019 15:08:57 +0200 Subject: [PATCH 058/109] Clarify that public prefix only applies to state variables and explain strings. --- docs/types/reference-types.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/types/reference-types.rst b/docs/types/reference-types.rst index 99d977f6c155..c640ca22d777 100644 --- a/docs/types/reference-types.rst +++ b/docs/types/reference-types.rst @@ -106,7 +106,7 @@ Array elements can be of any type, including mapping or struct. The general restrictions for types apply, in that mappings can only be stored in the ``storage`` data location and publicly-visible functions need parameters that are :ref:`ABI types `. -It is possible to mark arrays ``public`` and have Solidity create a :ref:`getter `. +It is possible to mark state variable arrays ``public`` and have Solidity create a :ref:`getter `. The numeric index becomes a required parameter for the getter. Accessing an array past its end causes a failing assertion. You can use the ``.push()`` method to append a new element at the end or assign to the ``.length`` :ref:`member ` to change the size (see below for caveats). @@ -119,10 +119,9 @@ Variables of type ``bytes`` and ``string`` are special arrays. A ``bytes`` is si but it is packed tightly in calldata and memory. ``string`` is equal to ``bytes`` but does not allow length or index access. -While Solidity does not have string manipulation functions, you can use -this implicit conversion for equivalent functionality. For example to compare -two strings ``keccak256(abi.encode(s1)) == keccak256(abi.encode(s2))``, or to -concatenate two strings already encoded with ``abi.encodePacked(s1, s2);``. +Solidity does not have string manipulation functions, but there are +third-party string libraries. You can also compare two strings by their keccak256-hash using +``keccak256(abi.encodePacked(s1)) == keccak256(abi.encodePacked(s2))`` and concatenate two strings using ``abi.encodePacked(s1, s2)``. You should use ``bytes`` over ``byte[]`` because it is cheaper, since ``byte[]`` adds 31 padding bytes between the elements. As a general rule, use ``bytes`` for arbitrary-length raw byte data and ``string`` for arbitrary-length From 77f407d45038f2547222675d0d32806f56bcdcdc Mon Sep 17 00:00:00 2001 From: Mathias Baumann Date: Tue, 5 Feb 2019 15:51:19 +0100 Subject: [PATCH 059/109] Fix check that mappings can only have storage location --- Changelog.md | 1 + libsolidity/analysis/TypeChecker.cpp | 15 +++++++++++---- ...ping_data_location_function_param_external.sol | 4 ++-- ...ping_data_location_function_param_internal.sol | 4 +++- ...apping_data_location_function_param_public.sol | 2 +- .../mapping/mapping_return_public_memory.sol | 2 +- 6 files changed, 19 insertions(+), 9 deletions(-) diff --git a/Changelog.md b/Changelog.md index 86787337e0f8..763312b52bf5 100644 --- a/Changelog.md +++ b/Changelog.md @@ -3,6 +3,7 @@ Bugfixes: * Code generator: Defensively pad allocation of creationCode and runtimeCode to multiples of 32 bytes. * Parser: Disallow empty import statements. + * Type Checker: Dissallow mappings with data locations other than 'storage' * Type system: Properly report packed encoded size for arrays and structs (mostly unused until now). diff --git a/libsolidity/analysis/TypeChecker.cpp b/libsolidity/analysis/TypeChecker.cpp index 6d887c454552..b5b7d9152cd4 100644 --- a/libsolidity/analysis/TypeChecker.cpp +++ b/libsolidity/analysis/TypeChecker.cpp @@ -334,10 +334,17 @@ bool TypeChecker::visit(FunctionDefinition const& _function) auto checkArgumentAndReturnParameter = [&](VariableDeclaration const& var) { if (type(var)->category() == Type::Category::Mapping) { - if (!type(var)->dataStoredIn(DataLocation::Storage)) - m_errorReporter.typeError(var.location(), "Mapping types can only have a data location of \"storage\"." ); - else if (!isLibraryFunction && _function.isPublic()) - m_errorReporter.typeError(var.location(), "Mapping types for parameters or return variables can only be used in internal or library functions."); + if (var.referenceLocation() != VariableDeclaration::Location::Storage) + { + if (!isLibraryFunction && _function.isPublic()) + m_errorReporter.typeError(var.location(), "Mapping types can only have a data location of \"storage\" and thus only be parameters or return variables for internal or library functions."); + else + m_errorReporter.typeError(var.location(), "Mapping types can only have a data location of \"storage\"." ); + } + else + { + solAssert(isLibraryFunction || !_function.isPublic(), "Mapping types for parameters or return variables can only be used in internal or library functions."); + } } else { diff --git a/test/libsolidity/syntaxTests/types/mapping/mapping_data_location_function_param_external.sol b/test/libsolidity/syntaxTests/types/mapping/mapping_data_location_function_param_external.sol index fa3757fa6eb4..d4a7b4599ec0 100644 --- a/test/libsolidity/syntaxTests/types/mapping/mapping_data_location_function_param_external.sol +++ b/test/libsolidity/syntaxTests/types/mapping/mapping_data_location_function_param_external.sol @@ -2,5 +2,5 @@ contract c { function f1(mapping(uint => uint) calldata) pure external returns (mapping(uint => uint) memory) {} } // ---- -// TypeError: (29-59): Mapping types for parameters or return variables can only be used in internal or library functions. -// TypeError: (84-112): Mapping types for parameters or return variables can only be used in internal or library functions. +// TypeError: (29-59): Mapping types can only have a data location of "storage" and thus only be parameters or return variables for internal or library functions. +// TypeError: (84-112): Mapping types can only have a data location of "storage" and thus only be parameters or return variables for internal or library functions. diff --git a/test/libsolidity/syntaxTests/types/mapping/mapping_data_location_function_param_internal.sol b/test/libsolidity/syntaxTests/types/mapping/mapping_data_location_function_param_internal.sol index 17f2f7128d51..58c105e860d4 100644 --- a/test/libsolidity/syntaxTests/types/mapping/mapping_data_location_function_param_internal.sol +++ b/test/libsolidity/syntaxTests/types/mapping/mapping_data_location_function_param_internal.sol @@ -1,4 +1,6 @@ contract c { - function f4(mapping(uint => uint) memory) pure internal {} + function f4(mapping(uint => uint) storage) pure internal {} + function f5(mapping(uint => uint) memory) pure internal {} } // ---- +// TypeError: (93-121): Mapping types can only have a data location of "storage". diff --git a/test/libsolidity/syntaxTests/types/mapping/mapping_data_location_function_param_public.sol b/test/libsolidity/syntaxTests/types/mapping/mapping_data_location_function_param_public.sol index 8c73b5ae949a..04932d48a072 100644 --- a/test/libsolidity/syntaxTests/types/mapping/mapping_data_location_function_param_public.sol +++ b/test/libsolidity/syntaxTests/types/mapping/mapping_data_location_function_param_public.sol @@ -2,4 +2,4 @@ contract c { function f3(mapping(uint => uint) memory) view public {} } // ---- -// TypeError: (29-57): Mapping types for parameters or return variables can only be used in internal or library functions. +// TypeError: (29-57): Mapping types can only have a data location of "storage" and thus only be parameters or return variables for internal or library functions. diff --git a/test/libsolidity/syntaxTests/types/mapping/mapping_return_public_memory.sol b/test/libsolidity/syntaxTests/types/mapping/mapping_return_public_memory.sol index 7378324a7c33..cf0c674293e6 100644 --- a/test/libsolidity/syntaxTests/types/mapping/mapping_return_public_memory.sol +++ b/test/libsolidity/syntaxTests/types/mapping/mapping_return_public_memory.sol @@ -3,4 +3,4 @@ contract C { } } // ---- -// TypeError: (51-79): Mapping types for parameters or return variables can only be used in internal or library functions. +// TypeError: (51-79): Mapping types can only have a data location of "storage" and thus only be parameters or return variables for internal or library functions. From e9a0d9921881575c09ad9794aa94407bada2688b Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 5 Feb 2019 16:46:13 +0000 Subject: [PATCH 060/109] Add optimiser test for triggering CopyMethod in ConstantOpimiser --- test/libsolidity/SolidityOptimizer.cpp | 37 ++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/test/libsolidity/SolidityOptimizer.cpp b/test/libsolidity/SolidityOptimizer.cpp index b5ce6f2af276..9c6f8f65e4dc 100644 --- a/test/libsolidity/SolidityOptimizer.cpp +++ b/test/libsolidity/SolidityOptimizer.cpp @@ -630,6 +630,43 @@ BOOST_AUTO_TEST_CASE(optimise_multi_stores) BOOST_CHECK_EQUAL(numInstructions(m_optimizedBytecode, Instruction::SSTORE), 8); } +BOOST_AUTO_TEST_CASE(optimise_constant_to_codecopy) +{ + char const* sourceCode = R"( + contract C { + // We use the state variable so that the functions won't be deemed identical + // and be optimised out to the same implementation. + uint a; + function f() public returns (uint) { + a = 1; + // This cannot be represented well with the `CalculateMethod`, + // hence the decision will be between `LiteralMethod` and `CopyMethod`. + return 0x1234123412431234123412412342112341234124312341234124; + } + function g() public returns (uint) { + a = 2; + return 0x1234123412431234123412412342112341234124312341234124; + } + function h() public returns (uint) { + a = 3; + return 0x1234123412431234123412412342112341234124312341234124; + } + function i() public returns (uint) { + a = 4; + return 0x1234123412431234123412412342112341234124312341234124; + } + } + )"; + compileBothVersions(sourceCode, 0, "C", 50); + compareVersions("f()"); + compareVersions("g()"); + compareVersions("h()"); + compareVersions("i()"); + // This is counting in the deployed code. + BOOST_CHECK_EQUAL(numInstructions(m_nonOptimizedBytecode, Instruction::CODECOPY), 0); + BOOST_CHECK_EQUAL(numInstructions(m_optimizedBytecode, Instruction::CODECOPY), 4); +} + BOOST_AUTO_TEST_SUITE_END() } From 257dbf1f6a6a1c580ba6c82837d4aa5e0a434b33 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 5 Feb 2019 19:33:53 +0000 Subject: [PATCH 061/109] Add test cases for libsolc with callbacks --- test/libsolidity/LibSolc.cpp | 80 +++++++++++++++++++++++++++++++++++- 1 file changed, 78 insertions(+), 2 deletions(-) diff --git a/test/libsolidity/LibSolc.cpp b/test/libsolidity/LibSolc.cpp index 62ec32f76090..ea725f433bff 100644 --- a/test/libsolidity/LibSolc.cpp +++ b/test/libsolidity/LibSolc.cpp @@ -40,9 +40,28 @@ namespace test namespace { -Json::Value compile(string const& _input) +/// TODO: share this between StandardCompiler.cpp +/// Helper to match a specific error type and message +bool containsError(Json::Value const& _compilerResult, string const& _type, string const& _message) { - string output(solidity_compile(_input.c_str(), nullptr)); + if (!_compilerResult.isMember("errors")) + return false; + + for (auto const& error: _compilerResult["errors"]) + { + BOOST_REQUIRE(error.isObject()); + BOOST_REQUIRE(error["type"].isString()); + BOOST_REQUIRE(error["message"].isString()); + if ((error["type"].asString() == _type) && (error["message"].asString() == _message)) + return true; + } + + return false; +} + +Json::Value compile(string const& _input, CStyleReadFileCallback _callback = nullptr) +{ + string output(solidity_compile(_input.c_str(), _callback)); Json::Value ret; BOOST_REQUIRE(jsonParseStrict(output, ret)); solidity_free(); @@ -89,6 +108,63 @@ BOOST_AUTO_TEST_CASE(standard_compilation) BOOST_CHECK(!result.isMember("contracts")); } +BOOST_AUTO_TEST_CASE(missing_callback) +{ + char const* input = R"( + { + "language": "Solidity", + "sources": { + "fileA": { + "content": "import \"missing.sol\"; contract A { }" + } + } + } + )"; + Json::Value result = compile(input); + BOOST_CHECK(result.isObject()); + + BOOST_CHECK(containsError(result, "ParserError", "Source \"missing.sol\" not found: File not supplied initially.")); +} + +BOOST_AUTO_TEST_CASE(with_callback) +{ + char const* input = R"( + { + "language": "Solidity", + "sources": { + "fileA": { + "content": "import \"found.sol\"; contract A { }" + } + } + } + )"; + + CStyleReadFileCallback callback{ + [](char const* _path, char** o_contents, char** o_error) + { + // Caller frees the pointers. + if (string(_path) == "found.sol") + { + static string content{"import \"missing.sol\"; contract B {}"}; + *o_contents = strdup(content.c_str()); + *o_error = nullptr; + } + else + { + static string errorMsg{"Missing file."}; + *o_error = strdup(errorMsg.c_str()); + *o_contents = nullptr; + } + } + }; + + Json::Value result = compile(input, callback); + BOOST_CHECK(result.isObject()); + + // This ensures that "found.sol" was properly loaded which triggered the second import statement. + BOOST_CHECK(containsError(result, "ParserError", "Source \"missing.sol\" not found: Missing file.")); +} + BOOST_AUTO_TEST_SUITE_END() } From 6b11be8c6921d3b8948f5521f6daa70806cc9471 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Mon, 4 Feb 2019 23:21:11 +0000 Subject: [PATCH 062/109] Mark apropriate methods protected in ConstantOptimiser --- libevmasm/ConstantOptimiser.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/libevmasm/ConstantOptimiser.h b/libevmasm/ConstantOptimiser.h index 181f9351e009..bd87c4dcec14 100644 --- a/libevmasm/ConstantOptimiser.h +++ b/libevmasm/ConstantOptimiser.h @@ -56,6 +56,9 @@ class ConstantOptimisationMethod Assembly& _assembly ); +protected: + /// This is the public API for the optimiser methods, but it doesn't need to be exposed to the caller. + struct Params { bool isCreation; ///< Whether this is called during contract creation or runtime. From 3cdcd06b663c442e4031210a07d700c5b7b66237 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 5 Feb 2019 20:26:47 +0000 Subject: [PATCH 063/109] Add last test case for callbacks in libsolc --- test/libsolidity/LibSolc.cpp | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/test/libsolidity/LibSolc.cpp b/test/libsolidity/LibSolc.cpp index ea725f433bff..89a502100509 100644 --- a/test/libsolidity/LibSolc.cpp +++ b/test/libsolidity/LibSolc.cpp @@ -133,7 +133,7 @@ BOOST_AUTO_TEST_CASE(with_callback) "language": "Solidity", "sources": { "fileA": { - "content": "import \"found.sol\"; contract A { }" + "content": "import \"found.sol\"; import \"notfound.sol\"; contract A { }" } } } @@ -149,12 +149,17 @@ BOOST_AUTO_TEST_CASE(with_callback) *o_contents = strdup(content.c_str()); *o_error = nullptr; } - else + else if (string(_path) == "missing.sol") { static string errorMsg{"Missing file."}; *o_error = strdup(errorMsg.c_str()); *o_contents = nullptr; } + else + { + *o_error = nullptr; + *o_contents = nullptr; + } } }; @@ -163,6 +168,9 @@ BOOST_AUTO_TEST_CASE(with_callback) // This ensures that "found.sol" was properly loaded which triggered the second import statement. BOOST_CHECK(containsError(result, "ParserError", "Source \"missing.sol\" not found: Missing file.")); + + // This should be placed due to the missing "notfound.sol" which sets both pointers to null. + BOOST_CHECK(containsError(result, "ParserError", "Source \"notfound.sol\" not found: File not found.")); } BOOST_AUTO_TEST_SUITE_END() From be22032141c85bbec5c25f1e2a10c69d3e395d37 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 5 Feb 2019 22:04:35 +0000 Subject: [PATCH 064/109] Clean up some includes in tests --- test/Metadata.cpp | 1 + test/libsolidity/LibSolc.cpp | 3 --- test/libsolidity/StandardCompiler.cpp | 3 +-- 3 files changed, 2 insertions(+), 5 deletions(-) diff --git a/test/Metadata.cpp b/test/Metadata.cpp index c130d3462a69..615bf1951424 100644 --- a/test/Metadata.cpp +++ b/test/Metadata.cpp @@ -22,6 +22,7 @@ #include #include #include +#include using namespace std; diff --git a/test/libsolidity/LibSolc.cpp b/test/libsolidity/LibSolc.cpp index 89a502100509..b94486ac8294 100644 --- a/test/libsolidity/LibSolc.cpp +++ b/test/libsolidity/LibSolc.cpp @@ -25,9 +25,6 @@ #include #include -#include -#include - using namespace std; namespace dev diff --git a/test/libsolidity/StandardCompiler.cpp b/test/libsolidity/StandardCompiler.cpp index 118d8210c0aa..0cd3958603c1 100644 --- a/test/libsolidity/StandardCompiler.cpp +++ b/test/libsolidity/StandardCompiler.cpp @@ -23,8 +23,7 @@ #include #include #include - -#include "../Metadata.h" +#include using namespace std; using namespace dev::eth; From 26de5684a276c36f31daec11538ee5e703b22ef4 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 5 Feb 2019 22:37:02 +0000 Subject: [PATCH 065/109] Move bytecodeSansMetadata(bytes) helper to test/Metadata --- test/Metadata.cpp | 11 +++++++++++ test/Metadata.h | 4 ++++ test/libsolidity/SolidityOptimizer.cpp | 8 +++----- 3 files changed, 18 insertions(+), 5 deletions(-) diff --git a/test/Metadata.cpp b/test/Metadata.cpp index 615bf1951424..2269dd497aaa 100644 --- a/test/Metadata.cpp +++ b/test/Metadata.cpp @@ -31,6 +31,17 @@ namespace dev namespace test { +bytes bytecodeSansMetadata(bytes const& _bytecode) +{ + unsigned size = _bytecode.size(); + if (size < 5) + return bytes{}; + size_t metadataSize = (_bytecode[size - 2] << 8) + _bytecode[size - 1]; + if (metadataSize != 0x29 || size < (metadataSize + 2)) + return bytes{}; + return bytes(_bytecode.begin(), _bytecode.end() - metadataSize - 2); +} + string bytecodeSansMetadata(string const& _bytecode) { /// The metadata hash takes up 43 bytes (or 86 characters in hex) diff --git a/test/Metadata.h b/test/Metadata.h index cd92ecd80c42..fb945b1610d6 100644 --- a/test/Metadata.h +++ b/test/Metadata.h @@ -19,6 +19,7 @@ * Metadata processing helpers. */ +#include #include namespace dev @@ -26,6 +27,9 @@ namespace dev namespace test { +/// Returns the bytecode with the metadata hash stripped out. +bytes bytecodeSansMetadata(bytes const& _bytecode); + /// Returns the bytecode with the metadata hash stripped out. std::string bytecodeSansMetadata(std::string const& _bytecode); diff --git a/test/libsolidity/SolidityOptimizer.cpp b/test/libsolidity/SolidityOptimizer.cpp index 9c6f8f65e4dc..22ca9182f84c 100644 --- a/test/libsolidity/SolidityOptimizer.cpp +++ b/test/libsolidity/SolidityOptimizer.cpp @@ -20,6 +20,7 @@ * Tests for the Solidity optimizer. */ +#include #include #include @@ -105,11 +106,8 @@ class OptimizerTestFramework: public SolidityExecutionFramework /// into account. size_t numInstructions(bytes const& _bytecode, boost::optional _which = boost::optional{}) { - BOOST_REQUIRE(_bytecode.size() > 5); - size_t metadataSize = (_bytecode[_bytecode.size() - 2] << 8) + _bytecode[_bytecode.size() - 1]; - BOOST_REQUIRE_MESSAGE(metadataSize == 0x29, "Invalid metadata size"); - BOOST_REQUIRE(_bytecode.size() >= metadataSize + 2); - bytes realCode = bytes(_bytecode.begin(), _bytecode.end() - metadataSize - 2); + bytes realCode = bytecodeSansMetadata(_bytecode); + BOOST_REQUIRE_MESSAGE(!realCode.empty(), "Invalid or missing metadata in bytecode."); size_t instructions = 0; solidity::eachInstruction(realCode, [&](Instruction _instr, u256 const&) { if (!_which || *_which == _instr) From 43ccc75293a48e9f0c6cafb24379a7d3bf407214 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 5 Feb 2019 22:53:25 +0000 Subject: [PATCH 066/109] Remove duplicated code in test/Metadata --- test/Metadata.cpp | 14 +------------- test/Metadata.h | 1 + 2 files changed, 2 insertions(+), 13 deletions(-) diff --git a/test/Metadata.cpp b/test/Metadata.cpp index 2269dd497aaa..65d5b079aa50 100644 --- a/test/Metadata.cpp +++ b/test/Metadata.cpp @@ -44,19 +44,7 @@ bytes bytecodeSansMetadata(bytes const& _bytecode) string bytecodeSansMetadata(string const& _bytecode) { - /// The metadata hash takes up 43 bytes (or 86 characters in hex) - /// /a165627a7a72305820([0-9a-f]{64})0029$/ - - if (_bytecode.size() < 88) - return _bytecode; - - if (_bytecode.substr(_bytecode.size() - 4, 4) != "0029") - return _bytecode; - - if (_bytecode.substr(_bytecode.size() - 86, 18) != "a165627a7a72305820") - return _bytecode; - - return _bytecode.substr(0, _bytecode.size() - 86); + return toHex(bytecodeSansMetadata(fromHex(_bytecode, WhenError::Throw))); } bool isValidMetadata(string const& _metadata) diff --git a/test/Metadata.h b/test/Metadata.h index fb945b1610d6..113a54bee35c 100644 --- a/test/Metadata.h +++ b/test/Metadata.h @@ -31,6 +31,7 @@ namespace test bytes bytecodeSansMetadata(bytes const& _bytecode); /// Returns the bytecode with the metadata hash stripped out. +/// Throws exception on invalid hex string. std::string bytecodeSansMetadata(std::string const& _bytecode); /// Expects a serialised metadata JSON and returns true if the From 532c55acbdc90ea9c15c03c87f56bc116182315f Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Tue, 5 Feb 2019 22:57:25 +0000 Subject: [PATCH 067/109] Add better sanity check to test/Metadata --- test/Metadata.cpp | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/test/Metadata.cpp b/test/Metadata.cpp index 65d5b079aa50..41e98b486412 100644 --- a/test/Metadata.cpp +++ b/test/Metadata.cpp @@ -37,7 +37,11 @@ bytes bytecodeSansMetadata(bytes const& _bytecode) if (size < 5) return bytes{}; size_t metadataSize = (_bytecode[size - 2] << 8) + _bytecode[size - 1]; - if (metadataSize != 0x29 || size < (metadataSize + 2)) + if (size < (metadataSize + 2)) + return bytes{}; + // Sanity check: assume the first byte is a fixed-size CBOR array with either 1 or 2 entries + unsigned char firstByte = _bytecode[size - metadataSize - 2]; + if (firstByte != 0xa1 && firstByte != 0xa2) return bytes{}; return bytes(_bytecode.begin(), _bytecode.end() - metadataSize - 2); } From 9f431339efef7f7f19ecd53922f2b6be764cfee9 Mon Sep 17 00:00:00 2001 From: Mathias Baumann Date: Tue, 5 Feb 2019 13:32:01 +0100 Subject: [PATCH 068/109] Fix crash for too large struct array indicies --- Changelog.md | 1 + libsolidity/analysis/TypeChecker.cpp | 23 ++++++++++++------- .../struct_array_noninteger_index.sol | 10 ++++++++ 3 files changed, 26 insertions(+), 8 deletions(-) create mode 100644 test/libsolidity/syntaxTests/indexing/struct_array_noninteger_index.sol diff --git a/Changelog.md b/Changelog.md index 763312b52bf5..6de017511a10 100644 --- a/Changelog.md +++ b/Changelog.md @@ -4,6 +4,7 @@ Bugfixes: * Code generator: Defensively pad allocation of creationCode and runtimeCode to multiples of 32 bytes. * Parser: Disallow empty import statements. * Type Checker: Dissallow mappings with data locations other than 'storage' + * Type Checker: Fix internal error when a struct array index does not fit into a uint256. * Type system: Properly report packed encoded size for arrays and structs (mostly unused until now). diff --git a/libsolidity/analysis/TypeChecker.cpp b/libsolidity/analysis/TypeChecker.cpp index b5b7d9152cd4..225735bac8c4 100644 --- a/libsolidity/analysis/TypeChecker.cpp +++ b/libsolidity/analysis/TypeChecker.cpp @@ -2205,15 +2205,22 @@ bool TypeChecker::visit(IndexAccess const& _access) resultType = make_shared(make_shared(DataLocation::Memory, typeType.actualType())); else { - expectType(*index, IntegerType::uint256()); - if (auto length = dynamic_cast(type(*index).get())) - resultType = make_shared(make_shared( - DataLocation::Memory, - typeType.actualType(), - length->literalValue(nullptr) - )); + u256 length = 1; + if (expectType(*index, IntegerType::uint256())) + { + if (auto indexValue = dynamic_cast(type(*index).get())) + length = indexValue->literalValue(nullptr); + else + m_errorReporter.fatalTypeError(index->location(), "Integer constant expected."); + } else - m_errorReporter.fatalTypeError(index->location(), "Integer constant expected."); + solAssert(m_errorReporter.hasErrors(), "Expected errors as expectType returned false"); + + resultType = make_shared(make_shared( + DataLocation::Memory, + typeType.actualType(), + length + )); } break; } diff --git a/test/libsolidity/syntaxTests/indexing/struct_array_noninteger_index.sol b/test/libsolidity/syntaxTests/indexing/struct_array_noninteger_index.sol new file mode 100644 index 000000000000..59010709dc94 --- /dev/null +++ b/test/libsolidity/syntaxTests/indexing/struct_array_noninteger_index.sol @@ -0,0 +1,10 @@ +contract test { + struct s { uint a; uint b;} + function f() pure public returns (byte) { + s[75555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555555]; + s[7]; + } +} + +// ---- +// TypeError: (101-241): Type int_const 7555...(132 digits omitted)...5555 is not implicitly convertible to expected type uint256. From 49f8fa4cfede8f989a9b8d6a625b44d239f133ee Mon Sep 17 00:00:00 2001 From: chriseth Date: Wed, 6 Feb 2019 12:26:54 +0100 Subject: [PATCH 069/109] Extract storing length to its own function. --- libsolidity/codegen/ABIFunctions.cpp | 44 ++++++++++++++++++++-------- libsolidity/codegen/ABIFunctions.h | 5 ++++ 2 files changed, 36 insertions(+), 13 deletions(-) diff --git a/libsolidity/codegen/ABIFunctions.cpp b/libsolidity/codegen/ABIFunctions.cpp index 85d71cb79daa..2c043b277f2d 100644 --- a/libsolidity/codegen/ABIFunctions.cpp +++ b/libsolidity/codegen/ABIFunctions.cpp @@ -621,12 +621,12 @@ string ABIFunctions::abiEncodingFunctionCalldataArray( Whiskers templ(R"( // -> function (start, length, pos) -> end { - // might update pos + pos := (pos, length) (start, pos, length) end := add(pos, (length)) } )"); - templ("storeLength", _to.isDynamicallySized() ? "mstore(pos, length) pos := add(pos, 0x20)" : ""); + templ("storeLength", arrayStoreLengthForEncodingFunction(toArrayType)); templ("functionName", functionName); templ("readableTypeNameFrom", _from.toString(true)); templ("readableTypeNameTo", _to.toString(true)); @@ -665,7 +665,7 @@ string ABIFunctions::abiEncodingFunctionSimpleArray( // -> function (value, pos) { let length := (value) - // might update pos + pos := (pos, length) let headStart := pos let tail := add(pos, mul(length, 0x20)) let srcPtr := (value) @@ -684,7 +684,7 @@ string ABIFunctions::abiEncodingFunctionSimpleArray( // -> function (value, pos) { let length := (value) - // might update pos + pos := (pos, length) let srcPtr := (value) for { let i := 0 } lt(i, length) { i := add(i, 1) } { @@ -702,10 +702,7 @@ string ABIFunctions::abiEncodingFunctionSimpleArray( templ("return", dynamic ? " -> end " : ""); templ("assignEnd", dynamic ? "end := pos" : ""); templ("lengthFun", arrayLengthFunction(_from)); - if (_to.isDynamicallySized()) - templ("storeLength", "mstore(pos, length) pos := add(pos, 0x20)"); - else - templ("storeLength", ""); + templ("storeLength", arrayStoreLengthForEncodingFunction(_to)); templ("dataAreaFun", arrayDataAreaFunction(_from)); templ("elementEncodedSize", toCompactHexWithPrefix(_to.baseType()->calldataEncodedSize())); @@ -828,7 +825,7 @@ string ABIFunctions::abiEncodingFunctionCompactStorageArray( // -> function (value, pos) { let length := (value) - // might update pos + pos := (pos, length) let originalPos := pos let srcPtr := (value) for { let i := 0 } lt(i, length) { i := add(i, ) } @@ -851,10 +848,7 @@ string ABIFunctions::abiEncodingFunctionCompactStorageArray( templ("return", dynamic ? " -> end " : ""); templ("assignEnd", dynamic ? "end := pos" : ""); templ("lengthFun", arrayLengthFunction(_from)); - if (_to.isDynamicallySized()) - templ("storeLength", "mstore(pos, length) pos := add(pos, 0x20)"); - else - templ("storeLength", ""); + templ("storeLength", arrayStoreLengthForEncodingFunction(_to)); templ("dataArea", arrayDataAreaFunction(_from)); templ("itemsPerSlot", to_string(itemsPerSlot)); string elementEncodedSize = toCompactHexWithPrefix(_to.baseType()->calldataEncodedSize()); @@ -1686,6 +1680,30 @@ string ABIFunctions::nextArrayElementFunction(ArrayType const& _type) }); } +string ABIFunctions::arrayStoreLengthForEncodingFunction(ArrayType const& _type) +{ + string functionName = "array_storeLengthForEncoding_" + _type.identifier(); + return createFunction(functionName, [&]() { + if (_type.isDynamicallySized()) + return Whiskers(R"( + function (pos, length) -> updated_pos { + mstore(pos, length) + updated_pos := add(pos, 0x20) + } + )") + ("functionName", functionName) + .render(); + else + return Whiskers(R"( + function (pos, length) -> updated_pos { + updated_pos := pos + } + )") + ("functionName", functionName) + .render(); + }); +} + string ABIFunctions::allocationFunction() { string functionName = "allocateMemory"; diff --git a/libsolidity/codegen/ABIFunctions.h b/libsolidity/codegen/ABIFunctions.h index c54432365490..47e40527fbcd 100644 --- a/libsolidity/codegen/ABIFunctions.h +++ b/libsolidity/codegen/ABIFunctions.h @@ -231,6 +231,11 @@ class ABIFunctions /// Only works for memory arrays and storage arrays that store one item per slot. std::string nextArrayElementFunction(ArrayType const& _type); + /// @returns the name of a function used during encoding that stores the length + /// if the array is dynamically sized. It returns the new encoding position. + /// If the array is not dynamically sized, does nothing and just returns the position again. + std::string arrayStoreLengthForEncodingFunction(ArrayType const& _type); + /// @returns the name of a function that allocates memory. /// Modifies the "free memory pointer" /// Arguments: size From 5c50e8fa995d8a6e04f717389c08b7a5dfe1c577 Mon Sep 17 00:00:00 2001 From: chriseth Date: Wed, 6 Feb 2019 15:46:59 +0100 Subject: [PATCH 070/109] Switch from Z3 to CVC4 as SMT solver for Ubuntu PPA. --- Changelog.md | 1 + scripts/release_ppa.sh | 6 +++--- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Changelog.md b/Changelog.md index 6de017511a10..1ec6faf1e7e7 100644 --- a/Changelog.md +++ b/Changelog.md @@ -20,6 +20,7 @@ Bugfixes: Build System: * Add support for continuous fuzzing via Google oss-fuzz + * Ubuntu PPA Packages: Use CVC4 as SMT solver instead of Z3 ### 0.5.3 (2019-01-22) diff --git a/scripts/release_ppa.sh b/scripts/release_ppa.sh index fcdc8d640f7a..d65d96a1cd61 100755 --- a/scripts/release_ppa.sh +++ b/scripts/release_ppa.sh @@ -70,7 +70,7 @@ cd $distribution if [ $distribution = STATIC ] then pparepo=ethereum-static - Z3DEPENDENCY="" + SMTDEPENDENCY="" CMAKE_OPTIONS="-DSOLC_LINK_STATIC=On" else if [ "$branch" = develop ] @@ -79,7 +79,7 @@ else else pparepo=ethereum fi - Z3DEPENDENCY="libz3-dev, + SMTDEPENDENCY="libcvc4-dev, " CMAKE_OPTIONS="" fi @@ -127,7 +127,7 @@ Source: solc Section: science Priority: extra Maintainer: Christian (Buildserver key) -Build-Depends: ${Z3DEPENDENCY}debhelper (>= 9.0.0), +Build-Depends: ${SMTDEPENDENCY}debhelper (>= 9.0.0), cmake, g++, git, From f90c6f57bb1ace984461fffa58e830844041bf70 Mon Sep 17 00:00:00 2001 From: Erik Kundt Date: Thu, 24 Jan 2019 10:48:01 +0100 Subject: [PATCH 071/109] Implements a test file parser. --- Changelog.md | 1 + test/CMakeLists.txt | 3 + test/libsolidity/util/TestFileParser.cpp | 376 ++++++++++++++++++ test/libsolidity/util/TestFileParser.h | 277 +++++++++++++ test/libsolidity/util/TestFileParserTests.cpp | 205 ++++++++++ 5 files changed, 862 insertions(+) create mode 100644 test/libsolidity/util/TestFileParser.cpp create mode 100644 test/libsolidity/util/TestFileParser.h create mode 100644 test/libsolidity/util/TestFileParserTests.cpp diff --git a/Changelog.md b/Changelog.md index 1ec6faf1e7e7..2872799427c0 100644 --- a/Changelog.md +++ b/Changelog.md @@ -21,6 +21,7 @@ Bugfixes: Build System: * Add support for continuous fuzzing via Google oss-fuzz * Ubuntu PPA Packages: Use CVC4 as SMT solver instead of Z3 + * Soltest: Add parser that is used in the file-based unit test environment. ### 0.5.3 (2019-01-22) diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index 10b78bdcd72b..e231a6f977e8 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -17,6 +17,8 @@ if (LLL) endif() file(GLOB libsolidity_sources "libsolidity/*.cpp") file(GLOB libsolidity_headers "libsolidity/*.h") +file(GLOB libsolidity_util_sources "libsolidity/util/*.cpp") +file(GLOB libsolidity_util_headers "libsolidity/util/*.h") add_executable(soltest ${sources} ${headers} ${contracts_sources} ${contracts_headers} @@ -26,6 +28,7 @@ add_executable(soltest ${sources} ${headers} ${libyul_sources} ${libyul_headers} ${liblll_sources} ${liblll_headers} ${libsolidity_sources} ${libsolidity_headers} + ${libsolidity_util_sources} ${libsolidity_util_headers} ) target_link_libraries(soltest PRIVATE libsolc yul solidity evmasm devcore ${Boost_UNIT_TEST_FRAMEWORK_LIBRARIES}) diff --git a/test/libsolidity/util/TestFileParser.cpp b/test/libsolidity/util/TestFileParser.cpp new file mode 100644 index 000000000000..bd793df81959 --- /dev/null +++ b/test/libsolidity/util/TestFileParser.cpp @@ -0,0 +1,376 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +using namespace dev; +using namespace langutil; +using namespace solidity; +using namespace dev::solidity::test; +using namespace std; + +namespace +{ + bool isDecimalDigit(char c) + { + return '0' <= c && c <= '9'; + } + bool isWhiteSpace(char c) + { + return c == ' ' || c == '\n' || c == '\t' || c == '\r'; + } + bool isIdentifierStart(char c) + { + return c == '_' || c == '$' || ('a' <= c && c <= 'z') || ('A' <= c && c <= 'Z'); + } + bool isIdentifierPart(char c) + { + return isIdentifierStart(c) || isDecimalDigit(c); + } +} + +vector TestFileParser::parseFunctionCalls() +{ + vector calls; + if (!accept(SoltToken::EOS)) + { + // TODO: check initial token state + expect(SoltToken::Unknown); + while (!accept(SoltToken::EOS)) + { + if (!accept(SoltToken::Whitespace)) + { + FunctionCall call; + + // f() + expect(SoltToken::Newline); + call.signature = parseFunctionSignature(); + + // f(), 314 ether + if (accept(SoltToken::Comma, true)) + call.value = parseFunctionCallValue(); + + // f(), 314 ether: 1, 1 + if (accept(SoltToken::Colon, true)) + call.arguments = parseFunctionCallArguments(); + + string comment = m_scanner.currentLiteral(); + if (accept(SoltToken::Comment, true)) + call.arguments.comment = comment; + + // -> 1 + expect(SoltToken::Newline); + expect(SoltToken::Arrow); + if (m_scanner.peekToken() != SoltToken::Newline) + { + call.expectations = parseFunctionCallExpectations(); + + string comment = m_scanner.currentLiteral(); + if (accept(SoltToken::Comment, true)) + call.expectations.comment = comment; + } + calls.emplace_back(std::move(call)); + } + else + m_scanner.scanNextToken(); + } + } + return calls; +} + +string TestFileParser::formatToken(SoltToken _token) +{ + switch (_token) + { +#define T(name, string, precedence) case SoltToken::name: return string; + SOLT_TOKEN_LIST(T, T) +#undef T + default: // Token::NUM_TOKENS: + return ""; + } +} + +bool TestFileParser::accept(SoltToken _token, bool const _expect) +{ + if (m_scanner.currentToken() == _token) + { + if (_expect) + expect(_token); + return true; + } + return false; +} + +bool TestFileParser::expect(SoltToken _token, bool const _advance) +{ + if (m_scanner.currentToken() != _token) + throw Error(Error::Type::ParserError, + "Unexpected " + formatToken(m_scanner.currentToken()) + ": \"" + + m_scanner.currentLiteral() + "\". " + + "Expected \"" + formatToken(_token) + "\"." + ); + if (_advance) + m_scanner.scanNextToken(); + return true; +} + +string TestFileParser::parseFunctionSignature() +{ + string signature = m_scanner.currentLiteral(); + expect(SoltToken::Identifier); + + signature += formatToken(SoltToken::LParen); + expect(SoltToken::LParen); + + while (!accept(SoltToken::RParen)) + { + signature += m_scanner.currentLiteral(); + expect(SoltToken::UInt); + while (accept(SoltToken::Comma)) + { + signature += m_scanner.currentLiteral(); + expect(SoltToken::Comma); + signature += m_scanner.currentLiteral(); + expect(SoltToken::UInt); + } + } + signature += formatToken(SoltToken::RParen); + expect(SoltToken::RParen); + return signature; +} + +u256 TestFileParser::parseFunctionCallValue() +{ + u256 value; + string literal = m_scanner.currentLiteral(); + expect(SoltToken::Number); + value = convertNumber(literal); + expect(SoltToken::Ether); + return value; +} + +FunctionCallArgs TestFileParser::parseFunctionCallArguments() +{ + FunctionCallArgs arguments; + + auto formattedBytes = parseABITypeLiteral(); + arguments.rawBytes += formattedBytes.first; + arguments.formats.emplace_back(std::move(formattedBytes.second)); + while (accept(SoltToken::Comma, true)) + { + auto formattedBytes = parseABITypeLiteral(); + arguments.rawBytes += formattedBytes.first; + arguments.formats.emplace_back(std::move(formattedBytes.second)); + } + return arguments; +} + +FunctionCallExpectations TestFileParser::parseFunctionCallExpectations() +{ + FunctionCallExpectations expectations; + string token = m_scanner.currentLiteral(); + + if (accept(SoltToken::Failure, true)) + expectations.status = false; + else + { + auto formattedBytes = parseABITypeLiteral(); + expectations.rawBytes += formattedBytes.first; + expectations.formats.emplace_back(std::move(formattedBytes.second)); + + while (accept(SoltToken::Comma, true)) + { + auto formattedBytes = parseABITypeLiteral(); + expectations.rawBytes += formattedBytes.first; + expectations.formats.emplace_back(std::move(formattedBytes.second)); + } + } + return expectations; +} + +pair TestFileParser::parseABITypeLiteral() +{ + try + { + u256 number; + ABIType abiType; + if (accept(SoltToken::Sub)) + { + abiType.type = ABIType::Type::SignedDec; + abiType.size = 32; + + expect(SoltToken::Sub); + number = convertNumber(parseNumber()) * -1; + } + else + if (accept(SoltToken::Number)) + { + abiType.type = ABIType::Type::UnsignedDec; + abiType.size = 32; + + number = convertNumber(parseNumber()); + } + + return make_pair(toBigEndian(number), abiType); + } + catch (std::exception const&) + { + throw Error(Error::Type::ParserError, "Number encoding invalid."); + } +} + +string TestFileParser::parseNumber() +{ + string literal = m_scanner.currentLiteral(); + expect(SoltToken::Number); + return literal; +} + +u256 TestFileParser::convertNumber(string const& _literal) +{ + try { + return u256{_literal}; + } + catch (std::exception const&) + { + throw Error(Error::Type::ParserError, "Number encoding invalid."); + } +} + +void TestFileParser::Scanner::readStream(istream& _stream) +{ + std::string line; + while (std::getline(_stream, line)) + m_line += line; + m_char = m_line.begin(); +} + +void TestFileParser::Scanner::scanNextToken() +{ + auto detectToken = [](std::string const& _literal = "") -> TokenDesc { + if (_literal == "ether") return TokenDesc{SoltToken::Ether, _literal}; + if (_literal == "uint256") return TokenDesc{SoltToken::UInt, _literal}; + if (_literal == "FAILURE") return TokenDesc{SoltToken::Failure, _literal}; + return TokenDesc{SoltToken::Identifier, _literal}; + }; + + auto selectToken = [this](SoltToken _token, std::string const& _literal = "") -> TokenDesc { + advance(); + return make_pair(_token, !_literal.empty() ? _literal : formatToken(_token)); + }; + + TokenDesc token = make_pair(SoltToken::Unknown, ""); + do { + switch(current()) + { + case '/': + advance(); + if (current() == '/') + token = selectToken(SoltToken::Newline); + break; + case '-': + if (peek() == '>') + { + advance(); + token = selectToken(SoltToken::Arrow); + } + else + token = selectToken(SoltToken::Sub); + break; + case ':': + token = selectToken(SoltToken::Colon); + break; + case '#': + token = selectToken(SoltToken::Comment, scanComment()); + break; + case ',': + token = selectToken(SoltToken::Comma); + break; + case '(': + token = selectToken(SoltToken::LParen); + break; + case ')': + token = selectToken(SoltToken::RParen); + break; + default: + if (isIdentifierStart(current())) + { + TokenDesc detectedToken = detectToken(scanIdentifierOrKeyword()); + token = selectToken(detectedToken.first, detectedToken.second); + } + else if (isDecimalDigit(current())) + token = selectToken(SoltToken::Number, scanNumber()); + else if (isWhiteSpace(current())) + token = selectToken(SoltToken::Whitespace); + else if (isEndOfLine()) + token = selectToken(SoltToken::EOS); + else + token = selectToken(SoltToken::Invalid); + break; + } + } + while (token.first == SoltToken::Whitespace); + + m_nextToken = token; + m_currentToken = token; +} + +string TestFileParser::Scanner::scanComment() +{ + string comment; + advance(); + + while (current() != '#') + { + comment += current(); + advance(); + } + return comment; +} + +string TestFileParser::Scanner::scanIdentifierOrKeyword() +{ + string identifier; + identifier += current(); + while (isIdentifierPart(peek())) + { + advance(); + identifier += current(); + } + return identifier; +} + +string TestFileParser::Scanner::scanNumber() +{ + string number; + number += current(); + while (isDecimalDigit(peek())) + { + advance(); + number += current(); + } + return number; +} diff --git a/test/libsolidity/util/TestFileParser.h b/test/libsolidity/util/TestFileParser.h new file mode 100644 index 000000000000..83e2c08b3821 --- /dev/null +++ b/test/libsolidity/util/TestFileParser.h @@ -0,0 +1,277 @@ +/* + This file is part of solidity. + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ + +#pragma once + +#include +#include +#include + +#include +#include +#include +#include +#include + +namespace dev +{ +namespace solidity +{ +namespace test +{ + +/** + * All SOLT (or SOLTest) tokens. + */ +#define SOLT_TOKEN_LIST(T, K) \ + T(Unknown, "unknown", 0) \ + T(Invalid, "invalid", 0) \ + T(EOS, "EOS", 0) \ + T(Whitespace, "_", 0) \ + /* punctuations */ \ + T(LParen, "(", 0) \ + T(RParen, ")", 0) \ + T(LBrack, "[", 0) \ + T(RBrack, "]", 0) \ + T(LBrace, "{", 0) \ + T(RBrace, "}", 0) \ + T(Sub, "-", 0) \ + T(Colon, ":", 0) \ + T(Comma, ",", 0) \ + T(Period, ".", 0) \ + T(Arrow, "->", 0) \ + T(Newline, "//", 0) \ + /* Literals & identifier */ \ + T(Comment, "comment", 0) \ + T(Number, "number", 0) \ + T(Identifier, "identifier", 0) \ + /* type keywords */ \ + K(Ether, "ether", 0) \ + K(UInt, "uint256", 0) \ + /* special keywords */ \ + K(Failure, "FAILURE", 0) \ + + +enum class SoltToken : unsigned int { +#define T(name, string, precedence) name, + SOLT_TOKEN_LIST(T, T) + NUM_TOKENS +#undef T +}; + +/** + * The purpose of the ABI type is the storage of type information + * retrieved while parsing a test. This information is used + * for the conversion of human-readable function arguments and + * return values to `bytes` and vice-versa. + * Defaults to an invalid 0-byte representation. + */ +struct ABIType +{ + enum Type { + UnsignedDec, + SignedDec, + Invalid + }; + ABIType(): type(ABIType::Invalid), size(0) { } + ABIType(Type _type, size_t _size): type(_type), size(_size) { } + + Type type; + size_t size; +}; + +using ABITypeList = std::vector; + +/** + * Represents the expected result of a function call after it has been executed. This may be a single + * return value or a comma-separated list of return values. It also contains the detected input + * formats used to convert the values to `bytes` needed for the comparison with the actual result + * of a call. In addition to that, it also stores the expected transaction status. + * An optional comment can be assigned. + */ +struct FunctionCallExpectations +{ + /// ABI encoded `bytes` of parsed expectations. This `bytes` + /// is compared to the actual result of a function call + /// and is taken into account while validating it. + bytes rawBytes; + /// Types that were used to encode `rawBytes`. Expectations + /// are usually comma seperated literals. Their type is auto- + /// detected and retained in order to format them later on. + ABITypeList formats; + /// Expected status of the transaction. It can be either + /// a REVERT or a different EVM failure (e.g. out-of-gas). + bool status = true; + /// A Comment that can be attached to the expectations, + /// that is retained and can be displayed. + std::string comment; +}; + +/** + * Represents the arguments passed to a function call. This can be a single + * argument or a comma-separated list of arguments. It also contains the detected input + * formats used to convert the arguments to `bytes` needed for the call. + * An optional comment can be assigned. + */ +struct FunctionCallArgs +{ + /// ABI encoded `bytes` of parsed parameters. This `bytes` + /// passed to the function call. + bytes rawBytes; + /// Types that were used to encode `rawBytes`. Parameters + /// are usually comma seperated literals. Their type is auto- + /// detected and retained in order to format them later on. + ABITypeList formats; + /// A Comment that can be attached to the expectations, + /// that is retained and can be displayed. + std::string comment; +}; + +/** + * Represents a function call read from an input stream. It contains the signature, the + * arguments, an optional ether value and an expected execution result. + */ +struct FunctionCall +{ + /// Signature of the function call, e.g. `f(uint256, uint256)`. + std::string signature; + /// Optional `ether` value that can be send with the call. + u256 value; + /// Object that holds all function parameters in their `bytes` + /// representations given by the contract ABI. + FunctionCallArgs arguments; + /// Object that holds all function call expectation in + /// their `bytes` representations given by the contract ABI. + /// They are checked against the actual results and their + /// `bytes` representation, as well as the transaction status. + FunctionCallExpectations expectations; +}; + +/** + * Class that is able to parse an additional and well-formed comment section in a Solidity + * source file used by the file-based unit test environment. For now, it parses function + * calls and their expected result after the call was made. + * + * - Function calls defined in blocks: + * // f(uint256, uint256): 1, 1 # Signature and comma-separated list of arguments # + * // -> 1, 1 # Expected result value # + * // g(), 2 ether # (Optional) Ether to be send with the call # + * // -> 2, 3 + * // h(uint256), 1 ether: 42 + * // -> FAILURE # If REVERT or other EVM failure was detected # + * ... + */ +class TestFileParser +{ +public: + /// Constructor that takes an input stream \param _stream to operate on + /// and creates the internal scanner. + TestFileParser(std::istream& _stream): m_scanner(_stream) {} + + /// Parses function calls blockwise and returns a list of function calls found. + /// Throws an exception if a function call cannot be parsed because of its + /// incorrect structure, an invalid or unsupported encoding + /// of its arguments or expected results. + std::vector parseFunctionCalls(); + + /// Prints a friendly string representation of \param _token. + static std::string formatToken(SoltToken _token); + +private: + /** + * Token scanner that is used internally to abstract away character traversal. + */ + class Scanner + { + public: + /// Constructor that takes an input stream \param _stream to operate on. + Scanner(std::istream& _stream) { readStream(_stream); } + + /// Reads input stream into a single line and resets the current iterator. + void readStream(std::istream& _stream); + + /// Reads character stream and creates token. + void scanNextToken(); + + SoltToken currentToken() { return m_currentToken.first; } + SoltToken peekToken() { return m_nextToken.first; } + + std::string currentLiteral() { return m_currentToken.second; } + + std::string scanComment(); + std::string scanIdentifierOrKeyword(); + std::string scanNumber(); + + private: + using TokenDesc = std::pair; + + /// Advances current position in the input stream. + void advance() { ++m_char; } + /// Returns the current character. + char current() const { return *m_char; } + /// Peeks the next character. + char peek() const { auto it = m_char; return *(it + 1); } + /// Returns true if the end of a line is reached, false otherwise. + bool isEndOfLine() const { return m_char == m_line.end(); } + + std::string m_line; + std::string::iterator m_char; + + std::string m_currentLiteral; + + TokenDesc m_currentToken; + TokenDesc m_nextToken; + }; + + bool accept(SoltToken _token, bool const _expect = false); + bool expect(SoltToken _token, bool const _advance = true); + + /// Parses a function call signature in the form of f(uint256, ...). + std::string parseFunctionSignature(); + + /// Parses the optional ether value that can be passed alongside the + /// function call arguments. Throws an InvalidEtherValueEncoding exception + /// if given value cannot be converted to `u256`. + u256 parseFunctionCallValue(); + + /// Parses a comma-separated list of arguments passed with a function call. + /// Does not check for a potential mismatch between the signature and the number + /// or types of arguments. + FunctionCallArgs parseFunctionCallArguments(); + + /// Parses the expected result of a function call execution. + FunctionCallExpectations parseFunctionCallExpectations(); + + /// Parses and converts the current literal to its byte representation and + /// preserves the chosen ABI type. Based on that type information, the driver of + /// this parser can format arguments, expectations and results. Supported types: + /// - unsigned and signed decimal number literals + /// Throws a ParserError if data is encoded incorrectly or + /// if data type is not supported. + std::pair parseABITypeLiteral(); + + /// Parses the current number literal. + std::string parseNumber(); + + /// Tries to convert \param _literal to `uint256` and throws if + /// if conversion failed. + u256 convertNumber(std::string const& _literal); + + /// A scanner instance + Scanner m_scanner; +}; + +} +} +} diff --git a/test/libsolidity/util/TestFileParserTests.cpp b/test/libsolidity/util/TestFileParserTests.cpp new file mode 100644 index 000000000000..88d4d4c12606 --- /dev/null +++ b/test/libsolidity/util/TestFileParserTests.cpp @@ -0,0 +1,205 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ +/** + * Unit tests for Solidity's test expectation parser. + */ + +#include +#include +#include +#include +#include +#include + +#include + +using namespace std; +using namespace dev::test; + +namespace dev +{ +namespace solidity +{ +namespace test +{ + +vector parse(string const& _source) +{ + istringstream stream{_source, ios_base::out}; + TestFileParser parser{stream}; + return parser.parseFunctionCalls(); +} + +BOOST_AUTO_TEST_SUITE(TestFileParserTest) + +BOOST_AUTO_TEST_CASE(smoke_test) +{ + char const* source = R"()"; + BOOST_CHECK_EQUAL(parse(source).size(), 0); +} + +BOOST_AUTO_TEST_CASE(simple_call_succees) +{ + char const* source = R"( + // f(uint256, uint256): 1, 1 + // -> + )"; + + auto const calls = parse(source); + BOOST_CHECK_EQUAL(calls.size(), 1); + + auto call = calls.at(0); + ABI_CHECK(call.arguments.rawBytes, toBigEndian(u256{1}) + toBigEndian(u256{1})); + BOOST_CHECK_EQUAL(call.signature, "f(uint256,uint256)"); +} + +BOOST_AUTO_TEST_CASE(non_existent_call_revert) +{ + char const* source = R"( + // i_am_not_there() + // -> FAILURE + )"; + auto const calls = parse(source); + BOOST_CHECK_EQUAL(calls.size(), 1); + + auto const& call = calls.at(0); + BOOST_CHECK_EQUAL(call.signature, "i_am_not_there()"); + BOOST_CHECK_EQUAL(call.expectations.status, false); +} + +BOOST_AUTO_TEST_CASE(call_comments) +{ + char const* source = R"( + // f() # This is a comment # + // -> 1 # This is another comment # + )"; + auto const calls = parse(source); + BOOST_CHECK_EQUAL(calls.size(), 1); + + auto const& call = calls.at(0); + BOOST_CHECK_EQUAL(call.signature, "f()"); + BOOST_CHECK_EQUAL(call.arguments.comment, " This is a comment "); + BOOST_CHECK_EQUAL(call.expectations.comment, " This is another comment "); + ABI_CHECK(call.expectations.rawBytes, toBigEndian(u256{1})); +} + +BOOST_AUTO_TEST_CASE(call_arguments) +{ + char const* source = R"( + // f(uint256), 314 ether: 5 # optional ether value # + // -> 4 + )"; + auto const calls = parse(source); + BOOST_CHECK_EQUAL(calls.size(), 1); + + auto const& call = calls.at(0); + BOOST_CHECK_EQUAL(call.signature, "f(uint256)"); + BOOST_CHECK_EQUAL(call.value, u256{314}); + ABI_CHECK(call.arguments.rawBytes, toBigEndian(u256{5})); + ABI_CHECK(call.expectations.rawBytes, toBigEndian(u256{4})); +} + +BOOST_AUTO_TEST_CASE(call_expectations_missing) +{ + char const* source = R"( + // f())"; + BOOST_CHECK_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_ether_value_expectations_missing) +{ + char const* source = R"( + // f(), 0)"; + BOOST_CHECK_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_arguments_invalid) +{ + char const* source = R"( + // f(uint256): abc + // -> 1 + )"; + BOOST_CHECK_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_ether_value_invalid) +{ + char const* source = R"( + // f(uint256), abc : 1 + // -> 1 + )"; + BOOST_CHECK_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_ether_type_invalid) +{ + char const* source = R"( + // f(uint256), 2 btc : 1 + // -> 1 + )"; + BOOST_CHECK_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_arguments_mismatch) +{ + char const* source = R"( + // f(uint256, uint256): 1 # This only throws at runtime # + // -> 1 + )"; + auto const calls = parse(source); + BOOST_CHECK_EQUAL(calls.size(), 1); + + auto const& call = calls.at(0); + BOOST_CHECK_EQUAL(call.signature, "f(uint256,uint256)"); + ABI_CHECK(call.arguments.rawBytes, toBigEndian(u256{1})); +} + +BOOST_AUTO_TEST_CASE(call_multiple_arguments) +{ + char const* source = R"( + // test(uint256, uint256): 1, 2 + // -> 1, 1 + )"; + auto const calls = parse(source); + BOOST_CHECK_EQUAL(calls.size(), 1); + + auto const& call = calls.at(0); + BOOST_CHECK_EQUAL(call.signature, "test(uint256,uint256)"); + ABI_CHECK(call.arguments.rawBytes, toBigEndian(u256{1}) + toBigEndian(u256{2})); +} + +BOOST_AUTO_TEST_CASE(call_multiple_arguments_mixed_format) +{ + char const* source = R"( + // test(uint256, uint256),314 ether: 1, -2 + // -> -1, 2 + )"; + auto const calls = parse(source); + BOOST_CHECK_EQUAL(calls.size(), 1); + + auto const& call = calls.at(0); + BOOST_CHECK_EQUAL(call.signature, "test(uint256,uint256)"); + BOOST_CHECK_EQUAL(call.value, u256{314}); + ABI_CHECK(call.arguments.rawBytes, toBigEndian(u256{1}) + toBigEndian(u256{-2})); + ABI_CHECK(call.expectations.rawBytes, toBigEndian(u256{-1}) + toBigEndian(u256{2})); +} + +BOOST_AUTO_TEST_SUITE_END() + +} +} +} From 7fa167977bd0e349becf93db8430e60541f0c807 Mon Sep 17 00:00:00 2001 From: Erik Kundt Date: Fri, 1 Feb 2019 06:29:34 +0100 Subject: [PATCH 072/109] Adds multi-line support for test file parser. --- test/libsolidity/util/TestFileParser.cpp | 121 ++++++------ test/libsolidity/util/TestFileParser.h | 111 ++++++++--- test/libsolidity/util/TestFileParserTests.cpp | 173 +++++++++++++++--- 3 files changed, 300 insertions(+), 105 deletions(-) diff --git a/test/libsolidity/util/TestFileParser.cpp b/test/libsolidity/util/TestFileParser.cpp index bd793df81959..40ea95b0df1f 100644 --- a/test/libsolidity/util/TestFileParser.cpp +++ b/test/libsolidity/util/TestFileParser.cpp @@ -65,33 +65,27 @@ vector TestFileParser::parseFunctionCalls() { FunctionCall call; - // f() expect(SoltToken::Newline); call.signature = parseFunctionSignature(); - // f(), 314 ether if (accept(SoltToken::Comma, true)) call.value = parseFunctionCallValue(); - - // f(), 314 ether: 1, 1 if (accept(SoltToken::Colon, true)) call.arguments = parseFunctionCallArguments(); - string comment = m_scanner.currentLiteral(); - if (accept(SoltToken::Comment, true)) - call.arguments.comment = comment; + call.displayMode = parseNewline(); + call.arguments.comment = parseComment(); - // -> 1 - expect(SoltToken::Newline); + if (accept(SoltToken::Newline, true)) + call.displayMode = FunctionCall::DisplayMode::MultiLine; expect(SoltToken::Arrow); - if (m_scanner.peekToken() != SoltToken::Newline) - { - call.expectations = parseFunctionCallExpectations(); - - string comment = m_scanner.currentLiteral(); - if (accept(SoltToken::Comment, true)) - call.expectations.comment = comment; - } + + call.expectations = parseFunctionCallExpectations(); + + if (accept(SoltToken::Newline, false)) + call.displayMode = parseNewline(); + call.expectations.comment = parseComment(); + calls.emplace_back(std::move(call)); } else @@ -127,7 +121,8 @@ bool TestFileParser::accept(SoltToken _token, bool const _expect) bool TestFileParser::expect(SoltToken _token, bool const _advance) { if (m_scanner.currentToken() != _token) - throw Error(Error::Type::ParserError, + throw Error + (Error::Type::ParserError, "Unexpected " + formatToken(m_scanner.currentToken()) + ": \"" + m_scanner.currentLiteral() + "\". " + "Expected \"" + formatToken(_token) + "\"." @@ -164,10 +159,7 @@ string TestFileParser::parseFunctionSignature() u256 TestFileParser::parseFunctionCallValue() { - u256 value; - string literal = m_scanner.currentLiteral(); - expect(SoltToken::Number); - value = convertNumber(literal); + u256 value = convertNumber(parseNumber()); expect(SoltToken::Ether); return value; } @@ -176,41 +168,46 @@ FunctionCallArgs TestFileParser::parseFunctionCallArguments() { FunctionCallArgs arguments; - auto formattedBytes = parseABITypeLiteral(); - arguments.rawBytes += formattedBytes.first; - arguments.formats.emplace_back(std::move(formattedBytes.second)); + auto param = parseParameter(); + if (param.abiType.type == ABIType::None) + throw Error(Error::Type::ParserError, "No argument provided."); + arguments.parameters.emplace_back(param); + while (accept(SoltToken::Comma, true)) - { - auto formattedBytes = parseABITypeLiteral(); - arguments.rawBytes += formattedBytes.first; - arguments.formats.emplace_back(std::move(formattedBytes.second)); - } + arguments.parameters.emplace_back(parseParameter()); return arguments; } FunctionCallExpectations TestFileParser::parseFunctionCallExpectations() { FunctionCallExpectations expectations; - string token = m_scanner.currentLiteral(); - if (accept(SoltToken::Failure, true)) - expectations.status = false; - else - { - auto formattedBytes = parseABITypeLiteral(); - expectations.rawBytes += formattedBytes.first; - expectations.formats.emplace_back(std::move(formattedBytes.second)); + auto param = parseParameter(); + if (param.abiType.type == ABIType::None) + return expectations; + expectations.parameters.emplace_back(param); - while (accept(SoltToken::Comma, true)) - { - auto formattedBytes = parseABITypeLiteral(); - expectations.rawBytes += formattedBytes.first; - expectations.formats.emplace_back(std::move(formattedBytes.second)); - } - } + while (accept(SoltToken::Comma, true)) + expectations.parameters.emplace_back(parseParameter()); + + /// We have always one virtual parameter in the parameter list. + /// If its type is FAILURE, the expected result is also a REVERT etc. + if (expectations.parameters.at(0).abiType.type != ABIType::Failure) + expectations.failure = false; return expectations; } +Parameter TestFileParser::parseParameter() +{ + Parameter parameter; + if (accept(SoltToken::Newline, true)) + parameter.format.newline = true; + auto literal = parseABITypeLiteral(); + parameter.rawBytes = literal.first; + parameter.abiType = literal.second; + return parameter; +} + pair TestFileParser::parseABITypeLiteral() { try @@ -219,21 +216,23 @@ pair TestFileParser::parseABITypeLiteral() ABIType abiType; if (accept(SoltToken::Sub)) { - abiType.type = ABIType::Type::SignedDec; - abiType.size = 32; - + abiType = ABIType{ABIType::SignedDec, 32}; expect(SoltToken::Sub); number = convertNumber(parseNumber()) * -1; } else + { if (accept(SoltToken::Number)) { - abiType.type = ABIType::Type::UnsignedDec; - abiType.size = 32; - + abiType = ABIType{ABIType::UnsignedDec, 32}; number = convertNumber(parseNumber()); } - + if (accept(SoltToken::Failure, true)) + { + abiType = ABIType{ABIType::Failure, 0}; + return make_pair(bytes{}, abiType); + } + } return make_pair(toBigEndian(number), abiType); } catch (std::exception const&) @@ -242,6 +241,21 @@ pair TestFileParser::parseABITypeLiteral() } } +solidity::test::FunctionCall::DisplayMode TestFileParser::parseNewline() +{ + if (accept(SoltToken::Newline, true)) + return FunctionCall::DisplayMode::MultiLine; + return FunctionCall::DisplayMode::SingleLine; +} + +string TestFileParser::parseComment() +{ + string comment = m_scanner.currentLiteral(); + if (accept(SoltToken::Comment, true)) + return comment; + return string{}; +} + string TestFileParser::parseNumber() { string literal = m_scanner.currentLiteral(); @@ -283,7 +297,8 @@ void TestFileParser::Scanner::scanNextToken() }; TokenDesc token = make_pair(SoltToken::Unknown, ""); - do { + do + { switch(current()) { case '/': diff --git a/test/libsolidity/util/TestFileParser.h b/test/libsolidity/util/TestFileParser.h index 83e2c08b3821..5c226986eabe 100644 --- a/test/libsolidity/util/TestFileParser.h +++ b/test/libsolidity/util/TestFileParser.h @@ -19,6 +19,7 @@ #include #include +#include #include #include #include @@ -53,7 +54,7 @@ namespace test T(Arrow, "->", 0) \ T(Newline, "//", 0) \ /* Literals & identifier */ \ - T(Comment, "comment", 0) \ + T(Comment, "#", 0) \ T(Number, "number", 0) \ T(Identifier, "identifier", 0) \ /* type keywords */ \ @@ -62,7 +63,6 @@ namespace test /* special keywords */ \ K(Failure, "FAILURE", 0) \ - enum class SoltToken : unsigned int { #define T(name, string, precedence) name, SOLT_TOKEN_LIST(T, T) @@ -82,40 +82,75 @@ struct ABIType enum Type { UnsignedDec, SignedDec, - Invalid + Failure, + None }; - ABIType(): type(ABIType::Invalid), size(0) { } + ABIType(): type(ABIType::None), size(0) { } ABIType(Type _type, size_t _size): type(_type), size(_size) { } Type type; size_t size; }; -using ABITypeList = std::vector; +/** + * Helper that can hold format information retrieved + * while scanning through a parameter list in sol_t. + */ +struct FormatInfo +{ + bool newline; +}; /** - * Represents the expected result of a function call after it has been executed. This may be a single - * return value or a comma-separated list of return values. It also contains the detected input - * formats used to convert the values to `bytes` needed for the comparison with the actual result - * of a call. In addition to that, it also stores the expected transaction status. - * An optional comment can be assigned. + * Parameter abstraction used for the encoding and decoding + * function parameter lists and expectation lists. + * A parameter list is usually a comma-separated list of literals. + * It should not be possible to call create a parameter holding + * an identifier, but if so, the ABI type would be invalid. */ -struct FunctionCallExpectations +struct Parameter { /// ABI encoded `bytes` of parsed expectations. This `bytes` /// is compared to the actual result of a function call /// and is taken into account while validating it. bytes rawBytes; /// Types that were used to encode `rawBytes`. Expectations - /// are usually comma seperated literals. Their type is auto- + /// are usually comma separated literals. Their type is auto- /// detected and retained in order to format them later on. - ABITypeList formats; + ABIType abiType; + /// Format info attached to the parameter. It handles newlines given + /// in the declaration of it. + FormatInfo format; +}; +using ParameterList = std::vector; + +/** + * Represents the expected result of a function call after it has been executed. This may be a single + * return value or a comma-separated list of return values. It also contains the detected input + * formats used to convert the values to `bytes` needed for the comparison with the actual result + * of a call. In addition to that, it also stores the expected transaction status. + * An optional comment can be assigned. + */ +struct FunctionCallExpectations +{ + /// Representation of the comma-separated (or empty) list of expectation parameters given + /// to a function call. + ParameterList parameters; /// Expected status of the transaction. It can be either /// a REVERT or a different EVM failure (e.g. out-of-gas). - bool status = true; + bool failure = true; /// A Comment that can be attached to the expectations, /// that is retained and can be displayed. std::string comment; + /// ABI encoded `bytes` of parsed parameters. This `bytes` + /// passed to the function call. + bytes rawBytes() const + { + bytes raw; + for (auto const& param: parameters) + raw += param.rawBytes; + return raw; + } }; /** @@ -126,16 +161,22 @@ struct FunctionCallExpectations */ struct FunctionCallArgs { - /// ABI encoded `bytes` of parsed parameters. This `bytes` - /// passed to the function call. - bytes rawBytes; /// Types that were used to encode `rawBytes`. Parameters - /// are usually comma seperated literals. Their type is auto- + /// are usually comma separated literals. Their type is auto- /// detected and retained in order to format them later on. - ABITypeList formats; + ParameterList parameters; /// A Comment that can be attached to the expectations, /// that is retained and can be displayed. std::string comment; + /// ABI encoded `bytes` of parsed parameters. This `bytes` + /// passed to the function call. + bytes rawBytes() const + { + bytes raw; + for (auto const& param: parameters) + raw += param.rawBytes; + return raw; + } }; /** @@ -156,6 +197,16 @@ struct FunctionCall /// They are checked against the actual results and their /// `bytes` representation, as well as the transaction status. FunctionCallExpectations expectations; + /// single / multi-line mode will be detected as follows: + /// every newline (//) in source results in a function call + /// that has its display mode set to multi-mode. Function and + /// result parameter lists are an exception: a single parameter + /// stores a format information that contains a newline definition. + enum DisplayMode { + SingleLine, + MultiLine + }; + DisplayMode displayMode = DisplayMode::SingleLine; }; /** @@ -196,6 +247,7 @@ class TestFileParser { public: /// Constructor that takes an input stream \param _stream to operate on. + /// It reads all lines into one single line, keeping the newlines. Scanner(std::istream& _stream) { readStream(_stream); } /// Reads input stream into a single line and resets the current iterator. @@ -253,19 +305,34 @@ class TestFileParser /// Parses the expected result of a function call execution. FunctionCallExpectations parseFunctionCallExpectations(); + /// Parses the next parameter in a comma separated list. + /// Takes a newly parsed, and type-annotated `bytes` argument, + /// appends it to the internal `bytes` buffer of the parameter. It can also + /// store newlines found in the source, that are needed to + /// format input and output of the interactive update. + Parameter parseParameter(); + /// Parses and converts the current literal to its byte representation and /// preserves the chosen ABI type. Based on that type information, the driver of /// this parser can format arguments, expectations and results. Supported types: - /// - unsigned and signed decimal number literals - /// Throws a ParserError if data is encoded incorrectly or + /// - unsigned and signed decimal number literals. + /// Returns invalid ABI type for empty literal. This is needed in order + /// to detect empty expectations. Throws a ParserError if data is encoded incorrectly or /// if data type is not supported. std::pair parseABITypeLiteral(); + /// Accepts a newline `//` and returns DisplayMode::MultiLine + /// if found, DisplayMode::SingleLine otherwise. + FunctionCall::DisplayMode parseNewline(); + + /// Parses a comment + std::string parseComment(); + /// Parses the current number literal. std::string parseNumber(); /// Tries to convert \param _literal to `uint256` and throws if - /// if conversion failed. + /// conversion fails. u256 convertNumber(std::string const& _literal); /// A scanner instance diff --git a/test/libsolidity/util/TestFileParserTests.cpp b/test/libsolidity/util/TestFileParserTests.cpp index 88d4d4c12606..620f41581f3a 100644 --- a/test/libsolidity/util/TestFileParserTests.cpp +++ b/test/libsolidity/util/TestFileParserTests.cpp @@ -57,44 +57,93 @@ BOOST_AUTO_TEST_CASE(simple_call_succees) char const* source = R"( // f(uint256, uint256): 1, 1 // -> + // # This call should not return a value, but still succeed. # )"; - auto const calls = parse(source); BOOST_CHECK_EQUAL(calls.size(), 1); auto call = calls.at(0); - ABI_CHECK(call.arguments.rawBytes, toBigEndian(u256{1}) + toBigEndian(u256{1})); + BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::MultiLine); BOOST_CHECK_EQUAL(call.signature, "f(uint256,uint256)"); + ABI_CHECK(call.arguments.rawBytes(), toBigEndian(u256{1}) + toBigEndian(u256{1})); + ABI_CHECK(call.expectations.rawBytes(), bytes{}); } -BOOST_AUTO_TEST_CASE(non_existent_call_revert) +BOOST_AUTO_TEST_CASE(simple_single_line_call_success) { char const* source = R"( - // i_am_not_there() - // -> FAILURE + // f(uint256): 1 -> # Does not expect a value. # + // f(uint256): 1 -> 1 # Expect return value. # + )"; + auto const calls = parse(source); + BOOST_CHECK_EQUAL(calls.size(), 2); + + auto call = calls.at(0); + BOOST_CHECK_EQUAL(call.signature, "f(uint256)"); + BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::SingleLine); + ABI_CHECK(call.arguments.rawBytes(), toBigEndian(u256{1})); + ABI_CHECK(call.expectations.rawBytes(), bytes{}); + + call = calls.at(1); + BOOST_CHECK_EQUAL(call.signature, "f(uint256)"); + BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::SingleLine); + ABI_CHECK(call.arguments.rawBytes(), toBigEndian(u256{1})); + ABI_CHECK(call.expectations.rawBytes(), toBigEndian(u256{1})); +} + +BOOST_AUTO_TEST_CASE(non_existent_call_revert_single_line) +{ + char const* source = R"( + // i_am_not_there() -> FAILURE )"; auto const calls = parse(source); BOOST_CHECK_EQUAL(calls.size(), 1); auto const& call = calls.at(0); + BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::SingleLine); BOOST_CHECK_EQUAL(call.signature, "i_am_not_there()"); - BOOST_CHECK_EQUAL(call.expectations.status, false); + BOOST_CHECK_EQUAL(call.expectations.failure, true); + BOOST_CHECK_EQUAL(call.expectations.parameters.at(0).abiType.type, ABIType::Failure); + ABI_CHECK(call.expectations.rawBytes(), bytes{}); } -BOOST_AUTO_TEST_CASE(call_comments) +BOOST_AUTO_TEST_CASE(non_existent_call_revert) { char const* source = R"( - // f() # This is a comment # - // -> 1 # This is another comment # + // i_am_not_there() + // -> FAILURE # This is can be either REVERT or a different EVM failure # )"; auto const calls = parse(source); BOOST_CHECK_EQUAL(calls.size(), 1); auto const& call = calls.at(0); - BOOST_CHECK_EQUAL(call.signature, "f()"); - BOOST_CHECK_EQUAL(call.arguments.comment, " This is a comment "); - BOOST_CHECK_EQUAL(call.expectations.comment, " This is another comment "); - ABI_CHECK(call.expectations.rawBytes, toBigEndian(u256{1})); + BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::MultiLine); + BOOST_CHECK_EQUAL(call.signature, "i_am_not_there()"); + BOOST_CHECK_EQUAL(call.expectations.parameters.at(0).abiType.type, ABIType::Failure); + ABI_CHECK(call.expectations.rawBytes(), bytes{}); + BOOST_CHECK_EQUAL(call.expectations.failure, true); +} + +BOOST_AUTO_TEST_CASE(call_comments) +{ + char const* source = R"( + // f() # Parameter comment # -> 1 # Expectation comment # + // f() # Parameter comment # + // -> 1 # Expectation comment # + )"; + auto const calls = parse(source); + BOOST_CHECK_EQUAL(calls.size(), 2); + + BOOST_CHECK_EQUAL(calls.at(0).displayMode, FunctionCall::DisplayMode::SingleLine); + BOOST_CHECK_EQUAL(calls.at(1).displayMode, FunctionCall::DisplayMode::MultiLine); + + for (auto const& call: calls) + { + BOOST_CHECK_EQUAL(call.signature, "f()"); + BOOST_CHECK_EQUAL(call.arguments.comment, " Parameter comment "); + BOOST_CHECK_EQUAL(call.expectations.comment, " Expectation comment "); + ABI_CHECK(call.expectations.rawBytes(), toBigEndian(u256{1})); + } } BOOST_AUTO_TEST_CASE(call_arguments) @@ -107,10 +156,44 @@ BOOST_AUTO_TEST_CASE(call_arguments) BOOST_CHECK_EQUAL(calls.size(), 1); auto const& call = calls.at(0); + BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::MultiLine); BOOST_CHECK_EQUAL(call.signature, "f(uint256)"); BOOST_CHECK_EQUAL(call.value, u256{314}); - ABI_CHECK(call.arguments.rawBytes, toBigEndian(u256{5})); - ABI_CHECK(call.expectations.rawBytes, toBigEndian(u256{4})); + BOOST_CHECK_EQUAL(call.expectations.failure, false); + ABI_CHECK(call.arguments.rawBytes(), toBigEndian(u256{5})); + ABI_CHECK(call.expectations.rawBytes(), toBigEndian(u256{4})); +} + +BOOST_AUTO_TEST_CASE(call_expectations_empty_single_line) +{ + char const* source = R"( + // _exp_() -> + )"; + auto const calls = parse(source); + BOOST_CHECK_EQUAL(calls.size(), 1); + + auto call = calls.at(0); + BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::SingleLine); + BOOST_CHECK_EQUAL(call.signature, "_exp_()"); + ABI_CHECK(call.arguments.rawBytes(), bytes{}); + ABI_CHECK(call.expectations.rawBytes(), bytes{}); +} + +BOOST_AUTO_TEST_CASE(call_expectations_empty_multiline) +{ + char const* source = R"( + // _exp_() + // -> + // # This call should not return a value, but still succeed. # + )"; + auto const calls = parse(source); + BOOST_CHECK_EQUAL(calls.size(), 1); + + auto call = calls.at(0); + BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::MultiLine); + BOOST_CHECK_EQUAL(call.signature, "_exp_()"); + ABI_CHECK(call.arguments.rawBytes(), bytes{}); + ABI_CHECK(call.expectations.rawBytes(), bytes{}); } BOOST_AUTO_TEST_CASE(call_expectations_missing) @@ -130,8 +213,7 @@ BOOST_AUTO_TEST_CASE(call_ether_value_expectations_missing) BOOST_AUTO_TEST_CASE(call_arguments_invalid) { char const* source = R"( - // f(uint256): abc - // -> 1 + // f(uint256): abc -> 1 )"; BOOST_CHECK_THROW(parse(source), langutil::Error); } @@ -139,8 +221,7 @@ BOOST_AUTO_TEST_CASE(call_arguments_invalid) BOOST_AUTO_TEST_CASE(call_ether_value_invalid) { char const* source = R"( - // f(uint256), abc : 1 - // -> 1 + // f(uint256), abc : 1 -> 1 )"; BOOST_CHECK_THROW(parse(source), langutil::Error); } @@ -148,8 +229,7 @@ BOOST_AUTO_TEST_CASE(call_ether_value_invalid) BOOST_AUTO_TEST_CASE(call_ether_type_invalid) { char const* source = R"( - // f(uint256), 2 btc : 1 - // -> 1 + // f(uint256), 2 btc : 1 -> 1 )"; BOOST_CHECK_THROW(parse(source), langutil::Error); } @@ -157,47 +237,80 @@ BOOST_AUTO_TEST_CASE(call_ether_type_invalid) BOOST_AUTO_TEST_CASE(call_arguments_mismatch) { char const* source = R"( - // f(uint256, uint256): 1 # This only throws at runtime # + // f(uint256): + // 1, 2 + // # This only throws at runtime # // -> 1 )"; auto const calls = parse(source); BOOST_CHECK_EQUAL(calls.size(), 1); auto const& call = calls.at(0); - BOOST_CHECK_EQUAL(call.signature, "f(uint256,uint256)"); - ABI_CHECK(call.arguments.rawBytes, toBigEndian(u256{1})); + BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::MultiLine); + BOOST_CHECK_EQUAL(call.signature, "f(uint256)"); + ABI_CHECK(call.arguments.rawBytes(), toBigEndian(u256{1}) + toBigEndian(u256{2})); + BOOST_CHECK_EQUAL(call.expectations.failure, false); } BOOST_AUTO_TEST_CASE(call_multiple_arguments) { char const* source = R"( - // test(uint256, uint256): 1, 2 - // -> 1, 1 + // test(uint256, uint256): + // 1, + // 2 + // -> 1, + // 1 )"; auto const calls = parse(source); BOOST_CHECK_EQUAL(calls.size(), 1); auto const& call = calls.at(0); + BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::MultiLine); BOOST_CHECK_EQUAL(call.signature, "test(uint256,uint256)"); - ABI_CHECK(call.arguments.rawBytes, toBigEndian(u256{1}) + toBigEndian(u256{2})); + ABI_CHECK(call.arguments.rawBytes(), toBigEndian(u256{1}) + toBigEndian(u256{2})); + BOOST_CHECK_EQUAL(call.expectations.failure, false); } BOOST_AUTO_TEST_CASE(call_multiple_arguments_mixed_format) { char const* source = R"( - // test(uint256, uint256),314 ether: 1, -2 + // test(uint256, uint256), 314 ether: + // 1, -2 // -> -1, 2 )"; auto const calls = parse(source); BOOST_CHECK_EQUAL(calls.size(), 1); auto const& call = calls.at(0); + BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::MultiLine); BOOST_CHECK_EQUAL(call.signature, "test(uint256,uint256)"); BOOST_CHECK_EQUAL(call.value, u256{314}); - ABI_CHECK(call.arguments.rawBytes, toBigEndian(u256{1}) + toBigEndian(u256{-2})); - ABI_CHECK(call.expectations.rawBytes, toBigEndian(u256{-1}) + toBigEndian(u256{2})); + ABI_CHECK(call.arguments.rawBytes(), toBigEndian(u256{1}) + toBigEndian(u256{-2})); + BOOST_CHECK_EQUAL(call.expectations.failure, false); + ABI_CHECK(call.expectations.rawBytes(), toBigEndian(u256{-1}) + toBigEndian(u256{2})); } +BOOST_AUTO_TEST_CASE(call_arguments_colon) +{ + char const* source = R"( + // h256(): + // -> 1 + )"; + BOOST_CHECK_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_arguments_newline_colon) +{ + char const* source = R"( + // h256() + // : + // -> 1 + )"; + BOOST_CHECK_THROW(parse(source), langutil::Error); +} + + + BOOST_AUTO_TEST_SUITE_END() } From 161b22bd137f6daaae9c9ca0ca8122487ad3e6e2 Mon Sep 17 00:00:00 2001 From: Erik Kundt Date: Sat, 2 Feb 2019 14:22:46 +0100 Subject: [PATCH 073/109] Cleans up test file parser and its tests. --- test/libsolidity/util/TestFileParser.cpp | 84 ++-- test/libsolidity/util/TestFileParser.h | 46 +-- test/libsolidity/util/TestFileParserTests.cpp | 378 +++++++++++------- 3 files changed, 295 insertions(+), 213 deletions(-) diff --git a/test/libsolidity/util/TestFileParser.cpp b/test/libsolidity/util/TestFileParser.cpp index 40ea95b0df1f..2c304e773b36 100644 --- a/test/libsolidity/util/TestFileParser.cpp +++ b/test/libsolidity/util/TestFileParser.cpp @@ -34,21 +34,13 @@ using namespace std; namespace { - bool isDecimalDigit(char c) - { - return '0' <= c && c <= '9'; - } - bool isWhiteSpace(char c) - { - return c == ' ' || c == '\n' || c == '\t' || c == '\r'; - } bool isIdentifierStart(char c) { return c == '_' || c == '$' || ('a' <= c && c <= 'z') || ('A' <= c && c <= 'Z'); } bool isIdentifierPart(char c) { - return isIdentifierStart(c) || isDecimalDigit(c); + return isIdentifierStart(c) || isdigit(c); } } @@ -57,33 +49,41 @@ vector TestFileParser::parseFunctionCalls() vector calls; if (!accept(SoltToken::EOS)) { - // TODO: check initial token state - expect(SoltToken::Unknown); + assert(m_scanner.currentToken() == SoltToken::Unknown); + m_scanner.scanNextToken(); + while (!accept(SoltToken::EOS)) { if (!accept(SoltToken::Whitespace)) { FunctionCall call; - expect(SoltToken::Newline); - call.signature = parseFunctionSignature(); + /// If this is not the first call in the test, + /// the last call to parseParameter could have eaten the + /// new line already. This could only be fixed with a one + /// token lookahead that checks parseParameter + /// if the next token is an identifier. + if (calls.empty()) + expect(SoltToken::Newline); + else + accept(SoltToken::Newline, true); + call.signature = parseFunctionSignature(); if (accept(SoltToken::Comma, true)) call.value = parseFunctionCallValue(); if (accept(SoltToken::Colon, true)) call.arguments = parseFunctionCallArguments(); - call.displayMode = parseNewline(); + if (accept(SoltToken::Newline, true)) + call.displayMode = FunctionCall::DisplayMode::MultiLine; + call.arguments.comment = parseComment(); if (accept(SoltToken::Newline, true)) call.displayMode = FunctionCall::DisplayMode::MultiLine; - expect(SoltToken::Arrow); + expect(SoltToken::Arrow); call.expectations = parseFunctionCallExpectations(); - - if (accept(SoltToken::Newline, false)) - call.displayMode = parseNewline(); call.expectations.comment = parseComment(); calls.emplace_back(std::move(call)); @@ -121,12 +121,12 @@ bool TestFileParser::accept(SoltToken _token, bool const _expect) bool TestFileParser::expect(SoltToken _token, bool const _advance) { if (m_scanner.currentToken() != _token) - throw Error - (Error::Type::ParserError, - "Unexpected " + formatToken(m_scanner.currentToken()) + ": \"" + - m_scanner.currentLiteral() + "\". " + - "Expected \"" + formatToken(_token) + "\"." - ); + throw Error( + Error::Type::ParserError, + "Unexpected " + formatToken(m_scanner.currentToken()) + ": \"" + + m_scanner.currentLiteral() + "\". " + + "Expected \"" + formatToken(_token) + "\"." + ); if (_advance) m_scanner.scanNextToken(); return true; @@ -143,13 +143,13 @@ string TestFileParser::parseFunctionSignature() while (!accept(SoltToken::RParen)) { signature += m_scanner.currentLiteral(); - expect(SoltToken::UInt); + expect(SoltToken::Identifier); while (accept(SoltToken::Comma)) { signature += m_scanner.currentLiteral(); expect(SoltToken::Comma); signature += m_scanner.currentLiteral(); - expect(SoltToken::UInt); + expect(SoltToken::Identifier); } } signature += formatToken(SoltToken::RParen); @@ -184,15 +184,18 @@ FunctionCallExpectations TestFileParser::parseFunctionCallExpectations() auto param = parseParameter(); if (param.abiType.type == ABIType::None) + { + expectations.failure = false; return expectations; - expectations.parameters.emplace_back(param); + } + expectations.result.emplace_back(param); while (accept(SoltToken::Comma, true)) - expectations.parameters.emplace_back(parseParameter()); + expectations.result.emplace_back(parseParameter()); /// We have always one virtual parameter in the parameter list. /// If its type is FAILURE, the expected result is also a REVERT etc. - if (expectations.parameters.at(0).abiType.type != ABIType::Failure) + if (expectations.result.at(0).abiType.type != ABIType::Failure) expectations.failure = false; return expectations; } @@ -212,8 +215,9 @@ pair TestFileParser::parseABITypeLiteral() { try { - u256 number; - ABIType abiType; + u256 number{0}; + ABIType abiType{ABIType::None, 0}; + if (accept(SoltToken::Sub)) { abiType = ABIType{ABIType::SignedDec, 32}; @@ -227,7 +231,7 @@ pair TestFileParser::parseABITypeLiteral() abiType = ABIType{ABIType::UnsignedDec, 32}; number = convertNumber(parseNumber()); } - if (accept(SoltToken::Failure, true)) + else if (accept(SoltToken::Failure, true)) { abiType = ABIType{ABIType::Failure, 0}; return make_pair(bytes{}, abiType); @@ -241,13 +245,6 @@ pair TestFileParser::parseABITypeLiteral() } } -solidity::test::FunctionCall::DisplayMode TestFileParser::parseNewline() -{ - if (accept(SoltToken::Newline, true)) - return FunctionCall::DisplayMode::MultiLine; - return FunctionCall::DisplayMode::SingleLine; -} - string TestFileParser::parseComment() { string comment = m_scanner.currentLiteral(); @@ -286,7 +283,6 @@ void TestFileParser::Scanner::scanNextToken() { auto detectToken = [](std::string const& _literal = "") -> TokenDesc { if (_literal == "ether") return TokenDesc{SoltToken::Ether, _literal}; - if (_literal == "uint256") return TokenDesc{SoltToken::UInt, _literal}; if (_literal == "FAILURE") return TokenDesc{SoltToken::Failure, _literal}; return TokenDesc{SoltToken::Identifier, _literal}; }; @@ -336,9 +332,9 @@ void TestFileParser::Scanner::scanNextToken() TokenDesc detectedToken = detectToken(scanIdentifierOrKeyword()); token = selectToken(detectedToken.first, detectedToken.second); } - else if (isDecimalDigit(current())) + else if (isdigit(current())) token = selectToken(SoltToken::Number, scanNumber()); - else if (isWhiteSpace(current())) + else if (isspace(current())) token = selectToken(SoltToken::Whitespace); else if (isEndOfLine()) token = selectToken(SoltToken::EOS); @@ -348,8 +344,6 @@ void TestFileParser::Scanner::scanNextToken() } } while (token.first == SoltToken::Whitespace); - - m_nextToken = token; m_currentToken = token; } @@ -382,7 +376,7 @@ string TestFileParser::Scanner::scanNumber() { string number; number += current(); - while (isDecimalDigit(peek())) + while (isdigit(peek())) { advance(); number += current(); diff --git a/test/libsolidity/util/TestFileParser.h b/test/libsolidity/util/TestFileParser.h index 5c226986eabe..cbc8717cbe86 100644 --- a/test/libsolidity/util/TestFileParser.h +++ b/test/libsolidity/util/TestFileParser.h @@ -59,7 +59,6 @@ namespace test T(Identifier, "identifier", 0) \ /* type keywords */ \ K(Ether, "ether", 0) \ - K(UInt, "uint256", 0) \ /* special keywords */ \ K(Failure, "FAILURE", 0) \ @@ -75,7 +74,9 @@ enum class SoltToken : unsigned int { * retrieved while parsing a test. This information is used * for the conversion of human-readable function arguments and * return values to `bytes` and vice-versa. - * Defaults to an invalid 0-byte representation. + * Defaults to None, a 0-byte representation. 0-bytes + * can also be interpreted as Failure, which means + * either a REVERT or another EVM failure. */ struct ABIType { @@ -85,11 +86,8 @@ struct ABIType Failure, None }; - ABIType(): type(ABIType::None), size(0) { } - ABIType(Type _type, size_t _size): type(_type), size(_size) { } - - Type type; - size_t size; + Type type = ABIType::None; + size_t size = 0; }; /** @@ -102,17 +100,19 @@ struct FormatInfo }; /** - * Parameter abstraction used for the encoding and decoding - * function parameter lists and expectation lists. + * Parameter abstraction used for the encoding and decoding of + * function parameter and expectation / return value lists. * A parameter list is usually a comma-separated list of literals. * It should not be possible to call create a parameter holding * an identifier, but if so, the ABI type would be invalid. */ struct Parameter { - /// ABI encoded `bytes` of parsed expectations. This `bytes` - /// is compared to the actual result of a function call - /// and is taken into account while validating it. + /// ABI encoded / decoded `bytes` of values. + /// These `bytes` are used to pass values to function calls + /// and also to store expected return vales. These are + /// compared to the actual result of a function call + /// and used for validating it. bytes rawBytes; /// Types that were used to encode `rawBytes`. Expectations /// are usually comma separated literals. Their type is auto- @@ -133,21 +133,22 @@ using ParameterList = std::vector; */ struct FunctionCallExpectations { - /// Representation of the comma-separated (or empty) list of expectation parameters given - /// to a function call. - ParameterList parameters; + /// Representation of the comma-separated (or empty) list of expectated result values + /// attached to the function call object. It is checked against the actual result of + /// a function call when used in test framework. + ParameterList result; /// Expected status of the transaction. It can be either /// a REVERT or a different EVM failure (e.g. out-of-gas). bool failure = true; /// A Comment that can be attached to the expectations, /// that is retained and can be displayed. std::string comment; - /// ABI encoded `bytes` of parsed parameters. This `bytes` - /// passed to the function call. + /// ABI encoded `bytes` of parsed expected return values. It is checked + /// against the actual result of a function call when used in test framework. bytes rawBytes() const { bytes raw; - for (auto const& param: parameters) + for (auto const& param: result) raw += param.rawBytes; return raw; } @@ -168,7 +169,7 @@ struct FunctionCallArgs /// A Comment that can be attached to the expectations, /// that is retained and can be displayed. std::string comment; - /// ABI encoded `bytes` of parsed parameters. This `bytes` + /// ABI encoded `bytes` of parsed parameters. These `bytes` /// passed to the function call. bytes rawBytes() const { @@ -257,8 +258,6 @@ class TestFileParser void scanNextToken(); SoltToken currentToken() { return m_currentToken.first; } - SoltToken peekToken() { return m_nextToken.first; } - std::string currentLiteral() { return m_currentToken.second; } std::string scanComment(); @@ -283,7 +282,6 @@ class TestFileParser std::string m_currentLiteral; TokenDesc m_currentToken; - TokenDesc m_nextToken; }; bool accept(SoltToken _token, bool const _expect = false); @@ -321,10 +319,6 @@ class TestFileParser /// if data type is not supported. std::pair parseABITypeLiteral(); - /// Accepts a newline `//` and returns DisplayMode::MultiLine - /// if found, DisplayMode::SingleLine otherwise. - FunctionCall::DisplayMode parseNewline(); - /// Parses a comment std::string parseComment(); diff --git a/test/libsolidity/util/TestFileParserTests.cpp b/test/libsolidity/util/TestFileParserTests.cpp index 620f41581f3a..30fb2d765a9e 100644 --- a/test/libsolidity/util/TestFileParserTests.cpp +++ b/test/libsolidity/util/TestFileParserTests.cpp @@ -23,7 +23,7 @@ #include #include #include -#include +#include #include @@ -37,6 +37,9 @@ namespace solidity namespace test { +using fmt = ExecutionFramework; +using Mode = FunctionCall::DisplayMode; + vector parse(string const& _source) { istringstream stream{_source, ios_base::out}; @@ -44,51 +47,44 @@ vector parse(string const& _source) return parser.parseFunctionCalls(); } +void testFunctionCall( + FunctionCall const& _call, + FunctionCall::DisplayMode _mode, + string _signature = "", + bool _failure = true, + bytes _arguments = bytes{}, + bytes _expectations = bytes{}, + u256 _value = 0, + string _argumentComment = "", + string _expectationComment = "" +) +{ + BOOST_REQUIRE_EQUAL(_call.expectations.failure, _failure); + BOOST_REQUIRE_EQUAL(_call.signature, _signature); + ABI_CHECK(_call.arguments.rawBytes(), _arguments); + ABI_CHECK(_call.expectations.rawBytes(), _expectations); + BOOST_REQUIRE_EQUAL(_call.displayMode, _mode); + BOOST_REQUIRE_EQUAL(_call.value, _value); + BOOST_REQUIRE_EQUAL(_call.arguments.comment, _argumentComment); + BOOST_REQUIRE_EQUAL(_call.expectations.comment, _expectationComment); +} + BOOST_AUTO_TEST_SUITE(TestFileParserTest) BOOST_AUTO_TEST_CASE(smoke_test) { char const* source = R"()"; - BOOST_CHECK_EQUAL(parse(source).size(), 0); -} - -BOOST_AUTO_TEST_CASE(simple_call_succees) -{ - char const* source = R"( - // f(uint256, uint256): 1, 1 - // -> - // # This call should not return a value, but still succeed. # - )"; - auto const calls = parse(source); - BOOST_CHECK_EQUAL(calls.size(), 1); - - auto call = calls.at(0); - BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::MultiLine); - BOOST_CHECK_EQUAL(call.signature, "f(uint256,uint256)"); - ABI_CHECK(call.arguments.rawBytes(), toBigEndian(u256{1}) + toBigEndian(u256{1})); - ABI_CHECK(call.expectations.rawBytes(), bytes{}); + BOOST_REQUIRE_EQUAL(parse(source).size(), 0); } -BOOST_AUTO_TEST_CASE(simple_single_line_call_success) +BOOST_AUTO_TEST_CASE(call_succees) { char const* source = R"( - // f(uint256): 1 -> # Does not expect a value. # - // f(uint256): 1 -> 1 # Expect return value. # + // success() -> )"; auto const calls = parse(source); - BOOST_CHECK_EQUAL(calls.size(), 2); - - auto call = calls.at(0); - BOOST_CHECK_EQUAL(call.signature, "f(uint256)"); - BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::SingleLine); - ABI_CHECK(call.arguments.rawBytes(), toBigEndian(u256{1})); - ABI_CHECK(call.expectations.rawBytes(), bytes{}); - - call = calls.at(1); - BOOST_CHECK_EQUAL(call.signature, "f(uint256)"); - BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::SingleLine); - ABI_CHECK(call.arguments.rawBytes(), toBigEndian(u256{1})); - ABI_CHECK(call.expectations.rawBytes(), toBigEndian(u256{1})); + BOOST_REQUIRE_EQUAL(calls.size(), 1); + testFunctionCall(calls.at(0), Mode::SingleLine, "success()", false); } BOOST_AUTO_TEST_CASE(non_existent_call_revert_single_line) @@ -97,141 +93,176 @@ BOOST_AUTO_TEST_CASE(non_existent_call_revert_single_line) // i_am_not_there() -> FAILURE )"; auto const calls = parse(source); - BOOST_CHECK_EQUAL(calls.size(), 1); - - auto const& call = calls.at(0); - BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::SingleLine); - BOOST_CHECK_EQUAL(call.signature, "i_am_not_there()"); - BOOST_CHECK_EQUAL(call.expectations.failure, true); - BOOST_CHECK_EQUAL(call.expectations.parameters.at(0).abiType.type, ABIType::Failure); - ABI_CHECK(call.expectations.rawBytes(), bytes{}); + BOOST_REQUIRE_EQUAL(calls.size(), 1); + testFunctionCall(calls.at(0), Mode::SingleLine, "i_am_not_there()", true); } -BOOST_AUTO_TEST_CASE(non_existent_call_revert) +BOOST_AUTO_TEST_CASE(call_arguments_success) { char const* source = R"( - // i_am_not_there() - // -> FAILURE # This is can be either REVERT or a different EVM failure # + // f(uint256): 1 + // -> )"; auto const calls = parse(source); - BOOST_CHECK_EQUAL(calls.size(), 1); - - auto const& call = calls.at(0); - BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::MultiLine); - BOOST_CHECK_EQUAL(call.signature, "i_am_not_there()"); - BOOST_CHECK_EQUAL(call.expectations.parameters.at(0).abiType.type, ABIType::Failure); - ABI_CHECK(call.expectations.rawBytes(), bytes{}); - BOOST_CHECK_EQUAL(call.expectations.failure, true); + BOOST_REQUIRE_EQUAL(calls.size(), 1); + testFunctionCall(calls.at(0), Mode::MultiLine, "f(uint256)", false, fmt::encodeArgs(u256{1})); } -BOOST_AUTO_TEST_CASE(call_comments) +BOOST_AUTO_TEST_CASE(call_arguments_comments_success) { char const* source = R"( - // f() # Parameter comment # -> 1 # Expectation comment # - // f() # Parameter comment # - // -> 1 # Expectation comment # + // f(uint256, uint256): 1, 1 + // -> + // # This call should not return a value, but still succeed. # )"; auto const calls = parse(source); - BOOST_CHECK_EQUAL(calls.size(), 2); - - BOOST_CHECK_EQUAL(calls.at(0).displayMode, FunctionCall::DisplayMode::SingleLine); - BOOST_CHECK_EQUAL(calls.at(1).displayMode, FunctionCall::DisplayMode::MultiLine); - - for (auto const& call: calls) - { - BOOST_CHECK_EQUAL(call.signature, "f()"); - BOOST_CHECK_EQUAL(call.arguments.comment, " Parameter comment "); - BOOST_CHECK_EQUAL(call.expectations.comment, " Expectation comment "); - ABI_CHECK(call.expectations.rawBytes(), toBigEndian(u256{1})); - } + BOOST_REQUIRE_EQUAL(calls.size(), 1); + testFunctionCall( + calls.at(0), + Mode::MultiLine, + "f(uint256,uint256)", + false, + fmt::encodeArgs(1, 1), + fmt::encodeArgs(), + 0, + "", + " This call should not return a value, but still succeed. " + ); } -BOOST_AUTO_TEST_CASE(call_arguments) +BOOST_AUTO_TEST_CASE(simple_single_line_call_comment_success) { char const* source = R"( - // f(uint256), 314 ether: 5 # optional ether value # - // -> 4 + // f(uint256): 1 -> # f(uint256) does not return a value. # + // f(uint256): 1 -> 1 )"; auto const calls = parse(source); - BOOST_CHECK_EQUAL(calls.size(), 1); - - auto const& call = calls.at(0); - BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::MultiLine); - BOOST_CHECK_EQUAL(call.signature, "f(uint256)"); - BOOST_CHECK_EQUAL(call.value, u256{314}); - BOOST_CHECK_EQUAL(call.expectations.failure, false); - ABI_CHECK(call.arguments.rawBytes(), toBigEndian(u256{5})); - ABI_CHECK(call.expectations.rawBytes(), toBigEndian(u256{4})); + BOOST_REQUIRE_EQUAL(calls.size(), 2); + + testFunctionCall( + calls.at(0), + Mode::SingleLine, + "f(uint256)", + false, + fmt::encodeArgs(1), + fmt::encodeArgs(), + 0, + "", + " f(uint256) does not return a value. " + ); + testFunctionCall(calls.at(1), Mode::SingleLine, "f(uint256)", false, fmt::encode(1), fmt::encode(1)); } -BOOST_AUTO_TEST_CASE(call_expectations_empty_single_line) +BOOST_AUTO_TEST_CASE(multiple_single_line) { char const* source = R"( - // _exp_() -> + // f(uint256): 1 -> 1 + // g(uint256): 1 -> )"; auto const calls = parse(source); - BOOST_CHECK_EQUAL(calls.size(), 1); + BOOST_REQUIRE_EQUAL(calls.size(), 2); - auto call = calls.at(0); - BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::SingleLine); - BOOST_CHECK_EQUAL(call.signature, "_exp_()"); - ABI_CHECK(call.arguments.rawBytes(), bytes{}); - ABI_CHECK(call.expectations.rawBytes(), bytes{}); + testFunctionCall(calls.at(0), Mode::SingleLine, "f(uint256)", false, fmt::encodeArgs(1), fmt::encodeArgs(1)); + testFunctionCall(calls.at(1), Mode::SingleLine, "g(uint256)", false, fmt::encodeArgs(1)); } -BOOST_AUTO_TEST_CASE(call_expectations_empty_multiline) +BOOST_AUTO_TEST_CASE(multiple_single_line_swapped) { char const* source = R"( - // _exp_() - // -> - // # This call should not return a value, but still succeed. # + // f(uint256): 1 -> + // g(uint256): 1 -> 1 )"; auto const calls = parse(source); - BOOST_CHECK_EQUAL(calls.size(), 1); + BOOST_REQUIRE_EQUAL(calls.size(), 2); + + testFunctionCall(calls.at(0), Mode::SingleLine, "f(uint256)", false, fmt::encodeArgs(1)); + testFunctionCall(calls.at(1), Mode::SingleLine, "g(uint256)", false, fmt::encodeArgs(1), fmt::encodeArgs(1)); - auto call = calls.at(0); - BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::MultiLine); - BOOST_CHECK_EQUAL(call.signature, "_exp_()"); - ABI_CHECK(call.arguments.rawBytes(), bytes{}); - ABI_CHECK(call.expectations.rawBytes(), bytes{}); } -BOOST_AUTO_TEST_CASE(call_expectations_missing) +BOOST_AUTO_TEST_CASE(non_existent_call_revert) { char const* source = R"( - // f())"; - BOOST_CHECK_THROW(parse(source), langutil::Error); + // i_am_not_there() + // -> FAILURE + )"; + auto const calls = parse(source); + BOOST_REQUIRE_EQUAL(calls.size(), 1); + testFunctionCall(calls.at(0), Mode::MultiLine, "i_am_not_there()", true); } -BOOST_AUTO_TEST_CASE(call_ether_value_expectations_missing) +BOOST_AUTO_TEST_CASE(call_expectations_empty_single_line) { char const* source = R"( - // f(), 0)"; - BOOST_CHECK_THROW(parse(source), langutil::Error); + // _exp_() -> + )"; + auto const calls = parse(source); + BOOST_REQUIRE_EQUAL(calls.size(), 1); + testFunctionCall(calls.at(0), Mode::SingleLine, "_exp_()", false); } -BOOST_AUTO_TEST_CASE(call_arguments_invalid) +BOOST_AUTO_TEST_CASE(call_expectations_empty_multiline) { char const* source = R"( - // f(uint256): abc -> 1 + // _exp_() + // -> )"; - BOOST_CHECK_THROW(parse(source), langutil::Error); + auto const calls = parse(source); + BOOST_REQUIRE_EQUAL(calls.size(), 1); + testFunctionCall(calls.at(0), Mode::MultiLine, "_exp_()", false); } -BOOST_AUTO_TEST_CASE(call_ether_value_invalid) +BOOST_AUTO_TEST_CASE(call_comments) { char const* source = R"( - // f(uint256), abc : 1 -> 1 + // f() # Parameter comment # -> 1 # Expectation comment # + // f() # Parameter comment # + // -> 1 # Expectation comment # )"; - BOOST_CHECK_THROW(parse(source), langutil::Error); + auto const calls = parse(source); + BOOST_REQUIRE_EQUAL(calls.size(), 2); + testFunctionCall( + calls.at(0), + Mode::SingleLine, + "f()", + false, + fmt::encodeArgs(), + fmt::encodeArgs(1), + 0, + " Parameter comment ", + " Expectation comment " + ); + testFunctionCall( + calls.at(1), + Mode::MultiLine, + "f()", + false, + fmt::encodeArgs(), + fmt::encodeArgs(1), + 0, + " Parameter comment ", + " Expectation comment " + ); } -BOOST_AUTO_TEST_CASE(call_ether_type_invalid) +BOOST_AUTO_TEST_CASE(call_arguments) { char const* source = R"( - // f(uint256), 2 btc : 1 -> 1 + // f(uint256), 314 ether: 5 # optional ether value # + // -> 4 )"; - BOOST_CHECK_THROW(parse(source), langutil::Error); + auto const calls = parse(source); + BOOST_REQUIRE_EQUAL(calls.size(), 1); + testFunctionCall( + calls.at(0), + Mode::MultiLine, + "f(uint256)", + false, + fmt::encodeArgs(5), + fmt::encodeArgs(4), + 314, + " optional ether value " + ); } BOOST_AUTO_TEST_CASE(call_arguments_mismatch) @@ -243,13 +274,17 @@ BOOST_AUTO_TEST_CASE(call_arguments_mismatch) // -> 1 )"; auto const calls = parse(source); - BOOST_CHECK_EQUAL(calls.size(), 1); - - auto const& call = calls.at(0); - BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::MultiLine); - BOOST_CHECK_EQUAL(call.signature, "f(uint256)"); - ABI_CHECK(call.arguments.rawBytes(), toBigEndian(u256{1}) + toBigEndian(u256{2})); - BOOST_CHECK_EQUAL(call.expectations.failure, false); + BOOST_REQUIRE_EQUAL(calls.size(), 1); + testFunctionCall( + calls.at(0), + Mode::MultiLine, + "f(uint256)", + false, + fmt::encodeArgs(1, 2), + fmt::encodeArgs(1), + 0, + " This only throws at runtime " + ); } BOOST_AUTO_TEST_CASE(call_multiple_arguments) @@ -262,13 +297,15 @@ BOOST_AUTO_TEST_CASE(call_multiple_arguments) // 1 )"; auto const calls = parse(source); - BOOST_CHECK_EQUAL(calls.size(), 1); - - auto const& call = calls.at(0); - BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::MultiLine); - BOOST_CHECK_EQUAL(call.signature, "test(uint256,uint256)"); - ABI_CHECK(call.arguments.rawBytes(), toBigEndian(u256{1}) + toBigEndian(u256{2})); - BOOST_CHECK_EQUAL(call.expectations.failure, false); + BOOST_REQUIRE_EQUAL(calls.size(), 1); + testFunctionCall( + calls.at(0), + Mode::MultiLine, + "test(uint256,uint256)", + false, + fmt::encodeArgs(1, 2), + fmt::encodeArgs(1, 1) + ); } BOOST_AUTO_TEST_CASE(call_multiple_arguments_mixed_format) @@ -279,15 +316,74 @@ BOOST_AUTO_TEST_CASE(call_multiple_arguments_mixed_format) // -> -1, 2 )"; auto const calls = parse(source); - BOOST_CHECK_EQUAL(calls.size(), 1); - - auto const& call = calls.at(0); - BOOST_CHECK_EQUAL(call.displayMode, FunctionCall::DisplayMode::MultiLine); - BOOST_CHECK_EQUAL(call.signature, "test(uint256,uint256)"); - BOOST_CHECK_EQUAL(call.value, u256{314}); - ABI_CHECK(call.arguments.rawBytes(), toBigEndian(u256{1}) + toBigEndian(u256{-2})); - BOOST_CHECK_EQUAL(call.expectations.failure, false); - ABI_CHECK(call.expectations.rawBytes(), toBigEndian(u256{-1}) + toBigEndian(u256{2})); + BOOST_REQUIRE_EQUAL(calls.size(), 1); + testFunctionCall( + calls.at(0), + Mode::MultiLine, + "test(uint256,uint256)", + false, + fmt::encodeArgs(1, -2), + fmt::encodeArgs(-1, 2), + 314 + ); +} + +BOOST_AUTO_TEST_CASE(call_signature) +{ + char const* source = R"( + // f(uint256, uint8, string) -> FAILURE + // f(invalid, xyz, foo) -> FAILURE + )"; + auto const calls = parse(source); + BOOST_REQUIRE_EQUAL(calls.size(), 2); + testFunctionCall(calls.at(0), Mode::SingleLine, "f(uint256,uint8,string)", true); + testFunctionCall(calls.at(1), Mode::SingleLine, "f(invalid,xyz,foo)", true); +} + +BOOST_AUTO_TEST_CASE(call_signature_invalid) +{ + char const* source = R"( + // f(uint8,) -> FAILURE + )"; + BOOST_REQUIRE_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_expectations_missing) +{ + char const* source = R"( + // f())"; + BOOST_REQUIRE_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_ether_value_expectations_missing) +{ + char const* source = R"( + // f(), 0)"; + BOOST_REQUIRE_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_arguments_invalid) +{ + char const* source = R"( + // f(uint256): abc -> 1 + )"; + BOOST_REQUIRE_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_ether_value_invalid) +{ + char const* source = R"( + // f(uint256), abc : 1 -> 1 + )"; + BOOST_REQUIRE_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_ether_type_invalid) +{ + char const* source = R"( + // f(uint256), 2 btc : 1 -> 1 + )"; + BOOST_REQUIRE_THROW(parse(source), langutil::Error); } BOOST_AUTO_TEST_CASE(call_arguments_colon) @@ -296,7 +392,7 @@ BOOST_AUTO_TEST_CASE(call_arguments_colon) // h256(): // -> 1 )"; - BOOST_CHECK_THROW(parse(source), langutil::Error); + BOOST_REQUIRE_THROW(parse(source), langutil::Error); } BOOST_AUTO_TEST_CASE(call_arguments_newline_colon) @@ -306,11 +402,9 @@ BOOST_AUTO_TEST_CASE(call_arguments_newline_colon) // : // -> 1 )"; - BOOST_CHECK_THROW(parse(source), langutil::Error); + BOOST_REQUIRE_THROW(parse(source), langutil::Error); } - - BOOST_AUTO_TEST_SUITE_END() } From c9c45780233e2363eaed5251f90b786624f4691f Mon Sep 17 00:00:00 2001 From: Erik Kundt Date: Tue, 5 Feb 2019 16:52:19 +0100 Subject: [PATCH 074/109] Adds support for tuples in test file parser. --- test/libsolidity/util/TestFileParser.cpp | 178 ++++++++++-------- test/libsolidity/util/TestFileParser.h | 55 ++++-- test/libsolidity/util/TestFileParserTests.cpp | 123 +++++++++++- 3 files changed, 256 insertions(+), 100 deletions(-) diff --git a/test/libsolidity/util/TestFileParser.cpp b/test/libsolidity/util/TestFileParser.cpp index 2c304e773b36..5f1a420e9a59 100644 --- a/test/libsolidity/util/TestFileParser.cpp +++ b/test/libsolidity/util/TestFileParser.cpp @@ -31,6 +31,7 @@ using namespace langutil; using namespace solidity; using namespace dev::solidity::test; using namespace std; +using namespace soltest; namespace { @@ -47,14 +48,14 @@ namespace vector TestFileParser::parseFunctionCalls() { vector calls; - if (!accept(SoltToken::EOS)) + if (!accept(Token::EOS)) { - assert(m_scanner.currentToken() == SoltToken::Unknown); + assert(m_scanner.currentToken() == Token::Unknown); m_scanner.scanNextToken(); - while (!accept(SoltToken::EOS)) + while (!accept(Token::EOS)) { - if (!accept(SoltToken::Whitespace)) + if (!accept(Token::Whitespace)) { FunctionCall call; @@ -64,63 +65,47 @@ vector TestFileParser::parseFunctionCalls() /// token lookahead that checks parseParameter /// if the next token is an identifier. if (calls.empty()) - expect(SoltToken::Newline); + expect(Token::Newline); else - accept(SoltToken::Newline, true); + accept(Token::Newline, true); call.signature = parseFunctionSignature(); - if (accept(SoltToken::Comma, true)) + if (accept(Token::Comma, true)) call.value = parseFunctionCallValue(); - if (accept(SoltToken::Colon, true)) + if (accept(Token::Colon, true)) call.arguments = parseFunctionCallArguments(); - if (accept(SoltToken::Newline, true)) + if (accept(Token::Newline, true)) call.displayMode = FunctionCall::DisplayMode::MultiLine; call.arguments.comment = parseComment(); - if (accept(SoltToken::Newline, true)) + if (accept(Token::Newline, true)) call.displayMode = FunctionCall::DisplayMode::MultiLine; - expect(SoltToken::Arrow); + expect(Token::Arrow); call.expectations = parseFunctionCallExpectations(); call.expectations.comment = parseComment(); calls.emplace_back(std::move(call)); } - else - m_scanner.scanNextToken(); } } return calls; } -string TestFileParser::formatToken(SoltToken _token) +bool TestFileParser::accept(soltest::Token _token, bool const _expect) { - switch (_token) - { -#define T(name, string, precedence) case SoltToken::name: return string; - SOLT_TOKEN_LIST(T, T) -#undef T - default: // Token::NUM_TOKENS: - return ""; - } -} - -bool TestFileParser::accept(SoltToken _token, bool const _expect) -{ - if (m_scanner.currentToken() == _token) - { - if (_expect) - expect(_token); - return true; - } - return false; + if (m_scanner.currentToken() != _token) + return false; + if (_expect) + return expect(_token); + return true; } -bool TestFileParser::expect(SoltToken _token, bool const _advance) +bool TestFileParser::expect(soltest::Token _token, bool const _advance) { - if (m_scanner.currentToken() != _token) + if (m_scanner.currentToken() != _token || m_scanner.currentToken() == Token::Invalid) throw Error( Error::Type::ParserError, "Unexpected " + formatToken(m_scanner.currentToken()) + ": \"" + @@ -135,32 +120,35 @@ bool TestFileParser::expect(SoltToken _token, bool const _advance) string TestFileParser::parseFunctionSignature() { string signature = m_scanner.currentLiteral(); - expect(SoltToken::Identifier); + expect(Token::Identifier); - signature += formatToken(SoltToken::LParen); - expect(SoltToken::LParen); + signature += formatToken(Token::LParen); + expect(Token::LParen); - while (!accept(SoltToken::RParen)) + string parameters; + if (!accept(Token::RParen, false)) + parameters = parseIdentifierOrTuple(); + + while (accept(Token::Comma)) { - signature += m_scanner.currentLiteral(); - expect(SoltToken::Identifier); - while (accept(SoltToken::Comma)) - { - signature += m_scanner.currentLiteral(); - expect(SoltToken::Comma); - signature += m_scanner.currentLiteral(); - expect(SoltToken::Identifier); - } + parameters += formatToken(Token::Comma); + expect(Token::Comma); + parameters += parseIdentifierOrTuple(); } - signature += formatToken(SoltToken::RParen); - expect(SoltToken::RParen); + if (accept(Token::Arrow, true)) + throw Error(Error::Type::ParserError, "Invalid signature detected: " + signature); + + signature += parameters; + + expect(Token::RParen); + signature += formatToken(Token::RParen); return signature; } u256 TestFileParser::parseFunctionCallValue() { u256 value = convertNumber(parseNumber()); - expect(SoltToken::Ether); + expect(Token::Ether); return value; } @@ -173,7 +161,7 @@ FunctionCallArgs TestFileParser::parseFunctionCallArguments() throw Error(Error::Type::ParserError, "No argument provided."); arguments.parameters.emplace_back(param); - while (accept(SoltToken::Comma, true)) + while (accept(Token::Comma, true)) arguments.parameters.emplace_back(parseParameter()); return arguments; } @@ -190,7 +178,7 @@ FunctionCallExpectations TestFileParser::parseFunctionCallExpectations() } expectations.result.emplace_back(param); - while (accept(SoltToken::Comma, true)) + while (accept(Token::Comma, true)) expectations.result.emplace_back(parseParameter()); /// We have always one virtual parameter in the parameter list. @@ -203,7 +191,7 @@ FunctionCallExpectations TestFileParser::parseFunctionCallExpectations() Parameter TestFileParser::parseParameter() { Parameter parameter; - if (accept(SoltToken::Newline, true)) + if (accept(Token::Newline, true)) parameter.format.newline = true; auto literal = parseABITypeLiteral(); parameter.rawBytes = literal.first; @@ -218,20 +206,20 @@ pair TestFileParser::parseABITypeLiteral() u256 number{0}; ABIType abiType{ABIType::None, 0}; - if (accept(SoltToken::Sub)) + if (accept(Token::Sub)) { abiType = ABIType{ABIType::SignedDec, 32}; - expect(SoltToken::Sub); + expect(Token::Sub); number = convertNumber(parseNumber()) * -1; } else { - if (accept(SoltToken::Number)) + if (accept(Token::Number)) { abiType = ABIType{ABIType::UnsignedDec, 32}; number = convertNumber(parseNumber()); } - else if (accept(SoltToken::Failure, true)) + else if (accept(Token::Failure, true)) { abiType = ABIType{ABIType::Failure, 0}; return make_pair(bytes{}, abiType); @@ -245,10 +233,35 @@ pair TestFileParser::parseABITypeLiteral() } } +string TestFileParser::parseIdentifierOrTuple() +{ + string identOrTuple; + + if (accept(Token::Identifier)) + { + identOrTuple = m_scanner.currentLiteral(); + expect(Token::Identifier); + return identOrTuple; + } + expect(Token::LParen); + identOrTuple += formatToken(Token::LParen); + identOrTuple += parseIdentifierOrTuple(); + + while (accept(Token::Comma)) + { + identOrTuple += formatToken(Token::Comma); + expect(Token::Comma); + identOrTuple += parseIdentifierOrTuple(); + } + expect(Token::RParen); + identOrTuple += formatToken(Token::RParen); + return identOrTuple; +} + string TestFileParser::parseComment() { string comment = m_scanner.currentLiteral(); - if (accept(SoltToken::Comment, true)) + if (accept(Token::Comment, true)) return comment; return string{}; } @@ -256,7 +269,7 @@ string TestFileParser::parseComment() string TestFileParser::parseNumber() { string literal = m_scanner.currentLiteral(); - expect(SoltToken::Number); + expect(Token::Number); return literal; } @@ -281,18 +294,21 @@ void TestFileParser::Scanner::readStream(istream& _stream) void TestFileParser::Scanner::scanNextToken() { - auto detectToken = [](std::string const& _literal = "") -> TokenDesc { - if (_literal == "ether") return TokenDesc{SoltToken::Ether, _literal}; - if (_literal == "FAILURE") return TokenDesc{SoltToken::Failure, _literal}; - return TokenDesc{SoltToken::Identifier, _literal}; + // Make code coverage happy. + assert(formatToken(Token::NUM_TOKENS) == ""); + + auto detectKeyword = [](std::string const& _literal = "") -> TokenDesc { + if (_literal == "ether") return TokenDesc{Token::Ether, _literal}; + if (_literal == "FAILURE") return TokenDesc{Token::Failure, _literal}; + return TokenDesc{Token::Identifier, _literal}; }; - auto selectToken = [this](SoltToken _token, std::string const& _literal = "") -> TokenDesc { + auto selectToken = [this](Token _token, std::string const& _literal = "") -> TokenDesc { advance(); return make_pair(_token, !_literal.empty() ? _literal : formatToken(_token)); }; - TokenDesc token = make_pair(SoltToken::Unknown, ""); + TokenDesc token = make_pair(Token::Unknown, ""); do { switch(current()) @@ -300,50 +316,50 @@ void TestFileParser::Scanner::scanNextToken() case '/': advance(); if (current() == '/') - token = selectToken(SoltToken::Newline); + token = selectToken(Token::Newline); + else + token = selectToken(Token::Invalid); break; case '-': if (peek() == '>') { advance(); - token = selectToken(SoltToken::Arrow); + token = selectToken(Token::Arrow); } else - token = selectToken(SoltToken::Sub); + token = selectToken(Token::Sub); break; case ':': - token = selectToken(SoltToken::Colon); + token = selectToken(Token::Colon); break; case '#': - token = selectToken(SoltToken::Comment, scanComment()); + token = selectToken(Token::Comment, scanComment()); break; case ',': - token = selectToken(SoltToken::Comma); + token = selectToken(Token::Comma); break; case '(': - token = selectToken(SoltToken::LParen); + token = selectToken(Token::LParen); break; case ')': - token = selectToken(SoltToken::RParen); + token = selectToken(Token::RParen); break; default: if (isIdentifierStart(current())) { - TokenDesc detectedToken = detectToken(scanIdentifierOrKeyword()); + TokenDesc detectedToken = detectKeyword(scanIdentifierOrKeyword()); token = selectToken(detectedToken.first, detectedToken.second); } else if (isdigit(current())) - token = selectToken(SoltToken::Number, scanNumber()); + token = selectToken(Token::Number, scanNumber()); else if (isspace(current())) - token = selectToken(SoltToken::Whitespace); + token = selectToken(Token::Whitespace); else if (isEndOfLine()) - token = selectToken(SoltToken::EOS); - else - token = selectToken(SoltToken::Invalid); + token = selectToken(Token::EOS); break; } } - while (token.first == SoltToken::Whitespace); + while (token.first == Token::Whitespace); m_currentToken = token; } diff --git a/test/libsolidity/util/TestFileParser.h b/test/libsolidity/util/TestFileParser.h index cbc8717cbe86..6b79008833fd 100644 --- a/test/libsolidity/util/TestFileParser.h +++ b/test/libsolidity/util/TestFileParser.h @@ -33,7 +33,7 @@ namespace test { /** - * All SOLT (or SOLTest) tokens. + * All soltest tokens. */ #define SOLT_TOKEN_LIST(T, K) \ T(Unknown, "unknown", 0) \ @@ -62,12 +62,30 @@ namespace test /* special keywords */ \ K(Failure, "FAILURE", 0) \ -enum class SoltToken : unsigned int { -#define T(name, string, precedence) name, - SOLT_TOKEN_LIST(T, T) - NUM_TOKENS -#undef T -}; +namespace soltest +{ + enum class Token : unsigned int { + #define T(name, string, precedence) name, + SOLT_TOKEN_LIST(T, T) + NUM_TOKENS + #undef T + }; + + /// Prints a friendly string representation of \param _token. + inline std::string formatToken(Token _token) + { + switch (_token) + { + #define T(name, string, precedence) case Token::name: return string; + SOLT_TOKEN_LIST(T, T) + #undef T + default: // Token::NUM_TOKENS: + return ""; + } + } +} + + /** * The purpose of the ABI type is the storage of type information @@ -92,7 +110,7 @@ struct ABIType /** * Helper that can hold format information retrieved - * while scanning through a parameter list in sol_t. + * while scanning through a parameter list in soltest. */ struct FormatInfo { @@ -133,7 +151,7 @@ using ParameterList = std::vector; */ struct FunctionCallExpectations { - /// Representation of the comma-separated (or empty) list of expectated result values + /// Representation of the comma-separated (or empty) list of expected result values /// attached to the function call object. It is checked against the actual result of /// a function call when used in test framework. ParameterList result; @@ -237,10 +255,8 @@ class TestFileParser /// of its arguments or expected results. std::vector parseFunctionCalls(); - /// Prints a friendly string representation of \param _token. - static std::string formatToken(SoltToken _token); - private: + using Token = soltest::Token; /** * Token scanner that is used internally to abstract away character traversal. */ @@ -257,7 +273,7 @@ class TestFileParser /// Reads character stream and creates token. void scanNextToken(); - SoltToken currentToken() { return m_currentToken.first; } + soltest::Token currentToken() { return m_currentToken.first; } std::string currentLiteral() { return m_currentToken.second; } std::string scanComment(); @@ -265,7 +281,7 @@ class TestFileParser std::string scanNumber(); private: - using TokenDesc = std::pair; + using TokenDesc = std::pair; /// Advances current position in the input stream. void advance() { ++m_char; } @@ -284,8 +300,8 @@ class TestFileParser TokenDesc m_currentToken; }; - bool accept(SoltToken _token, bool const _expect = false); - bool expect(SoltToken _token, bool const _advance = true); + bool accept(soltest::Token _token, bool const _expect = false); + bool expect(soltest::Token _token, bool const _advance = true); /// Parses a function call signature in the form of f(uint256, ...). std::string parseFunctionSignature(); @@ -319,7 +335,12 @@ class TestFileParser /// if data type is not supported. std::pair parseABITypeLiteral(); - /// Parses a comment + /// Recursively parses an identifier or a tuple definition that contains identifiers + /// and / or parentheses like `((uint, uint), (uint, (uint, uint)), uint)`. + std::string parseIdentifierOrTuple(); + + /// Parses a comment that is defined like this: + /// # A nice comment. # std::string parseComment(); /// Parses the current number literal. diff --git a/test/libsolidity/util/TestFileParserTests.cpp b/test/libsolidity/util/TestFileParserTests.cpp index 30fb2d765a9e..5b89341da4e8 100644 --- a/test/libsolidity/util/TestFileParserTests.cpp +++ b/test/libsolidity/util/TestFileParserTests.cpp @@ -265,6 +265,61 @@ BOOST_AUTO_TEST_CASE(call_arguments) ); } +BOOST_AUTO_TEST_CASE(call_arguments_tuple) +{ + char const* source = R"( + // f((uint256, bytes32), uint256) -> + // f((uint8), uint8) -> + )"; + auto const calls = parse(source); + BOOST_REQUIRE_EQUAL(calls.size(), 2); + testFunctionCall(calls.at(0), Mode::SingleLine, "f((uint256,bytes32),uint256)", false); + testFunctionCall(calls.at(1), Mode::SingleLine, "f((uint8),uint8)", false); +} + +BOOST_AUTO_TEST_CASE(call_arguments_tuple_of_tuples) +{ + char const* source = R"( + // f(((uint256, bytes32), bytes32), uint256) + // # f(S memory s, uint256 b) # + // -> + )"; + auto const calls = parse(source); + BOOST_REQUIRE_EQUAL(calls.size(), 1); + testFunctionCall( + calls.at(0), + Mode::MultiLine, + "f(((uint256,bytes32),bytes32),uint256)", + false, + fmt::encodeArgs(), + fmt::encodeArgs(), + 0, + " f(S memory s, uint256 b) " + ); +} + +BOOST_AUTO_TEST_CASE(call_arguments_recursive_tuples) +{ + char const* source = R"( + // f(((((bytes, bytes, bytes), bytes), bytes), bytes), bytes) -> + // f(((((bytes, bytes, (bytes)), bytes), bytes), (bytes, bytes)), (bytes, bytes)) -> + )"; + auto const calls = parse(source); + BOOST_REQUIRE_EQUAL(calls.size(), 2); + testFunctionCall( + calls.at(0), + Mode::SingleLine, + "f(((((bytes,bytes,bytes),bytes),bytes),bytes),bytes)", + false + ); + testFunctionCall( + calls.at(1), + Mode::SingleLine, + "f(((((bytes,bytes,(bytes)),bytes),bytes),(bytes,bytes)),(bytes,bytes))", + false + ); +} + BOOST_AUTO_TEST_CASE(call_arguments_mismatch) { char const* source = R"( @@ -333,18 +388,58 @@ BOOST_AUTO_TEST_CASE(call_signature) char const* source = R"( // f(uint256, uint8, string) -> FAILURE // f(invalid, xyz, foo) -> FAILURE - )"; + )"; auto const calls = parse(source); BOOST_REQUIRE_EQUAL(calls.size(), 2); testFunctionCall(calls.at(0), Mode::SingleLine, "f(uint256,uint8,string)", true); testFunctionCall(calls.at(1), Mode::SingleLine, "f(invalid,xyz,foo)", true); } +BOOST_AUTO_TEST_CASE(call_newline_invalid) +{ + char const* source = R"( + / + )"; + BOOST_REQUIRE_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_invalid) +{ + char const* source = R"( + / f() -> + )"; + BOOST_REQUIRE_THROW(parse(source), langutil::Error); +} + BOOST_AUTO_TEST_CASE(call_signature_invalid) { char const* source = R"( // f(uint8,) -> FAILURE - )"; + )"; + BOOST_REQUIRE_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_arguments_tuple_invalid) +{ + char const* source = R"( + // f((uint8,) -> FAILURE + )"; + BOOST_REQUIRE_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_arguments_tuple_invalid_empty) +{ + char const* source = R"( + // f(uint8, ()) -> FAILURE + )"; + BOOST_REQUIRE_THROW(parse(source), langutil::Error); +} + +BOOST_AUTO_TEST_CASE(call_arguments_tuple_invalid_parantheses) +{ + char const* source = R"( + // f((uint8,() -> FAILURE + )"; BOOST_REQUIRE_THROW(parse(source), langutil::Error); } @@ -370,6 +465,14 @@ BOOST_AUTO_TEST_CASE(call_arguments_invalid) BOOST_REQUIRE_THROW(parse(source), langutil::Error); } +BOOST_AUTO_TEST_CASE(call_arguments_invalid_decimal) +{ + char const* source = R"( + // sig(): 0.h3 -> + )"; + BOOST_REQUIRE_THROW(parse(source), langutil::Error); +} + BOOST_AUTO_TEST_CASE(call_ether_value_invalid) { char const* source = R"( @@ -378,6 +481,14 @@ BOOST_AUTO_TEST_CASE(call_ether_value_invalid) BOOST_REQUIRE_THROW(parse(source), langutil::Error); } +BOOST_AUTO_TEST_CASE(call_ether_value_invalid_decimal) +{ + char const* source = R"( + // sig(): 0.1hd ether -> + )"; + BOOST_REQUIRE_THROW(parse(source), langutil::Error); +} + BOOST_AUTO_TEST_CASE(call_ether_type_invalid) { char const* source = R"( @@ -405,6 +516,14 @@ BOOST_AUTO_TEST_CASE(call_arguments_newline_colon) BOOST_REQUIRE_THROW(parse(source), langutil::Error); } +BOOST_AUTO_TEST_CASE(call_arrow_missing) +{ + char const* source = R"( + // h256() + )"; + BOOST_REQUIRE_THROW(parse(source), langutil::Error); +} + BOOST_AUTO_TEST_SUITE_END() } From ee28cb65a6dd9b83fae8f5f71f6179e8d1d10f49 Mon Sep 17 00:00:00 2001 From: Mathias Baumann Date: Wed, 6 Feb 2019 20:43:53 +0100 Subject: [PATCH 075/109] Add tests to increase coverage of TypeChecker --- .../inheritance/reference_non_base_ctor.sol | 6 ++++++ .../396_invalid_mobile_type.sol | 4 +++- .../585_abi_decode_with_unsupported_types.sol | 8 ++++++++ .../586_invalid_types_for_constructor_call.sol | 12 ++++++++++++ .../587_event_param_type_outside_storage.sol | 7 +++++++ .../588_inline_assembly_two_stack_slots.sol | 10 ++++++++++ .../single_return_mismatching_type.sol | 9 +++++++++ .../abidecode/abi_decode_invalid_arg_type.sol | 8 ++++++++ 8 files changed, 63 insertions(+), 1 deletion(-) create mode 100644 test/libsolidity/syntaxTests/inheritance/reference_non_base_ctor.sol create mode 100644 test/libsolidity/syntaxTests/nameAndTypeResolution/585_abi_decode_with_unsupported_types.sol create mode 100644 test/libsolidity/syntaxTests/nameAndTypeResolution/586_invalid_types_for_constructor_call.sol create mode 100644 test/libsolidity/syntaxTests/nameAndTypeResolution/587_event_param_type_outside_storage.sol create mode 100644 test/libsolidity/syntaxTests/nameAndTypeResolution/588_inline_assembly_two_stack_slots.sol create mode 100644 test/libsolidity/syntaxTests/returnExpressions/single_return_mismatching_type.sol create mode 100644 test/libsolidity/syntaxTests/specialFunctions/abidecode/abi_decode_invalid_arg_type.sol diff --git a/test/libsolidity/syntaxTests/inheritance/reference_non_base_ctor.sol b/test/libsolidity/syntaxTests/inheritance/reference_non_base_ctor.sol new file mode 100644 index 000000000000..6f1932ffbe02 --- /dev/null +++ b/test/libsolidity/syntaxTests/inheritance/reference_non_base_ctor.sol @@ -0,0 +1,6 @@ +contract X {} +contract D { + constructor() X(5) public {} +} +// ---- +// TypeError: (45-49): Referenced declaration is neither modifier nor base class. diff --git a/test/libsolidity/syntaxTests/nameAndTypeResolution/396_invalid_mobile_type.sol b/test/libsolidity/syntaxTests/nameAndTypeResolution/396_invalid_mobile_type.sol index 536dd317ca10..d1f91a2371ea 100644 --- a/test/libsolidity/syntaxTests/nameAndTypeResolution/396_invalid_mobile_type.sol +++ b/test/libsolidity/syntaxTests/nameAndTypeResolution/396_invalid_mobile_type.sol @@ -1,8 +1,10 @@ contract C { function f() public { // Invalid number + 78901234567890123456789012345678901234567890123456789345678901234567890012345678012345678901234567; [1, 78901234567890123456789012345678901234567890123456789345678901234567890012345678012345678901234567]; } } // ---- -// TypeError: (93-191): Invalid rational number. +// TypeError: (89-187): Invalid rational number. +// TypeError: (205-303): Invalid rational number. diff --git a/test/libsolidity/syntaxTests/nameAndTypeResolution/585_abi_decode_with_unsupported_types.sol b/test/libsolidity/syntaxTests/nameAndTypeResolution/585_abi_decode_with_unsupported_types.sol new file mode 100644 index 000000000000..f2ad3f5635fb --- /dev/null +++ b/test/libsolidity/syntaxTests/nameAndTypeResolution/585_abi_decode_with_unsupported_types.sol @@ -0,0 +1,8 @@ +contract C { + struct s { uint a; uint b; } + function f() pure public { + abi.decode("", (s)); + } +} +// ---- +// TypeError: (98-99): Decoding type struct C.s memory not supported. diff --git a/test/libsolidity/syntaxTests/nameAndTypeResolution/586_invalid_types_for_constructor_call.sol b/test/libsolidity/syntaxTests/nameAndTypeResolution/586_invalid_types_for_constructor_call.sol new file mode 100644 index 000000000000..b1416ddec5fa --- /dev/null +++ b/test/libsolidity/syntaxTests/nameAndTypeResolution/586_invalid_types_for_constructor_call.sol @@ -0,0 +1,12 @@ +contract C { + constructor(bytes32 _arg) public { + } +} + +contract A { + function f() public { + new C((1234)); + } +} +// ---- +// TypeError: (115-121): Invalid type for argument in function call. Invalid implicit conversion from int_const 1234 to bytes32 requested. diff --git a/test/libsolidity/syntaxTests/nameAndTypeResolution/587_event_param_type_outside_storage.sol b/test/libsolidity/syntaxTests/nameAndTypeResolution/587_event_param_type_outside_storage.sol new file mode 100644 index 000000000000..c076017a87af --- /dev/null +++ b/test/libsolidity/syntaxTests/nameAndTypeResolution/587_event_param_type_outside_storage.sol @@ -0,0 +1,7 @@ +contract c { + event e(uint indexed a, mapping(uint => uint) indexed b, bool indexed c, uint indexed d, uint indexed e) anonymous; +} +// ---- +// TypeError: (41-72): Type is required to live outside storage. +// TypeError: (41-72): Internal or recursive type is not allowed as event parameter type. +// TypeError: (17-132): More than 4 indexed arguments for anonymous event. diff --git a/test/libsolidity/syntaxTests/nameAndTypeResolution/588_inline_assembly_two_stack_slots.sol b/test/libsolidity/syntaxTests/nameAndTypeResolution/588_inline_assembly_two_stack_slots.sol new file mode 100644 index 000000000000..c7084a0eef95 --- /dev/null +++ b/test/libsolidity/syntaxTests/nameAndTypeResolution/588_inline_assembly_two_stack_slots.sol @@ -0,0 +1,10 @@ +contract C { + function f() pure external { + function() external two_stack_slots; + assembly { + let x := two_stack_slots + } + } +} +// ---- +// TypeError: (132-147): Only types that use one stack slot are supported. diff --git a/test/libsolidity/syntaxTests/returnExpressions/single_return_mismatching_type.sol b/test/libsolidity/syntaxTests/returnExpressions/single_return_mismatching_type.sol new file mode 100644 index 000000000000..0a9b993ee265 --- /dev/null +++ b/test/libsolidity/syntaxTests/returnExpressions/single_return_mismatching_type.sol @@ -0,0 +1,9 @@ +contract C +{ + function g() public pure returns (uint) + { + return "string"; + } +} +// ---- +// TypeError: (78-86): Return argument type literal_string "string" is not implicitly convertible to expected type (type of first return variable) uint256. diff --git a/test/libsolidity/syntaxTests/specialFunctions/abidecode/abi_decode_invalid_arg_type.sol b/test/libsolidity/syntaxTests/specialFunctions/abidecode/abi_decode_invalid_arg_type.sol new file mode 100644 index 000000000000..e418390c5fb7 --- /dev/null +++ b/test/libsolidity/syntaxTests/specialFunctions/abidecode/abi_decode_invalid_arg_type.sol @@ -0,0 +1,8 @@ +contract C { + function f() public pure { + abi.decode(uint, uint); + } +} +// ---- +// TypeError: (57-61): Invalid type for argument in function call. Invalid implicit conversion from type(uint256) to bytes memory requested. +// TypeError: (63-67): The second argument to "abi.decode" has to be a tuple of types. From 5fb79f5e371f30433685d78b90739f2d98f4ff02 Mon Sep 17 00:00:00 2001 From: Mathias Baumann Date: Wed, 6 Feb 2019 13:11:04 +0100 Subject: [PATCH 076/109] Turn unreachable error statements into asserts --- libsolidity/analysis/TypeChecker.cpp | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/libsolidity/analysis/TypeChecker.cpp b/libsolidity/analysis/TypeChecker.cpp index 225735bac8c4..4b1f28cda6cb 100644 --- a/libsolidity/analysis/TypeChecker.cpp +++ b/libsolidity/analysis/TypeChecker.cpp @@ -287,8 +287,7 @@ void TypeChecker::endVisit(UsingForDirective const& _usingFor) bool TypeChecker::visit(StructDefinition const& _struct) { for (ASTPointer const& member: _struct.members()) - if (!type(*member)->canBeStored()) - m_errorReporter.typeError(member->location(), "Type cannot be used in struct."); + solAssert(type(*member)->canBeStored(), "Type cannot be used in struct."); // Check recursion, fatal error if detected. auto visitor = [&](StructDefinition const& _struct, CycleDetector& _cycleDetector, size_t _depth) @@ -615,8 +614,7 @@ void TypeChecker::endVisit(FunctionTypeName const& _funType) { FunctionType const& fun = dynamic_cast(*_funType.annotation().type); if (fun.kind() == FunctionType::Kind::External) - if (!fun.canBeUsedExternally(false)) - m_errorReporter.typeError(_funType.location(), "External function type uses internal types."); + solAssert(fun.canBeUsedExternally(false), "External function type uses internal types."); } bool TypeChecker::visit(InlineAssembly const& _inlineAssembly) @@ -887,8 +885,7 @@ bool TypeChecker::visit(VariableDeclarationStatement const& _statement) if (ref->dataStoredIn(DataLocation::Storage)) { string errorText{"Uninitialized storage pointer."}; - if (varDecl.referenceLocation() == VariableDeclaration::Location::Unspecified) - errorText += " Did you mean ' memory " + varDecl.name() + "'?"; + solAssert(varDecl.referenceLocation() != VariableDeclaration::Location::Unspecified, "Expected a specified location at this point"); solAssert(m_scope, ""); m_errorReporter.declarationError(varDecl.location(), errorText); } @@ -956,10 +953,7 @@ bool TypeChecker::visit(VariableDeclarationStatement const& _statement) solAssert(false, ""); } else if (*var.annotation().type == TupleType()) - m_errorReporter.typeError( - var.location(), - "Cannot declare variable with void (empty tuple) type." - ); + solAssert(false, "Cannot declare variable with void (empty tuple) type."); else if (valueComponentType->category() == Type::Category::RationalNumber) { string typeName = var.annotation().type->toString(true); From fc1825825040e10f2a8ebd8cc1a2aadda5302b84 Mon Sep 17 00:00:00 2001 From: chriseth Date: Thu, 7 Feb 2019 12:02:15 +0100 Subject: [PATCH 077/109] Remove type check that is covered by assertions. --- libsolidity/codegen/ABIFunctions.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libsolidity/codegen/ABIFunctions.cpp b/libsolidity/codegen/ABIFunctions.cpp index 2c043b277f2d..1652d427800b 100644 --- a/libsolidity/codegen/ABIFunctions.cpp +++ b/libsolidity/codegen/ABIFunctions.cpp @@ -569,7 +569,7 @@ string ABIFunctions::abiEncodingFunction( )"); templ("functionName", functionName); - if (_from.dataStoredIn(DataLocation::Storage) && to.isValueType()) + if (_from.dataStoredIn(DataLocation::Storage)) { // special case: convert storage reference type to value type - this is only // possible for library calls where we just forward the storage reference From ef6a76ce672378a95803409c84c69e121393052e Mon Sep 17 00:00:00 2001 From: Christian Parpart Date: Fri, 1 Feb 2019 15:39:18 +0100 Subject: [PATCH 078/109] libdevcore: Introduces a generalized AnsiColorized, an improved FormattedScope a future commit/PR could replace existing code to use AnsiColorized and remove the old implementation of FormattedScope. --- libdevcore/AnsiColorized.h | 91 ++++++++++++++++++++++++++++++++++++++ libdevcore/CMakeLists.txt | 1 + 2 files changed, 92 insertions(+) create mode 100644 libdevcore/AnsiColorized.h diff --git a/libdevcore/AnsiColorized.h b/libdevcore/AnsiColorized.h new file mode 100644 index 000000000000..0ee54ab0adca --- /dev/null +++ b/libdevcore/AnsiColorized.h @@ -0,0 +1,91 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ + +#pragma once + +#include +#include + +namespace dev +{ + +namespace formatting +{ + +// control codes +static constexpr char const* RESET = "\033[0m"; +static constexpr char const* INVERSE = "\033[7m"; +static constexpr char const* BOLD = "\033[1m"; +static constexpr char const* BRIGHT = BOLD; + +// standard foreground colors +static constexpr char const* BLACK = "\033[30m"; +static constexpr char const* RED = "\033[31m"; +static constexpr char const* GREEN = "\033[32m"; +static constexpr char const* YELLOW = "\033[33m"; +static constexpr char const* BLUE = "\033[34m"; +static constexpr char const* MAGENTA = "\033[35m"; +static constexpr char const* CYAN = "\033[36m"; +static constexpr char const* WHITE = "\033[37m"; + +// standard background colors +static constexpr char const* BLACK_BACKGROUND = "\033[40m"; +static constexpr char const* RED_BACKGROUND = "\033[41m"; +static constexpr char const* GREEN_BACKGROUND = "\033[42m"; +static constexpr char const* YELLOW_BACKGROUND = "\033[43m"; +static constexpr char const* BLUE_BACKGROUND = "\033[44m"; +static constexpr char const* MAGENTA_BACKGROUND = "\033[45m"; +static constexpr char const* CYAN_BACKGROUND = "\033[46m"; +static constexpr char const* WHITE_BACKGROUND = "\033[47m"; + +// 256-bit-colors (incomplete set) +static constexpr char const* RED_BACKGROUND_256 = "\033[48;5;160m"; +static constexpr char const* ORANGE_BACKGROUND_256 = "\033[48;5;166m"; + +} + +/// AnsiColorized provides a convenience helper to colorize ostream with formatting-reset assured. +class AnsiColorized +{ +public: + AnsiColorized(std::ostream& _os, bool const _enabled, std::vector&& _formatting): + m_stream{_os}, m_enabled{_enabled}, m_codes{std::move(_formatting)} + { + if (m_enabled) + for (auto const& code: m_codes) + m_stream << code; + } + + ~AnsiColorized() + { + if (m_enabled) + m_stream << formatting::RESET; + } + + template + std::ostream& operator<<(T&& _t) + { + return m_stream << std::forward(_t); + } + +private: + std::ostream& m_stream; + bool m_enabled; + std::vector m_codes; +}; + +} diff --git a/libdevcore/CMakeLists.txt b/libdevcore/CMakeLists.txt index 193fa41d0325..b92cb5db5492 100644 --- a/libdevcore/CMakeLists.txt +++ b/libdevcore/CMakeLists.txt @@ -1,5 +1,6 @@ set(sources Algorithms.h + AnsiColorized.h Assertions.h Common.h CommonData.cpp From 3d4b0f45daa27aa89dd2a3763119f717e42dc598 Mon Sep 17 00:00:00 2001 From: Christian Parpart Date: Tue, 27 Nov 2018 00:21:53 +0100 Subject: [PATCH 079/109] liblangutil: refactors SourceReferenceFormatter error formatting for pretty and colored output. * Refactors output format in a way it is (or should at least be) more readable. (NB.: As source of inspiration, I chose the rustc compiler output.) * Adds color support to the stream output. * Also improves multiline source formatting (i.e. truncating too long lines, like done with single lines already) * solc: adds flags --color (force terminal colors) and --no-color (disable autodetection) * solc: adds --new-reporter to give output in *new* formatting (colored or not) * Changelog adapted accordingly. --- Changelog.md | 3 +- liblangutil/CMakeLists.txt | 2 + liblangutil/SourceReferenceFormatter.h | 13 +- liblangutil/SourceReferenceFormatterHuman.cpp | 136 ++++++++++++++++++ liblangutil/SourceReferenceFormatterHuman.h | 80 +++++++++++ solc/CommandLineInterface.cpp | 40 +++++- solc/CommandLineInterface.h | 2 + 7 files changed, 264 insertions(+), 12 deletions(-) create mode 100644 liblangutil/SourceReferenceFormatterHuman.cpp create mode 100644 liblangutil/SourceReferenceFormatterHuman.h diff --git a/Changelog.md b/Changelog.md index 1ec6faf1e7e7..89143ee5942b 100644 --- a/Changelog.md +++ b/Changelog.md @@ -13,7 +13,8 @@ Language Features: Compiler Features: * C API (``libsolc`` / raw ``soljson.js``): Introduce ``solidity_free`` method which releases all internal buffers to save memory. - + * Commandline interface: Adds new option ``--new-reporter`` for improved diagnostics formatting + along with ``--color`` and ``--no-color`` for colorized output to be forced (or explicitly disabled). Bugfixes: diff --git a/liblangutil/CMakeLists.txt b/liblangutil/CMakeLists.txt index b172108b3454..4539376f0986 100644 --- a/liblangutil/CMakeLists.txt +++ b/liblangutil/CMakeLists.txt @@ -16,6 +16,8 @@ set(sources SourceReferenceExtractor.h SourceReferenceFormatter.cpp SourceReferenceFormatter.h + SourceReferenceFormatterHuman.cpp + SourceReferenceFormatterHuman.h Token.cpp Token.h UndefMacros.h diff --git a/liblangutil/SourceReferenceFormatter.h b/liblangutil/SourceReferenceFormatter.h index 9f05f430352b..c01b360eb2f2 100644 --- a/liblangutil/SourceReferenceFormatter.h +++ b/liblangutil/SourceReferenceFormatter.h @@ -44,11 +44,14 @@ class SourceReferenceFormatter m_stream(_stream) {} + virtual ~SourceReferenceFormatter() = default; + /// Prints source location if it is given. - void printSourceLocation(SourceLocation const* _location); - void printSourceLocation(SourceReference const& _ref); - void printExceptionInformation(dev::Exception const& _error, std::string const& _category); - void printExceptionInformation(SourceReferenceExtractor::Message const& _msg); + virtual void printSourceLocation(SourceReference const& _ref); + virtual void printExceptionInformation(SourceReferenceExtractor::Message const& _msg); + + virtual void printSourceLocation(SourceLocation const* _location); + virtual void printExceptionInformation(dev::Exception const& _error, std::string const& _category); static std::string formatExceptionInformation( dev::Exception const& _exception, @@ -62,7 +65,7 @@ class SourceReferenceFormatter return errorOutput.str(); } -private: +protected: /// Prints source name if location is given. void printSourceName(SourceReference const& _ref); diff --git a/liblangutil/SourceReferenceFormatterHuman.cpp b/liblangutil/SourceReferenceFormatterHuman.cpp new file mode 100644 index 000000000000..f246c2641bdf --- /dev/null +++ b/liblangutil/SourceReferenceFormatterHuman.cpp @@ -0,0 +1,136 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ +/** + * Formatting functions for errors referencing positions and locations in the source. + */ + +#include +#include +#include +#include +#include + +using namespace std; +using namespace dev; +using namespace dev::formatting; +using namespace langutil; + +AnsiColorized SourceReferenceFormatterHuman::normalColored() const +{ + return AnsiColorized(m_stream, m_colored, {WHITE}); +} + +AnsiColorized SourceReferenceFormatterHuman::frameColored() const +{ + return AnsiColorized(m_stream, m_colored, {BOLD, BLUE}); +} + +AnsiColorized SourceReferenceFormatterHuman::errorColored() const +{ + return AnsiColorized(m_stream, m_colored, {BOLD, RED}); +} + +AnsiColorized SourceReferenceFormatterHuman::messageColored() const +{ + return AnsiColorized(m_stream, m_colored, {BOLD, WHITE}); +} + +AnsiColorized SourceReferenceFormatterHuman::secondaryColored() const +{ + return AnsiColorized(m_stream, m_colored, {BOLD, CYAN}); +} + +AnsiColorized SourceReferenceFormatterHuman::highlightColored() const +{ + return AnsiColorized(m_stream, m_colored, {YELLOW}); +} + +AnsiColorized SourceReferenceFormatterHuman::diagColored() const +{ + return AnsiColorized(m_stream, m_colored, {BOLD, YELLOW}); +} + +void SourceReferenceFormatterHuman::printSourceLocation(SourceReference const& _ref) +{ + if (_ref.position.line < 0) + return; // Nothing we can print here + + int const leftpad = static_cast(log10(max(_ref.position.line, 1))) + 1; + + // line 0: source name + frameColored() << string(leftpad, ' ') << "--> "; + m_stream << _ref.sourceName << ":" << (_ref.position.line + 1) << ":" << (_ref.position.column + 1) << ": " << '\n'; + + if (!_ref.multiline) + { + int const locationLength = _ref.endColumn - _ref.startColumn; + + // line 1: + m_stream << string(leftpad, ' '); + frameColored() << " |" << '\n'; + + // line 2: + frameColored() << (_ref.position.line + 1) << " | "; + m_stream << _ref.text.substr(0, _ref.startColumn); + highlightColored() << _ref.text.substr(_ref.startColumn, locationLength); + m_stream << _ref.text.substr(_ref.endColumn) << '\n'; + + // line 3: + m_stream << string(leftpad, ' '); + frameColored() << " | "; + for_each( + _ref.text.cbegin(), + _ref.text.cbegin() + _ref.startColumn, + [this](char ch) { m_stream << (ch == '\t' ? '\t' : ' '); } + ); + diagColored() << string(locationLength, '^') << '\n'; + } + else + { + // line 1: + m_stream << string(leftpad, ' '); + frameColored() << " |" << '\n'; + + // line 2: + frameColored() << (_ref.position.line + 1) << " | "; + m_stream << _ref.text.substr(0, _ref.startColumn); + highlightColored() << _ref.text.substr(_ref.startColumn) << '\n'; + + // line 3: + frameColored() << string(leftpad, ' ') << " | "; + m_stream << string(_ref.startColumn, ' '); + diagColored() << "^ (Relevant source part starts here and spans across multiple lines).\n"; + } +} + +void SourceReferenceFormatterHuman::printExceptionInformation(SourceReferenceExtractor::Message const& _msg) +{ + // exception header line + errorColored() << _msg.category; + messageColored() << ": " << _msg.primary.message << '\n'; + + printSourceLocation(_msg.primary); + + for (auto const& secondary: _msg.secondary) + { + secondaryColored() << "Note"; + messageColored() << ": " << secondary.message << '\n'; + printSourceLocation(secondary); + } + + m_stream << '\n'; +} diff --git a/liblangutil/SourceReferenceFormatterHuman.h b/liblangutil/SourceReferenceFormatterHuman.h new file mode 100644 index 000000000000..a5e8038946e4 --- /dev/null +++ b/liblangutil/SourceReferenceFormatterHuman.h @@ -0,0 +1,80 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ +/** + * Formatting functions for errors referencing positions and locations in the source. + */ + +#pragma once + +#include +#include // SourceReferenceFormatterBase + +#include + +#include +#include +#include + +namespace dev +{ +struct Exception; // forward +} + +namespace langutil +{ + +struct SourceLocation; +struct SourceReference; + +class SourceReferenceFormatterHuman: public SourceReferenceFormatter +{ +public: + SourceReferenceFormatterHuman(std::ostream& _stream, bool colored): + SourceReferenceFormatter{_stream}, m_colored{colored} + {} + + void printSourceLocation(SourceReference const& _ref) override; + void printExceptionInformation(SourceReferenceExtractor::Message const& _msg) override; + using SourceReferenceFormatter::printExceptionInformation; + + static std::string formatExceptionInformation( + dev::Exception const& _exception, + std::string const& _name, + bool colored = false + ) + { + std::ostringstream errorOutput; + + SourceReferenceFormatterHuman formatter(errorOutput, colored); + formatter.printExceptionInformation(_exception, _name); + return errorOutput.str(); + } + +private: + dev::AnsiColorized normalColored() const; + dev::AnsiColorized frameColored() const; + dev::AnsiColorized errorColored() const; + dev::AnsiColorized messageColored() const; + dev::AnsiColorized secondaryColored() const; + dev::AnsiColorized highlightColored() const; + dev::AnsiColorized diagColored() const; + +private: + bool m_colored; +}; + +} diff --git a/solc/CommandLineInterface.cpp b/solc/CommandLineInterface.cpp index 4bf24901b491..8fb3abb47e1c 100644 --- a/solc/CommandLineInterface.cpp +++ b/solc/CommandLineInterface.cpp @@ -35,6 +35,7 @@ #include #include #include +#include #include #include @@ -46,6 +47,8 @@ #include #include +#include + #include #include #include @@ -134,6 +137,9 @@ static string const g_strStrictAssembly = "strict-assembly"; static string const g_strPrettyJson = "pretty-json"; static string const g_strVersion = "version"; static string const g_strIgnoreMissingFiles = "ignore-missing"; +static string const g_strColor = "color"; +static string const g_strNoColor = "no-color"; +static string const g_strNewReporter = "new-reporter"; static string const g_argAbi = g_strAbi; static string const g_argPrettyJson = g_strPrettyJson; @@ -169,6 +175,9 @@ static string const g_argStrictAssembly = g_strStrictAssembly; static string const g_argVersion = g_strVersion; static string const g_stdinFileName = g_stdinFileNameStr; static string const g_argIgnoreMissingFiles = g_strIgnoreMissingFiles; +static string const g_argColor = g_strColor; +static string const g_argNoColor = g_strNoColor; +static string const g_argNewReporter = g_strNewReporter; /// Possible arguments to for --combined-json static set const g_combinedJsonArgs @@ -652,6 +661,9 @@ Allowed options)", po::value()->value_name("path(s)"), "Allow a given path for imports. A list of paths can be supplied by separating them with a comma." ) + (g_argColor.c_str(), "Force colored output.") + (g_argNoColor.c_str(), "Explicitly disable colored output, disabling terminal auto-detection.") + (g_argNewReporter.c_str(), "Enables new diagnostics reporter.") (g_argIgnoreMissingFiles.c_str(), "Ignore missing files."); po::options_description outputComponents("Output Components"); outputComponents.add_options() @@ -691,6 +703,14 @@ Allowed options)", return false; } + if (m_args.count(g_argColor) && m_args.count(g_argNoColor)) + { + serr() << "Option " << g_argColor << " and " << g_argNoColor << " are mutualy exclusive." << endl; + return false; + } + + m_coloredOutput = !m_args.count(g_argNoColor) && (isatty(STDERR_FILENO) || m_args.count(g_argColor)); + if (m_args.count(g_argHelp) || (isatty(fileno(stdin)) && _argc == 1)) { sout() << desc; @@ -858,7 +878,11 @@ bool CommandLineInterface::processInput() m_compiler.reset(new CompilerStack(fileReader)); - SourceReferenceFormatter formatter(serr(false)); + unique_ptr formatter; + if (m_args.count(g_argNewReporter)) + formatter = make_unique(serr(false), m_coloredOutput); + else + formatter = make_unique(serr(false)); try { @@ -881,7 +905,7 @@ bool CommandLineInterface::processInput() for (auto const& error: m_compiler->errors()) { g_hasOutput = true; - formatter.printExceptionInformation( + formatter->printExceptionInformation( *error, (error->type() == Error::Type::Warning) ? "Warning" : "Error" ); @@ -893,7 +917,7 @@ bool CommandLineInterface::processInput() catch (CompilerError const& _exception) { g_hasOutput = true; - formatter.printExceptionInformation(_exception, "Compiler error"); + formatter->printExceptionInformation(_exception, "Compiler error"); return false; } catch (InternalCompilerError const& _exception) @@ -915,7 +939,7 @@ bool CommandLineInterface::processInput() else { g_hasOutput = true; - formatter.printExceptionInformation(_error, _error.typeName()); + formatter->printExceptionInformation(_error, _error.typeName()); } return false; @@ -1221,12 +1245,16 @@ bool CommandLineInterface::assemble( for (auto const& sourceAndStack: assemblyStacks) { auto const& stack = sourceAndStack.second; - SourceReferenceFormatter formatter(serr(false)); + unique_ptr formatter; + if (m_args.count(g_argNewReporter)) + formatter = make_unique(serr(false), m_coloredOutput); + else + formatter = make_unique(serr(false)); for (auto const& error: stack.errors()) { g_hasOutput = true; - formatter.printExceptionInformation( + formatter->printExceptionInformation( *error, (error->type() == Error::Type::Warning) ? "Warning" : "Error" ); diff --git a/solc/CommandLineInterface.h b/solc/CommandLineInterface.h index ff294adc81fb..2b74f3e5ef8c 100644 --- a/solc/CommandLineInterface.h +++ b/solc/CommandLineInterface.h @@ -109,6 +109,8 @@ class CommandLineInterface std::unique_ptr m_compiler; /// EVM version to use EVMVersion m_evmVersion; + /// Whether or not to colorize diagnostics output. + bool m_coloredOutput = true; }; } From 637546850fb9b1d83e68755c31a67cbe41ba85e0 Mon Sep 17 00:00:00 2001 From: Leonardo Alt Date: Tue, 22 Jan 2019 14:34:29 +0100 Subject: [PATCH 080/109] [SMTChecker] Add mod operator --- libsolidity/formal/CVC4Interface.cpp | 2 ++ libsolidity/formal/SolverInterface.h | 5 +++++ libsolidity/formal/Z3Interface.cpp | 2 ++ 3 files changed, 9 insertions(+) diff --git a/libsolidity/formal/CVC4Interface.cpp b/libsolidity/formal/CVC4Interface.cpp index e7c8f0159638..71f657471500 100644 --- a/libsolidity/formal/CVC4Interface.cpp +++ b/libsolidity/formal/CVC4Interface.cpp @@ -163,6 +163,8 @@ CVC4::Expr CVC4Interface::toCVC4Expr(Expression const& _expr) return m_context.mkExpr(CVC4::kind::MULT, arguments[0], arguments[1]); else if (n == "/") return m_context.mkExpr(CVC4::kind::INTS_DIVISION_TOTAL, arguments[0], arguments[1]); + else if (n == "mod") + return m_context.mkExpr(CVC4::kind::INTS_MODULUS, arguments[0], arguments[1]); else if (n == "select") return m_context.mkExpr(CVC4::kind::SELECT, arguments[0], arguments[1]); else if (n == "store") diff --git a/libsolidity/formal/SolverInterface.h b/libsolidity/formal/SolverInterface.h index 76991a58f540..80b48e0f6749 100644 --- a/libsolidity/formal/SolverInterface.h +++ b/libsolidity/formal/SolverInterface.h @@ -141,6 +141,7 @@ class Expression {"-", 2}, {"*", 2}, {"/", 2}, + {"mod", 2}, {"select", 2}, {"store", 3} }; @@ -246,6 +247,10 @@ class Expression { return Expression("/", std::move(_a), std::move(_b), Kind::Int); } + friend Expression operator%(Expression _a, Expression _b) + { + return Expression("mod", std::move(_a), std::move(_b), Kind::Int); + } Expression operator()(std::vector _arguments) const { solAssert( diff --git a/libsolidity/formal/Z3Interface.cpp b/libsolidity/formal/Z3Interface.cpp index 4cbc3271947e..4d7ea8347d6d 100644 --- a/libsolidity/formal/Z3Interface.cpp +++ b/libsolidity/formal/Z3Interface.cpp @@ -162,6 +162,8 @@ z3::expr Z3Interface::toZ3Expr(Expression const& _expr) return arguments[0] * arguments[1]; else if (n == "/") return arguments[0] / arguments[1]; + else if (n == "mod") + return z3::mod(arguments[0], arguments[1]); else if (n == "select") return z3::select(arguments[0], arguments[1]); else if (n == "store") From 4058978b3ba12ea1cb66f6364b1b6bc85958b032 Mon Sep 17 00:00:00 2001 From: Leonardo Alt Date: Mon, 4 Feb 2019 14:39:45 +0100 Subject: [PATCH 081/109] Require Z3 >= 4.6.0 --- cmake/FindZ3.cmake | 18 +++++++++++++++++- libsolidity/CMakeLists.txt | 2 +- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/cmake/FindZ3.cmake b/cmake/FindZ3.cmake index ad34cbc3b42a..bdd8ce72ff0f 100644 --- a/cmake/FindZ3.cmake +++ b/cmake/FindZ3.cmake @@ -1,8 +1,24 @@ if (USE_Z3) find_path(Z3_INCLUDE_DIR NAMES z3++.h PATH_SUFFIXES z3) find_library(Z3_LIBRARY NAMES z3) + find_program(Z3_EXECUTABLE z3 PATH_SUFFIXES bin) + + if(Z3_INCLUDE_DIR AND Z3_LIBRARY AND Z3_EXECUTABLE) + execute_process (COMMAND ${Z3_EXECUTABLE} -version + OUTPUT_VARIABLE libz3_version_str + ERROR_QUIET + OUTPUT_STRIP_TRAILING_WHITESPACE) + + string(REGEX REPLACE "^Z3 version ([0-9.]+).*" "\\1" + Z3_VERSION_STRING "${libz3_version_str}") + unset(libz3_version_str) + endif() + mark_as_advanced(Z3_VERSION_STRING z3_DIR) + include(FindPackageHandleStandardArgs) - find_package_handle_standard_args(Z3 DEFAULT_MSG Z3_LIBRARY Z3_INCLUDE_DIR) + find_package_handle_standard_args(Z3 + REQUIRED_VARS Z3_LIBRARY Z3_INCLUDE_DIR + VERSION_VAR Z3_VERSION_STRING) if (NOT TARGET Z3::Z3) add_library(Z3::Z3 UNKNOWN IMPORTED) diff --git a/libsolidity/CMakeLists.txt b/libsolidity/CMakeLists.txt index 8c2ab3475ad8..6cc77c4e3be0 100644 --- a/libsolidity/CMakeLists.txt +++ b/libsolidity/CMakeLists.txt @@ -102,7 +102,7 @@ set(sources parsing/Token.h ) -find_package(Z3 QUIET) +find_package(Z3 4.6.0) if (${Z3_FOUND}) add_definitions(-DHAVE_Z3) message("Z3 SMT solver found. This enables optional SMT checking with Z3.") From bea695f3fc661b48e1a5929fe4af8fedb615ede4 Mon Sep 17 00:00:00 2001 From: chriseth Date: Wed, 6 Feb 2019 15:50:36 +0100 Subject: [PATCH 082/109] Changelog entry for Z3 version. --- Changelog.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Changelog.md b/Changelog.md index 89143ee5942b..a08d22ecb2c1 100644 --- a/Changelog.md +++ b/Changelog.md @@ -21,6 +21,7 @@ Bugfixes: Build System: * Add support for continuous fuzzing via Google oss-fuzz + * SMT: If using Z3, require version 4.6.0 or newer. * Ubuntu PPA Packages: Use CVC4 as SMT solver instead of Z3 From 62f707610e8b300668d27595e5bb3f265431acf6 Mon Sep 17 00:00:00 2001 From: Alex Beregszaszi Date: Mon, 28 Jan 2019 13:51:47 +0000 Subject: [PATCH 083/109] Run clang tests with ASAN --- .circleci/config.yml | 49 +++++++++++++++++++++++++++++++-- cmake/EthCompilerSettings.cmake | 1 - 2 files changed, 46 insertions(+), 4 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index a5b0a6117f9e..a73770798d81 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -165,14 +165,14 @@ jobs: - test/soltest - test/tools/solfuzzer - build_x86_clang7: + build_x86_clang7_asan: docker: - image: buildpack-deps:cosmic environment: TERM: xterm CC: /usr/bin/clang-7 CXX: /usr/bin/clang++-7 - CMAKE_OPTIONS: -DLLL=ON + CMAKE_OPTIONS: -DSANITIZE=address -DCMAKE_BUILD_TYPE=Debug steps: - checkout - run: @@ -183,6 +183,13 @@ jobs: ./scripts/install_obsolete_jsoncpp_1_7_4.sh - run: *setup_prerelease_commit_hash - run: *run_build + - store_artifacts: *solc_artifact + - persist_to_workspace: + root: build + paths: + - solc/solc + - test/soltest + - test/tools/solfuzzer build_x86_mac: macos: @@ -283,6 +290,36 @@ jobs: path: test_results/ destination: test_results/ + test_x86_clang7_asan: + docker: + - image: buildpack-deps:cosmic + environment: + TERM: xterm + steps: + - checkout + - attach_workspace: + at: build + - run: + name: Install dependencies + command: | + apt-get -qq update + apt-get -qy install llvm-7-dev libcvc4-dev libleveldb1v5 python-pip + # This is needed to resolve the symbols. Since we're using clang7 in the build, we must use the appropriate symbolizer. + update-alternatives --install /usr/bin/llvm-symbolizer llvm-symbolizer /usr/bin/llvm-symbolizer-7 1 + - run: mkdir -p test_results + - run: + name: Run tests with ASAN + command: | + ulimit -a + # Increase stack size because ASan makes stack frames bigger and that breaks our assumptions (in tests). + ulimit -s 16384 + build/test/soltest --logger=JUNIT,test_suite,test_results/result.xml -- --no-ipc --testpath test + - store_test_results: + path: test_results/ + - store_artifacts: + path: test_results/ + destination: test_results/ + test_x86_archlinux: docker: - image: archlinux/base @@ -366,12 +403,18 @@ workflows: - build_emscripten - build_x86_linux: *build_on_tags - build_x86_linux_cxx17: *build_on_tags - - build_x86_clang7: *build_on_tags + - build_x86_clang7_asan: *build_on_tags - build_x86_mac: *build_on_tags - test_x86_linux: <<: *build_on_tags requires: - build_x86_linux + - test_x86_clang7_asan: + filters: + branches: + only: develop + requires: + - build_x86_clang7_asan - test_x86_mac: <<: *build_on_tags requires: diff --git a/cmake/EthCompilerSettings.cmake b/cmake/EthCompilerSettings.cmake index 99c03af00411..60ebe88634fb 100644 --- a/cmake/EthCompilerSettings.cmake +++ b/cmake/EthCompilerSettings.cmake @@ -74,7 +74,6 @@ if (("${CMAKE_CXX_COMPILER_ID}" MATCHES "GNU") OR ("${CMAKE_CXX_COMPILER_ID}" MA # into errors, which makes sense. # http://stackoverflow.com/questions/21617158/how-to-silence-unused-command-line-argument-error-with-clang-without-disabling-i add_compile_options(-Qunused-arguments) - elseif(EMSCRIPTEN) set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} --memory-init-file 0") # Leave only exported symbols as public and aggressively remove others From b105497d1b957d6ed4354695c8a47d83b977ad32 Mon Sep 17 00:00:00 2001 From: Chris Ward Date: Fri, 8 Feb 2019 08:45:43 +0000 Subject: [PATCH 084/109] Remove Frequently Asked Questions --- docs/frequently-asked-questions.rst | 6 ------ docs/index.rst | 1 - 2 files changed, 7 deletions(-) delete mode 100644 docs/frequently-asked-questions.rst diff --git a/docs/frequently-asked-questions.rst b/docs/frequently-asked-questions.rst deleted file mode 100644 index 9dc368abadb6..000000000000 --- a/docs/frequently-asked-questions.rst +++ /dev/null @@ -1,6 +0,0 @@ -########################### -Frequently Asked Questions -########################### - -If your question is not answered here, please talk to us on -`gitter `_ or file an `issue `_. diff --git a/docs/index.rst b/docs/index.rst index 16745c07c5a6..bdd4667045c7 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -87,5 +87,4 @@ Contents common-patterns.rst bugs.rst contributing.rst - frequently-asked-questions.rst lll.rst From 57c1c8b46cd4bebfb77540cbb8d52349f2e145f1 Mon Sep 17 00:00:00 2001 From: Mathias Baumann Date: Thu, 7 Feb 2019 15:53:11 +0100 Subject: [PATCH 085/109] Ensure function parameter names always matches parameter types length --- libsolidity/analysis/TypeChecker.cpp | 19 +++----- libsolidity/ast/Types.cpp | 65 ++++++++++++++++++++++------ libsolidity/ast/Types.h | 12 ++++- 3 files changed, 68 insertions(+), 28 deletions(-) diff --git a/libsolidity/analysis/TypeChecker.cpp b/libsolidity/analysis/TypeChecker.cpp index 4b1f28cda6cb..3c1bfbae5b35 100644 --- a/libsolidity/analysis/TypeChecker.cpp +++ b/libsolidity/analysis/TypeChecker.cpp @@ -1678,17 +1678,10 @@ void TypeChecker::typeCheckFunctionGeneralChecks( { auto const& parameterNames = _functionType->parameterNames(); - // Check for expected number of named arguments - if (parameterNames.size() != argumentNames.size()) - { - m_errorReporter.typeError( - _functionCall.location(), - parameterNames.size() > argumentNames.size() ? - "Some argument names are missing." : - "Too many arguments." - ); - return; - } + solAssert( + parameterNames.size() == argumentNames.size(), + "Unexpected parameter length mismatch!" + ); // Check for duplicate argument names { @@ -1972,8 +1965,8 @@ void TypeChecker::endVisit(NewExpression const& _newExpression) _newExpression.annotation().type = make_shared( TypePointers{make_shared(256)}, TypePointers{type}, - strings(), - strings(), + strings(1, ""), + strings(1, ""), FunctionType::Kind::ObjectCreation, false, StateMutability::Pure diff --git a/libsolidity/ast/Types.cpp b/libsolidity/ast/Types.cpp index 308c0fe5edda..3f35d4547cae 100644 --- a/libsolidity/ast/Types.cpp +++ b/libsolidity/ast/Types.cpp @@ -1836,8 +1836,8 @@ MemberList::MemberMap ArrayType::nativeMembers(ContractDefinition const*) const members.emplace_back("pop", make_shared( TypePointers{}, TypePointers{}, - strings{string()}, - strings{string()}, + strings{}, + strings{}, FunctionType::Kind::ArrayPop )); } @@ -2191,7 +2191,7 @@ FunctionTypePointer StructType::constructorType() const paramTypes, TypePointers{copyForLocation(DataLocation::Memory, false)}, paramNames, - strings(), + strings(1, ""), FunctionType::Kind::Internal ); } @@ -2423,6 +2423,16 @@ FunctionType::FunctionType(FunctionDefinition const& _function, bool _isInternal m_returnParameterNames.push_back(var->name()); m_returnParameterTypes.push_back(var->annotation().type); } + + solAssert( + m_parameterNames.size() == m_parameterTypes.size(), + "Parameter names list must match parameter types list!" + ); + + solAssert( + m_returnParameterNames.size() == m_returnParameterTypes.size(), + "Return parameter names list must match return parameter types list!" + ); } FunctionType::FunctionType(VariableDeclaration const& _varDecl): @@ -2479,6 +2489,15 @@ FunctionType::FunctionType(VariableDeclaration const& _varDecl): )); m_returnParameterNames.emplace_back(""); } + + solAssert( + m_parameterNames.size() == m_parameterTypes.size(), + "Parameter names list must match parameter types list!" + ); + solAssert( + m_returnParameterNames.size() == m_returnParameterTypes.size(), + "Return parameter names list must match return parameter types list!" + ); } FunctionType::FunctionType(EventDefinition const& _event): @@ -2491,9 +2510,20 @@ FunctionType::FunctionType(EventDefinition const& _event): m_parameterNames.push_back(var->name()); m_parameterTypes.push_back(var->annotation().type); } + + solAssert( + m_parameterNames.size() == m_parameterTypes.size(), + "Parameter names list must match parameter types list!" + ); + solAssert( + m_returnParameterNames.size() == m_returnParameterTypes.size(), + "Return parameter names list must match return parameter types list!" + ); } FunctionType::FunctionType(FunctionTypeName const& _typeName): + m_parameterNames(_typeName.parameterTypes().size(), ""), + m_returnParameterNames(_typeName.returnParameterTypes().size(), ""), m_kind(_typeName.visibility() == VariableDeclaration::Visibility::External ? Kind::External : Kind::Internal), m_stateMutability(_typeName.stateMutability()) { @@ -2519,6 +2549,15 @@ FunctionType::FunctionType(FunctionTypeName const& _typeName): ); m_returnParameterTypes.push_back(t->annotation().type); } + + solAssert( + m_parameterNames.size() == m_parameterTypes.size(), + "Parameter names list must match parameter types list!" + ); + solAssert( + m_returnParameterNames.size() == m_returnParameterTypes.size(), + "Return parameter names list must match return parameter types list!" + ); } FunctionTypePointer FunctionType::newExpressionType(ContractDefinition const& _contract) @@ -2856,8 +2895,8 @@ MemberList::MemberMap FunctionType::nativeMembers(ContractDefinition const*) con make_shared( parseElementaryTypeVector({"uint"}), TypePointers{copyAndSetGasOrValue(false, true)}, - strings(), - strings(), + strings(1, ""), + strings(1, ""), Kind::SetValue, false, StateMutability::NonPayable, @@ -2873,8 +2912,8 @@ MemberList::MemberMap FunctionType::nativeMembers(ContractDefinition const*) con make_shared( parseElementaryTypeVector({"uint"}), TypePointers{copyAndSetGasOrValue(true, false)}, - strings(), - strings(), + strings(1, ""), + strings(1, ""), Kind::SetGas, false, StateMutability::NonPayable, @@ -3377,7 +3416,7 @@ MemberList::MemberMap MagicType::nativeMembers(ContractDefinition const*) const TypePointers(), TypePointers{make_shared(DataLocation::Memory)}, strings{}, - strings{}, + strings{1, ""}, FunctionType::Kind::ABIEncode, true, StateMutability::Pure @@ -3386,7 +3425,7 @@ MemberList::MemberMap MagicType::nativeMembers(ContractDefinition const*) const TypePointers(), TypePointers{make_shared(DataLocation::Memory)}, strings{}, - strings{}, + strings{1, ""}, FunctionType::Kind::ABIEncodePacked, true, StateMutability::Pure @@ -3394,8 +3433,8 @@ MemberList::MemberMap MagicType::nativeMembers(ContractDefinition const*) const {"encodeWithSelector", make_shared( TypePointers{make_shared(4)}, TypePointers{make_shared(DataLocation::Memory)}, - strings{}, - strings{}, + strings{1, ""}, + strings{1, ""}, FunctionType::Kind::ABIEncodeWithSelector, true, StateMutability::Pure @@ -3403,8 +3442,8 @@ MemberList::MemberMap MagicType::nativeMembers(ContractDefinition const*) const {"encodeWithSignature", make_shared( TypePointers{make_shared(DataLocation::Memory, true)}, TypePointers{make_shared(DataLocation::Memory)}, - strings{}, - strings{}, + strings{1, ""}, + strings{1, ""}, FunctionType::Kind::ABIEncodeWithSignature, true, StateMutability::Pure diff --git a/libsolidity/ast/Types.h b/libsolidity/ast/Types.h index 53109de156fc..bd249c633eea 100644 --- a/libsolidity/ast/Types.h +++ b/libsolidity/ast/Types.h @@ -1012,8 +1012,8 @@ class FunctionType: public Type ): FunctionType( parseElementaryTypeVector(_parameterTypes), parseElementaryTypeVector(_returnParameterTypes), - strings(), - strings(), + strings(_parameterTypes.size(), ""), + strings(_returnParameterTypes.size(), ""), _kind, _arbitraryParameters, _stateMutability @@ -1050,6 +1050,14 @@ class FunctionType: public Type m_bound(_bound), m_declaration(_declaration) { + solAssert( + m_parameterNames.size() == m_parameterTypes.size(), + "Parameter names list must match parameter types list!" + ); + solAssert( + m_returnParameterNames.size() == m_returnParameterTypes.size(), + "Return parameter names list must match return parameter types list!" + ); solAssert( !m_bound || !m_parameterTypes.empty(), "Attempted construction of bound function without self type" From d41ffd1dcfd1a4c41065c5e163ba52c52d5e9754 Mon Sep 17 00:00:00 2001 From: Mathias Baumann Date: Mon, 11 Feb 2019 11:23:20 +0100 Subject: [PATCH 086/109] Rename & move tests to get rid of old number prefix in name --- .../tuple.sol} | 0 .../creating_memory_array.sol} | 0 .../creating_struct.sol} | 0 3 files changed, 0 insertions(+), 0 deletions(-) rename test/libsolidity/syntaxTests/nameAndTypeResolution/{249_tuple_compound_assignment.sol => compoundAssignment/tuple.sol} (100%) rename test/libsolidity/syntaxTests/nameAndTypeResolution/{266_invalid_args_creating_memory_array.sol => invalidArgs/creating_memory_array.sol} (100%) rename test/libsolidity/syntaxTests/nameAndTypeResolution/{267_invalid_args_creating_struct.sol => invalidArgs/creating_struct.sol} (100%) diff --git a/test/libsolidity/syntaxTests/nameAndTypeResolution/249_tuple_compound_assignment.sol b/test/libsolidity/syntaxTests/nameAndTypeResolution/compoundAssignment/tuple.sol similarity index 100% rename from test/libsolidity/syntaxTests/nameAndTypeResolution/249_tuple_compound_assignment.sol rename to test/libsolidity/syntaxTests/nameAndTypeResolution/compoundAssignment/tuple.sol diff --git a/test/libsolidity/syntaxTests/nameAndTypeResolution/266_invalid_args_creating_memory_array.sol b/test/libsolidity/syntaxTests/nameAndTypeResolution/invalidArgs/creating_memory_array.sol similarity index 100% rename from test/libsolidity/syntaxTests/nameAndTypeResolution/266_invalid_args_creating_memory_array.sol rename to test/libsolidity/syntaxTests/nameAndTypeResolution/invalidArgs/creating_memory_array.sol diff --git a/test/libsolidity/syntaxTests/nameAndTypeResolution/267_invalid_args_creating_struct.sol b/test/libsolidity/syntaxTests/nameAndTypeResolution/invalidArgs/creating_struct.sol similarity index 100% rename from test/libsolidity/syntaxTests/nameAndTypeResolution/267_invalid_args_creating_struct.sol rename to test/libsolidity/syntaxTests/nameAndTypeResolution/invalidArgs/creating_struct.sol From feae01f0424e8b04ce6748cb0de0f46e20b0c13a Mon Sep 17 00:00:00 2001 From: Mathias Baumann Date: Mon, 11 Feb 2019 11:23:56 +0100 Subject: [PATCH 087/109] Add tests to increase coverage of TypeChecker --- .../compoundAssignment/incomp_types.sol | 7 +++++++ .../tuple_invalid_inline_array_type.sol | 8 ++++++++ .../invalidArgs/creating_struct_members_skipped.sol | 9 +++++++++ .../invalidArgs/explicit_conversions.sol | 9 +++++++++ 4 files changed, 33 insertions(+) create mode 100644 test/libsolidity/syntaxTests/nameAndTypeResolution/compoundAssignment/incomp_types.sol create mode 100644 test/libsolidity/syntaxTests/nameAndTypeResolution/compoundAssignment/tuple_invalid_inline_array_type.sol create mode 100644 test/libsolidity/syntaxTests/nameAndTypeResolution/invalidArgs/creating_struct_members_skipped.sol create mode 100644 test/libsolidity/syntaxTests/nameAndTypeResolution/invalidArgs/explicit_conversions.sol diff --git a/test/libsolidity/syntaxTests/nameAndTypeResolution/compoundAssignment/incomp_types.sol b/test/libsolidity/syntaxTests/nameAndTypeResolution/compoundAssignment/incomp_types.sol new file mode 100644 index 000000000000..7f2139bf75d1 --- /dev/null +++ b/test/libsolidity/syntaxTests/nameAndTypeResolution/compoundAssignment/incomp_types.sol @@ -0,0 +1,7 @@ +contract C { + function f() public returns (uint a, uint b) { + a += (1, 1); + } +} +// ---- +// TypeError: (72-83): Operator += not compatible with types uint256 and tuple(int_const 1,int_const 1) diff --git a/test/libsolidity/syntaxTests/nameAndTypeResolution/compoundAssignment/tuple_invalid_inline_array_type.sol b/test/libsolidity/syntaxTests/nameAndTypeResolution/compoundAssignment/tuple_invalid_inline_array_type.sol new file mode 100644 index 000000000000..17892ec05da4 --- /dev/null +++ b/test/libsolidity/syntaxTests/nameAndTypeResolution/compoundAssignment/tuple_invalid_inline_array_type.sol @@ -0,0 +1,8 @@ +contract C { + function f() pure public { + uint x; + (x, ) = ([100e100]); + } +} +// ---- +// TypeError: (78-85): Invalid mobile type. diff --git a/test/libsolidity/syntaxTests/nameAndTypeResolution/invalidArgs/creating_struct_members_skipped.sol b/test/libsolidity/syntaxTests/nameAndTypeResolution/invalidArgs/creating_struct_members_skipped.sol new file mode 100644 index 000000000000..1e954ab6c392 --- /dev/null +++ b/test/libsolidity/syntaxTests/nameAndTypeResolution/invalidArgs/creating_struct_members_skipped.sol @@ -0,0 +1,9 @@ +contract C { + struct S { uint a; uint b; mapping(uint=>uint) c; } + + function f() public { + S memory s = S({a: 1}); + } +} +// ---- +// TypeError: (117-126): Wrong argument count for struct constructor: 1 arguments given but expected 2. Members that have to be skipped in memory: c diff --git a/test/libsolidity/syntaxTests/nameAndTypeResolution/invalidArgs/explicit_conversions.sol b/test/libsolidity/syntaxTests/nameAndTypeResolution/invalidArgs/explicit_conversions.sol new file mode 100644 index 000000000000..c81b60351621 --- /dev/null +++ b/test/libsolidity/syntaxTests/nameAndTypeResolution/invalidArgs/explicit_conversions.sol @@ -0,0 +1,9 @@ +contract test { + function f() public { + uint(1, 1); + uint({arg:1}); + } +} +// ---- +// TypeError: (50-60): Exactly one argument expected for explicit type conversion. +// TypeError: (70-83): Type conversion cannot allow named arguments. From 1b9d30f05fdf879e65579865e6c779863b756b9f Mon Sep 17 00:00:00 2001 From: chriseth Date: Tue, 22 Jan 2019 12:24:54 +0100 Subject: [PATCH 088/109] Packed encoding. --- libsolidity/codegen/ABIFunctions.cpp | 227 +++++++++++++++++++++------ libsolidity/codegen/ABIFunctions.h | 24 ++- 2 files changed, 198 insertions(+), 53 deletions(-) diff --git a/libsolidity/codegen/ABIFunctions.cpp b/libsolidity/codegen/ABIFunctions.cpp index 1652d427800b..a672457cb387 100644 --- a/libsolidity/codegen/ABIFunctions.cpp +++ b/libsolidity/codegen/ABIFunctions.cpp @@ -574,20 +574,66 @@ string ABIFunctions::abiEncodingFunction( // special case: convert storage reference type to value type - this is only // possible for library calls where we just forward the storage reference solAssert(_options.encodeAsLibraryTypes, ""); + solAssert(_options.padded && !_options.dynamicInplace, "Non-padded / inplace encoding for library call requested."); solAssert(to == IntegerType::uint256(), ""); templ("cleanupConvert", "value"); } else { + string cleanupConvert; if (_from == to) - templ("cleanupConvert", cleanupFunction(_from) + "(value)"); + cleanupConvert = cleanupFunction(_from) + "(value)"; else - templ("cleanupConvert", conversionFunction(_from, to) + "(value)"); + cleanupConvert = conversionFunction(_from, to) + "(value)"; + if (!_options.padded) + cleanupConvert = leftAlignFunction(to) + "(" + cleanupConvert + ")"; + templ("cleanupConvert", cleanupConvert); } return templ.render(); }); } +string ABIFunctions::abiEncodeAndReturnUpdatedPosFunction( + Type const& _givenType, + Type const& _targetType, + ABIFunctions::EncodingOptions const& _options +) +{ + string functionName = + "abi_encodeUpdatedPos_" + + _givenType.identifier() + + "_to_" + + _targetType.identifier() + + _options.toFunctionNameSuffix(); + return createFunction(functionName, [&]() { + string encoder = abiEncodingFunction(_givenType, _targetType, _options); + if (_targetType.isDynamicallyEncoded()) + return Whiskers(R"( + function (value, pos) -> updatedPos { + updatedPos := (value, pos) + } + )") + ("functionName", functionName) + ("encode", encoder) + .render(); + else + { + unsigned encodedSize = _targetType.calldataEncodedSize(_options.padded); + solAssert(encodedSize != 0, "Invalid encoded size."); + return Whiskers(R"( + function (value, pos) -> updatedPos { + (value, pos) + updatedPos := add(pos, ) + } + )") + ("functionName", functionName) + ("encode", encoder) + ("encodedSize", toCompactHexWithPrefix(encodedSize)) + .render(); + } + }); +} + string ABIFunctions::abiEncodingFunctionCalldataArray( Type const& _from, Type const& _to, @@ -623,15 +669,15 @@ string ABIFunctions::abiEncodingFunctionCalldataArray( function (start, length, pos) -> end { pos := (pos, length) (start, pos, length) - end := add(pos, (length)) + end := add(pos, ) } )"); - templ("storeLength", arrayStoreLengthForEncodingFunction(toArrayType)); + templ("storeLength", arrayStoreLengthForEncodingFunction(toArrayType, _options)); templ("functionName", functionName); templ("readableTypeNameFrom", _from.toString(true)); templ("readableTypeNameTo", _to.toString(true)); templ("copyFun", copyToMemoryFunction(true)); - templ("roundUpFun", roundUpFunction()); + templ("lengthPadded", _options.padded ? roundUpFunction() + "(length)" : "length"); return templ.render(); }); } @@ -659,8 +705,9 @@ string ABIFunctions::abiEncodingFunctionSimpleArray( bool dynamic = _to.isDynamicallyEncoded(); bool dynamicBase = _to.baseType()->isDynamicallyEncoded(); bool inMemory = _from.dataStoredIn(DataLocation::Memory); + bool const usesTail = dynamicBase && !_options.dynamicInplace; Whiskers templ( - dynamicBase ? + usesTail ? R"( // -> function (value, pos) { @@ -688,9 +735,8 @@ string ABIFunctions::abiEncodingFunctionSimpleArray( let srcPtr := (value) for { let i := 0 } lt(i, length) { i := add(i, 1) } { - (, pos) + pos := (, pos) srcPtr := (srcPtr) - pos := add(pos, ) } } @@ -702,17 +748,13 @@ string ABIFunctions::abiEncodingFunctionSimpleArray( templ("return", dynamic ? " -> end " : ""); templ("assignEnd", dynamic ? "end := pos" : ""); templ("lengthFun", arrayLengthFunction(_from)); - templ("storeLength", arrayStoreLengthForEncodingFunction(_to)); + templ("storeLength", arrayStoreLengthForEncodingFunction(_to, _options)); templ("dataAreaFun", arrayDataAreaFunction(_from)); - templ("elementEncodedSize", toCompactHexWithPrefix(_to.baseType()->calldataEncodedSize())); EncodingOptions subOptions(_options); subOptions.encodeFunctionFromStack = false; - templ("encodeToMemoryFun", abiEncodingFunction( - *_from.baseType(), - *_to.baseType(), - subOptions - )); + subOptions.padded = true; + templ("encodeToMemoryFun", abiEncodeAndReturnUpdatedPosFunction(*_from.baseType(), *_to.baseType(), subOptions)); templ("arrayElementAccess", inMemory ? "mload(srcPtr)" : _from.baseType()->isValueType() ? "sload(srcPtr)" : "srcPtr" ); templ("nextArrayElement", nextArrayElementFunction(_from)); return templ.render(); @@ -742,15 +784,16 @@ string ABIFunctions::abiEncodingFunctionMemoryByteArray( Whiskers templ(R"( function (value, pos) -> end { let length := (value) - mstore(pos, length) - (add(value, 0x20), add(pos, 0x20), length) - end := add(add(pos, 0x20), (length)) + pos := (pos, length) + (add(value, 0x20), pos, length) + end := add(pos, ) } )"); templ("functionName", functionName); templ("lengthFun", arrayLengthFunction(_from)); + templ("storeLength", arrayStoreLengthForEncodingFunction(_to, _options)); templ("copyFun", copyToMemoryFunction(false)); - templ("roundUpFun", roundUpFunction()); + templ("lengthPadded", _options.padded ? roundUpFunction() + "(length)" : "length"); return templ.render(); }); } @@ -784,28 +827,30 @@ string ABIFunctions::abiEncodingFunctionCompactStorageArray( case 0 { // short byte array let length := and(div(slotValue, 2), 0x7f) - mstore(pos, length) - mstore(add(pos, 0x20), and(slotValue, not(0xff))) - ret := add(pos, 0x40) + pos := (pos, length) + mstore(pos, and(slotValue, not(0xff))) + ret := add(pos, ) } case 1 { // long byte array let length := div(slotValue, 2) - mstore(pos, length) - pos := add(pos, 0x20) + pos := (pos, length) let dataPos := (value) let i := 0 for { } lt(i, length) { i := add(i, 0x20) } { mstore(add(pos, i), sload(dataPos)) dataPos := add(dataPos, 1) } - ret := add(pos, i) + ret := add(pos, ) } } )"); templ("functionName", functionName); templ("readableTypeNameFrom", _from.toString(true)); templ("readableTypeNameTo", _to.toString(true)); + templ("storeLength", arrayStoreLengthForEncodingFunction(_to, _options)); + templ("lengthPaddedShort", _options.padded ? "0x20" : "length"); + templ("lengthPaddedLong", _options.padded ? "i" : "length"); templ("arrayDataSlot", arrayDataAreaFunction(_from)); return templ.render(); } @@ -819,7 +864,7 @@ string ABIFunctions::abiEncodingFunctionCompactStorageArray( size_t storageBytes = _from.baseType()->storageBytes(); size_t itemsPerSlot = 32 / storageBytes; // This always writes full slot contents to memory, which might be - // more than desired, i.e. it writes beyond the end of memory. + // more than desired, i.e. it always writes beyond the end of memory. Whiskers templ( R"( // -> @@ -848,14 +893,16 @@ string ABIFunctions::abiEncodingFunctionCompactStorageArray( templ("return", dynamic ? " -> end " : ""); templ("assignEnd", dynamic ? "end := pos" : ""); templ("lengthFun", arrayLengthFunction(_from)); - templ("storeLength", arrayStoreLengthForEncodingFunction(_to)); + templ("storeLength", arrayStoreLengthForEncodingFunction(_to, _options)); templ("dataArea", arrayDataAreaFunction(_from)); templ("itemsPerSlot", to_string(itemsPerSlot)); + // We use padded size because array elements are always padded. string elementEncodedSize = toCompactHexWithPrefix(_to.baseType()->calldataEncodedSize()); templ("elementEncodedSize", elementEncodedSize); EncodingOptions subOptions(_options); subOptions.encodeFunctionFromStack = false; + subOptions.padded = true; string encodeToMemoryFun = abiEncodingFunction( *_from.baseType(), *_to.baseType(), @@ -910,7 +957,12 @@ string ABIFunctions::abiEncodingFunctionStruct( templ("readableTypeNameFrom", _from.toString(true)); templ("readableTypeNameTo", _to.toString(true)); templ("return", dynamic ? " -> end " : ""); - templ("assignEnd", dynamic ? "end := tail" : ""); + if (dynamic && _options.dynamicInplace) + templ("assignEnd", "end := pos"); + else if (dynamic && !_options.dynamicInplace) + templ("assignEnd", "end := tail"); + else + templ("assignEnd", ""); // to avoid multiple loads from the same slot for subsequent members templ("init", fromStorage ? "let slotValue := 0" : ""); u256 previousSlotOffset(-1); @@ -960,27 +1012,38 @@ string ABIFunctions::abiEncodingFunctionStruct( members.back()["retrieveValue"] = "mload(add(value, " + sourceOffset + "))"; } - Whiskers encodeTempl( - dynamicMember ? - string(R"( - mstore(add(pos, ), sub(tail, pos)) - tail := (memberValue, tail) - )") : - string(R"( - (memberValue, add(pos, )) - )") - ); - encodeTempl("encodingOffset", toCompactHexWithPrefix(encodingOffset)); - encodingOffset += dynamicMember ? 0x20 : memberTypeTo->calldataEncodedSize(); - EncodingOptions subOptions(_options); subOptions.encodeFunctionFromStack = false; - encodeTempl("abiEncode", abiEncodingFunction(*memberTypeFrom, *memberTypeTo, subOptions)); + // Like with arrays, struct members are always padded. + subOptions.padded = true; + + string encode; + if (_options.dynamicInplace) + encode = Whiskers{"pos := (memberValue, pos)"} + ("encode", abiEncodeAndReturnUpdatedPosFunction(*memberTypeFrom, *memberTypeTo, subOptions)) + .render(); + else + { + Whiskers encodeTempl( + dynamicMember ? + string(R"( + mstore(add(pos, ), sub(tail, pos)) + tail := (memberValue, tail) + )") : + "(memberValue, add(pos, ))" + ); + encodeTempl("encodingOffset", toCompactHexWithPrefix(encodingOffset)); + encodingOffset += dynamicMember ? 0x20 : memberTypeTo->calldataEncodedSize(); + encodeTempl("abiEncode", abiEncodingFunction(*memberTypeFrom, *memberTypeTo, subOptions)); + encode = encodeTempl.render(); + } + members.back()["encode"] = encode; - members.back()["encode"] = encodeTempl.render(); members.back()["memberName"] = member.name; } templ("members", members); + if (_options.dynamicInplace) + solAssert(encodingOffset == 0, "In-place encoding should enforce zero head size."); templ("headSize", toCompactHexWithPrefix(encodingOffset)); return templ.render(); }); @@ -1007,9 +1070,10 @@ string ABIFunctions::abiEncodingFunctionStringLiteral( if (_to.isDynamicallySized()) { + solAssert(_to.category() == Type::Category::Array, ""); Whiskers templ(R"( function (pos) -> end { - mstore(pos, ) + pos := (pos, ) <#word> mstore(add(pos, ), ) @@ -1020,12 +1084,17 @@ string ABIFunctions::abiEncodingFunctionStringLiteral( // TODO this can make use of CODECOPY for large strings once we have that in Yul size_t words = (value.size() + 31) / 32; - templ("overallSize", to_string(32 + words * 32)); templ("length", to_string(value.size())); + templ("storeLength", arrayStoreLengthForEncodingFunction(dynamic_cast(_to), _options)); + if (_options.padded) + templ("overallSize", to_string(words * 32)); + else + templ("overallSize", to_string(value.size())); + vector> wordParams(words); for (size_t i = 0; i < words; ++i) { - wordParams[i]["offset"] = to_string(32 + i * 32); + wordParams[i]["offset"] = to_string(i * 32); wordParams[i]["wordValue"] = "0x" + h256(value.substr(32 * i, 32), h256::AlignLeft).hex(); } templ("word", wordParams); @@ -1427,6 +1496,66 @@ string ABIFunctions::copyToMemoryFunction(bool _fromCalldata) }); } +string ABIFunctions::leftAlignFunction(Type const& _type) +{ + string functionName = string("leftAlign_") + _type.identifier(); + return createFunction(functionName, [&]() { + Whiskers templ(R"( + function (value) -> aligned { + + } + )"); + templ("functionName", functionName); + switch (_type.category()) + { + case Type::Category::Address: + templ("body", "aligned := " + leftAlignFunction(IntegerType(160)) + "(value)"); + break; + case Type::Category::Integer: + { + IntegerType const& type = dynamic_cast(_type); + if (type.numBits() == 256) + templ("body", "aligned := value"); + else + templ("body", "aligned := " + shiftLeftFunction(256 - type.numBits()) + "(value)"); + break; + } + case Type::Category::RationalNumber: + solAssert(false, "Left align requested for rational number."); + break; + case Type::Category::Bool: + templ("body", "aligned := " + leftAlignFunction(IntegerType(8)) + "(value)"); + break; + case Type::Category::FixedPoint: + solUnimplemented("Fixed point types not implemented."); + break; + case Type::Category::Array: + case Type::Category::Struct: + solAssert(false, "Left align requested for non-value type."); + break; + case Type::Category::FixedBytes: + templ("body", "aligned := value"); + break; + case Type::Category::Contract: + templ("body", "aligned := " + leftAlignFunction(AddressType::address()) + "(value)"); + break; + case Type::Category::Enum: + { + unsigned storageBytes = dynamic_cast(_type).storageBytes(); + templ("body", "aligned := " + leftAlignFunction(IntegerType(8 * storageBytes)) + "(value)"); + break; + } + case Type::Category::InaccessibleDynamic: + solAssert(false, "Left align requested for inaccessible dynamic type."); + break; + default: + solAssert(false, "Left align of type " + _type.identifier() + " requested."); + } + + return templ.render(); + }); +} + string ABIFunctions::shiftLeftFunction(size_t _numBits) { solAssert(_numBits < 256, ""); @@ -1680,11 +1809,11 @@ string ABIFunctions::nextArrayElementFunction(ArrayType const& _type) }); } -string ABIFunctions::arrayStoreLengthForEncodingFunction(ArrayType const& _type) +string ABIFunctions::arrayStoreLengthForEncodingFunction(ArrayType const& _type, EncodingOptions const& _options) { - string functionName = "array_storeLengthForEncoding_" + _type.identifier(); + string functionName = "array_storeLengthForEncoding_" + _type.identifier() + _options.toFunctionNameSuffix(); return createFunction(functionName, [&]() { - if (_type.isDynamicallySized()) + if (_type.isDynamicallySized() && !_options.dynamicInplace) return Whiskers(R"( function (pos, length) -> updated_pos { mstore(pos, length) diff --git a/libsolidity/codegen/ABIFunctions.h b/libsolidity/codegen/ABIFunctions.h index 47e40527fbcd..9aaddba4d58b 100644 --- a/libsolidity/codegen/ABIFunctions.h +++ b/libsolidity/codegen/ABIFunctions.h @@ -89,7 +89,9 @@ class ABIFunctions private: struct EncodingOptions { - /// Pad/signextend value types and bytes/string to multiples of 32 bytes. + /// Pad/signextend value types and bytes/string to multiples of 32 bytes. + /// If false, data is always left-aligned. + /// Note that this is always re-set to true for the elements of arrays and structs. bool padded = true; /// Store arrays and structs in place without "data pointer" and do not store the length. bool dynamicInplace = false; @@ -134,6 +136,14 @@ class ABIFunctions Type const& _targetType, EncodingOptions const& _options ); + /// @returns the name of a function that internally calls `abiEncodingFunction` + /// but always returns the updated encoding position, even if the type is + /// statically encoded. + std::string abiEncodeAndReturnUpdatedPosFunction( + Type const& _givenType, + Type const& _targetType, + EncodingOptions const& _options + ); /// Part of @a abiEncodingFunction for array target type and given calldata array. std::string abiEncodingFunctionCalldataArray( Type const& _givenType, @@ -212,6 +222,10 @@ class ABIFunctions /// Pads with zeros and might write more than exactly length. std::string copyToMemoryFunction(bool _fromCalldata); + /// @returns the name of a function that takes a (cleaned) value of the given value type and + /// left-aligns it, usually for use in non-padded encoding. + std::string leftAlignFunction(Type const& _type); + std::string shiftLeftFunction(size_t _numBits); std::string shiftRightFunction(size_t _numBits); /// @returns the name of a function that rounds its input to the next multiple @@ -232,9 +246,11 @@ class ABIFunctions std::string nextArrayElementFunction(ArrayType const& _type); /// @returns the name of a function used during encoding that stores the length - /// if the array is dynamically sized. It returns the new encoding position. - /// If the array is not dynamically sized, does nothing and just returns the position again. - std::string arrayStoreLengthForEncodingFunction(ArrayType const& _type); + /// if the array is dynamically sized (and the options do not request in-place encoding). + /// It returns the new encoding position. + /// If the array is not dynamically sized (or in-place encoding was requested), + /// does nothing and just returns the position again. + std::string arrayStoreLengthForEncodingFunction(ArrayType const& _type, EncodingOptions const& _options); /// @returns the name of a function that allocates memory. /// Modifies the "free memory pointer" From a08f4f43fa64e10cc43f66fc55824e21827373b5 Mon Sep 17 00:00:00 2001 From: chriseth Date: Wed, 23 Jan 2019 11:32:25 +0100 Subject: [PATCH 089/109] Add packed encoder V2. --- libsolidity/codegen/ABIFunctions.cpp | 68 +++++++++++++++++++++++++++ libsolidity/codegen/ABIFunctions.h | 16 +++++-- libsolidity/codegen/CompilerUtils.cpp | 19 ++++++-- libsolidity/codegen/CompilerUtils.h | 3 +- 4 files changed, 98 insertions(+), 8 deletions(-) diff --git a/libsolidity/codegen/ABIFunctions.cpp b/libsolidity/codegen/ABIFunctions.cpp index a672457cb387..1d532f5dac3f 100644 --- a/libsolidity/codegen/ABIFunctions.cpp +++ b/libsolidity/codegen/ABIFunctions.cpp @@ -107,6 +107,74 @@ string ABIFunctions::tupleEncoder( }); } +string ABIFunctions::tupleEncoderPacked( + TypePointers const& _givenTypes, + TypePointers const& _targetTypes +) +{ + EncodingOptions options; + options.encodeAsLibraryTypes = false; + options.encodeFunctionFromStack = true; + options.padded = false; + options.dynamicInplace = true; + + string functionName = string("abi_encode_tuple_packed_"); + for (auto const& t: _givenTypes) + functionName += t->identifier() + "_"; + functionName += "_to_"; + for (auto const& t: _targetTypes) + functionName += t->identifier() + "_"; + functionName += options.toFunctionNameSuffix(); + + return createExternallyUsedFunction(functionName, [&]() { + solAssert(!_givenTypes.empty(), ""); + + // Note that the values are in reverse due to the difference in calling semantics. + Whiskers templ(R"( + function (pos ) -> end { + + end := pos + } + )"); + templ("functionName", functionName); + string valueParams; + string encodeElements; + size_t stackPos = 0; + for (size_t i = 0; i < _givenTypes.size(); ++i) + { + solAssert(_givenTypes[i], ""); + solAssert(_targetTypes[i], ""); + size_t sizeOnStack = _givenTypes[i]->sizeOnStack(); + string valueNames = ""; + for (size_t j = 0; j < sizeOnStack; j++) + { + valueNames += "value" + to_string(stackPos) + ", "; + valueParams = ", value" + to_string(stackPos) + valueParams; + stackPos++; + } + bool dynamic = _targetTypes[i]->isDynamicallyEncoded(); + Whiskers elementTempl( + dynamic ? + string(R"( + pos := ( pos) + )") : + string(R"( + ( pos) + pos := add(pos, ) + )") + ); + elementTempl("values", valueNames); + if (!dynamic) + elementTempl("calldataEncodedSize", to_string(_targetTypes[i]->calldataEncodedSize(false))); + elementTempl("abiEncode", abiEncodingFunction(*_givenTypes[i], *_targetTypes[i], options)); + encodeElements += elementTempl.render(); + } + templ("valueParams", valueParams); + templ("encodeElements", encodeElements); + + return templ.render(); + }); +} string ABIFunctions::tupleDecoder(TypePointers const& _types, bool _fromMemory) { string functionName = string("abi_decode_tuple_"); diff --git a/libsolidity/codegen/ABIFunctions.h b/libsolidity/codegen/ABIFunctions.h index 9aaddba4d58b..a8a3c64e52d9 100644 --- a/libsolidity/codegen/ABIFunctions.h +++ b/libsolidity/codegen/ABIFunctions.h @@ -60,16 +60,26 @@ class ABIFunctions /// The values represent stack slots. If a type occupies more or less than one /// stack slot, it takes exactly that number of values. /// Returns a pointer to the end of the area written in memory. - /// Does not allocate memory (does not change the memory head pointer), but writes + /// Does not allocate memory (does not change the free memory pointer), but writes /// to memory starting at $headStart and an unrestricted amount after that. - /// Assigns the end of encoded memory either to $value0 or (if that is not present) - /// to $headStart. std::string tupleEncoder( TypePointers const& _givenTypes, TypePointers const& _targetTypes, bool _encodeAsLibraryTypes = false ); + /// @returns name of an assembly function to encode values of @a _givenTypes + /// with packed encoding into memory, converting the types to @a _targetTypes on the fly. + /// Parameters are: ... , i.e. + /// the layout on the stack is ... with + /// the top of the stack on the right. + /// The values represent stack slots. If a type occupies more or less than one + /// stack slot, it takes exactly that number of values. + /// Returns a pointer to the end of the area written in memory. + /// Does not allocate memory (does not change the free memory pointer), but writes + /// to memory starting at memPos and an unrestricted amount after that. + std::string tupleEncoderPacked(TypePointers const& _givenTypes, TypePointers const& _targetTypes); + /// @returns name of an assembly function to ABI-decode values of @a _types /// into memory. If @a _fromMemory is true, decodes from memory instead of /// from calldata. diff --git a/libsolidity/codegen/CompilerUtils.cpp b/libsolidity/codegen/CompilerUtils.cpp index 6cfb077783d1..676dd5b602ee 100644 --- a/libsolidity/codegen/CompilerUtils.cpp +++ b/libsolidity/codegen/CompilerUtils.cpp @@ -348,11 +348,15 @@ void CompilerUtils::encodeToMemory( if (_givenTypes.empty()) return; - else if (_padToWordBoundaries && !_copyDynamicDataInPlace && encoderV2) + if (encoderV2) { // Use the new Yul-based encoding function + solAssert( + _padToWordBoundaries != _copyDynamicDataInPlace, + "Non-padded and in-place encoding can only be combined." + ); auto stackHeightBefore = m_context.stackHeight(); - abiEncodeV2(_givenTypes, targetTypes, _encodeAsLibraryTypes); + abiEncodeV2(_givenTypes, targetTypes, _encodeAsLibraryTypes, _padToWordBoundaries); solAssert(stackHeightBefore - m_context.stackHeight() == sizeOnStack(_givenTypes), ""); return; } @@ -466,15 +470,22 @@ void CompilerUtils::encodeToMemory( void CompilerUtils::abiEncodeV2( TypePointers const& _givenTypes, TypePointers const& _targetTypes, - bool _encodeAsLibraryTypes + bool _encodeAsLibraryTypes, + bool _padToWordBoundaries ) { + if (!_padToWordBoundaries) + solAssert(!_encodeAsLibraryTypes, "Library calls cannot be packed."); + // stack: <$value0> <$value1> ... <$value(n-1)> <$headStart> auto ret = m_context.pushNewTag(); moveIntoStack(sizeOnStack(_givenTypes) + 1); - string encoderName = m_context.abiFunctions().tupleEncoder(_givenTypes, _targetTypes, _encodeAsLibraryTypes); + string encoderName = + _padToWordBoundaries ? + m_context.abiFunctions().tupleEncoder(_givenTypes, _targetTypes, _encodeAsLibraryTypes) : + m_context.abiFunctions().tupleEncoderPacked(_givenTypes, _targetTypes); m_context.appendJumpTo(m_context.namedTag(encoderName)); m_context.adjustStackOffset(-int(sizeOnStack(_givenTypes)) - 1); m_context << ret.tag(); diff --git a/libsolidity/codegen/CompilerUtils.h b/libsolidity/codegen/CompilerUtils.h index 6bde2e8bd4b7..d095e05f99ac 100644 --- a/libsolidity/codegen/CompilerUtils.h +++ b/libsolidity/codegen/CompilerUtils.h @@ -159,7 +159,8 @@ class CompilerUtils void abiEncodeV2( TypePointers const& _givenTypes, TypePointers const& _targetTypes, - bool _encodeAsLibraryTypes = false + bool _encodeAsLibraryTypes = false, + bool _padToWordBoundaries = true ); /// Decodes data from ABI encoding into internal encoding. If @a _fromMemory is set to true, From 0c2d623ee46e04c9d16e5681bda7d5f10acec1a9 Mon Sep 17 00:00:00 2001 From: chriseth Date: Thu, 24 Jan 2019 13:52:06 +0100 Subject: [PATCH 090/109] Enable struct encoding. --- libsolidity/ast/Types.cpp | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/libsolidity/ast/Types.cpp b/libsolidity/ast/Types.cpp index 308c0fe5edda..04d2602d9e2f 100644 --- a/libsolidity/ast/Types.cpp +++ b/libsolidity/ast/Types.cpp @@ -415,7 +415,7 @@ MemberList const& Type::members(ContractDefinition const* _currentScope) const return *m_members[_currentScope]; } -TypePointer Type::fullEncodingType(bool _inLibraryCall, bool _encoderV2, bool _packed) const +TypePointer Type::fullEncodingType(bool _inLibraryCall, bool _encoderV2, bool) const { TypePointer encodingType = mobileType(); if (encodingType) @@ -423,7 +423,7 @@ TypePointer Type::fullEncodingType(bool _inLibraryCall, bool _encoderV2, bool _p if (encodingType) encodingType = encodingType->encodingType(); // Structs are fine in the following circumstances: - // - ABIv2 without packed encoding or, + // - ABIv2 or, // - storage struct for a library if (_inLibraryCall && encodingType->dataStoredIn(DataLocation::Storage)) return encodingType; @@ -431,7 +431,7 @@ TypePointer Type::fullEncodingType(bool _inLibraryCall, bool _encoderV2, bool _p while (auto const* arrayType = dynamic_cast(baseType.get())) baseType = arrayType->baseType(); if (dynamic_cast(baseType.get())) - if (!_encoderV2 || _packed) + if (!_encoderV2) return TypePointer(); return encodingType; } From a8d0ef4bad8f77b737b2004d7d2e855598ef1286 Mon Sep 17 00:00:00 2001 From: chriseth Date: Thu, 24 Jan 2019 14:40:22 +0100 Subject: [PATCH 091/109] Allow indexed structs in events with encoder v2. --- libsolidity/analysis/TypeChecker.cpp | 10 ---------- libsolidity/codegen/ExpressionCompiler.cpp | 13 +++++++++---- 2 files changed, 9 insertions(+), 14 deletions(-) diff --git a/libsolidity/analysis/TypeChecker.cpp b/libsolidity/analysis/TypeChecker.cpp index 4b1f28cda6cb..ed0af364278c 100644 --- a/libsolidity/analysis/TypeChecker.cpp +++ b/libsolidity/analysis/TypeChecker.cpp @@ -578,17 +578,7 @@ bool TypeChecker::visit(EventDefinition const& _eventDef) for (ASTPointer const& var: _eventDef.parameters()) { if (var->isIndexed()) - { numIndexed++; - if ( - _eventDef.sourceUnit().annotation().experimentalFeatures.count(ExperimentalFeature::ABIEncoderV2) && - dynamic_cast(type(*var).get()) - ) - m_errorReporter.typeError( - var->location(), - "Indexed reference types cannot yet be used with ABIEncoderV2." - ); - } if (!type(*var)->canLiveOutsideStorage()) m_errorReporter.typeError(var->location(), "Type is required to live outside storage."); if (!type(*var)->interfaceType(false)) diff --git a/libsolidity/codegen/ExpressionCompiler.cpp b/libsolidity/codegen/ExpressionCompiler.cpp index c1079ed32b31..942363461237 100644 --- a/libsolidity/codegen/ExpressionCompiler.cpp +++ b/libsolidity/codegen/ExpressionCompiler.cpp @@ -731,6 +731,7 @@ bool ExpressionCompiler::visit(FunctionCall const& _functionCall) } arguments.front()->accept(*this); utils().fetchFreeMemoryPointer(); + solAssert(function.parameterTypes().front()->isValueType(), ""); utils().packedEncode( {arguments.front()->annotation().type}, {function.parameterTypes().front()} @@ -744,28 +745,32 @@ bool ExpressionCompiler::visit(FunctionCall const& _functionCall) _functionCall.expression().accept(*this); auto const& event = dynamic_cast(function.declaration()); unsigned numIndexed = 0; + TypePointers paramTypes = function.parameterTypes(); // All indexed arguments go to the stack for (unsigned arg = arguments.size(); arg > 0; --arg) if (event.parameters()[arg - 1]->isIndexed()) { ++numIndexed; arguments[arg - 1]->accept(*this); - if (auto const& arrayType = dynamic_pointer_cast(function.parameterTypes()[arg - 1])) + if (auto const& referenceType = dynamic_pointer_cast(paramTypes[arg - 1])) { utils().fetchFreeMemoryPointer(); utils().packedEncode( {arguments[arg - 1]->annotation().type}, - {arrayType} + {referenceType} ); utils().toSizeAfterFreeMemoryPointer(); m_context << Instruction::KECCAK256; } else + { + solAssert(paramTypes[arg - 1]->isValueType(), ""); utils().convertType( *arguments[arg - 1]->annotation().type, - *function.parameterTypes()[arg - 1], + *paramTypes[arg - 1], true ); + } } if (!event.isAnonymous()) { @@ -782,7 +787,7 @@ bool ExpressionCompiler::visit(FunctionCall const& _functionCall) { arguments[arg]->accept(*this); nonIndexedArgTypes.push_back(arguments[arg]->annotation().type); - nonIndexedParamTypes.push_back(function.parameterTypes()[arg]); + nonIndexedParamTypes.push_back(paramTypes[arg]); } utils().fetchFreeMemoryPointer(); utils().abiEncode(nonIndexedArgTypes, nonIndexedParamTypes); From 7684d886ae1655edf50a17082d3cd608613f494e Mon Sep 17 00:00:00 2001 From: chriseth Date: Wed, 23 Jan 2019 17:24:54 +0100 Subject: [PATCH 092/109] Semantic tests. --- test/libsolidity/SolidityEndToEndTest.cpp | 329 +++++++++++++++++++++- 1 file changed, 325 insertions(+), 4 deletions(-) diff --git a/test/libsolidity/SolidityEndToEndTest.cpp b/test/libsolidity/SolidityEndToEndTest.cpp index e738eb04bc2c..d721bcbda988 100644 --- a/test/libsolidity/SolidityEndToEndTest.cpp +++ b/test/libsolidity/SolidityEndToEndTest.cpp @@ -30,6 +30,8 @@ #include +#include + #include #include @@ -13478,13 +13480,332 @@ BOOST_AUTO_TEST_CASE(abi_encodePacked) y[0] = "e"; require(y[0] == "e"); } + function f4() public pure returns (bytes memory) { + string memory x = "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz"; + return abi.encodePacked(uint16(0x0701), x, uint16(0x1201)); + } + function f_literal() public pure returns (bytes memory) { + return abi.encodePacked(uint8(0x01), "abc", uint8(0x02)); + } + function f_calldata() public pure returns (bytes memory) { + return abi.encodePacked(uint8(0x01), msg.data, uint8(0x02)); + } + } + )"; + for (auto v2: {false, true}) + { + compileAndRun(string(v2 ? "pragma experimental ABIEncoderV2;\n" : "") + sourceCode, 0, "C"); + ABI_CHECK(callContractFunction("f0()"), encodeArgs(0x20, 0)); + ABI_CHECK(callContractFunction("f1()"), encodeArgs(0x20, 2, "\x01\x02")); + ABI_CHECK(callContractFunction("f2()"), encodeArgs(0x20, 5, "\x01" "abc" "\x02")); + ABI_CHECK(callContractFunction("f3()"), encodeArgs(0x20, 5, "\x01" "abc" "\x02")); + ABI_CHECK(callContractFunction("f4()"), encodeArgs( + 0x20, + 2 + 26 + 26 + 2, + "\x07\x01" "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz" "\x12\x01" + )); + ABI_CHECK(callContractFunction("f_literal()"), encodeArgs(0x20, 5, "\x01" "abc" "\x02")); + ABI_CHECK(callContractFunction("f_calldata()"), encodeArgs(0x20, 6, "\x01" "\xa5\xbf\xa1\xee" "\x02")); + } +} + +BOOST_AUTO_TEST_CASE(abi_encodePacked_from_storage) +{ + char const* sourceCode = R"( + contract C { + uint24[9] small_fixed; + int24[9] small_fixed_signed; + uint24[] small_dyn; + uint248[5] large_fixed; + uint248[] large_dyn; + bytes bytes_storage; + function sf() public returns (bytes memory) { + small_fixed[0] = 0xfffff1; + small_fixed[2] = 0xfffff2; + small_fixed[5] = 0xfffff3; + small_fixed[8] = 0xfffff4; + return abi.encodePacked(uint8(0x01), small_fixed, uint8(0x02)); + } + function sd() public returns (bytes memory) { + small_dyn.length = 9; + small_dyn[0] = 0xfffff1; + small_dyn[2] = 0xfffff2; + small_dyn[5] = 0xfffff3; + small_dyn[8] = 0xfffff4; + return abi.encodePacked(uint8(0x01), small_dyn, uint8(0x02)); + } + function sfs() public returns (bytes memory) { + small_fixed_signed[0] = -2; + small_fixed_signed[2] = 0xffff2; + small_fixed_signed[5] = -200; + small_fixed_signed[8] = 0xffff4; + return abi.encodePacked(uint8(0x01), small_fixed_signed, uint8(0x02)); + } + function lf() public returns (bytes memory) { + large_fixed[0] = 2**248-1; + large_fixed[1] = 0xfffff2; + large_fixed[2] = 2**248-2; + large_fixed[4] = 0xfffff4; + return abi.encodePacked(uint8(0x01), large_fixed, uint8(0x02)); + } + function ld() public returns (bytes memory) { + large_dyn.length = 5; + large_dyn[0] = 2**248-1; + large_dyn[1] = 0xfffff2; + large_dyn[2] = 2**248-2; + large_dyn[4] = 0xfffff4; + return abi.encodePacked(uint8(0x01), large_dyn, uint8(0x02)); + } + function bytes_short() public returns (bytes memory) { + bytes_storage = "abcd"; + return abi.encodePacked(uint8(0x01), bytes_storage, uint8(0x02)); + } + function bytes_long() public returns (bytes memory) { + bytes_storage = "0123456789012345678901234567890123456789"; + return abi.encodePacked(uint8(0x01), bytes_storage, uint8(0x02)); + } + } + )"; + for (auto v2: {false, true}) + { + compileAndRun(string(v2 ? "pragma experimental ABIEncoderV2;\n" : "") + sourceCode, 0, "C"); + bytes payload = encodeArgs(0xfffff1, 0, 0xfffff2, 0, 0, 0xfffff3, 0, 0, 0xfffff4); + bytes encoded = encodeArgs(0x20, 0x122, "\x01" + asString(payload) + "\x02"); + ABI_CHECK(callContractFunction("sf()"), encoded); + ABI_CHECK(callContractFunction("sd()"), encoded); + ABI_CHECK(callContractFunction("sfs()"), encodeArgs(0x20, 0x122, "\x01" + asString(encodeArgs( + u256(-2), 0, 0xffff2, 0, 0, u256(-200), 0, 0, 0xffff4 + )) + "\x02")); + payload = encodeArgs( + u256("0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"), + 0xfffff2, + u256("0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe"), + 0, + 0xfffff4 + ); + ABI_CHECK(callContractFunction("lf()"), encodeArgs(0x20, 5 * 32 + 2, "\x01" + asString(encodeArgs(payload)) + "\x02")); + ABI_CHECK(callContractFunction("ld()"), encodeArgs(0x20, 5 * 32 + 2, "\x01" + asString(encodeArgs(payload)) + "\x02")); + ABI_CHECK(callContractFunction("bytes_short()"), encodeArgs(0x20, 6, "\x01" "abcd\x02")); + ABI_CHECK( + callContractFunction("bytes_long()"), + encodeArgs(0x20, 42, "\x01" "0123456789012345678901234567890123456789\x02") + ); + } +} + +BOOST_AUTO_TEST_CASE(abi_encodePacked_from_memory) +{ + char const* sourceCode = R"( + contract C { + function sf() public pure returns (bytes memory) { + uint24[9] memory small_fixed; + small_fixed[0] = 0xfffff1; + small_fixed[2] = 0xfffff2; + small_fixed[5] = 0xfffff3; + small_fixed[8] = 0xfffff4; + return abi.encodePacked(uint8(0x01), small_fixed, uint8(0x02)); + } + function sd() public pure returns (bytes memory) { + uint24[] memory small_dyn = new uint24[](9); + small_dyn[0] = 0xfffff1; + small_dyn[2] = 0xfffff2; + small_dyn[5] = 0xfffff3; + small_dyn[8] = 0xfffff4; + return abi.encodePacked(uint8(0x01), small_dyn, uint8(0x02)); + } + function sfs() public pure returns (bytes memory) { + int24[9] memory small_fixed_signed; + small_fixed_signed[0] = -2; + small_fixed_signed[2] = 0xffff2; + small_fixed_signed[5] = -200; + small_fixed_signed[8] = 0xffff4; + return abi.encodePacked(uint8(0x01), small_fixed_signed, uint8(0x02)); + } + function lf() public pure returns (bytes memory) { + uint248[5] memory large_fixed; + large_fixed[0] = 2**248-1; + large_fixed[1] = 0xfffff2; + large_fixed[2] = 2**248-2; + large_fixed[4] = 0xfffff4; + return abi.encodePacked(uint8(0x01), large_fixed, uint8(0x02)); + } + function ld() public pure returns (bytes memory) { + uint248[] memory large_dyn = new uint248[](5); + large_dyn[0] = 2**248-1; + large_dyn[1] = 0xfffff2; + large_dyn[2] = 2**248-2; + large_dyn[4] = 0xfffff4; + return abi.encodePacked(uint8(0x01), large_dyn, uint8(0x02)); + } + } + )"; + for (auto v2: {false, true}) + { + compileAndRun(string(v2 ? "pragma experimental ABIEncoderV2;\n" : "") + sourceCode, 0, "C"); + bytes payload = encodeArgs(0xfffff1, 0, 0xfffff2, 0, 0, 0xfffff3, 0, 0, 0xfffff4); + bytes encoded = encodeArgs(0x20, 0x122, "\x01" + asString(payload) + "\x02"); + ABI_CHECK(callContractFunction("sf()"), encoded); + ABI_CHECK(callContractFunction("sd()"), encoded); + ABI_CHECK(callContractFunction("sfs()"), encodeArgs(0x20, 0x122, "\x01" + asString(encodeArgs( + u256(-2), 0, 0xffff2, 0, 0, u256(-200), 0, 0, 0xffff4 + )) + "\x02")); + payload = encodeArgs( + u256("0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"), + 0xfffff2, + u256("0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe"), + 0, + 0xfffff4 + ); + ABI_CHECK(callContractFunction("lf()"), encodeArgs(0x20, 5 * 32 + 2, "\x01" + asString(encodeArgs(payload)) + "\x02")); + ABI_CHECK(callContractFunction("ld()"), encodeArgs(0x20, 5 * 32 + 2, "\x01" + asString(encodeArgs(payload)) + "\x02")); + } +} + +BOOST_AUTO_TEST_CASE(abi_encodePacked_functionPtr) +{ + char const* sourceCode = R"( + contract C { + C other = C(0x1112131400000000000011121314000000000087); + function testDirect() public view returns (bytes memory) { + return abi.encodePacked(uint8(8), other.f, uint8(2)); + } + function testFixedArray() public view returns (bytes memory) { + function () external pure returns (bytes memory)[1] memory x; + x[0] = other.f; + return abi.encodePacked(uint8(8), x, uint8(2)); + } + function testDynamicArray() public view returns (bytes memory) { + function () external pure returns (bytes memory)[] memory x = new function() external pure returns (bytes memory)[](1); + x[0] = other.f; + return abi.encodePacked(uint8(8), x, uint8(2)); + } + function f() public pure returns (bytes memory) {} + } + )"; + for (auto v2: {false, true}) + { + compileAndRun(string(v2 ? "pragma experimental ABIEncoderV2;\n" : "") + sourceCode, 0, "C"); + string directEncoding = asString(fromHex("08" "1112131400000000000011121314000000000087" "26121ff0" "02")); + ABI_CHECK(callContractFunction("testDirect()"), encodeArgs(0x20, directEncoding.size(), directEncoding)); + string arrayEncoding = asString(fromHex("08" "1112131400000000000011121314000000000087" "26121ff0" "0000000000000000" "02")); + ABI_CHECK(callContractFunction("testFixedArray()"), encodeArgs(0x20, arrayEncoding.size(), arrayEncoding)); + ABI_CHECK(callContractFunction("testDynamicArray()"), encodeArgs(0x20, arrayEncoding.size(), arrayEncoding)); + } +} + +BOOST_AUTO_TEST_CASE(abi_encodePackedV2_structs) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + struct S { + uint8 a; + int16 b; + uint8[2] c; + int16[] d; + } + S s; + event E(S indexed); + constructor() public { + s.a = 0x12; + s.b = -7; + s.c[0] = 2; + s.c[1] = 3; + s.d.length = 2; + s.d[0] = -7; + s.d[1] = -8; + } + function testStorage() public returns (bytes memory) { + emit E(s); + return abi.encodePacked(uint8(0x33), s, uint8(0x44)); + } + function testMemory() public returns (bytes memory) { + S memory m = s; + emit E(m); + return abi.encodePacked(uint8(0x33), m, uint8(0x44)); + } } )"; compileAndRun(sourceCode, 0, "C"); - ABI_CHECK(callContractFunction("f0()"), encodeArgs(0x20, 0)); - ABI_CHECK(callContractFunction("f1()"), encodeArgs(0x20, 2, "\x01\x02")); - ABI_CHECK(callContractFunction("f2()"), encodeArgs(0x20, 5, "\x01" "abc" "\x02")); - ABI_CHECK(callContractFunction("f3()"), encodeArgs(0x20, 5, "\x01" "abc" "\x02")); + bytes structEnc = encodeArgs(int(0x12), u256(-7), int(2), int(3), u256(-7), u256(-8)); + string encoding = "\x33" + asString(structEnc) + "\x44"; + ABI_CHECK(callContractFunction("testStorage()"), encodeArgs(0x20, encoding.size(), encoding)); + BOOST_REQUIRE_EQUAL(m_logs[0].topics.size(), 2); + BOOST_CHECK_EQUAL(m_logs[0].topics[0], dev::keccak256(string("E((uint8,int16,uint8[2],int16[]))"))); + BOOST_CHECK_EQUAL(m_logs[0].topics[1], dev::keccak256(asString(structEnc))); + ABI_CHECK(callContractFunction("testMemory()"), encodeArgs(0x20, encoding.size(), encoding)); + BOOST_REQUIRE_EQUAL(m_logs[0].topics.size(), 2); + BOOST_CHECK_EQUAL(m_logs[0].topics[0], dev::keccak256(string("E((uint8,int16,uint8[2],int16[]))"))); + BOOST_CHECK_EQUAL(m_logs[0].topics[1], dev::keccak256(asString(structEnc))); +} + +BOOST_AUTO_TEST_CASE(abi_encodePackedV2_nestedArray) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + struct S { + uint8 a; + int16 b; + } + event E(S[2][][3] indexed); + function testNestedArrays() public returns (bytes memory) { + S[2][][3] memory x; + x[1] = new S[2][](2); + x[1][0][0].a = 1; + x[1][0][0].b = 2; + x[1][0][1].a = 3; + x[1][1][1].b = 4; + emit E(x); + return abi.encodePacked(uint8(0x33), x, uint8(0x44)); + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + bytes structEnc = encodeArgs(1, 2, 3, 0, 0, 0, 0, 4); + string encoding = "\x33" + asString(structEnc) + "\x44"; + ABI_CHECK(callContractFunction("testNestedArrays()"), encodeArgs(0x20, encoding.size(), encoding)); + BOOST_REQUIRE_EQUAL(m_logs[0].topics.size(), 2); + BOOST_CHECK_EQUAL(m_logs[0].topics[0], dev::keccak256(string("E((uint8,int16)[2][][3])"))); + BOOST_CHECK_EQUAL(m_logs[0].topics[1], dev::keccak256(asString(structEnc))); +} + +BOOST_AUTO_TEST_CASE(abi_encodePackedV2_arrayOfStrings) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + string[] x; + event E(string[] indexed); + constructor() public { + x.length = 2; + x[0] = "abc"; + x[1] = "0123456789012345678901234567890123456789"; + } + function testStorage() public returns (bytes memory) { + emit E(x); + return abi.encodePacked(uint8(0x33), x, uint8(0x44)); + } + function testMemory() public returns (bytes memory) { + string[] memory y = x; + emit E(y); + return abi.encodePacked(uint8(0x33), y, uint8(0x44)); + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + bytes arrayEncoding = encodeArgs("abc", "0123456789012345678901234567890123456789"); + // This pads to multiple of 32 bytes + string encoding = "\x33" + asString(arrayEncoding) + "\x44"; + BOOST_CHECK_EQUAL(encoding.size(), 2 + 32 * 3); + ABI_CHECK(callContractFunction("testStorage()"), encodeArgs(0x20, encoding.size(), encoding)); + BOOST_REQUIRE_EQUAL(m_logs[0].topics.size(), 2); + BOOST_CHECK_EQUAL(m_logs[0].topics[0], dev::keccak256(string("E(string[])"))); + BOOST_CHECK_EQUAL(m_logs[0].topics[1], dev::keccak256(asString(arrayEncoding))); + ABI_CHECK(callContractFunction("testMemory()"), encodeArgs(0x20, encoding.size(), encoding)); + BOOST_REQUIRE_EQUAL(m_logs[0].topics.size(), 2); + BOOST_CHECK_EQUAL(m_logs[0].topics[0], dev::keccak256(string("E(string[])"))); + BOOST_CHECK_EQUAL(m_logs[0].topics[1], dev::keccak256(asString(arrayEncoding))); } BOOST_AUTO_TEST_CASE(abi_encode_with_selector) From 299e3b5388cb51df654d7ab01cd18f74f07278a9 Mon Sep 17 00:00:00 2001 From: chriseth Date: Thu, 24 Jan 2019 15:37:25 +0100 Subject: [PATCH 093/109] Changelog entry. --- Changelog.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Changelog.md b/Changelog.md index 5e049967f39e..0f8a2bbc9ffa 100644 --- a/Changelog.md +++ b/Changelog.md @@ -12,6 +12,7 @@ Language Features: Compiler Features: + * Implement packed encoding for ABIEncoderV2. * C API (``libsolc`` / raw ``soljson.js``): Introduce ``solidity_free`` method which releases all internal buffers to save memory. * Commandline interface: Adds new option ``--new-reporter`` for improved diagnostics formatting along with ``--color`` and ``--no-color`` for colorized output to be forced (or explicitly disabled). From 227addfcefd017bc22bb473ddbd8ca6eb4ed5e76 Mon Sep 17 00:00:00 2001 From: chriseth Date: Thu, 24 Jan 2019 15:44:42 +0100 Subject: [PATCH 094/109] Syntax tests. --- test/libsolidity/syntaxTests/events/event_array_indexed_v2.sol | 1 - .../syntaxTests/events/event_nested_array_indexed_v2.sol | 1 - test/libsolidity/syntaxTests/events/event_struct_indexed_v2.sol | 1 - ...coding_structs_abiv2.sol => abi_encodePacked_structs_v2.sol} | 2 -- .../syntaxTests/specialFunctions/abi_encode_structs_abiv2.sol | 2 -- 5 files changed, 7 deletions(-) rename test/libsolidity/syntaxTests/specialFunctions/{types_with_unspecified_encoding_structs_abiv2.sol => abi_encodePacked_structs_v2.sol} (75%) diff --git a/test/libsolidity/syntaxTests/events/event_array_indexed_v2.sol b/test/libsolidity/syntaxTests/events/event_array_indexed_v2.sol index 3f729a6a5cb0..773706e363df 100644 --- a/test/libsolidity/syntaxTests/events/event_array_indexed_v2.sol +++ b/test/libsolidity/syntaxTests/events/event_array_indexed_v2.sol @@ -4,4 +4,3 @@ contract c { } // ---- // Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. -// TypeError: (59-73): Indexed reference types cannot yet be used with ABIEncoderV2. diff --git a/test/libsolidity/syntaxTests/events/event_nested_array_indexed_v2.sol b/test/libsolidity/syntaxTests/events/event_nested_array_indexed_v2.sol index f05b884e9bc2..5ed4b496ad16 100644 --- a/test/libsolidity/syntaxTests/events/event_nested_array_indexed_v2.sol +++ b/test/libsolidity/syntaxTests/events/event_nested_array_indexed_v2.sol @@ -4,4 +4,3 @@ contract c { } // ---- // Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. -// TypeError: (59-75): Indexed reference types cannot yet be used with ABIEncoderV2. diff --git a/test/libsolidity/syntaxTests/events/event_struct_indexed_v2.sol b/test/libsolidity/syntaxTests/events/event_struct_indexed_v2.sol index a1d8cf04e950..334fa539b919 100644 --- a/test/libsolidity/syntaxTests/events/event_struct_indexed_v2.sol +++ b/test/libsolidity/syntaxTests/events/event_struct_indexed_v2.sol @@ -5,4 +5,3 @@ contract c { } // ---- // Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. -// TypeError: (85-94): Indexed reference types cannot yet be used with ABIEncoderV2. diff --git a/test/libsolidity/syntaxTests/specialFunctions/types_with_unspecified_encoding_structs_abiv2.sol b/test/libsolidity/syntaxTests/specialFunctions/abi_encodePacked_structs_v2.sol similarity index 75% rename from test/libsolidity/syntaxTests/specialFunctions/types_with_unspecified_encoding_structs_abiv2.sol rename to test/libsolidity/syntaxTests/specialFunctions/abi_encodePacked_structs_v2.sol index 387028259420..0ae504665615 100644 --- a/test/libsolidity/syntaxTests/specialFunctions/types_with_unspecified_encoding_structs_abiv2.sol +++ b/test/libsolidity/syntaxTests/specialFunctions/abi_encodePacked_structs_v2.sol @@ -12,5 +12,3 @@ contract C { } // ---- // Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. -// TypeError: (191-192): This type cannot be encoded. -// TypeError: (194-195): This type cannot be encoded. diff --git a/test/libsolidity/syntaxTests/specialFunctions/abi_encode_structs_abiv2.sol b/test/libsolidity/syntaxTests/specialFunctions/abi_encode_structs_abiv2.sol index d6cf60e47f25..1b491b7fd85e 100644 --- a/test/libsolidity/syntaxTests/specialFunctions/abi_encode_structs_abiv2.sol +++ b/test/libsolidity/syntaxTests/specialFunctions/abi_encode_structs_abiv2.sol @@ -14,5 +14,3 @@ contract C { } // ---- // Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. -// TypeError: (235-236): This type cannot be encoded. -// TypeError: (238-239): This type cannot be encoded. From fe2429de9f6c5594b1a7cef347fab0966902ca7f Mon Sep 17 00:00:00 2001 From: Mathias Baumann Date: Mon, 4 Feb 2019 17:13:41 +0000 Subject: [PATCH 095/109] Packed Encoding: Disallow types in v2 that aren't allowed in v1 --- libsolidity/analysis/TypeChecker.cpp | 21 +++++++++---- test/libsolidity/SolidityEndToEndTest.cpp | 30 +++++++------------ .../abi_encodePacked_structs_v2.sol | 2 ++ .../specialFunctions/abi_encode_structs.sol | 4 +-- .../abi_encode_structs_abiv2.sol | 7 +++++ .../encodePacked_array_of_structs.sol | 2 +- .../encodePacked_dynamic_string_array_v2.sol | 13 ++++++++ ...ypes_with_unspecified_encoding_structs.sol | 4 +-- 8 files changed, 52 insertions(+), 31 deletions(-) create mode 100644 test/libsolidity/syntaxTests/specialFunctions/encodePacked_dynamic_string_array_v2.sol diff --git a/libsolidity/analysis/TypeChecker.cpp b/libsolidity/analysis/TypeChecker.cpp index ed0af364278c..dc6536499013 100644 --- a/libsolidity/analysis/TypeChecker.cpp +++ b/libsolidity/analysis/TypeChecker.cpp @@ -48,9 +48,9 @@ using namespace dev::solidity; namespace { -bool typeSupportedByOldABIEncoder(Type const& _type) +bool typeSupportedByOldABIEncoder(Type const& _type, bool _isLibraryCall) { - if (_type.dataStoredIn(DataLocation::Storage)) + if (_isLibraryCall && _type.dataStoredIn(DataLocation::Storage)) return true; if (_type.category() == Type::Category::Struct) return false; @@ -58,7 +58,7 @@ bool typeSupportedByOldABIEncoder(Type const& _type) { auto const& arrayType = dynamic_cast(_type); auto base = arrayType.baseType(); - if (!typeSupportedByOldABIEncoder(*base) || (base->category() == Type::Category::Array && base->isDynamicallySized())) + if (!typeSupportedByOldABIEncoder(*base, _isLibraryCall) || (base->category() == Type::Category::Array && base->isDynamicallySized())) return false; } return true; @@ -355,7 +355,7 @@ bool TypeChecker::visit(FunctionDefinition const& _function) if ( _function.isPublic() && !_function.sourceUnit().annotation().experimentalFeatures.count(ExperimentalFeature::ABIEncoderV2) && - !typeSupportedByOldABIEncoder(*type(var)) + !typeSupportedByOldABIEncoder(*type(var), isLibraryFunction) ) m_errorReporter.typeError( var.location(), @@ -475,7 +475,7 @@ bool TypeChecker::visit(VariableDeclaration const& _variable) { vector unsupportedTypes; for (auto const& param: getter.parameterTypes() + getter.returnParameterTypes()) - if (!typeSupportedByOldABIEncoder(*param)) + if (!typeSupportedByOldABIEncoder(*param, false /* isLibrary */)) unsupportedTypes.emplace_back(param->toString()); if (!unsupportedTypes.empty()) m_errorReporter.typeError(_variable.location(), @@ -585,7 +585,7 @@ bool TypeChecker::visit(EventDefinition const& _eventDef) m_errorReporter.typeError(var->location(), "Internal or recursive type is not allowed as event parameter type."); if ( !_eventDef.sourceUnit().annotation().experimentalFeatures.count(ExperimentalFeature::ABIEncoderV2) && - !typeSupportedByOldABIEncoder(*type(*var)) + !typeSupportedByOldABIEncoder(*type(*var), false /* isLibrary */) ) m_errorReporter.typeError( var->location(), @@ -1550,6 +1550,15 @@ void TypeChecker::typeCheckABIEncodeFunctions( } } + if (isPacked && !typeSupportedByOldABIEncoder(*argType, false /* isLibrary */)) + { + m_errorReporter.typeError( + arguments[i]->location(), + "Type not supported in packed mode." + ); + continue; + } + if (!argType->fullEncodingType(false, abiEncoderV2, !_functionType->padArguments())) m_errorReporter.typeError( arguments[i]->location(), diff --git a/test/libsolidity/SolidityEndToEndTest.cpp b/test/libsolidity/SolidityEndToEndTest.cpp index d721bcbda988..fb480459dc73 100644 --- a/test/libsolidity/SolidityEndToEndTest.cpp +++ b/test/libsolidity/SolidityEndToEndTest.cpp @@ -13715,25 +13715,22 @@ BOOST_AUTO_TEST_CASE(abi_encodePackedV2_structs) s.d[0] = -7; s.d[1] = -8; } - function testStorage() public returns (bytes memory) { + function testStorage() public { emit E(s); - return abi.encodePacked(uint8(0x33), s, uint8(0x44)); } - function testMemory() public returns (bytes memory) { + function testMemory() public { S memory m = s; emit E(m); - return abi.encodePacked(uint8(0x33), m, uint8(0x44)); } } )"; compileAndRun(sourceCode, 0, "C"); bytes structEnc = encodeArgs(int(0x12), u256(-7), int(2), int(3), u256(-7), u256(-8)); - string encoding = "\x33" + asString(structEnc) + "\x44"; - ABI_CHECK(callContractFunction("testStorage()"), encodeArgs(0x20, encoding.size(), encoding)); + ABI_CHECK(callContractFunction("testStorage()"), encodeArgs()); BOOST_REQUIRE_EQUAL(m_logs[0].topics.size(), 2); BOOST_CHECK_EQUAL(m_logs[0].topics[0], dev::keccak256(string("E((uint8,int16,uint8[2],int16[]))"))); BOOST_CHECK_EQUAL(m_logs[0].topics[1], dev::keccak256(asString(structEnc))); - ABI_CHECK(callContractFunction("testMemory()"), encodeArgs(0x20, encoding.size(), encoding)); + ABI_CHECK(callContractFunction("testMemory()"), encodeArgs()); BOOST_REQUIRE_EQUAL(m_logs[0].topics.size(), 2); BOOST_CHECK_EQUAL(m_logs[0].topics[0], dev::keccak256(string("E((uint8,int16,uint8[2],int16[]))"))); BOOST_CHECK_EQUAL(m_logs[0].topics[1], dev::keccak256(asString(structEnc))); @@ -13749,7 +13746,7 @@ BOOST_AUTO_TEST_CASE(abi_encodePackedV2_nestedArray) int16 b; } event E(S[2][][3] indexed); - function testNestedArrays() public returns (bytes memory) { + function testNestedArrays() public { S[2][][3] memory x; x[1] = new S[2][](2); x[1][0][0].a = 1; @@ -13757,14 +13754,12 @@ BOOST_AUTO_TEST_CASE(abi_encodePackedV2_nestedArray) x[1][0][1].a = 3; x[1][1][1].b = 4; emit E(x); - return abi.encodePacked(uint8(0x33), x, uint8(0x44)); } } )"; compileAndRun(sourceCode, 0, "C"); bytes structEnc = encodeArgs(1, 2, 3, 0, 0, 0, 0, 4); - string encoding = "\x33" + asString(structEnc) + "\x44"; - ABI_CHECK(callContractFunction("testNestedArrays()"), encodeArgs(0x20, encoding.size(), encoding)); + ABI_CHECK(callContractFunction("testNestedArrays()"), encodeArgs()); BOOST_REQUIRE_EQUAL(m_logs[0].topics.size(), 2); BOOST_CHECK_EQUAL(m_logs[0].topics[0], dev::keccak256(string("E((uint8,int16)[2][][3])"))); BOOST_CHECK_EQUAL(m_logs[0].topics[1], dev::keccak256(asString(structEnc))); @@ -13782,27 +13777,22 @@ BOOST_AUTO_TEST_CASE(abi_encodePackedV2_arrayOfStrings) x[0] = "abc"; x[1] = "0123456789012345678901234567890123456789"; } - function testStorage() public returns (bytes memory) { + function testStorage() public { emit E(x); - return abi.encodePacked(uint8(0x33), x, uint8(0x44)); } - function testMemory() public returns (bytes memory) { + function testMemory() public { string[] memory y = x; emit E(y); - return abi.encodePacked(uint8(0x33), y, uint8(0x44)); } } )"; compileAndRun(sourceCode, 0, "C"); bytes arrayEncoding = encodeArgs("abc", "0123456789012345678901234567890123456789"); - // This pads to multiple of 32 bytes - string encoding = "\x33" + asString(arrayEncoding) + "\x44"; - BOOST_CHECK_EQUAL(encoding.size(), 2 + 32 * 3); - ABI_CHECK(callContractFunction("testStorage()"), encodeArgs(0x20, encoding.size(), encoding)); + ABI_CHECK(callContractFunction("testStorage()"), encodeArgs()); BOOST_REQUIRE_EQUAL(m_logs[0].topics.size(), 2); BOOST_CHECK_EQUAL(m_logs[0].topics[0], dev::keccak256(string("E(string[])"))); BOOST_CHECK_EQUAL(m_logs[0].topics[1], dev::keccak256(asString(arrayEncoding))); - ABI_CHECK(callContractFunction("testMemory()"), encodeArgs(0x20, encoding.size(), encoding)); + ABI_CHECK(callContractFunction("testMemory()"), encodeArgs()); BOOST_REQUIRE_EQUAL(m_logs[0].topics.size(), 2); BOOST_CHECK_EQUAL(m_logs[0].topics[0], dev::keccak256(string("E(string[])"))); BOOST_CHECK_EQUAL(m_logs[0].topics[1], dev::keccak256(asString(arrayEncoding))); diff --git a/test/libsolidity/syntaxTests/specialFunctions/abi_encodePacked_structs_v2.sol b/test/libsolidity/syntaxTests/specialFunctions/abi_encodePacked_structs_v2.sol index 0ae504665615..a10791d24ddc 100644 --- a/test/libsolidity/syntaxTests/specialFunctions/abi_encodePacked_structs_v2.sol +++ b/test/libsolidity/syntaxTests/specialFunctions/abi_encodePacked_structs_v2.sol @@ -12,3 +12,5 @@ contract C { } // ---- // Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. +// TypeError: (191-192): Type not supported in packed mode. +// TypeError: (194-195): Type not supported in packed mode. diff --git a/test/libsolidity/syntaxTests/specialFunctions/abi_encode_structs.sol b/test/libsolidity/syntaxTests/specialFunctions/abi_encode_structs.sol index d9eebee428d3..45c9d9e01fd7 100644 --- a/test/libsolidity/syntaxTests/specialFunctions/abi_encode_structs.sol +++ b/test/libsolidity/syntaxTests/specialFunctions/abi_encode_structs.sol @@ -13,5 +13,5 @@ contract C { // ---- // TypeError: (131-132): This type cannot be encoded. // TypeError: (134-135): This type cannot be encoded. -// TypeError: (200-201): This type cannot be encoded. -// TypeError: (203-204): This type cannot be encoded. +// TypeError: (200-201): Type not supported in packed mode. +// TypeError: (203-204): Type not supported in packed mode. diff --git a/test/libsolidity/syntaxTests/specialFunctions/abi_encode_structs_abiv2.sol b/test/libsolidity/syntaxTests/specialFunctions/abi_encode_structs_abiv2.sol index 1b491b7fd85e..69a22f1a2874 100644 --- a/test/libsolidity/syntaxTests/specialFunctions/abi_encode_structs_abiv2.sol +++ b/test/libsolidity/syntaxTests/specialFunctions/abi_encode_structs_abiv2.sol @@ -5,6 +5,10 @@ contract C { S s; struct T { uint y; } T t; + function e() public view { + S memory st; + abi.encodePacked(st); + } function f() public view { abi.encode(s, t); } @@ -14,3 +18,6 @@ contract C { } // ---- // Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. +// TypeError: (193-195): Type not supported in packed mode. +// TypeError: (323-324): Type not supported in packed mode. +// TypeError: (326-327): Type not supported in packed mode. diff --git a/test/libsolidity/syntaxTests/specialFunctions/encodePacked_array_of_structs.sol b/test/libsolidity/syntaxTests/specialFunctions/encodePacked_array_of_structs.sol index 036e108aa5bc..6bbef79eff9f 100644 --- a/test/libsolidity/syntaxTests/specialFunctions/encodePacked_array_of_structs.sol +++ b/test/libsolidity/syntaxTests/specialFunctions/encodePacked_array_of_structs.sol @@ -6,4 +6,4 @@ contract C { } } // ---- -// TypeError: (116-117): This type cannot be encoded. +// TypeError: (116-117): Type not supported in packed mode. diff --git a/test/libsolidity/syntaxTests/specialFunctions/encodePacked_dynamic_string_array_v2.sol b/test/libsolidity/syntaxTests/specialFunctions/encodePacked_dynamic_string_array_v2.sol new file mode 100644 index 000000000000..aa7af3c4b7c3 --- /dev/null +++ b/test/libsolidity/syntaxTests/specialFunctions/encodePacked_dynamic_string_array_v2.sol @@ -0,0 +1,13 @@ +contract C { + string[] s; + function f() public pure { + string[] memory m; + abi.encodePacked(m); + } + function g() public pure { + abi.encodePacked(s); + } +} +// ---- +// TypeError: (112-113): Type not supported in packed mode. +// TypeError: (178-179): Type not supported in packed mode. diff --git a/test/libsolidity/syntaxTests/specialFunctions/types_with_unspecified_encoding_structs.sol b/test/libsolidity/syntaxTests/specialFunctions/types_with_unspecified_encoding_structs.sol index a1d3f5affd9b..9fcc30396db5 100644 --- a/test/libsolidity/syntaxTests/specialFunctions/types_with_unspecified_encoding_structs.sol +++ b/test/libsolidity/syntaxTests/specialFunctions/types_with_unspecified_encoding_structs.sol @@ -9,5 +9,5 @@ contract C { } } // ---- -// TypeError: (156-157): This type cannot be encoded. -// TypeError: (159-160): This type cannot be encoded. +// TypeError: (156-157): Type not supported in packed mode. +// TypeError: (159-160): Type not supported in packed mode. From 66b24225fb891e381117a23fe27a91ad33d80745 Mon Sep 17 00:00:00 2001 From: Mathias Baumann Date: Thu, 7 Feb 2019 15:53:49 +0100 Subject: [PATCH 096/109] Remove duplicate if-check --- libsolidity/analysis/TypeChecker.cpp | 4 ++-- test/libsolidity/SolidityTypes.cpp | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/libsolidity/analysis/TypeChecker.cpp b/libsolidity/analysis/TypeChecker.cpp index 3c1bfbae5b35..8df235851003 100644 --- a/libsolidity/analysis/TypeChecker.cpp +++ b/libsolidity/analysis/TypeChecker.cpp @@ -1234,10 +1234,10 @@ bool TypeChecker::visit(TupleExpression const& _tuple) if (!dynamic_cast(*types[i]).mobileType()) m_errorReporter.fatalTypeError(components[i]->location(), "Invalid rational number."); - if (_tuple.isInlineArray()) - solAssert(!!types[i], "Inline array cannot have empty components"); if (_tuple.isInlineArray()) { + solAssert(!!types[i], "Inline array cannot have empty components"); + if ((i == 0 || inlineArrayType) && !types[i]->mobileType()) m_errorReporter.fatalTypeError(components[i]->location(), "Invalid mobile type."); diff --git a/test/libsolidity/SolidityTypes.cpp b/test/libsolidity/SolidityTypes.cpp index 251835fe23f8..050fdaf2e25e 100644 --- a/test/libsolidity/SolidityTypes.cpp +++ b/test/libsolidity/SolidityTypes.cpp @@ -197,7 +197,7 @@ BOOST_AUTO_TEST_CASE(type_identifiers) TypePointer keccak256fun = make_shared(strings{}, strings{}, FunctionType::Kind::KECCAK256); BOOST_CHECK_EQUAL(keccak256fun->identifier(), "t_function_keccak256_nonpayable$__$returns$__$"); - FunctionType metaFun(TypePointers{keccak256fun}, TypePointers{s.type()}); + FunctionType metaFun(TypePointers{keccak256fun}, TypePointers{s.type()}, strings{""}, strings{""}); BOOST_CHECK_EQUAL(metaFun.identifier(), "t_function_internal_nonpayable$_t_function_keccak256_nonpayable$__$returns$__$_$returns$_t_type$_t_struct$_Struct_$3_storage_ptr_$_$"); TypePointer m = make_shared(Type::fromElementaryTypeName("bytes32"), s.type()); From cebeb4076c2a17953b57a72ec2aa56840a46f6dc Mon Sep 17 00:00:00 2001 From: Christian Parpart Date: Mon, 11 Feb 2019 15:27:08 +0100 Subject: [PATCH 097/109] Fixes compilation on Windows where STDERR_FILENO seems not to be present. --- solc/CommandLineInterface.cpp | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/solc/CommandLineInterface.cpp b/solc/CommandLineInterface.cpp index 8fb3abb47e1c..c0b5f974a3ee 100644 --- a/solc/CommandLineInterface.cpp +++ b/solc/CommandLineInterface.cpp @@ -60,10 +60,15 @@ #else // unix #include #endif + #include #include #include +#if !defined(STDERR_FILENO) + #define STDERR_FILENO 2 +#endif + using namespace std; using namespace langutil; namespace po = boost::program_options; From d099d55c6689726d73439f0dc1e7af22520f5c1d Mon Sep 17 00:00:00 2001 From: chriseth Date: Mon, 11 Feb 2019 15:29:58 +0100 Subject: [PATCH 098/109] Use master branch from solc-js. --- test/solcjsTests.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/solcjsTests.sh b/test/solcjsTests.sh index ab263178803e..93eac32e532a 100755 --- a/test/solcjsTests.sh +++ b/test/solcjsTests.sh @@ -39,8 +39,8 @@ VERSION="$2" DIR=$(mktemp -d) ( - echo "Preparing solc-js (0.5.0)..." - git clone --depth 1 --branch v0.5.0 https://github.com/ethereum/solc-js "$DIR" + echo "Preparing solc-js (master)..." + git clone --depth 1 --branch master https://github.com/ethereum/solc-js "$DIR" cd "$DIR" # disable "prepublish" script which downloads the latest version # (we will replace it anyway and it is often incorrectly cached From 0e4912a2038720fcce46d2ee369b4783041d7223 Mon Sep 17 00:00:00 2001 From: Daniel Kirchner Date: Tue, 5 Feb 2019 20:29:57 +0100 Subject: [PATCH 099/109] ABIEncoderV2: Implement calldata structs without dynamically encoded members. --- Changelog.md | 1 + libsolidity/analysis/TypeChecker.cpp | 10 +- libsolidity/ast/Types.cpp | 18 ++ libsolidity/ast/Types.h | 1 + libsolidity/codegen/ABIFunctions.cpp | 37 ++- libsolidity/codegen/ABIFunctions.h | 2 + libsolidity/codegen/CompilerUtils.cpp | 20 +- libsolidity/codegen/ExpressionCompiler.cpp | 18 ++ test/libsolidity/SolidityEndToEndTest.cpp | 242 ++++++++++++++++++ .../override/calldata_memory_struct.sol | 4 - .../syntaxTests/structs/array_calldata.sol | 2 - .../syntaxTests/structs/calldata.sol | 1 - .../structs/calldata_array_assign.sol | 10 + .../syntaxTests/structs/calldata_assign.sol | 8 + .../structs/calldata_struct_function_type.sol | 9 + .../structs/memory_to_calldata.sol | 12 + 16 files changed, 377 insertions(+), 18 deletions(-) create mode 100644 test/libsolidity/syntaxTests/structs/calldata_array_assign.sol create mode 100644 test/libsolidity/syntaxTests/structs/calldata_assign.sol create mode 100644 test/libsolidity/syntaxTests/structs/calldata_struct_function_type.sol create mode 100644 test/libsolidity/syntaxTests/structs/memory_to_calldata.sol diff --git a/Changelog.md b/Changelog.md index 0f8a2bbc9ffa..2bf09226dabc 100644 --- a/Changelog.md +++ b/Changelog.md @@ -9,6 +9,7 @@ Bugfixes: Language Features: + * Allow calldata structs without dynamically encoded members with ABIEncoderV2. Compiler Features: diff --git a/libsolidity/analysis/TypeChecker.cpp b/libsolidity/analysis/TypeChecker.cpp index e0ac3f9d5ef0..0f63d804866c 100644 --- a/libsolidity/analysis/TypeChecker.cpp +++ b/libsolidity/analysis/TypeChecker.cpp @@ -371,10 +371,12 @@ bool TypeChecker::visit(FunctionDefinition const& _function) if ( !m_scope->isInterface() && - baseType->category() == Type::Category::Struct && baseType->dataStoredIn(DataLocation::CallData) ) - m_errorReporter.typeError(var->location(), "Calldata structs are not yet supported."); + if (auto const* structType = dynamic_cast(baseType.get())) + if (structType->isDynamicallyEncoded()) + m_errorReporter.typeError(var->location(), "Dynamically encoded calldata structs are not yet supported."); + checkArgumentAndReturnParameter(*var); var->accept(*this); } @@ -2071,8 +2073,8 @@ bool TypeChecker::visit(MemberAccess const& _memberAccess) exprType->toString() + " (expected " + funType->selfType()->toString() + ")." ); - if (exprType->category() == Type::Category::Struct) - annotation.isLValue = true; + if (auto const* structType = dynamic_cast(exprType.get())) + annotation.isLValue = !structType->dataStoredIn(DataLocation::CallData); else if (exprType->category() == Type::Category::Array) { auto const& arrayType(dynamic_cast(*exprType)); diff --git a/libsolidity/ast/Types.cpp b/libsolidity/ast/Types.cpp index cca632446e64..aeb59dfca809 100644 --- a/libsolidity/ast/Types.cpp +++ b/libsolidity/ast/Types.cpp @@ -2053,6 +2053,24 @@ unsigned StructType::calldataEncodedSize(bool) const return size; } +unsigned StructType::calldataOffsetOfMember(std::string const& _member) const +{ + unsigned offset = 0; + for (auto const& member: members(nullptr)) + { + solAssert(member.type->canLiveOutsideStorage(), ""); + if (member.name == _member) + return offset; + { + // Struct members are always padded. + unsigned memberSize = member.type->calldataEncodedSize(true); + solAssert(memberSize != 0, ""); + offset += memberSize; + } + } + solAssert(false, "Struct member not found."); +} + bool StructType::isDynamicallyEncoded() const { solAssert(!recursive(), ""); diff --git a/libsolidity/ast/Types.h b/libsolidity/ast/Types.h index bd249c633eea..ea6792398664 100644 --- a/libsolidity/ast/Types.h +++ b/libsolidity/ast/Types.h @@ -852,6 +852,7 @@ class StructType: public ReferenceType std::pair const& storageOffsetsOfMember(std::string const& _name) const; u256 memoryOffsetOfMember(std::string const& _name) const; + unsigned calldataOffsetOfMember(std::string const& _name) const; StructDefinition const& structDefinition() const { return m_struct; } diff --git a/libsolidity/codegen/ABIFunctions.cpp b/libsolidity/codegen/ABIFunctions.cpp index 1d532f5dac3f..0c7a43c3771f 100644 --- a/libsolidity/codegen/ABIFunctions.cpp +++ b/libsolidity/codegen/ABIFunctions.cpp @@ -1246,7 +1246,16 @@ string ABIFunctions::abiDecodingFunction(Type const& _type, bool _fromMemory, bo return abiDecodingFunctionArray(*arrayType, _fromMemory); } else if (auto const* structType = dynamic_cast(decodingType.get())) - return abiDecodingFunctionStruct(*structType, _fromMemory); + { + if (structType->dataStoredIn(DataLocation::CallData)) + { + solAssert(!_fromMemory, ""); + solUnimplementedAssert(!structType->isDynamicallyEncoded(), "Dynamically encoded calldata structs are not yet implemented."); + return abiDecodingFunctionCalldataStruct(*structType); + } + else + return abiDecodingFunctionStruct(*structType, _fromMemory); + } else if (auto const* functionType = dynamic_cast(decodingType.get())) return abiDecodingFunctionFunctionType(*functionType, _fromMemory, _forUseOnStack); else @@ -1423,15 +1432,37 @@ string ABIFunctions::abiDecodingFunctionByteArray(ArrayType const& _type, bool _ }); } +string ABIFunctions::abiDecodingFunctionCalldataStruct(StructType const& _type) +{ + solAssert(_type.dataStoredIn(DataLocation::CallData), ""); + solAssert(_type.calldataEncodedSize(true) != 0, ""); + string functionName = + "abi_decode_" + + _type.identifier(); + + return createFunction(functionName, [&]() { + Whiskers w{R"( + // + function (offset, end) -> value { + if slt(sub(end, offset), ) { revert(0, 0) } + value := offset + } + )"}; + w("functionName", functionName); + w("readableTypeName", _type.toString(true)); + w("minimumSize", to_string(_type.calldataEncodedSize(true))); + return w.render(); + }); +} + string ABIFunctions::abiDecodingFunctionStruct(StructType const& _type, bool _fromMemory) { + solAssert(!_type.dataStoredIn(DataLocation::CallData), ""); string functionName = "abi_decode_" + _type.identifier() + (_fromMemory ? "_fromMemory" : ""); - solUnimplementedAssert(!_type.dataStoredIn(DataLocation::CallData), ""); - return createFunction(functionName, [&]() { Whiskers templ(R"( // diff --git a/libsolidity/codegen/ABIFunctions.h b/libsolidity/codegen/ABIFunctions.h index a8a3c64e52d9..8d8be9d713de 100644 --- a/libsolidity/codegen/ABIFunctions.h +++ b/libsolidity/codegen/ABIFunctions.h @@ -222,6 +222,8 @@ class ABIFunctions std::string abiDecodingFunctionCalldataArray(ArrayType const& _type); /// Part of @a abiDecodingFunction for byte array types. std::string abiDecodingFunctionByteArray(ArrayType const& _type, bool _fromMemory); + /// Part of @a abiDecodingFunction for calldata struct types. + std::string abiDecodingFunctionCalldataStruct(StructType const& _type); /// Part of @a abiDecodingFunction for struct types. std::string abiDecodingFunctionStruct(StructType const& _type, bool _fromMemory); /// Part of @a abiDecodingFunction for array types. diff --git a/libsolidity/codegen/CompilerUtils.cpp b/libsolidity/codegen/CompilerUtils.cpp index 676dd5b602ee..f8c8b3a889d0 100644 --- a/libsolidity/codegen/CompilerUtils.cpp +++ b/libsolidity/codegen/CompilerUtils.cpp @@ -918,8 +918,7 @@ void CompilerUtils::convertType( auto& targetType = dynamic_cast(_targetType); auto& typeOnStack = dynamic_cast(_typeOnStack); solAssert( - targetType.location() != DataLocation::CallData && - typeOnStack.location() != DataLocation::CallData + targetType.location() != DataLocation::CallData , ""); switch (targetType.location()) { @@ -933,9 +932,9 @@ void CompilerUtils::convertType( break; case DataLocation::Memory: // Copy the array to a free position in memory, unless it is already in memory. - if (typeOnStack.location() != DataLocation::Memory) + switch (typeOnStack.location()) { - solAssert(typeOnStack.location() == DataLocation::Storage, ""); + case DataLocation::Storage: // stack: m_context << typeOnStack.memorySize(); allocateMemory(); @@ -955,6 +954,19 @@ void CompilerUtils::convertType( storeInMemoryDynamic(*targetMemberType, true); } m_context << Instruction::POP << Instruction::POP; + break; + case DataLocation::CallData: + { + solUnimplementedAssert(!typeOnStack.isDynamicallyEncoded(), ""); + m_context << Instruction::DUP1; + m_context << Instruction::CALLDATASIZE; + m_context << Instruction::SUB; + abiDecode({targetType.shared_from_this()}, false); + break; + } + case DataLocation::Memory: + // nothing to do + break; } break; case DataLocation::CallData: diff --git a/libsolidity/codegen/ExpressionCompiler.cpp b/libsolidity/codegen/ExpressionCompiler.cpp index 942363461237..1e7a84e65305 100644 --- a/libsolidity/codegen/ExpressionCompiler.cpp +++ b/libsolidity/codegen/ExpressionCompiler.cpp @@ -1380,6 +1380,24 @@ bool ExpressionCompiler::visit(MemberAccess const& _memberAccess) setLValue(_memberAccess, *_memberAccess.annotation().type); break; } + case DataLocation::CallData: + { + solUnimplementedAssert(!type.isDynamicallyEncoded(), ""); + m_context << type.calldataOffsetOfMember(member) << Instruction::ADD; + // For non-value types the calldata offset is returned directly. + if (_memberAccess.annotation().type->isValueType()) + { + solAssert(_memberAccess.annotation().type->calldataEncodedSize(false) > 0, ""); + CompilerUtils(m_context).loadFromMemoryDynamic(*_memberAccess.annotation().type, true, true, false); + } + else + solAssert( + _memberAccess.annotation().type->category() == Type::Category::Array || + _memberAccess.annotation().type->category() == Type::Category::Struct, + "" + ); + break; + } default: solAssert(false, "Illegal data location for struct."); } diff --git a/test/libsolidity/SolidityEndToEndTest.cpp b/test/libsolidity/SolidityEndToEndTest.cpp index fb480459dc73..31d9be7a14e0 100644 --- a/test/libsolidity/SolidityEndToEndTest.cpp +++ b/test/libsolidity/SolidityEndToEndTest.cpp @@ -8013,6 +8013,248 @@ BOOST_AUTO_TEST_CASE(struct_named_constructor) ABI_CHECK(callContractFunction("s()"), encodeArgs(u256(1), true)); } +BOOST_AUTO_TEST_CASE(calldata_struct) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + struct S { uint256 a; uint256 b; } + function f(S calldata s) external pure returns (uint256 a, uint256 b) { + a = s.a; + b = s.b; + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + + ABI_CHECK(callContractFunction("f((uint256,uint256))", encodeArgs(u256(42), u256(23))), encodeArgs(u256(42), u256(23))); +} + +BOOST_AUTO_TEST_CASE(calldata_struct_and_ints) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + struct S { uint256 a; uint256 b; } + function f(uint256 a, S calldata s, uint256 b) external pure returns (uint256, uint256, uint256, uint256) { + return (a, s.a, s.b, b); + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + + ABI_CHECK(callContractFunction("f(uint256,(uint256,uint256),uint256)", encodeArgs(u256(1), u256(2), u256(3), u256(4))), encodeArgs(u256(1), u256(2), u256(3), u256(4))); +} + + +BOOST_AUTO_TEST_CASE(calldata_structs) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + struct S1 { uint256 a; uint256 b; } + struct S2 { uint256 a; } + function f(S1 calldata s1, S2 calldata s2, S1 calldata s3) + external pure returns (uint256 a, uint256 b, uint256 c, uint256 d, uint256 e) { + a = s1.a; + b = s1.b; + c = s2.a; + d = s3.a; + e = s3.b; + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + + ABI_CHECK(callContractFunction("f((uint256,uint256),(uint256),(uint256,uint256))", encodeArgs(u256(1), u256(2), u256(3), u256(4), u256(5))), encodeArgs(u256(1), u256(2), u256(3), u256(4), u256(5))); +} + +BOOST_AUTO_TEST_CASE(calldata_struct_array_member) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + struct S { uint256 a; uint256[2] b; uint256 c; } + function f(S calldata s) external pure returns (uint256 a, uint256 b0, uint256 b1, uint256 c) { + a = s.a; + b0 = s.b[0]; + b1 = s.b[1]; + c = s.c; + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + + ABI_CHECK(callContractFunction("f((uint256,uint256[2],uint256))", encodeArgs(u256(42), u256(1), u256(2), u256(23))), encodeArgs(u256(42), u256(1), u256(2), u256(23))); +} + +BOOST_AUTO_TEST_CASE(calldata_array_of_struct) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + struct S { uint256 a; uint256 b; } + function f(S[] calldata s) external pure returns (uint256 l, uint256 a, uint256 b, uint256 c, uint256 d) { + l = s.length; + a = s[0].a; + b = s[0].b; + c = s[1].a; + d = s[1].b; + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + + ABI_CHECK(callContractFunction("f((uint256,uint256)[])", encodeArgs(u256(0x20), u256(2), u256(1), u256(2), u256(3), u256(4))), encodeArgs(u256(2), u256(1), u256(2), u256(3), u256(4))); +} + +BOOST_AUTO_TEST_CASE(calldata_array_of_struct_to_memory) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + struct S { uint256 a; uint256 b; } + function f(S[] calldata s) external pure returns (uint256 l, uint256 a, uint256 b, uint256 c, uint256 d) { + S[] memory m = s; + l = m.length; + a = m[0].a; + b = m[0].b; + c = m[1].a; + d = m[1].b; + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + + ABI_CHECK(callContractFunction("f((uint256,uint256)[])", encodeArgs(u256(0x20), u256(2), u256(1), u256(2), u256(3), u256(4))), encodeArgs(u256(2), u256(1), u256(2), u256(3), u256(4))); +} + + +BOOST_AUTO_TEST_CASE(calldata_struct_to_memory) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + struct S { uint256 a; uint256 b; } + function f(S calldata s) external pure returns (uint256, uint256) { + S memory m = s; + return (m.a, m.b); + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + + ABI_CHECK(callContractFunction("f((uint256,uint256))", encodeArgs(u256(42), u256(23))), encodeArgs(u256(42), u256(23))); +} + +BOOST_AUTO_TEST_CASE(nested_calldata_struct) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + struct S1 { uint256 a; uint256 b; } + struct S2 { uint256 a; uint256 b; S1 s; uint256 c; } + function f(S2 calldata s) external pure returns (uint256 a, uint256 b, uint256 sa, uint256 sb, uint256 c) { + return (s.a, s.b, s.s.a, s.s.b, s.c); + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + + ABI_CHECK(callContractFunction("f((uint256,uint256,(uint256,uint256),uint256))", encodeArgs(u256(1), u256(2), u256(3), u256(4), u256(5))), encodeArgs(u256(1), u256(2), u256(3), u256(4), u256(5))); +} + +BOOST_AUTO_TEST_CASE(nested_calldata_struct_to_memory) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + struct S1 { uint256 a; uint256 b; } + struct S2 { uint256 a; uint256 b; S1 s; uint256 c; } + function f(S2 calldata s) external pure returns (uint256 a, uint256 b, uint256 sa, uint256 sb, uint256 c) { + S2 memory m = s; + return (m.a, m.b, m.s.a, m.s.b, m.c); + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + + ABI_CHECK(callContractFunction("f((uint256,uint256,(uint256,uint256),uint256))", encodeArgs(u256(1), u256(2), u256(3), u256(4), u256(5))), encodeArgs(u256(1), u256(2), u256(3), u256(4), u256(5))); +} + +BOOST_AUTO_TEST_CASE(calldata_struct_short) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + struct S { uint256 a; uint256 b; } + function f(S calldata) external pure returns (uint256) { + return msg.data.length; + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + + // double check that the valid case goes through + ABI_CHECK(callContractFunction("f((uint256,uint256))", u256(1), u256(2)), encodeArgs(0x44)); + + ABI_CHECK(callContractFunctionNoEncoding("f((uint256,uint256))", bytes(63,0)), encodeArgs()); + ABI_CHECK(callContractFunctionNoEncoding("f((uint256,uint256))", bytes(33,0)), encodeArgs()); + ABI_CHECK(callContractFunctionNoEncoding("f((uint256,uint256))", bytes(32,0)), encodeArgs()); + ABI_CHECK(callContractFunctionNoEncoding("f((uint256,uint256))", bytes(31,0)), encodeArgs()); + ABI_CHECK(callContractFunctionNoEncoding("f((uint256,uint256))", bytes()), encodeArgs()); +} + +BOOST_AUTO_TEST_CASE(calldata_struct_cleaning) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + struct S { uint8 a; bytes1 b; } + function f(S calldata s) external pure returns (uint256 a, bytes32 b) { + uint8 tmp1 = s.a; + bytes1 tmp2 = s.b; + assembly { + a := tmp1 + b := tmp2 + } + + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + + // double check that the valid case goes through + ABI_CHECK(callContractFunction("f((uint8,bytes1))", u256(0x12), bytes{0x34} + bytes(31,0)), encodeArgs(0x12, bytes{0x34} + bytes(31,0))); + ABI_CHECK(callContractFunction("f((uint8,bytes1))", u256(0x1234), bytes{0x56, 0x78} + bytes(30,0)), encodeArgs(0x34, bytes{0x56} + bytes(31,0))); + ABI_CHECK(callContractFunction("f((uint8,bytes1))", u256(-1), u256(-1)), encodeArgs(0xFF, bytes{0xFF} + bytes(31,0))); +} + +BOOST_AUTO_TEST_CASE(calldata_struct_function_type) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + struct S { function (uint) external returns (uint) fn; } + function f(S calldata s) external returns (uint256) { + return s.fn(42); + } + function g(uint256 a) external returns (uint256) { + return a * 3; + } + function h(uint256 a) external returns (uint256) { + return 23; + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + + bytes fn_C_g = m_contractAddress.asBytes() + FixedHash<4>(dev::keccak256("g(uint256)")).asBytes() + bytes(8,0); + bytes fn_C_h = m_contractAddress.asBytes() + FixedHash<4>(dev::keccak256("h(uint256)")).asBytes() + bytes(8,0); + ABI_CHECK(callContractFunctionNoEncoding("f((function))", fn_C_g), encodeArgs(42 * 3)); + ABI_CHECK(callContractFunctionNoEncoding("f((function))", fn_C_h), encodeArgs(23)); +} + BOOST_AUTO_TEST_CASE(literal_strings) { char const* sourceCode = R"( diff --git a/test/libsolidity/syntaxTests/inheritance/override/calldata_memory_struct.sol b/test/libsolidity/syntaxTests/inheritance/override/calldata_memory_struct.sol index b81e3859badc..42aebf304c4f 100644 --- a/test/libsolidity/syntaxTests/inheritance/override/calldata_memory_struct.sol +++ b/test/libsolidity/syntaxTests/inheritance/override/calldata_memory_struct.sol @@ -15,7 +15,3 @@ contract B is A { } // ---- // Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. -// TypeError: (102-112): Calldata structs are not yet supported. -// TypeError: (146-156): Calldata structs are not yet supported. -// TypeError: (198-208): Calldata structs are not yet supported. -// TypeError: (250-260): Calldata structs are not yet supported. diff --git a/test/libsolidity/syntaxTests/structs/array_calldata.sol b/test/libsolidity/syntaxTests/structs/array_calldata.sol index 3aac5606b3b0..9e1071a0e5ab 100644 --- a/test/libsolidity/syntaxTests/structs/array_calldata.sol +++ b/test/libsolidity/syntaxTests/structs/array_calldata.sol @@ -6,5 +6,3 @@ contract Test { } // ---- // Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. -// TypeError: (89-101): Calldata structs are not yet supported. -// TypeError: (131-145): Calldata structs are not yet supported. diff --git a/test/libsolidity/syntaxTests/structs/calldata.sol b/test/libsolidity/syntaxTests/structs/calldata.sol index dadf6e4fc9f8..8d584e5772b0 100644 --- a/test/libsolidity/syntaxTests/structs/calldata.sol +++ b/test/libsolidity/syntaxTests/structs/calldata.sol @@ -5,4 +5,3 @@ contract Test { } // ---- // Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. -// TypeError: (89-99): Calldata structs are not yet supported. diff --git a/test/libsolidity/syntaxTests/structs/calldata_array_assign.sol b/test/libsolidity/syntaxTests/structs/calldata_array_assign.sol new file mode 100644 index 000000000000..70e537ad64b3 --- /dev/null +++ b/test/libsolidity/syntaxTests/structs/calldata_array_assign.sol @@ -0,0 +1,10 @@ +pragma experimental ABIEncoderV2; +contract Test { + struct S { int[3] a; } + function f(S calldata s, int[3] calldata a) external { + s.a = a; + } +} +// ---- +// Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. +// TypeError: (144-147): Expression has to be an lvalue. diff --git a/test/libsolidity/syntaxTests/structs/calldata_assign.sol b/test/libsolidity/syntaxTests/structs/calldata_assign.sol new file mode 100644 index 000000000000..f44730b4f329 --- /dev/null +++ b/test/libsolidity/syntaxTests/structs/calldata_assign.sol @@ -0,0 +1,8 @@ +pragma experimental ABIEncoderV2; +contract Test { + struct S { int a; } + function f(S calldata s) external { s.a = 4; } +} +// ---- +// Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. +// TypeError: (114-117): Expression has to be an lvalue. diff --git a/test/libsolidity/syntaxTests/structs/calldata_struct_function_type.sol b/test/libsolidity/syntaxTests/structs/calldata_struct_function_type.sol new file mode 100644 index 000000000000..6a3c2d8447dd --- /dev/null +++ b/test/libsolidity/syntaxTests/structs/calldata_struct_function_type.sol @@ -0,0 +1,9 @@ +pragma experimental ABIEncoderV2; +contract C { + struct S { function (uint) external returns (uint) fn; } + function f(S calldata s) external returns (uint256 a) { + return s.fn(42); + } +} +// ---- +// Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. diff --git a/test/libsolidity/syntaxTests/structs/memory_to_calldata.sol b/test/libsolidity/syntaxTests/structs/memory_to_calldata.sol new file mode 100644 index 000000000000..09b3ef294ba0 --- /dev/null +++ b/test/libsolidity/syntaxTests/structs/memory_to_calldata.sol @@ -0,0 +1,12 @@ +pragma experimental ABIEncoderV2; +contract Test { + struct S { int a; } + function f(S calldata s) external { s = S(2); } + function g(S calldata s) external { S memory m; s = m; } +} +// ---- +// Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. +// TypeError: (114-115): Expression has to be an lvalue. +// TypeError: (118-122): Type struct Test.S memory is not implicitly convertible to expected type struct Test.S calldata. +// TypeError: (178-179): Expression has to be an lvalue. +// TypeError: (182-183): Type struct Test.S memory is not implicitly convertible to expected type struct Test.S calldata. From 08977af8437b39658d8c7ab1f00b3a1c583f61ee Mon Sep 17 00:00:00 2001 From: Mathias Baumann Date: Mon, 11 Feb 2019 15:13:16 +0100 Subject: [PATCH 100/109] Rename & move test to move away from number prefix --- .../constructor_call.sol} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename test/libsolidity/syntaxTests/nameAndTypeResolution/{586_invalid_types_for_constructor_call.sol => invalidTypes/constructor_call.sol} (100%) diff --git a/test/libsolidity/syntaxTests/nameAndTypeResolution/586_invalid_types_for_constructor_call.sol b/test/libsolidity/syntaxTests/nameAndTypeResolution/invalidTypes/constructor_call.sol similarity index 100% rename from test/libsolidity/syntaxTests/nameAndTypeResolution/586_invalid_types_for_constructor_call.sol rename to test/libsolidity/syntaxTests/nameAndTypeResolution/invalidTypes/constructor_call.sol From 259d803387a9bbe301c7fc13565a1e18ef0961bb Mon Sep 17 00:00:00 2001 From: Mathias Baumann Date: Wed, 6 Feb 2019 20:42:38 +0100 Subject: [PATCH 101/109] Conditional Expression: Delay invalid type fatal error Check the whole conditional first and then output errors for both, the true and false expressions. --- libsolidity/analysis/TypeChecker.cpp | 44 ++++++++++++------- .../invalidTypes/conditional_expression.sol | 16 +++++++ 2 files changed, 45 insertions(+), 15 deletions(-) create mode 100644 test/libsolidity/syntaxTests/nameAndTypeResolution/invalidTypes/conditional_expression.sol diff --git a/libsolidity/analysis/TypeChecker.cpp b/libsolidity/analysis/TypeChecker.cpp index e0ac3f9d5ef0..07c4346dbdf8 100644 --- a/libsolidity/analysis/TypeChecker.cpp +++ b/libsolidity/analysis/TypeChecker.cpp @@ -1058,25 +1058,39 @@ bool TypeChecker::visit(Conditional const& _conditional) TypePointer trueType = type(_conditional.trueExpression())->mobileType(); TypePointer falseType = type(_conditional.falseExpression())->mobileType(); + + TypePointer commonType; + if (!trueType) - m_errorReporter.fatalTypeError(_conditional.trueExpression().location(), "Invalid mobile type."); + m_errorReporter.typeError(_conditional.trueExpression().location(), "Invalid mobile type in true expression."); + else + commonType = trueType; + if (!falseType) - m_errorReporter.fatalTypeError(_conditional.falseExpression().location(), "Invalid mobile type."); + m_errorReporter.typeError(_conditional.falseExpression().location(), "Invalid mobile type in false expression."); + else + commonType = falseType; - TypePointer commonType = Type::commonType(trueType, falseType); - if (!commonType) + if (!trueType && !falseType) + BOOST_THROW_EXCEPTION(FatalError()); + else if (trueType && falseType) { - m_errorReporter.typeError( - _conditional.location(), - "True expression's type " + - trueType->toString() + - " doesn't match false expression's type " + - falseType->toString() + - "." - ); - // even we can't find a common type, we have to set a type here, - // otherwise the upper statement will not be able to check the type. - commonType = trueType; + commonType = Type::commonType(trueType, falseType); + + if (!commonType) + { + m_errorReporter.typeError( + _conditional.location(), + "True expression's type " + + trueType->toString() + + " doesn't match false expression's type " + + falseType->toString() + + "." + ); + // even we can't find a common type, we have to set a type here, + // otherwise the upper statement will not be able to check the type. + commonType = trueType; + } } _conditional.annotation().type = commonType; diff --git a/test/libsolidity/syntaxTests/nameAndTypeResolution/invalidTypes/conditional_expression.sol b/test/libsolidity/syntaxTests/nameAndTypeResolution/invalidTypes/conditional_expression.sol new file mode 100644 index 000000000000..4cb3f62924c2 --- /dev/null +++ b/test/libsolidity/syntaxTests/nameAndTypeResolution/invalidTypes/conditional_expression.sol @@ -0,0 +1,16 @@ +contract C { + function o(byte) public pure {} + function f() public { + o(true ? 99**99 : 99); + o(true ? 99 : 99**99); + + o(true ? 99**99 : 99**99); + } +} +// ---- +// TypeError: (92-98): Invalid mobile type in true expression. +// TypeError: (85-103): Invalid type for argument in function call. Invalid implicit conversion from uint8 to bytes1 requested. +// TypeError: (128-134): Invalid mobile type in false expression. +// TypeError: (116-134): Invalid type for argument in function call. Invalid implicit conversion from uint8 to bytes1 requested. +// TypeError: (155-161): Invalid mobile type in true expression. +// TypeError: (164-170): Invalid mobile type in false expression. From b34e104173d9b56c41368f079f24187a46ff2aeb Mon Sep 17 00:00:00 2001 From: chriseth Date: Thu, 31 Jan 2019 14:44:57 +0100 Subject: [PATCH 102/109] Make dialect a shared pointer. --- libsolidity/interface/AssemblyStack.cpp | 2 +- libyul/optimiser/Suite.cpp | 82 ++++++++++++------------- libyul/optimiser/Suite.h | 2 +- test/libyul/YulOptimizerTest.cpp | 2 +- 4 files changed, 44 insertions(+), 44 deletions(-) diff --git a/libsolidity/interface/AssemblyStack.cpp b/libsolidity/interface/AssemblyStack.cpp index 6d30f5f73c09..a09de3c3880e 100644 --- a/libsolidity/interface/AssemblyStack.cpp +++ b/libsolidity/interface/AssemblyStack.cpp @@ -134,7 +134,7 @@ void AssemblyStack::optimize(yul::Object& _object) for (auto& subNode: _object.subObjects) if (auto subObject = dynamic_cast(subNode.get())) optimize(*subObject); - yul::OptimiserSuite::run(*languageToDialect(m_language), *_object.code, *_object.analysisInfo); + yul::OptimiserSuite::run(languageToDialect(m_language), *_object.code, *_object.analysisInfo); } MachineAssemblyObject AssemblyStack::assemble(Machine _machine, bool _optimize) const diff --git a/libyul/optimiser/Suite.cpp b/libyul/optimiser/Suite.cpp index aa1d4a139511..3fb5a212c204 100644 --- a/libyul/optimiser/Suite.cpp +++ b/libyul/optimiser/Suite.cpp @@ -50,7 +50,7 @@ using namespace dev; using namespace yul; void OptimiserSuite::run( - Dialect const& _dialect, + shared_ptr const& _dialect, Block& _ast, AsmAnalysisInfo const& _analysisInfo, set const& _externallyUsedIdentifiers @@ -58,52 +58,52 @@ void OptimiserSuite::run( { set reservedIdentifiers = _externallyUsedIdentifiers; - Block ast = boost::get(Disambiguator(_dialect, _analysisInfo, reservedIdentifiers)(_ast)); + Block ast = boost::get(Disambiguator(*_dialect, _analysisInfo, reservedIdentifiers)(_ast)); (VarDeclInitializer{})(ast); (FunctionHoister{})(ast); (BlockFlattener{})(ast); (FunctionGrouper{})(ast); EquivalentFunctionCombiner::run(ast); - UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); (ForLoopInitRewriter{})(ast); (BlockFlattener{})(ast); - StructuralSimplifier{_dialect}(ast); + StructuralSimplifier{*_dialect}(ast); - NameDispenser dispenser{_dialect, ast}; + NameDispenser dispenser{*_dialect, ast}; for (size_t i = 0; i < 4; i++) { - ExpressionSplitter{_dialect, dispenser}(ast); + ExpressionSplitter{*_dialect, dispenser}(ast); SSATransform::run(ast, dispenser); - RedundantAssignEliminator::run(_dialect, ast); - RedundantAssignEliminator::run(_dialect, ast); + RedundantAssignEliminator::run(*_dialect, ast); + RedundantAssignEliminator::run(*_dialect, ast); - ExpressionSimplifier::run(_dialect, ast); - CommonSubexpressionEliminator{_dialect}(ast); - StructuralSimplifier{_dialect}(ast); + ExpressionSimplifier::run(*_dialect, ast); + CommonSubexpressionEliminator{*_dialect}(ast); + StructuralSimplifier{*_dialect}(ast); (BlockFlattener{})(ast); SSATransform::run(ast, dispenser); - RedundantAssignEliminator::run(_dialect, ast); - RedundantAssignEliminator::run(_dialect, ast); - UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); - CommonSubexpressionEliminator{_dialect}(ast); - UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); + RedundantAssignEliminator::run(*_dialect, ast); + RedundantAssignEliminator::run(*_dialect, ast); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); + CommonSubexpressionEliminator{*_dialect}(ast); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); SSAReverser::run(ast); - CommonSubexpressionEliminator{_dialect}(ast); - UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); + CommonSubexpressionEliminator{*_dialect}(ast); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); ExpressionJoiner::run(ast); ExpressionJoiner::run(ast); - ExpressionInliner(_dialect, ast).run(); - UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); + ExpressionInliner(*_dialect, ast).run(); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); - ExpressionSplitter{_dialect, dispenser}(ast); + ExpressionSplitter{*_dialect, dispenser}(ast); SSATransform::run(ast, dispenser); - RedundantAssignEliminator::run(_dialect, ast); - RedundantAssignEliminator::run(_dialect, ast); - CommonSubexpressionEliminator{_dialect}(ast); + RedundantAssignEliminator::run(*_dialect, ast); + RedundantAssignEliminator::run(*_dialect, ast); + CommonSubexpressionEliminator{*_dialect}(ast); (FunctionGrouper{})(ast); EquivalentFunctionCombiner::run(ast); @@ -111,33 +111,33 @@ void OptimiserSuite::run( (BlockFlattener{})(ast); SSATransform::run(ast, dispenser); - RedundantAssignEliminator::run(_dialect, ast); - RedundantAssignEliminator::run(_dialect, ast); - ExpressionSimplifier::run(_dialect, ast); - StructuralSimplifier{_dialect}(ast); + RedundantAssignEliminator::run(*_dialect, ast); + RedundantAssignEliminator::run(*_dialect, ast); + ExpressionSimplifier::run(*_dialect, ast); + StructuralSimplifier{*_dialect}(ast); (BlockFlattener{})(ast); - CommonSubexpressionEliminator{_dialect}(ast); + CommonSubexpressionEliminator{*_dialect}(ast); SSATransform::run(ast, dispenser); - RedundantAssignEliminator::run(_dialect, ast); - RedundantAssignEliminator::run(_dialect, ast); - UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); - CommonSubexpressionEliminator{_dialect}(ast); + RedundantAssignEliminator::run(*_dialect, ast); + RedundantAssignEliminator::run(*_dialect, ast); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); + CommonSubexpressionEliminator{*_dialect}(ast); } ExpressionJoiner::run(ast); - Rematerialiser::run(_dialect, ast); - UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); + Rematerialiser::run(*_dialect, ast); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); ExpressionJoiner::run(ast); - UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); ExpressionJoiner::run(ast); - UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); SSAReverser::run(ast); - CommonSubexpressionEliminator{_dialect}(ast); - UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); + CommonSubexpressionEliminator{*_dialect}(ast); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); ExpressionJoiner::run(ast); - Rematerialiser::run(_dialect, ast); - UnusedPruner::runUntilStabilised(_dialect, ast, reservedIdentifiers); + Rematerialiser::run(*_dialect, ast); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); _ast = std::move(ast); } diff --git a/libyul/optimiser/Suite.h b/libyul/optimiser/Suite.h index 376e9889bd51..8aeef76084cc 100644 --- a/libyul/optimiser/Suite.h +++ b/libyul/optimiser/Suite.h @@ -38,7 +38,7 @@ class OptimiserSuite { public: static void run( - Dialect const& _dialect, + std::shared_ptr const& _dialect, Block& _ast, AsmAnalysisInfo const& _analysisInfo, std::set const& _externallyUsedIdentifiers = {} diff --git a/test/libyul/YulOptimizerTest.cpp b/test/libyul/YulOptimizerTest.cpp index 306721a0259d..7b0cb100460d 100644 --- a/test/libyul/YulOptimizerTest.cpp +++ b/test/libyul/YulOptimizerTest.cpp @@ -242,7 +242,7 @@ bool YulOptimizerTest::run(ostream& _stream, string const& _linePrefix, bool con UnusedPruner::runUntilStabilised(*m_dialect, *m_ast); } else if (m_optimizerStep == "fullSuite") - OptimiserSuite::run(*m_dialect, *m_ast, *m_analysisInfo); + OptimiserSuite::run(m_dialect, *m_ast, *m_analysisInfo); else { FormattedScope(_stream, _formatted, {formatting::BOLD, formatting::RED}) << _linePrefix << "Invalid optimizer step: " << m_optimizerStep << endl; From 5a34743d88f6afbd39a18e4dcf7375a0b7d233c8 Mon Sep 17 00:00:00 2001 From: chriseth Date: Mon, 4 Feb 2019 14:01:05 +0100 Subject: [PATCH 103/109] Allow optimizer steps to run on FunctionDefinition and group suite. --- libyul/optimiser/NameCollector.cpp | 7 + libyul/optimiser/NameCollector.h | 1 + libyul/optimiser/Rematerialiser.cpp | 11 ++ libyul/optimiser/Rematerialiser.h | 2 + libyul/optimiser/Suite.cpp | 127 +++++++++++------- libyul/optimiser/UnusedPruner.cpp | 26 +++- libyul/optimiser/UnusedPruner.h | 13 ++ .../yulOptimizerTests/fullSuite/abi2.yul | 38 +++--- .../fullSuite/abi_example1.yul | 52 +++---- .../yulOptimizerTests/fullSuite/aztec.yul | 50 +++---- .../yulOptimizerTests/fullSuite/medium.yul | 6 +- 11 files changed, 209 insertions(+), 124 deletions(-) diff --git a/libyul/optimiser/NameCollector.cpp b/libyul/optimiser/NameCollector.cpp index f90798271600..5195b8d8fdc2 100644 --- a/libyul/optimiser/NameCollector.cpp +++ b/libyul/optimiser/NameCollector.cpp @@ -60,6 +60,13 @@ map ReferencesCounter::countReferences(Block const& _block) return counter.references(); } +map ReferencesCounter::countReferences(FunctionDefinition const& _function) +{ + ReferencesCounter counter; + counter(_function); + return counter.references(); +} + map ReferencesCounter::countReferences(Expression const& _expression) { ReferencesCounter counter; diff --git a/libyul/optimiser/NameCollector.h b/libyul/optimiser/NameCollector.h index c177a39962e6..7e21c03f0f3f 100644 --- a/libyul/optimiser/NameCollector.h +++ b/libyul/optimiser/NameCollector.h @@ -59,6 +59,7 @@ class ReferencesCounter: public ASTWalker virtual void operator()(FunctionCall const& _funCall); static std::map countReferences(Block const& _block); + static std::map countReferences(FunctionDefinition const& _function); static std::map countReferences(Expression const& _expression); std::map const& references() const { return m_references; } diff --git a/libyul/optimiser/Rematerialiser.cpp b/libyul/optimiser/Rematerialiser.cpp index 56f6e99cccee..34a23c1338ae 100644 --- a/libyul/optimiser/Rematerialiser.cpp +++ b/libyul/optimiser/Rematerialiser.cpp @@ -35,12 +35,23 @@ void Rematerialiser::run(Dialect const& _dialect, Block& _ast) Rematerialiser{_dialect, _ast}(_ast); } +void Rematerialiser::run(Dialect const& _dialect, FunctionDefinition& _function) +{ + Rematerialiser{_dialect, _function}(_function); +} + Rematerialiser::Rematerialiser(Dialect const& _dialect, Block& _ast): DataFlowAnalyzer(_dialect), m_referenceCounts(ReferencesCounter::countReferences(_ast)) { } +Rematerialiser::Rematerialiser(Dialect const& _dialect, FunctionDefinition& _function): + DataFlowAnalyzer(_dialect), + m_referenceCounts(ReferencesCounter::countReferences(_function)) +{ +} + void Rematerialiser::visit(Expression& _e) { if (_e.type() == typeid(Identifier)) diff --git a/libyul/optimiser/Rematerialiser.h b/libyul/optimiser/Rematerialiser.h index 731697c883a3..04b09f05cf36 100644 --- a/libyul/optimiser/Rematerialiser.h +++ b/libyul/optimiser/Rematerialiser.h @@ -39,9 +39,11 @@ class Rematerialiser: public DataFlowAnalyzer { public: static void run(Dialect const& _dialect, Block& _ast); + static void run(Dialect const& _dialect, FunctionDefinition& _function); protected: Rematerialiser(Dialect const& _dialect, Block& _ast); + Rematerialiser(Dialect const& _dialect, FunctionDefinition& _function); using ASTModifier::visit; void visit(Expression& _e) override; diff --git a/libyul/optimiser/Suite.cpp b/libyul/optimiser/Suite.cpp index 3fb5a212c204..8a104c39699e 100644 --- a/libyul/optimiser/Suite.cpp +++ b/libyul/optimiser/Suite.cpp @@ -69,60 +69,91 @@ void OptimiserSuite::run( (ForLoopInitRewriter{})(ast); (BlockFlattener{})(ast); StructuralSimplifier{*_dialect}(ast); + (BlockFlattener{})(ast); + + // None of the above can make stack problems worse. NameDispenser dispenser{*_dialect, ast}; for (size_t i = 0; i < 4; i++) { - ExpressionSplitter{*_dialect, dispenser}(ast); - SSATransform::run(ast, dispenser); - RedundantAssignEliminator::run(*_dialect, ast); - RedundantAssignEliminator::run(*_dialect, ast); - - ExpressionSimplifier::run(*_dialect, ast); - CommonSubexpressionEliminator{*_dialect}(ast); - StructuralSimplifier{*_dialect}(ast); - (BlockFlattener{})(ast); - SSATransform::run(ast, dispenser); - RedundantAssignEliminator::run(*_dialect, ast); - RedundantAssignEliminator::run(*_dialect, ast); - UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); - CommonSubexpressionEliminator{*_dialect}(ast); - UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); - - SSAReverser::run(ast); - CommonSubexpressionEliminator{*_dialect}(ast); - UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); - - ExpressionJoiner::run(ast); - ExpressionJoiner::run(ast); - ExpressionInliner(*_dialect, ast).run(); - UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); - - ExpressionSplitter{*_dialect, dispenser}(ast); - SSATransform::run(ast, dispenser); - RedundantAssignEliminator::run(*_dialect, ast); - RedundantAssignEliminator::run(*_dialect, ast); - CommonSubexpressionEliminator{*_dialect}(ast); - - (FunctionGrouper{})(ast); - EquivalentFunctionCombiner::run(ast); - FullInliner{ast, dispenser}.run(); - (BlockFlattener{})(ast); - - SSATransform::run(ast, dispenser); - RedundantAssignEliminator::run(*_dialect, ast); - RedundantAssignEliminator::run(*_dialect, ast); - ExpressionSimplifier::run(*_dialect, ast); - StructuralSimplifier{*_dialect}(ast); - (BlockFlattener{})(ast); - CommonSubexpressionEliminator{*_dialect}(ast); - SSATransform::run(ast, dispenser); - RedundantAssignEliminator::run(*_dialect, ast); - RedundantAssignEliminator::run(*_dialect, ast); - UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); - CommonSubexpressionEliminator{*_dialect}(ast); + { + // Turn into SSA and simplify + ExpressionSplitter{*_dialect, dispenser}(ast); + SSATransform::run(ast, dispenser); + RedundantAssignEliminator::run(*_dialect, ast); + RedundantAssignEliminator::run(*_dialect, ast); + + ExpressionSimplifier::run(*_dialect, ast); + CommonSubexpressionEliminator{*_dialect}(ast); + } + + { + // still in SSA, perform structural simplification + StructuralSimplifier{*_dialect}(ast); + (BlockFlattener{})(ast); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); + } + { + // simplify again + CommonSubexpressionEliminator{*_dialect}(ast); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); + } + + { + // reverse SSA + SSAReverser::run(ast); + CommonSubexpressionEliminator{*_dialect}(ast); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); + + ExpressionJoiner::run(ast); + ExpressionJoiner::run(ast); + } + + // should have good "compilability" property here. + + { + // run functional expression inliner + ExpressionInliner(*_dialect, ast).run(); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); + } + + { + // Turn into SSA again and simplify + ExpressionSplitter{*_dialect, dispenser}(ast); + SSATransform::run(ast, dispenser); + RedundantAssignEliminator::run(*_dialect, ast); + RedundantAssignEliminator::run(*_dialect, ast); + CommonSubexpressionEliminator{*_dialect}(ast); + } + + { + // run full inliner + (FunctionGrouper{})(ast); + EquivalentFunctionCombiner::run(ast); + FullInliner{ast, dispenser}.run(); + (BlockFlattener{})(ast); + } + + { + // SSA plus simplify + SSATransform::run(ast, dispenser); + RedundantAssignEliminator::run(*_dialect, ast); + RedundantAssignEliminator::run(*_dialect, ast); + ExpressionSimplifier::run(*_dialect, ast); + StructuralSimplifier{*_dialect}(ast); + (BlockFlattener{})(ast); + CommonSubexpressionEliminator{*_dialect}(ast); + SSATransform::run(ast, dispenser); + RedundantAssignEliminator::run(*_dialect, ast); + RedundantAssignEliminator::run(*_dialect, ast); + UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); + CommonSubexpressionEliminator{*_dialect}(ast); + } } + + // Make source short and pretty. + ExpressionJoiner::run(ast); Rematerialiser::run(*_dialect, ast); UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); diff --git a/libyul/optimiser/UnusedPruner.cpp b/libyul/optimiser/UnusedPruner.cpp index 365b255c5fe4..65db8025beef 100644 --- a/libyul/optimiser/UnusedPruner.cpp +++ b/libyul/optimiser/UnusedPruner.cpp @@ -35,10 +35,15 @@ using namespace yul; UnusedPruner::UnusedPruner(Dialect const& _dialect, Block& _ast, set const& _externallyUsedFunctions): m_dialect(_dialect) { - ReferencesCounter counter; - counter(_ast); + m_references = ReferencesCounter::countReferences(_ast); + for (auto const& f: _externallyUsedFunctions) + ++m_references[f]; +} - m_references = counter.references(); +UnusedPruner::UnusedPruner(Dialect const& _dialect, FunctionDefinition& _function, set const& _externallyUsedFunctions): + m_dialect(_dialect) +{ + m_references = ReferencesCounter::countReferences(_function); for (auto const& f: _externallyUsedFunctions) ++m_references[f]; } @@ -116,6 +121,21 @@ void UnusedPruner::runUntilStabilised( } } +void UnusedPruner::runUntilStabilised( + Dialect const& _dialect, + FunctionDefinition& _function, + set const& _externallyUsedFunctions +) +{ + while (true) + { + UnusedPruner pruner(_dialect, _function, _externallyUsedFunctions); + pruner(_function); + if (!pruner.shouldRunAgain()) + return; + } +} + bool UnusedPruner::used(YulString _name) const { return m_references.count(_name) && m_references.at(_name) > 0; diff --git a/libyul/optimiser/UnusedPruner.h b/libyul/optimiser/UnusedPruner.h index 72279d4aa0e8..4f81b950d23c 100644 --- a/libyul/optimiser/UnusedPruner.h +++ b/libyul/optimiser/UnusedPruner.h @@ -46,6 +46,11 @@ class UnusedPruner: public ASTModifier Block& _ast, std::set const& _externallyUsedFunctions = {} ); + UnusedPruner( + Dialect const& _dialect, + FunctionDefinition& _function, + std::set const& _externallyUsedFunctions = {} + ); using ASTModifier::operator(); void operator()(Block& _block) override; @@ -60,6 +65,14 @@ class UnusedPruner: public ASTModifier std::set const& _externallyUsedFunctions = {} ); + // Run the pruner until the code does not change anymore. + // Only run on the given function. + static void runUntilStabilised( + Dialect const& _dialect, + FunctionDefinition& _functionDefinition, + std::set const& _externallyUsedFunctions = {} + ); + private: bool used(YulString _name) const; void subtractReferences(std::map const& _subtrahend); diff --git a/test/libyul/yulOptimizerTests/fullSuite/abi2.yul b/test/libyul/yulOptimizerTests/fullSuite/abi2.yul index 14dafdb19598..2dd3e4c11c85 100644 --- a/test/libyul/yulOptimizerTests/fullSuite/abi2.yul +++ b/test/libyul/yulOptimizerTests/fullSuite/abi2.yul @@ -1073,12 +1073,12 @@ // fullSuite // { // let _2 := mload(1) -// let _153 := mload(0) -// if slt(sub(_2, _153), 64) +// let _134 := mload(0) +// if slt(sub(_2, _134), 64) // { // revert(0, 0) // } -// sstore(0, and(calldataload(_153), 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF)) +// sstore(0, and(calldataload(_134), 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF)) // let x0, x1, x2, x3, x4 := abi_decode_tuple_t_addresst_uint256t_bytes_calldata_ptrt_enum$_Operation_$1949(mload(7), mload(8)) // sstore(x1, x0) // sstore(x3, x2) @@ -1093,40 +1093,40 @@ // value0_57 := and(calldataload(headStart_55), 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF) // value1_58 := calldataload(add(headStart_55, 32)) // let offset_62 := calldataload(add(headStart_55, 64)) -// let _200 := 0xffffffffffffffff -// if gt(offset_62, _200) +// let _181 := 0xffffffffffffffff +// if gt(offset_62, _181) // { // revert(value4, value4) // } -// let _202 := add(headStart_55, offset_62) -// if iszero(slt(add(_202, 0x1f), dataEnd_56)) +// let _183 := add(headStart_55, offset_62) +// if iszero(slt(add(_183, 0x1f), dataEnd_56)) // { // revert(value4, value4) // } -// let abi_decode_length_15_244 := calldataload(_202) -// if gt(abi_decode_length_15_244, _200) +// let abi_decode_length_15_225 := calldataload(_183) +// if gt(abi_decode_length_15_225, _181) // { // revert(value4, value4) // } -// if gt(add(add(_202, abi_decode_length_15_244), 32), dataEnd_56) +// if gt(add(add(_183, abi_decode_length_15_225), 32), dataEnd_56) // { // revert(value4, value4) // } -// value2_59 := add(_202, 32) -// value3 := abi_decode_length_15_244 -// let _205 := calldataload(add(headStart_55, 96)) -// if iszero(lt(_205, 3)) +// value2_59 := add(_183, 32) +// value3 := abi_decode_length_15_225 +// let _186 := calldataload(add(headStart_55, 96)) +// if iszero(lt(_186, 3)) // { // revert(value4, value4) // } -// value4 := _205 +// value4 := _186 // } // function abi_encode_tuple_t_bytes32_t_address_t_uint256_t_bytes32_t_enum$_Operation_$1949_t_uint256_t_uint256_t_uint256_t_address_t_address_t_uint256__to_t_bytes32_t_address_t_uint256_t_bytes32_t_uint8_t_uint256_t_uint256_t_uint256_t_address_t_address_t_uint256_(headStart_252, value10_253, value9_254, value8_255, value7_256, value6_257, value5_258, value4_259, value3_260, value2_261, value1_262, value0_263) -> tail_264 // { // tail_264 := add(headStart_252, 352) // mstore(headStart_252, value0_263) -// let _409 := 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -// mstore(add(headStart_252, 32), and(value1_262, _409)) +// let _382 := 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF +// mstore(add(headStart_252, 32), and(value1_262, _382)) // mstore(add(headStart_252, 64), value2_261) // mstore(add(headStart_252, 96), value3_260) // if iszero(lt(value4_259, 3)) @@ -1137,8 +1137,8 @@ // mstore(add(headStart_252, 160), value5_258) // mstore(add(headStart_252, 192), value6_257) // mstore(add(headStart_252, 224), value7_256) -// mstore(add(headStart_252, 256), and(value8_255, _409)) -// mstore(add(headStart_252, 288), and(value9_254, _409)) +// mstore(add(headStart_252, 256), and(value8_255, _382)) +// mstore(add(headStart_252, 288), and(value9_254, _382)) // mstore(add(headStart_252, 320), value10_253) // } // } diff --git a/test/libyul/yulOptimizerTests/fullSuite/abi_example1.yul b/test/libyul/yulOptimizerTests/fullSuite/abi_example1.yul index 03b9c2e4cf5d..fca26bb7ce6b 100644 --- a/test/libyul/yulOptimizerTests/fullSuite/abi_example1.yul +++ b/test/libyul/yulOptimizerTests/fullSuite/abi_example1.yul @@ -460,12 +460,12 @@ // { // let _1 := 0x20 // let _2 := 0 -// let _218 := mload(_2) +// let _168 := mload(_2) // let abi_encode_pos := _1 -// let abi_encode_length_68 := mload(_218) +// let abi_encode_length_68 := mload(_168) // mstore(_1, abi_encode_length_68) // abi_encode_pos := 64 -// let abi_encode_srcPtr := add(_218, _1) +// let abi_encode_srcPtr := add(_168, _1) // let abi_encode_i_69 := _2 // for { // } @@ -474,45 +474,45 @@ // abi_encode_i_69 := add(abi_encode_i_69, 1) // } // { -// let _579 := mload(abi_encode_srcPtr) -// let abi_encode_pos_71_671 := abi_encode_pos -// let abi_encode_srcPtr_73_673 := _579 -// let abi_encode_i_74_674 := _2 +// let _491 := mload(abi_encode_srcPtr) +// let abi_encode_pos_71_583 := abi_encode_pos +// let abi_encode_srcPtr_73_585 := _491 +// let abi_encode_i_74_586 := _2 // for { // } -// lt(abi_encode_i_74_674, 0x3) +// lt(abi_encode_i_74_586, 0x3) // { -// abi_encode_i_74_674 := add(abi_encode_i_74_674, 1) +// abi_encode_i_74_586 := add(abi_encode_i_74_586, 1) // } // { -// mstore(abi_encode_pos_71_671, and(mload(abi_encode_srcPtr_73_673), 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF)) -// abi_encode_srcPtr_73_673 := add(abi_encode_srcPtr_73_673, _1) -// abi_encode_pos_71_671 := add(abi_encode_pos_71_671, _1) +// mstore(abi_encode_pos_71_583, and(mload(abi_encode_srcPtr_73_585), 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF)) +// abi_encode_srcPtr_73_585 := add(abi_encode_srcPtr_73_585, _1) +// abi_encode_pos_71_583 := add(abi_encode_pos_71_583, _1) // } // abi_encode_srcPtr := add(abi_encode_srcPtr, _1) // abi_encode_pos := add(abi_encode_pos, 0x60) // } -// let _220 := mload(64) -// let _221 := mload(_1) -// if slt(sub(_220, _221), 128) +// let _170 := mload(64) +// let _171 := mload(_1) +// if slt(sub(_170, _171), 128) // { // revert(_2, _2) // } -// let abi_decode_offset_64 := calldataload(add(_221, 64)) +// let abi_decode_offset_64 := calldataload(add(_171, 64)) // let abi_decode__74 := 0xffffffffffffffff // if gt(abi_decode_offset_64, abi_decode__74) // { // revert(_2, _2) // } -// let abi_decode_value2_314 := abi_decode_t_array$_t_uint256_$dyn_memory_ptr(add(_221, abi_decode_offset_64), _220) -// let abi_decode_offset_65 := calldataload(add(_221, 96)) +// let abi_decode_value2_264 := abi_decode_t_array$_t_uint256_$dyn_memory_ptr(add(_171, abi_decode_offset_64), _170) +// let abi_decode_offset_65 := calldataload(add(_171, 96)) // if gt(abi_decode_offset_65, abi_decode__74) // { // revert(_2, _2) // } -// let abi_decode_value3_315 := abi_decode_t_array$_t_array$_t_uint256_$2_memory_$dyn_memory_ptr(add(_221, abi_decode_offset_65), _220) -// sstore(calldataload(_221), calldataload(add(_221, _1))) -// sstore(abi_decode_value2_314, abi_decode_value3_315) +// let abi_decode_value3_265 := abi_decode_t_array$_t_array$_t_uint256_$2_memory_$dyn_memory_ptr(add(_171, abi_decode_offset_65), _170) +// sstore(calldataload(_171), calldataload(add(_171, _1))) +// sstore(abi_decode_value2_264, abi_decode_value3_265) // sstore(_2, abi_encode_pos) // function abi_decode_t_array$_t_array$_t_uint256_$2_memory_$dyn_memory_ptr(offset_3, end_4) -> array_5 // { @@ -544,10 +544,10 @@ // revert(0, 0) // } // let abi_decode_dst_15 := allocateMemory(array_allocation_size_t_array$_t_uint256_$2_memory(0x2)) -// let abi_decode_dst_15_1154 := abi_decode_dst_15 +// let abi_decode_dst_15_990 := abi_decode_dst_15 // let abi_decode_src_16 := src_8 -// let abi_decode__238 := add(src_8, 0x40) -// if gt(abi_decode__238, end_4) +// let abi_decode__188 := add(src_8, 0x40) +// if gt(abi_decode__188, end_4) // { // revert(0, 0) // } @@ -563,9 +563,9 @@ // abi_decode_dst_15 := add(abi_decode_dst_15, _16) // abi_decode_src_16 := add(abi_decode_src_16, _16) // } -// mstore(dst_7, abi_decode_dst_15_1154) +// mstore(dst_7, abi_decode_dst_15_990) // dst_7 := add(dst_7, _16) -// src_8 := abi_decode__238 +// src_8 := abi_decode__188 // } // } // function abi_decode_t_array$_t_uint256_$dyn_memory_ptr(offset_27, end_28) -> array_29 diff --git a/test/libyul/yulOptimizerTests/fullSuite/aztec.yul b/test/libyul/yulOptimizerTests/fullSuite/aztec.yul index 868c6b012da8..b89cc855754d 100644 --- a/test/libyul/yulOptimizerTests/fullSuite/aztec.yul +++ b/test/libyul/yulOptimizerTests/fullSuite/aztec.yul @@ -254,7 +254,7 @@ // hashCommitments(validateJo_notes, validateJo_n) // let validateJo_b := add(0x300, mul(validateJo_n, validateJo__6)) // let validateJo_i := 0 -// let validateJo_i_1306 := validateJo_i +// let validateJo_i_1218 := validateJo_i // for { // } // lt(validateJo_i, validateJo_n) @@ -263,10 +263,10 @@ // } // { // let validateJo__34 := 0x20 -// let validateJo__351 := add(validateJo__10, mul(validateJo_i, 0xc0)) -// let validateJo_noteIndex := add(validateJo__351, 0x24) -// let validateJo_k := validateJo_i_1306 -// let validateJo_a := calldataload(add(validateJo__351, 0x44)) +// let validateJo__329 := add(validateJo__10, mul(validateJo_i, 0xc0)) +// let validateJo_noteIndex := add(validateJo__329, 0x24) +// let validateJo_k := validateJo_i_1218 +// let validateJo_a := calldataload(add(validateJo__329, 0x44)) // let validateJo_c := validateJo_challenge // let validateJo__39 := add(validateJo_i, 0x01) // switch eq(validateJo__39, validateJo_n) @@ -284,30 +284,30 @@ // switch gt(validateJo__39, validateJo_m) // case 1 { // validateJo_kn := addmod(validateJo_kn, sub(validateJo_gen_order, validateJo_k), validateJo_gen_order) -// let validateJo_x := mod(mload(validateJo_i_1306), validateJo_gen_order) +// let validateJo_x := mod(mload(validateJo_i_1218), validateJo_gen_order) // validateJo_k := mulmod(validateJo_k, validateJo_x, validateJo_gen_order) // validateJo_a := mulmod(validateJo_a, validateJo_x, validateJo_gen_order) // validateJo_c := mulmod(validateJo_challenge, validateJo_x, validateJo_gen_order) -// mstore(validateJo_i_1306, keccak256(validateJo_i_1306, validateJo__34)) +// mstore(validateJo_i_1218, keccak256(validateJo_i_1218, validateJo__34)) // } // case 0 { // validateJo_kn := addmod(validateJo_kn, validateJo_k, validateJo_gen_order) // } // let validateJo__52 := 0x40 -// calldatacopy(0xe0, add(validateJo__351, 164), validateJo__52) -// calldatacopy(validateJo__34, add(validateJo__351, 100), validateJo__52) +// calldatacopy(0xe0, add(validateJo__329, 164), validateJo__52) +// calldatacopy(validateJo__34, add(validateJo__329, 100), validateJo__52) // let validateJo__61 := 0x120 // mstore(validateJo__61, sub(validateJo_gen_order, validateJo_c)) // let validateJo__62 := 0x60 // mstore(validateJo__62, validateJo_k) // mstore(0xc0, validateJo_a) // let validateJo__65 := 0x1a0 -// let validateJo_result := call(gas(), 7, validateJo_i_1306, 0xe0, validateJo__62, validateJo__65, validateJo__52) -// let validateJo_result_303 := and(validateJo_result, call(gas(), 7, validateJo_i_1306, validateJo__34, validateJo__62, validateJo__61, validateJo__52)) +// let validateJo_result := call(gas(), 7, validateJo_i_1218, 0xe0, validateJo__62, validateJo__65, validateJo__52) +// let validateJo_result_303 := and(validateJo_result, call(gas(), 7, validateJo_i_1218, validateJo__34, validateJo__62, validateJo__61, validateJo__52)) // let validateJo__80 := 0x160 -// let validateJo_result_304 := and(validateJo_result_303, call(gas(), 7, validateJo_i_1306, validateJo__6, validateJo__62, validateJo__80, validateJo__52)) -// let validateJo_result_305 := and(validateJo_result_304, call(gas(), 6, validateJo_i_1306, validateJo__61, validateJo__6, validateJo__80, validateJo__52)) -// validateJo_result := and(validateJo_result_305, call(gas(), 6, validateJo_i_1306, validateJo__80, validateJo__6, validateJo_b, validateJo__52)) +// let validateJo_result_304 := and(validateJo_result_303, call(gas(), 7, validateJo_i_1218, validateJo__6, validateJo__62, validateJo__80, validateJo__52)) +// let validateJo_result_305 := and(validateJo_result_304, call(gas(), 6, validateJo_i_1218, validateJo__61, validateJo__6, validateJo__80, validateJo__52)) +// validateJo_result := and(validateJo_result_305, call(gas(), 6, validateJo_i_1218, validateJo__80, validateJo__6, validateJo_b, validateJo__52)) // if eq(validateJo_i, validateJo_m) // { // mstore(0x260, mload(validateJo__34)) @@ -319,14 +319,14 @@ // { // mstore(validateJo__62, validateJo_c) // let validateJo__120 := 0x220 -// let validateJo_result_307 := and(validateJo_result, call(gas(), 7, validateJo_i_1306, validateJo__34, validateJo__62, validateJo__120, validateJo__52)) -// let validateJo_result_308 := and(validateJo_result_307, call(gas(), 6, validateJo_i_1306, validateJo__120, validateJo__6, 0x260, validateJo__52)) -// validateJo_result := and(validateJo_result_308, call(gas(), 6, validateJo_i_1306, validateJo__65, validateJo__6, 0x1e0, validateJo__52)) +// let validateJo_result_307 := and(validateJo_result, call(gas(), 7, validateJo_i_1218, validateJo__34, validateJo__62, validateJo__120, validateJo__52)) +// let validateJo_result_308 := and(validateJo_result_307, call(gas(), 6, validateJo_i_1218, validateJo__120, validateJo__6, 0x260, validateJo__52)) +// validateJo_result := and(validateJo_result_308, call(gas(), 6, validateJo_i_1218, validateJo__65, validateJo__6, 0x1e0, validateJo__52)) // } // if iszero(validateJo_result) // { -// mstore(validateJo_i_1306, 400) -// revert(validateJo_i_1306, validateJo__34) +// mstore(validateJo_i_1218, 400) +// revert(validateJo_i_1218, validateJo__34) // } // validateJo_b := add(validateJo_b, validateJo__52) // } @@ -336,13 +336,13 @@ // } // if iszero(eq(mod(keccak256(validateJo__24, add(validateJo_b, 0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffd60)), validateJo_gen_order), validateJo_challenge)) // { -// mstore(validateJo_i_1306, 404) -// revert(validateJo_i_1306, 0x20) +// mstore(validateJo_i_1218, 404) +// revert(validateJo_i_1218, 0x20) // } -// mstore(validateJo_i_1306, 0x01) -// return(validateJo_i_1306, 0x20) -// mstore(validateJo_i_1306, 404) -// revert(validateJo_i_1306, 0x20) +// mstore(validateJo_i_1218, 0x01) +// return(validateJo_i_1218, 0x20) +// mstore(validateJo_i_1218, 404) +// revert(validateJo_i_1218, 0x20) // function validatePairing(t2) // { // let t2_x_1 := calldataload(t2) diff --git a/test/libyul/yulOptimizerTests/fullSuite/medium.yul b/test/libyul/yulOptimizerTests/fullSuite/medium.yul index 1d07cd03e613..48fb617e93dd 100644 --- a/test/libyul/yulOptimizerTests/fullSuite/medium.yul +++ b/test/libyul/yulOptimizerTests/fullSuite/medium.yul @@ -21,8 +21,8 @@ // { // let allocate__19 := 0x40 // mstore(allocate__19, add(mload(allocate__19), 0x20)) -// let allocate_p_35_39 := mload(allocate__19) -// mstore(allocate__19, add(allocate_p_35_39, allocate__19)) -// mstore(add(allocate_p_35_39, 96), 2) +// let allocate_p_33_37 := mload(allocate__19) +// mstore(allocate__19, add(allocate_p_33_37, allocate__19)) +// mstore(add(allocate_p_33_37, 96), 2) // mstore(allocate__19, 0x20) // } From 83083d220868ea37adca54fbb6e690d3c3994ce1 Mon Sep 17 00:00:00 2001 From: chriseth Date: Mon, 4 Feb 2019 17:30:29 +0100 Subject: [PATCH 104/109] Stack compressor. --- libyul/CMakeLists.txt | 2 + libyul/optimiser/Rematerialiser.cpp | 32 ++++-- libyul/optimiser/Rematerialiser.h | 25 +++- libyul/optimiser/StackCompressor.cpp | 107 ++++++++++++++++++ libyul/optimiser/StackCompressor.h | 47 ++++++++ libyul/optimiser/Suite.cpp | 5 + test/libyul/YulOptimizerTest.cpp | 8 ++ .../yulOptimizerTests/fullSuite/aztec.yul | 59 ++++------ .../stackCompressor/inlineInBlock.yul | 10 ++ .../stackCompressor/inlineInFunction.yul | 16 +++ .../stackCompressor/noInline.yul | 13 +++ test/tools/yulopti.cpp | 7 +- 12 files changed, 282 insertions(+), 49 deletions(-) create mode 100644 libyul/optimiser/StackCompressor.cpp create mode 100644 libyul/optimiser/StackCompressor.h create mode 100644 test/libyul/yulOptimizerTests/stackCompressor/inlineInBlock.yul create mode 100644 test/libyul/yulOptimizerTests/stackCompressor/inlineInFunction.yul create mode 100644 test/libyul/yulOptimizerTests/stackCompressor/noInline.yul diff --git a/libyul/CMakeLists.txt b/libyul/CMakeLists.txt index 13cd0c37f957..2c7fdf8a9a02 100644 --- a/libyul/CMakeLists.txt +++ b/libyul/CMakeLists.txt @@ -94,6 +94,8 @@ add_library(yul optimiser/Semantics.h optimiser/SimplificationRules.cpp optimiser/SimplificationRules.h + optimiser/StackCompressor.cpp + optimiser/StackCompressor.h optimiser/StructuralSimplifier.cpp optimiser/StructuralSimplifier.h optimiser/Substitution.cpp diff --git a/libyul/optimiser/Rematerialiser.cpp b/libyul/optimiser/Rematerialiser.cpp index 34a23c1338ae..de1b91439297 100644 --- a/libyul/optimiser/Rematerialiser.cpp +++ b/libyul/optimiser/Rematerialiser.cpp @@ -30,25 +30,39 @@ using namespace std; using namespace dev; using namespace yul; -void Rematerialiser::run(Dialect const& _dialect, Block& _ast) +void Rematerialiser::run(Dialect const& _dialect, Block& _ast, set _varsToAlwaysRematerialize) { - Rematerialiser{_dialect, _ast}(_ast); + Rematerialiser{_dialect, _ast, std::move(_varsToAlwaysRematerialize)}(_ast); } -void Rematerialiser::run(Dialect const& _dialect, FunctionDefinition& _function) +void Rematerialiser::run( + Dialect const& _dialect, + FunctionDefinition& _function, + set _varsToAlwaysRematerialize +) { - Rematerialiser{_dialect, _function}(_function); + Rematerialiser{_dialect, _function, std::move(_varsToAlwaysRematerialize)}(_function); } -Rematerialiser::Rematerialiser(Dialect const& _dialect, Block& _ast): +Rematerialiser::Rematerialiser( + Dialect const& _dialect, + Block& _ast, + set _varsToAlwaysRematerialize +): DataFlowAnalyzer(_dialect), - m_referenceCounts(ReferencesCounter::countReferences(_ast)) + m_referenceCounts(ReferencesCounter::countReferences(_ast)), + m_varsToAlwaysRematerialize(std::move(_varsToAlwaysRematerialize)) { } -Rematerialiser::Rematerialiser(Dialect const& _dialect, FunctionDefinition& _function): +Rematerialiser::Rematerialiser( + Dialect const& _dialect, + FunctionDefinition& _function, + set _varsToAlwaysRematerialize +): DataFlowAnalyzer(_dialect), - m_referenceCounts(ReferencesCounter::countReferences(_function)) + m_referenceCounts(ReferencesCounter::countReferences(_function)), + m_varsToAlwaysRematerialize(std::move(_varsToAlwaysRematerialize)) { } @@ -64,7 +78,7 @@ void Rematerialiser::visit(Expression& _e) auto const& value = *m_value.at(name); size_t refs = m_referenceCounts[name]; size_t cost = CodeCost::codeCost(value); - if (refs <= 1 || cost == 0 || (refs <= 5 && cost <= 1)) + if (refs <= 1 || cost == 0 || (refs <= 5 && cost <= 1) || m_varsToAlwaysRematerialize.count(name)) { assertThrow(m_referenceCounts[name] > 0, OptimizerException, ""); for (auto const& ref: m_references[name]) diff --git a/libyul/optimiser/Rematerialiser.h b/libyul/optimiser/Rematerialiser.h index 04b09f05cf36..b568b5dbd156 100644 --- a/libyul/optimiser/Rematerialiser.h +++ b/libyul/optimiser/Rematerialiser.h @@ -38,17 +38,34 @@ namespace yul class Rematerialiser: public DataFlowAnalyzer { public: - static void run(Dialect const& _dialect, Block& _ast); - static void run(Dialect const& _dialect, FunctionDefinition& _function); + static void run( + Dialect const& _dialect, + Block& _ast, + std::set _varsToAlwaysRematerialize = {} + ); + static void run( + Dialect const& _dialect, + FunctionDefinition& _function, + std::set _varsToAlwaysRematerialize = {} + ); protected: - Rematerialiser(Dialect const& _dialect, Block& _ast); - Rematerialiser(Dialect const& _dialect, FunctionDefinition& _function); + Rematerialiser( + Dialect const& _dialect, + Block& _ast, + std::set _varsToAlwaysRematerialize = {} + ); + Rematerialiser( + Dialect const& _dialect, + FunctionDefinition& _function, + std::set _varsToAlwaysRematerialize = {} + ); using ASTModifier::visit; void visit(Expression& _e) override; std::map m_referenceCounts; + std::set m_varsToAlwaysRematerialize; }; } diff --git a/libyul/optimiser/StackCompressor.cpp b/libyul/optimiser/StackCompressor.cpp new file mode 100644 index 000000000000..bbf34ff61e02 --- /dev/null +++ b/libyul/optimiser/StackCompressor.cpp @@ -0,0 +1,107 @@ +/*( + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ +/** + * Optimisation stage that aggressively rematerializes certain variables ina a function to free + * space on the stack until it is compilable. + */ + +#include + +#include +#include +#include +#include +#include +#include + +#include + +#include + +using namespace std; +using namespace dev; +using namespace yul; + +namespace +{ + +template +void eliminateVariables(shared_ptr const& _dialect, ASTNode& _node, size_t _numVariables) +{ + SSAValueTracker ssaValues; + ssaValues(_node); + + map references = ReferencesCounter::countReferences(_node); + + set> rematCosts; + for (auto const& ssa: ssaValues.values()) + { + if (!MovableChecker{*_dialect, *ssa.second}.movable()) + continue; + size_t numRef = references[ssa.first]; + size_t cost = 0; + if (numRef > 1) + cost = CodeCost::codeCost(*ssa.second) * (numRef - 1); + rematCosts.insert(make_pair(cost, ssa.first)); + } + + // Select at most _numVariables + set varsToEliminate; + for (auto const& costs: rematCosts) + { + if (varsToEliminate.size() >= _numVariables) + break; + varsToEliminate.insert(costs.second); + } + + Rematerialiser::run(*_dialect, _node, std::move(varsToEliminate)); + UnusedPruner::runUntilStabilised(*_dialect, _node); +} + +} + +bool StackCompressor::run(shared_ptr const& _dialect, Block& _ast) +{ + yulAssert( + _ast.statements.size() > 0 && _ast.statements.at(0).type() == typeid(Block), + "Need to run the function grouper before the stack compressor." + ); + for (size_t iterations = 0; iterations < 4; iterations++) + { + map stackSurplus = CompilabilityChecker::run(_dialect, _ast); + if (stackSurplus.empty()) + return true; + + if (stackSurplus.count(YulString{})) + { + yulAssert(stackSurplus.at({}) > 0, "Invalid surplus value."); + eliminateVariables(_dialect, boost::get(_ast.statements.at(0)), stackSurplus.at({})); + } + + for (size_t i = 1; i < _ast.statements.size(); ++i) + { + FunctionDefinition& fun = boost::get(_ast.statements[i]); + if (!stackSurplus.count(fun.name)) + continue; + + yulAssert(stackSurplus.at(fun.name) > 0, "Invalid surplus value."); + eliminateVariables(_dialect, fun, stackSurplus.at(fun.name)); + } + } + return false; +} + diff --git a/libyul/optimiser/StackCompressor.h b/libyul/optimiser/StackCompressor.h new file mode 100644 index 000000000000..11c9aa437142 --- /dev/null +++ b/libyul/optimiser/StackCompressor.h @@ -0,0 +1,47 @@ +/* + This file is part of solidity. + + solidity is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + solidity is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with solidity. If not, see . +*/ +/** + * Optimisation stage that aggressively rematerializes certain variables ina a function to free + * space on the stack until it is compilable. + */ + +#pragma once + +#include + +namespace yul +{ + +struct Dialect; +struct Block; +struct FunctionDefinition; + +/** + * Optimisation stage that aggressively rematerializes certain variables in a function to free + * space on the stack until it is compilable. + * + * Prerequisite: Disambiguator, Function Grouper + */ +class StackCompressor +{ +public: + /// Try to remove local variables until the AST is compilable. + /// @returns true if it was successful. + static bool run(std::shared_ptr const& _dialect, Block& _ast); +}; + +} diff --git a/libyul/optimiser/Suite.cpp b/libyul/optimiser/Suite.cpp index 8a104c39699e..ab4cfd9c2077 100644 --- a/libyul/optimiser/Suite.cpp +++ b/libyul/optimiser/Suite.cpp @@ -37,6 +37,7 @@ #include #include #include +#include #include #include #include @@ -170,5 +171,9 @@ void OptimiserSuite::run( Rematerialiser::run(*_dialect, ast); UnusedPruner::runUntilStabilised(*_dialect, ast, reservedIdentifiers); + (FunctionGrouper{})(ast); + StackCompressor::run(_dialect, ast); + (BlockFlattener{})(ast); + _ast = std::move(ast); } diff --git a/test/libyul/YulOptimizerTest.cpp b/test/libyul/YulOptimizerTest.cpp index 7b0cb100460d..fa2d5c775b0f 100644 --- a/test/libyul/YulOptimizerTest.cpp +++ b/test/libyul/YulOptimizerTest.cpp @@ -42,6 +42,7 @@ #include #include #include +#include #include #include #include @@ -241,6 +242,13 @@ bool YulOptimizerTest::run(ostream& _stream, string const& _linePrefix, bool con CommonSubexpressionEliminator{*m_dialect}(*m_ast); UnusedPruner::runUntilStabilised(*m_dialect, *m_ast); } + else if (m_optimizerStep == "stackCompressor") + { + disambiguate(); + (FunctionGrouper{})(*m_ast); + StackCompressor::run(m_dialect, *m_ast); + (BlockFlattener{})(*m_ast); + } else if (m_optimizerStep == "fullSuite") OptimiserSuite::run(m_dialect, *m_ast, *m_analysisInfo); else diff --git a/test/libyul/yulOptimizerTests/fullSuite/aztec.yul b/test/libyul/yulOptimizerTests/fullSuite/aztec.yul index b89cc855754d..c61a7cddfa6d 100644 --- a/test/libyul/yulOptimizerTests/fullSuite/aztec.yul +++ b/test/libyul/yulOptimizerTests/fullSuite/aztec.yul @@ -234,10 +234,8 @@ // let validateJo__6 := 0x80 // mstore(validateJo__6, 7673901602397024137095011250362199966051872585513276903826533215767972925880) // mstore(0xa0, 8489654445897228341090914135473290831551238522473825886865492707826370766375) -// let validateJo__10 := calldataload(0x04) -// let validateJo_notes := add(0x04, validateJo__10) // let validateJo_m := calldataload(0x24) -// let validateJo_n := calldataload(validateJo_notes) +// let validateJo_n := calldataload(add(0x04, calldataload(0x04))) // let validateJo_gen_order := 0x30644e72e131a029b85045b68181585d2833e84879b9709143e1f593f0000001 // let validateJo_challenge := mod(calldataload(0x44), validateJo_gen_order) // if gt(validateJo_m, validateJo_n) @@ -246,12 +244,11 @@ // revert(0x00, 0x20) // } // let validateJo_kn := calldataload(add(calldatasize(), 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff40)) -// let validateJo__24 := 0x2a0 -// mstore(validateJo__24, caller()) +// mstore(0x2a0, caller()) // mstore(0x2c0, validateJo_kn) // mstore(0x2e0, validateJo_m) // validateJo_kn := mulmod(sub(validateJo_gen_order, validateJo_kn), validateJo_challenge, validateJo_gen_order) -// hashCommitments(validateJo_notes, validateJo_n) +// hashCommitments(add(0x04, calldataload(0x04)), validateJo_n) // let validateJo_b := add(0x300, mul(validateJo_n, validateJo__6)) // let validateJo_i := 0 // let validateJo_i_1218 := validateJo_i @@ -262,14 +259,11 @@ // validateJo_i := add(validateJo_i, 0x01) // } // { -// let validateJo__34 := 0x20 -// let validateJo__329 := add(validateJo__10, mul(validateJo_i, 0xc0)) -// let validateJo_noteIndex := add(validateJo__329, 0x24) +// let validateJo__329 := add(calldataload(0x04), mul(validateJo_i, 0xc0)) // let validateJo_k := validateJo_i_1218 // let validateJo_a := calldataload(add(validateJo__329, 0x44)) // let validateJo_c := validateJo_challenge -// let validateJo__39 := add(validateJo_i, 0x01) -// switch eq(validateJo__39, validateJo_n) +// switch eq(add(validateJo_i, 0x01), validateJo_n) // case 1 { // validateJo_k := validateJo_kn // if eq(validateJo_m, validateJo_n) @@ -278,55 +272,50 @@ // } // } // case 0 { -// validateJo_k := calldataload(validateJo_noteIndex) +// validateJo_k := calldataload(add(validateJo__329, 0x24)) // } -// validateCommitment(validateJo_noteIndex, validateJo_k, validateJo_a) -// switch gt(validateJo__39, validateJo_m) +// validateCommitment(add(validateJo__329, 0x24), validateJo_k, validateJo_a) +// switch gt(add(validateJo_i, 0x01), validateJo_m) // case 1 { // validateJo_kn := addmod(validateJo_kn, sub(validateJo_gen_order, validateJo_k), validateJo_gen_order) // let validateJo_x := mod(mload(validateJo_i_1218), validateJo_gen_order) // validateJo_k := mulmod(validateJo_k, validateJo_x, validateJo_gen_order) // validateJo_a := mulmod(validateJo_a, validateJo_x, validateJo_gen_order) // validateJo_c := mulmod(validateJo_challenge, validateJo_x, validateJo_gen_order) -// mstore(validateJo_i_1218, keccak256(validateJo_i_1218, validateJo__34)) +// mstore(validateJo_i_1218, keccak256(validateJo_i_1218, 0x20)) // } // case 0 { // validateJo_kn := addmod(validateJo_kn, validateJo_k, validateJo_gen_order) // } // let validateJo__52 := 0x40 // calldatacopy(0xe0, add(validateJo__329, 164), validateJo__52) -// calldatacopy(validateJo__34, add(validateJo__329, 100), validateJo__52) -// let validateJo__61 := 0x120 -// mstore(validateJo__61, sub(validateJo_gen_order, validateJo_c)) -// let validateJo__62 := 0x60 -// mstore(validateJo__62, validateJo_k) +// calldatacopy(0x20, add(validateJo__329, 100), validateJo__52) +// mstore(0x120, sub(validateJo_gen_order, validateJo_c)) +// mstore(0x60, validateJo_k) // mstore(0xc0, validateJo_a) -// let validateJo__65 := 0x1a0 -// let validateJo_result := call(gas(), 7, validateJo_i_1218, 0xe0, validateJo__62, validateJo__65, validateJo__52) -// let validateJo_result_303 := and(validateJo_result, call(gas(), 7, validateJo_i_1218, validateJo__34, validateJo__62, validateJo__61, validateJo__52)) -// let validateJo__80 := 0x160 -// let validateJo_result_304 := and(validateJo_result_303, call(gas(), 7, validateJo_i_1218, validateJo__6, validateJo__62, validateJo__80, validateJo__52)) -// let validateJo_result_305 := and(validateJo_result_304, call(gas(), 6, validateJo_i_1218, validateJo__61, validateJo__6, validateJo__80, validateJo__52)) -// validateJo_result := and(validateJo_result_305, call(gas(), 6, validateJo_i_1218, validateJo__80, validateJo__6, validateJo_b, validateJo__52)) +// let validateJo_result := call(gas(), 7, validateJo_i_1218, 0xe0, 0x60, 0x1a0, validateJo__52) +// let validateJo_result_303 := and(validateJo_result, call(gas(), 7, validateJo_i_1218, 0x20, 0x60, 0x120, validateJo__52)) +// let validateJo_result_304 := and(validateJo_result_303, call(gas(), 7, validateJo_i_1218, validateJo__6, 0x60, 0x160, validateJo__52)) +// let validateJo_result_305 := and(validateJo_result_304, call(gas(), 6, validateJo_i_1218, 0x120, validateJo__6, 0x160, validateJo__52)) +// validateJo_result := and(validateJo_result_305, call(gas(), 6, validateJo_i_1218, 0x160, validateJo__6, validateJo_b, validateJo__52)) // if eq(validateJo_i, validateJo_m) // { -// mstore(0x260, mload(validateJo__34)) +// mstore(0x260, mload(0x20)) // mstore(0x280, mload(validateJo__52)) // mstore(0x1e0, mload(0xe0)) // mstore(0x200, sub(0x30644e72e131a029b85045b68181585d97816a916871ca8d3c208c16d87cfd47, mload(0x100))) // } // if gt(validateJo_i, validateJo_m) // { -// mstore(validateJo__62, validateJo_c) -// let validateJo__120 := 0x220 -// let validateJo_result_307 := and(validateJo_result, call(gas(), 7, validateJo_i_1218, validateJo__34, validateJo__62, validateJo__120, validateJo__52)) -// let validateJo_result_308 := and(validateJo_result_307, call(gas(), 6, validateJo_i_1218, validateJo__120, validateJo__6, 0x260, validateJo__52)) -// validateJo_result := and(validateJo_result_308, call(gas(), 6, validateJo_i_1218, validateJo__65, validateJo__6, 0x1e0, validateJo__52)) +// mstore(0x60, validateJo_c) +// let validateJo_result_307 := and(validateJo_result, call(gas(), 7, validateJo_i_1218, 0x20, 0x60, 0x220, validateJo__52)) +// let validateJo_result_308 := and(validateJo_result_307, call(gas(), 6, validateJo_i_1218, 0x220, validateJo__6, 0x260, validateJo__52)) +// validateJo_result := and(validateJo_result_308, call(gas(), 6, validateJo_i_1218, 0x1a0, validateJo__6, 0x1e0, validateJo__52)) // } // if iszero(validateJo_result) // { // mstore(validateJo_i_1218, 400) -// revert(validateJo_i_1218, validateJo__34) +// revert(validateJo_i_1218, 0x20) // } // validateJo_b := add(validateJo_b, validateJo__52) // } @@ -334,7 +323,7 @@ // { // validatePairing(0x64) // } -// if iszero(eq(mod(keccak256(validateJo__24, add(validateJo_b, 0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffd60)), validateJo_gen_order), validateJo_challenge)) +// if iszero(eq(mod(keccak256(0x2a0, add(validateJo_b, 0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffd60)), validateJo_gen_order), validateJo_challenge)) // { // mstore(validateJo_i_1218, 404) // revert(validateJo_i_1218, 0x20) diff --git a/test/libyul/yulOptimizerTests/stackCompressor/inlineInBlock.yul b/test/libyul/yulOptimizerTests/stackCompressor/inlineInBlock.yul new file mode 100644 index 000000000000..4e594eee4a60 --- /dev/null +++ b/test/libyul/yulOptimizerTests/stackCompressor/inlineInBlock.yul @@ -0,0 +1,10 @@ +{ + let x := 8 + let y := calldataload(calldataload(9)) + mstore(y, add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(y, 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1)) +} +// ---- +// stackCompressor +// { +// mstore(calldataload(calldataload(9)), add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(calldataload(calldataload(9)), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1)) +// } diff --git a/test/libyul/yulOptimizerTests/stackCompressor/inlineInFunction.yul b/test/libyul/yulOptimizerTests/stackCompressor/inlineInFunction.yul new file mode 100644 index 000000000000..fc89a8f77194 --- /dev/null +++ b/test/libyul/yulOptimizerTests/stackCompressor/inlineInFunction.yul @@ -0,0 +1,16 @@ +{ + let x := 8 + function f() { + let y := calldataload(calldataload(9)) + mstore(y, add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(y, 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1)) + } +} +// ---- +// stackCompressor +// { +// let x := 8 +// function f() +// { +// mstore(calldataload(calldataload(9)), add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(add(calldataload(calldataload(9)), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1), 1)) +// } +// } diff --git a/test/libyul/yulOptimizerTests/stackCompressor/noInline.yul b/test/libyul/yulOptimizerTests/stackCompressor/noInline.yul new file mode 100644 index 000000000000..d357269349c3 --- /dev/null +++ b/test/libyul/yulOptimizerTests/stackCompressor/noInline.yul @@ -0,0 +1,13 @@ +{ + let x := 8 + function f() { let y := 9 } +} +// ---- +// stackCompressor +// { +// let x := 8 +// function f() +// { +// let y := 9 +// } +// } diff --git a/test/tools/yulopti.cpp b/test/tools/yulopti.cpp index 7203ef541ef2..f77c630ff60d 100644 --- a/test/tools/yulopti.cpp +++ b/test/tools/yulopti.cpp @@ -48,6 +48,7 @@ #include #include #include +#include #include #include @@ -130,7 +131,8 @@ class YulOpti cout << "(q)quit/(f)flatten/(c)se/initialize var(d)ecls/(x)plit/(j)oin/(g)rouper/(h)oister/" << endl; cout << " (e)xpr inline/(i)nline/(s)implify/(u)nusedprune/ss(a) transform/" << endl; cout << " (r)edundant assign elim./re(m)aterializer/f(o)r-loop-pre-rewriter/" << endl; - cout << " s(t)ructural simplifier/equi(v)alent function combiner/ssa re(V)erser? " << endl; + cout << " s(t)ructural simplifier/equi(v)alent function combiner/ssa re(V)erser/? " << endl; + cout << " stack com(p)ressor? " << endl; cout.flush(); int option = readStandardInputChar(); cout << ' ' << char(option) << endl; @@ -192,6 +194,9 @@ class YulOpti case 'V': SSAReverser::run(*m_ast); break; + case 'p': + StackCompressor::run(m_dialect, *m_ast); + break; default: cout << "Unknown option." << endl; } From 53f9be8a23c8e80579f6b22ceda20878681f2920 Mon Sep 17 00:00:00 2001 From: Christian Parpart Date: Mon, 11 Feb 2019 16:00:24 +0100 Subject: [PATCH 105/109] Adapts tests/ to use AnsiColorized (the generalized/moved version of FormattedScope, which in turn is removed in this PR) --- test/libsolidity/ASTJSONTest.cpp | 15 +++--- test/libsolidity/ASTJSONTest.h | 2 +- test/libsolidity/FormattedScope.h | 68 ------------------------- test/libsolidity/SMTCheckerJSONTest.cpp | 6 +-- test/libsolidity/SyntaxTest.cpp | 16 +++--- test/libsolidity/SyntaxTest.h | 2 +- test/libyul/ObjectCompilerTest.cpp | 8 +-- test/libyul/YulOptimizerTest.cpp | 14 ++--- test/tools/isoltest.cpp | 21 ++++---- 9 files changed, 43 insertions(+), 109 deletions(-) delete mode 100644 test/libsolidity/FormattedScope.h diff --git a/test/libsolidity/ASTJSONTest.cpp b/test/libsolidity/ASTJSONTest.cpp index be482d99830e..605fbfc0ce23 100644 --- a/test/libsolidity/ASTJSONTest.cpp +++ b/test/libsolidity/ASTJSONTest.cpp @@ -17,6 +17,7 @@ #include #include +#include #include #include #include @@ -26,10 +27,10 @@ #include #include -using namespace dev; -using namespace solidity; +using namespace dev::solidity; using namespace dev::solidity::test; -using namespace dev::solidity::test::formatting; +using namespace dev::formatting; +using namespace dev; using namespace std; namespace fs = boost::filesystem; using namespace boost::unit_test; @@ -116,7 +117,7 @@ bool ASTJSONTest::run(ostream& _stream, string const& _linePrefix, bool const _f if (m_expectation != m_result) { string nextIndentLevel = _linePrefix + " "; - FormattedScope(_stream, _formatted, {BOLD, CYAN}) << _linePrefix << "Expected result:" << endl; + AnsiColorized(_stream, _formatted, {BOLD, CYAN}) << _linePrefix << "Expected result:" << endl; { istringstream stream(m_expectation); string line; @@ -124,7 +125,7 @@ bool ASTJSONTest::run(ostream& _stream, string const& _linePrefix, bool const _f _stream << nextIndentLevel << line << endl; } _stream << endl; - FormattedScope(_stream, _formatted, {BOLD, CYAN}) << _linePrefix << "Obtained result:" << endl; + AnsiColorized(_stream, _formatted, {BOLD, CYAN}) << _linePrefix << "Obtained result:" << endl; { istringstream stream(m_result); string line; @@ -148,7 +149,7 @@ bool ASTJSONTest::run(ostream& _stream, string const& _linePrefix, bool const _f if (m_expectationLegacy != m_resultLegacy) { string nextIndentLevel = _linePrefix + " "; - FormattedScope(_stream, _formatted, {BOLD, CYAN}) << _linePrefix << "Expected result (legacy):" << endl; + AnsiColorized(_stream, _formatted, {BOLD, CYAN}) << _linePrefix << "Expected result (legacy):" << endl; { istringstream stream(m_expectationLegacy); string line; @@ -156,7 +157,7 @@ bool ASTJSONTest::run(ostream& _stream, string const& _linePrefix, bool const _f _stream << nextIndentLevel << line << endl; } _stream << endl; - FormattedScope(_stream, _formatted, {BOLD, CYAN}) << _linePrefix << "Obtained result (legacy):" << endl; + AnsiColorized(_stream, _formatted, {BOLD, CYAN}) << _linePrefix << "Obtained result (legacy):" << endl; { istringstream stream(m_resultLegacy); string line; diff --git a/test/libsolidity/ASTJSONTest.h b/test/libsolidity/ASTJSONTest.h index dcdaf221027a..fb63d719808b 100644 --- a/test/libsolidity/ASTJSONTest.h +++ b/test/libsolidity/ASTJSONTest.h @@ -17,7 +17,7 @@ #pragma once -#include +#include #include #include diff --git a/test/libsolidity/FormattedScope.h b/test/libsolidity/FormattedScope.h deleted file mode 100644 index 923404f0cca8..000000000000 --- a/test/libsolidity/FormattedScope.h +++ /dev/null @@ -1,68 +0,0 @@ -/* - This file is part of solidity. - - solidity is free software: you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation, either version 3 of the License, or - (at your option) any later version. - - solidity is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with solidity. If not, see . -*/ - -#pragma once - -#include - -#include -#include - -namespace dev -{ -namespace solidity -{ -namespace test -{ - -namespace formatting -{ - -static constexpr char const* RESET = "\033[0m"; -static constexpr char const* RED = "\033[1;31m"; -static constexpr char const* GREEN = "\033[1;32m"; -static constexpr char const* YELLOW = "\033[1;33m"; -static constexpr char const* CYAN = "\033[1;36m"; -static constexpr char const* BOLD = "\033[1m"; -static constexpr char const* RED_BACKGROUND = "\033[48;5;160m"; -static constexpr char const* ORANGE_BACKGROUND = "\033[48;5;166m"; -static constexpr char const* INVERSE = "\033[7m"; - -} - -class FormattedScope: boost::noncopyable -{ -public: - /// @arg _formatting List of formatting strings (e.g. colors) defined in the formatting namespace. - FormattedScope(std::ostream& _stream, bool const _enabled, std::vector const& _formatting): - m_stream(_stream), m_enabled(_enabled) - { - if (m_enabled) - for (auto const& format: _formatting) - m_stream << format; - } - ~FormattedScope() { if (m_enabled) m_stream << formatting::RESET; } - template - std::ostream& operator<<(T&& _t) { return m_stream << std::forward(_t); } -private: - std::ostream& m_stream; - bool m_enabled; -}; - -} -} -} diff --git a/test/libsolidity/SMTCheckerJSONTest.cpp b/test/libsolidity/SMTCheckerJSONTest.cpp index e9204cc4e973..18a5bc41002e 100644 --- a/test/libsolidity/SMTCheckerJSONTest.cpp +++ b/test/libsolidity/SMTCheckerJSONTest.cpp @@ -28,10 +28,10 @@ #include #include -using namespace dev; -using namespace solidity; using namespace dev::solidity::test; -using namespace dev::solidity::test::formatting; +using namespace dev::solidity; +using namespace dev::formatting; +using namespace dev; using namespace std; using namespace boost::unit_test; diff --git a/test/libsolidity/SyntaxTest.cpp b/test/libsolidity/SyntaxTest.cpp index c47ea599c788..f13b2e79cde2 100644 --- a/test/libsolidity/SyntaxTest.cpp +++ b/test/libsolidity/SyntaxTest.cpp @@ -24,11 +24,11 @@ #include #include -using namespace dev; using namespace langutil; -using namespace solidity; +using namespace dev::solidity; using namespace dev::solidity::test; -using namespace dev::solidity::test::formatting; +using namespace dev::formatting; +using namespace dev; using namespace std; namespace fs = boost::filesystem; using namespace boost::unit_test; @@ -100,9 +100,9 @@ bool SyntaxTest::printExpectationAndError(ostream& _stream, string const& _lineP if (m_expectations != m_errorList) { string nextIndentLevel = _linePrefix + " "; - FormattedScope(_stream, _formatted, {BOLD, CYAN}) << _linePrefix << "Expected result:" << endl; + AnsiColorized(_stream, _formatted, {BOLD, CYAN}) << _linePrefix << "Expected result:" << endl; printErrorList(_stream, m_expectations, nextIndentLevel, _formatted); - FormattedScope(_stream, _formatted, {BOLD, CYAN}) << _linePrefix << "Obtained result:" << endl; + AnsiColorized(_stream, _formatted, {BOLD, CYAN}) << _linePrefix << "Obtained result:" << endl; printErrorList(_stream, m_errorList, nextIndentLevel, _formatted); return false; } @@ -127,7 +127,7 @@ void SyntaxTest::printSource(ostream& _stream, string const& _linePrefix, bool c if (isWarning) { if (sourceFormatting[i] == formatting::RESET) - sourceFormatting[i] = formatting::ORANGE_BACKGROUND; + sourceFormatting[i] = formatting::ORANGE_BACKGROUND_256; } else sourceFormatting[i] = formatting::RED_BACKGROUND; @@ -166,12 +166,12 @@ void SyntaxTest::printErrorList( ) { if (_errorList.empty()) - FormattedScope(_stream, _formatted, {BOLD, GREEN}) << _linePrefix << "Success" << endl; + AnsiColorized(_stream, _formatted, {BOLD, GREEN}) << _linePrefix << "Success" << endl; else for (auto const& error: _errorList) { { - FormattedScope scope(_stream, _formatted, {BOLD, (error.type == "Warning") ? YELLOW : RED}); + AnsiColorized scope(_stream, _formatted, {BOLD, (error.type == "Warning") ? YELLOW : RED}); _stream << _linePrefix; _stream << error.type << ": "; } diff --git a/test/libsolidity/SyntaxTest.h b/test/libsolidity/SyntaxTest.h index 12c14087b50e..69394f4af861 100644 --- a/test/libsolidity/SyntaxTest.h +++ b/test/libsolidity/SyntaxTest.h @@ -18,9 +18,9 @@ #pragma once #include -#include #include #include +#include #include #include diff --git a/test/libyul/ObjectCompilerTest.cpp b/test/libyul/ObjectCompilerTest.cpp index e60f718d6fa8..d4a3a55d1d20 100644 --- a/test/libyul/ObjectCompilerTest.cpp +++ b/test/libyul/ObjectCompilerTest.cpp @@ -17,7 +17,7 @@ #include -#include +#include #include @@ -67,7 +67,7 @@ bool ObjectCompilerTest::run(ostream& _stream, string const& _linePrefix, bool c AssemblyStack stack(EVMVersion(), AssemblyStack::Language::StrictAssembly); if (!stack.parseAndAnalyze("source", m_source)) { - FormattedScope(_stream, _formatted, {formatting::BOLD, formatting::RED}) << _linePrefix << "Error parsing source." << endl; + AnsiColorized(_stream, _formatted, {formatting::BOLD, formatting::RED}) << _linePrefix << "Error parsing source." << endl; printErrors(_stream, stack.errors()); return false; } @@ -91,9 +91,9 @@ bool ObjectCompilerTest::run(ostream& _stream, string const& _linePrefix, bool c if (m_expectation != m_obtainedResult) { string nextIndentLevel = _linePrefix + " "; - FormattedScope(_stream, _formatted, {formatting::BOLD, formatting::CYAN}) << _linePrefix << "Expected result:" << endl; + AnsiColorized(_stream, _formatted, {formatting::BOLD, formatting::CYAN}) << _linePrefix << "Expected result:" << endl; printIndented(_stream, m_expectation, nextIndentLevel); - FormattedScope(_stream, _formatted, {formatting::BOLD, formatting::CYAN}) << _linePrefix << "Obtained result:" << endl; + AnsiColorized(_stream, _formatted, {formatting::BOLD, formatting::CYAN}) << _linePrefix << "Obtained result:" << endl; printIndented(_stream, m_obtainedResult, nextIndentLevel); return false; } diff --git a/test/libyul/YulOptimizerTest.cpp b/test/libyul/YulOptimizerTest.cpp index 306721a0259d..434c954860b6 100644 --- a/test/libyul/YulOptimizerTest.cpp +++ b/test/libyul/YulOptimizerTest.cpp @@ -17,8 +17,6 @@ #include -#include - #include #include @@ -52,6 +50,8 @@ #include #include +#include + #include #include @@ -245,7 +245,7 @@ bool YulOptimizerTest::run(ostream& _stream, string const& _linePrefix, bool con OptimiserSuite::run(*m_dialect, *m_ast, *m_analysisInfo); else { - FormattedScope(_stream, _formatted, {formatting::BOLD, formatting::RED}) << _linePrefix << "Invalid optimizer step: " << m_optimizerStep << endl; + AnsiColorized(_stream, _formatted, {formatting::BOLD, formatting::RED}) << _linePrefix << "Invalid optimizer step: " << m_optimizerStep << endl; return false; } @@ -254,10 +254,10 @@ bool YulOptimizerTest::run(ostream& _stream, string const& _linePrefix, bool con if (m_expectation != m_obtainedResult) { string nextIndentLevel = _linePrefix + " "; - FormattedScope(_stream, _formatted, {formatting::BOLD, formatting::CYAN}) << _linePrefix << "Expected result:" << endl; + AnsiColorized(_stream, _formatted, {formatting::BOLD, formatting::CYAN}) << _linePrefix << "Expected result:" << endl; // TODO could compute a simple diff with highlighted lines printIndented(_stream, m_expectation, nextIndentLevel); - FormattedScope(_stream, _formatted, {formatting::BOLD, formatting::CYAN}) << _linePrefix << "Obtained result:" << endl; + AnsiColorized(_stream, _formatted, {formatting::BOLD, formatting::CYAN}) << _linePrefix << "Obtained result:" << endl; printIndented(_stream, m_obtainedResult, nextIndentLevel); return false; } @@ -291,7 +291,7 @@ bool YulOptimizerTest::parse(ostream& _stream, string const& _linePrefix, bool c m_ast = yul::Parser(errorReporter, m_dialect).parse(scanner, false); if (!m_ast || !errorReporter.errors().empty()) { - FormattedScope(_stream, _formatted, {formatting::BOLD, formatting::RED}) << _linePrefix << "Error parsing source." << endl; + AnsiColorized(_stream, _formatted, {formatting::BOLD, formatting::RED}) << _linePrefix << "Error parsing source." << endl; printErrors(_stream, errorReporter.errors()); return false; } @@ -305,7 +305,7 @@ bool YulOptimizerTest::parse(ostream& _stream, string const& _linePrefix, bool c ); if (!analyzer.analyze(*m_ast) || !errorReporter.errors().empty()) { - FormattedScope(_stream, _formatted, {formatting::BOLD, formatting::RED}) << _linePrefix << "Error analyzing source." << endl; + AnsiColorized(_stream, _formatted, {formatting::BOLD, formatting::RED}) << _linePrefix << "Error analyzing source." << endl; printErrors(_stream, errorReporter.errors()); return false; } diff --git a/test/tools/isoltest.cpp b/test/tools/isoltest.cpp index e5578045ea41..c15183f8f613 100644 --- a/test/tools/isoltest.cpp +++ b/test/tools/isoltest.cpp @@ -16,6 +16,7 @@ */ #include +#include #include #include @@ -38,7 +39,7 @@ using namespace dev; using namespace dev::solidity; using namespace dev::solidity::test; -using namespace dev::solidity::test::formatting; +using namespace dev::formatting; using namespace std; namespace po = boost::program_options; namespace fs = boost::filesystem; @@ -110,7 +111,7 @@ TestTool::Result TestTool::process() bool success; std::stringstream outputMessages; - (FormattedScope(cout, m_formatted, {BOLD}) << m_name << ": ").flush(); + (AnsiColorized(cout, m_formatted, {BOLD}) << m_name << ": ").flush(); try { @@ -119,33 +120,33 @@ TestTool::Result TestTool::process() } catch(boost::exception const& _e) { - FormattedScope(cout, m_formatted, {BOLD, RED}) << + AnsiColorized(cout, m_formatted, {BOLD, RED}) << "Exception during syntax test: " << boost::diagnostic_information(_e) << endl; return Result::Exception; } catch (std::exception const& _e) { - FormattedScope(cout, m_formatted, {BOLD, RED}) << + AnsiColorized(cout, m_formatted, {BOLD, RED}) << "Exception during syntax test: " << _e.what() << endl; return Result::Exception; } catch (...) { - FormattedScope(cout, m_formatted, {BOLD, RED}) << + AnsiColorized(cout, m_formatted, {BOLD, RED}) << "Unknown exception during syntax test." << endl; return Result::Exception; } if (success) { - FormattedScope(cout, m_formatted, {BOLD, GREEN}) << "OK" << endl; + AnsiColorized(cout, m_formatted, {BOLD, GREEN}) << "OK" << endl; return Result::Success; } else { - FormattedScope(cout, m_formatted, {BOLD, RED}) << "FAIL" << endl; + AnsiColorized(cout, m_formatted, {BOLD, RED}) << "FAIL" << endl; - FormattedScope(cout, m_formatted, {BOLD, CYAN}) << " Contract:" << endl; + AnsiColorized(cout, m_formatted, {BOLD, CYAN}) << " Contract:" << endl; m_test->printSource(cout, " ", m_formatted); cout << endl << outputMessages.str() << endl; @@ -305,7 +306,7 @@ boost::optional runTestSuite( TestStats stats = TestTool::processPath(_testCaseCreator, _basePath, _subdirectory, _formatted); cout << endl << _name << " Test Summary: "; - FormattedScope(cout, _formatted, {BOLD, stats ? GREEN : RED}) << + AnsiColorized(cout, _formatted, {BOLD, stats ? GREEN : RED}) << stats.successCount << "/" << stats.testCount; @@ -389,7 +390,7 @@ Allowed options)", } cout << endl << "Summary: "; - FormattedScope(cout, formatted, {BOLD, global_stats ? GREEN : RED}) << + AnsiColorized(cout, formatted, {BOLD, global_stats ? GREEN : RED}) << global_stats.successCount << "/" << global_stats.testCount; cout << " tests successful." << endl; From 9e32aa7510ffa4dc713c2b75059db20f3158e6b0 Mon Sep 17 00:00:00 2001 From: Daniel Kirchner Date: Mon, 11 Feb 2019 17:13:12 +0100 Subject: [PATCH 106/109] Disallow calldata arrays with dynamically encoded base types in TypeChecker. --- libsolidity/analysis/TypeChecker.cpp | 10 ++++++++++ test/libsolidity/SolidityEndToEndTest.cpp | 16 ++++++++++++++++ test/libsolidity/syntaxTests/array/calldata.sol | 6 ++++++ .../syntaxTests/array/calldata_dynamic.sol | 6 ++++++ .../syntaxTests/array/calldata_multi.sol | 6 ++++++ .../syntaxTests/array/calldata_multi_dynamic.sol | 7 +++++++ .../syntaxTests/structs/array_calldata.sol | 1 + 7 files changed, 52 insertions(+) create mode 100644 test/libsolidity/syntaxTests/array/calldata.sol create mode 100644 test/libsolidity/syntaxTests/array/calldata_dynamic.sol create mode 100644 test/libsolidity/syntaxTests/array/calldata_multi.sol create mode 100644 test/libsolidity/syntaxTests/array/calldata_multi_dynamic.sol diff --git a/libsolidity/analysis/TypeChecker.cpp b/libsolidity/analysis/TypeChecker.cpp index e5919560677a..c716d673030f 100644 --- a/libsolidity/analysis/TypeChecker.cpp +++ b/libsolidity/analysis/TypeChecker.cpp @@ -366,6 +366,16 @@ bool TypeChecker::visit(FunctionDefinition const& _function) for (ASTPointer const& var: _function.parameters()) { TypePointer baseType = type(*var); + if (auto const* arrayType = dynamic_cast(baseType.get())) + { + baseType = arrayType->baseType(); + if ( + !m_scope->isInterface() && + baseType->dataStoredIn(DataLocation::CallData) && + baseType->isDynamicallyEncoded() + ) + m_errorReporter.typeError(var->location(), "Calldata arrays with dynamically encoded base types are not yet supported."); + } while (auto const* arrayType = dynamic_cast(baseType.get())) baseType = arrayType->baseType(); diff --git a/test/libsolidity/SolidityEndToEndTest.cpp b/test/libsolidity/SolidityEndToEndTest.cpp index 31d9be7a14e0..010a05d739a7 100644 --- a/test/libsolidity/SolidityEndToEndTest.cpp +++ b/test/libsolidity/SolidityEndToEndTest.cpp @@ -8013,6 +8013,22 @@ BOOST_AUTO_TEST_CASE(struct_named_constructor) ABI_CHECK(callContractFunction("s()"), encodeArgs(u256(1), true)); } +BOOST_AUTO_TEST_CASE(calldata_array) +{ + char const* sourceCode = R"( + pragma experimental ABIEncoderV2; + contract C { + function f(uint[2] calldata s) external pure returns (uint256 a, uint256 b) { + a = s[0]; + b = s[1]; + } + } + )"; + compileAndRun(sourceCode, 0, "C"); + + ABI_CHECK(callContractFunction("f(uint256[2])", encodeArgs(u256(42), u256(23))), encodeArgs(u256(42), u256(23))); +} + BOOST_AUTO_TEST_CASE(calldata_struct) { char const* sourceCode = R"( diff --git a/test/libsolidity/syntaxTests/array/calldata.sol b/test/libsolidity/syntaxTests/array/calldata.sol new file mode 100644 index 000000000000..68276a8ec5b5 --- /dev/null +++ b/test/libsolidity/syntaxTests/array/calldata.sol @@ -0,0 +1,6 @@ +pragma experimental ABIEncoderV2; +contract Test { + function f(uint[3] calldata) external { } +} +// ---- +// Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. diff --git a/test/libsolidity/syntaxTests/array/calldata_dynamic.sol b/test/libsolidity/syntaxTests/array/calldata_dynamic.sol new file mode 100644 index 000000000000..bba2541f9919 --- /dev/null +++ b/test/libsolidity/syntaxTests/array/calldata_dynamic.sol @@ -0,0 +1,6 @@ +pragma experimental ABIEncoderV2; +contract Test { + function f(uint[] calldata) external { } +} +// ---- +// Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. diff --git a/test/libsolidity/syntaxTests/array/calldata_multi.sol b/test/libsolidity/syntaxTests/array/calldata_multi.sol new file mode 100644 index 000000000000..570f3e9427d4 --- /dev/null +++ b/test/libsolidity/syntaxTests/array/calldata_multi.sol @@ -0,0 +1,6 @@ +pragma experimental ABIEncoderV2; +contract Test { + function f(uint[3][4] calldata) external { } +} +// ---- +// Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. diff --git a/test/libsolidity/syntaxTests/array/calldata_multi_dynamic.sol b/test/libsolidity/syntaxTests/array/calldata_multi_dynamic.sol new file mode 100644 index 000000000000..a732b44626bb --- /dev/null +++ b/test/libsolidity/syntaxTests/array/calldata_multi_dynamic.sol @@ -0,0 +1,7 @@ +pragma experimental ABIEncoderV2; +contract Test { + function f(uint[][] calldata) external { } +} +// ---- +// Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. +// TypeError: (65-82): Calldata arrays with dynamically encoded base types are not yet supported. diff --git a/test/libsolidity/syntaxTests/structs/array_calldata.sol b/test/libsolidity/syntaxTests/structs/array_calldata.sol index 9e1071a0e5ab..8da097884cf0 100644 --- a/test/libsolidity/syntaxTests/structs/array_calldata.sol +++ b/test/libsolidity/syntaxTests/structs/array_calldata.sol @@ -6,3 +6,4 @@ contract Test { } // ---- // Warning: (0-33): Experimental features are turned on. Do not use experimental features on live deployments. +// TypeError: (131-145): Calldata arrays with dynamically encoded base types are not yet supported. From 091bbdb459df7a54d426b56caf1e2afbcc09f2e6 Mon Sep 17 00:00:00 2001 From: chriseth Date: Tue, 12 Feb 2019 10:53:59 +0100 Subject: [PATCH 107/109] More tests for overloaded events. --- test/libsolidity/SolidityEndToEndTest.cpp | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/test/libsolidity/SolidityEndToEndTest.cpp b/test/libsolidity/SolidityEndToEndTest.cpp index 31d9be7a14e0..e47973dd08b5 100644 --- a/test/libsolidity/SolidityEndToEndTest.cpp +++ b/test/libsolidity/SolidityEndToEndTest.cpp @@ -3746,17 +3746,22 @@ BOOST_AUTO_TEST_CASE(events_with_same_name) event Deposit(); event Deposit(address _addr); event Deposit(address _addr, uint _amount); + event Deposit(address _addr, bool _flag); function deposit() public returns (uint) { emit Deposit(); return 1; } function deposit(address _addr) public returns (uint) { emit Deposit(_addr); - return 1; + return 2; } function deposit(address _addr, uint _amount) public returns (uint) { emit Deposit(_addr, _amount); - return 1; + return 3; + } + function deposit(address _addr, bool _flag) public returns (uint) { + emit Deposit(_addr, _flag); + return 4; } } )"; @@ -3770,19 +3775,26 @@ BOOST_AUTO_TEST_CASE(events_with_same_name) BOOST_REQUIRE_EQUAL(m_logs[0].topics.size(), 1); BOOST_CHECK_EQUAL(m_logs[0].topics[0], dev::keccak256(string("Deposit()"))); - ABI_CHECK(callContractFunction("deposit(address)", c_loggedAddress), encodeArgs(u256(1))); + ABI_CHECK(callContractFunction("deposit(address)", c_loggedAddress), encodeArgs(u256(2))); BOOST_REQUIRE_EQUAL(m_logs.size(), 1); BOOST_CHECK_EQUAL(m_logs[0].address, m_contractAddress); BOOST_CHECK(m_logs[0].data == encodeArgs(c_loggedAddress)); BOOST_REQUIRE_EQUAL(m_logs[0].topics.size(), 1); BOOST_CHECK_EQUAL(m_logs[0].topics[0], dev::keccak256(string("Deposit(address)"))); - ABI_CHECK(callContractFunction("deposit(address,uint256)", c_loggedAddress, u256(100)), encodeArgs(u256(1))); + ABI_CHECK(callContractFunction("deposit(address,uint256)", c_loggedAddress, u256(100)), encodeArgs(u256(3))); BOOST_REQUIRE_EQUAL(m_logs.size(), 1); BOOST_CHECK_EQUAL(m_logs[0].address, m_contractAddress); BOOST_CHECK(m_logs[0].data == encodeArgs(c_loggedAddress, 100)); BOOST_REQUIRE_EQUAL(m_logs[0].topics.size(), 1); BOOST_CHECK_EQUAL(m_logs[0].topics[0], dev::keccak256(string("Deposit(address,uint256)"))); + + ABI_CHECK(callContractFunction("deposit(address,bool)", c_loggedAddress, false), encodeArgs(u256(4))); + BOOST_REQUIRE_EQUAL(m_logs.size(), 1); + BOOST_CHECK_EQUAL(m_logs[0].address, m_contractAddress); + BOOST_CHECK(m_logs[0].data == encodeArgs(c_loggedAddress, false)); + BOOST_REQUIRE_EQUAL(m_logs[0].topics.size(), 1); + BOOST_CHECK_EQUAL(m_logs[0].topics[0], dev::keccak256(string("Deposit(address,bool)"))); } BOOST_AUTO_TEST_CASE(events_with_same_name_inherited_emit) From 7751fa740f699af0f2eb87f7b7b4cc072cd32382 Mon Sep 17 00:00:00 2001 From: Bhargava Shastry Date: Wed, 6 Feb 2019 14:39:43 +0100 Subject: [PATCH 108/109] Ensure we fail gracefully when user tries to optimize Yul code --- Changelog.md | 1 + libsolidity/interface/AssemblyStack.cpp | 3 ++- solc/CommandLineInterface.cpp | 6 ++---- test/cmdlineTests.sh | 3 +-- 4 files changed, 6 insertions(+), 7 deletions(-) diff --git a/Changelog.md b/Changelog.md index 2bf09226dabc..d3083b861fae 100644 --- a/Changelog.md +++ b/Changelog.md @@ -6,6 +6,7 @@ Bugfixes: * Type Checker: Dissallow mappings with data locations other than 'storage' * Type Checker: Fix internal error when a struct array index does not fit into a uint256. * Type system: Properly report packed encoded size for arrays and structs (mostly unused until now). + * Commandline interface: Allow yul optimizer only for strict assembly. Language Features: diff --git a/libsolidity/interface/AssemblyStack.cpp b/libsolidity/interface/AssemblyStack.cpp index a09de3c3880e..a600cbee1af4 100644 --- a/libsolidity/interface/AssemblyStack.cpp +++ b/libsolidity/interface/AssemblyStack.cpp @@ -84,7 +84,8 @@ bool AssemblyStack::parseAndAnalyze(std::string const& _sourceName, std::string void AssemblyStack::optimize() { - solAssert(m_language != Language::Assembly, "Optimization requested for loose assembly."); + if (m_language != Language::StrictAssembly) + solUnimplemented("Optimizer for both loose assembly and Yul is not yet implemented"); solAssert(m_analysisSuccessful, "Analysis was not successful."); m_analysisSuccessful = false; optimize(*m_parserResult); diff --git a/solc/CommandLineInterface.cpp b/solc/CommandLineInterface.cpp index c0b5f974a3ee..5485d3b055de 100644 --- a/solc/CommandLineInterface.cpp +++ b/solc/CommandLineInterface.cpp @@ -861,13 +861,11 @@ bool CommandLineInterface::processInput() return false; } } - if (optimize && inputLanguage == Input::Assembly) + if (optimize && inputLanguage != Input::StrictAssembly) { serr() << - "Optimizer cannot be used for loose assembly. Use --" << + "Optimizer can only be used for strict assembly. Use --" << g_strStrictAssembly << - " or --" << - g_strYul << "." << endl; return false; diff --git a/test/cmdlineTests.sh b/test/cmdlineTests.sh index 1ef1b320b3f0..9a3e53111a4e 100755 --- a/test/cmdlineTests.sh +++ b/test/cmdlineTests.sh @@ -332,13 +332,12 @@ printTask "Testing assemble, yul, strict-assembly and optimize..." # Test options above in conjunction with --optimize. # Using both, --assemble and --optimize should fail. ! echo '{}' | "$SOLC" - --assemble --optimize &>/dev/null + ! echo '{}' | "$SOLC" - --yul --optimize &>/dev/null # Test yul and strict assembly output # Non-empty code results in non-empty binary representation with optimizations turned off, # while it results in empty binary representation with optimizations turned on. test_solc_assembly_output "{ let x:u256 := 0:u256 }" "{ let x:u256 := 0:u256 }" "--yul" - test_solc_assembly_output "{ let x:u256 := 0:u256 }" "{ }" "--yul --optimize" - test_solc_assembly_output "{ let x := 0 }" "{ let x := 0 }" "--strict-assembly" test_solc_assembly_output "{ let x := 0 }" "{ }" "--strict-assembly --optimize" ) From 1990d7034ac7bb2adeed22def9658859e2995ff8 Mon Sep 17 00:00:00 2001 From: Erik Kundt Date: Tue, 12 Feb 2019 11:55:58 +0100 Subject: [PATCH 109/109] Updates Changelog and buglist for the 0.5.4 release. --- Changelog.md | 24 +++++++++++------------- docs/bugs_by_version.json | 4 ++++ 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/Changelog.md b/Changelog.md index d3083b861fae..8f533ecc0ecb 100644 --- a/Changelog.md +++ b/Changelog.md @@ -1,32 +1,30 @@ -### 0.5.4 (unreleased) - -Bugfixes: - * Code generator: Defensively pad allocation of creationCode and runtimeCode to multiples of 32 bytes. - * Parser: Disallow empty import statements. - * Type Checker: Dissallow mappings with data locations other than 'storage' - * Type Checker: Fix internal error when a struct array index does not fit into a uint256. - * Type system: Properly report packed encoded size for arrays and structs (mostly unused until now). - * Commandline interface: Allow yul optimizer only for strict assembly. - +### 0.5.4 (2019-02-12) Language Features: * Allow calldata structs without dynamically encoded members with ABIEncoderV2. Compiler Features: - * Implement packed encoding for ABIEncoderV2. + * ABIEncoderV2: Implement packed encoding. * C API (``libsolc`` / raw ``soljson.js``): Introduce ``solidity_free`` method which releases all internal buffers to save memory. - * Commandline interface: Adds new option ``--new-reporter`` for improved diagnostics formatting + * Commandline Interface: Adds new option ``--new-reporter`` for improved diagnostics formatting along with ``--color`` and ``--no-color`` for colorized output to be forced (or explicitly disabled). + Bugfixes: + * Code Generator: Defensively pad allocation of creationCode and runtimeCode to multiples of 32 bytes. + * Commandline Interface: Allow yul optimizer only for strict assembly. + * Parser: Disallow empty import statements. + * Type Checker: Disallow mappings with data locations other than ``storage``. + * Type Checker: Fix internal error when a struct array index does not fit into a uint256. + * Type System: Properly report packed encoded size for arrays and structs (mostly unused until now). Build System: * Add support for continuous fuzzing via Google oss-fuzz * SMT: If using Z3, require version 4.6.0 or newer. - * Ubuntu PPA Packages: Use CVC4 as SMT solver instead of Z3 * Soltest: Add parser that is used in the file-based unit test environment. + * Ubuntu PPA Packages: Use CVC4 as SMT solver instead of Z3 ### 0.5.3 (2019-01-22) diff --git a/docs/bugs_by_version.json b/docs/bugs_by_version.json index e1581293b87e..531dcd46279a 100644 --- a/docs/bugs_by_version.json +++ b/docs/bugs_by_version.json @@ -624,5 +624,9 @@ "0.5.3": { "bugs": [], "released": "2019-01-22" + }, + "0.5.4": { + "bugs": [], + "released": "2019-02-12" } } \ No newline at end of file