feat: implement dynamic arrays #2556

charles-cooper · 2021-12-04T01:59:29Z

What I did

This commit implements Solidity-ABI-compatible dynamic arrays in vyper.

Dynamic arrays have the type DynArray[<type>, <maxlen>]. They are
implemented similarly to bytestrings, with a single length word followed
by the data, <len> <data...>.

How I did it

Threaded through typechecker, added abi encoder/decoder and added the relevant functionality in codegen

How to verify it

See tests

Description for the changelog

add dynamic arrays to vyper

Cute Animal Picture

lgtm-com · 2021-12-06T21:05:48Z

This pull request introduces 1 alert when merging 2eba8e0 into fac2152 - view on LGTM.com

new alerts:

1 for Unused import

lgtm-com · 2021-12-07T21:20:41Z

This pull request introduces 2 alerts when merging bea42c4 into fac2152 - view on LGTM.com

new alerts:

2 for Unused import

lgtm-com · 2021-12-07T21:52:05Z

This pull request introduces 2 alerts when merging b1c93ec into fac2152 - view on LGTM.com

new alerts:

2 for Unused import

lgtm-com · 2021-12-13T20:19:42Z

This pull request introduces 1 alert when merging 8bdbcdd into 654b21d - view on LGTM.com

new alerts:

1 for Unused import

lgtm-com · 2021-12-13T21:08:01Z

This pull request introduces 1 alert when merging 04fb957 into 654b21d - view on LGTM.com

new alerts:

1 for Unused import

lgtm-com · 2021-12-14T02:38:36Z

This pull request introduces 2 alerts when merging 3fa97c2 into 654b21d - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Unused import

lgtm-com · 2021-12-14T02:57:53Z

This pull request introduces 3 alerts and fixes 2 when merging b398f15 into 654b21d - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Unused import
1 for Syntax error

fixed alerts:

1 for `__eq__` not overridden when adding attributes
1 for Use of the return value of a procedure

lgtm-com · 2021-12-20T22:31:05Z

This pull request introduces 4 alerts when merging c33e36d into c843b68 - view on LGTM.com

new alerts:

2 for Variable defined multiple times
1 for Unused local variable
1 for Unused import

also, rename to memory_bytes_required because get_size_of_type is almost always used in the context of trying to find out how many bytes has been or should be allocated for a type in memory.

strip dead location param get rid of very old deprecation warning

fubuloubu · 2022-01-04T05:01:11Z

tests/grammar/vyper.lark

@@ -74,11 +74,12 @@ event_def: _EVENT_DECL NAME ":" ( event_body | _PASS )

 // Types
 array_def: (NAME | array_def) "[" (DEC_NUMBER | NAME) "]"
-tuple_def: "(" ( NAME | array_def | tuple_def ) ( "," ( NAME | array_def | tuple_def ) )* [","] ")"
+dyn_array_def: "DynArray" "[" (NAME | array_def | dyn_array_def) "," (DEC_NUMBER | NAME) "]"
+tuple_def: "(" ( NAME | array_def | dyn_array_def | tuple_def ) ( "," ( NAME | array_def | dyn_array_def | tuple_def ) )* [","] ")"


Tuple of DynArrays shouldn't cause any issues right?

Theoretically no, we should add it to list of things to QA

vyper/old_codegen/lll_node.py

fubuloubu · 2022-01-04T05:05:20Z

vyper/old_codegen/lll_node.py

+                _check(repeat_count is None or repeat_count.valency == 1, repeat_count)
+                _check(counter_ptr.valency == 1, counter_ptr)
+                _check(start.valency == 1, start)
+                _check(body.valency == 0, body)


alright so maybe saves a little code here

vyper/semantics/validation/annotation.py

vyper/semantics/validation/utils.py

tests/parser/types/test_dynamic_array.py

vyper/old_codegen/parser_utils.py

lgtm-com · 2022-01-06T02:07:32Z

This pull request introduces 1 alert when merging dc05585 into 4385110 - view on LGTM.com

new alerts:

1 for Unused local variable

This reverts commit 2e5d05e.

fubuloubu · 2022-01-06T04:14:43Z

vyper/semantics/validation/utils.py

@@ -20,7 +20,11 @@
 from vyper.semantics.namespace import get_namespace
 from vyper.semantics.types.abstract import IntegerAbstractType
 from vyper.semantics.types.bases import BaseTypeDefinition
-from vyper.semantics.types.indexable.sequence import ArrayDefinition, TupleDefinition
+from vyper.semantics.types.indexable.sequence import (
+    ArrayDefinition,


Should this be StaticArrayDefinition to match?

Could be, I didn't want to change the existing type name

Haha https://github.com/vyperlang/vyper/pull/2556/files#diff-19b7b9e98dc724b69e793038c0d43aff56b4b5c9636a37ffef6bff7332a29432R57

This commit implements Solidity-ABI-compatible dynamic arrays in vyper. Dynamic arrays have the type `DynArray[<type>, <maxlen>]`. They are implemented similarly to bytestrings, with a single length word followed by the data, <len> <data...>. It also adds some code to clarify between the sizes of objects when they are in memory vs in storage. This commit also refactors a lot of code dealing with encoding and byte copying. Byte copying is cheaper now since previously the logic was ``` for i in range(maxlen // 32): if i > len: break sstore(dst + i, mload(src + 32*i)) ``` This was necessary because the `repeat` opcode in LLL requires a max bound. Now, we have repeat(len, maxlen), which still enforces the max bound but breaks once it hits the minimum of `len` and `maxlen`. Now the byte copier loop has the logic of: ``` for i in range( min(len, maxlen) ): sstore(dst + i, mload(src + 32*i)) ``` It also adds a method to the LLLnode API, `cache_when_complex`. This abstracts the logic for checking when an LLL expression needs to be cached using a `with` expression. This was used to simplify the ABI encoder and several routines related to copying (including `make_setter`).

this fixes an array index clamp bug introduced in #2556 where negative numbers are not clamped properly it also - streamlines logic by taking advantage of downstream optimizations - blocks user indexing into `empty` arrays - cleans up optimizer code - adds various `slice` tests (in preparation for rewrite)

charles-cooper force-pushed the dynamic_arrays branch 3 times, most recently from ce872df to 2eba8e0 Compare December 6, 2021 20:46

charles-cooper force-pushed the dynamic_arrays branch from 2eba8e0 to bea42c4 Compare December 7, 2021 21:00

charles-cooper force-pushed the dynamic_arrays branch from c9a573a to 8bdbcdd Compare December 13, 2021 19:58

charles-cooper force-pushed the dynamic_arrays branch from 8bdbcdd to 04fb957 Compare December 13, 2021 20:47

charles-cooper force-pushed the dynamic_arrays branch from 04fb957 to 3fa97c2 Compare December 14, 2021 02:19

charles-cooper force-pushed the dynamic_arrays branch from b398f15 to c33e36d Compare December 20, 2021 22:12

charles-cooper added 14 commits December 21, 2021 11:24

rename canonical_type to canonical_abi_type

910b1a3

rename get_size_of_type for bytes/words clarity

ed4d46d

also, rename to memory_bytes_required because get_size_of_type is almost always used in the context of trying to find out how many bytes has been or should be allocated for a type in memory.

start adding dynamic array to ast

bd80758

add DArrayType; rename ListType to SArrayType

0d8e147

lint setup.py

032bc4e

wip: codegen for dynamic arrays

2c19f6d

refactor parse_type

1ec9d9e

strip dead location param get rid of very old deprecation warning

fix parse_type for dyn array

1072e79

fix some codegen

6c3a405

fix byte_slice_copier

87eade4

simplify byte_array_copier

5e8ec66

add an abstraction for caching complex expressions

d9babb6

minor clean up for LLLnode constructor

da497eb

fix lint and add some notes

f0ac551

charles-cooper force-pushed the dynamic_arrays branch from 80272a9 to b7b5e18 Compare January 3, 2022 17:42

charles-cooper changed the title ~~[WIP] dynamic arrays~~ feat: implement dynamic arrays Jan 3, 2022

charles-cooper marked this pull request as ready for review January 3, 2022 19:34

charles-cooper requested a review from fubuloubu January 3, 2022 19:34

fubuloubu reviewed Jan 4, 2022

View reviewed changes

charles-cooper mentioned this pull request Jan 4, 2022

error with specific form of array construction for all types #2587

Closed

fubuloubu reviewed Jan 4, 2022

View reviewed changes

tests/parser/types/test_dynamic_array.py Show resolved Hide resolved

tests/parser/types/test_dynamic_array.py Outdated Show resolved Hide resolved

vyper/old_codegen/parser_utils.py Show resolved Hide resolved

tserg mentioned this pull request Jan 4, 2022

fix: error handling for string and byte arrays #2584

Merged

fubuloubu approved these changes Jan 5, 2022

View reviewed changes

charles-cooper added 4 commits January 5, 2022 15:52

Merge branch 'master' into dynamic_arrays

1784343

constant dynarrays work

039d594

make empty(DynArray[]) work

2e5d05e

add check for repeat dyn bound == 0

dc05585

charles-cooper added 2 commits January 6, 2022 03:58

Revert "make empty(DynArray[]) work"

de65cc8

This reverts commit 2e5d05e.

make empty(DynArray[]) work again

658df8f

fubuloubu approved these changes Jan 6, 2022

View reviewed changes

charles-cooper added 5 commits January 6, 2022 15:47

handle empty DynArray in lark grammar

4d656e9

add some tests for empty dyn arrays

7432148

add more empty dynarray tests

5ed4177

add oob tests and heterogeneously shaped dynarrays

cc7d5aa

charles-cooper force-pushed the dynamic_arrays branch from 3f07f9a to f01415d Compare January 6, 2022 21:27

add comment for repeat 0-checking

809aff8

charles-cooper merged commit ef1e6e4 into vyperlang:master Jan 6, 2022

charles-cooper deleted the dynamic_arrays branch January 6, 2022 22:18

charles-cooper mentioned this pull request Jan 6, 2022

feat: support address code attribute #2583

Merged

charles-cooper mentioned this pull request Feb 11, 2022

fix: array index clamp bug #2645

Merged

fubuloubu mentioned this pull request Mar 14, 2022

VIP: Dynamic Arrays w/ Sizing #1440

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement dynamic arrays #2556

feat: implement dynamic arrays #2556

charles-cooper commented Dec 4, 2021 •

edited

Loading

lgtm-com bot commented Dec 6, 2021

lgtm-com bot commented Dec 7, 2021

lgtm-com bot commented Dec 7, 2021

lgtm-com bot commented Dec 13, 2021

lgtm-com bot commented Dec 13, 2021

lgtm-com bot commented Dec 14, 2021

lgtm-com bot commented Dec 14, 2021

lgtm-com bot commented Dec 20, 2021

fubuloubu Jan 4, 2022

charles-cooper Jan 4, 2022

fubuloubu Jan 4, 2022

lgtm-com bot commented Jan 6, 2022

fubuloubu Jan 6, 2022

charles-cooper Jan 6, 2022

charles-cooper Jan 6, 2022

feat: implement dynamic arrays #2556

feat: implement dynamic arrays #2556

Conversation

charles-cooper commented Dec 4, 2021 • edited Loading

What I did

How I did it

How to verify it

Description for the changelog

Cute Animal Picture

lgtm-com bot commented Dec 6, 2021

lgtm-com bot commented Dec 7, 2021

lgtm-com bot commented Dec 7, 2021

lgtm-com bot commented Dec 13, 2021

lgtm-com bot commented Dec 13, 2021

lgtm-com bot commented Dec 14, 2021

lgtm-com bot commented Dec 14, 2021

lgtm-com bot commented Dec 20, 2021

fubuloubu Jan 4, 2022

Choose a reason for hiding this comment

charles-cooper Jan 4, 2022

Choose a reason for hiding this comment

fubuloubu Jan 4, 2022

Choose a reason for hiding this comment

lgtm-com bot commented Jan 6, 2022

fubuloubu Jan 6, 2022

Choose a reason for hiding this comment

charles-cooper Jan 6, 2022

Choose a reason for hiding this comment

charles-cooper Jan 6, 2022

Choose a reason for hiding this comment

charles-cooper commented Dec 4, 2021 •

edited

Loading