fix: type check for constants #2603

tserg · 2022-01-16T15:40:33Z

What I did

Based on the issue, the following code compiles when it should not:

MY_CONSTANT: constant(uint256[3]) = [1,2,3]

@external
def foo() -> uint8[3]:
    return MY_CONSTANT

While looking into this, I discovered two other similar code snippets that also compile when they should not:

MY_CONSTANT: constant(uint256[3]) = [1,2,3]

@external
def foo() -> uint8:
    return MY_CONSTANT[0]

MY_LIST: constant(uint256[6]) = [1,2,3]

@external
def ix(i: uint256) -> uint8:
    return MY_LIST[i]

The bug does not appear to be limited to constant arrays, however. The following code snippet also compiles when it should not:

MY_CONSTANT: constant(uint256) = 1

@external
def foo() -> uint8:
    return MY_CONSTANT

How I did it

Propagate the type definition of a constant when it is folded (i.e. replaced by a new AST node) by appending this information as a string to the new node's _metadata attribute using the type key. This value is then used to narrow down the possible type of a node from a list of possible types.

How to verify it

See tests.

Description for the changelog

Fix type checking for constants

Cute Animal Picture

codecov-commenter · 2022-01-16T15:53:28Z

Codecov Report

Merging #2603 (758b3a0) into master (bbbd861) will decrease coverage by 0.05%.
The diff coverage is 95.00%.

@@            Coverage Diff             @@
##           master    #2603      +/-   ##
==========================================
- Coverage   86.59%   86.53%   -0.06%     
==========================================
  Files          91       92       +1     
  Lines        9438     9472      +34     
  Branches     2362     2403      +41     
==========================================
+ Hits         8173     8197      +24     
- Misses        777      785       +8     
- Partials      488      490       +2

Impacted Files	Coverage Δ
vyper/ast/folding.py	`95.14% <94.11%> (-0.51%)`	⬇️
vyper/semantics/validation/local.py	`91.45% <100.00%> (-0.79%)`	⬇️
vyper/semantics/validation/utils.py	`90.95% <100.00%> (+0.08%)`	⬆️
vyper/semantics/validation/annotation.py	`91.78% <0.00%> (-1.37%)`	⬇️
vyper/codegen/core.py	`83.11% <0.00%> (ø)`
vyper/codegen/types/convert.py	`61.29% <0.00%> (ø)`
vyper/compiler/output.py	`94.64% <0.00%> (+0.03%)`	⬆️
vyper/codegen/expr.py	`80.69% <0.00%> (+0.07%)`	⬆️
vyper/codegen/abi_encoder.py	`85.85% <0.00%> (+0.14%)`	⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bbbd861...758b3a0. Read the comment docs.

charles-cooper

@tserg thanks for the contribution! I think this PR will be cleaner if you reuse existing machinery for parsing types, and then follow the convention of putting type objects (e.g. some kind of TypeDefinition) into node._metadata["type"]

vyper/ast/folding.py

charles-cooper · 2022-01-17T16:21:24Z

vyper/ast/folding.py

-        return new_node.from_node(old_node, value=new_node.value)
+        new_node = new_node.from_node(old_node, value=new_node.value)
+        if type:
+            new_node._metadata["type"] = type


isn't node._metadata["type"] usually some kind of type object, not a string?

Yes, it is usually a TypeDefinition. My initial thinking was that TypeDefinition objects are assigned at a later stage under semantics, and it could be inappropriate/messy/premature to import those classes and helper functions into the ast module. However, as you pointed out, the inconsistency/messiness ends up in the _metadata["type"] instead.

Right that's very interesting, I forgot type checking happens after constant folding. We should consider switching the order of the phases.

I tried moving the type check phase before constant folding. The good news is that it resolves the issue being addressed in this PR. The bad news is that there are about another 100 test cases that fail.

Some of the type-checking issues raised are due to:

Values for built-in constants such as ZERO_ADDRESS not being injected, and therefore not available during type checking.

Values for user-defined constants in cases where they are used as subscript or loop ranges not being injected, and therefore not available during type checking.

The remaining issues will require more investigation (e.g. type mismatches, method id must be folded, missing "type" key in metadata).

At the moment, it seems that switching phases is more complex than originally thought. I was thinking that we may need to split up the constant folding phase, but even that may be insufficient.

In that case, we should not use the "type" key because having "type" be polymorphic is really messy. We should think about a more long-term solution but for now I think we can attach a "constant_annotation", which we can inspect during the annotation / type checking phase. Also I would just attach the entire annotation (AST node) instead of converting it to a string.

vyper/ast/folding.py

tserg · 2022-01-18T03:59:51Z

Update: I tried to refactor the parametrizing of distinct integer types for the negative test cases to reduce duplication and potentially in view of a much bigger list of integer types. However, it seems that pytest has an existing limitation regarding how fixtures can be used in parametrize. IMO, the best approach at this moment is to rely on the workaround in the first post of this thread (pytest-dev/pytest#349).

lgtm-com · 2022-01-18T04:10:29Z

This pull request introduces 1 alert when merging 63eb93b into 2735f02 - view on LGTM.com

new alerts:

1 for Except block handles 'BaseException'

tests/parser/types/test_lists.py

tests/conftest.py

tests/parser/types/numbers/test_constants.py

lgtm-com · 2022-01-18T06:07:09Z

This pull request introduces 1 alert when merging a96340c into 2735f02 - view on LGTM.com

new alerts:

1 for Except block handles 'BaseException'

vyper/ast/folding.py

…folding

vyper/semantics/validation/utils.py

charles-cooper · 2022-01-22T14:14:24Z

vyper/semantics/validation/utils.py

+                propagated_type = propagated_type.value_type
+
+            for _, t in enumerate(types_list):
+                if type(propagated_type) is type(t):


direct type comparison always jumps out at me .. can you help me understand what the intention of this loop is?

The intention is to ensure that the propagated type is one of the possible types retrieved for that node so as to maintain some logical consistency with what the get_possible_types_from_node function is already doing, and also potentially as a failsafe in some edge case where the propagated type may not be one of the possible types retrieved.

In this approach, we would need to do a type comparison because the definition classes are created as different objects even if they both represent the same base type.

The test suite still passes if we replace this loop with the following statement:

types_list = [propagated_type]

If this is the preferred option, then we can shift this segment for constant typedef propagation to the beginning of the function so that the propagated type is taken to be the default possible type and returned immediately. There would be no need to retrieve the possible types for this node. We can view this function as then differentiating between nodes with constant typedef propagation metadata and nodes without.

An alternative to type comparison in the existing approach is to compare the __repr__() string representations.

vyper/ast/folding.py

@external

previously the constant folding stage would forget the annotated types of user-defined constants, letting the typechecker treat them like normal literals. this would allow code like the following to compile: ``` MY_CONSTANT: constant(uint256) = 1 @external def foo() -> uint8: return MY_CONSTANT ``` this commit propagates the annotated types of user-defined constants so that they get typechecked properly.

tserg added 3 commits January 16, 2022 18:04

propagate typedef in node metadata for folded constants

6551b82

restrict possible node types if typedef is propagated

05af622

update for loop range value validation for constant

f9df360

fix: add test cases for type checking of constants

109a08d

tserg marked this pull request as ready for review January 17, 2022 07:32

fubuloubu requested a review from charles-cooper January 17, 2022 15:50

charles-cooper requested changes Jan 17, 2022

View reviewed changes

tserg added 2 commits January 18, 2022 07:34

replace type string with TypeDefinition objects

420af97

test: refactor distinct integer type parametrization into pytest fixture

63eb93b

fubuloubu reviewed Jan 18, 2022

View reviewed changes

tests/parser/types/test_lists.py Outdated Show resolved Hide resolved

tests/conftest.py Outdated Show resolved Hide resolved

tests/parser/types/numbers/test_constants.py Outdated Show resolved Hide resolved

test: revert parametrization to inpage itertools

a96340c

fubuloubu approved these changes Jan 18, 2022

View reviewed changes

vyper/ast/folding.py Outdated Show resolved Hide resolved

use VyperException for try except in type derivation during constant …

7378218

…folding

fubuloubu approved these changes Jan 19, 2022

View reviewed changes

tserg added 2 commits January 22, 2022 16:21

propagate ast annotation

7b62dea

test: add tests for dynarray

ed869fc

fubuloubu approved these changes Jan 22, 2022

View reviewed changes

charles-cooper reviewed Jan 22, 2022

View reviewed changes

vyper/semantics/validation/utils.py Outdated Show resolved Hide resolved

charles-cooper reviewed Jan 22, 2022

View reviewed changes

vyper/semantics/validation/utils.py Outdated Show resolved Hide resolved

charles-cooper reviewed Jan 22, 2022

View reviewed changes

early termination for propagate type

1ed9d1a

charles-cooper reviewed Feb 7, 2022

View reviewed changes

vyper/ast/folding.py Outdated Show resolved Hide resolved

tserg added 4 commits February 7, 2022 12:20

extract vyper type during constant folding

ba00005

add tests for constant nested lists

5036102

move extraction of typedef to replace_constant helper

758b3a0

move extraction of typedef to user-defined constants

2d4a6eb

charles-cooper approved these changes Feb 8, 2022

View reviewed changes

charles-cooper merged commit a90f645 into vyperlang:master Feb 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: type check for constants #2603

fix: type check for constants #2603

tserg commented Jan 16, 2022 •

edited

Loading

codecov-commenter commented Jan 16, 2022 •

edited

Loading

charles-cooper left a comment

charles-cooper Jan 17, 2022

tserg Jan 17, 2022

charles-cooper Jan 17, 2022

tserg Jan 18, 2022 •

edited

Loading

charles-cooper Jan 19, 2022

tserg commented Jan 18, 2022

lgtm-com bot commented Jan 18, 2022

lgtm-com bot commented Jan 18, 2022

charles-cooper Jan 22, 2022

tserg Jan 22, 2022 •

edited

Loading

tserg Jan 22, 2022

fix: type check for constants #2603

fix: type check for constants #2603

Conversation

tserg commented Jan 16, 2022 • edited Loading

What I did

How I did it

How to verify it

Description for the changelog

Cute Animal Picture

codecov-commenter commented Jan 16, 2022 • edited Loading

Codecov Report

charles-cooper left a comment

Choose a reason for hiding this comment

charles-cooper Jan 17, 2022

Choose a reason for hiding this comment

tserg Jan 17, 2022

Choose a reason for hiding this comment

charles-cooper Jan 17, 2022

Choose a reason for hiding this comment

tserg Jan 18, 2022 • edited Loading

Choose a reason for hiding this comment

charles-cooper Jan 19, 2022

Choose a reason for hiding this comment

tserg commented Jan 18, 2022

lgtm-com bot commented Jan 18, 2022

lgtm-com bot commented Jan 18, 2022

charles-cooper Jan 22, 2022

Choose a reason for hiding this comment

tserg Jan 22, 2022 • edited Loading

Choose a reason for hiding this comment

tserg Jan 22, 2022

Choose a reason for hiding this comment

tserg commented Jan 16, 2022 •

edited

Loading

codecov-commenter commented Jan 16, 2022 •

edited

Loading

tserg Jan 18, 2022 •

edited

Loading

tserg Jan 22, 2022 •

edited

Loading