Skip to content

Conversation

@alexmojaki
Copy link
Collaborator

For example nameof(self.x) == 'x'. I think this is a pretty intuitive enhancement. The bytecode version already does this.

@alexmojaki
Copy link
Collaborator Author

alexmojaki commented Aug 4, 2020

pylint says Unnecessary "elif" after "return" (no-else-return):

def _node_name(node):
    if isinstance(node, ast.Name):
        return node.id
    elif isinstance(node, ast.Attribute):
        return node.attr
    else:
        raise VarnameRetrievingError(
            f"Can only get name of a variable or attribute, "
            f"not {ast.dump(node)}"
        )

Personally I think it looks nice and consistent when the clauses are short - easy to mentally parse quickly. Should I change it? pylint is pretty fussy, it's brave to put that in CI.

@pwwang
Copy link
Owner

pwwang commented Aug 4, 2020

It is... but just want to keep some uniformed style if there are more PRs coming in.

You can simply ignore the pattern:

def _node_name(node):
    # pylint: disable=no-else-return
    if isinstance(node, ast.Name):
        return node.id
    elif isinstance(node, ast.Attribute): 
        return node.attr
    else: 
        raise VarnameRetrievingError(
            f"Can only get name of a variable or attribute, "
            f"not {ast.dump(node)}"
        )

@alexmojaki
Copy link
Collaborator Author

Do you like the general idea of the PR? I probably need more tests before merging.

@pwwang
Copy link
Owner

pwwang commented Aug 4, 2020

Do you like the general idea of the PR? I probably need more tests before merging.

I love it!

@alexmojaki
Copy link
Collaborator Author

In the process of doing this I've noticed some problems with how nameof works. I see that if executing doesn't give you the actual node you use the statement, and then you use _lookfor_child_nameof to find the actual call. But _lookfor_child_nameof just returns the first call it finds which look vaguely right, and it's easy for it to get it wrong. For example here:

def test_original_nameof():
    x = 1
    y = []
    assert len(y) != original_nameof(x) == 'x'

it finds the call to len(y) first so pytest says AssertionError: assert 'y' == 'x' because original_nameof(x) returns 'y'. And it's not just that _lookfor_child_nameof isn't implemented correctly, it's that this is an incredibly hard problem to solve - I would know!

The reason executing doesn't find the correct node in this case and elsewhere in tests is that pytest modifies the bytecode to make the assertions show nice error messages like the one above. Here's an example of what that looks like under the hood:

def test_bird():
    assert original_nameof(nameof) == 'nameof'
    import dis
    dis.dis(test_bird)

Here's the output from dis:

              0 LOAD_GLOBAL              0 (original_nameof)
              2 LOAD_GLOBAL              1 (nameof)
              4 CALL_FUNCTION            1
              6 STORE_FAST               0 (@py_assert2)
              8 LOAD_CONST               1 ('nameof')
             10 STORE_FAST               1 (@py_assert5)
             12 LOAD_FAST                0 (@py_assert2)
             14 LOAD_FAST                1 (@py_assert5)
             16 COMPARE_OP               2 (==)
             18 STORE_FAST               2 (@py_assert4)
             20 LOAD_FAST                2 (@py_assert4)
             22 LOAD_CONST               0 (None)
             24 COMPARE_OP               8 (is)
             26 POP_JUMP_IF_FALSE       72
             28 LOAD_CONST               2 (0)
             30 LOAD_CONST               3 (('PytestAssertRewriteWarning',))
             32 IMPORT_NAME              2 (_pytest.warning_types)
             34 IMPORT_FROM              3 (PytestAssertRewriteWarning)
             36 STORE_FAST               3 (PytestAssertRewriteWarning)
             38 POP_TOP
             40 LOAD_CONST               2 (0)
             42 LOAD_CONST               4 (('warn_explicit',))
             44 IMPORT_NAME              4 (warnings)
             46 IMPORT_FROM              5 (warn_explicit)
             48 STORE_FAST               4 (warn_explicit)
             50 POP_TOP
             52 LOAD_FAST                4 (warn_explicit)
             54 LOAD_FAST                3 (PytestAssertRewriteWarning)
             56 LOAD_CONST               5 ('asserting the value None, please use "assert is None"')
             58 CALL_FUNCTION            1
             60 LOAD_CONST               0 (None)
             62 LOAD_CONST               6 ('/Users/alexhall/Desktop/python/python-varname/tests/test_varname.py')
             64 LOAD_CONST               7 (39)
             66 LOAD_CONST               8 (('category', 'filename', 'lineno'))
             68 CALL_FUNCTION_KW         4
             70 POP_TOP
        >>   72 LOAD_FAST                2 (@py_assert4)
             74 POP_JUMP_IF_TRUE       214
             76 LOAD_GLOBAL              6 (@pytest_ar)
             78 LOAD_METHOD              7 (_call_reprcompare)
             80 LOAD_CONST               9 (('==',))
             82 LOAD_FAST                2 (@py_assert4)
             84 BUILD_TUPLE              1
             86 LOAD_CONST              10 (('%(py3)s\n{%(py3)s = %(py0)s(%(py1)s)\n} == %(py6)s',))
             88 LOAD_FAST                0 (@py_assert2)
             90 LOAD_FAST                1 (@py_assert5)
             92 BUILD_TUPLE              2
             94 CALL_METHOD              4
             96 LOAD_CONST              11 ('original_nameof')
             98 LOAD_GLOBAL              8 (@py_builtins)
            100 LOAD_METHOD              9 (locals)
            102 CALL_METHOD              0
            104 COMPARE_OP               6 (in)
            106 POP_JUMP_IF_TRUE       118
            108 LOAD_GLOBAL              6 (@pytest_ar)
            110 LOAD_METHOD             10 (_should_repr_global_name)
            112 LOAD_GLOBAL              0 (original_nameof)
            114 CALL_METHOD              1
            116 POP_JUMP_IF_FALSE      128
        >>  118 LOAD_GLOBAL              6 (@pytest_ar)
            120 LOAD_METHOD             11 (_saferepr)
            122 LOAD_GLOBAL              0 (original_nameof)
            124 CALL_METHOD              1
            126 JUMP_FORWARD             2 (to 130)
        >>  128 LOAD_CONST              11 ('original_nameof')
        >>  130 LOAD_CONST               1 ('nameof')
            132 LOAD_GLOBAL              8 (@py_builtins)
            134 LOAD_METHOD              9 (locals)
            136 CALL_METHOD              0
            138 COMPARE_OP               6 (in)
            140 POP_JUMP_IF_TRUE       152
            142 LOAD_GLOBAL              6 (@pytest_ar)
            144 LOAD_METHOD             10 (_should_repr_global_name)
            146 LOAD_GLOBAL              1 (nameof)
            148 CALL_METHOD              1
            150 POP_JUMP_IF_FALSE      162
        >>  152 LOAD_GLOBAL              6 (@pytest_ar)
            154 LOAD_METHOD             11 (_saferepr)
            156 LOAD_GLOBAL              1 (nameof)
            158 CALL_METHOD              1
            160 JUMP_FORWARD             2 (to 164)
        >>  162 LOAD_CONST               1 ('nameof')
        >>  164 LOAD_GLOBAL              6 (@pytest_ar)
            166 LOAD_METHOD             11 (_saferepr)
            168 LOAD_FAST                0 (@py_assert2)
            170 CALL_METHOD              1
            172 LOAD_GLOBAL              6 (@pytest_ar)
            174 LOAD_METHOD             11 (_saferepr)
            176 LOAD_FAST                1 (@py_assert5)
            178 CALL_METHOD              1
            180 LOAD_CONST              12 (('py0', 'py1', 'py3', 'py6'))
            182 BUILD_CONST_KEY_MAP      4
            184 BINARY_MODULO
            186 STORE_FAST               5 (@py_format7)
            188 LOAD_CONST              13 ('assert %(py8)s')
            190 LOAD_CONST              14 ('py8')
            192 LOAD_FAST                5 (@py_format7)
            194 BUILD_MAP                1
            196 BINARY_MODULO
            198 STORE_FAST               6 (@py_format9)
            200 LOAD_GLOBAL             12 (AssertionError)
            202 LOAD_GLOBAL              6 (@pytest_ar)
            204 LOAD_METHOD             13 (_format_explanation)
            206 LOAD_FAST                6 (@py_format9)
            208 CALL_METHOD              1
            210 CALL_FUNCTION            1
            212 RAISE_VARARGS            1
        >>  214 LOAD_CONST               0 (None)
            216 DUP_TOP
            218 STORE_FAST               0 (@py_assert2)
            220 DUP_TOP
            222 STORE_FAST               2 (@py_assert4)
            224 STORE_FAST               1 (@py_assert5)

 40         226 LOAD_CONST               2 (0)
            228 LOAD_CONST               0 (None)
            230 IMPORT_NAME             14 (dis)
            232 STORE_FAST               7 (dis)

 41         234 LOAD_FAST                7 (dis)
            236 LOAD_METHOD             14 (dis)
            238 LOAD_GLOBAL             15 (test_bird)
            240 CALL_METHOD              1
            242 POP_TOP
            244 LOAD_CONST               0 (None)
            246 RETURN_VALUE

Obviously it's very complicated and I don't know what exactly it's doing but for example at the beginning it calculates original_nameof(nameof) as normal and then stores that in a variable called @py_assert2 which it can retrieve to show a nice error message if the assertion fails.

executing sees that the bytecode doesn't look the way it expects and returns None as the node. In general it doesn't support nodes inside pytest assertions.

The bytecode version is also affected by this. assert _bytecode_nameof(lam.a) == "a" fails because _bytecode_nameof(lam.a) returns @py_assert2. I've only observed this problem for attributes, I guess because pytest doesn't need to make extra variables just to store a variable, but I can't guarantee anything. Besides, pytest isn't the only case to consider.

I suggest that:

  • If executing finds the source code but can't find the exact node, trust that it knows what it's doing. Something is fishy. Don't try to find the node yourself from just a statement.
  • Only use the bytecode version of nameof when there's no source code.
  • Check that the result is a valid variable name in normal Python.
  • Document that the library is not perfect and won't work well in the presence of other magic such as pytest.
  • Use unittest.TestCase.assertEqual etc. when needed. pytest assertions will break nameof.

@pwwang
Copy link
Owner

pwwang commented Aug 4, 2020

Good catch. I am wondering if we can do a strict check for such a situation and make it work. For example, we tell people that it doesn't work if there are more than 2 items to compare (len(y) != original_nameof(x) == 'x'). But for two-item comparison, we can still make it work as normal. Currently, in my mind, I am not sure about the reliability of the latter case. We probably have to look at python's grammar file to enumerate all the cases.

@pwwang
Copy link
Owner

pwwang commented Aug 4, 2020

Well, even for 2-item comparisons, I am guessing we are still failing this:

assert nameof(a) != nameof(b)

with pytest.

I have added a test, which proves my guessing:

    def test_both():
        a = 1
        b = 1
>       assert varname_module.nameof(a) != varname_module.nameof(b)

tests/test_varname.py:479:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (1,), result = 'a', @py_assert3 = 2, @py_assert5 = 'b'

    def nameof(*args):
        """Test both implementations at the same time"""
        result = original_nameof(*args, caller=2)
        if len(args) == 1:
>           assert result == _bytecode_nameof(caller=2)
E           AssertionError: assert 'a' == 'b'
E             - b
E             + a

tests/test_varname.py:22: AssertionError

Failing on original_nameof as well:

    def test_both():
        a = 1
        b = 1
>       assert original_nameof(a) != original_nameof(b)
E       AssertionError: assert 'a' != 'a'
E        +  where 'a' = original_nameof(1)
E        +  and   'a' = original_nameof(1)

tests/test_varname.py:479: AssertionError

@pwwang
Copy link
Owner

pwwang commented Aug 4, 2020

I agree with all your suggestions. Even though we could make it work somehow when there is one nameof called in the statement or expression, I guess it doesn't worth it to support just this special case. What do you think? Is it worth supporting if there is only one nameof (I guess it probably has to be one general function) called?

@alexmojaki
Copy link
Collaborator Author

alexmojaki commented Aug 4, 2020

It's quite easy to check that there's exactly one ast.Call in a statement - you look for all nodes using ast.walk. And it would work unless the user did something really weird like __add__ = nameof.

My worry is users being confused by code working fine one moment and then not working when they make a change that seems innocent. I think it's better to just find out as soon as possible that nameof will not work under certain conditions. When executing finds the source but not the node, show a message like "Couldn't retrieve the call node. This may happen if you're using some other AST magic at the same time, such as pytest, ipython, macropy, or birdseye".

@pwwang
Copy link
Owner

pwwang commented Aug 4, 2020

We should just probably drop support for such a case when executing not able to retrieve the exact node. It's confusing when this works assert nameof(a) == 'a' and this doesn't: assert nameof(a) != nameof(b). And it might be even more confusing to explain this in the exception message.

@alexmojaki
Copy link
Collaborator Author

Yes, it sounds like we agree.

@pwwang
Copy link
Owner

pwwang commented Aug 4, 2020

Yes. Do you want to do another PR to drop the support first, and then switch back to this one?

@alexmojaki
Copy link
Collaborator Author

I think I'll make another PR first.

@alexmojaki alexmojaki requested a review from pwwang August 11, 2020 20:14
@pwwang pwwang merged commit 529a90b into master Aug 11, 2020
@pwwang pwwang deleted the attributes branch August 13, 2020 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants