Wrong token range given for list with trailing comma

[LiraNuna]

Hello,

Great library! I've been using it to implement this pre-commit hook that ensures trailing commas exist.

Here is a simple repro case to illustrate an issue I'm facing:

import ast, asttokens


class TestcaseVisitor(ast.NodeVisitor):
    def __init__(self, atok):
        self.atok = atok

    def visit_List(self, node):
        super().generic_visit(node)

        tokens = self.atok.token_range(node.first_token, node.last_token, include_extra=True)
        for token in tokens:
            print(token)


atok = asttokens.ASTTokens("""
[
    1,
]
""", parse=True)
visitor = TestcaseVisitor(atok)
visitor.visit(atok.tree)

This will output

OP:'['
NL:'\n'
NUMBER:'1'

Where the expected result should be

OP:'['
NL:'\n'
NUMBER:'1'
OP:','
NL:'\n'
OP:']'

Please note that removing the trailing comma will not reproduce the bug.
This issue is not limited to a single param - as long as there's a trailing comma. The following will produce similar a issue:

[
    1,
    2,
]

[
    1,
    2,
    3,
]

and so on

[LiraNuna]

It seems like the problem is not limited to just List nodes but to Set and Dict nodes as well.

{
    1,
}

will also reproduce this issue. However, Tuple will not. This code will not reproduce the issue:

(
    1,
)

[dsagal]

Thank you for finding the bug and collecting all the details. I'll take a look.

[dsagal]

Could you check out branch 'trailing_commas' to verify if it fixes the problem you found? I think it should also resolve Issue #2.

[LiraNuna]

The branch seems to fix it! Thanks for the quick resolution and for the great library.

[dsagal]

Excellent! I'll close this issue then.