ParserSyntaxError when a module have a "from django.utils import timezone" import #421

luzfcb · 2021-06-14T14:51:44Z

Describe the bug

I'm migrating a medium-sized codebase, and I found this strange behavior: if the code has a "from django.utils import timezone" import, a ParserSyntaxError is raised

$ pre-commit run -a
djcodemod................................................................Failed
- hook id: djcodemod
- exit code: 1

Running codemods: AssignmentTagTransformer, CookieDateTransformer, FixedOffsetTransformer, FloatRangeFormFieldTransformer, FloatRangeModelFieldTransformer, InlineHasAddPermissionsTransformer, ModelsPermalinkTransformer, OnDeleteTransformer, QuerySetPaginatorTransformer, SignalDisconnectWeakTransformer, URLResolversTransformer
Executing codemod...
Finished codemodding 62 files!
 - Transformed 62 files successfully.
 - Skipped 0 files.
 - Failed to codemod 0 files.
 - 0 warnings were generated.
Running codemods: AssignmentTagTransformer, CookieDateTransformer, FixedOffsetTransformer, FloatRangeFormFieldTransformer, FloatRangeModelFieldTransformer, InlineHasAddPermissionsTransformer, ModelsPermalinkTransformer, OnDeleteTransformer, QuerySetPaginatorTransformer, SignalDisconnectWeakTransformer, URLResolversTransformer
Executing codemod...
Codemodding /home/luzfcb/projects/big_project/foobar/tests/test_managers.py
Traceback (most recent call last):
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/base_parser.py", line 152, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: ReservedString(import)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_cli.py", line 295, in _execute_transform
    output_tree = transformer.transform_module(input_tree)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_command.py", line 88, in transform_module
    tree = self._instantiate_and_run(transform, tree)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_command.py", line 58, in _instantiate_and_run
    return inst.transform_module(tree)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_codemod.py", line 108, in transform_module
    return self.transform_module_impl(tree_with_metadata)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/_visitor.py", line 32, in transform_module_impl
    return tree.visit(self)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_nodes/module.py", line 91, in visit
    result = super(Module, self).visit(visitor)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_nodes/base.py", line 235, in visit
    leave_result = visitor.on_leave(self, with_updated_children)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/matchers/_visitors.py", line 512, in on_leave
    retval = CSTTransformer.on_leave(self, original_node, updated_node)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_visitors.py", line 72, in on_leave
    updated_node = leave_func(original_node, updated_node)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/visitors/_add_imports.py", line 393, in leave_Module
    *[
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/codemod/visitors/_add_imports.py", line 394, in <listcomp>
    parse_statement(
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/entrypoints.py", line 100, in parse_statement
    result = _parse(
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/entrypoints.py", line 51, in _parse
    result = parser.parse()
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/base_parser.py", line 111, in parse
    self._add_token(token)
  File "/home/luzfcb/.cache/pre-commit/repof_bg4wfe/py_env-python3/lib/python3.9/site-packages/libcst/_parser/base_parser.py", line 187, in _add_token
    raise ParserSyntaxError(
libcst._exceptions.ParserSyntaxError: Syntax Error @ 1:7.
Incomplete input. Encountered 'import', but expected '.', '...', or 'NAME'.

from  import datetime
      ^

Failed to codemod /home/luzfcb/projects/big_project/foobar/tests/test_managers.py

To Reproduce

pipx install "django-codemod==1.5.5"

echo "from django.utils import timezone" > foobar.py

djcodemod run --deprecated-in 1.9 --deprecated-in 1.10 --deprecated-in 1.11 --deprecated-in 2.0 --deprecated-in 2.1 --deprecated-in 2.2 foobar.py

Additional context

The text was updated successfully, but these errors were encountered:

browniebroke · 2021-06-14T21:40:29Z

Hi @luzfcb! Thanks for the report. That's a stange one, I cannot reproduce with the CLI installed in my development environment for the project, however, I'm able to do so if I install django-codemod with pipx, like you do.

I've removed a few options you've used and I tracked down the bug to --deprecated-in 2.2, and then to the FixedOffsetTransformer codemodder. You'd get the bug with this:

djcodemod run --codemod FixedOffsetTransformer foobar.py

I'll take a look, but in the meantime you can avoid passing --deprecated-in 2.2...

Ref #421

jefftriplett · 2021-06-16T13:51:45Z

I came here to report this bug too. Using the latest version:

❯ python --version
Python 3.7.10

❯ django-admin --version
3.0.14

❯ djcodemod run --removed-in=3.1 ./blog/models.py
Running codemods: FixedOffsetTransformer, FloatRangeFormFieldTransformer, FloatRangeModelFieldTransformer, QuerySetPaginatorTransformer
Executing codemod...
Codemodding /Users/jefftriplett/.virtualenvs/trailhawks.com/src/trailhawks.com-git/blog/models.py
Traceback (most recent call last):
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_parser/base_parser.py", line 152, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: ReservedString(import)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/codemod/_cli.py", line 295, in _execute_transform
    output_tree = transformer.transform_module(input_tree)
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/codemod/_command.py", line 88, in transform_module
    tree = self._instantiate_and_run(transform, tree)
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/codemod/_command.py", line 58, in _instantiate_and_run
    return inst.transform_module(tree)
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/codemod/_codemod.py", line 108, in transform_module
    return self.transform_module_impl(tree_with_metadata)
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/codemod/_visitor.py", line 32, in transform_module_impl
    return tree.visit(self)
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_nodes/module.py", line 91, in visit
    result = super(Module, self).visit(visitor)
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_nodes/base.py", line 235, in visit
    leave_result = visitor.on_leave(self, with_updated_children)
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/matchers/_visitors.py", line 512, in on_leave
    retval = CSTTransformer.on_leave(self, original_node, updated_node)
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_visitors.py", line 72, in on_leave
    updated_node = leave_func(original_node, updated_node)
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/codemod/visitors/_add_imports.py", line 404, in leave_Module
    for module, aliases in module_and_alias_mapping.items()
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/codemod/visitors/_add_imports.py", line 405, in <listcomp>
    if module != "__future__"
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_parser/entrypoints.py", line 105, in parse_statement
    detect_default_newline=False,
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_parser/entrypoints.py", line 51, in _parse
    result = parser.parse()
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_parser/base_parser.py", line 111, in parse
    self._add_token(token)
  File "/Users/jefftriplett/.virtualenvs/trailhawks.com/lib/python3.7/site-packages/libcst/_parser/base_parser.py", line 191, in _add_token
    raw_column=token.start_pos[1],
libcst._exceptions.ParserSyntaxError: Syntax Error @ 1:7.
Incomplete input. Encountered 'import', but expected '.', '...', or 'NAME'.

from  import datetime
      ^

Failed to codemod /Users/jefftriplett/.virtualenvs/trailhawks.com/src/trailhawks.com-git/blog/models.py

If it's helpful, I have a public project you can reproduce it on: https://github.com/trailhawks/trailhawks.com

browniebroke · 2021-06-16T17:27:25Z

Yes I think I've nailed down the problem to this section, it's just wrong (it replaces the import for the parent module, even if the thing being codemodded isn't used in the file):

django-codemod/django_codemod/visitors/base.py

Lines 167 to 175 in 7942036

    
           if self.old_parent_module_parts != self.new_parent_module_parts: 
        
               # import statement needs updating 
        
               AddImportsVisitor.add_needed_import( 
        
                   context=self.context, 
        
                   module=".".join(self.new_parent_module_parts), 
        
                   obj=self.new_parent_name, 
        
                   asname=import_alias.evaluated_alias, 
        
               ) 
        
               continue

I'll probably disable this feature and push a new release (hopefully tomorrow) until I can make a fix, this is probably breaking a lot of other things.

browniebroke · 2021-06-17T08:43:11Z

I think the issue can be avoided by using django-codemod<1.4 since it was most likely introduced in #347

As per #421, this is not working currently, so I'm putting this feature behind an experimental feature flag until it's fixed properly.

browniebroke · 2021-06-17T15:36:14Z

The problem should no longer happen in 1.5.6, I've disabled the problematic feature for now. I'll try to fix it properly later, not really have time at the moment...

luzfcb added the bug Something isn't working label Jun 14, 2021

luzfcb changed the title ~~ParserSyntaxError when the module have a "from django.utils import timezone" import~~ ParserSyntaxError when a module have a "from django.utils import timezone" import Jun 14, 2021

browniebroke pushed a commit that referenced this issue Jun 14, 2021

test: test case reproducing bug with FixedOffsetTransformer

6805683

Ref #421

browniebroke added a commit that referenced this issue Jun 14, 2021

test: test case reproducing bug with FixedOffsetTransformer

a4cc90e

Ref #421

browniebroke mentioned this issue Jun 14, 2021

Fix ParserSyntaxError when a module has from django.utils import timezone #422

Merged

browniebroke added a commit that referenced this issue Jun 14, 2021

test: test case reproducing bug with FixedOffsetTransformer

1b480c1

Ref #421

browniebroke added a commit that referenced this issue Jun 17, 2021

fix: disable rename parent module imported

1f93024

As per #421, this is not working currently, so I'm putting this feature behind an experimental feature flag until it's fixed properly.

browniebroke closed this as completed in #422 Jun 17, 2021

browniebroke added a commit that referenced this issue Jun 17, 2021

fix: disable rename parent module imported

b717fbb

As per #421, this is not working currently, so I'm putting this feature behind an experimental feature flag until it's fixed properly.

browniebroke mentioned this issue Jun 27, 2021

fix: rewrite parent module renaming logic #429

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ParserSyntaxError when a module have a "from django.utils import timezone" import #421

ParserSyntaxError when a module have a "from django.utils import timezone" import #421

luzfcb commented Jun 14, 2021 •

edited

browniebroke commented Jun 14, 2021

jefftriplett commented Jun 16, 2021 •

edited

browniebroke commented Jun 16, 2021 •

edited

browniebroke commented Jun 17, 2021

browniebroke commented Jun 17, 2021

ParserSyntaxError when a module have a "from django.utils import timezone" import #421

ParserSyntaxError when a module have a "from django.utils import timezone" import #421

Comments

luzfcb commented Jun 14, 2021 • edited

browniebroke commented Jun 14, 2021

jefftriplett commented Jun 16, 2021 • edited

browniebroke commented Jun 16, 2021 • edited

browniebroke commented Jun 17, 2021

browniebroke commented Jun 17, 2021

luzfcb commented Jun 14, 2021 •

edited

jefftriplett commented Jun 16, 2021 •

edited

browniebroke commented Jun 16, 2021 •

edited