Codemod crashes on my code

[bjackman]

If I checkout git@github.com:bjackman/git-dissect.git at commit c5ab19c8097ea22 (if you do this and expose your mind to the code in there, please forgive me)

And run hypothesis codemod ., my fans spin for a while (not sure how long, maybe a minute or 2?), then I get this

multiprocessing.pool.RemoteTraceback:                                                                                                                                                                                                                                              
"""                                                                                                                                                                                                                                                                                
Traceback (most recent call last):                                                                                                                                                                                                                                                 
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker                                                                                                                                                                                                          
    result = (True, func(*args, **kwds))                                                                                                                                                                                                                                           
  File "/home/brendan/src/git-dissect/.venv/lib/python3.10/site-packages/hypothesis/extra/cli.py", line 169, in _refactor                                                                                                                                                          
    newcode = func(oldcode)                                                                                                                                                                                                                                                        
  File "/home/brendan/src/git-dissect/.venv/lib/python3.10/site-packages/hypothesis/extra/codemods.py", line 67, in refactor                                                                                                                                                       
    mod = cst.parse_module(code)                                                                                                                                                                                                                                                   
  File "/home/brendan/src/git-dissect/.venv/lib/python3.10/site-packages/libcst/_parser/entrypoints.py", line 109, in parse_module                                                                                                                                                 
    result = _parse(                                                                                                                                                                                                                                                               
  File "/home/brendan/src/git-dissect/.venv/lib/python3.10/site-packages/libcst/_parser/entrypoints.py", line 55, in _parse                                                                                                                                                        
    return parse(source_str)                                                                                                                                                                                                                                                       
libcst._exceptions.ParserSyntaxError: Syntax Error @ 2:1.                                                                                                                                                                                                                          
parser error: error at 2:8: expected one of ,, NAME                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                   
def f(*): pass                                                                                                                                                                                                                                                                     
^                                                                                                                                                                                                                                                                                  
"""                                                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                   
The above exception was the direct cause of the following exception:                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                                                                                 
  File "/home/brendan/src/git-dissect/.venv/bin/hypothesis", line 8, in <module>                                                                                                                                                                                                   
    sys.exit(main())                                                                                                                                                                                                                                                               
  File "/home/brendan/src/git-dissect/.venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__                                                                                                                                                                    
    return self.main(*args, **kwargs)                                                                                                                                                                                                                                              
  File "/home/brendan/src/git-dissect/.venv/lib/python3.10/site-packages/click/core.py", line 1078, in main                                                                                                                                                                        
    rv = self.invoke(ctx)                                                                                                                                                                                                                                                          
  File "/home/brendan/src/git-dissect/.venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke                                                                                                                                                                      
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                                                                        
  File "/home/brendan/src/git-dissect/.venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke                                                                                                                                                                      
    return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                                                                 
  File "/home/brendan/src/git-dissect/.venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke                                                                                                                                                                       
    return __callback(*args, **kwargs)                                                                                                                                                                                                                                             
  File "/home/brendan/src/git-dissect/.venv/lib/python3.10/site-packages/hypothesis/extra/cli.py", line 215, in codemod                                                                                                                                                            
    for msg in pool.imap_unordered(                                                                                                                                                                                                                                                
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 873, in next                                                                                                                                                                                                            
    raise value                                                                                                                                                                                                                                                                    
libcst._exceptions.ParserSyntaxError: Syntax Error @ 2:1.                                                                                                                                                                                                                          
parser error: error at 2:8: expected one of ,, NAME                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                   
def f(*): pass                                                                                                                                                                                                                                                                     
^           

This is not a problem for me, I was only running that command out of pure curiosity! This is just an FYI, I guess Something Is Wrong.

---

Here are the relevant versions of stuff (I am using venv). I was surprised there is no hypothesis[cli] in the pip freeze output. I assume the [cli] thing is some Pip magic that I am ignorant of.

$ pip freeze
attrs==23.1.0
black==23.9.1
click==8.1.7
exceptiongroup==1.1.3
hypothesis==6.87.1
importlab==0.8
iniconfig==2.0.0
Jinja2==3.1.2
libcst==1.0.1
markdown-it-py==3.0.0
MarkupSafe==2.1.3
mdurl==0.1.2
mypy==1.5.1
mypy-extensions==1.0.0
networkx==3.1
ninja==1.11.1
packaging==23.2
pathspec==0.11.2
platformdirs==3.10.0
pluggy==1.3.0
pycnite==2023.9.18
pydot==1.4.2
Pygments==2.16.1
pyparsing==3.1.1
pytest==7.4.2
pytype==2023.9.27
PyYAML==6.0.1
rich==13.6.0
sortedcontainers==2.4.0
tabulate==0.9.0
toml==0.10.2
tomli==2.0.1
typing-inspect==0.9.0
typing_extensions==4.8.0

$ python --version
Python 3.10.12

$ cat /etc/issue
Ubuntu 22.04.3 LTS \n \l

I hope this is useful/interesting!

[Zac-HD]

I think your problem is simply that def f(*): pass is a SyntaxError, i.e. not valid Python at all.

I guess we could do better by warning on files which are invalid syntax. It might also be a nice performance upgrade to skip the processing if "hypothesis" not in source.

[bjackman]

Aha, it had not occured to me to check, I assumed that def f(*): pass was something Hypothesis generated! Next time I'm back at my PC I'll have to double check, but you are probably right.

[bjackman]

Actually no, the def f(*): pass is not in my code:

$ git grep 'def f'
git_dissect.py:    def from_string(cls, s):
git_dissect_test.py:      def f(node):

[Zac-HD]

Huh. Can you provide an input file which triggers this? It sounds like we have a real bug somewhere!

[bjackman]

Yep you can check out the repo and commit I listed on the original report. I just ran it on the whole directory but there are only a couple of files in it.

…

On Mon, 2 Oct 2023, 23:15 Zac Hatfield-Dodds, ***@***.***> wrote: Huh. Can you provide an input file which triggers this? It sounds like we have a real big somewhere! — Reply to this email directly, view it on GitHub <#3759 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAWNWKZ2GLGXFO3UXBVLJZLX5MVHFAVCNFSM6AAAAAA5OREEWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBTG43TONJRGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[bjackman]

Hey Zac did you intend to close the issue? I think that PR was originally marked as fixing it but then IIUC you changed your mind in your previous comment.

[Zac-HD]

Oops, yeah - the PR turns this crash into a warning so we can process other files, but doesn't fully fix it.

[Zac-HD]

Unfortunately I still can't reproduce this. I've tried running codemods on the files, on the whole directory, from the Python interface, etc., and it plainly works.

Combined with your report that it takes several minutes for this to trigger, I suspect that you have something like a symlink involved, which is causing the codemods to be applied to a much larger set of files than you expected. If we can find a reproducing file which triggers this issue I'll be happy to reopen again, but I think it really is fixed.