Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

urllib3 bdist unittest error with nuitka #413

Open
tommyli3318 opened this issue Jun 1, 2019 · 5 comments

Comments

Projects
2 participants
@tommyli3318
Copy link
Member

commented Jun 1, 2019

Platform: Windows, python 3.7.3 x32
Nuitka commit 99b496e
Urllib3 commit 52c827532d8b434e6bd344fad7ee69b9d247792b

Steps to reproduce:

git clone https://github.com/urllib3/urllib3.git
python -m pip install -r dev-requirements.txt
python setup.py bdist_nuitka
python -m pip install dist/urllib3-1.25.3-cp37-cp37m-win32.whl
rm -rf src
python -m pytest --disable-warnings

Segmentation fault in test\test_no_ssl.py, after enabling --debug, this assertion triggers

@kayhayen kayhayen self-assigned this Jun 3, 2019

@kayhayen

This comment has been minimized.

Copy link
Member

commented Jun 3, 2019

Thanks for the report, @tommyli3318

The code in question attempts to load the module from sys.modules after the module body ran. It then wants to set __spec__ on it for Python 3.4or higher and touch up theloaderof modules, so thatpkgutil.itermodules` could work (that code is not yet used). The assumption that that it will always be there, may not hold true though.

I believe this happens in case of an exception. Now to narrow done things further, you have several options in this instance, and one thing to research:

a) When a module throws an exception on the module body, is it in sys.modules afterwards. Try it for Python2 and Python3 and let me know. This will let us know if the assertions is valid. If it is not, we need to change it from an assertion to an if (result != NULL) ... statement rather.

b) What you should also try, is to put --trace-execution into the compilation, that will make it output everything it does. Be prepared for massive output, which is in its entirety useless, but the last lines are typically an indication of what was attempted. I expect an import statement to be the last thing before the assertion triggers.

c) You could try and execute the failing test as a binary, for that of course, you need to call the test_xxx function in that test case. That may or may not work so well, but trims down the test by a lot, esp. for the above approach.

d) And when it's import related, e.g. now we want to know the module that is causing the crash, you can do --python-flag=-v and it will trace its imports. Do not combine with the other tracing, or else you are barely going to see it. You have to read it, and follow it, what I expect to happen might be that a module is imported, then attempts and import that fails, gives an exception, that exits the module, which then crashes things.

@tommyli3318

This comment has been minimized.

Copy link
Member Author

commented Jun 4, 2019

a)

When a module throws an exception on the module body, is it in sys.modules afterwards

I am not following what you mean, can you please elaborate?

b)
--trace-execution goes into nuitka/disutils/bdist_nuitka.py right?
This is the bottommost output

Execute: C:\Users\Tommy\urllib3-issue\urllib3\build\lib\urllib3\packages\six.py:181  'STATEMENT_CONDITIONAL'>                                                                   
Execute: C:\Users\Tommy\urllib3-issue\urllib3\build\lib\urllib3\packages\six.py:183  'STATEMENT_RETURN_NONE'>                                                                   
Execute: C:\Users\Tommy\urllib3-issue\urllib3\build\lib\urllib3\packages\six.py:180  'STATEMENT_RELEASE_VARIABLE' with {'variable':  }>                                  
Execute: C:\Users\Tommy\urllib3-issue\urllib3\build\lib\urllib3\packages\six.py:180  'STATEMENT_RELEASE_VARIABLE' with {'variable':  le'>}>                                    
Execute: C:\Users\Tommy\urllib3-issue\urllib3\build\lib\urllib3\packages\six.py:180  'STATEMENT_RELEASE_VARIABLE' with {'variable':  }>                    
Execute: C:\Users\Tommy\urllib3-issue\urllib3\build\lib\urllib3\packages\six.py:180  'STATEMENT_RETURN'>

There's no tracing output before the segmentation fault though?

d)
Again there doesn't seem to be anything wrong leading up to the segmentation fault

Loading urllib3.util.retry                                                                 
Loaded urllib3.util.retry                                                                  
import urllib3.util.url # considering responsibility                                       
import urllib3.util.url # claimed responsibility (compiled)                                
Loading urllib3.util.url                                                                   
Loaded urllib3.util.url                                                                    
Loaded urllib3.util                                                                        
import urllib3._collections # considering responsibility                                   
import urllib3._collections # claimed responsibility (compiled)                            
Loading urllib3._collections                                                               
Loaded urllib3._collections                                                                
Loaded urllib3.connection                                                                  
import urllib3.request # considering responsibility                                        
import urllib3.request # claimed responsibility (compiled)                                 
Loading urllib3.request                                                                    
import urllib3.filepost # considering responsibility                                       
import urllib3.filepost # claimed responsibility (compiled)                                
Loading urllib3.filepost                                                                   
import urllib3.fields # considering responsibility                                         
import urllib3.fields # claimed responsibility (compiled)                                  
Loading urllib3.fields                                                                     
import mimetypes # considering responsibility                                              
import mimetypes # denied responsibility                                                   
Loaded urllib3.fields                                                                      
Loaded urllib3.filepost                                                                    
import urllib3.packages.six.moves.urllib # considering responsibility                      
import urllib3.packages.six.moves.urllib # denied responsibility                           
import urllib3.packages.six.moves.urllib.parse # considering responsibility                
import urllib3.packages.six.moves.urllib.parse # denied responsibility                     
Loaded urllib3.request                                                                     
import urllib3.response # considering responsibility                                       
import urllib3.response # claimed responsibility (compiled)
@kayhayen

This comment has been minimized.

Copy link
Member

commented Jun 7, 2019

My request was geared at code like this:

try:
  ìmport a
except Exception:
  import sys
  print("a" in sys.modules")
# module code of a
raise TypeError

When a program catches that exception, is it normally in sys.modules with standard Python, or is it not. That is the assumption violated.

I am suspecting, six is doing in that line 180 an import, can you confirm. Maybe of the module that crashes it. Although I am not seeing that. Did you use --debug in your traces? Because if you don't, the error might occur only later.

@tommyli3318

This comment has been minimized.

Copy link
Member Author

commented Jun 8, 2019

When a program catches that exception, is it normally in sys.modules with standard Python, or is it not. That is the assumption violated.

Just tested, both python 2 and 3 returned False for module in sys.modules after an exception in module.

Did you use --debug in your traces? Because if you don't, the error might occur only later.

Using --debug removes all the tracing output for some reason (as in, it crashes with the dialog box without any tracing output, even with --trace-execution enabled).

@kayhayen kayhayen added the develop label Jun 16, 2019

@kayhayen

This comment has been minimized.

Copy link
Member

commented Jun 16, 2019

Based on your finding that it must not be assumed to be there, I changed the C code to not do it. Please retry with current develop code.

@kayhayen kayhayen added this to To do in Planning via automation Jul 1, 2019

@kayhayen kayhayen moved this from To do to Done in Planning Jul 1, 2019

@kayhayen kayhayen added this to the 0.6.5 milestone Jul 15, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.