Skip to content
This repository has been archived by the owner. It is now read-only.

SystemError: Handle is not initialized. in abc.py #1187

Closed
dxtodorovic opened this issue Mar 19, 2015 · 3 comments
Closed

SystemError: Handle is not initialized. in abc.py #1187

dxtodorovic opened this issue Mar 19, 2015 · 3 comments
Labels

Comments

@dxtodorovic
Copy link

@dxtodorovic dxtodorovic commented Mar 19, 2015

I ran into a non-reproducible bug "SystemError: Handle is not initialized". It is very similar to the bug already posted here, it seems to have something to do with the garbage collector.

Stacktrace:

C:\Program Files (x86)\IronPython 2.7\ipy.exe -X:FullFrames -X:ShowClrExceptions -X:ExceptionDetail script_name.py


Traceback (most recent call last):
  ...
  File "D:\Scripts\jenkins.py", line 308, in get_build_definitions_names
    return self._remote.jobs.keys()
  File "D:\Scripts\jenkins.py", line 116, in jobs
    return GJobs(self)
  File "D:\Scripts\lib\jenkinsapi\jobs.py", line 21, in __init__
    self.jenkins.poll()
  File "D:\Scripts\lib\jenkinsapi\jenkinsbase.py", line 56, in poll
    self._data = self._poll()
  File "D:\Scripts\lib\jenkinsapi\jenkinsbase.py", line 62, in _poll
    return self.get_data(url)
  File "D:\Scripts\lib\jenkinsapi\jenkinsbase.py", line 66, in get_data
    response = requester.get_url(url, params)
  File "D:\Scripts\lib\jenkinsapi\utils\requester.py", line 95, in get_url
    return requests.get(self._update_url_scheme(url), **requestKwargs)
  File "D:\Scripts\lib\requests\api.py", line 59, in get
    return request('get', url, **kwargs)
  File "D:\Scripts\lib\requests\api.py", line 48, in request
    return session.request(method=method, url=url, **kwargs)
  File "D:\Scripts\lib\requests\sessions.py", line 437, in request
    prep = self.prepare_request(req)
  File "D:\Scripts\lib\requests\sessions.py", line 363, in prepare_request
    p.prepare(
  File "D:\Scripts\lib\requests\models.py", line 300, in prepare
    self.prepare_auth(auth, url)
  File "D:\Scripts\lib\requests\models.py", line 480, in prepare_auth
    r = auth(self)
  File "D:\Scripts\casauth.py", line 23, in __call__
    res = self._get_cas_auth_page(res, self._session)
  File "D:\Scripts\casauth.py", line 48, in _get_cas_auth_page
    return session.get(redirection_path, verify=False)
  File "D:\Scripts\lib\requests\sessions.py", line 463, in get
    return self.request('GET', url, **kwargs)
  File "D:\Scripts\lib\requests\sessions.py", line 451, in request
    resp = self.send(prep, **send_kwargs)
  File "D:\Scripts\lib\requests\sessions.py", line 583, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "D:\Scripts\lib\requests\sessions.py", line 182, in resolve_redirects
    resp = self.send(
  File "D:\Scripts\lib\requests\sessions.py", line 557, in send
    r = adapter.send(request, **kwargs)
  File "D:\Scripts\lib\requests\adapters.py", line 352, in send
    resp = conn.urlopen(
  File "D:\Scripts\lib\requests\packages\urllib3\connectionpool.py", line 525, in urlopen
    response = HTTPResponse.from_httplib(httplib_response,
  File "D:\Scripts\lib\requests\packages\urllib3\response.py", line 276, in from_httplib
    headers = HTTPHeaderDict()
  File "D:\Scripts\lib\requests\packages\urllib3\_collections.py", line 149, in __init__
    self.update(headers, **kwargs)
  File "C:\Program Files (x86)\IronPython 2.7\Lib\_abcoll.py", line 492, in update
    if isinstance(other, Mapping):
  File "C:\Program Files (x86)\IronPython 2.7\Lib\abc.py", line 132, in __instancecheck__
    if subclass is not None and subclass in cls._abc_cache:
  File "C:\Program Files (x86)\IronPython 2.7\Lib\_weakrefset.py", line 73, in __contains__
    return wr in self.data
SystemError: Handle is not initialized.

I was unable to deduce what causes this particular bug to appear and it seems to happen in a seemingly random fashion.

@andressommerhoff

This comment has been minimized.

Copy link

@andressommerhoff andressommerhoff commented Apr 13, 2015

I manage to reproduce the error by trying the offending code 100.000 times. Below the script (Can someone else reproduce it by running the script?). Please note that I isolate the error to WeakSet.__contains__() of _weakrefset.py file (So the script is clean of all the abc.py stuff). More details about this bug description at:

https://mail.python.org/pipermail/ironpython-users/2015-April/017442.html

Looking forward some IronPython mantainer can debug inside the C# code to tackle the root of the problem!

#
# file: testWeakSetError.py
#

#EXPLANATION ABOUT THE TEST SCRIPT:
#    This test script is somehow based in the way abc.py class handle 
#  a weakSet as a cache for speeding up the ABCMeta.__instancecheck__(),
#  which overrides the buildin isinstance(). The abc.py check the cache by 
#  testing "test=classX in WeakSetObjAsCache", which fails when 
#  the seldom exception is raised inside WeakSet.__contains__().

#    The boolean test is a simple "test = item in WeakSetObject" which performs 
#  as expected 99.9% of the time and the result is True. "item" is a 
#  normal (not weak) reference, so it should not be killed 
#  by gc.collection, and "item" was added to WeakSetObject before. This
#  boolean test is inside a loop that repeat the test 100.000 times. On every
#  cycle of the loop, is it also added a new instance of 
#  a big (RAM) object (just 2kb to 12kb each obj), which will be 
#  stay as a weakref inside the WeakSet. Every 1000 cycles a gc.collection()
#  is manually performed, which should clean some of this big objects. The 
#  error (exception) start to happen around cycle 12.000th (sometimes sooner).
#  The error consist in an untrapped exception of the type:
#       (ValueError) or (SystemError)
#  After 100.000 cycles I count around 10 or 20 errors. For a real python
#  application, that is enough frequent to stop the execution of a medium size
#  optimization problem (Pulp library make intensive use of "isinstance()", 
#  and what buildin function use WeakSet under the hook as a cache.) 

import gc, sys, traceback 
from _weakrefset import WeakSet

#Big Object -> takes 2 to 12kb of RAM!
class BigObjClass(object):
    def __init__(self,id):
        self.useRAM = str(id)*1000 # Less than <100 or bigger than >10000, then no bug shows up (not knowing why?) 

item       = BigObjClass(id="AA")
weakSetObj = WeakSet()
weakSetObj.add(item) # adding an item instance 

errorcount=0
for i in xrange(100000): #<- Do several loops until you get the error
    #Simulating ram intensive python script (it must be inside the loop, otherwise no error shows up)
    t = BigObjClass(id=i) # A new instance, takes 2 to 12kb of RAM!
    weakSetObj.add(t)  #<- Adding it to WeakSet. It is a weakref so the 2 to 12kb of RAM should be retrieved when gc.collect()

    #The boolean Test
    try:
        test = item in weakSetObj  #<- OFENDING CODE: THIS TEST IS WHERE SOMETIMES THE ERROR IS PRODUCED!!!
        assert test==True
    except (SystemError, ValueError) as e: # Exception as e: 
        errorcount += 1
        (extype, exvalue, extraceback) = sys.exc_info()

        print "\n####################################################"
        print "Error in Loop",i,"coming from WeakSet.__contains__:"
        print type(e), e
        #print "------------- TRACEBACK -------------"
        traceback.print_exception(extype,exvalue,extraceback,limit=10,file=sys.stdout)
        print "####################################################\n"

    # Garbage Collection from time to time (necessary to get the error)
    if i%1000==0:  # If too small (i.e. <100) the error is not shown! If too big -> memoryout exception
        howmuch = gc.collect() #should collect the BigObjs created in the loop, but not "item" as it is normal referenced   
        print "loop",i, " doing gc.collect(): ",howmuch

print
print "In total:", errorcount, "errors in 100000 tests = %",  errorcount/100000.0*100      
print "End"        
@amas008

This comment has been minimized.

Copy link

@amas008 amas008 commented Nov 17, 2015

For your information, this WeakSet.contains() error is impacting at least one other user of SolverStudio and PuLP. They see it in Iron Python 2.7.5 but not 2.7.4.1000 for some reason. Andrew

@kunom

This comment has been minimized.

Copy link
Contributor

@kunom kunom commented Sep 10, 2016

I guess, this can now be closed as #1198 is merged.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
7 participants
You can’t perform that action at this time.