Device or resource busy #3440

xoriole · 2018-02-05T16:07:32Z

From reporter (Tribler 7.0.0)

crash occured shortly after displaying first channels after first startup

Traceback (most recent call last):
File "/usr/share/tribler/TriblerGUI/event_request_manager.py", line 99, in on_read_data
raise RuntimeError(json_dict["event"]["text"])
RuntimeError: [Failure instance: Traceback: <class 'socket.error'>: [Errno 16] Device or resource busy
/usr/lib/python2.7/threading.py:801:__bootstrap_inner
/usr/lib/python2.7/threading.py:754:run
/usr/lib/python2.7/site-packages/twisted/_threads/_threadworker.py:46:work
/usr/lib/python2.7/site-packages/twisted/_threads/_team.py:190:doWork
--- <exception caught here> ---
/usr/lib/python2.7/site-packages/twisted/python/threadpool.py:250:inContext
/usr/lib/python2.7/site-packages/twisted/python/threadpool.py:266:<lambda>
/usr/lib/python2.7/site-packages/twisted/python/context.py:122:callWithContext
/usr/lib/python2.7/site-packages/twisted/python/context.py:85:callWithContext
/usr/lib/python2.7/site-packages/twisted/internet/_resolver.py:124:get
]

non-Jedi · 2018-05-02T22:17:16Z

Per conversation on NixOS/nixpkgs#36065, it seems this bug was introduced some time between 7.0.0 rc3 and the 7.0.0 release. I was going to take a look and see if I could find anything, but there are quite a few commits there. Might want to control your rc process a little better.

qstokkink · 2018-06-22T10:31:27Z

The error seems to be here:

https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/_resolver.py#L123

My best guess is that this is the line that would cause this:

tribler/Tribler/Core/TorrentChecker/session.py

Line 523 in a088d01

    
           self.ip_resolve_deferred = self.register_task("resolve", reactor.resolve(self._tracker_address[0]))

devos50 · 2018-06-22T10:51:28Z

@non-Jedi thank you for your report. We are currently in the process of reproducing and fixing this bug within our environment. We will keep you updated!

qstokkink · 2018-06-26T08:51:26Z

Can someone with NixOS send me the output of this script?

import urllib2, socket, urlparse
from twisted.internet import defer, reactor, threads


def error(failure):
    print str(failure)


def run():
    urls = ["https://www.google.com/", "http://www.python.org/", "https://github.com/"]
    try:
        deferreds = []
        for url in (urls):
            deferreds.append(reactor.resolve(url))
            deferreds[-1].addErrback(error)
        threads.blockingCallFromThread(reactor, lambda: defer.DeferredList(deferreds))
        print "Finished Twisted resolution"
    except:
        import traceback
        traceback.print_exc()
        print "Twisted reactor.resolve failed"

    try:
        for url in (urls):
            socket.gethostbyname(urlparse.urlparse(urllib2.urlopen(url).geturl()).hostname)
        print "Finished socket resolution"
    except:
        import traceback
        traceback.print_exc()
        print "Socket gethostbyname failed"
        
    reactor.callFromThread(reactor.stop)

reactor.callInThread(run)
reactor.run()

non-Jedi · 2018-06-26T12:00:49Z

@xvapx I'm not sure how to make a python environment with nix that looks like the one tribler sees. Maybe you have a better idea.

xvapx · 2018-06-26T21:11:13Z

@non-Jedi Yes, nixos generates a wrapper for tribler which exports all the needed environment variables and paths, you can actually run cat $(which tribler) to see the entire environment in which tribler will run.

@qstokkink output:

[Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.DNSLookupError'>: DNS lookup failed: https://github.com/.
]
[Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.DNSLookupError'>: DNS lookup failed: https://www.google.com/.
]
[Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.DNSLookupError'>: DNS lookup failed: http://www.python.org/.
]
Finished Twisted resolution
Finished socket resolution

Is there any more test more I can do to help?

qstokkink · 2018-06-27T07:59:49Z

@xvapx Thanks for testing: this apparently is not the issue making Tribler crash on Nix though.

One side-effect of testing this, is that we discovered that reactor.resolve is pretty bad at what it does.

xvapx · 2018-06-27T08:10:22Z

@qstokkink No problem, I'm testing 7.1.0-exp2 to see if something has changed on nixos, I'll report back.

xvapx · 2018-06-28T07:37:54Z

I'm working on it, having to fix some dependencies in nixpkgs.

xvapx · 2018-06-28T09:54:38Z

@qstokkink @non-Jedi
7.1.0-exp2 is no longer crashing in nixos, I'm testing that it actually works and will PR in nixpkgs.

EDIT: Lol, after about 10 minutes running, I just wrote this comment and tribler crashed with [Errno 16] Device or resource busy

xvapx · 2018-06-28T09:57:18Z

crash log:

Traceback (most recent call last):
  File "/home/xvapx/tmp/tribler/tmp/tribler/TriblerGUI/event_request_manager.py", line 127, in on_read_data
    raise RuntimeError(json_dict["event"]["text"])
RuntimeError: [Failure instance: Traceback: <class 'socket.error'>: [Errno 16] Device or resource busy
/nix/store/d3aa78jf7pm9lk6ad67x4f21a068dxn6-python-2.7.15/lib/python2.7/threading.py:801:__bootstrap_inner
/nix/store/d3aa78jf7pm9lk6ad67x4f21a068dxn6-python-2.7.15/lib/python2.7/threading.py:754:run
/nix/store/hfq5vjpvsx9j38jc30fkg0r095m4lq3r-python2.7-Twisted-18.4.0/lib/python2.7/site-packages/twisted/_threads/_threadworker.py:46:work
/nix/store/hfq5vjpvsx9j38jc30fkg0r095m4lq3r-python2.7-Twisted-18.4.0/lib/python2.7/site-packages/twisted/_threads/_team.py:190:doWork
--- <exception caught here> ---
/nix/store/hfq5vjpvsx9j38jc30fkg0r095m4lq3r-python2.7-Twisted-18.4.0/lib/python2.7/site-packages/twisted/python/threadpool.py:250:inContext
/nix/store/hfq5vjpvsx9j38jc30fkg0r095m4lq3r-python2.7-Twisted-18.4.0/lib/python2.7/site-packages/twisted/python/threadpool.py:266:<lambda>
/nix/store/hfq5vjpvsx9j38jc30fkg0r095m4lq3r-python2.7-Twisted-18.4.0/lib/python2.7/site-packages/twisted/python/context.py:122:callWithContext
/nix/store/hfq5vjpvsx9j38jc30fkg0r095m4lq3r-python2.7-Twisted-18.4.0/lib/python2.7/site-packages/twisted/python/context.py:85:callWithContext
/nix/store/hfq5vjpvsx9j38jc30fkg0r095m4lq3r-python2.7-Twisted-18.4.0/lib/python2.7/site-packages/twisted/internet/_resolver.py:124:get
]

qstokkink · 2018-06-28T10:18:23Z

It seems the error is with Twisted: https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/_resolver.py#L123 seems to only catch gaierror and not a socket.error.

If we fix this on our end, it would be a nasty patch.

qstokkink · 2018-06-28T10:27:10Z

We appear to already have a nasty workaround method which we can add to: https://github.com/Tribler/tribler/blob/next/Tribler/Core/Session.py#L149 .

qstokkink · 2018-06-28T11:47:11Z

This should be fixed now. We'll keep you posted on the build completion.

qstokkink · 2018-06-28T14:44:41Z

Actually, release candidate 3 should be out tomorrow evening.

xvapx · 2018-06-28T19:33:55Z

Great, I'll test it ASAP, thanks a lot!

xvapx · 2018-06-29T18:49:23Z

@qstokkink
Sorry, bad news:

Traceback (most recent call last):
  File "/home/xvapx/tmp/tribler/tmp/tribler/TriblerGUI/event_request_manager.py", line 127, in on_read_data
    raise RuntimeError(json_dict["event"]["text"])
RuntimeError: [Failure instance: Traceback: <class 'socket.error'>: [Errno 16] Device or resource busy
/nix/store/d3aa78jf7pm9lk6ad67x4f21a068dxn6-python-2.7.15/lib/python2.7/threading.py:801:__bootstrap_inner
/nix/store/d3aa78jf7pm9lk6ad67x4f21a068dxn6-python-2.7.15/lib/python2.7/threading.py:754:run
/nix/store/hfq5vjpvsx9j38jc30fkg0r095m4lq3r-python2.7-Twisted-18.4.0/lib/python2.7/site-packages/twisted/_threads/_threadworker.py:46:work
/nix/store/hfq5vjpvsx9j38jc30fkg0r095m4lq3r-python2.7-Twisted-18.4.0/lib/python2.7/site-packages/twisted/_threads/_team.py:190:doWork
--- <exception caught here> ---
/nix/store/hfq5vjpvsx9j38jc30fkg0r095m4lq3r-python2.7-Twisted-18.4.0/lib/python2.7/site-packages/twisted/python/threadpool.py:250:inContext
/nix/store/hfq5vjpvsx9j38jc30fkg0r095m4lq3r-python2.7-Twisted-18.4.0/lib/python2.7/site-packages/twisted/python/threadpool.py:266:<lambda>
/nix/store/hfq5vjpvsx9j38jc30fkg0r095m4lq3r-python2.7-Twisted-18.4.0/lib/python2.7/site-packages/twisted/python/context.py:122:callWithContext
/nix/store/hfq5vjpvsx9j38jc30fkg0r095m4lq3r-python2.7-Twisted-18.4.0/lib/python2.7/site-packages/twisted/python/context.py:85:callWithContext
/nix/store/hfq5vjpvsx9j38jc30fkg0r095m4lq3r-python2.7-Twisted-18.4.0/lib/python2.7/site-packages/twisted/internet/_resolver.py:124:get
]

I've sent the report, it's the same bug, version 7.1.0-exp3, on NixOS.

xvapx · 2018-07-03T19:01:21Z

Great, thanks a lot!

devos50 · 2018-07-04T09:20:02Z

@xvapx could you try out this build? https://jenkins.tribler.org/job/Build_Tribler_next/297/

xvapx · 2018-07-04T18:39:35Z

Yep, I'll try and report back

xvapx · 2018-07-05T09:19:13Z

@devos50 It's been running for more than one hour now, without a single crash.

viric · 2020-05-01T10:44:04Z

This crashes again in Tribler 7.4.4 (on NixOS release-20.03). Same thing.

Traceback (most recent call last):
  File "/nix/store/3rwchz4m7qbjm43x57pq7dsg3zzaz25a-tribler-7.4.4/TriblerGUI/event_request_manager.py", line 114, in on_read_data
    raise RuntimeError(json_dict["event"]["text"])
RuntimeError: [Failure instance: Traceback: <class 'OSError'>: [Errno 16] Device or resource busy
/nix/store/z4y3q5nsm1hhk7g18ywpnsdgqcgsvc2i-python3-3.7.6/lib/python3.7/threading.py:926:_bootstrap_inner
/nix/store/z4y3q5nsm1hhk7g18ywpnsdgqcgsvc2i-python3-3.7.6/lib/python3.7/threading.py:870:run
/nix/store/98n0hbsr0022jarj6xal83w5wldiyrqb-python3.7-Twisted-20.3.0/lib/python3.7/site-packages/twisted/_threads/_threadworker.py:46:work
/nix/store/98n0hbsr0022jarj6xal83w5wldiyrqb-python3.7-Twisted-20.3.0/lib/python3.7/site-packages/twisted/_threads/_team.py:190:doWork
--- <exception caught here> ---
/nix/store/98n0hbsr0022jarj6xal83w5wldiyrqb-python3.7-Twisted-20.3.0/lib/python3.7/site-packages/twisted/python/threadpool.py:250:inContext
/nix/store/98n0hbsr0022jarj6xal83w5wldiyrqb-python3.7-Twisted-20.3.0/lib/python3.7/site-packages/twisted/python/threadpool.py:266:<lambda>
/nix/store/98n0hbsr0022jarj6xal83w5wldiyrqb-python3.7-Twisted-20.3.0/lib/python3.7/site-packages/twisted/python/context.py:122:callWithContext
/nix/store/98n0hbsr0022jarj6xal83w5wldiyrqb-python3.7-Twisted-20.3.0/lib/python3.7/site-packages/twisted/python/context.py:85:callWithContext
/nix/store/98n0hbsr0022jarj6xal83w5wldiyrqb-python3.7-Twisted-20.3.0/lib/python3.7/site-packages/twisted/internet/_resolver.py:124:get
/nix/store/z4y3q5nsm1hhk7g18ywpnsdgqcgsvc2i-python3-3.7.6/lib/python3.7/socket.py:752:getaddrinfo
]```

viric · 2020-05-01T13:22:01Z

This doesn't happen in 7.5.0-rc5 so I guess it's all fine.

xoriole added the type: bug label Feb 5, 2018

xoriole added this to the V7.1: The token micro-economy milestone Feb 7, 2018

This was referenced Feb 24, 2018

Possible package updates 2018-02-23 NixOS/nixpkgs#35444

Closed

tribler: 7.0.0-rc3 -> 7.0.1 NixOS/nixpkgs#36065

Closed

qstokkink added the needs feedback label Jun 26, 2018

qstokkink closed this as completed Jun 28, 2018

qstokkink reopened this Jun 28, 2018

qstokkink removed the needs feedback label Jun 28, 2018

qstokkink assigned devos50 and qstokkink Jun 28, 2018

qstokkink mentioned this issue Jun 28, 2018

Added workaround for resolve socket.error 16 #3696

Merged

devos50 closed this as completed in #3696 Jun 28, 2018

qstokkink reopened this Jun 30, 2018

devos50 mentioned this issue Jul 3, 2018

Fixed crash during pstate upgrade to 7.1 #3711

Merged

qstokkink closed this as completed in #3711 Jul 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Device or resource busy #3440

Device or resource busy #3440

xoriole commented Feb 5, 2018

non-Jedi commented May 2, 2018

qstokkink commented Jun 22, 2018 •

edited

Loading

devos50 commented Jun 22, 2018

qstokkink commented Jun 26, 2018

non-Jedi commented Jun 26, 2018

xvapx commented Jun 26, 2018

qstokkink commented Jun 27, 2018

xvapx commented Jun 27, 2018

xvapx commented Jun 28, 2018

xvapx commented Jun 28, 2018 •

edited

Loading

xvapx commented Jun 28, 2018

qstokkink commented Jun 28, 2018

qstokkink commented Jun 28, 2018

qstokkink commented Jun 28, 2018

qstokkink commented Jun 28, 2018

xvapx commented Jun 28, 2018

xvapx commented Jun 29, 2018

xvapx commented Jul 3, 2018

devos50 commented Jul 4, 2018

xvapx commented Jul 4, 2018

xvapx commented Jul 5, 2018

viric commented May 1, 2020 •

edited

Loading

viric commented May 1, 2020

Device or resource busy #3440

Device or resource busy #3440

Comments

xoriole commented Feb 5, 2018

non-Jedi commented May 2, 2018

qstokkink commented Jun 22, 2018 • edited Loading

devos50 commented Jun 22, 2018

qstokkink commented Jun 26, 2018

non-Jedi commented Jun 26, 2018

xvapx commented Jun 26, 2018

qstokkink commented Jun 27, 2018

xvapx commented Jun 27, 2018

xvapx commented Jun 28, 2018

xvapx commented Jun 28, 2018 • edited Loading

xvapx commented Jun 28, 2018

qstokkink commented Jun 28, 2018

qstokkink commented Jun 28, 2018

qstokkink commented Jun 28, 2018

qstokkink commented Jun 28, 2018

xvapx commented Jun 28, 2018

xvapx commented Jun 29, 2018

xvapx commented Jul 3, 2018

devos50 commented Jul 4, 2018

xvapx commented Jul 4, 2018

xvapx commented Jul 5, 2018

viric commented May 1, 2020 • edited Loading

viric commented May 1, 2020

qstokkink commented Jun 22, 2018 •

edited

Loading

xvapx commented Jun 28, 2018 •

edited

Loading

viric commented May 1, 2020 •

edited

Loading