Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting error "OSError: [Errno 98] Address already in use" when making lots of requests. #3

Closed
blark opened this issue Jun 8, 2017 · 1 comment

Comments

@blark
Copy link

blark commented Jun 8, 2017

I am making tons of requests at one time, for example say 100k. I am using a semaphore to limit the requests, but I'm still getting the address already in use error.

I am pretty sure the answer lies in setting SO_REUSEPORT or SO_REUSEADDR somewhere, but I'm not 100% sure where to do this. I'm using asyncio and uvloop to make these requests with async_dns. If you can help me I'd appreciate it! THANKS!

Here is the (very rough) code:

import asyncio
import uvloop
from async_dns.resolver import ProxyResolver
from async_dns import types

tasks = []
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
resolver = ProxyResolver()

async def lookup(name):
    with (await sem):
        response = await resolver.query(name, types.A)
        try:
            if response.__dict__['an']:
                print(response)
        except AttributeError:
            pass

with open('subdomains.txt') as f:
    names = f.read().splitlines()

print("Looking up {} subdomains...".format(len(names)))

for n in names:
    task = asyncio.ensure_future(lookup('{}.testdomain.com'.format(n)))
    tasks.append(task)

sem = asyncio.Semaphore(500)
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
@gera2ld gera2ld closed this as completed in bc3f3e2 Jun 8, 2017
@blark
Copy link
Author

blark commented Jun 9, 2017

Running the code above now results in

AttributeError: 'NoneType' object has no attribute 'an'

I believe that all requests for domains that don't exist now result in the same error...

@gera2ld gera2ld reopened this Jun 9, 2017
@gera2ld gera2ld closed this as completed in 5a08968 Jun 9, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants