Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

all_tickers.py is stuck #24

Open
dropcunt opened this issue Jan 26, 2021 · 6 comments
Open

all_tickers.py is stuck #24

dropcunt opened this issue Jan 26, 2021 · 6 comments

Comments

@dropcunt
Copy link

I think there is a problem with the urlib.request. Might need to add a header

@tmhieu99
Copy link

tmhieu99 commented May 1, 2021

Hi @dropcunt, I got the same problem. Have you found a way to fix it?

@dkubanyi
Copy link

Hi @tmhieu99 and @dropcunt, I stumbled upon the same problem and the solution is actually very easy. The exchange where the script is fetching the data from changed its routing, which is why it gets stuck. In all_tickers.py, you need to replace the part where you fetch the exchange data to this:

for exchange in ["NASDAQ", "NYSE", "AMEX"]:
    # this is the changed URL
    url = "https://api.nasdaq.com/api/screener/stocks?offset=0&exchange={}&download=true"

    repeat_times = 10
    
    for _ in range(repeat_times):
        response = urlopen(url.format(exchange))

I was then able to successfully download the tickers

@tmhieu99
Copy link

Hi @dkubanyi, tt worked now. Thanks for your solution.

@vedantk281007
Copy link

@dkubanyi could you send the full code block?
When I do this, it still does not work.

@glitchawy
Copy link

can someone explain because it still doesn't work after changing the code

@dkubanyi
Copy link

@glitchawy took a look at it again, and even though I can't test the rest of the process, I can give you a head start - this implementation fetched the data from nasdaq's json api and wrote it into a csv file:

def get_tickers(percent):
    """Keep the top percent market-cap companies."""
    assert isinstance(percent, int)
    for exchange in ["nasdaq", "nyse", "amex"]:
        repeat_times = 10 # repeat downloading in case of http error

        for _ in range(repeat_times):
            try:
                url = "https://api.nasdaq.com/api/screener/stocks?tableonly=true&limit=3296&exchange={}".format(exchange)
                headers = {
                    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0",
                }

                print("Downloading tickers from {}: {}...".format(exchange, url))

                response = requests.get(url, headers=headers)
                j = response.json()

                table = j['data']['table']
                table_headers = table['headers']

                with open('input/tickerList.csv', 'w', newline='') as f_output:
                    csv_output = csv.DictWriter(f_output, fieldnames=table_headers.values(), extrasaction='ignore')
                    csv_output.writeheader()

                    for table_row in table['rows']:
                        csv_row = {table_headers.get(key, None): value for key, value in table_row.items()}
                        csv_output.writerow(csv_row)
            except:
                continue

It's just quick and dirty fix, feel free to adjust it to your needs and preferences.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants