Add cookie & crumb to requests

[ValueRaider]

Does this fix info 404 error? VOTE HERE

---

I stumbled upon simple code to get cookie & crumb from Yahoo: https://stackoverflow.com/questions/76065035/yahoo-finance-v7-api-now-requiring-cookies-python

It might fix issues some users are having. Try it and report back (instructions #1080)

Update:

Fixed this "basic" strategy, then added second strategy for getting cookie & crumb #1657 (comment)

Hopefully fixes:

• .info returns 404 on all tickers #1729

Status:

Big change so letting people stress-test this. Concerned by Asia reporting "doesn't fix".

Credits:
@bot-unit
@psychoz971
@Jshen123
anyone else?

[vismoh2010]

@ValueRaider This fixes all my issues mentioned in #1649! Please merge this ASAP.

[ValueRaider]

Code will need fault-tolerance, so failure to get cookie or crumb doesn't stop use.

[ValueRaider]

Updated with @bot-unit 's crumb fetch #1729 (comment). Also, store crumb in utils.py for global reuse that actually works (class members wasn't working).

Works great except _fetch_complementary() failing:

  File "yfinance/data.py", line 149, in get
    response = self._session.get(
...
  File ".local/lib/python3.11/site-packages/requests/utils.py", line 993, in prepend_scheme_if_needed
    parsed = parse_url(url)
             ^^^^^^^^^^^^^^
  File ".local/lib/python3.11/site-packages/urllib3/util/url.py", line 402, in parse_url
    if not _SCHEME_RE.search(url):
           ^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'int'

[ms82494]

I don't know if this is the right place to leave this comment. But I installed and tried this feature branch today from the US and I run into the following error:

In [2]: msft = yf.Ticker("MSFT")

In [3]: msft.info
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 msft.info

File ~/miniconda3/envs/mergent/lib/python3.10/site-packages/yfinance/ticker.py:142, in Ticker.info(self)
    140 @property
    141 def info(self) -> dict:
--> 142     return self.get_info()

File ~/miniconda3/envs/mergent/lib/python3.10/site-packages/yfinance/base.py:1655, in TickerBase.get_info(self, proxy)
   1653 def get_info(self, proxy=None) -> dict:
   1654     self._quote.proxy = proxy
-> 1655     data = self._quote.info
   1656     return data

File ~/miniconda3/envs/mergent/lib/python3.10/site-packages/yfinance/scrapers/quote.py:570, in Quote.info(self)
    567 @property
    568 def info(self) -> dict:
    569     if self._info is None:
--> 570         self._fetch(self.proxy)
    571         self._fetch_complementary(self.proxy)  # Failing, don't know why. Help!
    573     return self._info

File ~/miniconda3/envs/mergent/lib/python3.10/site-packages/yfinance/scrapers/quote.py:599, in Quote._fetch(self, proxy)
    597 modules = ['financialData', 'quoteType', 'defaultKeyStatistics', 'assetProfile', 'summaryDetail']
    598 params_dict = {"modules": modules, "ssl": "true"}
--> 599 result = self._data.get_raw_json(
    600     _BASIC_URL_ + f"/{self._data.ticker}", params=params_dict, proxy=proxy
    601 )
    602 result["quoteSummary"]["result"][0]["symbol"] = self._data.ticker
    603 query1_info = next(
    604     (info for info in result.get("quoteSummary", {}).get("result", []) if info["symbol"] == self._data.ticker),
    605     None,
    606 )

File ~/miniconda3/envs/mergent/lib/python3.10/site-packages/yfinance/data.py:172, in TickerData.get_raw_json(self, url, user_agent_headers, params, proxy, timeout)
    171 def get_raw_json(self, url, user_agent_headers=None, params=None, proxy=None, timeout=30):
--> 172     response = self.get(url, user_agent_headers=user_agent_headers, params=params, proxy=proxy, timeout=timeout)
    173     response.raise_for_status()
    174     return response.json()

File ~/miniconda3/envs/mergent/lib/python3.10/site-packages/yfinance/data.py:147, in TickerData.get(self, url, user_agent_headers, params, cookies, proxy, timeout)
    144     params = {}
    145 if 'crumb' not in params:
    146     # params['crumb'] = self._get_crumb()
--> 147     params['crumb'] = self._get_crumb_botunit()
    149 response = self._session.get(
    150     url=url,
    151     params=params,
   (...)
    154     timeout=timeout,
    155     headers=user_agent_headers or self.user_agent_headers)
    156 return response

File ~/miniconda3/envs/mergent/lib/python3.10/site-packages/yfinance/data.py:109, in TickerData._get_crumb_botunit(self, proxy, timeout)
    107 soup = BeautifulSoup(response.content, 'html.parser')
    108 csrfTokenInput = soup.find('input', attrs={'name': 'csrfToken'})
--> 109 csrfToken = csrfTokenInput['value']
    110 sessionIdInput = soup.find('input', attrs={'name': 'sessionId'})
    111 sessionId = sessionIdInput['value']

TypeError: 'NoneType' object is not subscriptable

[ValueRaider]

@ms82494 @gene-git I wonder if this branch only works outside USA, and inside USA needs different code to get crumb. Can you experiment with data.py, see if there's a solution for USA?

[gene-git]

@ValueRaider - i need some guidance what you suggest I try in data.py.
happy to test some things - but not sure what to try.

[gene-git]

If I just put the url into my browser I get back an 11 character string.

url is from data.py : https://query1.finance.yahoo.com/v1/test/getcrumb

So that doesn't seem to be a problem - bug must be elsewhere.

Edit: so seems to me

_get_crumb() works

while

_get_crumb_botunit() gets error at line 108 - doesn't seem to have much to do with _get_crumb() or even any crumb best I can tell. The failure occurs parsing this:

url = https://guce.yahoo.com/consent

Sorry I have no clue what this part of the code is trying to do.

As I said in the last message (that got deleted) - this url returns a ton of stuff but none of that includes 'csrfToken'
which is why the next line produces csrfTokenInput = None and thats the problem.

what is csrfToken and why would it be in the .../consent' url?
Does this have something to do with crumb somehow?

[ValueRaider]

(I deleted because you're repeating stack trace above you)
Here is what https://guce.yahoo.com/consent returns for me in Europe: guce-yahoo-com-consent-html.txt
@bot-unit maybe you can help

[ms82494]

I don't know if this helps, but I'm located in the US (California), and when I go to https://guce.yahoo.com/consent I get this result:

(base) [catweazle@karwendel ~]$ curl -s -L -D - https://guce.yahoo.com/consent -o guce.txt -w'%{url_effective}\n'
HTTP/1.1 302 Found
Connection: keep-alive
Set-Cookie: GUC=AQEBCAFlQs5ldUIYkwOz&s=AQAAAMb0hisq&g=ZUGCkg; Expires=Thu, 31 Oct 2024 10:40:39 GMT; Domain=yahoo.com; Path=/; Secure
Set-Cookie: A1=d=AQABBIiCQWUCECtRuaMCReMRZ4xrdbtm0fIFEgEBCAHOQmV1Zdwr0iMA_eMBAAcIiIJBZUxUFFA&S=AQAAAkFX6HAlKvGgcrse_FHLMnw; Expires=Wed, 30 Oct 2024 22:41:12 GMT; Max-Age=31536000; Domain=.yahoo.com; Path=/; SameSite=Lax; Secure; HttpOnly
Set-Cookie: A3=d=AQABBIiCQWUCECtRuaMCReMRZ4xrdbtm0fIFEgEBCAHOQmV1Zdwr0iMA_eMBAAcIiIJBZUxUFFA&S=AQAAAkFX6HAlKvGgcrse_FHLMnw; Expires=Wed, 30 Oct 2024 22:41:12 GMT; Max-Age=31536000; Domain=.yahoo.com; Path=/; SameSite=None; Secure; HttpOnly
Set-Cookie: A1S=d=AQABBIiCQWUCECtRuaMCReMRZ4xrdbtm0fIFEgEBCAHOQmV1Zdwr0iMA_eMBAAcIiIJBZUxUFFA&S=AQAAAkFX6HAlKvGgcrse_FHLMnw; Domain=.yahoo.com; Path=/; SameSite=Lax; Secure
Server: guce
Strict-Transport-Security: max-age=31536000; includeSubDomains
Location: https://www.yahoo.com?guccounter=1
Content-Length: 0
Date: Tue, 31 Oct 2023 22:41:12 GMT

HTTP/2 200 
expect-ct: max-age=31536000, report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-expect-ct-report-only
referrer-policy: no-referrer-when-downgrade
strict-transport-security: max-age=31536000
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
content-type: text/html; charset=utf-8
date: Tue, 31 Oct 2023 22:41:13 GMT
x-envoy-upstream-service-time: 44
server: ATS
age: 0
cache-control: no-store, no-cache, max-age=0, private
expires: -1
content-security-policy: frame-ancestors 'self' https://*.builtbygirls.com https://*.rivals.com https://*.engadget.com https://*.intheknow.com https://*.autoblog.com https://*.techcrunch.com https://*.yahoo.com https://*.aol.com https://*.huffingtonpost.com https://*.oath.com https://*.search.yahoo.com https://*.pnr.ouryahoo.com https://pnr.ouryahoo.com https://*.search.aol.com https://*.search.huffpost.com https://*.onesearch.com https://*.verizonmedia.com https://*.publishing.oath.com https://*.autoblog.com; sandbox allow-forms allow-same-origin allow-scripts allow-popups allow-popups-to-escape-sandbox allow-presentation; report-uri https://csp.yahoo.com/beacon/csp?src=ats&site=frontpage&region=US&lang=en-US&device=desktop&yrid=62bc3m5ik30k9&partner=;

https://www.yahoo.com/?guccounter=1
(base) [catweazle@karwendel ~]$ 

After the redirect, I'm basically on www.yahoo.com. Here's the content of guce.txt. It doesn't contain the csrfToken value, which is causing the crumb retrieval to fail for me.

[ms82494]

I then opened a VPN tunnel to Germany, and went to https://guce.yahoo.com/consent through that. This redirects to https://consent.yahoo.com/v2/collectConsent. See here:

(base) [catweazle@karwendel ~]$ curl -s -L -D - https://guce.yahoo.com/consent -o guce.txt -w'%{url_effective}\n'
HTTP/1.1 302 Found
Connection: keep-alive
Server: guce
Strict-Transport-Security: max-age=31536000; includeSubDomains
Location: https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_73c71438-e32d-470a-9b7f-aa69c7fd803e
Content-Length: 0
Date: Tue, 31 Oct 2023 22:56:27 GMT

HTTP/1.1 200 OK
Expires: 0
Cache-Control: no-cache, no-store, must-revalidate
Content-Security-Policy-Report-Only: default-src 'none'; block-all-mixed-content; connect-src 'self'; frame-ancestors 'none'; img-src 'self' https://s.yimg.com; media-src 'none'; script-src 'self' 'nonce-bVZSEFrngDV51YZEsYRKNQEDDBZs9x8g' https://s.yimg.com; style-src 'self' 'nonce-bVZSEFrngDV51YZEsYRKNQEDDBZs9x8g' https://s.yimg.com; font-src 'self'; object-src 'none'; frame-src 'none'; report-uri https://csp.yahoo.com/beacon/csp?src=guce
Server: guce
X-XSS-Protection: 1; mode=block
Pragma: no-cache
X-Frame-Options: DENY
Referrer-Policy: strict-origin-when-cross-origin
Date: Tue, 31 Oct 2023 22:56:29 GMT
Connection: keep-alive
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
Transfer-Encoding: chunked
Content-Type: text/html;charset=UTF-8

https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_73c71438-e32d-470a-9b7f-aa69c7fd803e
(base) [catweazle@karwendel ~]$ 

For comparison, here's the source of the returned page: guce.txt. This time I do have the csrfToken that's referenced in the @botunit version of the crumb retrieval method.

[gene-git]

It would be helpful to know what we are actually looking for - the code looks for csrfToken which is not in the returned data.

grep -i csrfToken /tmp/guce.txt

returns nothing - seems to be the source of the "problem"

Do you all see anything like that in the returned output?

File I get is pretty long -

$ wc /tmp/guce.txt
653 43825 1358109 /tmp/guce.txt

[ValueRaider]

Remember the goal: mimic what happens when you manually visit finance.yahoo.com. Firefox -> F12 -> Network tab -> F5

[gene-git]

okay got it - from the US we don't get the guce csrfToken - as you all had already surmised.

[ms82494]

@ValueRaider , @gene-git : I'm knee-deep in data.py. I'm now able to get a crumb, but I still get this error:

HTTPError: 401 Client Error: Unauthorized for url: https://query2.finance.yahoo.com/v10/finance/quoteSummary/MSFT?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true&crumb=GSA9Mz%2F83QX

I'll try more stuff later.

[bot-unit]

(I deleted because you're repeating stack trace above you) Here is what https://guce.yahoo.com/consent returns for me in Europe: guce-yahoo-com-consent-html.txt @bot-unit maybe you can help

It's look like Yahoo has different politics for cookie for different country. I can test only for europe and if cookie isn't set, they ask to me to be agree with cookies. (redirect to https://guce.yahoo.com/consent)

The process look like:

1. Ask you if agree to set cookie (maybe we can skip this step for other country)
2. Set cookie (just request finance.yahoo.com)
3. Get crumb with setted cookie

We can try in this order: 2, if not success, then 1 and 2. And 3 is last. How i test, crumb is valid all time while cookie is valid too

[bot-unit]

So in my code i am using two functions before doing any yahoo requests (in Session object of Requests):

def _get_cookie(self):
    self.session = requests.Session()
    response = self.session.get('https://guce.yahoo.com/consent', headers=self.user_agent_headers)
    soup = BeautifulSoup(response.content, 'html.parser')
    csrfTokenInput = soup.find('input', attrs={'name': 'csrfToken'})
    csrfToken = csrfTokenInput['value']
    sessionIdInput = soup.find('input', attrs={'name': 'sessionId'})
    sessionId = sessionIdInput['value']
    originalDoneUrl = 'https://finance.yahoo.com/'
    namespace = 'yahoo'
    data = {
        'agree': ['agree', 'agree'],
        'consentUUID': 'default',
        'sessionId': sessionId,
        'csrfToken': csrfToken,
        'originalDoneUrl': originalDoneUrl,
        'namespace': namespace,
    }
    self.session.post(f'https://consent.yahoo.com/v2/collectConsent?sessionId={sessionId}', data=data, headers=self.user_agent_headers)
    self.session.get(f'https://guce.yahoo.com/copyConsent?sessionId={sessionId}', headers=self.user_agent_headers)
    self.cookies = True

def _update_crumb(self, try_count=0):
    if not self.cookies:
        self._get_cookie()
    crumb = self.session.get('https://query2.finance.yahoo.com/v1/test/getcrumb', headers=self.user_agent_headers).text
    if crumb == '':
        if try_count:
            raise ValueError("Too many requests")
        self.cookies = False
        self._update_crumb(1)
    self.crumb = crumb
    self.crumb_timestamp = time.time()

The first, i create request (https://guce.yahoo.com/consent) for form to agree with cookies, yahoo gives me session id. Then i post my agreement with session id (https://consent.yahoo.com/v2/collectConsent?sessionId=). After that i request cookies with my session id (https://guce.yahoo.com/copyConsent?sessionId=). The last step is get crumb (https://query2.finance.yahoo.com/v1/test/getcrumb)

This code is working for me everyday. My software recreate new Session object of requests every day, so i repeat all steps.

[bot-unit]

https://query2.finance.yahoo.com/v1/test/getcrumb
return empty string if cookies is not set
return Too many request if cookie invalid
return something like this: zLoP.Fp75Uh

USA users can check if they need cookies or something else: just delete cookie from browser and then open https://query2.finance.yahoo.com/v1/test/getcrumb

[ms82494]

Ticker(...).info works fine for me with the most recent commit. But Ticker(...).history doesn't. I guess the crumb parameter is still missing from the URL?

In [20]: yfConn.history(period='1mo')
DEBUG    Entering history()
DEBUG     GOOG: Yahoo GET parameters: {'range': '1mo', 'interval': '1d', 'includePrePost': False, 'events': 'div,splits,capitalGains'}
DEBUG     Entering get()
DEBUG      get(): https://query2.finance.yahoo.com/v8/finance/chart/GOOG
ERROR     GOOG: No price data found, symbol may be delisted (period=1mo)
DEBUG    Exiting history()
Out[20]: 
Empty DataFrame
Columns: [Open, High, Low, Close, Adj Close, Volume]
Index: []

[pipreaper]

installed from Git:
~/PycharmProjects/get_prospects$ pip3 install git+https://github.com/ranaroussi/yfinance.git@feature/cookie-and-crumb

Works for me (UK)
Many Thanks

..... stopped working on .info dictionary again but .get_fast_info() still works!

[ValueRaider]

Vote if works instead of replying (see top post).

@pipreaper We need more info. Is there an error message? And post output with yf.enable_debug_mode()

[pipreaper]

As of 4.15am UK time it is working on .info. If it stops working I will post the debug output.

import yfinance as yf
import sys
yf.enable_debug_mode()

print('run')
print(sys.version)

s_ticker = 'CSCO'
ticker = yf.Ticker(s_ticker)
# data = ticker.get_fast_info()
data = ticker.info

/home/robert/PycharmProjects/get_prospects/env/bin/python /home/robert/PycharmProjects/get_prospects/tests/a_cookie and crumb/test_it.py
DEBUG get_raw_json(): https://query2.finance.yahoo.com/v10/finance/quoteSummary/CSCO
DEBUG Entering get()
DEBUG url=https://query2.finance.yahoo.com/v10/finance/quoteSummary/CSCO
DEBUG params={'modules': 'financialData,quoteType,defaultKeyStatistics,assetProfile,summaryDetail', 'ssl': 'true'}
DEBUG Entering _get_cookie_and_crumb()
DEBUG cookie_mode = 'basic'
DEBUG Entering _get_cookie_and_crumb_basic()
DEBUG Entering _get_cookie_basic()
DEBUG loaded persistent cookie
DEBUG Exiting _get_cookie_basic()
DEBUG Entering _get_crumb_basic()
DEBUG Entering _get_cookie_basic()
DEBUG reusing cookie
DEBUG Exiting _get_cookie_basic()
run
3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
DEBUG crumb = 'n9BMUI5oIqh'
DEBUG Exiting _get_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb()
DEBUG Exiting get()
DEBUG Entering get()
DEBUG url=https://query1.finance.yahoo.com/ws/fundamentals-timeseries/v1/finance/timeseries/CSCO?symbol=CSCO&type=trailingPegRatio&period1=1683676800&period2=1699488000
DEBUG params=None
DEBUG Entering _get_cookie_and_crumb()
DEBUG cookie_mode = 'basic'
DEBUG Entering _get_cookie_and_crumb_basic()
DEBUG Entering _get_cookie_basic()
DEBUG reusing cookie
DEBUG Exiting _get_cookie_basic()
DEBUG Entering _get_crumb_basic()
DEBUG reusing crumb
DEBUG Exiting _get_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb_basic()
DEBUG Exiting _get_cookie_and_crumb()
DEBUG Exiting get()

[crusader1977]

@pipreaper

I'm located in Canada, and I was getting the following when trying to call .info

File "C:\Users\money\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\requests\models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://query2.finance.yahoo.com/v6/finance/quoteSummary/HRIF.TO?modules=financialData&modules=quoteType&modules=defaultKeyStatistics&modules=assetProfile&modules=summaryDetail&ssl=true

I installed your branch per your instructions, and not only did it solve this problem but it also fixed the cookie & crumb issues reported in many of the "failed to decrypt yahoo data response" issues that started with #1407 and have been reported in several other issues for a while now as well! IMHO the workaround to revert to the much older v6 URL was a bad decision.

In any case, all errors I was getting are gone using your branch; it should be merged with the main branch to make it into the next release ASAP!

[pipreaper]

All (UK), working well on .history and .info.

Problem occurs if I enter an erroneous Ticker (I have a search box in an App). After the erroneous call to 'NEX.L' fetching history fails to retrieve data for following valid ticker entries.

code to reproduce.txt
run debugging.txt

[ValueRaider]

@pipreaper fixed

[pipreaper]

@pipreaper fixed

confirm running my code issue has been solved

[brian-dm100]

#1657 (comment)

National Express (NEX.L) has been renamed Mobico Group Plc (MCG.L) effective from early June 2023.

[ValueRaider]

@brian-dm100 yfinance should handle & continue not blowup.

[brian-dm100]

@brian-dm100 yfinance should handle & continue not blowup.

Maybe. I use an 'exception' and enclosed 'while true: try:' to present the input box again until a correct ticker is entered.

[ValueRaider]

I've moved data fetching into a singleton class, so one session one cookie shared by threads. Can people stress-test yf.download, and any other parallel loads.

[crusader1977]

No issues for me on yf.download with 46 tickers

[Jshen123]

Small bug fix
https://github.com/ranaroussi/yfinance/pull/1741/files

Not a bug, caused by Python 3.8

[antonior]

Worked perfectly well for me in Brazil. For both history() and get_info() for 400+ tickers.

[ValueRaider]

Ima wait until Friday evening passes, that seems to be when Yahoo makes breaking changes. Then official release.