YfData requesting fc.yahoo.com on every call

[jhodsdon]

Describe bug

Hello, as everyone says. The devs are this project are heros. With 0.2.59 i noticed fc.yahoo.com being hit every call. I noticed that the crumb does not seem to be reused at times — if cached here —

https://github.com/ranaroussi/yfinance/blame/4c23d339bc88ce4b70ba0dd4add7e38743242833/yfinance/data.py#L351C23-L351C23

I find fc.yahoo.com only needs to be hit once and the crumb successfully reused.

Simple code that reproduces your problem

import time
import types
from urllib.parse import urlparse

import yfinance
from curl_cffi import Session

# yfinance.enable_debug_mode()

log = print


def verbose_session(session):
    """
    Monkey-patch a curl_cffi.requests.Session instance to emit DEBUG logs
    showing method, hostname, path, and duration for each request.
    """
    # grab the original bound method
    orig_request = session.request

    def _request_with_logging(self, method, url, *args, **kwargs):
        parsed = urlparse(url)
        host = parsed.hostname or "<unknown-host>"
        path = parsed.path or "/"
        query = parsed.query

        start_ts = time.time()

        # perform the real request
        resp = orig_request(method, url, *args, **kwargs)

        # End log
        duration_ms = int((time.time() - start_ts) * 1000)
        log(
            f"CURL_CFFI -- {method.upper()} - {resp.status_code} - {duration_ms}ms - {host} - {path} - {query}"
        )

        return resp

    session.request = types.MethodType(_request_with_logging, session)
    return session


session = Session(impersonate="chrome")
verbose_session(session)

for _ in range(10):
    ticker = yfinance.Ticker("AAPL", session=session)

    price = ticker.fast_info.last_price
    print(price)

Debug log from yf.enable_debug_mode()

CURL_CFFI -- GET - 404 - 134ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 404 - 20ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 131ms - query1.finance.yahoo.com - /v1/test/getcrumb -
CURL_CFFI -- GET - 200 - 139ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
CURL_CFFI -- GET - 404 - 16ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 87ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
198.4250030517578
CURL_CFFI -- GET - 404 - 13ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 25ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
CURL_CFFI -- GET - 404 - 20ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 13ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
198.4250030517578
CURL_CFFI -- GET - 404 - 17ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 18ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
CURL_CFFI -- GET - 404 - 14ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 15ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
198.4250030517578
CURL_CFFI -- GET - 404 - 15ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 19ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
CURL_CFFI -- GET - 404 - 17ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 16ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
198.4250030517578
CURL_CFFI -- GET - 404 - 14ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 17ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
CURL_CFFI -- GET - 404 - 14ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 16ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
198.4250030517578
CURL_CFFI -- GET - 404 - 18ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 16ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
CURL_CFFI -- GET - 404 - 23ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 17ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
198.4250030517578
CURL_CFFI -- GET - 404 - 14ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 18ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
CURL_CFFI -- GET - 404 - 20ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 18ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
198.4250030517578
CURL_CFFI -- GET - 404 - 13ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 16ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
CURL_CFFI -- GET - 404 - 19ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 16ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
198.4250030517578
CURL_CFFI -- GET - 404 - 14ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 16ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
CURL_CFFI -- GET - 404 - 15ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 15ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
198.4250030517578
CURL_CFFI -- GET - 404 - 20ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 20ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
CURL_CFFI -- GET - 404 - 13ms - fc.yahoo.com - / -
CURL_CFFI -- GET - 200 - 16ms - query2.finance.yahoo.com - /v8/finance/chart/AAPL -
198.4250030517578

Bad data proof

No response

yfinance version

0.2.59

Python version

No response

Operating system

No response

[ValueRaider]

Show proof of reuse, with yf.enable_debug_mode()

[jhodsdon]

Show proof of reuse, with yf.enable_debug_mode()

Updated the issue with a monkey patched logging of the reused curl_cffi.Session. There is no debugging in the method that requests fc.yahoo.com

[ValueRaider]

bug confirmed, fix on the way

[ValueRaider]

Fix in #2483

[ValueRaider]

Try latest release

[jhodsdon]

Fixed! Tested using 0.2.61. Thank you!