Set cookies and crumb #205

dpguthrie · 2023-07-17T04:28:25Z

Initialize session with required cookies
Create larger list of rotating user-agents with related headers
Added webdriver-manager for selenium functionality
Changed from flit to poetry
Added additional package files that will help with testing and development
Added function in base client class to retrieve a crumb and then add as a query parameter

ValueRaider · 2023-07-17T11:19:11Z

I quickly tried this but get exception:
yq.Ticker('AMZN', username='redacted', password='redacted')

File ".../yahooquery/yahooquery/login.py", line 27, in init
self.driver = webdriver.Chrome(
^^^^^^^^^^^^^^^^^
TypeError: WebDriver.init() got an unexpected keyword argument 'service'

selenium=3.141.0 webdriver_manager=3.8.6

dpguthrie · 2023-07-17T12:39:06Z

selenium=3.141.0 webdriver_manager=3.8.6

@ValueRaider Can you try upgrading selenium?

ValueRaider · 2023-07-17T12:51:10Z

Selenium 4 just generates a different exception:

requests.exceptions.RetryError: HTTPSConnectionPool(host='query2.finance.yahoo.com', port=443): Max retries exceeded with url: /v1/test/getcrumb (Caused by ResponseError('too many 500 error responses'))

I prefer v3 because calls with v4 hang for 2 minutes, I've no idea why

dpguthrie · 2023-07-17T13:08:20Z

Hmm this looks like it’s no longer a problem with selenium but a problem now with getting the crumb from that endpoint after you’ve logged in.

dpguthrie · 2023-07-17T13:08:45Z

Do you get the same error when not passing in a username and password?

ValueRaider · 2023-07-17T14:43:09Z

Without username & password, same "max retries" error and now with either version of Selenium. Note: does not happen with yahooquery 2.3.2 (latest official)

yahooquery/base.py

dpguthrie · 2023-07-18T19:28:12Z

@ValueRaider I wasn't able to reproduce your error. I did change the code slightly though so it may be working for you now. I also deployed it temporarily to the streamlit app where it seems to be working again.

ValueRaider · 2023-07-18T20:47:31Z

Same behaviour, "Max retries" errors. What are other people experiencing?

dpguthrie · 2023-07-18T23:52:50Z

Same behaviour, "Max retries" errors. What are other people experiencing?

That’s weird. I have it working locally and I have this code running in the streamlit app as well - https://yahooquery.streamlit.app/

dpguthrie · 2023-07-20T13:02:12Z

@ValueRaider I’m wondering if it has something to do with where you’re based. I’m naively navigating to finance.yahoo.com to retrieve cookies - but I believe you’re based in the UK and would actually navigate to uk.finance.yahoo.com. Curious if you pulled this code down and just changed this if it would start working for you.

ValueRaider · 2023-07-20T13:11:18Z

I don't quite understand where I would make the change. But to check if worth investigating, I logged in with Firefox Debugger active and don't see UK-specific URLs, but the same as you e.g. https://query2.finance.yahoo.com/v1/test/getcrumb

EDIT: let me check header for anything UK-specific ...
EDIT: Some progress. Copying over my headers from browser solves the exception, but crumb is empty string

dpguthrie · 2023-07-20T13:22:12Z

yahooquery/utils/__init__.py

+def setup_session_with_cookies_and_crumb(session: Session):
+    headers = {**random.choice(HEADERS), **addl_headers}
+    session.headers = headers
+    response = session.get('https://finance.yahoo.com', hooks={'response': get_crumb})


@ValueRaider This is the place where you would modify to (I'm guessing) https://uk.finance.yahoo.com

I'd also be curious what headers you're seeing on your side when that request is being made.

Adding 'uk' = no difference.

But, adding key headers['Host'] = 'query1.finance.yahoo.com' did address exception. Crumb an empty string, maybe because I'm not logging in? I can't login because I updated Chrome yesterday and webdriver complaining it's too new, but that's my problem.

I wonder if it has something to do with accepting their cookies. I tried navigating to the uk site and got their consent screen.

I’m guessing without accepting that then YF is unable to actually set the appropriate cookies.

Also, this code doesn’t use selenium. It’s only used to login to YF (and not even sure that’s working right now as it looks like they’ve added a captcha)

The getcrumb response.text contains string "HTTP Status 403 - Forbidden". Is that typical of Yahoo expecting a cookie?

Yes, that makes sense if the session making that request isn't set up with the appropriate cookies. Those cookies are supposed to be set when making the initial request to the YF home page.

Sorry wrong part of code, that 403 happens in setup_session_with_cookies_and_crumb() on https://finance.yahoo.com

That's really surprising! I'm not sure what would make that a forbidden request just to the YF home page.

cmjordan42 · 2023-07-22T00:31:27Z

Hey y'all, interested in this one as I currently use all_modules in an occasional batch run that computes a ton of single security analytics. Sounds like we're still pretty far from having a solution we like here, and that maybe Yahoo is deliberately discouraging it. Given that, I'm probably gonna switch over to a bunch of individual targeted calls to get what's in all_modules on Sunday.

dpguthrie · 2023-07-22T20:24:21Z

Hey y'all, interested in this one as I currently use all_modules in an occasional batch run that computes a ton of single security analytics. Sounds like we're still pretty far from having a solution we like here, and that maybe Yahoo is deliberately discouraging it. Given that, I'm probably gonna switch over to a bunch of individual targeted calls to get what's in all_modules on Sunday.

@cmjordan42 Where are you based? My somewhat uneducated theory right now is that this would work for anyone based in the U.S, which is why it works for me and it works inside of the streamlit app. If you are U.S. based, you could take this branch for a spin and see if you're able to access the all_modules property in that way.

Another somewhat uneducated guess - I think the solution to this is to almost have country/region specific headers that map to what they would be if that person navigated to the YF home page on a browser. I think what's happening right now is instead of going directly to the home page, it first navigates to consent.yahoo.com and has you accept cookies, which you can't do using simply requests.

cmjordan42 · 2023-07-22T21:35:29Z

I'm in US - EST. I'd be happy to test drive if it would be helpful.

cmjordan42 · 2023-07-23T20:05:53Z

Just tested with this branch, it appears to work fine. Definitely an improvement even if some regions have more complicated issues. May as well publish it to get all_modules working for some people.

However, it occasionally fails with something a response along the lines of:

"For input string: "-91000.0000000002""

cmjordan42 · 2023-07-23T20:06:42Z

And that appears to be transient; if you request the same security again, it may work. Not sure if that transience is on the yahooquery side or the Yahoo side.

dpguthrie · 2023-07-24T16:26:42Z

@ValueRaider, @cmjordan42 Would you mind giving this another go? Made some changes last night that hopefully both handles errors but also provides a fallback option with selenium (if it's installed) to retrieve cookies/crumb.

ValueRaider · 2023-07-24T18:19:47Z

No change, and I tried changing URL and header parameters. These are my browser headers btw:

{
    "Host": "query1.finance.yahoo.com",
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0",
    "Accept": "*/*",
    "Accept-Language": "en-GB,en;q=0.5",
    "Accept-Encoding": "gzip, deflate, br",
    "Content-Type": "text/plain",
    "Origin": "https://finance.yahoo.com",
    "Referer": "https://finance.yahoo.com/"
}

dpguthrie · 2023-07-24T19:18:12Z

@ValueRaider Is this where the error is happening (the explicit request to retrieve the crumb)?

ValueRaider · 2023-07-24T19:57:57Z

That line returns an empty string. This line returns 403 Forbidden.

ValueRaider · 2023-08-07T19:58:52Z

Does this work for you in USA? https://stackoverflow.com/questions/76065035/yahoo-finance-v7-api-now-requiring-cookies-python
Works for me in Europe.

… feat/set-cookie-and-crumb

dpguthrie added 7 commits July 16, 2023 16:48

Modifications to switch to poetry

5fa0fc8

Make selenium pkgs optional for poetry

d029573

Add function to get crumb, set in init

ae4d124

Add webdriver manager for selenium

4780894

Access public instead of private method

c376be2

Initialize session with cookies, need appropriate headers

5876ec9

Change to appropriate branch

ab161a6

Account for FuturesSession when setting up for cookies

a56c77d

dpguthrie added 3 commits July 17, 2023 21:05

Updates for selenium 4

cf7e74e

Update optional deps

c3d2641

Update to get crumb when setting up session

1ca552c

ericwbzhang reviewed Jul 18, 2023

View reviewed changes

yahooquery/base.py Outdated Show resolved Hide resolved

dpguthrie mentioned this pull request Jul 18, 2023

Ticker('aapl').all_modules broken #206

Closed

dpguthrie mentioned this pull request Jul 19, 2023

API calls returning 'Invalid Cookie' #203

Open

dpguthrie commented Jul 20, 2023

View reviewed changes

dpguthrie added 3 commits July 23, 2023 22:03

Add better error handling when unable to find crumb

a058dd8

Create variable indicating if user has optional dependency

4d7896b

Add fallback if crumb, cookies retrieval fails

89e0abc

maread99 mentioned this pull request Jul 24, 2023

Ticker.history return messy tz info #208

Open

Ge0rg3 mentioned this pull request Jul 28, 2023

Fix Intermittent Crumb Collection Issue #212

Closed

dpguthrie added 4 commits October 28, 2023 10:34

Add warning

8c9d295

Merge branch 'master' of https://github.com/dpguthrie/yahooquery into…

1f438f4

… feat/set-cookie-and-crumb

Update how cookie is obtained, remove selenium backup

30077f7

Remove unnecesasry imports

231a19c

dpguthrie merged commit 5da0b49 into master Oct 28, 2023
1 check passed

dpguthrie deleted the feat/set-cookie-and-crumb branch October 28, 2023 17:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set cookies and crumb #205

Set cookies and crumb #205

dpguthrie commented Jul 17, 2023

ValueRaider commented Jul 17, 2023 •

edited

dpguthrie commented Jul 17, 2023 •

edited

ValueRaider commented Jul 17, 2023 •

edited

dpguthrie commented Jul 17, 2023

dpguthrie commented Jul 17, 2023

ValueRaider commented Jul 17, 2023

dpguthrie commented Jul 18, 2023

ValueRaider commented Jul 18, 2023

dpguthrie commented Jul 18, 2023

dpguthrie commented Jul 20, 2023

ValueRaider commented Jul 20, 2023 •

edited

dpguthrie Jul 20, 2023

dpguthrie Jul 20, 2023

ValueRaider Jul 20, 2023 •

edited

dpguthrie Jul 20, 2023

dpguthrie Jul 20, 2023

ValueRaider Jul 20, 2023

dpguthrie Jul 20, 2023

ValueRaider Jul 20, 2023

dpguthrie Jul 20, 2023

cmjordan42 commented Jul 22, 2023

dpguthrie commented Jul 22, 2023

cmjordan42 commented Jul 22, 2023

cmjordan42 commented Jul 23, 2023

cmjordan42 commented Jul 23, 2023

dpguthrie commented Jul 24, 2023

ValueRaider commented Jul 24, 2023

dpguthrie commented Jul 24, 2023 •

edited

ValueRaider commented Jul 24, 2023

ValueRaider commented Aug 7, 2023 •

edited

Set cookies and crumb #205

Set cookies and crumb #205

Conversation

dpguthrie commented Jul 17, 2023

ValueRaider commented Jul 17, 2023 • edited

dpguthrie commented Jul 17, 2023 • edited

ValueRaider commented Jul 17, 2023 • edited

dpguthrie commented Jul 17, 2023

dpguthrie commented Jul 17, 2023

ValueRaider commented Jul 17, 2023

dpguthrie commented Jul 18, 2023

ValueRaider commented Jul 18, 2023

dpguthrie commented Jul 18, 2023

dpguthrie commented Jul 20, 2023

ValueRaider commented Jul 20, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ValueRaider Jul 20, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmjordan42 commented Jul 22, 2023

dpguthrie commented Jul 22, 2023

cmjordan42 commented Jul 22, 2023

cmjordan42 commented Jul 23, 2023

cmjordan42 commented Jul 23, 2023

dpguthrie commented Jul 24, 2023

ValueRaider commented Jul 24, 2023

dpguthrie commented Jul 24, 2023 • edited

ValueRaider commented Jul 24, 2023

ValueRaider commented Aug 7, 2023 • edited

ValueRaider commented Jul 17, 2023 •

edited

dpguthrie commented Jul 17, 2023 •

edited

ValueRaider commented Jul 17, 2023 •

edited

ValueRaider commented Jul 20, 2023 •

edited

ValueRaider Jul 20, 2023 •

edited

dpguthrie commented Jul 24, 2023 •

edited

ValueRaider commented Aug 7, 2023 •

edited