Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Script appears broken - 401 Unauthorized #33

Closed
phish0r opened this issue Apr 19, 2023 · 37 comments
Closed

Script appears broken - 401 Unauthorized #33

phish0r opened this issue Apr 19, 2023 · 37 comments

Comments

@phish0r
Copy link

phish0r commented Apr 19, 2023

/home/ticker.sh AAPL
jq: error (at :1): Cannot iterate over null (null)

appears the script is broken

@phish0r
Copy link
Author

phish0r commented Apr 19, 2023

nevermind, script is not broken. appears to have something to do with the rare google outage that happened today

@phish0r phish0r closed this as completed Apr 19, 2023
@g4570n
Copy link

g4570n commented Apr 19, 2023

I am getting the same error trying to use the script.

jq: error (at <stdin>:1): Cannot iterate over null (null)

Does it work for you or does it keep giving the error?

@phish0r
Copy link
Author

phish0r commented Apr 19, 2023 via email

@sfsam
Copy link

sfsam commented Apr 19, 2023

I am also getting this error. I think the Yahoo finance API is failing and giving jq bad data.

@phish0r phish0r reopened this Apr 20, 2023
@phish0r
Copy link
Author

phish0r commented Apr 20, 2023

This appears to be an ongoing issue

@pstadler
Copy link
Owner

Yahoo changed something in their finance API. Tried to debug it, but there’s no quick fix.
Basically, they now expect another query parameter which is available from within a browser session. Will need to dig into it a little deeper.

@pstadler
Copy link
Owner

Here’s how other people are handling this. Two extra calls to obtain a session and fetch a crumb.

Implementation for us would look something like this: If crumb is unavailable or API returns 401, obtain new session, fetch crumb (store it in temporary file for subsequent requests) and repeat the initial call to fetch quotes.

@pstadler pstadler changed the title script appears broken Script appears broken - 401 Unauthorized Apr 20, 2023
@pstadler
Copy link
Owner

Please test this and let me know whether it works and which OS you're using: #35

@appatalks
Copy link
Sponsor

Oh!
I also just created a quick pull request, testing yours too.

@appatalks
Copy link
Sponsor

Please test this and let me know whether it works and which OS you're using: #35

I got this error at the moment, though I do have this running with root via /usr/local/sbin/:

"mktemp: too few X's in template 'pstadler-ticker-sh'"

@pstadler
Copy link
Owner

Please test this and let me know whether it works and which OS you're using: #35

I got this error at the moment, though I do have this running with root via /usr/local/sbin/:

"mktemp: too few X's in template 'pstadler-ticker-sh'"

Linux?

@appatalks
Copy link
Sponsor

Please test this and let me know whether it works and which OS you're using: #35

I got this error at the moment, though I do have this running with root via /usr/local/sbin/:
"mktemp: too few X's in template 'pstadler-ticker-sh'"

Linux?

Yes, I'm currently on Arch :)

@pstadler
Copy link
Owner

So what I‘m ultimately trying to do is to persist sessions across executions. #35 works just fine on macOS, but mktemp requires some sorts of random bits which is quite annoying. I should have some time later tonight to work out a solution that works across platforms.

@pstadler
Copy link
Owner

pstadler commented Apr 20, 2023

Updated PR, tested with macOS and Arch. This should now work in most environments.

@appatalks
Copy link
Sponsor

I can confirm working well on my particular environment. Great work and Thank you! @pstadler

@pstadler
Copy link
Owner

merged into master. please pull the latest version.

@tomaszg7
Copy link

tomaszg7 commented Apr 20, 2023

For me (on Gentoo) the old version started to work now (it was broken a few hours ago). New version however doesn't produce any output. No error message, no quotes, nothing.

Edit: here's content of my crumb file:

<!doctype html public "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
    <title>Yahoo! - Error report</title>
    <style>
        /* nn4 hide */ /*/*/
        body {
        font: small/ 1.2em arial, helvetica, clean, sans-serif;
        font: x-small;
        text-align: center;
        }

        table {
        font-size: inherit;
        font: x-small;
        }

        html>body {
        font: 83%/ 1.2em arial, helvetica, clean, sans-serif;
        }

        input {
        font-size: 100%;
        vertical-align: middle;
        }

        p,form {
        margin: 0;
        padding: 0;
        }

        p {
        padding-bottom: 6px;
        margin-bottom: 10px;
        }

        #doc {
        width: 48.5em;
        margin: 0 auto;
        border: 1px solid #fff;
        text-align: center;
        }

        #ygma {
        text-align: right;
        margin-bottom: 53px
        }

        #ygma img {
        float: left;
        }

        #ygma div {
        border-bottom: 1px solid #ccc;
        padding-bottom: 8px;
        margin-left: 152px;
        }

        #bd {
        clear: both;
        text-align: left;
        width: 75%;
        margin: 0 auto 20px;
        }

        h1 {
        font-size: 135%;
        text-align: center;
        margin: 0 0 15px;
        }

        legend {
        display: none;
        }

        fieldset {
        border: 0 solid #fff;
        padding: .8em 0 .8em 4.5em;
        }

        form {
        position: relative;
        background: #eee;
        margin-bottom: 15px;
        border: 1px solid #ccc;
        border-width: 1px 0;
        }

        #s1p {
        width: 15em;
        margin-right: .1em;
        }

        form span {
        position: absolute;
        left: 70%;
        top: .8em;
        }

        form a {
        font: 78%/ 1.2em arial;
        display: block;
        padding-left: .8em;
        white-space: nowrap;
        background: url(https://s.yimg.com/lq/a/i/s/bullet.gif)
        no-repeat left center;
        }

        form .sep {
        display: none;
        }

        .more {
        text-align: center;
        }

        #ft {
        padding-top: 10px;
        border-top: 1px solid #999;
        }

        #ft p {
        text-align: center;
        font: 78% arial;
        }
        /* end nn4 hide */
    </style>
</head>
<body>
<div id="doc">
    <div id="ygma">
        <a href="http://www.yahoo.com">
            <img src="https://s.yimg.com/lq/a/i/yahoo.gif" width="147"
                 height="31" border="0" alt="Yahoo!">
        </a>

        <div>
            <a href="http://www.yahoo.com">Yahoo!</a>
            - <a href="http://help.yahoo.com">Help</a>
        </div>
    </div>
    <div id="bd">
        <h1>HTTP Status 500 - Server Error</h1>

        <p>
            <b>type</b>
            Exception report
        </p>

        <p>
            <b>message</b>
            <u>Server Error</u>
        </p>

        <p>
            <b>description</b>
            <u>Server Error</u>
        </p>

        <p>
            
        </p>
    </div>
</div>
</body>
</html>

@appatalks
Copy link
Sponsor

appatalks commented Apr 20, 2023

For me (on Gentoo) the old version started to work now (it was broken a few hours ago). New version however doesn't produce any output. No error message, no quotes, nothing.

Edit: here's content of my crumb file:

<!doctype html public "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
... ```

I wonder if its giving a 500 cause of the cookie file? Does that get generated okay?

@tomaszg7
Copy link

Cookie file:

# Netscape HTTP Cookie File
# https://curl.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

.yahoo.com	TRUE	/	TRUE	1682020284	GUCS	AQOsjLu3

@appatalks
Copy link
Sponsor

appatalks commented Apr 20, 2023

Yep, I think it might be that,

I was able to recreate your error with your cookies file, but my file worked okay:

$ curl --silent -b cookies.txt "https://query1.finance.yahoo.com/v1/test/getcrumb"
500 Error
vs
$ curl --silent -b cookies.txt.old "https://query1.finance.yahoo.com/v1/test/getcrumb"
fk22Yb6tiAY

However, my cookies.txt file has a lot more content in there, for example (removing what might be sensitive):

# Netscape HTTP Cookie File
# https://curl.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

.yahoo.com	TRUE	/	TRUE	0	A1S	d=-----
#HttpOnly_.yahoo.com	TRUE	/	TRUE	1713575851	A3	d=-------
#HttpOnly_.yahoo.com	TRUE	/	TRUE	1713575851	A1	d=-------

I am hoping this helps the dev :)

@pstadler
Copy link
Owner

That‘s mighty odd. Thanks for helping out each other here 👍🏻

@pstadler
Copy link
Owner

The HttpOnly cookies are missing. what‘s your curl version?

@tomaszg7
Copy link

curl 7.88.1 (x86_64-pc-linux-gnu) libcurl/7.88.1 OpenSSL/1.1.1t zlib/1.2.13 brotli/1.0.9 c-ares/1.19.0 nghttp2/1.51.0
Release-Date: 2023-02-20
Protocols: dict file ftp ftps http https imap imaps mqtt pop3 pop3s rtsp smtp smtps tftp
Features: AsynchDNS brotli HTTP2 HTTPS-proxy IPv6 Largefile libz NTLM SSL threadsafe TLS-SRP UnixSockets

@appatalks
Copy link
Sponsor

appatalks commented Apr 20, 2023

Here's mine for context:

$ curl --version
curl 7.87.0 (x86_64-pc-linux-gnu) libcurl/7.87.0 OpenSSL/3.0.7 zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.10.0 nghttp2/1.51.0
Release-Date: 2022-12-21
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Edit: went ahead and updated, and still working okay for me with:
$ curl --version
curl 8.0.1 (x86_64-pc-linux-gnu) libcurl/8.0.1 OpenSSL/3.0.7 zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.4 libpsl/0.21.2 (+libidn2/2.3.4) libssh2/1.10.0 nghttp2/1.51.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

@tomaszg7
Copy link

Tried running curl command without silent flag and nothing weird happens. Here's output

$  curl  --output /tmp/yyy --cookie-jar /tmp/xxx "https://finance.yahoo.com"   -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"   
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

The file /tmp/yyy is empty and /tmp/xxx is similar to the one I previously shown.

@tomaszg7
Copy link

Also tried updating curl and enabling some extra features, but no change:

curl 8.0.1 (x86_64-pc-linux-gnu) libcurl/8.0.1 OpenSSL/1.1.1t zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 c-ares/1.19.0 libidn2/2.3.4 nghttp2/1.51.0
Release-Date: 2023-03-20
Protocols: dict file ftp ftps http https imap imaps mqtt pop3 pop3s rtsp smtp smtps tftp
Features: AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL threadsafe TLS-SRP UnixSockets zstd

Tried it also on another machine (also a similar Gentoo box, but connected to different ISP) and the result was the same.

@appatalks
Copy link
Sponsor

This one is a real head scratcher for me

@pstadler
Copy link
Owner

@tomaszg7 please run this and paste the result:

curl -v "https://finance.yahoo.com" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"

@tomaszg7
Copy link

*   Trying 87.248.106.204:443...
* Connected to finance.yahoo.com (87.248.106.204) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=Sunnyvale; O=Oath Holdings Inc.; CN=*.api.fantasysports.yahoo.com
*  start date: Apr 10 00:00:00 2023 GMT
*  expire date: May 31 23:59:59 2023 GMT
*  subjectAltName: host "finance.yahoo.com" matched cert's "*.yahoo.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: finance.yahoo.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8]
* Using Stream ID: 1 (easy handle 0x55e9eda3d050)
> GET / HTTP/2
> Host: finance.yahoo.com
> user-agent: curl/7.88.1
> accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 307
< date: Fri, 21 Apr 2023 06:31:42 GMT
< strict-transport-security: max-age=31536000
< server: ATS
< cache-control: no-store
< content-type: text/html; charset=utf-8
< content-language: en
< expect-ct: max-age=31536000, report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-expect-ct-report-only"
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< location: https://guce.yahoo.com/consent?brandType=nonEu&gcrumb=BlQKcX8&done=https%3A%2F%2Ffinance.yahoo.com%2F
< set-cookie: GUCS=AQZUCnF_; Max-Age=1800; Domain=.yahoo.com; Path=/; Secure
< content-length: 0
<
* Connection #0 to host finance.yahoo.com left intact

@pstadler
Copy link
Owner

that’s it, you’re getting redirected:
< location: https://guce.yahoo.com/consent?brandType=nonEu&gcrumb=BlQKcX8&done=https%3A%2F%2Ffinance.yahoo.com%2F

@appatalks
Copy link
Sponsor

appatalks commented Apr 21, 2023

Very interesting!

Something about a consent page for GDPR consent. I wonder if its a geo-location thing that you might be running in-to that present that redirect..

Out of my comfort zone here, but there might be a way to inject a header to accept the consent cookie and then continue on with the call.

Would you be able to give this a try? :

curl -v "https://finance.yahoo.com" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8" -H 'cookie: notice_behavior=expressed,eu; notice_gdpr_prefs=0,1,2:1a8b5228dd7ff0717196863a5d28ce6c;'

(https://stackoverflow.com/questions/64859826/spring-webclient-how-to-skip-accept-cookie-consent-request)

Additional Ref:
https://stackoverflow.com/questions/64912909/getting-consent-cookies-for-website-login-using-python-requests-cmp-euconsent-v
https://adinserter.pro/faq/gdpr-compliance-cookies-consent

@pstadler
Copy link
Owner

Image from iOS

@pstadler
Copy link
Owner

people are reporting that the API is working again without crumb. Can you confirm this?

@tomaszg7
Copy link

Here's the output:

$ curl -v "https://finance.yahoo.com" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8" -H 'cookie: notice_behavior=expressed,eu; notice_gdpr_prefs=0,1,2:1a8b5228dd7ff0717196863a5d28ce6c;'
*   Trying 87.248.119.252:443...
* Connected to finance.yahoo.com (87.248.119.252) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Client hello (1):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Server hello (2):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Certificate (11):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, CERT verify (15):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Finished (20):
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=Sunnyvale; O=Oath Holdings Inc.; CN=*.api.fantasysports.yahoo.com
*  start date: Apr 10 00:00:00 2023 GMT
*  expire date: May 31 23:59:59 2023 GMT
*  subjectAltName: host "finance.yahoo.com" matched cert's "*.yahoo.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: finance.yahoo.com]
* h2h3 [user-agent: curl/7.87.0]
* h2h3 [accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8]
* h2h3 [cookie: notice_behavior=expressed,eu; notice_gdpr_prefs=0,1,2:1a8b5228dd7ff0717196863a5d28ce6c;]
* Using Stream ID: 1 (easy handle 0x5561bae30080)
> GET / HTTP/2
> Host: finance.yahoo.com
> user-agent: curl/7.87.0
> accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
> cookie: notice_behavior=expressed,eu; notice_gdpr_prefs=0,1,2:1a8b5228dd7ff0717196863a5d28ce6c;
> 
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 307 
< date: Fri, 21 Apr 2023 16:42:59 GMT
< strict-transport-security: max-age=31536000
< server: ATS
< cache-control: no-store
< content-type: text/html; charset=utf-8
< content-language: en
< expect-ct: max-age=31536000, report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-expect-ct-report-only"
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< set-cookie: notice_behavior=DELETE; Expires=Fri, 21 Apr 2023 16:32:59 GMT; Max-Age=0; Domain=finance.yahoo.com; Path=; Secure; HttpOnly
< set-cookie: notice_behavior=DELETE; Expires=Fri, 21 Apr 2023 16:32:59 GMT; Max-Age=0; Domain=yahoo.com; Path=; Secure; HttpOnly
< set-cookie: notice_gdpr_prefs=DELETE; Expires=Fri, 21 Apr 2023 16:32:59 GMT; Max-Age=0; Domain=finance.yahoo.com; Path=; Secure; HttpOnly
< set-cookie: notice_gdpr_prefs=DELETE; Expires=Fri, 21 Apr 2023 16:32:59 GMT; Max-Age=0; Domain=yahoo.com; Path=; Secure; HttpOnly
< location: https://guce.yahoo.com/consent?brandType=nonEu&gcrumb=Rxwfl3c&done=https%3A%2F%2Ffinance.yahoo.com%2F
< set-cookie: GUCS=AUccH5d3; Max-Age=1800; Domain=.yahoo.com; Path=/; Secure
< content-length: 0
< 
* Connection #0 to host finance.yahoo.com left intact

(It's from another machine with another ISP, but still in the same general area).

I confirm that API is working without the crumb, I reported that a few posts back ;) I just checked and it still works.

@appatalks
Copy link
Sponsor

appatalks commented Apr 21, 2023

people are reporting that the API is working again without crumb. Can you confirm this?

I am able to confirm too :)

commented out the following to test:


CRUMB_FILE="${SESSION_DIR}/crumb.txt"
 curl --silent -b "$COOKIE_FILE" "https://query1.finance.yahoo.com/v1/test/getcrumb" \
 "$CRUMB_FILE"

and adjusted to this:

fetch_quotes () { curl --silent -b "$COOKIE_FILE" "$API_ENDPOINT&fields=$fields&symbols=$symbols" }

Edit: So odd, I also removed the cookie and it works like that too lol

fetch_quotes () {
  #curl --silent -b "$COOKIE_FILE" "$API_ENDPOINT&fields=$fields&symbols=$symbols"

to

  curl --silent "$API_ENDPOINT&fields=$fields&symbols=$symbols"
}

pstadler added a commit that referenced this issue Apr 21, 2023
@pstadler
Copy link
Owner

pstadler commented Apr 21, 2023

Happily reverted this change!

I confirm that API is working without the crumb, I reported that a few posts back ;) I just checked and it still works.

@tomaszg7 I ignored your remark at that time, as I observed the API experiencing intermittent issues for days, if not weeks, before it eventually completely broke down. I thought that these disruptions might have been due to A/B testing or similar activities, and I didn't expect it to recover. It seems the joke's on me this time.

If the issue reoccurs, I would be grateful to reach out to both you and @servingbaby for assistance. Thank you for your collaboration!

@tomaszg7
Copy link

tomaszg7 commented May 5, 2023

Well, apparently we're back to square one. The script is again broken. Maybe they'll fix it once more in a day or two...

pstadler added a commit that referenced this issue May 31, 2023
* acquire yahoo finance session (fixes #33)

* make temp dir creation work cross platform
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants