Quote Sources

[bpschuck]

General discussion about exploring new sources for quote data.

[bpschuck]

Issues with YahooJSON Quote Source

Recent issues with the source used by the YahooJSON.pm module, there is a need to explore new data sources in case Yahoo blocks access to all of its URLs that return JSON. At some point in the future the module may need to be replaced by one that does a web scrape of pages such as https://finance.yahoo.com/quote/F?p=F&.tsrc=fin-srch.

Other sources such as the Marketstack API may be considered.

Those users with trading accounts with brokers like E*Trade may benefit from modules that utilize their APIs. These are a bit more complicated, since access mechanisms like SAML may be required. This would likely require access credentials that users would need to configure locally and securely.

[bpschuck]

Some APIs to check out

• https://twelvedata.com/pricing
• https://financeapi.net/
• https://polygon.io/

[vincentl]

All four of these examples are similar to IEXCloud - API key + REST API + json response. One interesting thing is Marketstack free tier doesn't support https - so if a user subscribes to the service, it would be nice to use the secure endpoint. This fits into the "feature" concept with Marketstack having both an "API_KEY" and "USE_HTTPS" feature field.

Any opinion on which of these sources to try first? I'm happy to open a PR for one of them.

[bpschuck]

I haven't looked too deeply into them yet. In my opinion which ever has the best free offering with respect to maximum number of queries and minimal throttling would be a good choice. Some time ago I started a World Trading Data module, and WTD was then renamed or acquired by Marketstack.

But rather than exploring new sources, maybe look at one of the two recent issues of modules recently not working. The Bourso one is likely a quick fix, just removing spaces from the "last" value. Tradeville is a bit more involved. Change of URL and likely a much different parse/extraction logic of the HTML.

Thanks.

[bpschuck]

Just posting this here so its handy. The v11 Yahoo API has multiple modules. Need to use more than one to get other data currently returned from the v6 API.

https://query2.finance.yahoo.com/v11/finance/quoteSummary/PDM?modules=summaryProfile,financialData,recommendationTrend,upgradeDowngradeHistory,earnings,defaultKeyStatistics,calendarEvents,esgScores,details

To replace the v6 URL we would not need all of these, but for EPS in addition to basic price data at least two are needed (I think summaryProfile and financialData).

[Wm-X]

I happened to look at FTfunds.pm this morning and I think with a bit of attention it might be usable for more than just UK Unit Trusts. If I do get it tidied I reckon it'll take at least 2 years for the people in charge here to implement the changes so no chance of it being overused :) (See Alphavantege.pm in issues)

[bpschuck]

Feel free to contribute, make those changes yourself and submit a pull request. Otherwise it may be 3 or 4 years before they get implemented.

[Wm-X]

There is at least one further issue, the providers of data don't know that F::Q is a once-a-day. Yahoo don't mind me getting data from them multiple times a day using their app or their website. I see adverts, news articles, etc and get the prices for free. Other projects have presumed prices per second are a right, AlphaVantage have reacted, Yahoo are reacting. How do we get back to "I want one price a day, hope that's OK" in the face of multiple hits per second from other projects ?

[bpschuck]

While the "once a day" retrieval is probably typical, don't assume that applies to every user of F::Q. I know of at least one day trader who uses F::Q to retrieve price data multiple times a day. If you have an issue with various data sources making access more difficult, take it up with them. Better yet, purchase your own feed from Factset or other provider and make that freely available for F::Q users.

[bpschuck]

@vincentl, et al,

Currently for AlphaVantage and the yet to be added TwelveData modules, we force throttling whether a user has a free or premium key. Maybe another parameter could be passed to those module, API_KEY_TYPE. Defaults to something that represents the free key, ("free", 0, ??), but if set to "premium" or some other value we make known as a choice (either by passing parameter or setting environment variable) the module bypasses the throttling mechanism.

[bpschuck]

Just a thought. I'm expecting Yahoo to pull the plug on using the v10/v11 URLs sometime down the road. Anybody want to start work on YahooWeb.pm that uses HTML::TableExtract and/or HTML::TreeBuilder (or other modules) to grab the needed data from URLs such as https://finance.yahoo.com/quote/AAPL?p=AAPL?

[bpschuck]

@vincentl - I am going to delete most, if not all, of the stale branches that are around 10 years old or more. Unless you can think of a really good reason to keep them around. Too bad GitHub did not have an "archive" option for branches. Although I can do something like:

git tag archive/<branchname> <branchname>
git branch -d <branchname>
git push --tags
git push origin :<branchname>

[bpschuck]

@vincentl, et al.,

I stumbled across MarketWatch as a viable candidate for relatively easy screen scraping.
Grabbing https://www.marketwatch.com/investing/stock/aapl outputs a nice parseable set of meta tags in the HTML head with the basic data:

<meta name="parsely-tags" content="PageType: Quotes, US:AAPL, Stock, US: U.S.: Nasdaq, NAS, page-Overview" />
<meta name="name" content="Apple Inc." />
<meta name="tickerSymbol" content="AAPL" />
<meta name="chartingSymbol" content="STOCK/US/XNAS/AAPL" />
<meta name="instrumentType" content="Stock" />
<meta name="exchange" content="U.S.: Nasdaq" />
<meta name="exchangeCountry" content="US" />
<meta name="exchangeIso" content="XNAS" />
<meta name="priceCurrency" content="USD" />
<meta name="price" content="$180.95" />
<meta name="priceChange" content="0.86" />
<meta name="priceChangePercent" content="0.48%" />
<meta name="quoteTime" content="Jun 2, 2023 4:00 p.m. EDT" />
<meta name="exchangeTimezone" content="(UTC-05:00) Eastern Time (US &amp; Canada)" />

The rest of the HTML contains a unordered list with other useful data:

<ul class="list list--kv list--col50">
                <li class="kv__item">
                    <small class="label">Open</small>
                    <span class="primary ">$181.03</span>
                    <span class="secondary no-value"></span>
                </li>
                <li class="kv__item">
                    <small class="label">Day Range</small>
                    <span class="primary ">179.26 - 181.78</span>
                    <span class="secondary no-value"></span>
                </li>      
                    <li class="kv__item">
                    <small class="label">EPS</small>
                    <span class="primary ">$5.89</span>
                    <span class="secondary no-value"></span>
                </li>      
.
.
.
</ul>

[bpschuck]

I have begun looking at screen scraping https://www.google.com/finance/quote/<symbol>. That page returns a list of URLs with the symbol for different exchanges. Assuming we want NYSE, NASDAQ, or NYSEAMERICAN (What Google uses for stocks on the American exchange), the module would then retrieve https://www.google.com/finance/quote/<symbol>:<exchange>. For the basic price data use HTML::TreeBuilder to get the <div tag with the data-last-price attribute. In that tag we have additional attributes data-exchange, data-currency-code, and data-last-normal-market-timestamp. The timestamp is in seconds since epoch. I already have a scratch script getting the link we want from the first link. The rest will be fairly trivial.

[bpschuck]

Yahoo appears to have closed the door on access to the finance.yahoo.com/v?? URLs that return JSON.

Sadly this means that the YahooJSON modules (there is one for currency rates) are going to be removed from the next release of F::Q.
If anyone can discover the trick to getting JSON securities data from Yahoo, please respond here or make the changes and submit a pull request.

I am going to be scarce the next couple weeks and not be much, if any, work on Finance::Quote. Sorry.
Thanks.

[jsjuni]

Longtime F::Q (via GnuCash) user here. Thank you.

Now that Yahoo has pulled the plug, I guess in the interim I'll revert to Alphavantage and deal with throttling. I'm willing to put some work into developing better alternatives.

I'm not fluent in Perl (despite sharing an office with Larry Wall at JPL around 1990) and don't know much about F::Q internals, but if someone sets up a stub that queries a source for something, I can do the grunt work to add whatever else it needs to get.

[jsjuni]

I remember addresses like that very well. In the mid-80s I was devvax!uclaee!ucla-an!steve or something close to that. In 1987 my advisor was on sabbatical in Holland and I set up really arcane sendmail rules to route email from our little DEC microVAX II via UUCP over SLIP to the Pyramid system in UCLA Electrical Engineering, from there via Internet to the Bitnet gateway at Penn State, from there to the EARN gateway somewhere in Europe, and from there to the VAX/VMS system at the University of Leiden. It was crazy but it worked.

I haven't talked to Larry in a long time. He was a lovely guy. I learned a lot from him.

Even if Alphvantage doesn't enforce it, it's clear they consider the rate limit a part of the deal. I'll take a look at what you've started for GoogleWeb.pm.

[bpschuck]

heh-heh. I also had to cobble internet->bitnet rules in Sendmail during my time at The American Physical Society (aps.org - physics stuff, not health/body stuff as most people assume). Fun stuff!

Larry and many of the other major players during Perl's heyday in the early/mid to late 90s and very early 2000s were all pretty much cool people. I've had electronic and IRL interactions with a bunch of them, all pretty much helpful, personable, and informative.

Generating a different API key would look to AV like a different user. Hence, no throttling. To the server side, it's a different user querying a different security.

GoogleWeb is decent one to look at for a source that isn't JSON. It will also help you to reverse engineer the module if you pull the HTML that the module parses for a particular security using curl or wget. Then view the HTML while reviewing the code.

Have fun!

[bpschuck]

Fudge! Seems AV does throttle based on source IP. That sucks.

[jsjuni]

I don't see BPSCHUCK/Finance-Quote-1.57_01.tar.gz on CPAN.

[bpschuck]

It's not showing on MetaCPAN or a normal list of modules because of the _01. Indicates it is not a production indexed version. If you look at the author tree on CPAN web you will find https://www.cpan.org/modules/by-authors/id/B/BP/BPSCHUCK/.

On a system with CPAN installed, it can be installed with the command cpan BPSCHUCK/Finance-Quote-1.57_01.tar.gz or cpanm BPSCHUCK/Finance-Quote-1.57_01.tar.gz if CPAN Minus is installed. Since then I pushed _02 to CPAN with a few other fixes.

[jsjuni]

I forked the repo and set up a little test rig set up so I can make changes to GoogleWeb.pm and inspect the JSON returned by some simple queries.

The last close, day range, etc. appear in blocks like this:

<div class="gyFHrc">
  <span data-is-tooltip-wrapper="true">
    <div class="mfs7Fc" jscontroller="e2jnoe" jsaction="mouseenter:tfO1Yc; focus:AHmuwe; blur:O22p3e; mouseleave:JywGue;    touchstart:p6p2H; touchend:yfqBxc;mlnRJb:fLiPzd;" aria-describedby="c522" data-tooltip-x-position="2" data-tooltip-anchor-boundary-type="2">Previous close</div>
    <div class="EY8ABd-OWXEXe-TAWMXe" role="tooltip" aria-hidden="true" id="c522">The last closing price</div>
  </span>
  <div class="P6K39c">$60.81</div>
</div>

I don't know anything about front-end code. Are those class attributes stable? Would it make sense to find all divs with class="ghFHrc" and build a hash mapping titles (e.g., "Previous close") to values (e.g., "$60.81") and then transforming as necessary to map to the F::Q result hash?

[jsjuni]

Or just iterate over them with a switch statement. The real question is a reliable means of finding those blocks.

[bpschuck]

The URL query in GoogleWeb, YahooWeb, and others does not return JSON, which is much easier to parse (The output of curl 'https://www.alphavantage.co/query?function=GLOBAL_QUOTE&symbol=IBM&apikey=demo' is JSON) . I also just recently had to tweak YahooWeb because the original code was finding the wrong tag for single (and probably double) character symbols, therefore returning bad price data.

Perl also does not have a switch statement. Well, kind of. It has when and given.

You might not be aware that the GoogleWeb, MarketWatch, and YahooWeb modules were hastily added in the last couple months to have alternate sources available as Yahoo seems to be blocking access URLs that return easily parsed JSON (see recent changes to the YahooJSON module over the last few months). The fix to YahooWeb I just mentioned is likely not going to be the last. These and other modules, while working, could probably benefit from more specific parsing and additional error checking for when parsing fails to find some values.

As for class attributes being stable, unfortunately that is an assumption for probably all the modules that parse HTML and CSS instead of defined JSON output. For many modules, history has shown they usually are stable usually don't break unless the web pages undergo somewhat major redesigns.

[bpschuck]

So I added a simple way to count the amount of times each fetch method was called in Quote.pm. It was in the v1.57 release. Since then here is the running count for the month of July 2023. Please note however, that early on I discovered that a single address was hitting the ukfunds method continuously and I stopped including that in the running totals early in the month.

7447	yahoo_json
6151	xetra
5633	ukfunds
4035	alphavantage
3720	yahooweb
 739	usa
 638	bourso
 624	googleweb
 624	mstaruk
 485	marketwatch
 444	aex
 426	tsp
 405	tradegate
 393	australia
 369	europe
 351	fool
 346	bloomberg
 341	asx
 290	ftfunds
 184	deka
 184	nseindia
 182	nasdaq
 178	fidelity
 175	amfiindia
 156	tsx
 135	indiamutual
 128	fidelity_direct
 127	unionfunds
 122	nyse
 118	canada
 113	bseindia
  78	morningstarch
  71	hu
  69	bse
  68	finanzpartner
  67	canadamutual
  64	tiaacref
  60	sinvestor
  56	tmx
  55	fundlibrary
  52	goldmoney
  52	tesouro_direto
  49	nzx
  47	dwsfunds
  45	morningstarjp
  43	troweprice_direct
  41	troweprice
  40	comdirect
  40	morningstaruk
  33	aufunds
  33	seb_funds
  32	india
  29	fondsweb
  28	six
  27	onvista
  27	twelvedata
  26	known_currencies
  25	fundata
  24	treasurydirect
  23	dutch
  23	za
  18	asegr
  18	france
  16	fetch_live_currencies
  13	bet
  13	bvb
  13	oslobors
  12	cse
  12	greece
  12	morningstarau
  11	bamosz
  10	hufund
  10	iexcloud
  10	romania
   9	hungary
   9	hustock
   9	tradeville

[bpschuck]

@jsjuni,

How goes the testing? If you want to get your hands dirty working on something that would be helpful, have a look at #324. The Bloomberg module recently broke due to web site changes. Aside from need to get and use cookies, the format of the HTML has changed.

Thanks.

[jsjuni]

I'm visiting family for a few days and haven't had much time to look. Will be back home Wednesday.

[bpschuck]

I did some poking around MarketStack. Contrary to what @vincentl said earlier, the MarketStack API is available through HTTPS. While the free tier 100 requests per month may seem limiting at first, they do offer a end-of day API that allows multiple symbols. many F::Q users (including GnuCash users) only update stock prices once a day. Thirty or thirty-One requests per month falls way below the 100 per month limitation.
I am likely going to release Finance::Quote v1.58 next Saturday, with no further changes except for release related housekeeping for the next week. But I may start working on a MarketWatch.pm module for v1.59.

[bpschuck]

The MS API shares the same shortcoming as AlphaVantage. No currency indicator. So most would assume the GBP.XLON may be GBP when the values returned are GBp/GBX. We could do the same kludge/workaround just added to AV. User adds '.X' to the symbol, the module drops the '.X' for the lookup and scales the price data.

[rsbrux]

I have no idea whether they provide a convenient API, but a website I find very useful for both funds and stocks is finanzen.net.

[bpschuck]

@rsbrux - It does look promising. But someone will need to step up and create a module.

[bpschuck]

Checked today, and Marketwatch.com no longer works...

It looks like stockdata.org is may be a decent API Key based source. Their free plan is 100 requests daily. If there are no volunteers, I may start cobbling a module together next week.

[bpschuck]

Another possible API source with 100 daily requests - https://www.marketdata.app/#pricing