New _chart module feedback

[gadicc]

Following from #238, #239, #328. In particular:

I prefer the way @PythonCreator27 outputs the data (arrays with { date: ... } rather than Objects with timestamp keys).

and let's discuss any other feedback.

[huned]

Agreed - I thought the output format @PythonCreator27 chose was generally easy to work with.

FWIW my client code combines the historical prices, dividends, and splits into a giant 2D array so I can operate on the entire chunk of data efficiently, but I'm not sure that's a common enough use case to consider just having the chart module output that format. I think as it is now is better because it's more explicit (and thus understandable) to users.

[gadicc]

Thanks, @huned.

Having also had some time to think about this, I've decided I'd like to offer both formats. With certain charting libraries, Yahoo's original input may indeed be easier to work with, and I'd like to avoid a situation where to convert to the array format only for the user to then need to write their own conversion code just go get back to the data we already had.

I imagine it will turn out similar to quote, so say:

yf2._chart('AAPL', { period1, return: "array" }); // default
yf2._chart('AAPL', { period1, return: "object" });

I was also considering merging into a single array too, and maybe we can indeed have an option for that. condense maybe? What exactly does your array look like, like this?

[
  { date: type: 'quote', /* ... */ },
  { date, type: 'split', /* ... */ },
  { date, type: 'dividend', /* ... */ },
 ]

Thanks again for all your patience and efforts to get things moving on this again... owing to the _ prefix I think / hope we can get this finished up and released in the coming days.

[huned]

Thanks for your work on this @gadicc.

I transform both quote and _chart output as follows:

[
   ['ticker', 'date',  'open', 'close', 'low', 'high', 'volume', 'dividend', 'split'],
   // and then rows of data, e.g.
   ['MMM', new Date(...), #, #, #, #, #, # or undefined, # or undefined],
   ...
]

Which I write to stdout as CSVs. I'm doing it this way so I can take advantage of many common command line tools (eg, cut, column) and so I can efficiently import into a SQL database. My gut tells me that my use case is probably uncommon and probably shouldn't be supported until more people want it? The hesitation is that this creates a lot more code, tests, and maintenance workload for you for a codepath that might not get used by all that many people.

Are there more users you can think of who might be able to provide feedback on whether you should build it or not?

[gadicc]

Oh yes, that's right, you did you say a 2D array. Ok, on the same page now.

Yes, I agree, that's probably a less commonly used format, and I thank you for your consideration of the project maintainers :)

Umm, I believe the only people interested in this are you, @PythonCreator27, @ZehCoque (hi from July! 😅) and now @felixsanz in #334 (intraday interval support).

I'm happy this is still released as _chart, the API is a bit finicky. Just from playing with the values a bit, came up with quite a few cases that needed schema fine tuning. Anyway, since you seem happy running off devel, let me know if it handled everything you threw it at, and we can at least get this into an interim release.

Particularly, tradingPeriods is a bit weird, and maybe we should transform it to something more useful. For example in this test):

{
              "tradingPeriods": {
                "pre": [
                  [
                    {
                      "timezone": "EST",
                      "start": 1637571600,
                      "end": 1637591400,
                      "gmtoffset": -18000
                    }
                  ]
                ],
                // ...
}

So it's in a double array. We should try find an example of more than 1 so we know where the other entries go. (That's in the original response from Yahoo, we transform the dates of course).

[huned]

Thanks @gadicc! I'll update my client code to use the devel branch and the new _chart module and will report back on any issues that arise over the next week or so.

Re the tradingPeriods property. I initially thought the array-of-array-containing-one-element thing might be related to securities traded in multiple markets, but I looked at a few (BIDU, BABA, BHP, BP) and all I saw had just the same one element nested arrays as you've observed.

However, an observation: The tradingPeriods data is ALMOST the same as currentTradingPeriods. You'll see that the timestamps are actually different between tradingPeriods and and currentTradingPeriod. However, in all cases, I see that the durations of each period are exactly the same:

(tradingPeriods.pre.end - tradingPeriods.pre.start) / 60 / 60 = 5.5 // 5.5 hours
(currentTradingPeriod.pre.end - currentTradingPeriod.pre.start) / 60 / 60 = 5.5

(tradingPeriods.regular.end - tradingPeriods.regular.start) / 60 / 60 = 6.5
(currentTradingPeriod.regular.end - currentTradingPeriod.regular.start) / 60 / 60 = 6.5

(tradingPeriods.post.end - tradingPeriods.post.start) / 60 / 60 = 4
(currentTradingPeriod.post.end - currentTradingPeriod.post.start) / 60 / 60 = 4

Yet, also, tradingPeriods timestamps and their corresponding currentTradingPeriod counterpart are offset by 24 hours

(tradingPeriods.post.end - currentTradingPeriod.post.end) / 60 / 60 = 24
// same 24 hour offset for `regular`, and `pre` as well

I'm not sure what all this means, but figured it'd be helpful to post the observation.

[gadicc]

Thanks, @huned, for your observations. Glad to have that recorded here. Ok sounds good, let's see what else comes up with some use 🙏

[gadicc]

For those following, this came out (still as _chart) in 2.1.0 today.

[huned]

Thanks @gadicc 🥇 Been using it daily without problems.

[gadicc]

Ok, great! Thanks amazing news! And really helpful to know. Thanks for reporting back 🙏

[glaucoheitor]

Hey guys, I started following this closely now and I'm super interested in it as well, my main goal in the future is to be able to request historical data of multiple symbols in one call to avoid Unauthorized error/bans.

I even contacted aforementioned @ZehCoque directly as we're both from the same country and have our own edge cases when it comes to Brazil's stock exchanges (.SA). We are working on similar projects and can benefit from combined knowledge.

I've been delving into the code of this project for the past couple hours and I feel more confident now. I'm not well versed into TypeScript yet but I can help on whatever is needed. I actually never participated actively in the developing of a community library (lone wolf programmer), so this could be a great learning for me as well.

[glaucoheitor]

So, I am getting a validation error with symbol NUBR33.SA, sometimes Yahoo have information, but sometimes, like now, they have no historical info, so firstTradeDate is null while it's expecting date.

I can, of course, skip validation or catch the error, but I just wanted to point that out if you decided to do something about it.

The following result did not validate with schema: #/definitions/ChartResultObject
[
  {
    keyword: 'yahooFinanceType',
    message: "Expecting date'ish but got null",
    params: { schema: 'date', data: null },
    instancePath: '/meta/firstTradeDate',
    schemaPath: '#/properties/firstTradeDate/yahooFinanceType',        
    data: null
  }
]

[glaucoheitor]

Hey guys, so, still talking about NUBR33.SA, I found something weird that I'm trying to figure out what Yahoo is doing.

If I request a yahooFinance.historical it only gives me one day (the same happens on the the website).

Now, if I request a yahooFinance._chart with 1d intervals, it only return one day (even with longer date ranges), but if I send a request with 5m up to 4h (not 1m) it return all the data with no problem.

[brandonros]

https://query1.finance.yahoo.com/v8/finance/chart/SPY?symbol=SPY&period1=1673409759&period2=1673582559&useYfid=true&interval=1m&includePrePost=true&events=div%7Csplit%7Cearn&lang=en-US&region=US&crumb=dOh3M4TA.5w&corsDomain=finance.yahoo.com

If you change period1 from 1673409759 (Tuesday, January 10, 2023 11:02:39 PM GMT-05:00) to 1673533800 (Thursday, January 12, 2023 9:30:00 AM GMT-05:00), the API doesn't properly return volume (0 instead of 7760866).

period1 as 1673533800:

  {
    "i": 0,
    "timestamp": "2023-01-12T14:30:00.000Z",
    "close": 396.7099914550781,
    "open": 396.6700134277344,
    "high": 396.9649963378906,
    "low": 396.3999938964844,
    "volume": 0
  },

versus

period1 as 1673409759:

{
    "i": 1234,
    "timestamp": "2023-01-12T14:30:00.000Z",
    "close": 396.7099914550781,
    "open": 396.6700134277344,
    "high": 396.9649963378906,
    "low": 396.3999938964844,
    "volume": 7760866
  },

Not a problem with your library since it's the underlying API, but makes using the API "confusing".