Data caching issue

[ASDWQad]

I have customized a Binance data source, but the API can only retrieve 1,000 data points each time. I have cached all the data from 2023/1/1 to 2024/1/1, but when calling the cache, it does not merge the kline return, only returning the cached 1,000 data points.

Can this problem be solved? Or I can only download the data to the local, and then customize the data source to import the local data.

[edtechre]

Hi @ASDWQad,

You should be able to reuse the existing cache mechanism by extending DataSourceCacheMixin. Can you post your code here?

[tsunamilx]

Hi @edtechre

I have another question about the cache:
When I query for some small amount of symbols (like 100), the cache works, but when I query for large amount of symbols (like 1000), it always calls the fetch function (meaning cache is not used). Is there any limit on the cache (like max size limit) so that the early data is being evicted?

And when I query the 1000 symbols one by one, it actually loads everything from cache.

# querying 1000 symbols at once fails the cache
d = ds.query(symbols, start, end, timeframe)

# querying 1000 symbols one by one works
for s in symobls:
  d = ds.query(s, start, end, timeframe)

[edtechre]

Thanks @tsunamilx, I am not aware of any cache limit. I will need to test this myself and report back.

[edtechre]

Hi @tsunamilx,

Looking at the code, there is no difference between calling query with all symbols or individually for each symbol. See data.py.

Perhaps your cache was invalidated somehow?