During horizontal scrolling, sometimes daily candles disappear

[haimivan]

Hi,

when the resolution of the plot is at the point where the daily candles are just visible, I observe their candle bodies to disappear during horizontal scrolling.

It would be very helpful if they would be visible in all situations.

Please find below the code and the referring video.

Thanks in advance!

code

#!/usr/bin/env python3

from datetime import date, timedelta
import finplot as fplt
import pandas as pd
import scipy.optimize
import yfinance as yf

now = date.today()
start_day = now - timedelta(days=59)
df = yf.download('GOOG', start_day.isoformat(), now.isoformat(), interval='5m')

import pytz

fplt.display_timezone = pytz.timezone('UTC')

# resample to daily candles, i.e. five 90-minute candles per business day
dfd = df.Open.resample('D').first().to_frame()
dfd['Close'] = df.Close.resample('D').last()
dfd['High'] = df.High.resample('D').max()
dfd['Low'] = df.Low.resample('D').min()

ax = fplt.create_plot('Alphabet Inc.', rows=1)

# plot down-sampled daily candles first
if True:
    daily_plot = fplt.candlestick_ochl(dfd.dropna(), candle_width=5)
    daily_plot.colors.update(dict(bull_body='#bfb', bull_shadow='#ada', bear_body='#fbc', bear_shadow='#dab'))
    daily_plot.x_offset = 3.03  # resample() gets us start of day, offset +1.1 (gap+off center wick)

# plot high resolution on top
fplt.candlestick_ochl(df[['Open', 'Close', 'High', 'Low']])

fplt.autoviewrestore(True)

fplt.show()

video + .ini file:

simplescreenrecorder-2021-01-02_07.50.26.zip

[highfestiva]

Looks like Level of Detail (LoD) is reducing the amount of drawn bodies. To allow for more bodies being drawn:

fplt.lod_candles = 30000

[haimivan]

Thanks a lot, this looks already better! Thanks!

I do not want to stress any functionality / code / use case etc., but if I take more data, it happens to me, that also after increasing the level of detail, something strange happens (namely: all candles disappear).

Even taking a value of 30_000_000 does not help here. And I can observe, that the faster I scroll, the more this effect becomes apparent.

This is the referring code and a small video for that.

Is there perhaps anything you could also do about this?

from datetime import date, timedelta
import finplot as fplt
import pandas as pd
import scipy.optimize
import yfinance as yf
import pytz

now = date.today()
start_day = now - timedelta(days=59)
df = yf.download('GOOG', start_day.isoformat(), now.isoformat(), interval='5m')


df.reset_index(inplace=True, drop=True)

if True:
    for runner in range(5):
        df = df.append(df)
df.reset_index(drop=True, inplace=True)
datelist = pd.date_range(pd.to_datetime('2020-12-23 00:00'), periods=df.shape[0], freq="5min").tolist()
df.index = datelist

print("df.shape=", df.shape)

fplt.display_timezone = pytz.timezone('UTC')

# resample to daily candles, i.e. five 90-minute candles per business day
dfd = df.Open.resample('D').first().to_frame()
dfd['Close'] = df.Close.resample('D').last()
dfd['High'] = df.High.resample('D').max()
dfd['Low'] = df.Low.resample('D').min()

ax = fplt.create_plot('Alphabet Inc.', rows=1)

print("BEFORE fplt.lod_candles=", fplt.lod_candles)
fplt.lod_candles = 30_000
print("AFTER fplt.lod_candles=", fplt.lod_candles)

# plot down-sampled daily candles first
if True:
    daily_plot = fplt.candlestick_ochl(dfd.dropna(), candle_width=int(24*60/5))
    daily_plot.colors.update(dict(bull_body='#bfb', bull_shadow='#ada', bear_body='#fbc', bear_shadow='#dab'))
    daily_plot.x_offset = 3.03  # resample() gets us start of day, offset +1.1 (gap+off center wick)

daily_plot.datasrc.is_sparse = True

# plot high resolution on top
fplt.candlestick_ochl(df[['Open', 'Close', 'High', 'Low']])

fplt.autoviewrestore(True)

fplt.show()

simplescreenrecorder-2021-01-04_22.57.13.zip

[highfestiva]

That's weird, I'm not able to reproduce, so must be something Macish. Are any errors printed out? Does the plots start working again if you scroll slower, or are they forever gone? I'm guessing this has something to do with how Qt's event loop is setup with threads etc. in MacOS. One possibility might be to only handle a keypress if the previous is done.

Could you try to make the above change and create a pull request? Thanks!

[haimivan]

Hi,

well, it is a Linux operating system I am working with (Ubuntu 16.04) under python 3.6.

$ pip freeze|grep -i qt
PyQt5==5.13.0
PyQt5-sip==12.8.1
PyQt5-stubs==5.14.2.2
pyqtgraph==0.11.1
QtAwesome==0.6.0
qtconsole==4.6.0
QtPy==1.9.0
QTPyLib==1.5.83

(update of PyQt5 to 5.15.2 seems to be impossible for me presently due to some error related to AttributeError: module 'sipbuild.api' has no attribute 'prepare_metadata_for_build_wheel' for PyQt5 5.15.0

Are any errors printed out?

Unfortunately, no errors are printed out.

Additionally, I ran the code in debug mode and set breakpoints in finplot whenever an exception is ignored with a pass or something similar.

At the following position this happens:

user32 = ctypes.windll.user32 (=>AttributeError: module 'ctypes' has no attribute 'windll')
(and at the creation of the .finplot directory which already exists...)

Does the plots start working again if you scroll slower, or are they forever gone?

If I go back to the approximate position where the candles disappeared they reappear (most the times).

Could you try to make the above change and create a pull request? Thanks!

Unfortunately, I do not see myself presently able to perform this kind of change... :-(

[highfestiva]

I'm using PyQt5==5.15.2 and Python 3.9.0. I've never seen this error before, and can't help you. Sorry!

[pssolanki111]

@haimivan

user32 = ctypes.windll.user32 (=>AttributeError: module 'ctypes' has no attribute 'windll')

if you're on linux, then ctypes.windll is a windows specific thing, which seems to be the reason behind this AttributeError

[highfestiva]

The user32 Windows library is only used for getting the screen width for setting the LoD to a value sensible for very wide screens. It should not affect you.

[highfestiva]

@haimivan Did you try force-installing a newer pyqt?

[haimivan]

@highfestiva
yes, I did so.

Now I have python3.8 active and PyQt5==5.15.2.

pip freeze|grep -i qt
PyQt5==5.15.2
PyQt5-sip==12.8.1
PyQt5-stubs==5.14.2.2
pyqtgraph==0.11.1

For my code reported above, it looks now better. That's very good!!!

I made a self-containing reproducible example where I can still see one thing:

1. zooming several times into the canvas
2. going to the right end of the data
3. and then pressing the cursor left key slowly until I reach the left end of the canvas.

After that, the very left (sparse) candles have disappeared (although the scaling is done accordingly already, what one can see as the continuous candles seem to become "smaller" - space above them is reserved for the sparse candles to appear).

Please find my code and the video in the following:

import finplot as fplt
import numpy as np
import pandas as pd
import time
np.random.seed(1)

ax1 = fplt.create_plot('title', rows=1)

length = 115000
# length = 55000

prices = (pd.Series(np.random.random(length)).rolling(30).mean() + .5).dropna()
dates = pd.date_range('01:00', periods=len(prices), freq='5min')

df = pd.DataFrame(prices)
df.index = dates

df1h = df.resample('1h').ohlc()
df1h.columns = ['open', 'high', 'low', 'close']

df1D = df1h.resample('1D', closed='left', label='left').agg({'open': 'first',
                                                             'high': 'max',
                                                             'low': 'min',
                                                             'close': 'last',
                                                             })

df1D.columns = ['open', 'high', 'low', 'close']

df1D_ = 1.12 * df1D.copy()

dfSomeCandles = df1D_.iloc[0:10]
dfSomeCandles = dfSomeCandles.append(df1D_.iloc[-20:-10])
dfSomeCandles = dfSomeCandles*1.2

print("df1D.reset_index().iloc[0].iloc[0], df1D.reset_index().iloc[-1].iloc[0], df1D.shape=", df1D.reset_index().iloc[0].iloc[0], df1D.reset_index().iloc[-1].iloc[0], df1D.shape)


COLUMN_ORDER_FOR_FINPLOT_LIST = ['open', 'close', 'high', 'low']

plt1 = fplt.candlestick_ochl(df1h[COLUMN_ORDER_FOR_FINPLOT_LIST], ax=ax1)


daily_plot = fplt.candlestick_ochl(df1D[COLUMN_ORDER_FOR_FINPLOT_LIST], candle_width=24, ax=ax1)
daily_plot.colors.update(dict(bull_body='#bfb', bull_shadow='#ada', bear_body='#fbc', bear_shadow='#dab'))

daily_plot2 = fplt.candlestick_ochl(dfSomeCandles[COLUMN_ORDER_FOR_FINPLOT_LIST], candle_width=24, ax=ax1)
daily_plot2.colors.update(dict(bull_body='#1f77b4', bull_shadow='#1f77b4', bear_body='#ffcc00', bear_shadow='#ffcc00'))

if False:
    daily_plot_ax2 = fplt.candlestick_ochl(df1D[COLUMN_ORDER_FOR_FINPLOT_LIST], ax=ax2)
    daily_plot_ax2.colors.update(dict(bull_body='#bfb', bull_shadow='#ada', bear_body='#fbc', bear_shadow='#dab'))


fplt.lod_candles = 30000

daily_plot2.datasrc.is_sparse = True
daily_plot.datasrc.is_sparse = True

# fplt.autoviewrestore()
fplt.show()

output:

df1D.reset_index().iloc[0].iloc[0], df1D.reset_index().iloc[-1].iloc[0], df1D.shape= 2021-01-07 00:00:00 2022-02-10 00:00:00 (400, 4)

simplescreenrecorder-2021-01-07_14.46.18.mp4

[highfestiva]

That is Qt not repainting things that ever went off-screen. I'll see what I can do to circumvent.

[highfestiva]

With aa47248 I managed to get it a lot better, but the remaining issue seems to be internal Qt or pyqtgraph, so considering this fixed.

[haimivan]

Thanks a lot for your effort. It must be quite frustrating to depend on the underlying 'infrastructure' in such a way.

Just for your information:

The left side of the data looks quite good now. Unfortunately, the right side still does not seem to work as good as the left side.

I executed the above script in "debug" mode (and just activated initially: fplt.autoviewrestore() as compared to the script above).

Observations:

• On the right side, the sparse candles unfortunately do not appear when scrolling slowly there
• during this test, your new code
  (lines:

                # we need something that renders
                df = self.df.loc[self.df.iloc[:,self.col_data_offset].notna(), :]

)
does not get accessed.

Here is the video (and leave it, if it is too complicated...)

simplescreenrecorder-2021-01-11_08.15.06.mp4

[highfestiva]

Added a fix for it in 4fc9640.

[haimivan]

Thanks a lot.

That fix seems to solve the missing candles when moving slowly!

Very good! 👍