Daylocator causes frozen computer when used with FuncAnimation

[Spock-to-Kirk]

Bug report

This is an existing bug that is already known but it might be prevented by using extra initialization steps.
I hope someone can guide me.

Error: "Locator attempting to generate 4018 ticks ([10775.0, ..., 14792.0]), which exceeds Locator.MAXTICKS (40)."

Day locator is generating far too many day or hour ticks (thousands). This is causing to slow down the code to freeze for minutes or to completely freeze computer. I have seen Daylocator working properly so it may only happen in conjunction with FuncAnimation. My code must update the graph every few seconds. I use FuncAnimation wich an update function to do that.
I think the combination of both causes this issue. If I cannot solve it I must stop using Matplotlib. More people might be looking for it. That is the reason for posting.

Conditions and versions

Using OS Ubuntu 18.04
Python: 3.6.9
Matplotlib: 3.3.3
Matplotlib backend: Qt5Agg

The bug

The bug is explained in detail here:
runningcode11 and here stackoverflow

It is caused by missing initialization in rrulelocator. I tried to set this by running SecondLocator, MinuteLocation, HourLocator before. This is not working.

Code for reproduction

#!/usr/bin/python3
# mathplotlib example


import warnings
warnings.filterwarnings( "ignore", module = "matplotlib\..*" )

import datetime
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
from matplotlib.animation import FuncAnimation


# timing variables
update_int = 100    # update interval in ms
rep_del = 1000      # delay after each animation run


# np line space does not support datetime objects. Function to create a np line space.
# start, end are date_time objects
def date_linespace(start, end, steps):
    
    delta = (end - start) / steps
    increments = range(0, steps) * np.array([delta]*steps)
    return start + increments


def init():
    print("init")
    # ax is of type AxesSubplot
    #ax.set_xlim(0, 2*np.pi)
    
    ax.set_xlim(start_datetime_object, end_datetime_object)
    print("x limit")
    print(ax.get_xlim())

    ax.set_ylim(0, 3)
    print("y limit")
    print(ax.get_ylim())

    return line,    # the comma makes the return value iterable



def update(frame):

    # frame is numpy float64
    print("this is a frame")
    y = fake_y_data.pop(0)    # take one element from the array

    xdata.append(frame)
    #ydata.append(np.sin(frame) + y_offset)
    ydata.append(y)
    
    print("x data")
    print(xdata)
    print("y data")
    print(ydata)
    line.set_data(xdata, ydata)

    return line,


# use two sub plots
fig, ax = plt.subplots(figsize=(5, 4))  # size of graphic
xdata, ydata = [], []
fake_y_data = [2,1,0,1,1,3,0,1,2,3,2,3,0,3,3,1,2,0,2,1]   # test data

# format line
line, = plt.plot([], [], '-r')     # - = solid line, r = red. Must be iterable ","

# circumventing a bug //// this is not working
seclocator = mdates.SecondLocator(interval=2000) 
minlocator = mdates.MinuteLocator(byminute=range(60), interval=1000) 
hourlocator = mdates.HourLocator(byhour=range(24), interval=1200)

seclocator.MAXTICKS  = 10
minlocator.MAXTICKS  = 10
hourlocator.MAXTICKS  = 10


# Major ticks every day
fmt_day = mdates.DayLocator(interval=1)
fmt_day.MAXTICKS = 40  # has NO effect
ax.xaxis.set_major_locator(fmt_day)

# Text in the x axis will be displayed in 'YYYY-mm-dd' format.
ax.xaxis.set_major_formatter(mdates.DateFormatter('%Y-%m-%d'))

# create x time series data: start > end in n frames
# init frames. X- coordinates of type: numpy.ndarray
start_datetime_object = datetime.datetime(2019, 8, 8)
end_datetime_object = datetime.datetime(2019, 8, 23)
frames = date_linespace(start_datetime_object, end_datetime_object, 10)

ani = FuncAnimation(fig, update, frames, init_func=init, blit=True, interval=update_int, repeat_delay=rep_del, repeat=False)

plt.title("my test graph")
plt.xlabel("x-axis")
plt.ylabel("y-axis")

# Right corner coordinates box
ax.format_xdata = mdates.DateFormatter('%Y-%m-%d')
ax.format_ydata = lambda x: f'kwh{x:.2f}'  # Format x value in kwh
ax.grid(True)

# Rotates and right aligns the x labels, and moves the bottom of the
# axes up to make room for them.
fig.autofmt_xdate()                                    # will rotate 30 deg to fit
plt.setp(ax.xaxis.get_majorticklabels(), rotation=30)  # rotate manually (not required)

# show the plot and loop
plt.show()

Actual outcome

The support for Qt4  was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
  module = self._original_import(*args, **kw)
Locator attempting to generate 4018 ticks ([10775.0, ..., 14792.0]), which exceeds Locator.MAXTICKS (40).
Locator attempting to generate 4018 ticks ([10775.0, ..., 14792.0]), which exceeds Locator.MAXTICKS (40).
Locator attempting to generate 4018 ticks ([10775.0, ..., 14792.0]), which exceeds Locator.MAXTICKS (40).
Locator attempting to generate 4018 ticks ([10775.0, ..., 14792.0]), which exceeds Locator.MAXTICKS (40).

Expected outcome

Proper initialization should avoid create all those ticks an memory corruption that is coming from it.

[dopplershift]

There's not so much an issue with animations here, but an issue with set_data(). I boiled it down to:

#!/usr/bin/python3
import datetime
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.dates as mdates

# np line space does not support datetime objects. Function to create a np line space.
# start, end are date_time objects
def date_linespace(start, end, steps):
    delta = (end - start) / steps
    increments = range(0, steps) * np.array([delta]*steps)
    return start + increments

# use two sub plots
fig, ax = plt.subplots(figsize=(5, 4))  # size of graphic

# Major ticks every day
fmt_day = mdates.DayLocator(interval=1)
fmt_day.MAXTICKS = 40  # has NO effect
ax.xaxis.set_major_locator(fmt_day)

# create x time series data: start > end in n frames
# init frames. X- coordinates of type: numpy.ndarray
start_datetime_object = datetime.datetime(2019, 8, 8)
end_datetime_object = datetime.datetime(2019, 8, 23)
frames = date_linespace(start_datetime_object, end_datetime_object, 10)
fake_y_data = [2,1,0,1,1,3,0,1,2,3,2,3,0,3,3,1,2,0,2,1]   # test data

# format line
line, = plt.plot([], [], '-r')     # - = solid line, r = red. Must be iterable ","
line.set_data(frames, fake_y_data[:len(frames)])

# This works fine
#line, = plt.plot(frames, fake_y_data[:len(frames)], '-r')

plt.show()

Plotting the data normally works fine, but it looks like set_data is doing something weird (probably related to our datetime/unit handling if I had to guess).

[jklymak]

Well, its even simpler:

import matplotlib.pyplot as plt
import numpy as np
import matplotlib.dates as mdates

fig, ax = plt.subplots()
fmt_day = mdates.DayLocator(interval=1)
fmt_day.MAXTICKS = 40  # has NO effect
ax.xaxis_date()
ax.xaxis.set_major_locator(fmt_day)

plt.show()

And this is just a consequence of 2000-2010 being the default axes limits for dates. To get around this, set your initial limits sensibly:

import matplotlib.pyplot as plt
import numpy as np
import matplotlib.dates as mdates

fig, ax = plt.subplots()
fmt_day = mdates.DayLocator(interval=1)
fmt_day.MAXTICKS = 40  # has NO effect
ax.xaxis_date()
ax.xaxis.set_major_locator(fmt_day)

ax.set_xlim(np.datetime64('2019-01-01'), np.datetime64('2019-02-01'))

plt.show()

I don't think there is anything we can do here. The default limit is the default limit, and since you have an empty axes, that is what you have to work with on the first draw. I would load the first draw with the first data point...

[Spock-to-Kirk]

@ jklymak Thank you for replying.

Yes that worked!!! Great

This is a big step for all who have similar issues. Let's describe correctly for future reference.
Time consumption by over populating x-axis by far too many ticks. It is caused by empty or incorrect defaults for tick ranges.
Solved by just setting the x-axis date range. If you know the date range, it is easy to set it.

The correct code is:

# Major ticks every day
fmt_day = mdates.DayLocator(interval=1)
ax.xaxis.set_major_locator(fmt_day)
ax.set_xlim(np.datetime64('2019-08-01'), np.datetime64('2019-09-01'))  # set limit for tick population

Thank you very much :-)

[jklymak]

Actually I'll re-open.

The 2000-2010 comes from DateLocator.nonsingular but there is no reason DayLocator etc couldn't define their own, more sensible, nonsingular. This is actually a pretty easy little project if someone wants to tackle it...

i.e. something like:

def nonsingular(self, vmin, vmax):
    if not np.isfinite(vmin) or not np.isfinite(vmax):
            # Except if there is no data, then use 2000-2010 as default.
            return (date2num(datetime.date(2000, 1, 1)),
                    date2num(datetime.date(2000, 2, 1)))
    else:
       return super().nonsingular(vmin, vmax)

where the date limits are appropriate for the smaller locators in dates.py

[jeffreypaul15]

Can I give this a shot?

[jeffreypaul15]

Actually I'll re-open.

The 2000-2010 comes from DateLocator.nonsingular but there is no reason DayLocator etc couldn't define their own, more sensible, nonsingular. This is actually a pretty easy little project if someone wants to tackle it...

i.e. something like:

def nonsingular(self, vmin, vmax):
    if not np.isfinite(vmin) or not np.isfinite(vmax):
            # Except if there is no data, then use 2000-2010 as default.
            return (date2num(datetime.date(2000, 1, 1)),
                    date2num(datetime.date(2000, 2, 1)))
    else:
       return super().nonsingular(vmin, vmax)

where the date limits are appropriate for the smaller locators in dates.py

Tinkering around with this, I found that the setting of default ticks happens in DateConverter.axisinfo() and not in DateLocator.nonsingular where the vmin and vmax is checked for being finite instead and then the defaults are set.
In the XAxis class (axis.py), this method is called and the defaults are set with the default converter which is a DateConverter object.

Would changing the default ticks over at DateConverter.axisinfo() be viable? @jklymak

[dmatos2012]

fig, ax = plt.subplots()
fmt_day = mdates.DayLocator(interval=1)
ax.xaxis_date()
ax.xaxis.set_major_locator(fmt_day)
plt.show() 

I am not sure this patch works here, as @jeffreypaul15 mentioned, default_limits is set in DateConverter.axisinfo(). I think the problem comes when it tries to plot and tries to return the value of the located ticks in DateLocator.tick_values(), where the data points are already > 1000 ticks and raises the log message, and since ax is empty, it returns the default_limits

I was thinking however if it was possible to do some checking when user does ax.xaxis.set_major_locator(fmt_day), since at this point we know the axes.viewLimit.intervalx, and if the user sets fmt_day as in this example, it will surely raise the log message, since its trying to plot daily ticks for 10 years (2000-2010). So for instance, if we know that default_limit is set and fmt_day is a DayLocator, we could perhaps default to do a more sensible limit (e,g 2000-2001) and avoid the warning log.

Or, perhaps, we could make sure that ax.set_xlim() is set adequately at all times, and if it isnt, then we could raise a warning such as

Warning: Attempting to generate ticks with default limits, make sure you set your axis limits appropriately

I tried to do something, but was not sure what the intended behavior was, and whether this is actually a bug. And also, my attempts were breaking current functionality 🤦, but thought at least I would describe what I found, but yeah I am not sure.

Thoughts @tacaswell @jklymak?

[jklymak]

@jeffreypaul15 @dmatos2012 my apologies, this fell off my radar.

I'm going to close this. yeah, its annoying, but the default limit is chosen at ax.xaxis_date() which has no way of knowing that the locator will end up being a Daily locator. The solution is to simply supply limits that make sense before attempting a draw.

fig, ax = plt.subplots()
fmt_day = mdates.DayLocator(interval=1)
fmt_day.MAXTICKS = 40  # has NO effect
ax.xaxis_date()
ax.xaxis.set_major_locator(fmt_day)
ax.set_xlim(np.datetime64('2001-01-01'), np.datetime64('2001-01-10'))

works fine.

however, if anyone strongly objects or has a way forward, happy to re-open.

[ba05]

@jeffreypaul15 @dmatos2012 my apologies, this fell off my radar.

I'm going to close this. yeah, its annoying, but the default limit is chosen at ax.xaxis_date() which has no way of knowing that the locator will end up being a Daily locator. The solution is to simply supply limits that make sense before attempting a draw.

fig, ax = plt.subplots()
fmt_day = mdates.DayLocator(interval=1)
fmt_day.MAXTICKS = 40  # has NO effect
ax.xaxis_date()
ax.xaxis.set_major_locator(fmt_day)
ax.set_xlim(np.datetime64('2001-01-01'), np.datetime64('2001-01-10'))

works fine.

however, if anyone strongly objects or has a way forward, happy to re-open.

I'm having the same/similar issue with matplotlib 3.5.2. Plotting hangs unless I add:

ax.set_xlim(np.datetime64(df.index[0]), np.datetime64(df.index[-1]))

It seems that matplotlib wants to explicitly know the starting and end x limits. Whatever default value causes the code to hang.

[tacaswell]

@ba05 The issue is that the default range is ~10 years (When we know nothing about the data we have to pick something). If you then set the locator to daily, we try to plot 3.65k ticks which is a very slow process (rendering text is expensive).

[jklymak]

I guess the argument could be made that the default range can be changed to 0-1 (days) like all other axes now that we have 1970 as the epoch. The historical problem with 0-1 was that with the old epoch it was an illegal date! The problem with changing it is that some folks may still use the old epoch, and back compatibility.

[ba05]

@ba05 The issue is that the default range is ~10 years (When we know nothing about the data we have to pick something). If you then set the locator to daily, we try to plot 3.65k ticks which is a very slow process (rendering text is expensive).

Can a warning be added when taking an inordinate amount of time to execute?

[dopplershift]

The problem is that it's running synchronously, so your options for issuing the warning are either before it runs, or after, the latter of which sounds kinda pointless.

[tacaswell]

We used to have warnings when trying to draw a large number of ticks, but I think we removed them because they were annoying users who really did want that many ticks...