Feature Request: Better Handling Nan and None Values

[Freed-Wu]

import plotext as plt
y = plt.sin()
import numpy as np
y[:40] = [np.nan] * 40
plt.scatter(y)
plt.show()

Expected: A plot which x range is [0, 200], but [0, 40] don't have any number.

Actual: A plot which x range is [40, 200].

[piccolomo]

Hi @Freed-Wu,

I am working on the issues. Regarding this one, no idea is coming to me on how to solve it. The function that I use to remove non numerical values litterally removes all of the non numerical values (nan or None).

non_numerical = lambda el:  el == None or (not isinstance(el, str) and math.isnan(el))

def remove_non_numerical(x, y): # it remove None and nan values but keeps strings for possible date or bar plots
   l = len(x)
   p = [i for i in range(l) if not (non_numerical(x[i]) or non_numerical(y[i]))]
   xn = [try_float(x[i]) for i in p]
   yn = [try_float(y[i]) for i in p]
   return xn, yn

any idea?

[Freed-Wu]

Take matplotlib as an example:

import numpy as np
from matplotlib import pyplot as plt
a = np.arange(10.)
a[2:5] = [np.nan] * 3
plt.plot(a, "b")
plt.show()

is equal to

import numpy as np
from matplotlib import pyplot as plt
a = np.arange(10.)
a[2:5] = [np.nan] * 3
plt.plot(range(0, 2), a[0:2], "b")
plt.plot(range(5, 10), a[5:10], "b")
plt.show()

So. I think just split the array to some parts by np.nan, and plot them
separately, which can solve this problem.

import numpy as np
import plotext as plt
a = np.arange(10.)
a[2:5] = [np.nan] * 3
plt.plot(a)
plt.show()

[piccolomo]

It's a good idea, but I think it would still produce the same result as in your initial code / post.

[piccolomo]

Also if in your latest code you use scatter, the plot works

import numpy as np
import plotext as plt
a = np.arange(10.)
a[2:5] = [np.nan] * 3
plt.scatter(a)
plt.show()

[Freed-Wu]

It's a good idea, but I think it would still produce the same result as in your initial code / post.

import numpy as np
import plotext as plt
a = np.arange(10.)
a[2:5] = [np.nan] * 3
plt.plot(range(0, 2), a[0:2], color="blue")
plt.plot(range(5, 10), a[5:10], color="blue")
plt.show()

1. a function to split the array to a list consisted of some arrays by np.nan
2. for each array in the list, plot them with the corresponding x coordinates.

[piccolomo]

If you let matplotlib deal with first code you sent, you get the same result as in plotext

import matplotlib.pyplot as plt; from plotext import sin
y = sin()
import numpy as np
y[:40] = [np.nan] * 40
plt.scatter(range(len(y)), y)
plt.show()

and in plotext

import plotext as plt; from plotext import sin
y = sin()
import numpy as np
y[:40] = [np.nan] * 40
plt.scatter(range(len(y)), y)
plt.show()

[piccolomo]

This is a slightly different issue, here the problem is the fact that plot() perceives a single signal and so draws lines between each point, while it would be better if it jumps the nan values. I would think about this, but its not trivial.

Take matplotlib as an example:

import numpy as np
from matplotlib import pyplot as plt
a = np.arange(10.)
a[2:5] = [np.nan] * 3
plt.plot(a, "b")
plt.show()

is equal to

import numpy as np
from matplotlib import pyplot as plt
a = np.arange(10.)
a[2:5] = [np.nan] * 3
plt.plot(range(0, 2), a[0:2], "b")
plt.plot(range(5, 10), a[5:10], "b")
plt.show()

So. I think just split the array to some parts by np.nan, and plot them separately, which can solve this problem.

import numpy as np
import plotext as plt
a = np.arange(10.)
a[2:5] = [np.nan] * 3
plt.plot(a)
plt.show()

[Freed-Wu]

If you let matplotlib deal with first code you sent, you get the same result as in plotext

Oh, I am sure. It should be expected behavior of matplotlib.

This is actually a slightly different issue

Right. 😄 It should be an extended version of the first problem: nan exists in the medium of an array, not beginning or end.

[piccolomo]

Done! Somehow I managed!

It was not trivial but easier then I thought. I will upload the code soon

[piccolomo]

Interestingly this solves also the first issue you posted

matplotlib would not handle it the same way

[Freed-Wu]

It was not trivial but easier then I thought. I will upload the code soon

Great!

matplotlib would not handle it the same way

Aha 😄. I'll ask a problem to matplotlib about is it a bug or an expected behavior.

[piccolomo]

Cool thanks a lot for the report. The solution did not involve splitting data luckily (cause otherwise I would have to split all colors, style, and other parameters, which would have been a pain), but the matplotlib example you posted was useful for me to understand anyway. Basically I let nan or None "percolate" inside the calculation until its time to be drawn, where they are simply discarded. I was expecting getting lost in tedious programming, but i got lucky.

[Freed-Wu]

The solution did not involve splitting data luckily (cause otherwise I would have to split all colors, style, and other parameters, which would have been a pain), but the matplotlib example you posted was useful for me to understand anyway.

Great! 👍

[Freed-Wu]

matplotlib/matplotlib#23797 (comment)

I think we auto-scale based on valid points, if there is a nan in the y-value it is not a valid point so we are auto-scaling the x limits to show you the valid data.

It should be the expected behavior of matplotlib. Does plotext support auto-scale?

[piccolomo]

No, the new code will not update the x or y limits, it just lets the nan or None value show up to plotting time when they are discarded. The reason why the x limits are preserved at the end is because the x data has all regular values and its limits are calculated naturally.

matplotlib/matplotlib#23797 (comment)

I think we auto-scale based on valid points, if there is a nan in the y-value it is not a valid point so we are auto-scaling the x limits to show you the valid data.

It should be the expected behavior of matplotlib. Does plotext support auto-scale?

[piccolomo]

Hi @Freed-Wu , I finally updated plotext including the changes you asked for.

The new version is available on GitHib for now and soon also on PyPi. To install follow the indications here.

Your changes are documented here and you have been credited here.

Any feedback is welcomed.

Thanks a lot for your inputs and all the best,
Savino