urgent bug after 1.5.0: offset of LineCollection when apply agg_filter

[Hualin]

so my plot is,

1. drawing regular scatterplot
2. drawing a network overlay using LineCollection with each edge to be blurred using a trick called agg_filter, which is a hidden feature not listed in the documentation of matplotlib.

the problem is this plot used to be fine before perhaps in version 1.4, but now after upgrade to 1.5.0 there is a offset of line collection and scatterplot. the plot is now look like this:

the 2D coordinate for scatterplot and line collection should be correct otherwise the old version should not draw it correctly.

I found there is no explanation in the official documentation of matplotlib of the Collection. set_agg_filter(filter_func) nor did I found the explanation of what is filter_fun is like and how it should be defined.

please tell me whether agg_filter is still working in Matplotlib, many thanks.

---

so basically I extract the relevant code to present:

## for drawing the overlay graph

# this line draw the network
pclt = LineCollection(segments = arwPair, linewidths = sizeArr, linestyles = markerArr)

from agg_filter import AggFilterHelper # a helper class wrapping the Gaussian filter

afh = AggFilterHelper("Blur")
pclt.set_agg_filter(afh.get_aggFilter()) # this LineCollection api to set filter
pclt.set_rasterized(True)

axx.add_collection(pclt) # add collection to axes object
# go on drawing

the agg_filter code is found in MailList of matplotlib which is never listed in Documentation but mentioned to be implemented but never actually implemented. so I guess it is a hidden feature, perhaps the code is outdated, but I really need to blur these lines in my plot in the new version of Matplotlib.

def smooth1d(x, window_len):
    # copied from http://www.scipy.org/Cookbook/SignalSmooth

    s=np.r_[2*x[0]-x[window_len:1:-1],x,2*x[-1]-x[-1:-window_len:-1]]
    w = np.hanning(window_len)
    y=np.convolve(w/w.sum(),s,mode='same')
    return y[window_len-1:-window_len+1]

def smooth2d(A, sigma=3):

    window_len = max(int(sigma), 3)*2+1
    A1 = np.array([smooth1d(x, window_len) for x in np.asarray(A)])
    A2 = np.transpose(A1)
    A3 = np.array([smooth1d(x, window_len) for x in A2])
    A4 = np.transpose(A3)

    return A4


class BaseFilter(object):
    def prepare_image(self, src_image, dpi, pad):
        ny, nx, depth = src_image.shape
        padded_src = np.zeros([pad*2+ny, pad*2+nx, depth], dtype="d")
        padded_src[pad:-pad, pad:-pad,:] = src_image[:,:,:]
        return padded_src

    def get_pad(self, dpi):
        return 0

    def __call__(self, im, dpi):
        pad = self.get_pad(dpi)
        padded_src = self.prepare_image(im, dpi, pad)
        tgt_image = self.process_image(padded_src, dpi)
        return tgt_image, -pad, -pad

class GaussianFilter(BaseFilter):
    "simple gauss filter"
    # Only this one I test and used in the application
    def __init__(self, sigma):
        self.sigma = sigma

    def get_pad(self, dpi):
        return int(self.sigma*3/72.*dpi)


    def process_image(self, padded_src, dpi):
        tgt_image = np.zeros_like(padded_src)
        # original 
        # aa = smooth2d(padded_src[:,:,-1]*self.alpha, self.sigma/72.*dpi)
        # tgt_image[:,:,-1] = aa
        # tgt_image[:,:,:-1] = self.color

        # My Try: hell yeah it works
        for i in range(4):
            aa = smooth2d(padded_src[:,:,i], self.sigma/72.*dpi)
            tgt_image[:,:,i] = aa

        return tgt_image

[Hualin]

while I guess I fixed it by removing the line pclt.set_rasterized(True). not sure what happened beneath coz there is no docs at all.

---

update:
wait, removing that line, the LineCollection is not blurred any more...

[tacaswell]

Probably a duplicate of #5960

[Hualin]

@tacaswell thank you for reply. I remember a few months ago I upgrade matplotlib and have this offset problem and then switch back to old version just to get my work done. I thought 1.5.1 would fix it.

May I ask, applying agg_filter to Artist object, is it a feature going to be fully documented or implemented? I mean I will not consider using offset box to mimic drawing lines by drawing images in order to have blur affect. I hope the bug can be fixed.

cheers.

[tacaswell]

We can not fix bugs we do not know about. Can you check if using #5718 fixes it for you?

[Hualin]

@tacaswell what do you mean you don't know about it? it is applying agg_filter to artist and the coordinate is wrong.

it is not only my code, the demo code of agg_filter provided in Mantplotlib official site can not run correctly also.

[Hualin]

previous stable version 1.4.3 result of running my code:

the network is blurred and coordinates is corrected.

[WeatherGod]

Hualin,

What we mean about "bugs we don't know about" is your statement that you
encountered the problem when you updated to version 1.5.0, and then simply
hoped that it would have been fixed for 1.5.1 without having reported it.
You even said that it is an undocumented feature, so it is likely not in
our test suite, either. The examples in our documentation are not checked
as rigorously as the images in our unit tests, so it is very easy for
errors like this to get past us in the release process.

Thank you for the latest image, as it would help us see exactly what is
wrong. The filtered image is flipped.

One of the big changes in v1.5.0 was to jettison the CXX bindings to AGG
and use the C++ library directly. Afterwards, we have found a few instances
were we have made mistakes in the rewrite. Some of them were fixed for
v1.5.1, others were deferred for the big rewrite in #5718. Can you confirm
if that is the case? Without a complete, self-contained example from you,
we can't confirm it for you.

On Fri, Feb 5, 2016 at 3:13 PM, Hualin notifications@github.com wrote:

previous stable version 1.4.3 result of running my code:

the network is blurred and coordinates is corrected.

[image: tsne_d_delaunay_plot pdf 1 page preview today at 8 12 00 pm]
https://cloud.githubusercontent.com/assets/1881213/12857911/c46360c0-cc44-11e5-9afd-2801584a8cea.png

—
Reply to this email directly or view it on GitHub
#5969 (comment)
.

[Hualin]

@WeatherGod

well I am not "simply hoping" anything fixed without report it. I just expect the version 1.5.1 could match what it has in the examples. I know no one can test all cases. it is common. agg_filter is very "lazy documented" not completely off record. Without proper docs I am not sure it is a supported or hidden feature (it has one line of explain and come with example, but not detail explained).

sorry I can't give you the whole project of my code here. it is too large. the procedure is simply is:

apply Gaussian blur to LineCollection by calling the method set_agg_filter and the filter code from agg_filter_demo, the coordinate of LineCollection is wrong, in version 1.5.0 is offset, in version 1.5.1 is flipped. (doesn't matter how wrong)

sorry I am not sure how exactly agg_filter is working in C level and I can't confirm it is the same case as #5718 , because I don't have that knowledge or experience to understand it.

I don't blame anybody of not fixing it now, I just wish it could be fixed in the future from it been reported. anyway, I can always use version 1.4.3 for my work only. I know Matplotlib is only a scientific plotting lib and it does most of the plot well. Now I have learned that for filter affect in visualisation, I should consider other libs in the future, perhaps Processing.

cheers.

[tacaswell]

@Hualin There seems to be some miscommunication here. As a project, we are very concerned about regressions; if we had known about this before we release 1.5.0 it would have been a blocking issue. However, we did not have an automated test of agg_filter and no one looked at the docs to notice that this was broken.

This will be fixed in v2.0 (the next planned major release) and has been milestoned for 1.5.2 (which may or may not happen) so it should be fixed 'soon'.

As for #5718 all you need to do is install mpl from source on that branch and verify if your code works or not.

[Hualin]

@tacaswell thanks it's very kind of you. looking forward the next major release.
: )

[efiring]

I'm closing this as a duplicate of #5960.

[tacaswell]

As an update, this will not be fixed in 1.5.2, but will be fixed in 2.0.