Artificial Events due to filter settings

[Silberbaer]

When I try to filter Chimera data to lower frequencies (<500kHz) with the filter order too high (8 for 500kHz but even at 4 for 250kHz and below) MOSAIC generates artificial events that are not seen in the raw data (see attached images). These events are very frequent (>50/s) and are spaced in multiples of 2.4ms so I am assuming it is some sort of chunking of the data that messes up the filter. My current blocksizeSec is set to 0.1 so that is way higher than the spacing of these events I am seeing.

All of these events have the same general shape which looks like this one:

Clearly not a real translocation event.

I plotted the difference of the absolute start times of the first 100 events to make clear that they are spaced evenly.

I am using the besselLowpassfilter for all of this

[Silberbaer]

I should mention that I do not get these events when the filter is set to for example 900kHz with a filter order of 8 (or 4) but if I go below that it seems to mess up. And the spacing between those events is not dependent on what the exact cutoff frequency is.

[abalijepalli]

What was the amplifier bandwidth and sampling rate?

We changed a few things about the filter recently and I wonder if that is causing some artifacts. @shadowk29, do you think it may be related to the switch to a non-causal filter?

[shadowk29]

It seems more like a data pipe issue, like pre-event data is getting dropped somewhere, though I suppose it's possible the non causal filter padding is the problem. The amplifier here is sampling at 4.1666Mhz with 1MHz bandwidth, so the sampling period is 0.24 us - 2.4 ms spacing means this is happening every 10000 samples exactly. That equal to the max event length in his config file, though that might be coincidence.

I'll poke the filter code and see if I can find anything, but the periodicity seems odd.

[shadowk29]

It turns out that @Silberbaer is actually using a version of mosaic that pre-dates the change to non-causal filter.

[shadowk29]

I think the events he is seeing are just filter transients at the start of data chunks being fed to the filter. The non-causal filter should not have this issue to updating should fix it.

[abalijepalli]

The filter code processes data in blocks controlled by the block size. If the block size was set to 0.1 s, I would expect to see repeats at 100k data points. I'll take a look at the event padding to make sure nothing is being dropped.

[shadowk29]

0.1s would be 416,666 points for chimera data. 10000 is MaxEventLength in adept settings, which is the only place I can think that number would be entering. Maybe it's feeding it MaxEventLength points each time?

[abalijepalli]

I looked at the TrajIO code and it appears that the 10000 points is the block size used to read more data from disk (https://github.com/usnistgov/mosaic/blob/devel-1.0/mosaic/trajio/metaTrajIO.py#L101). I am still uncertain about the cause of the edge effects from the filter.

[shadowk29]

It is most likely the transient arising from the filter used - it looks as though it's padded with zeros on the left and does not have enough time to settle completely before the real data begins. @Silberbaer confirmed that the updated code does not have this issue, so it is most likely something to do with the old filter. The timing with the start of new data chunks basically confirms this in my mind, so I think this can be closed.