Python Decimation Filter with Aligning Frames

[Inch4Tk]

Required Info
Camera Model	D435
Firmware Version	05.09.14.00
Operating System & Version	Ubuntu 16.04
Kernel Version (Linux Only)	4.15.0-33-generic
Platform	PC
SDK Version	2.13.0
Language	python
Segment	Robot

Issue Description

I am trying to get the decimation filter combined with aligning frames to run, but could not make it work. I did not find any python documentation/issues/examples regarding this. As per this issue #1207 I wanted to first post-process before aligning.

Here is my current code. Everything works fine without using the filter.

# frameset of color and depth
frames = self.pipeline.wait_for_frames()
depth = frames.get_depth_frame()
# Apply filters to depth
depth = self.dec_filter.process(depth)

# TODO: Combine filtered depth with frames
# ????

# Align the depth frame to color frame
aligned_frames = self.align.process(frames)

# Get aligned frames
aligned_depth_frame = aligned_frames.get_depth_frame()
color_frame = aligned_frames.get_color_frame()

# Validate that both frames are valid
if not aligned_depth_frame or not color_frame:
    continue

I thought I figured out a way that could work, but currently the binding is bugged so I don't know if that would work:

#...
depth = self.filter_chain(depth)
list_of_frames = [rs.frame(frames.get_color_frame()), depth]
source = rs.frame_source()
frames = source.allocate_composite_frame(list_of_frames)

# Align the depth frame to color frame
aligned_frames = self.align.process(frames)
#...

Which gives me this:
TypeError: pyrealsense2.frame_source: No constructor defined!

So my question is: Am I on the right track? Or even better, Is there a way to do it without using allocate_composite_frame?

[dorodnic]

Since 2.16 there is a simpler way to do this.
Something like:

fs = pipe.wait_for_frames()
fs = fs.apply_filter(align).apply_filter(decimate);

The idea is to have single apply_filter method that can compose any operation, so that you don't have to deal with splitting and re-building the frameset.
I can't check the code right now, but if you run into problems please let us know and we will post a full example.

[Inch4Tk]

So I finally had a chance to try it on a realsense device after upgrading to 2.16. But currently this does not seem to work with python:

AttributeError: 'pyrealsense2.composite_frame' object has no attribute 'apply_filter'.

Is there any other way to apply the decimation filter before applying align? Or possibly another way to rebuild the frameset manually?

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hi @Inch4Tk,

If the decimation filter applied before align, that means the image size is reduce to 1/4. So far, it's not like the current implementation.
Any reason you like to implement this way?

[Inch4Tk]

Thanks for your reply @RealSense-Customer-Engineering . Here in this Issue, evp-mp #1207 (comment) explains in point 3 that he recommends applying filters before aligning.

We recommend to apply post-processing filters before aligning depth to color to reduce aliasing. This is the way it implemented in ''realsense-viewer''s pointcloud view.

That is why I wanted to do it that way. I also wanted to avoid having to rescale the aligned depth or color image myself to get it aligned again.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hi @ev-mp,

Do you have more suggestions about the implementation in Python?

[freemanlo]

@ev-mp #1207 (comment) explains in point 3 that he recommends applying filters before aligning. A python example code is required to have the correct the filters before align.

[RZz96]

Since 2.16 there is a simpler way to do this.
Something like:

fs = pipe.wait_for_frames()
fs = fs.apply_filter(align).apply_filter(decimate);

The idea is to have single apply_filter method that can compose any operation, so that you don't have to deal with splitting and re-building the frameset.
I can't check the code right now, but if you run into problems please let us know and we will post a full example.

Hi
I used it and i got error saying that " object has no attribute 'apply_filter' "

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Ticket being closed due to inactivity for 30+ days

[zamirkhan]

I would also like to see a working python example of alignment and post-processing together. This issue was closed without one being provided. The suggest apply_filter method does not appear to exist as shown.

[darrell-rg]

This worked for me when i modified the align-dept2color.py example. both frames come out 640x480

decimate = rs.decimation_filter(8)
align_to = rs.stream.color
align = rs.align(align_to)

# Streaming loop
try:
    while True:
        # Get frameset of color and depth
        frames = pipeline.wait_for_frames()
        decimated = decimate.process(frames).as_frameset()
        # Align the depth frame to color frame
        aligned_frames = align.process(decimated)

[ev-mp]

A reference python snippet was provided by @darrell-rg

[Boatsure]

This worked for me when I modified the align-dept2color.py example. both frames come out 640x480

decimate = rs.decimation_filter(8)
align_to = rs.stream.color
align = rs.align(align_to)

# Streaming loop
try:
    while True:
        # Get frameset of color and depth
        frames = pipeline.wait_for_frames()
        decimated = decimate.process(frames).as_frameset()
        # Align the depth frame to color frame
        aligned_frames = align.process(decimated)

This script does work. So the size of frames doesn't change after the decimation filter, is that correct? How about " the Decimation Filter will reduce spatial resolution preserving z-accuracy and performing some rudamentary hole-filling."

[Chris45215]

How about " the Decimation Filter will reduce spatial resolution preserving z-accuracy and performing some rudamentary hole-filling."

Is there any update on this aspect? I am trying to use the decimation filter specifically to reduce the resolution of the frame - and that seems like the most intuitive use of the filter. But, I am not getting a reduced frame size, I'm just getting the voxels in little grids of 3x3, 4x4, 5x5, etc.

As per #10716 (comment), the last line of that opening post says that the frame size should be reduced if the decimation filter is applied AFTER alignment - but I can't get that to work on my system. I get errors a few lines later, in the open3d.geometry.RGBDImage.create_from_color_and_depth() function.

[Bash226]

Can anyone please show me the result(image) after this(applying filter)?

[MartyG-RealSense]

Hi @Bash226 An animated image in Intel's post-processing guide at the link below shows before (original) and after (post-processing) when applying filters such as Decimation.

https://dev.intelrealsense.com/docs/rs-post-processing