Android - Problem with GLSurfaceView after saving aligned colorized depth image

[Xantiks]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	D415
Firmware Version	05.12.15.50
Operating System & Version	Android 10
Platform	Android 10
SDK Version	2.49.0
Language	Java
Segment	Smartphone

Issue Description

Hi,

I'm trying to use the Align class to align the depth frame to color frame and then save the image as .jpg. The issue I'm facing is that when I use applyFilter to the Frameset I get the correct image, but the GLSurfaceView somehow gets updated with the aligned image.
This is the code for uploading frameset to the GLSurfaceView, which is unchanged:

Runnable mStreaming = new Runnable() {
        @Override
        public void run() {
            try {
                try(FrameReleaser fr = new FrameReleaser()) {
                    FrameSet framesProcessed = mPipeline.waitForFrames().releaseWith(fr).applyFilter(mColorizer).releaseWith(fr);
                    mGLSurfaceView.upload(framesProcessed);
                }
                mHandler.post(mStreaming);
            }
            catch (Exception e) {
                Log.e(TAG, "streaming, error: " + e.getMessage());
            }
        }
    };

This function is used for saving RGB and colorized depth image as .jpg files and saving raw depth data in .npy file. It is called in an AsyncTask if that is of any relevance.

public void snapshot(String profile, String folderPath) {
        try {
            try(FrameReleaser fr = new FrameReleaser()) {
                try(FrameSet frames = mPipeline.waitForFrames().releaseWith(fr)) {
                    Frame depth = frames.first(StreamType.DEPTH).releaseWith(fr);
                    DepthFrame depthFrame = depth.as(Extension.DEPTH_FRAME);
                    depthFrame.releaseWith(fr);
                    // saving raw depth data
                    Mat mDepth = new Mat(depthFrame.getHeight(), depthFrame.getWidth(), CvType.CV_16UC1);
                    int size = (int) (mDepth.total() * mDepth.elemSize());
                    byte [] return_buff_depth = new byte[size];
                    depthFrame.getData(return_buff_depth);
                    short [] shorts = new short[size/2];
                    ByteBuffer.wrap(return_buff_depth).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts);
                    mDepth.put(0,0,shorts);
                    Path path = Paths.get(folderPath, String.format("%spixelDepthData%s.npy", profile, currentDateAndTime));
                    NpyFile.write(path, shorts);
                    // applying Colorizer
                    try(FrameSet processed = frames.applyFilter(mColorizer).releaseWith(fr)) {
                        Frame depthColored = processed.first(StreamType.DEPTH).releaseWith(fr);
                        DepthFrame depthColorFrame = depthColored.as(Extension.DEPTH_FRAME);
                        depthColorFrame.releaseWith(fr);
                        Frame color = processed.first(StreamType.COLOR).releaseWith(fr);
                        VideoFrame colorFrame = color.as(Extension.VIDEO_FRAME);
                        colorFrame.releaseWith(fr);

                        // capturing color image
                        Mat mRGB = new Mat(colorFrame.getHeight(), colorFrame.getWidth(), CvType.CV_8UC3);
                        byte[] return_buff = new byte[colorFrame.getDataSize()];
                        colorFrame.getData(return_buff);
                        mRGB.put(0, 0, return_buff);

                        Bitmap bmp = Bitmap.createBitmap(mRGB.cols(), mRGB.rows(), Bitmap.Config.ARGB_8888);
                        org.opencv.android.Utils.matToBitmap(mRGB, bmp);

                        File imgFile = new File(folderPath, profile + "colorFrame-" + currentDateAndTime + ".jpg");
                        try {
                            FileOutputStream out2 = new FileOutputStream(imgFile);
                            bmp.compress(Bitmap.CompressFormat.JPEG, 100, out2);
                            out2.flush();
                            out2.close();
                        } catch (Exception e) {
                            e.printStackTrace();
                        }

                        // capturing colorized depth image
                        Mat mDepthColorized = new Mat(depthColorFrame.getHeight(), depthColorFrame.getWidth(), CvType.CV_8UC3);
                        byte[] return_buff_DepthColor = new byte[depthColorFrame.getDataSize()];
                        depthColorFrame.getData(return_buff_DepthColor);
                        mDepthColorized.put(0, 0, return_buff_DepthColor);

                        Bitmap bmpDepthColor = Bitmap.createBitmap(mDepthColorized.cols(), mDepthColorized.rows(), Bitmap.Config.ARGB_8888);
                        org.opencv.android.Utils.matToBitmap(mDepthColorized, bmpDepthColor);

                        File imgFileDepthColor = new File(folderPath, profile + "depthFrame-" + currentDateAndTime + ".jpg");
                        try {
                            FileOutputStream out3 = new FileOutputStream(imgFileDepthColor);
                            bmpDepthColor.compress(Bitmap.CompressFormat.JPEG, 100, out3);
                            out3.flush();
                            out3.close();
                        } catch (Exception e) {
                            e.printStackTrace();
                        }
                       .....

This is how my screen looks like when this code is used:

Once I add the Align filter like this in the 3rd line of the snapshot function:
FrameSet frames = mPipeline.waitForFrames().releaseWith(fr).applyFilter(mAlign).releaseWith(fr),
a single image is added to the GLSurfaceView every time, which results in this:

In case I would take lets say 20 pictures, I would have 20 images stacked on top of each other. I really have no idea what I'm missing and how can this happen by just adding a .applyFilter(Align), so if you could help me out I would really appreciate it.

Thank you in advance.

[MartyG-RealSense]

Hi @Xantiks A useful starting point in investigating your issue may be to refer you to #5520 for insights and scripting regarding using align in an Android application.

[Xantiks]

After some trial and error, I managed to solve the problem by just not using Colorizer on the frameset that is being uploaded to the GLSurfaceView. So there is no more stacking of images on the screen and the saved images are aligned as expected.

[MartyG-RealSense]

That's excellent to hear, @Xantiks - thanks very much for the update :)

[oddityyyy]

@Xantiks Hi, could you share where is the function void snapshot(String profile, String folderPath) called? I am having trouble saving the color image and depth image locally, could you help me? Thanks.

[Xantiks]

Hi @oddityyyy, I just call the function when a button is pressed. Maybe you can check out the recording example and see if it helps you out.

[oddityyyy]

嗨@oddityyyy，我只是在按下按钮时调用该函数。也许您可以查看录音该函数，看看它是否对您有帮助。

The recording example sets two buttons, mStartRecordFab and mStopRecordFab. We only need one button, that is, press the button once to take a picture. So I just need to replace the toggleRecording() method triggered by the button with the snapshot() method you wrote. And not change other parts. Did I understand it correctly? Could you help me see it? thanks！ o(￣▽￣)ブ

`mStartRecordFab.setOnClickListener(new View.OnClickListener() {

        @Override
        public void onClick(View view) {
            toggleRecording();
        }
    });
    mStopRecordFab.setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(View view) {
            toggleRecording();
        }
    });`

[Xantiks]

Yes, that should do it.

[oddityyyy]

是的，应该漂亮。

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/android/examples/capture/src/main/java/com/intel/realsense/capture
Hello, I made some changes to the above example as follows
MainActivity.zip
But after I press the button, the application crashes.
Can you help me or share your entire file, please, this problem has troubled me for a long time and I haven't solved it for a long time. I am new to Android Studio, please forgive me.
thanks

[Xantiks]

I'm also a beginner at this, but I skimmed through your code and at the start I see that you are missing the PERMISSIONS_REQUEST_WRITE for writing the data as you can see in the recording example. I can't really know what crashes your app, but you probably need this.

[oddityyyy]

我也是这方面的初学者，但我浏览了您的代码，开始一我发现您缺少PERMISSIONS_REQUEST_WRITE关系编写数据的方法，您如在录制示例中所见，我真的不知道是什么导致了您的应用程序崩溃，但您可能需要这个。

Sorry, I tried the suggestions you gave, but the application still crashes. I think there is a problem with the code I wrote. Could I learn from the rest of the code you wrote?

[mayur310]

I simply copied the code and attempted to run it, but it displayed the "Connect A RealSense Camera" message when I connected my Intel RealSense camera. And I am using D415 realsense.