Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extracting a rectified image for android device #11514

Closed
Tikitaka02 opened this issue Mar 3, 2023 · 31 comments
Closed

Extracting a rectified image for android device #11514

Tikitaka02 opened this issue Mar 3, 2023 · 31 comments
Labels

Comments

@Tikitaka02
Copy link

Hello,

I am using D435 depth camera. I could compile the sample application in the repo for an android device using android studio, but I would like to extract rectified frames. How can I do it? Or is there any detailed documentation for the realsense sdk for android?

I appreciate your answer in advance.

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Mar 3, 2023

Hi @Tikitaka02 Almost all RealSense image formats except Y16 are already rectified by default so you do not need to write a special routine to rectify them. The rectification is applied to the image inside the camera hardware before the image is sent along the USB cable.

There are example programs for the SDK's Android wrapper at the link below.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/android/examples

If you are aiming to extract RealSense Android camera images to PNG image file then #9989 may be a helpful reference.

@Tikitaka02
Copy link
Author

Thank you for your message. I meant rectified images as not only the undistorted images, but also the combined images from 2 cameras.

@MartyG-RealSense
Copy link
Collaborator

For an undistorted image the easiest method would be to use Y16 format (available as infrared and RGB), as it does not have a distortion model applied to it. It is a monochrome image though.

There is not an unrectified format available for depth.

Unrectified RGB can also be obtained by accessing the camera's raw RGB frames and converting them to a usable RGB format with OpenCV's cvtColor instruction.

When you mention two cameras, do you mean two D435 cameras, or aligning two images (depth and colour) on the same camera, please

@Tikitaka02
Copy link
Author

I meant the 2 views in the same camera. I will explain the background. I noticed that there 4 sensors on the camera. 1 for infrared, 2 for depth, and 1 for normal rgb camera. I could place the normal rgb screen and depth screen side by side like the example named capture. However, the 2 screens have different perspective since I think the depth screen is the combination of 2 sensors. I would like to have the combined view as rgb image.

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Mar 3, 2023

The stereo depth image is constructed from images produced by the left and right infrared sensors. These are raw left and right infrared frames in the camera hardware. Depth is not generated from the Infrared and Infrared 2 streams, which is why depth can be enabled even if the Infrared and Infrared 2 streams are inactive.

You can align the depth and RGB screeens into a single combined image. There are few references though for how to perform depth-color alignment in Android. Two of the best available references are #4716 and #5520

As well as depth to color alignment (where the depth image is mapped onto the RGB image), you can also perform color to depth alignment, where the RGB image is mapped onto the depth image.

@Tikitaka02
Copy link
Author

Thank you so much. I could align the view. Would it be possible to render some points and lines on the aligned view?

@MartyG-RealSense
Copy link
Collaborator

I carefully researched your question about rendering points and lines. Thanks very much for your patience.

RealSense Android applications usually do not have visual elements such as lines and shapes overlaid on the image. Conceivably you could draw lines and points if you could interface your Android application with OpenCV to make use of its draw instructions, as discussed at #6717

@Tikitaka02
Copy link
Author

Thank you for the answer. Now I can draw on the display, but I would like to only display rgb stream and put the depth stream background so that I can still get depth information. How can I do it?

@MartyG-RealSense
Copy link
Collaborator

You could perhaps create two separate pipelines (pipeline1 and pipeline2), each with their own configuration (config1 and config2). Put the color stream definition in the config1 definition and the depth stream in the config2 definition. Start both pipeline1 and pipeline2 so that both are streaming data, but only visually render pipeline1 (the color one).

@Tikitaka02
Copy link
Author

But I would like to align the color and depth also.

@MartyG-RealSense
Copy link
Collaborator

Possibly you could put the depth code within a true / false bool condition so that when 'depth = true' the depth information is allowed to be produced, and if 'depth = false' then the depth information is not allowed to be produced.

@Tikitaka02
Copy link
Author

So what I have done is like this

private void configAndStart() throws Exception {
    try(Config config  = new Config())
    {
        // Set the format corresponding to bitmap
        config.enableStream(StreamType.COLOR, 640, 480, StreamFormat.RGBA8);
        config.enableStream(StreamType.DEPTH, 640, 480);
        // try statement needed here to release resources allocated by the Pipeline:start() method
        try(PipelineProfile pp = mPipeline.start(config)){

        }

    }
}

and

Runnable mStreaming = new Runnable() {
        @Override
        public void run() {
            try {
                try(FrameReleaser fr = new FrameReleaser()) {
                    try (FrameSet frames = mPipeline.waitForFrames(1000).releaseWith(fr)) {

                        try (VideoFrame videoFrame = frames.first(StreamType.COLOR).releaseWith(fr).as(Extension.VIDEO_FRAME);) {

                            try (Frame f = frames.first(StreamType.DEPTH)) {
                                f.releaseWith(fr);
                                DepthFrame depthFrame = f.as(Extension.DEPTH_FRAME);

                                extrinsic = videoFrame.getProfile().getExtrinsicTo(depthFrame.getProfile());
                                float[] rot = extrinsic.getRotation();
                                float[] trans = extrinsic.getTranslation();
                                for (int i =0; i<rot.length; i++) {
                                    Log.e("CamParams", "intRot "+String.valueOf(i)+" : "+String.valueOf(rot[i]));
                                }
                                for (int i =0; i<trans.length; i++) {
                                    Log.e("CamParams", "intTrans "+String.valueOf(i)+" : "+String.valueOf(trans[0]));
                                }

                                intrinsicDepth = depthFrame.getProfile().getIntrinsic();
                                float fxD = intrinsicDepth.getFx();
                                float fyD = intrinsicDepth.getFy();
                                float PpxD = intrinsicDepth.getPpx();
                                float PpyD = intrinsicDepth.getPpy();

                                Log.e("CamParamsDepth", "intFx: "+String.valueOf(fxD));
                                Log.e("CamParamsDepth", "intFy: "+String.valueOf(fyD));
                                Log.e("CamParamsDepth", "intPpx: "+String.valueOf(PpxD));
                                Log.e("CamParamsDepth", "intPpy: "+String.valueOf(PpyD));
                            }

                            intrinsic = videoFrame.getProfile().getIntrinsic();
                            float fx = intrinsic.getFx();
                            float fy = intrinsic.getFy();
                            float Ppx = intrinsic.getPpx();
                            float Ppy = intrinsic.getPpy();

                            Log.e("CamParamsColor", "intFx: "+String.valueOf(fx));
                            Log.e("CamParamsColor", "intFy: "+String.valueOf(fy));
                            Log.e("CamParamsColor", "intPpx: "+String.valueOf(Ppx));
                            Log.e("CamParamsColor", "intPpy: "+String.valueOf(Ppy));

                            Log.e("bitmapSize","Height"+ String.valueOf(image.getHeight()));
                            Log.e("bitmapSize: ","Width" + String.valueOf(image.getWidth()));
                        }
                        mGLSurfaceView.upload(frames);
                    }
                }
                mHandler.post(mStreaming);
            }
            catch (Exception e) {
                Log.e(TAG, "streaming, error: " + e.getMessage());
            }
        }
    };

I could extract camera extrinsics and intrinsics from both depth an color views, but how can I find the pixcel correspondance to know the specific pixel in color view is corresponding to that pixel in depth view?

@MartyG-RealSense
Copy link
Collaborator

The SDK has an instruction called rs2_project_color_pixel_to_depth_pixel that can convert a single specific color pixel coordinate to a depth pixel coordinate in order to obtain 3D XYZ without using alignment or pointcloud.

@Tikitaka02
Copy link
Author

Would it be possible to paste the instruction page here? I cannot find it...

@MartyG-RealSense
Copy link
Collaborator

There are not Android references about using rs2_project_color_pixel_to_depth_pixel with Android, unfortunately. kafan1986 at #5766 where you just commented will likely have Android programming knowledge about using this instruction.

@Tikitaka02
Copy link
Author

I found the function. It is in Utils.java in librealsense. However, the function requires depth scale information. How can I get it?

@MartyG-RealSense
Copy link
Collaborator

For almost all 400 Series depth cameras except D405, the default depth scale value is 0.001 and does not change unless deliberately changed by the program or a user input. So this value can be hard-coded into a script instead of retrieving the scale value from the camera in real-time.

@Tikitaka02
Copy link
Author

Got it thanks! Also, what do the depth max and min range mean? When I try to obtain them without hard code, there are so many options to choose from.

@MartyG-RealSense
Copy link
Collaborator

I have previously looked at depth_min and depth_max at #10484 (comment)

@Tikitaka02
Copy link
Author

Thank you so much. I have one additional question. How can I change the orientation of view? Now I have 640px width and 480px height, but I would like to make it vertucal view like 360px width and 480px height without chaging the context meaning if a person is stands upright on a frame, the person should stand upright on a frame after changing the orientation.

@MartyG-RealSense
Copy link
Collaborator

You could crop the image horizontally so that the width of the image is reduced but the height remains the same. It is not a simple thing to do though.

A RealSense user at #8512 (comment) cropped a colorized depth image and shared the C++ code of their solution.

A different approach that another RealSense user described at #2016 (comment) was to define a bounding box on their depth image and exclude from the depth image all of the coordinates outside of the defined box.

@Tikitaka02
Copy link
Author

Sorry for this repetitive question. I was also wondering if there are any concerns about using the d435 camera under a glass screen.

@MartyG-RealSense
Copy link
Collaborator

It's no trouble at all. :)

It should be no problem to use the D435 behind a glass screen so long as both sides of the screen have an anti-reflective coating applied to them. This is to avoid back-reflections from the camera's laser. The image below is of a D435 in a custom housing for underwater use.

image

@Tikitaka02
Copy link
Author

Hello, Thank you for the information. Another question. Would it be possible to access only an RGB camera without using sdk in android?

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Mar 23, 2023

RealSense cameras are UVC 1.5 compliant and so should be able to access an RGB stream as though they were an ordinary video webcam. By default the /video channel number for RGB on a D435 is /video2 (depth is /video0 and infrared is /video1), as advised at #3416 (comment)

On Windows, RGB can be accessed in video-using applications just by having the camera RGB driver installed in the Device Manager without having to have the SDK installed. On Linux raw RealSense camera data can be accessed with standard Linux tools by building the SDK from source code with the V4L2 backend (-DFORCE_LIBUVC=OFF)

I do not have information though regarding how RGB might be accessed through Android without the SDK. Usually, the RealSense AAR is required to be included in an Android project in order to grant device permissions to access the RealSense camera.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/android#build-with-gradle

@Tikitaka02
Copy link
Author

ok then would it be possible to use other surface view? I know the SDK uses OpenGL, but I would like to use other views to draw something on canvas with frames from the RGB camera for example.

@MartyG-RealSense
Copy link
Collaborator

The two main examples of RealSense Android applications that are able to render RGB onscreen are Capture (which you mentioned earlier in this case) and the RsCamera demo app on the Google Play store.

https://play.google.com/store/apps/details?id=com.intel.realsense.camera

The project source code for the RsCamera app is available at the link below as the Camera tool.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/android/tools/camera

@Tikitaka02
Copy link
Author

Hello, I have 2 additional questions.

If I want to place the intel camera behind a glass (thickness 2.81mm), what is the exact type of anti-reflection treatment I have to place? There are a lot of types available in the market. Can I apply the treatment only in the area around the camera?

If I want to place the intel camera behind a glass (thickness 2.81mm), what is the acceptable distance between the camera lens and the glass surface to get a reliable measurement?

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Mar 28, 2023

There are not recommendations for a particular type of anti-reflective coating or for a minimum / maximum distance of the glass from the camera lens. Yes, you could apply the coating only in the area around the camera.

If you would prefer not to coat the glass, you could alternatively try applying a thin-film linear polarization filter over the camera lenses on the outside of the camera. This filter dampens glare from reflections in the scene such as window glass, like the 'without / with filter' image below.

image

Section 4.4 When to use polarizers and waveplates of Intel's white-paper guide about use of optical filters with 400 Series cameras has more information about linear polarization filters.

https://dev.intelrealsense.com/docs/optical-filters-for-intel-realsense-depth-cameras-d400#4-the-use-of-optical-filters

Any polarization filter that is linear will work, though the circular filters in 3D glasses will not. This makes the filters inexpensive to purchase. You can search stores such as Amazon for the term linear polarizing filter sheet

@MartyG-RealSense
Copy link
Collaborator

Hi @Tikitaka02 Do you require further assistance with this case, please? Thanks!

@MartyG-RealSense
Copy link
Collaborator

Case closed due to no further comments received.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants