alignment fails when enable_device(serialid) is used

[omerbrandis]

hello,

i've encountered the following using d415 , firmware 5.9.2, sdk 2.10.1:

alignment of rgb and depth works using examples code when streams are enabled as following:
enable_stream(RS2_STREAM_DEPTH, -1,640, 480, RS2_FORMAT_Z16 , 6);
enable_stream(RS2_STREAM_COLOR, -1,640, 480, RS2_FORMAT_RGB8, 6);

(
wait_for_frames();
rs2::align align(RS2_STREAM_COLOR);
align.process(..)
.get_depth_frame();
.first(RS2_STREAM_COLOR);
.
.
.
)

but when i add
enable_device(serialid); ( after the enable streams commands, and obviously before the wait for frames and jalignment commands)
the alignment fails. ( there is about a 10 pixel offset).

i need to use enable_device since my app will need to handle several cameras at once ( connected to different usb ports obviously...)

please advise.
Omer.

[zivsha]

Sound unrelated...when you enable_device you simply state to the pipeline's config to use that device upon the next pipeline.start(config) call. If the device you choose causes problem with Align then something is wrong with that device or with Align when using that device.

Try to leave a single camera connected at each moment and see if align works. There should be no difference in this case if you use enable_device or not, if you do see a difference then I'm totally wrong it is a bug.
Either way try to isolate the issue

[omerbrandis]

Hello zivsha,

Thank you for trying to help,

It seems i have not been 100% clear, so i will try to clearify.

My app works with one camera ( aligns the rgb to depth) correctly when i do not use the enable_device command .

My next step is to prepare the app to support multiple cameras,
For that i have added the enable_device(serialid) after the 2 enable_stream commands, i have not yet added/connected an additional camera,
But now the alignment does not work ( it does not return/raise an error or exception, it simply does not perform full/complete alignment, there is about 10 pixel differernce),

Thanks,
Omer.

[zivsha]

You were clear, that's how I understood it. Can you post the entire code from the creation of rs2::config up to align.process?

[omerbrandis]

hello Ziv.

i'm afraid Im not comfortable with posting all the code to a public forum , i've shared the pseudo code in the initial post, as far as i can tell , there is not much more going on in the code from a realsense perspective.
i can add that most of the variables used are defined as data members in a wrapper class, and that i "burn off" a few frames before i "save"/store the frames i want (in the class data members) for later use.
( but the "store" of the result of wait_for_frames in a data member, align stored frames , get aligned frames and write to file , is a serial process , no additional wait for frames in between the align and the get aligned frames ) .

if you are part of the intel's realsense support team, can we continue this on some private & secure ticketing system?

i've uploaded 4 images that show the problem.

its not easy to spot, but if you focus your attention on the right edge of the bigger apple in bad_rgb2+3, you will see about a 5 pixel difference.
the only difference in the code between the good example and bad example is the explicit enabling of the device with the serial id "myconfig.enable_device(cameraid)" ( also minor movement of the camera).

granted, its hard to spot, only about 5 pixel difference , but its there.
yesterday it was very noticable at about 10 pixels, but i did not save those images.

Omer.

[omerbrandis]

hi,

I see that the file upload did not go as planned, the filenames have not been preserved,

so in this post i'll just upload the example of the bad alignment. ( about 5 pixel difference)

Omer.

[omerbrandis]

ok, i think i have a lead.

i also have a load camera parameters from json file ( myconfig.enable...-> pipiline.start -> burn off a few frames -> get_device+load_json -> waitForFrames -> align -> store to files.)

when the config.enable phase includes enable_Device the load_json parms are honored ( i see the rgb result as green since i've set one of the parms accordingly ( don't rememer its name right now),
but the alignment is not 100%.

when the config.enable phase does not include the explicit enable_device, it seems that the subsequent load_json is ignored, the result rgb image does not have a green hue, and the alignment is better.

so i guess the questions now become
a. is there a parm that can affect the aligment ? ( if so , which one ?)
b. why must i explicitly enable a device in order to use the load_json method?

Omer.

[omerbrandis]

i seems that the align.process was aligning "old" frames, not the newest/latest ones i was interested in.
( i added a wait-until-keypressed function between the burn-frames phase and the last wait_for_frames,
changed the scene before the keypress , and saw that the resutls i was getting were of the old "burnt" frames).

in other words, it seems that i have a very basic misunderstanding of the way this api works.
i'm trying to capture and process still images from the streams.
i've reviewed a few forum posts that suggest frames can't be stored localy in memory.
in other words that align.process(frameset) does not work on a specific ( in my case , latest) frameset that is passed to it as a parameter.

must/can i discard the old frames i'm not interested in ?
(rs2_release_frame(LastCapturedFrames[...].get()); )

should i stop/start the pipeline instead?

Omer.