indi_gphoto compatibility with libraw-2.0

[knallio]

Resolving #173 helped compiling the indi_gphoto driver, but there seem to be some issues left.

While taking pictures worked last night without any issues, the fitsheader in the images now states
BAYERPAT= 'GBRG ' / Bayer color pattern as opposed to BAYERPAT= 'RGGB ' / Bayer color pattern with libraw-0.19 (which as far as I know is correct).

I expect that is also due to changes in the libraw data structure and probably all metadata from the image should be checked.
I'll try to find the cause and also check the rest of the data and update this issue accordingly.

[knro]

Thanks for catching this as well! Looking forward to what you find out.

[knallio]

As far as I can tell the bayer pattern is the only thing which has changed. And this seems to be a change in libraw itself, see e.g. https://www.libraw.org/comment/5873#comment-5873

As far as I understood they changed the way COLOR() works, but to be honest, I did not yet fully understand everything.

[knallio]

After experimenting for far too long, I somehow understood what libraw changed (more or less), but not really why and what the correct way to deal with it is. And that is what libraw recommends... fixing the application since the application supposedly did it wrong in the first place. Codewise it boils down to top_margin being 52 for 0.20 instead of 51 for 0.19 so the bayer matrix is wrong by one row. Or might it be correct since the converted image then also starts one row later?? But that seems strange to me as well...

Hopefully you know more about raw files than I do ;-)

Just being curious: The fits file has the following fields :

XBAYROFF=                    0 / X offset of Bayer array                        
YBAYROFF=                    0 / Y offset of Bayer array  

Is there something which can be done with those?

[knro]

This is a hint for clients to start debayering one-row and/or one-column off, exactly for these types of scenarios. You can try it with Ekos. For libraw, I added a patch to remove their changes when building for the PPA at least... this is not a long-term solution, but it's a quick hack for now.

So does the color seem correct with 0.2? if you put Y offset 1, are they correct? Maybe the solution is to set YBAYEROFF to 1 when detecting 0.2? this way I can revert the hack.

[knallio]

I will test a bit tomorrow (hopefully). Tonight I will take pictures in native (raw) format, to avoid any possible issues.

[knallio]

Now I am totally confused. I took three pictures with all red, blue, and green and zoomed in at the unmodified fits files (see attached pictures) with gimp and siril.

Red:

Green:

Blue:

For me that looks like an RGGB image.

But if I debayer it while opening the picture with siril it works when using the GBRG from the header and not with RGGB. Opening the file with kstars and auto debayering (which also uses GBRG from the header as far as I can tell) also works.

What am I missing?

[knallio]

Hmm. In which direction is the bayer pattern defined? top-down or bottom-up?

I tried again with libraw-0.19 and took an all-red image:

So the RGGB looking file with 0.20 is debayered correctly with GBRG and the GBRG-looking file with 0.19 is correctly with RGGB. For me this would only make sense if the bayerpattern is applied bottom-up...

BTW, whatever direction, it might be a good idea to use ROWORDER keyword for fits files in indi?

So my current conclusion is, that the pixel pattern changes with libraw-0.20 and the bayer pattern changes accordingly, so in the end the colors in the final image are correct. But I stil don't understand, how and why the pixel pattern in the file can change. Does this mean libraw extracts different parts from the sensor-area depending on the version? I would have thought that the image area would be well defined somewhere...

[knro]

Thank you for the detailed investigation. If the final result is correct colors, I guess that all that matters. I will remove my patch to disable these libraw changes then.

[knallio]

If the final result is correct colors, I guess that all that matters.

On the one hand yes. On the other hand users must be aware that they have to update all their calibration frames as soon as they switch to libraw-0.20. It gets worse if they use different machines with different software versions... Maybe a not updated raspberry pi for taking the Light frames and an up-to-date computer for dark/bias frames.

That could be a real pain...

[knallio]

Just saw that this is not a new issue at all:

LibRaw/LibRaw#252

And since I wanted to understand the issue some more, I experimented with different margins and also the original canon raw converter, and the issue might return in the future again:

So far both dcraw and libraw had the image area hardcoded in their source. For the 600D which I use this was set to a top_margin of 51, which was compatible to an RGGB bayer mask. Now libraw shifts that margin to 52 (and the bayer mask accordingly), which is in a way a step in the right direction. Because the "real" top margin should be (according to the canon specific metadata) 56.

These hardcoded values are also the reason why e.g. dcraw converts my canon raw files to an image-size of 5202 x 3465, while the "advertised" image size by canon is 5184 x 3456. Which corresponds to the dimensions of the fits-file written by indi.

So my conclusion is that it is all a huge mess, but as long as all frames are taken/converted with the same tool and the same version, everything should be fine. Maybe I should invest in a dedicated astro camera...

I am closing this issue for now.