indi-mi: missing FITS headers and properties in ekos

[peter-englmaier]

Is your feature request related to a problem? Please describe.

The Moravian Instruments C4-160000 camera is a dual-gain CMOS camera, which requires taking Dark and Light Frames in two gain settings (read modes). Ideally, I would like to take and preprocess the images with ekos, but currently this is not possible. Instead, SIPS.exe is needed at least to pre-process the images. However, when images are taken with ekos, the sips.exe cannot process because of missing FITS headers indicating the read mode. Likewise, any other program with preprocessing (i.e. PixInsight) will need to know what read mode was used for processing.

As it is right now, the indi-mi driver is useless for the new C4-160000 camera.

Describe the solution you'd like
These calibration images are critical for calibration, and we need a few more features to make them work in an automated way:

• please add "Read Mode" from indi_mi Main Control panel to the "Custom Capture Properties" in ekos. This makes it possible to program sequences which automatically take i.e. DARK images in the various Read Modes.
• please add FITS headers for: GAIN and DATAMAX according to this table:

Read Mode	GAIN	DATAMAX
16-bit HDR	0.82	65535
12-bit hi-gain	0.82	4095
12-bit lo-gain	18.69	4095
"16-bit" lo-gain	0.82	65535

The FITS headers are needed to identify the read mode and also make it possible to pre-process the images with the SIPS.exe from MI.

The FITS header make identification easier and also make output compatible with the MI SIPS.exe program, required for calibration. Perhaps adding a "READMODE" fits header would also help some users.

To fully pre-process the images in ekos, processing must take two DARK and FLAT frames into account. When the measured value (in the 16-bit HDR image) is less or equal to a threshold value, use the 12-bit-hi-gain DARK/FLAT image... Otherwise, use the "16-bit" lo-gain DARK/FLAT images.
The threshold for the C4-160000 camera is 3600 ADU. This is the same procedure taken by the camera, when the HDR mode is used. The documentation for this is in the C4 Camera User's Guide page 43ff:
https://www.gxccd.com/download/C4%20Manual.pdf

Describe alternatives you've considered

Currently, my only alternative is to use SIPS.exe instead of Ekos. Or, edit the fits header after recording the files with Ekos.

Additional context

I think this new kind of 2-gain cameras will become more common and ekos should be able to pre-process such data as well. Of course, we can make a script to do it outside ekos, but it would be much nicer to have this feature buildin.

[knro]

@sWski Can you please take a look?

[sWski]

Hi,

as for adding GAIN, DATAMAX and possibly READMODE to the FITS header, that's not a problem at all because libgxccd can already return those values. Note that for some cameras the DATAMAX depends on actual parameters, e.g. read mode, binning, etc. For example, the mentioned C4-16000 camera, if it reads in "12-bit hi-gain" or "12-bit lo-gain" read-mode, calculates DATAMAX as 4095 * binning_x * binning_y, so the table above is valid only if binning is set to 1x1.

I discussed this with a colleague who develops our SIPS software and he said, that in fact SIPS does NOT use the FITS header during the calibration at all. It doesn't even understand the READMODE header because it doesn't exist in the FITS specification. You have to specify directly which frame was downloaded in which read mode in the Calibration dialog, as you can see below:

We believe that the SIPS Calibration tool should work normally even with images obtained with KStars/Ekos and current indi-mi driver.

---

As for the calibration in KStars/Ekos, that's up to you to decide and develop such tool. I think our company was the first to describe quite well how to work with this chip and calibrate it and it's all in the linked PDF above.

If one wanted to fully automate this (without the need to choose frames in specific read modes), it might be necessary to add the READMODE parameter to FITS, but what value would it have if it is not in the standard? In addition, each camera has different numbering of reading modes...

[peter-englmaier]

Dear Jakub

I just bought the C4-160000EC and C1-5000A camera and naturally, I like to get the most out of them ;-) I fully understand that Moravian has pioneered the proper usage of this chip and enjoyed the articles on your website. I specifically have chosen Moravian over another brand, because of this!

Yes, you are right, the table is only for binning 1x1. I have not yet tried other binnings.

When I used SIPS to calibrate data obtained with ekos, I got errors about mismatching bit depth, so I tried to figure out why. However, I could not see in the file, what kind of read mode was used. I had some success when adding GAIN and DATAMAX values to the kstars files and using them to calibrate in SIPS, but I may just have been lucky.

For any fits produced with ekos, I like to know all the settings of the camera by just looking at the fits headers.

The READMODE header would help to identify and automatically sort files in software like PixInsight. It can be set to a string value making sense for the camera at hand, not provide something meaningful for every camera. So, in principle you can give a short string, i.e.: hdr16b, lo12b, hi12b, hi16b. For other cameras, you may not need to set the READMODE header at all. I think, it is only useful for cameras where we need to take different DARKs or FLATs with different read modes. With the header, one could just make all the required calibration images and then sort things out in other software like PixInsight. If we want to push a new "standard", we could set some more headers, i.e.: CHIPTYPE=GSENSE4040 in all kind of images, HDRTHRES=3600 in all hdr images, indicating the value used to combine hi-gain and lo-gain images. It is a constant, but maybe a future slightly modified camera firmware or model needs is going to change it? And of course GAIN and DATAMAX, because they are set in SIPS and software written to process SIPS images might already depend on it.

Perhaps adding yet more headers would be useful too: the preflash time / numclear information might sometimes also be important to record. So one can check anytime, if all DARKs have been taken with the same settings.

I just found out about READOUTM from SBIG cameras, but could not find any definition of its values. So better not use READOUTM.

[peter-englmaier]

Dear Jakub
Just to confirm: yes, I can calibrate in SIPS data obtained with ekos. After looking at the driver code, I do not see any place where FITS headers are set. Those things must happen downstream, probably in the ekos imager module.
Nevertheless, I find it quite critical to save those details in the header and will look into how kstars can be modified.

Closing this, because I do not see how this can be fixed on the driver side.
Best, Peter

[knro]

Actually, FITS header keywords are done on the driver side and not client side.

[sWski]

Dear Peter,

driver can add/update FITS headers, but those you see are added by INDI itself. I can add various headers that we are discussing here to the driver code. I'm just not sure how to deal with those non-standard keywords.

[peter-englmaier]

Great! Ok, I just cannot find the place in the code. What do you need to know? I can do some research to save your time, if I know what you like to know before adding new headers. I also have limited know how about writing indi drivers, but I am just starting with it (too little to make PR for existing stuff). So, let me reopen this ;-)

[sWski]

Peter,

• GAIN and DATAMAX are clear.
• READMODE - found, that QHY uses this and they write it as number. Should we do it same way they did?
• Preflash does not exists in FITS and no INDI driver writes it, so we need to create those keywords.
• CHIPTYPE and HDRTHRES - same as Preflash, we can push this, as no INDI driver writes it.

@knro what do you think?

[peter-englmaier]

Sorry, I was offline for a while.
Yes, READMODE as a number is fine and is actually preferred. The text version could be the comment field (for humans).

Preflash is kind of important if you need to compare data taken in different sessions. As I understand it, the preflash helps to correct for RBI, but at the price of higher noise. So people will not always have it turned on. They might have forgotten what was used when looking at the data at a later time or when doing some DARKs. It seems also to be of high interest for scientific applications to have good "control" of used preflash settings through the processing pipeline.

CHIPTYPE and HDRTHRES might become more significant in the future, if more and more chips like this become available. Makes it easier to adapt post-processing in other software.

[knro]

@sWski If there is no standard, we can do it in any way. I think we can go ahead and add some keywords to the FITS header as it is requested by several users now.