ZWO ASI6200MC indi_asi_ccd failure somewhere in /lib/aarch64-linux-gnu/libASICamera2.so

[jmocenigo]

Bug Description:
Attempt to take exposure results in repeated failures that appear in log as follows:

[2021-04-18T09:45:23.680 EDT INFO ][   org.kde.kstars.ekos.capture] - "Capturing 1.000-second  image..."
[2021-04-18T09:45:23.684 EDT DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI6200MC Pro : "[DEBUG] StartExposure->setexp : 1.000s "
[2021-04-18T09:45:34.527 EDT DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI6200MC Pro : "[DEBUG] ASIGetExpStatus failed. Restarting exposure... "

Failure is somewhere in libASICamera2.so called from line 340 of asi_ccd.cpp:
ASI_ERROR_CODE ret = ASIGetExpStatus(mCameraInfo.CameraID, &status);

Environment:

• Hardware is 8GB Raspberry Pi4 with boot and root filesystems on USB3 SSD
• OS is 64-bit Ubuntu-Mate 20.0.4 LTS
• Camera is ZWO ASI6200MC (61 Megapixel sensor) using USB3
• Using Telescope Simulator and Camera, nothing else for this test

Failure occurs with stable indi/kstars and nightly-build version and self-built from git clone.
Failure does not occur 100% of the time, but definitely above 95% of the time.
Failure also occurred using raspbian beta 64-bit OS.
Failures happen with USB bandwidth at 40 and at 80.
Failures happen regardless of FITS viewer settings (tried all reasonable combinations including off)
.
I am able to capture images reliably using 32-bit astroberry on RPi4 (FITS view off)

To Reproduce:

1. Use Linux 64-bit (e.g., Ubuntu-Mate) on RPi4 (8GB or 4GB)
2. Create profile using local mode and just telescope simulator and ZWO ASI6200MC
3. My camera was not connected to any equipment (other than computer) and had sensor cover on.
4. Take any exposure. If it happens to succeed, then try a series of 10 exposure - it is bound to fail.

Expected behavior:

1. Message that capture is starting (timer counts down the time)
2. spinning cursor with log message "Capture failed. (etc)"
3. Multiple retries (that all fail)
4. Eventually "CCD capture aborted." message

Pertinent part of log file is above - nothing to be gained by attaching file.

[knro]

Is this with GIT? the ASI camera SDK was rolled back a few days ago due to reported crashes.

[jmocenigo]

I did a git clone within the last 24 hours.

…

Sent from my iPhone

On Apr 18, 2021, at 10:49 AM, Jasem Mutlaq ***@***.***> wrote:  Is this with GIT? the ASI camera SDK was rolled back a few days ago due to reported crashes. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[jmocenigo]

Thu Apr 15 17:53:28 2021 +0300 libasi/armv6/libASICamera2.bin Thu Apr 15 17:53:28 2021 +0300 libasi/armv7/libASICamera2.bin Thu Apr 15 17:53:28 2021 +0300 libasi/armv8/libASICamera2.bin Thu Apr 15 17:53:28 2021 +0300 libasi/mac/libASICamera2.bin Thu Apr 15 17:53:28 2021 +0300 libasi/x64/libASICamera2.bin Thu Apr 15 17:53:28 2021 +0300 libasi/x86/libASICamera2.bin

…

On Apr 18, 2021, at 11:45 AM, John Mocenigo ***@***.***> wrote: I did a git clone within the last 24 hours. Sent from my iPhone > On Apr 18, 2021, at 10:49 AM, Jasem Mutlaq ***@***.***> wrote: > >  > > Is this with GIT? the ASI camera SDK was rolled back a few days ago due to reported crashes. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub <#376 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACVI5BRF3F2Q3J3LQF2N36TTJLWPVANCNFSM43EJBTVA>. >

[jmocenigo]

To be a little more clear regarding my previous message: $ git log -1 --format="%ad" -- libasi/armv8/libASICamera2.bin Thu Apr 15 17:53:28 2021 +0300 I wanted to add that my suspicion is some sort of threading issue, which is really the only way I can explain that it sometimes works, but usually doesn’t (even during the same session, namely get one image, works; repeat, works; repeat fails - all in the same camera connection session). I tried attaching gdb to the indi_asi_ccd process and stepping through and sometimes it worked that way and sometimes not as well, but I had a (anecdotally) higher success rate, which again makes me think it is a threading issue since gdb does a Heisenberg on thread behavior. Would capturing a failure situation in gdb help you out even if what I capture looks like the stuff below (since, obviously, there is no source info)? It is not easy to do so I rather not spend the time if it is not going to be useful. (gdb) 0x0000ffff998010d8 in ASIGetExpStatus () from /lib/aarch64-linux-gnu/libASICamera2.so (gdb) 0x0000ffff998010dc in ASIGetExpStatus () from /lib/aarch64-linux-gnu/libASICamera2.so (gdb) 0x0000ffff998010f4 in ASIGetExpStatus () from /lib/aarch64-linux-gnu/libASICamera2.so (gdb) 0x0000ffff998010f8 in ASIGetExpStatus () from /lib/aarch64-linux-gnu/libASICamera2.so (gdb) 0x0000ffff998010fc in ASIGetExpStatus () from /lib/aarch64-linux-gnu/libASICamera2.so (gdb) 0x0000ffff99801100 in ASIGetExpStatus () from /lib/aarch64-linux-gnu/libASICamera2.so

…

On Apr 18, 2021, at 12:22 PM, John Mocenigo ***@***.***> wrote: Thu Apr 15 17:53:28 2021 +0300 libasi/armv6/libASICamera2.bin Thu Apr 15 17:53:28 2021 +0300 libasi/armv7/libASICamera2.bin Thu Apr 15 17:53:28 2021 +0300 libasi/armv8/libASICamera2.bin Thu Apr 15 17:53:28 2021 +0300 libasi/mac/libASICamera2.bin Thu Apr 15 17:53:28 2021 +0300 libasi/x64/libASICamera2.bin Thu Apr 15 17:53:28 2021 +0300 libasi/x86/libASICamera2.bin > On Apr 18, 2021, at 11:45 AM, John Mocenigo ***@***.*** ***@***.***>> wrote: > > I did a git clone within the last 24 hours. > > Sent from my iPhone > >> On Apr 18, 2021, at 10:49 AM, Jasem Mutlaq ***@***.*** ***@***.***>> wrote: >> >>  >> >> Is this with GIT? the ASI camera SDK was rolled back a few days ago due to reported crashes. >> >> — >> You are receiving this because you authored the thread. >> Reply to this email directly, view it on GitHub <#376 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACVI5BRF3F2Q3J3LQF2N36TTJLWPVANCNFSM43EJBTVA>. >>

[jmocenigo]

P.S. I ordered a “real” Linux box (a System76 Meerkat) that should arrive Wednesday, so that should at least confirm that it is not a RPi4-specific issue.

…

On Apr 18, 2021, at 7:23 PM, John Mocenigo ***@***.***> wrote: To be a little more clear regarding my previous message: $ git log -1 --format="%ad" -- libasi/armv8/libASICamera2.bin Thu Apr 15 17:53:28 2021 +0300 I wanted to add that my suspicion is some sort of threading issue, which is really the only way I can explain that it sometimes works, but usually doesn’t (even during the same session, namely get one image, works; repeat, works; repeat fails - all in the same camera connection session). I tried attaching gdb to the indi_asi_ccd process and stepping through and sometimes it worked that way and sometimes not as well, but I had a (anecdotally) higher success rate, which again makes me think it is a threading issue since gdb does a Heisenberg on thread behavior. Would capturing a failure situation in gdb help you out even if what I capture looks like the stuff below (since, obviously, there is no source info)? It is not easy to do so I rather not spend the time if it is not going to be useful. (gdb) 0x0000ffff998010d8 in ASIGetExpStatus () from /lib/aarch64-linux-gnu/libASICamera2.so (gdb) 0x0000ffff998010dc in ASIGetExpStatus () from /lib/aarch64-linux-gnu/libASICamera2.so (gdb) 0x0000ffff998010f4 in ASIGetExpStatus () from /lib/aarch64-linux-gnu/libASICamera2.so (gdb) 0x0000ffff998010f8 in ASIGetExpStatus () from /lib/aarch64-linux-gnu/libASICamera2.so (gdb) 0x0000ffff998010fc in ASIGetExpStatus () from /lib/aarch64-linux-gnu/libASICamera2.so (gdb) 0x0000ffff99801100 in ASIGetExpStatus () from /lib/aarch64-linux-gnu/libASICamera2.so > On Apr 18, 2021, at 12:22 PM, John Mocenigo ***@***.*** ***@***.***>> wrote: > > Thu Apr 15 17:53:28 2021 +0300 libasi/armv6/libASICamera2.bin > Thu Apr 15 17:53:28 2021 +0300 libasi/armv7/libASICamera2.bin > Thu Apr 15 17:53:28 2021 +0300 libasi/armv8/libASICamera2.bin > Thu Apr 15 17:53:28 2021 +0300 libasi/mac/libASICamera2.bin > Thu Apr 15 17:53:28 2021 +0300 libasi/x64/libASICamera2.bin > Thu Apr 15 17:53:28 2021 +0300 libasi/x86/libASICamera2.bin > >> On Apr 18, 2021, at 11:45 AM, John Mocenigo ***@***.*** ***@***.***>> wrote: >> >> I did a git clone within the last 24 hours. >> >> Sent from my iPhone >> >>> On Apr 18, 2021, at 10:49 AM, Jasem Mutlaq ***@***.*** ***@***.***>> wrote: >>> >>>  >>> >>> Is this with GIT? the ASI camera SDK was rolled back a few days ago due to reported crashes. >>> >>> — >>> You are receiving this because you authored the thread. >>> Reply to this email directly, view it on GitHub <#376 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACVI5BRF3F2Q3J3LQF2N36TTJLWPVANCNFSM43EJBTVA>. >>> >

[jmocenigo]

Hi. I pulled the INDI core/lib and KStars sources from GitHub today (within the last hour), built and installed and now it is working consistently. I am running on a System 76 Meerkat Linux (Ubuntu 20.04 Mate) with the ZWO ASI6200 and ran 20 images in sequence, followed by 60 images in sequence. All FITS viewer options except "Single Window Capture" and "Limited Resources Mode" were turned on. To whomever did whatever, Thanks!

[gogolander]

Same problem on a different camera: ASI 294MM Pro on a Ubuntu 20.04 minipc; controlled remotely by a KStar on a remote PC.
The problem appears whenever the exposition bin is 2x2 or 1x1, which makes plate solving and focusing troublesome.

[praetp]

I am still seeing this with my ZWO ASI 533 mc pro. After 165 images, suddenly got this failure.

[2022-04-21T01:13:31.796 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] Uploading file. Ext: fits, Size: 18103680, sendImage? No, saveImage? Yes "
[2022-04-21T01:13:31.920 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Image saved to /camera/Light/L_Astronomik/M51_Light_L_Astronomik_60_secs_2022-04-21T01-13-31_139.fits "
[2022-04-21T01:13:31.927 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] Upload complete "
[2022-04-21T01:13:32.038 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] StartExposure->setexp : 60.000s "
[2022-04-21T01:13:32.041 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Taking a 60 seconds frame... "
[2022-04-21T01:14:32.889 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Exposure done, downloading image... "
[2022-04-21T01:14:32.941 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Download complete. "
[2022-04-21T01:14:33.005 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] Uploading file. Ext: fits, Size: 18103680, sendImage? No, saveImage? Yes "
[2022-04-21T01:14:33.103 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Image saved to /camera/Light/L_Astronomik/M51_Light_L_Astronomik_60_secs_2022-04-21T01-14-33_140.fits "
[2022-04-21T01:14:33.112 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] Upload complete "
[2022-04-21T01:14:33.612 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] StartExposure->setexp : 60.000s "
[2022-04-21T01:14:33.613 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Taking a 60 seconds frame... "
[2022-04-21T01:15:34.368 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Exposure done, downloading image... "
[2022-04-21T01:15:34.369 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Download complete. "
[2022-04-21T01:15:34.435 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] Uploading file. Ext: fits, Size: 18103680, sendImage? No, saveImage? Yes "
[2022-04-21T01:15:34.509 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Image saved to /camera/Light/L_Astronomik/M51_Light_L_Astronomik_60_secs_2022-04-21T01-15-34_141.fits "
[2022-04-21T01:15:34.516 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] Upload complete "
[2022-04-21T01:15:41.775 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] StartExposure->setexp : 60.000s "
[2022-04-21T01:15:41.776 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Taking a 60 seconds frame... "
[2022-04-21T01:16:43.835 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] ASIGetExpStatus failed. Restarting exposure... "
[2022-04-21T01:16:43.836 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] StartExposure->setexp : 60.000s "
[2022-04-21T01:16:43.839 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Taking a 60 seconds frame... "
[2022-04-21T01:16:44.837 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] ASIGetExpStatus failed. Restarting exposure... "
[2022-04-21T01:16:44.837 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] StartExposure->setexp : 60.000s "
[2022-04-21T01:16:44.838 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Taking a 60 seconds frame... "
[2022-04-21T01:16:45.839 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[ERROR] Exposure failed after 3 attempts. "
[2022-04-21T01:16:45.961 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] StartExposure->setexp : 60.000s "
[2022-04-21T01:16:45.961 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Taking a 60 seconds frame... "
[2022-04-21T01:16:45.962 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] Aborting exposure... "
[2022-04-21T01:16:46.256 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] StartExposure->setexp : 60.000s "
[2022-04-21T01:16:46.257 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Taking a 60 seconds frame... "
[2022-04-21T01:16:47.258 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] ASIGetExpStatus failed. Restarting exposure... "
[2022-04-21T01:16:47.259 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] StartExposure->setexp : 60.000s "
[2022-04-21T01:16:47.261 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Taking a 60 seconds frame... "
[2022-04-21T01:16:48.258 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] ASIGetExpStatus failed. Restarting exposure... "
[2022-04-21T01:16:48.258 CEST DEBG ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro : "[DEBUG] StartExposure->setexp : 60.000s "
[2022-04-21T01:16:48.259 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[INFO] Taking a 60 seconds frame... "
[2022-04-21T01:16:49.259 CEST INFO ][           org.kde.kstars.indi] - ZWO CCD ASI533MC Pro :  "[ERROR] Exposure failed after 3 attempts. "