Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lx200_10micron not correctly reporting the parking status #1582

Closed
fenriques opened this issue Dec 29, 2021 · 48 comments
Closed

lx200_10micron not correctly reporting the parking status #1582

fenriques opened this issue Dec 29, 2021 · 48 comments
Assignees
@d33psky
Labels
bug

Comments

@fenriques
Copy link
Contributor

When parking a 10micron mount the light representing the park status never changes.
It is always green, never busy (yellow) even when the mount is slewing.

The issue was tracked down by Wolfgang in this forum thread where we provide some examples and details:
https://indilib.org/forum/mounts/10549-10micron-lx200-different-parking-behavior-when-using-scheduler-or-indi-driver.html

The issue is more than the color itself: the scheduler for example relies on this status to close the dome/roof.
When the scheduler issues the command to park the mount, immediately after it sees the mount as parked (while the mount is still moving) and the roof begins dangerously to move.

I do not know if this affects 10micron only or all the lx200 family.

Ferrante
@fenriques fenriques added the bug label Dec 29, 2021
@knro knro closed this as completed in 03e588e Dec 30, 2021
@fenriques
Copy link
Contributor Author

hi Jasem,
I've tested with the latest version installed and the issue is still there.
The park light is always green regardless of the mount slewing, (un)parking, tracking or stopped.

@knro
Copy link
Contributor

knro commented Jan 18, 2022

Something must be resetting the TrackState then. I suspect in ReadScopeStatus starting in like 467. It could be the are are in parking but the state is overwritten here.

@d33psky
Copy link
Contributor

d33psky commented Jan 18, 2022

Just a note: I'll try to reproduce and report back.

@d33psky d33psky reopened this Jan 27, 2022
@fenriques
Copy link
Contributor Author

hi d33psky,
thanks for taking the time to look into this issue.
Can I contribute in testing and/or give further information?

Ferrante

@d33psky
Copy link
Contributor

d33psky commented Mar 2, 2022

Hi Ferrante, yes please when I have something to test ... I'm still poking in the dark with this issue.

@mads0100
Copy link

mads0100 commented Apr 1, 2022

I'm seeing strange parking issues inside of Scheduler. Essentially it doesn't park, sometimes. Is there something I can do to help? What logs do you need me to gather? I should have clear skies all night.

@knro
Copy link
Contributor

knro commented Apr 1, 2022

Enable mount + scheduler logging.

@mads0100
Copy link

mads0100 commented Apr 1, 2022 via email

@mads0100
Copy link

mads0100 commented Apr 3, 2022

Hey Jasem,

So the issue seems to be that INDI can't tell the 10u to unpark. When I try to let the scheduler do it, it just saying unparking and repeats itself over and over again. If I go into MW4 and tell it to unpark and then run scheduler, it'll work. But, if scheduler seems a delay and tells it to park... it'll go into that cycle and fail. Not quite sure what the fix is. I know MW4 is 'native' speaking to the mount. I don't know what happened to the driver that would cause this. I hope the logs help; I'll keep gathering them till you tell me not to. If there is something specific you guys need to see, please let me know and I'll do it. It's clear for the next few days at least.

Chris

[log_07-53
log_19-29-17.txt

log_18-40-14.txt
log_22-53-58.txt

log_19-48-05.txt
-09.txt](https://github.com/indilib/indi/files/8405504/log_07-53-09.txt)
log_08-13-14.txt

@d33psky
Copy link
Contributor

d33psky commented Apr 4, 2022

I see this in one of the logs

[2022-04-01T21:00:16.549 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Scheduler is awake. Jobs shall be started when ready..."
[2022-04-01T21:00:19.557 CDT INFO ][ org.kde.kstars.ekos.scheduler] - Unparking mount in progress...
[2022-04-01T21:00:19.560 CDT INFO ][           org.kde.kstars.indi] - LX200 10micron :  "[INFO] Unparking. "
[2022-04-01T21:00:19.561 CDT INFO ][           org.kde.kstars.indi] - LX200 10micron :  "[INFO] Mount is unparked. "
[2022-04-01T21:00:19.564 CDT INFO ][           org.kde.kstars.indi] - WatchDog :  "[INFO] Mount is Unparked "
[2022-04-01T21:00:19.825 CDT INFO ][           org.kde.kstars.indi] - LX200 10micron :  "[INFO] Mount is parked. "
[2022-04-01T21:00:19.826 CDT INFO ][     org.kde.kstars.ekos.align] - Target updated to JNow RA: "11h 57m 43s" DE: " 89° 45' 17\""
[2022-04-01T21:00:19.828 CDT INFO ][           org.kde.kstars.indi] - WatchDog :  "[INFO] Mount is Parked "
[2022-04-01T`
[2022-04-01T21:00:20.987 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount parked."
[2022-04-01T21:00:22.048 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount parked."
[2022-04-01T21:00:23.058 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount parked."
...

I've never seen that before. ?!

The 21:00:19.825 message [ org.kde.kstars.indi] - LX200 10micron : "[INFO] Mount is parked. " should be the clue. No idea yet why this happens.
The mount got polled a second later where it reported being unparked with :
21:00:20.964 CDT INFO ][ org.kde.kstars.indi] - LX200 10micron : "[INFO] Gstat changed from 5 to 0 "
So the end state here is that the driver knew that the mount was unparked, yet the scheduler did not know this.

@mads0100
Copy link

mads0100 commented Apr 4, 2022 via email

@mads0100
Copy link

mads0100 commented Apr 4, 2022 via email

@mads0100
Copy link

mads0100 commented Apr 9, 2022 via email

@knro
Copy link
Contributor

knro commented Apr 10, 2022

This is not an issue in Ekos, the driver should handle parking and unparking gracefully. Not sure if @d33psky has any suggestion on how to resolve this issue.

@knro
Copy link
Contributor

knro commented Jun 9, 2022

Anyway I can help in resolving this issue? @d33psky any suggestions on how to proceed?

@mads0100
Copy link

mads0100 commented Jun 9, 2022 via email

@d33psky
Copy link
Contributor

d33psky commented Jun 9, 2022

Maybe this would make the system more robust: the unpark command should not return either until the periodic poll confirms the mount is unparked, or until 1 poll period has passed.
Thoughts ?

@fenriques
Copy link
Contributor Author

from what I understood (mainly thanks to Wolfgang) it's simply that in the driver the 'park' light is not in sync with the messages and the mount status.
E.g. the mount is parking, the log shows 'parking' but the light is green (should be yellow until mount is parked).
The issue is that the scheduler reads the light and not the log. So it parks the dome while mount is still moving.

I think that mads0100 adds here another issue but not sure is related to this.

ferrante

@mads0100
Copy link

mads0100 commented Jun 10, 2022 via email

@mads0100
Copy link

I'm still experiencing this with the newest release. It's become more unstable; when I polar aligned tonight, it would take an image, I would manual slew (it doesn't give me speeds and doesn't work), after a few seconds it would decide to park itself. I couldn't figure out a reason for why; I tried killing MW4; tried unparking with the mount module and Ekos control panel. If I let it park and then slewed back to where it was, it often times didn't do it again. But, sometimes it did.

Super frustrating. If you need logs or whatever please ask. It's still doing the same thing where it hops between parking states. I wish it would just poll the mount instead of trying to 'force' whatever state is in Ekos; Ekos seems to have an issue with telling the mount to 'unpark' and 'track'.

@d33psky
Copy link
Contributor

d33psky commented Aug 14, 2022

Hi mads0100, yes mount logs would be good because what you describe sounds different from what was pasted here on April 4th, as such it may be a new problem.
I've never seen the mount park by itself like how you described. Even if you have MW4 up and running next to EKOS that should not happen. I use MW4 myself too to make pointing models, it's awesome.

@mads0100
Copy link

mads0100 commented Aug 15, 2022 via email

@d33psky
Copy link
Contributor

d33psky commented Aug 16, 2022

So you're saying from a parked position you press 'Tracking -> on' which unparks and starts tracking. (I never tried that, I always select 'Parking -> Unpark' to unpark).

And when from a parked position you press 'Parking -> Unpark' nothing happens in Either EKOS or MW4 ? That is weird because that definitely works for me. If I do not do this step then I cannot perform GOTOs.

I'll have to check what happens if I unpark from MW4, whether INDI sees that. Gstat should change at least.

I'm on a laptop, not on a PI. In MW4 I use 4x4 binned 2s exposures at 1657mm with an asi1600 and those all resolve with success within 2 to 3 seconds or so.

@mads0100
Copy link

mads0100 commented Aug 16, 2022 via email

@fenriques
Copy link
Contributor Author

Hi all,
as far as I can see the issue is still there and it's quite dangerous if you are not aware of it: a roof or dome can close when the mount is still slewing and unparked.

I looked into the lx200_10micron and its parent classes. It seems that the TrackState is correctly set in lx200_10micron but the TrackStateSP is not updated anywhere in the class.

TrackStateSP is not managed by the parent class inditelescope either because I guess that the different TrackStates are mount specific: in fact other lx200 classes have extensive code to set TrackStateSP e.g. lx200_OnStep.cpp from line 2178 on.

The issue is then that the Scheduler reads TrackStateSP and not TrackState to trigger actions.

Does this make sense?
Ferrante

@knro
Copy link
Contributor

knro commented Oct 3, 2022

TrackState is updated in NewRaDec in INDI::Telescope so that's Ok. Regarding scheduler, that's a different topic, but I don't see where TrackStateSP is used? can you point it out?

@fenriques
Copy link
Contributor Author

fenriques commented Oct 3, 2022 •
edited

hi Jasem,
I didn't looked into the Scheduler code. My comment was based on the log and what Wolfgang commented in the forum thread:
https://indilib.org/forum/mounts/10549-10micron-lx200-different-parking-behavior-when-using-scheduler-or-indi-driver.html#78897

What I see in the log is that the trackState is updated correctly in the driver:
"[INFO] Gstat changed from 0 to 2 " (from tracking to parking)
but the Scheduler ignores it and immediately (1sec after) logs the status as parked:
[ org.kde.kstars.ekos.scheduler] - "Mount parked."

ferrante

@mads0100
Copy link

mads0100 commented Oct 3, 2022 via email

@knro
Copy link
Contributor

knro commented Oct 4, 2022

You can park and watch the log for the state changes. Just INDI, not related to Ekos.

@mads0100
Copy link

logs.tar.gz

Logs attached for the last... long time. They all had mount/INDI enabled.

@knro
Copy link
Contributor

knro commented Nov 22, 2022

@d33psky Any ideas given the logs above?

@d33psky
Copy link
Contributor

d33psky commented Nov 26, 2022

Oh, I missed those. Looking through them RN. I see several weird things that make no sense. Will get back later with a summary that hopefully provides the needed clue.

@d33psky
Copy link
Contributor

d33psky commented Nov 26, 2022

Ok I think I scraped some clue from the logs :

Here we see INDI connecting to a parked mount, perfectly normal :

[2022-10-03T18:40:19.843 CDT INFO ][           org.kde.kstars.indi] - LX200 10micron :  "[INFO] Gstat initialized at 5 "
[2022-10-03T18:40:19.845 CDT INFO ][           org.kde.kstars.indi] - LX200 10micron :  "[INFO] Mount is parked. "

The driver info gets picked up by the higher mount class :

[2022-10-03T18:40:20.279 CDT INFO ][     org.kde.kstars.ekos.mount] - "Telescope info updated successfully."
[2022-10-03T18:40:21.016 CDT DEBG ][     org.kde.kstars.ekos.mount] - Mount status changed from  "Idle"  to  "Parked"

Then the scheduler considers 3 targets and starts with sleeping :

[2022-10-03T18:47:27.381 CDT DEBG ][ org.kde.kstars.ekos.scheduler] - "Greedy Scheduler scheduling next job IC 1805 at 20:19"
[2022-10-03T18:47:28.452 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Sleeping until observation job IC 1805 is ready at Mon Oct 3 20:19:27 2022..."

Then something weird happens. There are NO indi driver log lines, but out of the blue this is logged :

[2022-10-03T18:53:18.120 CDT DEBG ][     org.kde.kstars.ekos.mount] - Mount status changed from  "Parked"  to  "Error"
[2022-10-03T18:53:18.128 CDT DEBG ][ org.kde.kstars.ekos.scheduler] - Mount State changed to 6
[2022-10-03T18:53:20.119 CDT DEBG ][     org.kde.kstars.ekos.mount] - Mount status changed from  "Error"  to  "Parked"
[2022-10-03T18:53:20.182 CDT DEBG ][ org.kde.kstars.ekos.scheduler] - Mount State changed to 5
[2022-10-03T20:19:27.118 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Scheduler is awake. Jobs shall be started when ready..."

That looks bad. I do not know what it means or if it even has consequences.
I consider this a bug, but it is outside the LX200 10micron device driver.

With the scheduler awakes at the announced timeslot the mount is instructed to unpark :

[2022-10-03T20:19:30.122 CDT DEBG ][ org.kde.kstars.ekos.scheduler] - "Checking Startup State (4)..."
[2022-10-03T20:19:30.123 CDT DEBG ][           org.kde.kstars.indi] - ISD:Telescope: UnParking...

[2022-10-03T20:19:30.123 CDT INFO ][ org.kde.kstars.ekos.scheduler] - Unparking mount in progress...
[2022-10-03T20:19:30.126 CDT INFO ][           org.kde.kstars.indi] - LX200 10micron :  "[INFO] Unparking. "
[2022-10-03T20:19:30.127 CDT INFO ][           org.kde.kstars.indi] - LX200 10micron :  "[INFO] Mount is unparked. "
[2022-10-03T20:19:30.134 CDT INFO ][           org.kde.kstars.indi] - WatchDog :  "[INFO] Mount is Unparked "

But right after that the driver reports the mount is parked again :

[2022-10-03T20:19:31.099 CDT INFO ][           org.kde.kstars.indi] - LX200 10micron :  "[INFO] Mount is parked. "
[2022-10-03T20:19:31.100 CDT INFO ][     org.kde.kstars.ekos.align] - Target updated to JNow RA: "19h 00m 00s" DE: " 89° 49' 31\""
[2022-10-03T20:19:31.102 CDT INFO ][           org.kde.kstars.indi] - WatchDog :  "[INFO] Mount is Parked "

This is a bug. And I see no Gstat lines anywhere near these lines.

What is probably a consequence for the scheduler about this unintended state change, it now logs :

[2022-10-03T20:19:31.122 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount parked."
[2022-10-03T20:19:32.122 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount parked."
[2022-10-03T20:19:33.122 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount parked."
[2022-10-03T20:19:34.122 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount parked."
[2022-10-03T20:19:35.123 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount parked."

every second, which appears to have stopped by itself almost 20 minutes later (no idea why) :

[2022-10-03T20:37:40.123 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount parked."
[2022-10-03T20:37:41.282 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount parked."
[2022-10-03T20:37:42.240 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount parked."
[2022-10-03T20:37:43.197 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount parked."
[2022-10-03T20:37:44.186 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount parked."
[2022-10-03T20:37:45.066 CDT INFO ][           org.kde.kstars.indi] - LX200 10micron :  "[INFO] Unparking. "
[2022-10-03T20:37:45.068 CDT INFO ][           org.kde.kstars.indi] - LX200 10micron :  "[INFO] Mount is unparked. "
[2022-10-03T20:37:45.070 CDT INFO ][           org.kde.kstars.indi] - WatchDog :  "[INFO] Mount is Unparked "
[2022-10-03T20:37:45.119 CDT DEBG ][     org.kde.kstars.ekos.mount] - Mount status changed from  "Parked"  to  "Idle"
[2022-10-03T20:37:45.125 CDT DEBG ][ org.kde.kstars.ekos.scheduler] - Mount State changed to 0
[2022-10-03T20:37:45.149 CDT INFO ][ org.kde.kstars.ekos.scheduler] - "Mount unparked."

And again a toggle from unparked to parked state out of the blue :

[2022-10-03T20:37:45.185 CDT INFO ][           org.kde.kstars.indi] - LX200 10micron :  "[INFO] Mount is parked. "
[2022-10-03T20:37:45.186 CDT INFO ][     org.kde.kstars.ekos.align] - Target updated to JNow RA: "20h 00m 00s" DE: " 89° 49' 31\""
[2022-10-03T20:37:45.188 CDT INFO ][           org.kde.kstars.indi] - WatchDog :  "[INFO] Mount is Parked "
[2022-10-03T20:37:46.119 CDT DEBG ][     org.kde.kstars.ekos.mount] - Mount status changed from  "Idle"  to  "Parked"

There are 0 Gstat lines here since the initialization at 2022-10-03T18:40:19.843 .
This means the mount never actually unparked.

This part

[2022-10-03T20:19:30.126 CDT INFO ][           org.kde.kstars.indi] - LX200 10micron :  "[INFO] Unparking. "
[2022-10-03T20:19:30.127 CDT INFO ][           org.kde.kstars.indi] - LX200 10micron :  "[INFO] Mount is unparked. "

need to be reliable. This is its implementation :

bool LX200_10MICRON::UnPark()
{
    // #:PO#
    // Unpark
    // Returns:nothing
    LOG_INFO("Unparking.");
    if (setStandardProcedureWithoutRead(fd, "#:PO#") < 0)
    {
        return false;
    }

    TrackState = SCOPE_IDLE;
    SetParked(false);
    return true;
}

And the LX200 protocol documentation shows that the command #:PO# returns
nothing which makes this function a happy-path implementation. I hate those.

We should make this function wait for a max specified time while we probe the
actual (un)park state with Gstat until the desired state is confirmed by the
mount. Which we already know from the logs never happens (no Gstat lines).

So the mount gets a command to unpark from the driver, yet it never arrives or
is never acted upon. The driver should of course handle this better, but the
real problem seems to be in the physical communication. This also explains why
I have never seen issues like these before.

Ferrante: Is this an ethernet connection or a wifi one (between the device driver
and the mount) ?

-- Hans

@fenriques
Copy link
Contributor Author

hi Hans,
in this thread are reported two different issues. The log you're referring to is from Chris, mine is referenced in the first post of this thread.

In any case my mount has a eth connection and I experienced the same issue on two different mounts (gm1000 and gm2000).

@mads0100
Copy link

mads0100 commented Nov 26, 2022 via email

@d33psky
Copy link
Contributor

d33psky commented Nov 27, 2022

ferrante: I have a PR in prep for forced ParkSP handling from the driver. Waiting for clear weather to be able to test.

@fenriques
Copy link
Contributor Author

great news Hans! thanks

@mads0100
Copy link

mads0100 commented Nov 27, 2022 via email

@d33psky
Copy link
Contributor

d33psky commented Nov 28, 2022 •
edited

Clouds but no rain, so I disabled the dome locking safety feature as that depends on my weather station to test.

Here's the results :

  • start in parked state: green
    This is done in the periodic LX200_10MICRON::ReadScopeStatus() case GSTAT_PARKED
  • park when parked: white/off
    This is done in inditelescope.cpp after line 1104 if (toPark && TrackState == SCOPE_PARKED
  • unpark when not allowed via dome: red
    This is done in inditelescope.cpp after line 1093 if (toPark == false && isLocked())
  • unpark when parked and allowed: green
    This is done in LX200_10MICRON::Unpark() and is unsafe/happy path. Needs improvement.
  • parking: yellow
    This is NEW in LX200_10MICRON::Park() via ParkSP.s
  • parked: green
    This is done in the periodic LX200_10MICRON::ReadScopeStatus() case GSTAT_PARKED

I have not tested a schedule yet but this yellow light is at least an improvement.

@mads0100
Copy link

mads0100 commented Nov 28, 2022 via email

@fenriques
Copy link
Contributor Author

Hi Hans,
The 'parking yellow' seems to be solving my issue.
Have you tried a long (≥10 sec) parking procedure? That's were my problem happened : the light turned green before reaching the park position.

All the other new features will improve the reliability of the driver. Thanks for taking care of that.

As soon as you submit the PR and it's merged I will be able to test it.

Ferrante

@d33psky
Copy link
Contributor

d33psky commented Nov 28, 2022

Hi Chris, I have not made a PR yet but if you're proficient in compiling stuff yourself and in a big hurry then you can use this branch on my fork of the indi repo to get to the code RN: https://github.com/d33psky/indi/tree/HL-issue1582-a

Hi Ferrante, yes I did a several seconds long slew initially to test a long slew park, and for that I had to open the roof so had to wait for the rains to pass.

I can make a PR RN of course but the happy-path unpark is not fixed up properly yet. And I'm not sure how to even; there is a Gstat state 3 for 'unparking' but I never see it (it always goes straight from 5 to 0 for my setup) :

[2022-11-28T22:57:33.799 CET INFO ][           org.kde.kstars.indi] - 10micron :  "[INFO] Unparking. "
[2022-11-28T22:57:33.799 CET DEBG ][           org.kde.kstars.indi] - 10micron : "[SCOPE] CMD <#:PO#> "
[2022-11-28T22:57:33.800 CET INFO ][           org.kde.kstars.indi] - 10micron :  "[INFO] Mount is unparked. "
[2022-11-28T22:57:33.800 CET INFO ][           org.kde.kstars.indi] - Dome Scripting Gateway :  "[INFO] Telescope status changed. Lock is set to: locked "
[2022-11-28T22:57:33.801 CET INFO ][           org.kde.kstars.indi] - WatchDog :  "[INFO] Mount is Unparked "
[2022-11-28T22:57:33.881 CET DEBG ][           org.kde.kstars.indi] - 10micron : "[SCOPE] CMD <#:Ginfo#> RES <15.065057,+39.61398,E,357.09639,+01.24001,2459912.41484943,0,0#> "
[2022-11-28T22:57:33.882 CET INFO ][           org.kde.kstars.indi] - 10micron :  "[INFO] Gstat changed from 5 to 0 "
[2022-11-28T22:57:33.882 CET DEBG ][           org.kde.kstars.indi] - 10micron : "[SCOPE] CMD <#:GS#> "
[2022-11-28T22:57:33.978 CET DEBG ][     org.kde.kstars.ekos.mount] - Mount status changed from  "Parked"  to  "Tracking"

It's annoying the mount does not signal when it is done unparking, maybe I can derive it from a tracking state (0), or a special state (like 6 or 7 or 8 up to 99)

@d33psky d33psky mentioned this issue Dec 15, 2022
Merged
@d33psky
Copy link
Contributor

d33psky commented Dec 15, 2022

I've made it a PR #1787

@mads0100
Copy link

mads0100 commented Dec 15, 2022 via email

@knro
Copy link
Contributor

knro commented Dec 24, 2022

is this confirmed fixed now?

@fenriques
Copy link
Contributor Author

I'm still testing it.
Issuing the park command from the driver tab gives the right behavior: the light turns yellow until the mount is parked and the 'parked' message in the log is printed when the mount is parked (and not just 1 sec after the park button is pressed).
And the roof moves only after the mount slew to park position.

Still I have to check the behavior from the scheduler.

@knro
Copy link
Contributor

knro commented Jan 15, 2023

I'm marking this as fixed. Please reopen if not working.

@knro knro closed this as completed Jan 15, 2023
@mads0100
Copy link

mads0100 commented Jan 15, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@d33psky d33psky

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@d33psky @knro @fenriques @mads0100