Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Ver 0.10.2) dagit's "Tick History" chart no longer works, just shows constant spinning circle #3626

Closed
pybokeh opened this issue Jan 29, 2021 · 16 comments
Assignees
Labels
type: bug Something isn't working

Comments

@pybokeh
Copy link

pybokeh commented Jan 29, 2021

Actually, the spinning circle is running even before turning on the schedule. In other words, when viewing the page for the first time, the spinning circle animation is already running.

image

My setup:

  • Windows 10
  • Local dagit and daemon instance
  • On-prem Postgresql ver 11.5
  • dagster_postgres ver 0.10.2
  • Chrome ver 87

I will report back using default sqlite db and see if the problem persists.

@pybokeh pybokeh added the type: bug Something isn't working label Jan 29, 2021
@pybokeh
Copy link
Author

pybokeh commented Jan 29, 2021

Switched to sqlite db and even waited for 2 runs to finish. The constant spinning circle remained:

image

@prha
Copy link
Member

prha commented Jan 29, 2021

I can't repro this locally... Is there an error in the JavaScript console?

Feel free to ping me (prha) in the elementl slack to help debug...

@prha prha self-assigned this Jan 29, 2021
@pybokeh
Copy link
Author

pybokeh commented Jan 29, 2021

I switched to personal Ubuntu. running a toy pipeline, everything local with default sqlite db, and using Firefox browser ver 84.0.2

Sorry, I have little JS dev knowledge. So I'm assuming you're wanting to see output from Firefox's web console. Initially I am getting no output in Firefox's web console:

image

Then I happen to look at the web console again and this showed up:

image

Zoomed in:

image

@prha
Copy link
Member

prha commented Jan 29, 2021

This was very helpful, thanks!

This hit a bug when there's no tick history and the schedule has never been turned on. I'll also get a reasonable message surfaced and push that out in 0.10.3.

@prha prha closed this as completed in d4c33e9 Jan 29, 2021
@prha prha reopened this Jan 29, 2021
@prha
Copy link
Member

prha commented Jan 29, 2021

Ah, I'll keep this open until 0.10.3 is released next Thursday

@pybokeh
Copy link
Author

pybokeh commented Feb 4, 2021

@prha Thanks! I look forward to the 0.10.3 release today or whenever it is ready.

@pybokeh
Copy link
Author

pybokeh commented Feb 4, 2021

I've upgraded, but the problem still remains.

On the first schedule run, I did get an error:

An error occurred while resolving field DaemonHealth.allDaemonStatuses
Traceback (most recent call last):
  File "d:\python38\envs\dagster_dev\lib\site-packages\graphql\execution\executor.py", line 452, in resolve_or_error
    return executor.execute(resolve_fn, source, info, **args)
  File "d:\python38\envs\dagster_dev\lib\site-packages\graphql\execution\executors\sync.py", line 16, in execute
    return fn(*args, **kwargs)
  File "d:\python38\envs\dagster_dev\lib\site-packages\dagster_graphql\schema\instance.py", line 93, in resolve_allDaemonStatuses
    return [
  File "d:\python38\envs\dagster_dev\lib\site-packages\dagster_graphql\schema\instance.py", line 94, in <listcomp>
    GrapheneDaemonStatus(daemon_status) for daemon_status in self._daemon_statuses.values()
  File "d:\python38\envs\dagster_dev\lib\site-packages\dagster_graphql\schema\instance.py", line 56, in __init__
    lastHeartbeatErrors=[
TypeError: 'NoneType' object is not iterable

I was curious if the existing data in the Postgres was maybe causing problems, so I deleted all the dagster-related tables, re-ran the schedule. Still no change, but no longer getting that error above.

No errors showing up in the web console either or in the dagit terminal or the daemon terminal:

image

@pybokeh
Copy link
Author

pybokeh commented Feb 4, 2021

Looks like some time between the 2nd and 3rd run, got something in the web console:

image

@prha
Copy link
Member

prha commented Feb 5, 2021

Sorry about that. I'm going to run a patch release to pick up the fix for this (unrelated to the original error). I think if you start the daemon using dagster-daemon run or run dagster-daemon wipe, this problem should go away.

@pybokeh
Copy link
Author

pybokeh commented Feb 5, 2021

Upgraded personal 'nix machine with all sqlite dbs removed, problem still remains with no output in web console after 2 runs:

image

Also did not get the TypeError: 'NoneType' object is not iterable error at any point.

@johannkm
Copy link
Contributor

johannkm commented Feb 5, 2021

Hi @pybokeh, this is a new error from the latest release. Thanks for raising this- will get a fix out asap

@pybokeh
Copy link
Author

pybokeh commented Feb 5, 2021

@prha No worries! I am amazed at what dagit does or what it is able to do with so much going on underneath the hood. Just to clarify, which problem should go away? The TypeError problem or the constantly spinning circle? I just did dagster-daemon wipe then did dagster-daemon run, spinning circle still remains. FYI, like before, the spinning circle was already animating when the schedule's page was first rendered.

@johannkm
Copy link
Contributor

johannkm commented Feb 5, 2021

I have a fix ready for the TypeError: 'NoneType' object is not iterable issue, but I don't have any update on the spinner issue unfortunately.

@prha
Copy link
Member

prha commented Feb 5, 2021

Hmm, curious why you're still getting the spinner. And I guess you're not getting any errors in the console? Would love to set up a time to dig into it with you.

@pybokeh
Copy link
Author

pybokeh commented Feb 5, 2021

@prha I just checked, after 17 runs, no output in the web console on my person machine.

image

I can be free after 4pm EST. You can hit me up on dagster's Slack (Daniel Kim). Yes it is strange. The tick history was working fine for me with 0.10.0 and 0.10.1. I am puzzled that this problem can not be duplicated I'm assuming.

@nancydyc nancydyc added this to To do in Scheduled Runs & gRPC via automation Feb 8, 2021
@prha prha closed this as completed in 33ac358 Feb 9, 2021
Scheduled Runs & gRPC automation moved this from To do to Done Feb 9, 2021
@prha
Copy link
Member

prha commented Feb 9, 2021

Reopening until this is out in the release, but the main underlying issue was around fetching ticks for schedules added to the repo using the lazy repository dictionary.

Short-term work around would be to use the list-based repository definition.

Fixing rev is here:
https://dagster.phacility.com/D6359

@prha prha reopened this Feb 9, 2021
Scheduled Runs & gRPC automation moved this from Done to In progress Feb 9, 2021
@prha prha closed this as completed Feb 11, 2021
Scheduled Runs & gRPC automation moved this from In progress to Done Feb 11, 2021
mrdavidlaing pushed a commit to mrdavidlaing/dagster that referenced this issue Feb 20, 2021
Summary:
We were not handling the jobState graphql queries correctly when schedules were loaded using the lazy repository dictionary definition.

This diff hides `jobs` (but does not throw) as a repository dictionary key and adds sensors.

It also merges schedules and sensors into jobs, making sure that the `jobState` query resolves correctly.

I changed some of the graphql test structure to load a lazy repository dictionary definition as one of the multi_location test variants.  Then I hooked up the jobs tests to query the secondary location repository and made sure it loaded correctly.

Should resolve dagster-io#3626

Test Plan: bk

Reviewers: dgibson, johann

Reviewed By: dgibson

Differential Revision: https://dagster.phacility.com/D6359
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug Something isn't working
Projects
No open projects
Development

No branches or pull requests

3 participants