Uncaught undefined results in Something went wrong pages in pipeline runs

[christianversloot]

Checks

• I added a descriptive title to this issue
• I have searched (google, github) for similar issues and couldn't find anything
• I have read and followed the docs and still think this is a bug

General

Since ZenML 0.50.0 it frequently happens that pipeline runs show this error:

It seems to be caused by referring to undefined somewhere:

That somewhere seems to be here in the build:

                        if ("step" === a.type && a.id === n.target && "running" === a.data.status) {
                            var r = e.find((function(e) {
                                return e.id === n.source
                            }
                            ));
                            "running" === e.find((function(e) {
----->				return "step_" + r.data.parent_step_id === e.id
                            }
                            )).data.status && (n.animated = !0)

Slightly above in the stack I'm seeing null === t || void 0 === t || t.forEach((function(n) {

This suggests it's happening here (and if not, here).

Here, n must be defined as initialNodes.find((node) => node.id === edge.source); compared with the passed graph edge. I'm not entirely sure why that's the case, but it seems like sometimes the console writes a set of nodes and edges...

{nodes: Array(11), edges: Array(13), rootStepId: 'step_1b3bd291-a2f0-4570-880a-63a036a46c25', runMetadata: Array(0), runId: '8848e65e-3beb-41a4-9532-bb45c62e6d0e'}

...which at that point is not present in the response of GET /api/v1/runs/<id>/graph.

The ones with step_02c60d8e-d551-47dd-921a-4af1d91bcc6f in the screenshot were present in that response, just not the one failing.

Strangely, this was exactly the step ID of a run I had open in another tab at that point in time or probably was just loading concurrently:

I'm just not sure how the response from tab 1 can end up in tab 2. Probably it can't, but maybe it's in state somewhere within tab 1, and then loaded as well? Not sure though.

Hope you can find and fix this, thanks a lot :)

[Cahllagerfeld]

I've spent some time digging into this, a summary from my side what I understood

• You had two tabs open, with two different Dags in it
• Then the UI for one of them failed, with an error that it was referring to data on an undefined object
• You opened the logs and saw that the response looked correct

My first feeling on how resposne from tab1 ends up in tab2 might be that the redux setup is configured to use localstorage. Localstorage is shared across tabs.

Does this issue also occur when only using a single tab?

[christianversloot]

Yes, like that. Regarding point 3:

• The API response looked correct indeed (in the network tab).
• Still (in the console tab) the UI attempted to refer to data on an undefined object.

If localstorage is used without a key prefix for a run, yes it could have definitely been the case that results got mixed.

[Cahllagerfeld]

Does this issue also occur in a single tab setup?

[christianversloot]

I haven't encountered it there, but I cannot state with certainty that it never happens because I'm having this problem intermittently, not all the time.

[christianversloot]

Can confirm now, it just happened in a situation where no other ZenML tab was open; same issue.

[christianversloot]

Interestingly:

1. localStorage.clear() logs me out, as expected.
2. F5 and immediately logging in again --> same issue.

In other words, it also happens while no other process could have interfered.

[Cahllagerfeld]

Okay, this is really strange, I'll need to spend some more time investigating this 🙏

[christianversloot]

For what it's worth, I mostly see it with AWS SageMaker runs which are long-lived. Not sure if it only happens there, though.

[Cahllagerfeld]

I got an error, and the described error screen, for runs that are in the running state. If I understand your comment correct

For what it's worth, I mostly see it with AWS SageMaker runs which are long-lived. Not sure if it only happens there, though.

Is this the case for you as well, that it only appears on running pipelines?

[christianversloot]

Yes, correct! And even more interestingly, when errors occur, they no longer occur after the pipeline has finished.

[Cahllagerfeld]

Sorry for the delay here...

I had some sort of the same experience. With the latest release, we changed how the polling is done while the status is running. Are you on the latest version of ZenML, and if yes is it also still occurring there?

[christianversloot]

No problem, glad it's being worked on.

Can confirm it is still occurring (just saw it with an AWS-based pipeline run) and yes we're running 0.53.1.

[christianversloot]

@Cahllagerfeld if possible do you have any update? The problem is still occurring and tbh quite annoying. Thanks! 😊

[Cahllagerfeld]

Hey @christianversloot, I'm super sorry,
I think this issue got lost in all the other things happening in the last weeks.

I will put it on my list for next week and keep you posted here!

[christianversloot]

Thank you! :)