Add feature to show names of layer dependencies in HLG JupyterLab repr

[aomirolis]

This PR implements the enhancement proposed by @gjoseph92 on #8567.

I opened a new PR (closed previous #9061) with the corrections suggested by @jsignell and @gjoseph92, thank you a lot for your quick response and feedback!

[GPUtester]

Can one of the admins verify this patch?

[pavithraes]

@aomirolis Thank you for opening this! For future reference, you can push update commits to the branch you use for the PR, it helps consolidate all discussion and feedback in the same place. :)

Some CI tests are failing with:

TypeError: Layer._repr_html_() missing 1 required positional argument: 'dependencies'

Since we added a new dependencies argument in this PR, could you please update all the tests where _repr_html_() is used?

[aomirolis]

Thank you! I'll keep that in mind.

[pavithraes]

@aomirolis Thanks for the updates!

@gjoseph92 Could you please take a look at this when you get a chance?

[jsignell]

This looks nice and clean but I don't actually see any dependency information in the output. Is it possible that that information isn't actually available until the graph is materialized @gjoseph92?

[gjoseph92]

Suggested change

	def _repr_html_(self, dependencies, layer_index="", highlevelgraph_key=""):
	def _repr_html_(self, layer_index="", highlevelgraph_key="", dependencies=()):

This should be an optional argument. Then you can revert your changes to all those tests as well.

I would have expected this to break rendering a Layer instance (as opposed to an HLG instance) in an actual notebook?

[aomirolis]

Thank you for the suggestion! May I ask why it should be this way? I will soon make the suggested changes.

[jsignell]

By adding dependencies as an optional kwarg at the end of the method, this PR is fully backwards compatible. So any existing code will continue to work even if dependencies is not passed.

[aomirolis]

Makes sense, thank you!

[gjoseph92]

@jsignell are you trying this out?

Is it possible that that information isn't actually available until the graph is materialized

That's definitely not the case, that's just part of the HLG structure. If it't not showing up, then I think something else is wrong.

[jsignell]

Yeah I was trying it out with:

import dask.array as da

arr1 = da.zeros(shape=(100,100), chunks=(5,10))
arr2 = arr1.T + 100
arr3 = arr1.dot(arr2)
arr3.dask

[aomirolis]

This code produces this output for me. You can see the dependencies of layer 4 at the last two lines. Am I missing something?

[jsignell]

So sorry for the noise @aomirolis my git branch checkout failed and I didn't notice. I see the intended behavior now just as in your screenshot.

[aomirolis]

No problem!!

[jsignell]

This looks great! Thanks @aomirolis for sticking with it!