Add support for model-explorer in ArmTester

[Erik-Lundell]

If model-explorer is installed, run it on the exported_graph using ArmTester.visualize(), or use the api the visualize module directly from the debug console.

Introduces two pytest configurations:
--model_explore_host : if set, tries connecting to to a running server
rather than starting a new one.
--model_explore_port : set the port of the above host. If not set, uses default model-explorer port option which searches for a running server.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/6750

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit d98cfe6 with merge base 7fcd0af ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[digantdesai]

Do you just test locally?

[Erik-Lundell]

Yeah, this has only been tested locally. I am not sure how to test in ci since it requires an external dependency

[digantdesai]

@Olivia-liu - what do you think of this vs. our SDK plans for visualization?

[Erik-Lundell]

Bump @Olivia-liu, some kind of visualization would be great, so would really appreciate merging unless you have other plans soon

[Olivia-liu]

@Erik-Lundell Sorry about the delay! This is great. Glad to see integration with model-explorer going on!

[digantdesai]

Let me get back to you @Erik-Lundell - apologies for the slow responses here.

[digantdesai]

@Olivia-liu - do you think we should make this a devtools utility which can be leveraged by other delegates and/or ET itself? Rationale is, such shared facilities tend to get better faster overtime.

[Olivia-liu]

@Olivia-liu - do you think we should make this a devtools utility which can be leveraged by other delegates and/or ET itself? Rationale is, such shared facilities tend to get better faster overtime.

@digantdesai I agree. Since this visualization isn't Arm specific, it makes sense to make it a shared utility. @Erik-Lundell Would you be able to move it under executorch/devtools? You can create a new directory executorch/devtools/visualization.

[Olivia-liu]

To change it to a shared utility that's not Arm delegate specific

[Erik-Lundell]

The idea here was to make it super convenient to use with our tester - the api for just visualizing an exported program in model-explorer is very minimal (basically just model_explorer.visualize_pytorch(exported_program, *options). The project has fixed a bug regarding reuse_server since this PR which this PR had to work around which makes it seem a bit more involved) so I don't think it makes sense to create a utility around it.

What I could do is add it to the xnnpack tester which we inherit from. But I also think I need some guidance on adding model-explorer as a new dependency, can I add it as an optional devtool dependency, should I handle if it is not installed, or something else.

[digantdesai]

I am not a devtools expert, @Olivia-liu is, but how I imagine this as,
(1) visualize(exported_program) should be a devtools utility - including all of visualize.py here
(2) This should not be any delegate specific
(3) This should not be limited to testers
(4) we should make sure it works smoothly i.e. we install right packages with ET setup, have proper docs, etc.

I remember Olivia had something like this but not on the Github yet, so just want to make sure we are aligned and not doing throwaway work here.

Also not want to block you @Erik-Lundell - so I would say just move this out of arm dir to devtools and get a stamp from @Olivia-liu .

[Erik-Lundell]

I am not a devtools expert, @Olivia-liu is, but how I imagine this as, (1) visualize(exported_program) should be a devtools utility - including all of visualize.py here (2) This should not be any delegate specific (3) This should not be limited to testers (4) we should make sure it works smoothly i.e. we install right packages with ET setup, have proper docs, etc.

I remember Olivia had something like this but not on the Github yet, so just want to make sure we are aligned and not doing throwaway work here.

Also not want to block you @Erik-Lundell - so I would say just move this out of arm dir to devtools and get a stamp from @Olivia-liu .

Thanks, I have some idea of how to go forward with this now. It'll take some time until I can make the changes, I'll mark the PR as a draft until then.

[Erik-Lundell]

Since the the new patch was completely different I made a new PR, see #7554 @Olivia-liu