New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GEXF conversion #54

Merged
merged 6 commits into from May 30, 2018

Conversation

Projects
None yet
3 participants
@jgosmann
Member

jgosmann commented Oct 12, 2017

(based on PR #56)

This PR adds a conversion of Nengo model to the GEXF format, which can be opened with Gephi, a visualization and explanatory analysis software for graphs. The motivation behind this is that Nengo GUI currently is not powerful enough to get a good overview over complex models (e.g., no support to color or temporarily hide objects, no possibility to identify inhibitory or modulatory connections).

A screenshot from a Nengo model visualized in Gephi:
screenshot_125402

The main class is GexfConverter, doing the basic conversion of a model. But usually there is a bunch of networks (like spa.State) where all the detail is too much. CollapsingGexfConverter thus extends GexfConverter to keep certain networks collapsed. Both classes use the DispatchTable descriptor to define conversion methods for different types of Nengo objects. This descriptor allows to overwrite methods in derived classes or on instances to customize the conversion.

Furthermore, there is a InspectiveLabeler class to obtain the labels for the graph nodes. This is done by going through the referrers of an object and looking for dictionaries where the key could be used as name and stack frames that contain the object. This might be mildly interesting for Nengo GUI (@tcstewar @tbekolay) as an alternate way to obtain names, but is probably a worse approach. Getting the referrers is a slow operation (because Python has to scan all objects). Also, we cannot get the name for objects that have been created in a function that finished excuting without assigning the object as an attribute. Even worse, those objects get a name like 'o' (depending on the context in which the labeler is executed). Currently, this affects only a few nodes in my model (so I won't spend time on this for now), but might be a limitation for other models.

The GEXF format supports a hierarchical graph representation that can be activated with the hierarchical=True flag. However, in the current Gephi version support for hierarchical graphs has been removed (last version with that support is 0.8.x which can still be run with JRE <= 1.7). Thus, it will flatten a hierarchical graph and leave an unconnected node for every network. The default hierarchical=False will write a flat graph and avoid those unconnected nodes.

@hunse

Cool! Seems like a good thing to have.

Could you put an example of how to visualize a model using Gephi in the examples? I think that would really help others start to make use of this.

with nengo.Network() as model:
ea = nengo.networks.EnsembleArray(10, 10)

assert ea # hide pylint unused warning

This comment has been minimized.

@hunse

hunse Oct 12, 2017

Contributor

Can't you just take the ea = out of the line above to get rid of the warning?

This comment has been minimized.

@jgosmann

jgosmann Oct 12, 2017

Member

Unfortunately not, because then the label in the generated XML will depend on the code determining the labels (due to the limitations of the labeler). This is definitely not clear from the code. Do you think explaining this in a comment makes sense?

This comment has been minimized.

@hunse

hunse Oct 16, 2017

Contributor

Yeah, maybe a comment or something would help.

So the XML has labels based on the variable names used in the code? Why not label things the same way we do in nengo_gui, using each object's label attribute? I remember we talked about this when designing nengo/nengo_gui, and decided that basing labels on variable names could be problematic, for example where ensembles are created in a for-loop or list comprehension, or some other way that is not variable_name = single_nengo_object.

This comment has been minimized.

@jgosmann

jgosmann Oct 16, 2017

Member

Yes, for-loops are another situation where this fails. I was assuming that the GUI requires the source code to figure out the labels, but I just looked at the source again and it seems I was wrong. There might still be situations for my particular model where the GUI gives worse names, but I have to verify that.

I wonder whether it is better to import NameFinder from the GUI or copy it over (code duplication vs. risk of changes in the GUI breaking this code).

This comment has been minimized.

@tbekolay

tbekolay Oct 16, 2017

Member

The GUI does use the source code; the name finder is pre-loaded with the locals dict which is filled by execing the Nengo script. The label of the object is only used when explicitly asked for (otherwise you get the unique identifier, which is something that you can eval in the context of locals to get that object).

'xmlns="http://www.gexf.net/1.3draft" '
'xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" '
'xsi:schemaLocation="http://www.gexf.net/1.3draft '
'http://www.gexf.net/1.3draft/gexf.xsd">'

This comment has been minimized.

@hunse

hunse Oct 12, 2017

Contributor

Does this preamble need to be here? It seems like the kind of thing that might change arbitrarily, and it would be nice if our tests didn't break if it does.

This comment has been minimized.

@jgosmann

jgosmann Oct 12, 2017

Member

Not necessarily ... though without it these might change arbitrarily to something wrong by accident?
I'm fine either way, if you think it should be removed from the tests, I'll do it.

This comment has been minimized.

@hunse

hunse Oct 16, 2017

Contributor

Oh, sorry, I just looked at the code again, and it's our code that determines these headers. I had assumed it was an external library that was specifying them. In that case, I'm fine to leave them in the tests, because if/when we change them in the main code, we can also change the tests. (It is another thing to keep updated, but that's not a big deal, and probably best to make sure the headers are not forgotten.)

@jgosmann

This comment has been minimized.

Member

jgosmann commented Oct 12, 2017

Could you put an example of how to visualize a model using Gephi in the examples?

I'm hoping to do a presentation in a lab meeting on this. Though, having an example in writing would certainly be nice, but also a bit of effort ...

@hunse

This comment has been minimized.

Contributor

hunse commented Oct 16, 2017

I'm hoping to do a presentation in a lab meeting on this. Though, having an example in writing would certainly be nice, but also a bit of effort ...

I think that effort is worth it. Whether you prepare the example now, or for the lab meeting in the future, an example is necessary if you want others to use this, IMO. The problem with only presenting it in a lab meeting is that a) the people who saw the presentation will forget the details, and not have anything to reference, b) anybody who didn't see the presentation (whether they were away, or someone outside our immediate lab group) won't have anything to work from. Examples become especially useful when things like the Nengo summer school come up, and we want to help introduce these extra tools to new people.

I'm fine to merge this without an example for now, so that it's easier for you to use and maintain, as long as the plan is to add an example eventually.

@jgosmann

This comment has been minimized.

Member

jgosmann commented Oct 16, 2017

I think that effort is worth it.

Yes, I completely agree. I just have to find the time ...

@jgosmann jgosmann referenced this pull request Oct 23, 2017

Closed

Add GEXF documentation #56

@jgosmann jgosmann self-assigned this Nov 13, 2017

@jgosmann jgosmann force-pushed the gexf branch 3 times, most recently from 62b0d89 to d4ae3b5 Nov 18, 2017

@jgosmann

This comment has been minimized.

Member

jgosmann commented Nov 22, 2017

Rebased this PR onto #56 (setting up documentation for nengo_extras) and added a tutorial on Nengo+Gephi.

@jgosmann jgosmann removed their assignment Nov 22, 2017

@tbekolay

I've rebased this and changed it slightly so it works with the new docs organization. Looks like there are a few Python 2.7 issues, which I should be able to fix. Aside from that, I would also like to convert guide.rst to a notebook and put the images somewhere else (outside the repository, to keep it small) but have them still show up in the notebook. Are all those changes OK with you @jgosmann ?

@jgosmann

This comment has been minimized.

Member

jgosmann commented May 28, 2018

Sounds good to me. It'd be nice if building the documentation with the images somewhere else does not involve much more effort and if it is clear how to change/update them.

@tbekolay

This comment has been minimized.

Member

tbekolay commented May 28, 2018

It'd be nice if building the documentation with the images somewhere else does not involve much more effort and if it is clear how to change/update them.

I was going to just upload them to imgur and include them with IPython.display.Image. Any objection to that, or alternatives?

@jgosmann

This comment has been minimized.

Member

jgosmann commented May 28, 2018

Doesn't imgur delete images after a while? I feel like it is not uncommon to come along broken imgur links. But that's just anecdotal evidence (at best).

@tbekolay

This comment has been minimized.

Member

tbekolay commented May 28, 2018

As of Feb 2015 images are only deleted upon request, as long as they're uploaded by a logged in user (which I would be).

@tbekolay tbekolay force-pushed the gexf branch 2 times, most recently from 2328169 to dc78585 May 29, 2018

@tbekolay tbekolay merged commit ac0f9cc into master May 30, 2018

1 check was pending

continuous-integration/travis-ci/push The Travis CI build is in progress
Details

@tbekolay tbekolay deleted the gexf branch May 30, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment