Hetio v0.2.3 breaks integration code

[veleritas]

Hi Daniel,

Just wanted to note that the new assertions you added into v0.2.3 about whether a node or edge have already been integrated into the network breaks earlier code in the integrate repository.

Looking at the blame for hetnet.py shows that those assertion lines were added only 21 days ago with commit d32ba2a, and are therefore very recent (I wasn't having these problems before).

For now I've just commented out the cells which add edges which already exist in the network, but I just wanted you to know that dhimmel/integrate/integrate.ipynb won't run directly anymore with the latest hetio release.

[dhimmel]

Eek. Okay I'll look into this. Sorry!

[veleritas]

The easy way might be to modify hetio so that if the node or edge already exists in the network, then the code just skips adding that node/edge again. This would probably be easier than modifying all of the integrate code to use try/catch statements to deal with all of the assertion errors.

[dhimmel]

@veleritas as you suspected, d32ba2a adds a check for duplicate edge additions. However, duplicate nodes additions previously threw an error. I was surprised when I looked through the code and saw that duplicate edges were allowed to be added.

I definitely think attempting to add a duplicate edge should throw an error... allowing additions of duplicates edges will almost certainly lead to a host of other bugs and confusing behaviors down the road.

The easy way might be to modify hetio so that if the node or edge already exists in the network, then the code just skips adding that node/edge again.

I'd rather implement these checks or error handing in integrate.ipynb. I'll look into where duplicate edges are being added in integrate.ipynb and attempt to resolve these issues. Remember the Zen:

Errors should never pass silently.

Hopefully, I can identify a solution that:

1. makes integrate.ipynb work with hetio>=0.2.3
2. reproduces Hetionet v1.0 exactly

I'm thinking that in the past when a duplicate edge was added, it would essentially overwrite the existing edge. So I'm crossing my fingers that drop_duplicates in pandas using keep='last' can save us here. Will let you know.

[dhimmel]

Here's the situation we're dealing with.

The following issues exposed by d32ba2a cause changes to the hetnet:

1. 2925 Gene-interacts-Gene edges are self-loops. This makes sense (for example homodimers)
2. Self to self gene pairs in output data dhimmel/erc#2: 2 ERC gene pairs were self-loops (this does not make sense, should be removed)

I believe, these changes will affect the hetnet compared to Hetionet v1.0. This is because previously the self-loops of bidirectional edges would be considered inverted and therefore not get exported. d32ba2a added support for self loops.

The following issues exposed by d32ba2a do not cause changes to the hetnet:

1. Duplicate gene pairs in output data dhimmel/erc#1: one ERC gene pair was duplicated. This does not make sense, Resolved by dropping the duplicate gene pair with keep='last'.
2. dhimmel/drugcentral@0e085dd: duplicated Compound-includes-PharmacologicClass edges. Resolved by removing duplicates upstream in dhimmel/drugcentral.

These issues became apparent due to the duplicate edge checking added in d32ba2a. I currently believe these fixes will not affect the hetnet compared to Hetionet v1.0.

I'm still investigating the issue.

[dhimmel]

Effect on Project Rephetio

d32ba2a enabled bidirectional self-loops in hetio. Here I'll assess the downstream effect on Project Rephetio of allowing self-loops in integrate.ipynb. 2925 new GiG and 2 new GcG edges would be added. When computing features, the path counts should stay the same as we exclude paths with duplicate nodes. However, degree-weighted path counts could change, since some nodes will now have a different degree.

Therefore I think it's best to filter self-loops in integrate.ipynb so we keep the old behavior. In future versions of Hetionet, we can remove the self-interaction filter.

@veleritas I implement a fix in dhimmel/integrate#12. If you adopt the changes where you're tagged, you should be able to use hetio v0.2.3 without any downstream effects. I recommend the upgrade since the old hetio behaviors were essentially bugs. Let me know if this makes sense.

[dhimmel]

I think there's nothing outstanding on this issue, so I'll close. Happy to reopen if it's still active.