M. Bostock co-occurrence matrix

[smartinsightsfromdata]

I've seen this very elegant example of adjacency / co-occurrence matrix designed by M. Bostock. Very impressive.

http://bost.ocks.org/mike/miserables/

I wonder if there is anything similar already achievable with any htmlwidgets?

I see quite a bit of interesting applications...

[timelyportfolio]

This will be even easier since Elijah Meeks has made a reusable layout. If I did it, I would most likely try to squeeze into networkD3. I'll let you know.

[Cat1796]

This would be a very worthy addition to networkD3. I can think of a number of uses, but in the finance context: think about default/non default of a bond (highly non-normal). Run a large monte-carlo analysis to generate outcomes. Plotting co-occurence of default of bonds within a portfolio helps visualize adjacency, which highlights risk factors/co-dependency in the portfolio. Just my two cents. I could certainly use something like this.

[timelyportfolio]

@Cat1796 and @smartinsightsfromdata, I'll be adding this to networkD3 as this week's htmlwidget. I have a very rough prototype in this feature/adjacency branch if you want an early preview.

devtools::install_github("timelyportfolio/networkD3@feature/adjacency")

@Cat1796, I'd absolutely love to see a bond default co-occurence example.

[smartinsightsfromdata]

awesome!

[timelyportfolio]

As I work through this, I have started to realize that an adjacency matrix really is just a subset of heatmap. In this case, we can just piggyback off of d3heatmap. @smartinsightsfromdata, @Cat1796 am I missing something if I adopt this thought process?

Here are some quick examples to help me think through the issue.

# instead of building something new, try to adapt d3heatmap for this purpose
#devtools::install_github("rstudio/d3heatmap")

library(d3heatmap)
library(igraph)

data(karate, package="igraphdata")
karate <- upgrade_graph(karate)

d3heatmap(
  get.adjacency(karate,sparse=FALSE),
  colors = "Blues",
  dendrogram="none"
)

d3heatmap(
  get.adjacency(karate,sparse=FALSE,attr="weight"),
  colors = "Blues",
  dendrogram="none"
)

Cl <- cor(longley)
d3heatmap(
  Cl,
  # play with colors to remember how
  colors = col_numeric("RdBu",c(-.2,1)),
  dendrogram="none"
)

[Cat1796]

Hi @timelyportfolio, I am not very familiar with d3heatmap, but having a look at your examples, it seems to be roughly the same. Still, a plot of a bond default co-occurence (naive example below with 6 independent bonds), d3heatmap seem to display bonds that have low co-occurence close together, when what we what really is the opposite. A larger simulation with dependent bonds would lead to several clusters in the bostock co-occurence matrix.

bonds_def_prob<-c(0.01,0.02,0.001,0.023,0.024,0.025)
nbbonds<-length(bondsdefprob)
nbsim<-10000
randomness<-matrix(runif(nbbonds*nbsim,0,1),ncol=nbbonds,nrow=nbsim,byrow=TRUE)
prob_matrix<-matrix(bonds_def_prob,nrow=nbsim,ncol=length(bonds_def_prob),byrow=TRUE)
defaults<-randomness<prob_matrix
co_oc<-t(defaults) %*% defaults
diag(co_oc)<-0

In any case, the animation of bostock above is way cooler than d3heatmap (which is very cool itself). Also, the concept of moving from ordering by name (no logic), concurrence (riskiness of bond), cluster (co-dependence) makes a huge amount of sense in the financial context.

[smartinsightsfromdata]

I just found this interesting post , where shiny and ggplot2 are used to represent co-occurrence examples. Unfortunately, while it is a good static representation of the concept, it is ... "static".

Let me clarify where do I come from. Recently I've been working on using graph databases & graph visualisations to represent securities in a stock exchange. A lot of studies have been done on this, typically some years ago with relatively "old" style visualisations.
In particular the following paper caught my attention:

Heterogeneous Economic Networks" by Wataru Souma, Yoshi Fujiwara and Hideaki Aoyama, appeared in “The complex network of economic interaction” Springer 2006.

The picture below comes from the paper. I started to think how interesting it would be if there were different ways to analyse visually types of aggregation at this level, quickly comparing one to the other.
Beside industry sector one could look at different ways to order / categorise shareholding companies, changing dynamically from one to the other, to see if there is any discernible insight that comes to light.

Could I do this with ggplot2 as per the example above? Possibly, but I was looking for something that would be able to:

• quickly transition from one state to the other, like M. Bostock matrix
• manage a large number of rows / columns (perhaps thousands of them?).

I recently asked @jcheng5 if d3heatmap was suitable to build the equivalent of Mike Bostock's Les Misérables Co-occurrence and as you can see in the post Joe reply was:

It's not currently possible. I wasn't really thinking of heatmap transitions when I started out, but it makes plenty of sense I guess.

[timelyportfolio]

I just realized I never followed up with a quick prototype I built http://www.buildingwidgets.com/blog/2015/10/22/week-42-adjacency-matrix.

[vyster]

Hello, I need to make https://bost.ocks.org/mike/miserables/ this from the co-occurrence matrix I have generated., Can you help??
https://drive.google.com/file/d/1c275QvW7XjY5llhBvExSn_XWgK5w7o2R/view?usp=sharing

[vyster]

Hello, I need to make https://bost.ocks.org/mike/miserables/ this from the co-occurrence matrix I have generated., Can you help??
https://drive.google.com/file/d/1c275QvW7XjY5llhBvExSn_XWgK5w7o2R/view?usp=sharing
I need a working code... https://gist.github.com/allisonking/c9f6ea277e4194cf7e13f49efb83b5b5 is exactly same as my problem but isn't complete.

[vyster]

I've seen this very elegant example of adjacency / co-occurrence matrix designed by M. Bostock. Very impressive.

http://bost.ocks.org/mike/miserables/

I wonder if there is anything similar already achievable with any htmlwidgets?

I see quite a bit of interesting applications...

Hello, I need to make https://bost.ocks.org/mike/miserables/ this from the co-occurrence matrix I have generated., Can you help??
https://drive.google.com/file/d/1c275QvW7XjY5llhBvExSn_XWgK5w7o2R/view?usp=sharing
I need a working code... https://gist.github.com/allisonking/c9f6ea277e4194cf7e13f49efb83b5b5 is exactly same as my problem but isn't complete.