[FLINK-3768] [gelly] Clustering Coefficient

[greghogan]

Provides an algorithm for local clustering coefficient and dependent functions for degree annotation, algorithm caching, and graph translation.

I worked to improve the performance of TriangleEnumerator. Perhaps the API has changed since Edge.reverse() is not in-place and the edges were not being sorted by degree. The JoinHint is also important so that the Triads are not spilled to disk.

On an AWS ec2.4xlarge (16 vcores, 30 GiB) I am seeing for the following timings of 5s, 29s, and 183s for TriangleListing. With TriangleEnumerator the timings are 7s, 45s, and 281s. Without the JoinHint the latter TriangleEnumerator timings are 58s and 347s.

Scale	ChecksumHashCode	Count
16	0x0000d9086985f4ce	15616010
18	0x0010eeb32a441365	82781436
20	0x014a9434bb57ddef	423780284

The command I had used to run the tests:

./bin/flink run -class org.apache.flink.graph.examples.TriangleListing ~/flink-gelly-examples_2.10-1.1-SNAPSHOT.jar --clip_and_flip false --output print --output hash --scale 16 --listing

[vasia]

Hi @greghogan,
thank you for the PR. This is a big addition! Are all the changes related to the clustering coefficient JIRA? Could it maybe be split into smaller PRs in order to make reviewing easier?
For big changes like this, it usually a good idea to create a design document to show the high-level approach and functionality to be added and discuss it with the community. Do you think it would make sense to do this?
Thank you!

[greghogan]

It's certainly nicer to review small PRs, but I also like to present the big picture and context in which the features will be used. What if I leave this here, break out the dependencies into new PRs, and then rebase this PR down if those features are accepted? I'm not quite sure what a design document would cover. There's not much sophisticated here as with SG or GSA.

[hsaputra]

Could add JavaDoc to ALL new Java or Scala classes added to source? Explanation why and how it is interact with other code is something we are looking for. Thanks.

[vasia]

Ideally, each PR should make a single change and reference a single JIRA issue. If you could break this into smaller changes, that'd be great. Otherwise, it would really help if you could give a more detailed description of changes and additions. Thanks!

[greghogan]

I have rebased this PR against the newly committed dependent features so it should be good for discussion.

[hsaputra]

Please address my comments about adding Javadoc header info to ALL new Java class added to source repo. I understand some of the classes are just for examples, but would be good to also have Javdoc explaining why the examples are added.
Thx much!

[hsaputra]

I know this is just an example, but would be nice to add Javadoc information to give high level information why this class exists.

[greghogan]

Done.

[hsaputra]

Thanks for the docs update.

[greghogan]

If there are no further review comments I'll look at merging this today.

[vasia]

Hi @greghogan,
I haven't had time to review this PR. If this is blocking you or is urgent for some reason, please go ahead. Otherwise, I could take a look later this week.