[FEATURE] Add hierarchical clustering

[eldariont]

This PR adds a second clustering method - agglomerative hierarchical clustering - and closes #54.

The idea

Agglomerative hierarchical clustering uses a "bottom-up" approach: Each junction starts in its own cluster. In every iteration, the two clusters with the lowest distance are merged. This process yields a dendogram (see the example here) that can be cut at a desired distance.

To break the problem of clustering thousands of junctions into multiple smaller problems, we first generate smaller partitions of junctions in close proximity (see partition_junctions() in src/modules/clustering/hierarchical_clustering_method.cpp).

Then, we cluster each partition separately (see hierarchical_clustering_method() in src/modules/clustering/hierarchical_clustering_method.cpp).

Implementation

We reuse an existing implementation of the hierarchical clustering algorithm from this repo which we add to the project as a submodule.

Review

It is probably a good idea to review this PR on a commit-by-commit basis because this makes it easier to follow the changes.

EDIT (irallia, 13.04.2021): As this is the last PR of the epic, it resolves #32 aswell.

[SGSSGene]

Changes look really great. 👍

There are two hard errors:

1. std::string const & = "blub" creates a reference onto a temporary reference...which makes any acces invalid (I believe, not sure why it didn't make any trouble).
2. std::numeric_limits<int>::infinity() should be replaced by ::max().
3. Cool c++ kids don't use new/delete 😂
   The rest of my comments can definitely be ignored if you don't like 😃

@file Ending: As far as I understand every line of a text file has to be finished by an '\n', otherwise it is technically not a text file and some programs will fail (I believe it is not even a valid c++ program, but no compiler complains). This actually should be cared for by your text editor automatically. Not sure what went wrong.

[eldariont]

Hi @SGSSGene, thanks for the very helpful review. I now incorporated most of your suggestions - with the exception of the "uncool" new/delete 😄

The hierarchical clustering functions that I use (as a dependency) expect arrays/pointers as parameters.

From lib/hclust/fastcluster.h:
int hclust_fast(int n, double* distmat, int method, int* merge, double* height);

The size of the arrays is computed during runtime based on the number of elements in each partition. Currently, the space for the arrays is dynamically allocated using new. Do you see an option that would get rid of the new/delete? The std::array constructor expects a constexpr as size and std::vector does not translate into a pointer AFAIK. So which other options are there?

[SGSSGene]

Hi @SGSSGene, thanks for the very helpful review. I now incorporated most of your suggestions - with the exception of the "uncool" new/delete smile

The hierarchical clustering functions that I use (as a dependency) expect arrays/pointers as parameters.

From lib/hclust/fastcluster.h:
int hclust_fast(int n, double* distmat, int method, int* merge, double* height);

The size of the arrays is computed during runtime based on the number of elements in each partition. Currently, the space for the arrays is dynamically allocated using new. Do you see an option that would get rid of the new/delete? The std::array constructor expects a constexpr as size and std::vector does not translate into a pointer AFAIK. So which other options are there?

Ah yes, I see. Checkout std::vector::data() https://en.cppreference.com/w/cpp/container/vector/data

You can use it as follows:

std::vector<int> myVector(10); // initializes the vector size 10
assert(myVector.size() == 10);
int* ptr = myVector.data();

At the end std::vector of course also calls new/delete, but it is much safer, since it handles many cases for exceptions and early returns. The general concept is called RAII and can be extended from memory to many other resources https://en.cppreference.com/w/cpp/language/raii

[eldariont]

Ah yes, I see. Checkout std::vector::data() https://en.cppreference.com/w/cpp/container/vector/data

You can use it as follows:

std::vector<int> myVector(10); // initializes the vector size 10
assert(myVector.size() == 10);
int* ptr = myVector.data();

At the end std::vector of course also calls new/delete, but it is much safer, since it handles many cases for exceptions and early returns. The general concept is called RAII and can be extended from memory to many other resources https://en.cppreference.com/w/cpp/language/raii

Great, I replaced new/delete with nice and safe std::vectors now. Is the assert(v.size() == n) required after I default-initialized the vector with std::vector<T> v(n)?

[SGSSGene]

Ah yes, I see. Checkout std::vector::data() https://en.cppreference.com/w/cpp/container/vector/data
You can use it as follows:

std::vector<int> myVector(10); // initializes the vector size 10
assert(myVector.size() == 10);
int* ptr = myVector.data();

At the end std::vector of course also calls new/delete, but it is much safer, since it handles many cases for exceptions and early returns. The general concept is called RAII and can be extended from memory to many other resources https://en.cppreference.com/w/cpp/language/raii

Great, I replaced new/delete with nice and safe std::vectors now. Is the assert(v.size() == n) required after I default-initialized the vector with std::vector<T> v(n)?

The assert(v.size() == n) is not required. I just put it there to "prove" that it is guaranteed that v.size() is always equal to n.

[SGSSGene]

Could you rebase this branch (or merge the origin/master into it)? there seems to some conflict in src/variant_detection/variant_detection.cpp

[eldariont]

Could you rebase this branch (or merge the origin/master into it)? there seems to some conflict in src/variant_detection/variant_detection.cpp

Hi @SGSSGene, I rebased the branch so it's ready for another review :)

[Irallia]

Thank you for this work!
I have started to look at some of the details of the changes. I will do a full review after the approved first review.
But thank you very much, it looks great! 🎉

[Irallia]

Because you said you weren't sure how to do this, I looked in the code of our other seqan tools and found this (seqan/mars):
https://github.com/seqan/mars/blob/master/CMakeLists.txt
Do you think it makes sense to extend our code in this way?

[Irallia]

no?

[eldariont]

Yes, it's much better to get rid of the git submodule because we just want to use the library instead of accessing the source code. I just pushed a commit that replaces the submodule with CMake's FetchContent to get the hierarchical clustering dependency. For FetchContent I needed to increase the CMake version to 3.11.

[Irallia]

still open?

[eldariont]

Hi @SGSSGene, I rebased the PR another time. As soon as you are satisfied with the changes, @Irallia can do the second review.

[SGSSGene]

I think your call against std::tie is a good call. It would harm performance.
I see that there are still a few for loops that could use more tidy up, but I think
it is not so important, it doesn't harm readability and too much premature optimizations might lead to waste of development time. So from my sight it gets a green light

[Irallia]

Wow, that was a lot to read and process. 😮
First of all, thank you! Thanks also for the detailed tests on the new method! ❤️
I have a few documentation suggestions and otherwise really only style issues. 💅
Content wise it looks great! I hope I understood everything 😄

[Irallia]

no?

[Irallia]

Thanks for the changes! LGFM now, just two misspellings.
I'll commit These last changes by myself or do you want to cleanup the history anyway?