QuickBundlesX #1434

Garyfallidis · 2018-02-16T02:34:20Z

This PR is replacing #1380
This PR includes the fastest algorithm ever for clustering tractograms!!! 🔥

Garyfallidis E. et al. QuickBundlesX: Sequential clustering of millions of streamlines in multiple levels of detail at record execution time. Proceedings of the, International Society of Magnetic Resonance in Medicine (ISMRM). Singapore, 4187, 2016.

MarcCote

In theory, QuickBundles(threshold) and QuickBundlesX(thresholds=[threshold]) should produce the same kind of results. Maybe we could refactor QuickBundles such that it uses the same Cython algorithm/code as QuickBundlesX (or maybe in a future PR). In any cases, we add some simple tests that check if the results are similar between QuickBundles(threshold) and QuickBundlesX(thresholds=[threshold]).

MarcCote · 2018-02-18T03:25:54Z

dipy/segment/clustering.py

+                            thresholds=self.thresholds,
+                            ordering=ordering)
+
+        #cluster_map.refdata = streamlines


Remove comment.

MarcCote · 2018-02-18T03:36:17Z

dipy/segment/clusteringspeed.pyx

+DEF BIGGEST_FLOAT = 3.4028235e+38  # np.finfo('f4').max
+DEF SMALLEST_FLOAT = -3.4028235e+38  # np.finfo('f4').max
+
+THRESHOLD_MULTIPLIER = 2.


MarcCote · 2018-02-18T03:39:23Z

dipy/segment/clusteringspeed.pyx

@@ -99,7 +402,7 @@ cdef class ClustersCentroid(Clusters):
            raise ValueError("'centroid_shape' must be a tuple or a int.")

        self._centroid_shape = tuple2shape(centroid_shape)
-
+        # self.aabb


Remove this line.

MarcCote · 2018-02-18T03:39:30Z

dipy/segment/clusteringspeed.pyx

@@ -175,6 +478,10 @@ cdef class ClustersCentroid(Clusters):
                converged &= fabs(centroid[n, d] - updated_centroid[n, d]) < self.eps
                centroid[n, d] = updated_centroid[n, d]

+        #cdef float * aabb = &self.centroids[id_cluster].aabb[0]


Remove this line.

MarcCote · 2018-02-18T03:47:30Z

dipy/segment/clusteringspeed.pyx

        self.metric = metric
        self.features_shape = tuple2shape(features_shape)
        self.threshold = threshold
        self.max_nb_clusters = max_nb_clusters
        self.clusters = ClustersCentroid(features_shape)
        self.features = np.empty(features_shape, dtype=DTYPE)
        self.features_flip = np.empty(features_shape, dtype=DTYPE)
+        self.bvh = bvh


Nowhere in the code, I could find Quickbundles instance with bvh=1. Maybe we should remove this functionality for now? Otherwise, we could do like for QuickbundlesX and always perform aabb checks?

MarcCote · 2018-02-18T03:49:11Z

dipy/segment/clusteringspeed.pyx

+        return self._build_clustermap()
+
+
+def evaluate_aabbb_checks():


I couldn't find a place where this function is used.

MarcCote · 2018-02-18T03:51:49Z

dipy/segment/tests/test_qbx.py

+    #test_show_qbx()
+    #test_3D_segments()
+    test_3D_points()
+


Newline missing?

MarcCote · 2018-02-18T03:53:34Z

dipy/segment/tests/test_qbx.py

+    thresholds = [4, 2, 1]
+    qbx_class = QuickBundlesX(thresholds)
+    qbx = qbx_class.cluster(points)
+    print(qbx)


What do we expect? Add assertions.

MarcCote · 2018-02-18T03:53:40Z

dipy/segment/tests/test_qbx.py

+    thresholds = [4, 2, 1]
+    qbx_class = QuickBundlesX(thresholds)
+    qbx = qbx_class.cluster(points)
+    print(qbx)


What do we expect? Add assertions.

codecov-io · 2018-02-18T17:13:36Z

Codecov Report

Merging #1434 into master will increase coverage by 0.02%.
The diff coverage is 91.54%.

@@            Coverage Diff             @@
##           master    #1434      +/-   ##
==========================================
+ Coverage   87.38%   87.41%   +0.02%     
==========================================
  Files         237      238       +1     
  Lines       30069    30282     +213     
  Branches     3232     3253      +21     
==========================================
+ Hits        26277    26472     +195     
- Misses       3043     3057      +14     
- Partials      749      753       +4

Impacted Files	Coverage Δ
dipy/segment/tests/test_quickbundles.py	`98.31% <100%> (+0.36%)`	⬆️
dipy/segment/clustering.py	`93.51% <79.41%> (-6.49%)`	⬇️
dipy/segment/tests/test_qbx.py	`96.77% <96.77%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 95a9cca...f81a4da. Read the comment docs.

Garyfallidis · 2018-02-18T22:55:20Z

Hey @MarcCote I agree on the bvh. Needs some investigation. Any clues on the travis error? Strangely only one bot is failing https://travis-ci.org/nipy/dipy/jobs/343108569

skoudoro · 2018-02-19T00:21:48Z

The last bot is failing because of the new version of Cython. Look at the issue #1435

Garyfallidis · 2018-02-20T15:11:38Z

@MarcCote there is a change in Cython that will require some large RF of the code. Cython stopped supporting memory views in structs.

Rb qbx2

MarcCote · 2018-03-01T13:13:13Z

dipy/segment/cythonutils.pyx

@@ -104,3 +111,46 @@ cdef int same_shape(Shape shape1, Shape shape2) nogil:
        same_shape &= shape1.dims[i] == shape2.dims[i]

    return same_shape
+
+
+cdef Data2D* create_memview(Py_ssize_t buffer_size, Py_ssize_t dims[MAX_NDIM]) nogil:


Do we only want to support 2D memview? If so, we could rename the function so that it is clearer it is only for 2D array.

ok, you are right, let's do it more generic

Finally, I choose the easy path: renaming. The second option needs more thinking and we need to move forward with the release

MarcCote · 2018-03-01T13:13:22Z

dipy/segment/cythonutils.pyx

+    """
+    free(&(memview[0][0, 0]))
+    memview[0] = None  # Necessary to decrease refcount
+    free(memview)


Newline missing.

Rb qbx2 2

Garyfallidis · 2018-03-01T19:57:57Z

@MarcCote I addressed all your comments. If Travis agrees do merge this PR so that the other PRs can can start becoming green.

Reducing code complexity

skoudoro · 2018-03-03T01:34:02Z

Yeah! Looks good and all tests are happy. I think it is ready to merge, can you have a last look @MarcCote? Thanks to you and @Garyfallidis!

MarcCote

After that, LGTM.

MarcCote · 2018-03-03T01:49:11Z

dipy/segment/tests/test_qbx.py

+def test_circle_parallel_fornix():
+
+    circle = streamlines_in_circle(100, step_size=2)
+


PEP8 - too many empty lines.

MarcCote · 2018-03-03T01:50:25Z

dipy/segment/clusteringspeed.pyx

-        self.level = levels[level]
-        self.traverse_postorder(self.root, self._fetch_level)
-        return self.clusters
+    # cdef void _fetch_level(self, CentroidNode* node):


let's delete those commented lines if everyone is happy with manipulating the TreeClusterMap python object instead of the QuickBundlesX Cython object.

MarcCote · 2018-03-03T01:50:46Z

dipy/segment/clusteringspeed.pxd

@@ -110,5 +110,5 @@ cdef class QuickBundlesX(object):
    cpdef object insert(self, Data2D datum, int datum_idx)
    cdef void traverse_postorder(self, CentroidNode* node, void (*visit)(QuickBundlesX, CentroidNode*))
    cdef void _dealloc_node(self, CentroidNode* node)
-    cdef void _fetch_level(self, CentroidNode* node)
+    # cdef void _fetch_level(self, CentroidNode* node)


Garyfallidis · 2018-03-03T02:27:59Z

Done @MarcCote !

MarcCote · 2018-03-03T04:00:34Z

Let's make all the PRs green again!

Garyfallidis · 2018-03-03T05:26:52Z

Super! @skoudoro please inform everyone to rebase their PRs. We can go green again :) Thank you both for all the help. That was a fast recovery with a sudden Cythonic change! Grrr.....

QuickBundlesX

Garyfallidis added 9 commits February 15, 2018 14:04

NF: Added QBX, QBO, QBXO

031634e

TEST: added test module for QuickBundlesX

89089a6

NF: added new speed versions of QBX, QBO, QBXO

626f31f

Added TreeCluster and TreeClusterMap

2dcb6b6

Updated clustering_algorithms.pyx

fc21fb0

Added Cythonized class of QBX in clusterringspeed.pyx

70fa188

Added missing definitions

c164d76

Temporary change in test_qbx.py

b34a66c

TEST: Updating tests to be py2/3 compatible

7723025

Garyfallidis mentioned this pull request Feb 16, 2018

WIP: QuickBundlesX #1380

Closed

Garyfallidis requested a review from MarcCote February 16, 2018 14:30

Garyfallidis added 3 commits February 16, 2018 11:42

Removed Online versions and simplified tests

c9a9792

Pep8 and refs

6667ba6

DOC: adding explanation on the QBX output

0853e66

MarcCote requested changes Feb 18, 2018

View reviewed changes

Garyfallidis added 4 commits February 18, 2018 00:10

TEST: started improving tests

d2897c3

TEST: test points ready

cc3df22

TEST: updating QBX tests

8a57ccd

TEST: updated test_3D_segments

db1b47f

Garyfallidis added 2 commits February 18, 2018 13:25

TEST: improved test of simple simulated bundle of 3 streamlines

5f412d2

TEST: Removed viz parts in test_qbx

9cab395

skoudoro and others added 5 commits February 28, 2018 21:41

add function create/free memview

55b68b6

manage Data2D to make cython 0.28 happy

50084a5

update doc

5fd00e0

update doc

ef83255

Merge pull request #51 from skoudoro/rb_qbx2

7ab7efb

Rb qbx2

MarcCote reviewed Mar 1, 2018

View reviewed changes

MarcCote mentioned this pull request Mar 1, 2018

BF: Avoid using memview in struct (Cython 0.28) #1439

Closed

skoudoro and others added 8 commits March 1, 2018 17:18

correction from @MarcCote : update benchmark + pep8 correction

49f9afc

replace xrange by range

5a9e113

from @MarcCote : adding a test to quickbundles

0577aff

from @MarcCote : update metric to the refactoring

222c54a

pep8 correction

e778ef8

renaming functions free/create_memview to free/create_memview_2d

53940ab

Merge pull request #52 from skoudoro/rb_qbx2

2061bc3

Rb qbx2 2

Corrected Marco's comments and removed bvh from QuickBundles

4e3667f

MarcCote and others added 4 commits March 1, 2018 22:25

Reducing code complexity

0e888bb

Merge pull request #53 from MarcCote/rb_qbx2

c081f2e

Reducing code complexity

Use Py2/3 compatible xrange and commented fetch_level declaration

d9bfbfb

Increased coverage in QBX

12cb3b7

MarcCote approved these changes Mar 3, 2018

View reviewed changes

More comments addressed

f81a4da

MarcCote merged commit 5a8ed85 into dipy:master Mar 3, 2018

skoudoro mentioned this pull request Mar 3, 2018

compilation failed with the new cython version (0.28) #1435

Closed

ShreyasFadnavis pushed a commit to ShreyasFadnavis/dipy that referenced this pull request Sep 20, 2018

Merge pull request dipy#1434 from Garyfallidis/rb_qbx2

08cec0a

QuickBundlesX

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

QuickBundlesX #1434

QuickBundlesX #1434

Garyfallidis commented Feb 16, 2018 •

edited

Loading

MarcCote left a comment

MarcCote Feb 18, 2018

MarcCote Feb 18, 2018

MarcCote Feb 18, 2018

MarcCote Feb 18, 2018

MarcCote Feb 18, 2018

MarcCote Feb 18, 2018

MarcCote Feb 18, 2018

MarcCote Feb 18, 2018

MarcCote Feb 18, 2018

codecov-io commented Feb 18, 2018 •

edited

Loading

Garyfallidis commented Feb 18, 2018

skoudoro commented Feb 19, 2018

Garyfallidis commented Feb 20, 2018

MarcCote Mar 1, 2018

skoudoro Mar 1, 2018

skoudoro Mar 1, 2018

MarcCote Mar 1, 2018

skoudoro Mar 1, 2018

Garyfallidis commented Mar 1, 2018

skoudoro commented Mar 3, 2018

MarcCote left a comment

MarcCote Mar 3, 2018

MarcCote Mar 3, 2018

MarcCote Mar 3, 2018

Garyfallidis commented Mar 3, 2018

MarcCote commented Mar 3, 2018

Garyfallidis commented Mar 3, 2018

		def test_circle_parallel_fornix():

		circle = streamlines_in_circle(100, step_size=2)

QuickBundlesX #1434

QuickBundlesX #1434

Conversation

Garyfallidis commented Feb 16, 2018 • edited Loading

MarcCote left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Feb 18, 2018 • edited Loading

Codecov Report

Garyfallidis commented Feb 18, 2018

skoudoro commented Feb 19, 2018

Garyfallidis commented Feb 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Garyfallidis commented Mar 1, 2018

skoudoro commented Mar 3, 2018

MarcCote left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Garyfallidis commented Mar 3, 2018

MarcCote commented Mar 3, 2018

Garyfallidis commented Mar 3, 2018

Garyfallidis commented Feb 16, 2018 •

edited

Loading

codecov-io commented Feb 18, 2018 •

edited

Loading