five more tests fail when pytest is run with four parallel tasks on Fedora 29 (rawhide)

[rathann]

Expected behaviour

All tests pass.

Actual behaviour

Compared to issue #36, five more tests fail when running in Fedora 29 (rawhide) build environment with parallel test execution: TestEncore.test_hes, TestEncore.test_hes_align, TestEncoreClustering.test_clustering_KMeans_direct, TestEncoreClustering.test_clustering_method_w_no_distance_matrix and TestEncoreClustering.test_clustering_two_methods_one_w_no_distance_matrix.

Code to reproduce the behaviour

$ PYTHONPATH=/builddir/build/BUILDROOT/python-MDAnalysis-0.18.0-1.fc29.x86_64/usr/lib64/python2.7/site-packages:/builddir/build/BUILDROOT/python-MDAnalysis-0.18.0-1.fc29.x86_64/usr/lib/python2.7/site-packages pytest-2 --numprocesses=auto analysis/test_encore.py
================================================= test session starts ==================================================
platform linux2 -- Python 2.7.15, pytest-3.6.2, py-1.5.4, pluggy-0.6.0
rootdir: /builddir/build/BUILD/MDAnalysis-0.18.0/MDAnalysisTests-0.18.0, inifile: setup.cfg
plugins: xdist-1.22.2, forked-0.2, hypothesis-3.49.0
gw0 [46] / gw1 [46] / gw2 [46] / gw3 [46]
scheduling tests via LoadScheduling
........F.F.F.........F......F................                                                                   [100%]
======================================================= FAILURES =======================================================
_____________________________________________ TestEncore.test_hes_to_self ______________________________________________
[gw2] linux2 -- Python 2.7.15 /usr/bin/python2

self = <MDAnalysisTests.analysis.test_encore.TestEncore object at 0x7f6b71e13690>, ens1 = <Universe with 3341 atoms>

    def test_hes_to_self(self, ens1):
        results, details = encore.hes([ens1, ens1])
        result_value = results[0, 1]
        expected_value = 0.
        assert_almost_equal(result_value, expected_value,
>                           err_msg="Harmonic Ensemble Similarity to itself not zero: {0:f}".format(result_value))
E       AssertionError: 
E       Arrays are not almost equal to 7 decimals
E       Harmonic Ensemble Similarity to itself not zero: -81.057878
E        ACTUAL: -81.05787844289566
E        DESIRED: 0.0

analysis/test_encore.py:232: AssertionError
______________________________________________ TestEncore.test_hes_align _______________________________________________
[gw0] linux2 -- Python 2.7.15 /usr/bin/python2

self = <MDAnalysisTests.analysis.test_encore.TestEncore object at 0x7fac0911e3d0>, ens1 = <Universe with 3341 atoms>
ens2 = <Universe with 3341 atoms>

    def test_hes_align(self, ens1, ens2):
        # This test is massively sensitive!
        # Get 5260 when masses were float32?
        results, details = encore.hes([ens1, ens2], align=True)
        result_value = results[0,1]
        expected_value = 2047.05
        assert_almost_equal(result_value, expected_value, decimal=-3,
>                           err_msg="Unexpected value for Harmonic Ensemble Similarity: {0:f}. Expected {1:f}.".format(result_value, expected_value))
E       AssertionError: 
E       Arrays are not almost equal to -3 decimals
E       Unexpected value for Harmonic Ensemble Similarity: -4364.809461. Expected 2047.050000.
E        ACTUAL: -4364.809461134429
E        DESIRED: 2047.05

analysis/test_encore.py:257: AssertionError
__________________________________________ TestEncore.test_hes_custom_weights __________________________________________
[gw3] linux2 -- Python 2.7.15 /usr/bin/python2

self = <MDAnalysisTests.analysis.test_encore.TestEncore object at 0x7f55f8b18e10>, ens1 = <Universe with 3341 atoms>
ens2 = <Universe with 3341 atoms>

    def test_hes_custom_weights(self, ens1, ens2):
        results, details = encore.hes([ens1, ens2], weights='mass')
        results_custom, details_custom = encore.hes([ens1, ens2],
                                                    weights=(ens1.select_atoms('name CA').masses,
                                                             ens2.select_atoms('name CA').masses))
        result_value = results[0, 1]
        result_value_custom = results_custom[0, 1]
>       assert_almost_equal(result_value, result_value_custom)
E       AssertionError: 
E       Arrays are not almost equal to 7 decimals
E        ACTUAL: -303572628.6677543
E        DESIRED: 4812781047.929773

analysis/test_encore.py:248: AssertionError
__________________________________ TestEncoreClustering.test_clustering_KMeans_direct __________________________________
[gw2] linux2 -- Python 2.7.15 /usr/bin/python2

self = <MDAnalysisTests.analysis.test_encore.TestEncoreClustering object at 0x7f6b6c3aafd0>
ens1 = <Universe with 3341 atoms>

    def test_clustering_KMeans_direct(self, ens1):
        pytest.importorskip('sklearn')
        clusters = 10
        method = encore.KMeans(clusters)
        coordinates = ens1.trajectory.timeseries(order='fac')
        coordinates = np.reshape(coordinates,
                                 (coordinates.shape[0], -1))
        cluster_assignment, details = method(coordinates)
>       assert len(set(cluster_assignment)) == clusters, \
                     "Unexpected result: {0}".format(cluster_assignment)
E       AssertionError: Unexpected result: [ 0  1  3  3  3  1  1  7  7 10 10 12 12 13 15 15 15 18 18 18]
E       assert 9 == 10
E        +  where 9 = len(set([0, 1, 3, 7, 10, 12, ...]))
E        +    where set([0, 1, 3, 7, 10, 12, ...]) = set(array([ 0,  1,  3,  3,  3,  1,  1,  7,  7, 10, 10, 12, 12, 13, 15, 15, 15,\n       18, 18, 18], dtype=int32))

analysis/test_encore.py:460: AssertionError
______________________ TestEncoreClustering.test_clustering_two_methods_one_w_no_distance_matrix _______________________
[gw1] linux2 -- Python 2.7.15 /usr/bin/python2

self = <MDAnalysisTests.analysis.test_encore.TestEncoreClustering object at 0x7f9f222d2fd0>
ens1 = <Universe with 3341 atoms>

    def test_clustering_two_methods_one_w_no_distance_matrix(self, ens1):
        pytest.importorskip('sklearn')
        cluster_collection = encore.cluster(
            [ens1],
            method=[encore.KMeans(17),
>                   encore.AffinityPropagationNative()])

analysis/test_encore.py:494: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../BUILDROOT/python-MDAnalysis-0.18.0-1.fc29.x86_64/usr/lib64/python2.7/site-packages/MDAnalysis/analysis/encore/clustering/cluster.py:234: in cluster
    metadata=metadata) for clusters in results]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = 
elements = array([ 0,  1,  2,  3,  4, 16,  6,  8,  8,  9, 10, 11,  4, 13, 14, 14,  0,
       18, 18, 19], dtype=int32)
metadata = {'ensemble_membership': array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])}

    def __init__(self, elements=None, metadata=None):
        """Class constructor. If elements is None, an empty cluster collection
                will be created. Otherwise, the constructor takes as input an
                iterable of ints, for instance:
    
                [ a, a, a, a, b, b, b, c, c, ... , z, z ]
    
                the variables a,b,c,...,z are cluster centroids, here as cluster
                element numbers (i.e. 3 means the 4th element of the ordered input
                for clustering). The array maps a correspondence between
                cluster elements (which are implicitly associated with the
                position in the array) with centroids, i. e. defines clusters.
                For instance:
    
                [ 1, 1, 1, 4, 4, 5 ]
    
                means that elements 0, 1, 2 form a cluster which has 1 as centroid,
                elements 3 and 4 form a cluster which has 4 as centroid, and
                element 5 has its own cluster.
    
    
                Parameters
                ----------
    
                elements : iterable of ints or None
                    clustering results. See the previous description for details
    
                metadata : {str:list, str:list,...} or None
                    metadata for the data elements. The list must be of the same
                    size as the elements array, with one value per element.
    
            """
        idn = 0
        if elements is None:
            self.clusters = None
            return
    
        if not len(set((type(el) for el in elements))) == 1:
            raise TypeError("all the elements must have the same type")
        self.clusters = []
        elements_array = np.array(elements)
        centroids = np.unique(elements_array)
        for i in centroids:
            if elements[i] != i:
                raise ValueError("element {0}, which is a centroid, doesn't \
>   belong to its own cluster".format(elements[i]))
E               ValueError: element 0, which is a centroid, doesn't belong to its own cluster

../../../../BUILDROOT/python-MDAnalysis-0.18.0-1.fc29.x86_64/usr/lib64/python2.7/site-packages/MDAnalysis/analysis/encore/clustering/ClusterCollection.py:194: ValueError
========================================= 5 failed, 41 passed in 22.07 seconds =========================================

Currently version of MDAnalysis:

0.18.0

[rathann]

One more:

___________________________ TestEncoreClustering.test_clustering_method_w_no_distance_matrix ___________________________
[gw0] linux2 -- Python 2.7.15 /usr/bin/python2

self = <MDAnalysisTests.analysis.test_encore.TestEncoreClustering object at 0x7fdf7dd7df10>
ens1 = <Universe with 3341 atoms>

    def test_clustering_method_w_no_distance_matrix(self, ens1):
        pytest.importorskip('sklearn')
        cluster_collection = encore.cluster(
            [ens1],
            method=encore.KMeans(10))
>       assert len(cluster_collection) == 10, \
                     "Unexpected result: {0}".format(cluster_collection)
E       AssertionError: Unexpected result: 0 (size:1,centroid:0): array([0])
E         1 (size:1,centroid:1): array([1])
E         2 (size:2,centroid:3): array([2, 3])
E         3 (size:3,centroid:4): array([4, 5, 6])
E         4 (size:4,centroid:7): array([ 7,  8, 11, 12])
E         5 (size:2,centroid:10): array([ 9, 10])
E         6 (size:2,centroid:13): array([13, 14])
E         7 (size:3,centroid:16): array([15, 16, 17])
E         8 (size:2,centroid:18): array([18, 19])
E         
E       assert 9 == 10
E        +  where 9 = len(0 (size:1,centroid:0): array([0])\n1 (size:1,centroid:1): array([1])\n2 (size:2,...ze:3,centroid:16): array([15, 16, 17])\n8 (size:2,centroid:18): array([18, 19])\n)

analysis/test_encore.py:453: AssertionError

[rathann]

And one more:

_________________________________________________ TestEncore.test_hes __________________________________________________
[gw1] linux2 -- Python 2.7.15 /usr/bin/python2

self = <MDAnalysisTests.analysis.test_encore.TestEncore object at 0x7fbf0b9b28d0>, ens1 = <Universe with 3341 atoms>
ens2 = <Universe with 3341 atoms>

    def test_hes(self, ens1, ens2):
        results, details = encore.hes([ens1, ens2], weights='mass')
        result_value = results[0, 1]
        min_bound = 1E5
>       assert result_value > min_bound, "Unexpected value for Harmonic " \
                                          "Ensemble Similarity: {0:f}. Expected {1:f}.".format(result_value, min_bound)
E       AssertionError: Unexpected value for Harmonic Ensemble Similarity: -13909450.252759. Expected 100000.000000.
E       assert -13909450.25275872 > 100000.0

analysis/test_encore.py:232: AssertionError

Sorry for the breadcrumbs style, but I got additional failures after disabling the previously failing ones and re-running the testsuite. Something is wrong when running tests in parallel.

[kain88-de]

Those tests are also fragile for us. One problem is that we can't reliable seed the RNG before these tests and they are sensitive to the chosen random numbers (because the test ensemble is very small)

[orbeckst]

These tests have been annoying us for a while.

Issue MDAnalysis/mdanalysis#1933 thinks that we could solve it with a decent RNG interface.

PR MDAnalysis/mdanalysis#1932 was a band-aid for some of these failures – maybe adding more band-aids as an interim "solution"?

[mtiberti]

Hi,
I'm not sure all of these are RNG-related (which is something I agree requires attention more in general)- especially seeing that some are in HES; do these fails happen only when tests are run in parallel? Does Travis use parallel tasks (I would think so as I see --numprocesses 2 in the command line)? I can really look into it starting around half of next week when my schedule will eventually free up a bit

[rathann]

@mtiberti Yes, the failures occur only when running in parallel. My machine has 4 cores, so I run the tests with --numprocesses 4.

[mtiberti]

can anyone confirm if this is still the case on a more recent Fedora version? Otherwise we can just close it