New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

R* tree fails KNNModel tests on OS X only #964

Closed
minxuancao opened this Issue Mar 26, 2017 · 14 comments

Comments

Projects
None yet
4 participants
@minxuancao

minxuancao commented Mar 26, 2017

After building mlpack, I ran bin/mlpack_test and the terminal returned 3 failures. I'm not sure whether I should ignore it. I followed the build from source directions on the mlpack website.

n242-140:build minxuan$ bin/mlpack_test
Running 759 test cases...
/Users/minxuan/mlpack-2.2.0/src/mlpack/tests/aknn_test.cpp:395: fatal error: in "AKNNTest/KNNModelTest": critical check std::abs((distancesExact[k]) - (distancesApprox[k])) <= (0.05) * std::abs(distancesExact[k]) has failed [0.059011698748245967 > 0.031223004815217881]
/Users/minxuan/mlpack-2.2.0/src/mlpack/tests/test_tools.hpp:61: fatal error: in "GMMTest/GMMTrainEMOneGaussian": difference{1.20512} between a[i]{8.337663734166346e-06} and b[i]{3.7810539028744946e-06} exceeds 1e-05%
/Users/minxuan/mlpack-2.2.0/src/mlpack/tests/knn_test.cpp:1138: fatal error: in "KNNTest/KNNModelTest": critical check neighbors[k] == baselineNeighbors[k] has failed [68 != 60]

*** 3 failures are detected in the test module "mlpackTest"

@sagarbhathwar

This comment has been minimized.

Show comment
Hide comment
@sagarbhathwar

sagarbhathwar Mar 27, 2017

Contributor

There is already an issue opened for this at #922

Contributor

sagarbhathwar commented Mar 27, 2017

There is already an issue opened for this at #922

@rcurtin

This comment has been minimized.

Show comment
Hide comment
@rcurtin

rcurtin Mar 27, 2017

Member

At least the GMMTest failure has been often seen elsewhere and that is indeed a random failure like those we are trying to fix in #922. But I have never seen the AKNNTest or KNNTest failures, and those tests (at least the KNNTest) should be deterministic and have no random component. Can you tell me more about your system to see if we can reproduce the problem?

Member

rcurtin commented Mar 27, 2017

At least the GMMTest failure has been often seen elsewhere and that is indeed a random failure like those we are trying to fix in #922. But I have never seen the AKNNTest or KNNTest failures, and those tests (at least the KNNTest) should be deterministic and have no random component. Can you tell me more about your system to see if we can reproduce the problem?

@minxuancao

This comment has been minimized.

Show comment
Hide comment
@minxuancao

minxuancao Mar 27, 2017

I am using macOS Sierra 10.12.3 and I installed mlpack 2.2.0

minxuancao commented Mar 27, 2017

I am using macOS Sierra 10.12.3 and I installed mlpack 2.2.0

@minxuancao

This comment has been minimized.

Show comment
Hide comment
@minxuancao

minxuancao Mar 27, 2017

I downloaded Armadillo and Boost from Homebrew.

minxuancao commented Mar 27, 2017

I downloaded Armadillo and Boost from Homebrew.

@rcurtin

This comment has been minimized.

Show comment
Hide comment
@rcurtin

rcurtin Mar 28, 2017

Member

I tried to reproduce on OS X Yosemite (all I have) but I did not see any of the same errors. Does anyone else have a Sierra box they could try to reproduce the AKNNTest and KNNTest failures on? I have Boost 1.59 and Armadillo 7.500.0 installed via MacPorts.

@minxuancao: this is a long shot, but are you sure that your RAM and hard drive don't have any issues?

Member

rcurtin commented Mar 28, 2017

I tried to reproduce on OS X Yosemite (all I have) but I did not see any of the same errors. Does anyone else have a Sierra box they could try to reproduce the AKNNTest and KNNTest failures on? I have Boost 1.59 and Armadillo 7.500.0 installed via MacPorts.

@minxuancao: this is a long shot, but are you sure that your RAM and hard drive don't have any issues?

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Mar 28, 2017

Member

@rcurtin I can try to reproduce the issue, @minxuancao can you tell us more about the boost and armadillo version you used?

Member

zoq commented Mar 28, 2017

@rcurtin I can try to reproduce the issue, @minxuancao can you tell us more about the boost and armadillo version you used?

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Mar 28, 2017

Member

Okay, build mlpack-2.2.0 and I can reproduce the errors.

Member

zoq commented Mar 28, 2017

Okay, build mlpack-2.2.0 and I can reproduce the errors.

@zoq

This comment has been minimized.

Show comment
Hide comment
@rcurtin

This comment has been minimized.

Show comment
Hide comment
@rcurtin

rcurtin Mar 28, 2017

Member

Can't reproduce with those datasets on either my Linux desktop or on Yosemite. I ran:

$ mlpack_knn -r referenceData.csv -q queryData.csv -k 1 -d d.csv -n n.csv
$ mlpack_knn -r referenceData.csv -q queryData.csv -k 1 -d dn.csv -n nn.csv -N
$ diff n.csv nn.csv
$ diff d.csv dn.csv

(i.e. run in dual-tree mode, then naive mode, and make sure the results are identical)

Member

rcurtin commented Mar 28, 2017

Can't reproduce with those datasets on either my Linux desktop or on Yosemite. I ran:

$ mlpack_knn -r referenceData.csv -q queryData.csv -k 1 -d d.csv -n n.csv
$ mlpack_knn -r referenceData.csv -q queryData.csv -k 1 -d dn.csv -n nn.csv -N
$ diff n.csv nn.csv
$ diff d.csv dn.csv

(i.e. run in dual-tree mode, then naive mode, and make sure the results are identical)

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Mar 28, 2017

Member

Yeah, I can't reproduce the problem with the executable, would be interesting to see if:

BOOST_AUTO_TEST_CASE(KNNModelTest)
{
  // Ensure that we can build an NSModel<NearestNeighborSearch> and get correct
  // results.
  typedef NSModel<NearestNeighborSort> KNNModel;

  arma::mat queryData;
  arma::mat referenceData;

  data::Load("queryData.csv", queryData);
  data::Load("referenceData.csv", referenceData);

  arma::mat referenceCopy(referenceData);
  arma::mat queryCopy(queryData);

  // Get a baseline.
  KNN knn(referenceData);
  arma::Mat<size_t> baselineNeighbors;
  arma::mat baselineDistances;
  knn.Search(queryData, 3, baselineNeighbors, baselineDistances);

  KNNModel model = KNNModel(KNNModel::TreeTypes::R_STAR_TREE, true);
  model.BuildModel(std::move(referenceCopy), 20, DUAL_TREE_MODE);

  arma::Mat<size_t> neighbors;
  arma::mat distances;

  model.Search(std::move(queryCopy), 3, neighbors, distances);

  for (size_t k = 0; k < distances.n_elem; ++k)
  {
    BOOST_REQUIRE_EQUAL(neighbors[k], baselineNeighbors[k]);
  }
}

works for you.

Member

zoq commented Mar 28, 2017

Yeah, I can't reproduce the problem with the executable, would be interesting to see if:

BOOST_AUTO_TEST_CASE(KNNModelTest)
{
  // Ensure that we can build an NSModel<NearestNeighborSearch> and get correct
  // results.
  typedef NSModel<NearestNeighborSort> KNNModel;

  arma::mat queryData;
  arma::mat referenceData;

  data::Load("queryData.csv", queryData);
  data::Load("referenceData.csv", referenceData);

  arma::mat referenceCopy(referenceData);
  arma::mat queryCopy(queryData);

  // Get a baseline.
  KNN knn(referenceData);
  arma::Mat<size_t> baselineNeighbors;
  arma::mat baselineDistances;
  knn.Search(queryData, 3, baselineNeighbors, baselineDistances);

  KNNModel model = KNNModel(KNNModel::TreeTypes::R_STAR_TREE, true);
  model.BuildModel(std::move(referenceCopy), 20, DUAL_TREE_MODE);

  arma::Mat<size_t> neighbors;
  arma::mat distances;

  model.Search(std::move(queryCopy), 3, neighbors, distances);

  for (size_t k = 0; k < distances.n_elem; ++k)
  {
    BOOST_REQUIRE_EQUAL(neighbors[k], baselineNeighbors[k]);
  }
}

works for you.

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Mar 28, 2017

Member

Also, in KNNModelTest I think we never test against the NAIVE_MODE since for (size_t j = 0; j < 2; ++j)and if (j == 2).

Member

zoq commented Mar 28, 2017

Also, in KNNModelTest I think we never test against the NAIVE_MODE since for (size_t j = 0; j < 2; ++j)and if (j == 2).

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Mar 28, 2017

Member

hm, but even with the test code I can't reproduce the error in every run.

Member

zoq commented Mar 28, 2017

hm, but even with the test code I can't reproduce the error in every run.

@rcurtin

This comment has been minimized.

Show comment
Hide comment
@rcurtin

rcurtin Apr 22, 2017

Member

Fascinating, I am able to reproduce it on OS X but not on a Linux system! Both are x86_64. It also looks like I exposed another bug somewhere in there:

unknown location:0: fatal error: in "KNNTest/KNNModelTest2": signal: SIGSEGV, si_code: 0 (memory access violation at address: 0x00000000)
/Users/ryan_curtin/src/mlpack-2.2.0/src/mlpack/tests/knn_test.cpp:1279: last checkpoint: "KNNModelTest2" entry.

as well as the original error of

/Users/ryan_curtin/src/mlpack-2.2.0/src/mlpack/tests/knn_test.cpp:1311: fatal error: in "KNNTest/KNNModelTest2": critical check neighbors[k] == baselineNeighbors[k] has failed [44 != 115]

This is probably a strange edge case for the R* tree that happens to only occur on OS X. When I have time I'll try and dig into what exactly the issue might be.

Member

rcurtin commented Apr 22, 2017

Fascinating, I am able to reproduce it on OS X but not on a Linux system! Both are x86_64. It also looks like I exposed another bug somewhere in there:

unknown location:0: fatal error: in "KNNTest/KNNModelTest2": signal: SIGSEGV, si_code: 0 (memory access violation at address: 0x00000000)
/Users/ryan_curtin/src/mlpack-2.2.0/src/mlpack/tests/knn_test.cpp:1279: last checkpoint: "KNNModelTest2" entry.

as well as the original error of

/Users/ryan_curtin/src/mlpack-2.2.0/src/mlpack/tests/knn_test.cpp:1311: fatal error: in "KNNTest/KNNModelTest2": critical check neighbors[k] == baselineNeighbors[k] has failed [44 != 115]

This is probably a strange edge case for the R* tree that happens to only occur on OS X. When I have time I'll try and dig into what exactly the issue might be.

@rcurtin rcurtin added this to the mlpack 2.2.2 milestone Apr 22, 2017

@rcurtin rcurtin changed the title from failures after build to R* tree fails KNNModel tests on OS X only Apr 22, 2017

@rcurtin

This comment has been minimized.

Show comment
Hide comment
@rcurtin

rcurtin May 4, 2017

Member

Hi there @minxuancao,

Thanks again for the report. This appeared to be a subtle bug with the implementation of the R* tree, but I have gone through in 1165dcc and some other commits and done a refactoring that fixes it. I can no longer produce the issue on my OS X system.

I'll release mlpack 2.2.2 with the fix and it should solve your problem.

Thanks!

Ryan

Member

rcurtin commented May 4, 2017

Hi there @minxuancao,

Thanks again for the report. This appeared to be a subtle bug with the implementation of the R* tree, but I have gone through in 1165dcc and some other commits and done a refactoring that fixes it. I can no longer produce the issue on my OS X system.

I'll release mlpack 2.2.2 with the fix and it should solve your problem.

Thanks!

Ryan

@rcurtin rcurtin closed this May 4, 2017

@rcurtin rcurtin added the R: fixed label May 4, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment