bin/allknn failing for some data sets #185

Closed
rcurtin opened this Issue Dec 29, 2014 · 6 comments

Projects

None yet

1 participant

@rcurtin
Member
rcurtin commented Dec 29, 2014

Reported by pram on 6 Jul 42072109 23:57 UTC
The binary is compiled in debug mode on no.cc. I am using the following files:
-r /net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensItems_50.csv (n = 3706, d -= 51)
-q /net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensUsers_50.csv (n = 6040, d = 51)

These are the commands I used and the following output:

  1. Dual-tree: ./bin/allknn -r
    /net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensItems_50.csv -q /net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensUsers_50.csv -k 1 -d /net/hg200/pram/wrkspc_data/maxip/movielens_nn_dist.csv -n /net/hg200/pram/wrkspc_data/maxip/movielens_nn.csv -v

Output:

[Compiled with debugging symbols.
[INFO ](DEBUG]) Loading '/net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensItems_5
0.csv' as CSV data.
[INFO ] Loaded reference data from '/net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensItems_50.csv'.
[INFO ] Building reference tree...
[INFO ] Loading '/net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensUsers_50.csv' as CSV data.
[INFO ] Loaded query data from '/net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensUsers_50.csv'.
[INFO ] Building query tree...
[INFO ] Tree built.
[INFO ] Computing 1 nearest neighbors...
[[bt](DEBUG]): (1) ./bin/allknn : mlpack::bound::HRectBound<2>::MinDistance(mlpack::bound::HRectBound<2> const&) const+0x36 [[bt](0x42497c]
[DEBUG]): (2) ./bin/allknn : double mlpack::neighbor::NearestNeighborSort::BestNodeToNodeDistance<mlpack::tree::BinarySpaceTree<mlpack::bound::HRectBound<2>, mlpack::neighbor::QueryStat<mlpack::neighbor::NearestNeighborSort>, arma::Mat<double> > >(mlpack::tree::BinarySpaceTree<mlpack::bound::HRectBound<2>, mlpack::neighbor::QueryStat<mlpack::neighbor::NearestNeighborSort>, arma::Mat<double> > const*, mlpack::tree::BinarySpaceTree<mlpack::bound::HRectBound<2>, mlpack::neighbor::QueryStat<mlpack::neighbor::NearestNeighborSort>, arma::Mat<double> > const*)+0x36 [[bt](0x424aee]
[DEBUG]): (3) ./bin/allknn : mlpack::neighbor::NeighborSearch<mlpack::neighbor::NearestNeighborSort, mlpack::metric::LMetric<2, false>, mlpack::tree::BinarySpaceTree<mlpack::bound::HRectBound<2>, mlpack::neighbor::QueryStat<mlpack::neighbor::NearestNeighborSort>, arma::Mat<double> > >::ComputeDualNeighborsRecursion(mlpack::tree::BinarySpaceTree<mlpack::bound::HRectBound<2>, mlpack::neighbor::QueryStat<mlpack::neighbor::NearestNeighborSort>, arma::Mat<double> >*, mlpack::tree::BinarySpaceTree<mlpack::bound::HRectBound<2>, mlpack::neighbor::QueryStat<mlpack::neighbor::NearestNeighborSort>, arma::Mat<double> >*, double, arma::Mat<unsigned long>&, arma::Mat<double>&)+0x44a [[bt](0x42c442]
[DEBUG]): (4) ./bin/allknn : mlpack::neighbor::NeighborSearch<mlpack::neighbor::NearestNeighborSort, mlpack::metric::LMetric<2, false>, mlpack::tree::BinarySpaceTree<mlpack::bound::HRectBound<2>, mlpack::neighbor::QueryStat<mlpack::neighbor::NearestNeighborSort>, arma::Mat<double> > >::ComputeDualNeighborsRecursion(mlpack::tree::BinarySpaceTree<mlpack::bound::HRectBound<2>, mlpack::neighbor::QueryStat<mlpack::neighbor::NearestNeighborSort>, arma::Mat<double> >*, mlpack::tree::BinarySpaceTree<mlpack::bound::HRectBound<2>, mlpack::neighbor::QueryStat<mlpack::neighbor::NearestNeighborSort>, arma::Mat<double> >*, double, arma::Mat<unsigned long>&, arma::Mat<double>&)+0x597 [[bt](0x42c58f]
[DEBUG]): (5) ./bin/allknn : mlpack::neighbor::NeighborSearch<mlpack::neighbor::NearestNeighborSort, mlpack::metric::LMetric<2, false>, mlpack::tree::BinarySpaceTree<mlpack::bound::HRectBound<2>, mlpack::neighbor::QueryStat<mlpack::neighbor::NearestNeighborSort>, arma::Mat<double> > >::Search(unsigned long, arma::Mat<unsigned long>&, arma::Mat<double>&)+0x63c [[bt](0x431f2c]
[DEBUG]): (6) ./bin/allknn : main+0x9db [[bt](0x417a81]
[DEBUG]): (7) /lib64/libc.so.6 : __libc_start_main+0xf4 [[bt](0x318ac1d994]
[DEBUG]): (8) ./bin/allknn : __gxx_personality_v0+0x1e9 [Assert Failed.
[INFO ](0x416a89]
[DEBUG]) 
[INFO ] Execution parameters:
[INFO ]   distances_file: /net/hg200/pram/wrkspc_data/maxip/movielens_nn_dist.csv
[INFO ]   help: false
[INFO ]   info: ""
[INFO ]   k: 1
[INFO ]   leaf_size: 20
[INFO ]   naive: false
[INFO ]   neighbors_file: /net/hg200/pram/wrkspc_data/maxip/movielens_nn.csv
[INFO ]   query_file: /net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensUsers_50.csv
[INFO ]   reference_file: /net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensItems_50.csv
[INFO ]   single_mode: false
[INFO ]   verbose: true
[INFO ] 
[INFO ] Program timers:
[INFO ]   computing_neighbors: 1327604346.859360s (15365 days, 18 hrs, 59 mins,6.8secs)
[INFO ]   total_time: 1.003714s
[INFO ]   tree_building: 0.106337s
  1. Naive: ./bin/allknn -r /net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensItems_50.csv -q /net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensUsers_50.csv -k 1 -d /net/hg200/pram/wrkspc_data/maxip/movielens_nn_dist.csv -n /net/hg200/pram/wrkspc_data/maxip/movielens_nn.csv -Nv

Output:

[Compiled with debugging symbols.
[INFO ](DEBUG]) Loading '/net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensItems_50.csv' as CSV data.
[INFO ] Loaded reference data from '/net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensItems_50.csv'.
[INFO ] Building reference tree...
[INFO ] Loading '/net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensUsers_50.csv' as CSV data.
[INFO ] Loaded query data from '/net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensUsers_50.csv'.
[INFO ] Building query tree...
[INFO ] Tree built.
[INFO ] Computing 1 nearest neighbors...

error: Mat::unsafe_col(): out of bounds

terminate called after throwing an instance of 'std::logic_error'
  what():  
Aborted

(gdb) run
error: Mat::unsafe_col(): out of bounds

terminate called after throwing an instance of 'std::logic_error'
  what():  

Program received signal SIGABRT, Aborted.
0x000000318ac30265 in raise () from /lib64/libc.so.6

(gdb) where
#0  0x000000318ac30265 in raise () from /lib64/libc.so.6
#1  0x000000318ac31d10 in abort () from /lib64/libc.so.6
#2  0x000000318dcbed14 in __gnu_cxx::__verbose_terminate_handler() ()
   from /usr/lib64/libstdc++.so.6
#3  0x000000318dcbce16 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x000000318dcbce43 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5  0x000000318dcbcf2a in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x0000000000422e78 in arma::arma_stop<char const*> (x=@0x7fffffffd128)
    at /usr/include/armadillo_bits/debug.hpp:117
#7  0x00000000004230de in arma::arma_check<char [(state=true, x=...)
    at /usr/include/armadillo_bits/debug.hpp:346
#8  0x000000000042baf2 in arma::Mat<double>::unsafe_col (this=0x7fffffffdab0, 
    col_num=0) at /usr/include/armadillo_bits/Mat_meat.hpp:1995
#9  0x000000000042bbca in mlpack::neighbor::NeighborSearch<mlpack::neighbor::NearestNeighborSort, mlpack::metric::LMetric<2, false>, mlpack::tree::BinarySpaceTree<mlpack::bound::HRectBound<2>, mlpack::neighbor::QueryStat<mlpack::neighbor::NearestNeighborSort>, arma::Mat<double> > >::ComputeBaseCase (this=0x6491c0, 
    queryNode=0x7fffffffde70, referenceNode=0x7fffffffdeb0, neighbors=..., 
    distances=...)
    at /net/hu12/pram/work_space/mlpack/trunk/src/mlpack/methods/neighbor_search/neighbor_search_impl.hpp:335
#10 0x000000000042c0dd in mlpack::neighbor::NeighborSearch<mlpack::neighbor::NearestNeighborSort, mlpack::metric::LMetric<2, false>, mlpack::tree::BinarySpaceTree<mlpack::bound::HRectBound<2>, mlpack::neighbor::QueryStat<mlpack::neighbor::NearestNeighborSort>, arma::Mat<double> > >::ComputeDualNeighborsRecursion (
    this=0x6491c0, queryNode=0x7fffffffde70, referenceNode=0x7fffffffdeb0, 
    lowerBound=0, neighbors=..., distances=...)
    at /net/hu12/pram/work_space/mlpack/trunk/src/mlpack/methods/neighbor_search/neighbor_search_impl.hpp:400
#11 0x0000000000431f2c in mlpack::neighbor::NeighborSearch<mlpack::neighbor::NearestNeighborSort, mlpack::metric::LMetric<2, false>, mlpack::tree::BinarySpaceTree<mlpack::bound::HRectBound<2>, mlpack::neighbor::QueryStat<mlpack::neighbor::NearestNeighborSort>, arma::Mat<double> > >::Search (this=0x6491c0, k=1, 
    resultingNeighbors=..., distances=...)
    at /net/hu12/pram/work_space/mlpack/trunk/src/mlpack/methods/neighbor_search/neighbor_search_impl.hpp:213
#12 0x0000000000417a81 in main (argc=12, argv=0x7fffffffe228)
    at /net/hu12/pram/work_space/mlpack/trunk/src/mlpack/methods/neighbor_search/allknn_main.cpp:164
  1. Single-tree:./bin/allknn -r /net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensItems_50.csv -q /net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensUsers_50.csv -k 1 -d /net/hg200/pram/wrkspc_data/maxip/movielens_nn_dist.csv -n /net/hg200/pram/wrkspc_data/maxip/movielens_nn.csv -sv

Output:

[DEBUG](33]>) Compiled with debugging symbols.
[INFO ] Loading '/net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensItems_50.csv' as CSV data.
[INFO ] Loaded reference data from '/net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensItems_50.csv'.
[INFO ] Building reference tree...
[INFO ] Loading '/net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensUsers_50.csv' as CSV data.
[INFO ] Loaded query data from '/net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensUsers_50.csv'.
[INFO ] Building query tree...
[INFO ] Tree built.
[INFO ] Computing 1 nearest neighbors...
[INFO ] Neighbors computed.
[INFO ] Re-mapping indices...
[INFO ] Saving CSV data to '/net/hg200/pram/wrkspc_data/maxip/movielens_nn_dist.csv'.
[INFO ] Saving CSV data to '/net/hg200/pram/wrkspc_data/maxip/movielens_nn.csv'.
[INFO ] 
[INFO ] Execution parameters:
[INFO ]   distances_file: /net/hg200/pram/wrkspc_data/maxip/movielens_nn_dist.csv
[INFO ]   help: false
[INFO ]   info: ""
[INFO ]   k: 1
[INFO ]   leaf_size: 20
[INFO ]   naive: false
[INFO ]   neighbors_file: /net/hg200/pram/wrkspc_data/maxip/movielens_nn.csv
[INFO ]   query_file: /net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensUsers_50.csv
[INFO ]   reference_file: /net/hg200/pram/datasets/ip_datasets/MovieLens/MovieLensItems_50.csv
[INFO ]   single_mode: true
[INFO ]   verbose: true
[INFO ] 
[INFO ] Program timers:
[INFO ]   computing_neighbors: 0.000012s
[INFO ]   total_time: 0.997204s
[INFO ]   tree_building: 0.105990s

Everything seems fine, but when you check the output files, they are empty:
-rw-r--r-- 1 pram fastlab 0 Jan 26 14:04 ../maxip/movielens_nn.csv
-rw-r--r-- 1 pram fastlab 0 Jan 26 14:04 ../maxip/movielens_nn_dist.csv

The binary has similar behavior with another data set I tried:

-r /net/hg200/pram/datasets/nn_datasets/optd_r.csv (n = 1379, d = 64)
-q /net/hg200/pram/datasets/nn_datasets/optd_q.csv (n = 450, d = 64)

Failing for Naive, and empty output files for single and dual-tree search.

@rcurtin rcurtin self-assigned this Dec 29, 2014
@rcurtin rcurtin added this to the mlpack 1.0.1 milestone Dec 29, 2014
@rcurtin rcurtin closed this Dec 29, 2014
@rcurtin
Member
rcurtin commented Dec 30, 2014

Commented by rcurtin on 14 Jan 42072128 00:24 UTC
Where can I find these datasets? I can't reproduce this particular bug with 'test_data_3_1000.csv' (but I managed to find another, which is a bit embarrassing...).

@rcurtin
Member
rcurtin commented Dec 30, 2014

Commented by pram on 14 Apr 42072138 22:05 UTC
Well I have given the paths to these datasets. They are all sitting on the raid (/net/hg200) so you should be able to access them (I think).

@rcurtin
Member
rcurtin commented Dec 30, 2014

Commented by rcurtin on 6 Apr 42072148 18:29 UTC
Oh... oops, I should have seen that. I'm going to solve #192 first and then this.

@rcurtin
Member
rcurtin commented Dec 30, 2014

Commented by rcurtin on 6 Dec 42072201 19:50 UTC
This may be the same bug as #192 but manifested differently (I thought they were completely different at first). I submitted r11292 which may fix this issue, but I have to run for the moment so I can't test it. Pari, if you re-test it and it works, can you resolve this bug? If it doesn't work... well, let me know and I'll keep fighting with it...

@rcurtin
Member
rcurtin commented Dec 30, 2014

Commented by pram on 21 Aug 42073235 09:06 UTC
Replying to rcurtin:

This may be the same bug as #192 but manifested differently (I thought they were completely different at first). I submitted r11292 which may fix this issue, but I have to run for the moment so I can't test it. Pari, if you re-test it and it works, can you resolve this bug? If it doesn't work... well, let me know and I'll keep fighting with it...

I run the NN search on the datasets mentioned previously and they seemed to work fine this time around without any issues. I have not checked the correctness of the results though. Should I still mark it as resolved or wait till it is tested?

@rcurtin
Member
rcurtin commented Dec 30, 2014

Commented by rcurtin on 5 Feb 42073342 02:06 UTC
The results are very likely to be correct (in the test I ran, they were correct, and the AllkNNTest/AllkFNTest report success), so I'll resolve it. The reason those tests didn't pick up this bug is because the bug was in allknn_main.cpp not the actual NeighborSearch class, which is what's tested. I guess it would be good to actually run the executables on some datasets to test them too, but our current testing framework doesn't quite allow for that and I'm not sure how to go about doing it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment