CSCMatrix result raise a ArrayIndexOutOfBoundsException

[elbaulp]

Hi, I posted a question on stackoverflow for this, but may be this is a proper place.

I am working with a sparse matrix (CSCMatrix) in scala using Breeze. After creating and filling the matrix:

val tempFeatures = new CSCMatrix.Builder[Boolean](rows = trainX(lp) size,
                                                  cols = NFeatures)
// Populate with tempFeatures.add(....)

When I call tempFeatures.result(), as said in its documentation I am getting an ArrayIndexOutOfBoundsException:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 514

The size of my Sparse matrix is 28x513, so the exception make sense. But I have no control on the method result(). Is there a way I could fix this?

The only thing related to this error I've found is this issue in Breeze github

Looking at the Breeze code, it seems that the exception its happening here:

    // In CSCMatrix.scala
    if(!colsEqual) {
      while(lastCol < col) {
        outCols(lastCol+1) = outDataIndex
        lastCol += 1
      }
    }

I am debugging to try to catch the bug, but it seems the problem is that my column size is 513, but internally that while loop is checking if lastCol (512) is less that col (516), That is the reason of the problem. May be one way to solve it is replacing the while with:

while(lastCol < _col) {

And then pad the rest of the array until col?

[dlwh]

can you possibly give me a snippet that reproduces?

On Wed, Sep 28, 2016 at 4:45 AM, Alejandro Alcalde <notifications@github.com

wrote:

Hi, I posted a question on stackoverflow for this
http://stackoverflow.com/questions/39746018/cscmatrix-result-raise-a-arrayindexoutofboundsexception,
but may be this is a proper place.

I am working with a sparse matrix (CSCMatrix) in scala using Breeze.
After creating and filling the matrix:

val tempFeatures = new CSCMatrix.Builder[Boolean](rows = trainX%28lp%29 size,
cols = NFeatures)
// Populate with tempFeatures.add(....)

When I call tempFeatures.result(), as said in its documentation
https://github.com/scalanlp/breeze/wiki/Data-Structures#creating-cscmatrices
I am getting an ArrayIndexOutOfBoundsException:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 514

The size of my Sparse matrix is 28x513, so the exception make sense. But
I have no control on the method result(). Is there a way I could fix this?

The only thing related to this error I've found is this issue in Breeze
github #320

Looking at the Breeze code, it seems that the exception its happening here:

// In CSCMatrix.scala
if(!colsEqual) {
  while(lastCol < col) {
    outCols(lastCol+1) = outDataIndex
    lastCol += 1
  }
}

I am debugging to try to catch the bug, but it seems the problem is that
my column size is 513, but internally that while loop is checking if
lastCol (512) is less that col (516), That is the reason of the problem.
May be one way to solve it is replacing the while with:

while(lastCol < _col) {

And then pad the rest of the array until col?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#589, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAloYHFpK0mFDz8IPH_zxp0dmQYh_Fnks5qulM8gaJpZM4KIuLM
.

[elbaulp]

Of course, I have added a print statement:

          val tempFeatures = new CSCMatrix.Builder[Boolean](rows = 28, cols = 513)

                    trainX.zipWithIndex foreach {
                      case ((_, vec), index) => vec foreach (int => {
                        println(s"$index, $int")
                        tempFeatures add(index,int, true)
                      })
                    }

          features getOrElseUpdate (lp, tempFeatures.result())

The result of the print statement:

2, 3
2, 36
2, 53
2, 57
2, 76
2, 76
2, 82
2, 82
2, 107
2, 127
2, 219
2, 234
2, 247
2, 289
2, 296
2, 296
2, 297
2, 297
2, 298
2, 298
2, 299
2, 299
2, 314
2, 345
2, 379
2, 399
1, 15
1, 23
1, 33
1, 33
1, 37
1, 37
1, 56
1, 56
1, 56
1, 56
1, 62
1, 62
1, 62
1, 62
1, 73
1, 98
1, 116
1, 120
1, 170
1, 190
1, 264
1, 299
1, 302
1, 302
1, 303
1, 303
1, 304
1, 305
1, 341
1, 354
1, 382
1, 409
1, 416
1, 417
1, 418
1, 419
1, 442
1, 462
0, 0
0, 22
0, 33
0, 37
0, 56
0, 56
0, 62
0, 62
0, 83
0, 97
0, 180
0, 195
0, 253
0, 295
0, 302
0, 302
0, 303
0, 303
0, 304
0, 304
0, 305
0, 305
0, 320
0, 351
0, 386
0, 406
0, 437
0, 467
5, 7
5, 16
5, 25
5, 25
5, 25
5, 29
5, 29
5, 29
5, 46
5, 46
5, 52
5, 52
5, 60
5, 75
5, 86
5, 86
5, 86
5, 90
5, 90
5, 90
5, 109
5, 109
5, 109
5, 109
5, 115
5, 115
5, 115
5, 115
5, 116
5, 155
5, 169
5, 169
5, 173
5, 173
5, 225
5, 246
5, 341
5, 348
5, 385
5, 409
5, 412
5, 412
5, 413
5, 413
5, 414
5, 415
5, 449
5, 464
5, 479
5, 516
5, 525
5, 526
5, 527
5, 528
6, 26
6, 40
6, 53
6, 53
6, 57
6, 57
6, 76
6, 76
6, 76
6, 76
6, 82
6, 82
6, 82
6, 82
6, 94
6, 129
6, 157
6, 162
6, 209
6, 229
6, 258
6, 293
6, 296
6, 296
6, 297
6, 297
6, 298
6, 299
6, 335
6, 348
6, 375
6, 400
6, 409
6, 410
6, 411
6, 412
3, 0
3, 22
3, 33
3, 33
3, 37
3, 37
3, 56
3, 56
3, 62
3, 62
3, 65
3, 100
3, 178
3, 197
3, 220
3, 220
3, 225
3, 225
3, 243
3, 243
3, 248
3, 248
3, 259
3, 294
3, 302
3, 303
3, 329
3, 349
3, 378
3, 413
3, 416
3, 416
3, 417
3, 417
3, 418
3, 419
3, 454
3, 467
4, 17
4, 38
4, 86
4, 130
4, 157
4, 162
4, 180
4, 185
4, 209
4, 229
4, 276
4, 291
4, 304
4, 346
4, 353
4, 353
4, 354
4, 354
4, 355
4, 355
4, 356
4, 356
4, 370
4, 402

And the stacktrace:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 514
    at breeze.linalg.CSCMatrix$Builder.result(CSCMatrix.scala:528)
    at com.elbauldelprogramador.nlp.parser.SVMParser$$anonfun$train$3$$anonfun$apply$11.apply(SVMParser.scala:161)
    at com.elbauldelprogramador.nlp.parser.SVMParser$$anonfun$train$3$$anonfun$apply$11.apply(SVMParser.scala:161)
    at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194)
    at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80)
    at com.elbauldelprogramador.nlp.parser.SVMParser$$anonfun$train$3.apply(SVMParser.scala:161)
    at com.elbauldelprogramador.nlp.parser.SVMParser$$anonfun$train$3.apply(SVMParser.scala:136)
    at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:103)
    at scala.collection.mutable.HashMap$$anon$1$$anonfun$foreach$2.apply(HashMap.scala:103)
    at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
    at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
    at scala.collection.mutable.HashMap$$anon$1.foreach(HashMap.scala:103)
    at com.elbauldelprogramador.nlp.parser.SVMParser.train(SVMParser.scala:136)
    at com.elbauldelprogramador.nlp.Main$.delayedEndpoint$com$elbauldelprogramador$nlp$Main$1(Main.scala:39)
    at com.elbauldelprogramador.nlp.Main$delayedInit$body.apply(Main.scala:27)
    at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
    at scala.App$$anonfun$main$1.apply(App.scala:76)
    at scala.App$$anonfun$main$1.apply(App.scala:76)
    at scala.collection.immutable.List.foreach(List.scala:381)
    at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
    at scala.App$class.main(App.scala:76)
    at com.elbauldelprogramador.nlp.Main$.main(Main.scala:27)
    at com.elbauldelprogramador.nlp.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

[dlwh]

heh, finally looked at this. You were giving it columns greater than what you said the max was. the failure was non obvious so I'm adding a better error message.

[elbaulp]

Thanks!

On Wed, Jan 4, 2017, 07:16 David Hall ***@***.***> wrote: Closed #589 <#589> via 8f36cb7 <8f36cb7> . — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#589 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAYGLaSbIMOBubOzj31JBdqnXMp2PaM-ks5rOzlagaJpZM4KIuLM> .

-- Alejandro Alcalde - elbauldelprogramador.com