Segmentation fault with SVC with probability = True and degree 3

[rshah1990]

I am using SVC with predict proba with below hyper-parameter on large data set. it is giving segmentation fault error. code works well for smaller data set.
I don't think its memory issue we are using 96 GB RAM. I have also monitored system using htop during run & I didn't gave impression of memory issue.

hyper-parameter :
SVC(kernel='poly',gamma=100,C=10,degree=3,decision_function_shape='ovo',probability=True,random_state=42,verbose=True)
No of Features : 768
No of rows : 2.1 million

[amueller]

Please provide self-contained example code, including imports and data (if possible), so that other contributors can just run it and reproduce your issue. Ideally your example code should be minimal.

[rth]

There were known segfaults in the polynomial SVC kernel in the past #6687 that should have been resolved, but maybe not everything was.

[rshah1990]

Dummy code

import numpy as np
features = np.random.rand(2100000,768) 
target = np.random.randint(low =100000,high=500000, size=2100000)
from sklearn.svm import SVC
svc = SVC(kernel='poly',gamma=100,C=10,degree=3,decision_function_shape='ovo',probability=True,random_state=42,verbose=True)
svc.fit(features,target)

[rshah1990]

Any update on this ? are you guys able to re-produce the issue?

[rth]

Yes, I can reproduce the segfault with the above example. The backtrace is,

#0  0x00007ffce605c026 in svm::sigmoid_train (l=8, dec_values=dec_values@entry=0x555556012620, labels=0x5555564f9c70,
    A=<error reading variable>, B=<error reading variable>) at sklearn/svm/src/libsvm/svm.cpp:1918
#1  0x00007ffce6063dac in svm::svm_binary_svc_probability (status=0x7fffffffc1ec, probB=<error reading variable>,
    probA=<error reading variable>, Cn=<optimized out>, Cp=<optimized out>, param=0x7fffffffc6b0, prob=0x7fffffffc070)
    at sklearn/svm/src/libsvm/svm.cpp:2176
#2  svm_train (prob=0x7fffffffc050, prob@entry=0x7fffffffc210, param=param@entry=0x7fffffffc6b0, status=status@entry=0x7fffffffc1ec)
    at sklearn/svm/src/libsvm/svm.cpp:2488
#3  0x00007ffce6046afd in __pyx_pf_7sklearn_3svm_6libsvm_fit (__pyx_v_X=__pyx_v_X@entry=0x7ffff6b40030,
    __pyx_v_Y=__pyx_v_Y@entry=0x7ffce4844990, __pyx_v_svm_type=__pyx_v_svm_type@entry=0, __pyx_v_kernel=__pyx_v_kernel@entry=0x7ffff69fe500,
    __pyx_v_degree=__pyx_v_degree@entry=3, __pyx_v_gamma=__pyx_v_gamma@entry=100, __pyx_v_coef0=__pyx_v_coef0@entry=0,
    __pyx_v_tol=__pyx_v_tol@entry=0.001, __pyx_v_C=__pyx_v_C@entry=10, __pyx_v_nu=__pyx_v_nu@entry=0,
    __pyx_v_epsilon=__pyx_v_epsilon@entry=0, __pyx_v_class_weight=__pyx_v_class_weight@entry=0x7ffce4844a30,
    __pyx_v_sample_weight=0x7ffce616d3f0, __pyx_v_shrinking=1, __pyx_v_probability=1, __pyx_v_cache_size=200, __pyx_v_max_iter=-1,
    __pyx_v_random_seed=1608637542, __pyx_self=<optimized out>) at sklearn/svm/libsvm.c:2881
#4  0x00007ffce604e263 in __pyx_pw_7sklearn_3svm_6libsvm_1fit (__pyx_self=<optimized out>, __pyx_args=<optimized out>,
    __pyx_kwds=<optimized out>) at sklearn/svm/libsvm.c:2335

more investigation is needed though to understand what is exactly wrong in sklearn/svm/src/libsvm/svm.cpp:sigmoid_train

[rth]

Given that it segfaults on random data with different initial seeds my guess it has more to with the large size of the data.

[rshah1990]

We have tried on r5a.24xlarge (96 core,768 GiB) machine available on ec2 instance. gave same issue.

[rth]

I mean not an issue with hardware but that 2million*768 will overflow for an int32. So maybe some index needs to be changed from int to long somewhere in the C++ libsvm code. I have not looked at the code in detail, so it's only a guess though.

[amueller]

@rth that would be my hunch as well.

[amueller]

There's also 400000 classes in the synthetic dataset. Does it still error with, say 2 or 10? It didn't segfault for me immediately with less classes but I don't have enough ram to actually try...

[rth]

There's also 400000 classes in the synthetic dataset. Does it still error with, say 2 or 10? It didn't segfault for me immediately with less classes but I don't have enough ram to actually try...

The stdout before the segfault was,

optimization finished, #iter = 16
obj = -0.000000, rho = 0.134267
nSV = 7, nBSV = 0
Total nSV = 7
..*.*  
optimization finished, #iter = 20
obj = -0.000000, rho = 0.260627
nSV = 6, nBSV = 0
Total nSV = 6
.*.*   
optimization finished, #iter = 17
obj = -0.000000, rho = -0.194731
nSV = 7, nBSV = 0
Total nSV = 7
..*.*  
optimization finished, #iter = 20
obj = -0.000000, rho = -0.295525
nSV = 6, nBSV = 0
Total nSV = 6
.*.*   
optimization finished, #iter = 13
obj = -0.000000, rho = -0.102764
nSV = 6, nBSV = 0
Total nSV = 6

Thread 1 "python" received signal SIGSEGV, Segmentation fault.

so I imagine it means that the first 5 classes when through fine.

[RyanWoodard]

I believe I have a fix for this.

Just wondering if that code above should be added to the tests? It takes quite a while to run.