Vowpal Wabbit failing import

[gaborfodor]

import vowpalwabbit # nothing happens

from vowpalwabbit import pyvw  #fails
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-10-76d9c58cae1b> in <module>()
----> 1 from vowpalwabbit import pyvw

/opt/conda/lib/python3.6/site-packages/vowpalwabbit/pyvw.py in <module>()
      1 from __future__ import division
----> 2 import pylibvw
      3 
      4 
      5 class SearchTask():

ImportError: dynamic module does not define module export function (PyInit_pylibvw)

from vowpalwabbit.sklearn_vw import VWClassifier # fails
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-11-14a6a5c77f64> in <module>()
----> 1 from vowpalwabbit.sklearn_vw import VWClassifier

/opt/conda/lib/python3.6/site-packages/vowpalwabbit/sklearn_vw.py in <module>()
     16 from sklearn.utils.validation import check_is_fitted
     17 from sklearn.externals import joblib
---> 18 from vowpalwabbit import pyvw
     19 
     20 

/opt/conda/lib/python3.6/site-packages/vowpalwabbit/pyvw.py in <module>()
      1 from __future__ import division
----> 2 import pylibvw
      3 
      4 
      5 class SearchTask():

ImportError: dynamic module does not define module export function (PyInit_pylibvw)

[nerdcha]

Thanks for flagging this. Looks like Vowpal Wabbit's build process is quite complex, and the patches we used for Python 3.4 haven't carried across to Python 3.6. I'm going to have to keep working on this.

[mlisovyi]

Was there a progress on VW integration? In the kernels on kaggle one gets:

import vowpalwabbit
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-ea6142880cc4> in <module>()
----> 1 import vowpalwabbit

ModuleNotFoundError: No module named 'vowpalwabbit'

[ghost]

@mlisovyi The last time I tried getting vowpal wabbit working was unsuccessful. I don't remember why so I'll put it on my to do list to try again.

[mlisovyi]

@crawforc3 Thanks a lot! this will be very helpful as vowpalwabbit is a handy tool and a kernel with nice intro will be helpful for beginners. The python binding of VW would be sufficient.
Are those problems related specifically to kaggle-docker or did you face some general issues?

[mlisovyi]

Ok, i managed to install relevant packages and VW in the docker image locally. This requires a few small updates to the VW Makefiles and there is already a relevant VW PR. I will wait for that to be merged (fingers crossed) and will add a PR request here.
One technical question:
what is the policy on compiler to be used? In order to implement VW I had to install and make use of g++ version 7.2.0 as shipped by conda in gxx_linux-64 package. Is that any kind of problem?

[sebbov]

No particular policy at this time. You're probably the first who wonders about that.

I believe that, so long it produces ABI compatible objects it interfaces with, it should be fine. If that isn't the case, I would expect you to notice issues with that in build failures or at runtime (make sure to run the tests!).

Some issues sometimes sneak through this process and we will roll back the culprit changes accordingly, reintroducing them (with increased test coverage) with a fix, when/if a fix is found.

[mlisovyi]

Thanks for the prompt feedback. Could you point me at the instructions on the tests required? This is my first experience with docker. The main README.md does not mention anything.

[ghost]

Hi @mlisovyi, we have some rudimentary tests in test_build.py. At a bare minimum, that script should run successfully.

[sebbov]

I also just updated the README.md with the build / test instructions. Thanks for letting me know if that covers it and works for you.

[mlisovyi]

The required PR in vowpal wabbit has been merged. Now i'm trying to add it o the kaggle docker file.

@sebbov are there a simplified procedure? Or practical suggestions on how to speed up building. I was running the build script for ~3 hours on a dual-core laptop with 2 threads per core and a reasonably good home internet connection and I managed to complete only a half of the build (approx. up to line 210). Afterwards I had to interrupt building, as I had to catch a train. Is there a chance to do a test build on a remote server or to build on top of the downloaded docker image? inside the build script I see a --use-cache command line option, but it is not clear how is it supposed to be used. Any advice is welcome!

[mlisovyi]

Just added the first VW kernel on the Titanic dataset, and it runs smooth: https://www.kaggle.com/mlisovyi/vowpal-wabbit-decides-who-lives-and-who-dies. I guess, this issue can be closed now

[ghost]

Awesome :) Closing this now but let us know if you run into any problems

[ghost]

@mlisovyi Re-opening because I am no longer able to install VW in our docker images. I think it may be related to VowpalWabbit/vowpal_wabbit@a300157

[mlisovyi]

@crawforc3 Can you share a log? Indeed, it was never checked in kaggle kernels, but vw developers claimed that it was checked in a docker image in some setup

[mlisovyi]

@crawforc3 is there a way to check it without download and built of the WHOLE image? It is way too large by now. I changed laptop and now would have to install everything from scratch, which is very demotivating, to be honest

[ghost]

I had to pin the previous version for it to work. Closing this for now.