Adding breaking NLI dataset as benchmark

[tomwesolowski]

Hello,

Let me congratulate to you on great research efforts. I've tried to use the code against breaking NLI ( https://github.com/BIU-NLP/Breaking_NLI ). I modified preprocess_data.py file and use build_sequence and CoreNLP methods to prepare data to feed into the model. I manually checked files like *_token.txt and '*_lemma.txt' and they look OK to me.

To evaluate a model on the new dataset, I changed gen.py file adding new TextIterator object and calling pred_acc function.

These are the results I get:
kim_accuracies_test 0.886
kim_accuracies_train 0.931
kim_accuracies_valid 0.887
kim_accuracies_breaking 0.109

Do you have any ideas why the number is so low?

Thanks

[lukecq1231]

Hi, I don't know why you get the low number. Actually, I tested KIM model on breaking NLI data, and I got a good number of 83.5%. You can find the detail experiment on our ACL paper: http://www.aclweb.org/anthology/P18-1224

[swarnaHub]

Hi @tomwesolowski , could you kindly share the model with me. I am unable to run the GPU code because of some cuda driver incompatibilities. I also cannot update the drivers because of some other constraints.

[tomwesolowski]

I've had similar issues. Installing Vonda helped. I needed to change a couple of cuda/theano files to be able to run it. I haven't changed anything in the code.

[swarnaHub]

If you have the model with you, perhaps you can upload it somewhere? Or, if you can point me to the exact fixes, that should also work.

[tomwesolowski]

The model is identical as in the repository. I just followed various pieces of advice on Theano blogs/websites to make it work. I don't remember what was it exactly, unfortunately. Best of luck

…

On Thu, 29 Nov 2018, 09:51 Swarnadeep Saha ***@***.*** wrote: If you have the model with you, perhaps you can upload it somewhere? Or, if you can point me to the exact fixes, that should also work. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAWjYH0u39z5iuqz5ZMJl8_apkpUspn9ks5uz6AqgaJpZM4VK9Ri> .

[jabalazs]

@swarnaHub I made it work after doing the following:

• Create a conda environment, install theano, pygpu, and numpy==1.12:

  $ conda create -n kim python=2.7.13
  $ conda activate kim
  (kim) $ conda install theano pygpu
  (kim) $ pip install numpy==1.12 
  

  This will install Theano 1.0.3 (Kim runs with this newer version).

  I also made sure that the version of numpy being imported by the python in my PATH was 1.12.0

  (kim) $ python -c "import numpy; print(numpy.version.version)"
  1.12.0
  

• I removed the Theano cache directory as suggested in Broken build Theano/Theano#3762 (comment)

  (kim) $ rm -rf ~/.theano
  

• I followed the steps recommended here: cannot compile with cuDNN Theano/Theano#5943 (comment)

• I exported the THEANO_FLAGS environment variable specified in https://github.com/lukecq1231/kim/blob/master/scripts/kim/train.sh, but changed device=gpu0 to device=cuda1 and removed lib.cnmem=0.9. Note that I used cuda1 but you could also use cuda0 in case you have only one gpu:

  (kim) $ export THEANO_FLAGS='mode=FAST_RUN,device=cuda1,floatX=float32,optimizer_including=cudnn,warn_float64=warn'
  

• Finally I get:

  (kim) $ python -c "import theano"
  WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
  Using cuDNN version 7103 on context None
  Mapped name None to device cuda1: TITAN Xp (0000:02:00.0)
  

---

Some more info for further reference:

• ~/.theanorc

  (kim) $ cat ~/.theanorc
  [cuda]
  # Set to where the cuda drivers are installed.
  # You might have to change this depending where your cuda driver/what version is installed.
  root=/usr/local/cuda/bin
  
  [dnn]
  include_path=/usr/local/cuda/include
  library_path=/usr/local/cuda/lib64
  

• LD_LIBRARY_PATH

  (kim) $ echo $LD_LIBRARY_PATH
  /usr/local/cuda/lib64:/usr/local/cuda/lib:/usr/local/cuda/extras/CUPTI/lib64
  

• cuda:

  (kim) $ nvcc --version
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2017 NVIDIA Corporation
  Built on Fri_Nov__3_21:07:56_CDT_2017
  Cuda compilation tools, release 9.1, V9.1.85
  

• pip

  (kim) $ pip list
  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
  Package       Version
  ------------- ------------------------
  certifi       2019.3.9
  colorama      0.3.9
  Lasagne       0.2.dev1
  Mako          1.0.7
  MarkupSafe    1.1.1
  mkl-fft       1.0.0
  mkl-random    1.0.1
  nose          1.3.7
  numpy         1.12.0
  opencv-python 4.0.0.21
  pip           19.0.3
  pygpu         0.7.6
  scipy         1.1.0
  setuptools    40.8.0
  six           1.11.0
  termcolor     1.1.0
  Theano        1.0.3+2.g3e47d39ac.dirty
  tldr          0.4.2
  wheel         0.33.1
  

• conda:

  (kim) $ conda list
  # Name                    Version                   Build  Channel
  anaconda                  custom           py27h4a00acb_0
  binutils_impl_linux-64    2.31.1               h6176602_1
  binutils_linux-64         2.31.1               h6176602_6
  blas                      1.0                         mkl
  ca-certificates           2019.1.23                     0
  certifi                   2019.3.9                 py27_0
  gcc_impl_linux-64         7.3.0                habb00fd_1
  gcc_linux-64              7.3.0                h553295d_6
  gxx_impl_linux-64         7.3.0                hdf63c60_1
  gxx_linux-64              7.3.0                h553295d_6
  intel-openmp              2019.3                      199
  libedit                   3.1                  heed3624_0
  libffi                    3.2.1                hd88cf55_4
  libgcc-ng                 8.2.0                hdf63c60_1
  libgfortran               3.0.0                         1
  libgfortran-ng            7.3.0                hdf63c60_0
  libgpuarray               0.7.6                h14c3975_0
  libopenblas               0.3.3                h5a2b251_3
  libstdcxx-ng              8.2.0                hdf63c60_1
  mako                      1.0.7                    pypi_0    pypi
  markupsafe                1.1.1            py27h7b6447c_0
  mkl                       2018.0.3                      1
  mkl-service               1.1.2                    py27_3
  mkl_fft                   1.0.1            py27h3010b51_0
  mkl_random                1.0.1            py27h629b387_0
  ncurses                   6.0                  h9df7e31_2
  nose                      1.3.7                    py27_2
  numpy                     1.12.0                   pypi_0    pypi
  openblas                  0.2.14                        4
  openssl                   1.0.2r               h7b6447c_0
  pip                       19.0.3                   py27_0
  pygpu                     0.7.6            py27h035aef0_0
  python                    2.7.13              heccc3f1_16
  readline                  7.0                  ha6073c6_4
  scipy                     1.1.0            py27hd20e5f9_0
  setuptools                40.8.0                   py27_0
  six                       1.12.0                   py27_0
  sqlite                    3.23.1               he433501_0
  theano                    1.0.3            py27hfd86e86_0
  tk                        8.6.8                hbc83047_0
  wheel                     0.33.1                   py27_0
  zlib                      1.2.11               h7b6447c_3