Several re-ranking issues in experiments-msmarco-document.md

[joelrorseth]

Hi, I'm following the instructions in docs/experiments-msmarco-document.md, and I am running into a few issues during the first re-ranking step Re-Ranking with monoT5 (https://github.com/castorini/pygaggle/blob/master/docs/experiments-msmarco-document.md#re-ranking-with-monot5).

Here are my environment details. Btw, I'm a UW CS grad student, and I'm running on the datasci gpu2 machine. I'd be happy to try to reproduce these issues on other school machines, if you know of any.

Ubuntu 18.04.5 LTS
Python 3.6.9
GPUs: 2x Tesla P40
Cuda compilation tools, release 10.2, V10.2.89

I'm getting the following warning about my CUDA version. It seems that TensorFlow 2.5.0 (from requirements.txt) requires CUDA 11.2, according to their website https://www.tensorflow.org/install/source#gpu. This doesn't seem to prevent anything from running, so this is just FYI. I think it would be great to list the required CUDA version in the README, or in this document I'm following.

2022-04-09 12:59:30.120693: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-04-09 12:59:30.120734: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

First, I got the following error because my JAVA_HOME was not set. This was an easy enough fix, for me I used export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64.

Traceback (most recent call last):
  File "/my-project/pygaggle/venv/lib/python3.6/site-packages/jnius/env.py", line 322, in get_jdk_home
    which('javac')
  File "/usr/lib/python3.6/posixpath.py", line 394, in realpath
    filename = os.fspath(filename)
TypeError: expected str, bytes or os.PathLike object, not NoneType

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/my-project/pygaggle/pygaggle/run/evaluate_document_ranker.py", line 11, in <module>
    from pygaggle.rerank.base import Reranker
  File "/my-project/pygaggle/pygaggle/rerank/base.py", line 5, in <module>
    from pyserini.search import JSimpleSearcherResult
  File "/my-project/pygaggle/venv/lib/python3.6/site-packages/pyserini/search/__init__.py", line 17, in <module>
    from ._base import JQuery, JDisjunctionMaxQueryGenerator, get_topics, get_topics_with_reader, get_qrels_file, get_qrels
  File "/my-project/pygaggle/venv/lib/python3.6/site-packages/pyserini/search/_base.py", line 26, in <module>
    from ..pyclass import autoclass
  File "/my-project/pygaggle/venv/lib/python3.6/site-packages/pyserini/pyclass.py", line 27, in <module>
    from jnius import autoclass, cast
  File "/my-project/pygaggle/venv/lib/python3.6/site-packages/jnius/__init__.py", line 43, in <module>
    from .reflect import *  # noqa
  File "/my-project/pygaggle/venv/lib/python3.6/site-packages/jnius/reflect.py", line 20, in <module>
    class Class(with_metaclass(MetaJavaClass, JavaClass)):
  File "/my-project/pygaggle/venv/lib/python3.6/site-packages/six.py", line 856, in __new__
    return meta(name, resolved_bases, d)
  File "jnius/jnius_export_class.pxi", line 119, in jnius.MetaJavaClass.__new__
  File "jnius/jnius_export_class.pxi", line 179, in jnius.MetaJavaClass.resolve_class
  File "jnius/jnius_env.pxi", line 11, in jnius.get_jnienv
  File "jnius/jnius_jvm_dlopen.pxi", line 102, in jnius.get_platform_jnienv
  File "jnius/jnius_jvm_dlopen.pxi", line 54, in jnius.create_jnienv
  File "/my-project/pygaggle/venv/lib/python3.6/site-packages/jnius/env.py", line 66, in get_java_setup
    JAVA_HOME = get_jdk_home(platform)
  File "/my-project/pygaggle/venv/lib/python3.6/site-packages/jnius/env.py", line 325, in get_jdk_home
    raise Exception('Unable to find javac')
Exception: Unable to find javac

Once I fixed the above error, I got the following error about missing faiss module:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/my-project/pygaggle/pygaggle/run/evaluate_document_ranker.py", line 11, in <module>
    from pygaggle.rerank.base import Reranker
  File "/my-project/pygaggle/pygaggle/rerank/base.py", line 5, in <module>
    from pyserini.search import JSimpleSearcherResult
  File "/my-project/pygaggle/venv/lib/python3.6/site-packages/pyserini/search/__init__.py", line 20, in <module>
    from ._impact_searcher import JImpactSearcherResult, ImpactSearcher
  File "/my-project/pygaggle/venv/lib/python3.6/site-packages/pyserini/search/_impact_searcher.py", line 28, in <module>
    from pyserini.encode import QueryEncoder, TokFreqQueryEncoder, UniCoilQueryEncoder, CachedDataQueryEncoder
  File "/my-project/pygaggle/venv/lib/python3.6/site-packages/pyserini/encode/__init__.py", line 17, in <module>
    from ._base import DocumentEncoder, QueryEncoder, JsonlCollectionIterator,\
  File "/my-project/pygaggle/venv/lib/python3.6/site-packages/pyserini/encode/_base.py", line 19, in <module>
    import faiss
ModuleNotFoundError: No module named 'faiss'

I tried pip install faiss, which threw another import error about _swigfaiss. According to the discussion here facebookresearch/faiss#821 (comment), you can get around this issue by using pip install faiss-cpu --no-cache.

This resolved my import issues, however now I'm stuck with the following spacy issue. It appears to be due to an oversight while migrating from spacy v2 to v3, see https://stackoverflow.com/questions/67906945/valueerror-nlp-add-pipe-now-takes-the-string-name-of-the-registered-component-f. Here is the trackback I got:

2022-04-09 12:59:31 [INFO] loader: Loading faiss with AVX2 support.
2022-04-09 12:59:31 [INFO] loader: Could not load library with AVX2 support due to:
ModuleNotFoundError("No module named 'faiss.swigfaiss_avx2'",)
2022-04-09 12:59:31 [INFO] loader: Loading faiss.
2022-04-09 12:59:31 [INFO] loader: Successfully loaded faiss.
2022-04-09 12:59:55 [INFO] evaluate_document_ranker: Preprocessing Queries & Docs:
100%|██████████| 25000/25000 [02:19<00:00, 179.36it/s]
2022-04-09 13:02:16 [INFO] msmarco: Expected P@1 for Random Ordering: 0.0008799999999999999
2022-04-09 13:02:16 [INFO] msmarco: Expected R@1000 for Random Ordering: 0.88
2022-04-09 13:02:16 [INFO] msmarco: Expected MRR for Random Ordering: 0.006587214357284643
2022-04-09 13:02:16 [INFO] msmarco: Expected MRR@10 for Random Ordering: 0.002577492063492063
2022-04-09 13:02:16 [INFO] msmarco: Existing MRR: 0.17985744638145454
2022-04-09 13:02:16 [INFO] msmarco: Existing MRR@10: 0.162
2022-04-09 13:02:16 [INFO] evaluate_document_ranker: Loading Ranker & Tokenizer:
Downloading: 100%|██████████| 1.84k/1.84k [00:00<00:00, 577kB/s]
Downloading: 100%|██████████| 892M/892M [00:21<00:00, 42.4MB/s]
Downloading: 100%|██████████| 1.20k/1.20k [00:00<00:00, 611kB/s]
Downloading: 100%|██████████| 792k/792k [00:00<00:00, 7.69MB/s]
Downloading: 100%|██████████| 1.39M/1.39M [00:00<00:00, 11.2MB/s]
Writing run: True
2022-04-09 13:02:56 [INFO] evaluate_document_ranker: Reranking:
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/my-project/pygaggle/pygaggle/run/evaluate_document_ranker.py", line 171, in <module>
    main()
  File "/my-project/pygaggle/pygaggle/run/evaluate_document_ranker.py", line 166, in main
    options.aggregate_method):
  File "/my-project/pygaggle/pygaggle/model/evaluate.py", line 182, in evaluate_by_segments
    segment_processor = SegmentProcessor()
  File "/my-project/pygaggle/pygaggle/data/segmentation.py", line 29, in __init__
    self.nlp.add_pipe(self.nlp.create_pipe("sentencizer"))
  File "/my-project/pygaggle/venv/lib/python3.6/site-packages/spacy/language.py", line 754, in add_pipe
    raise ValueError(err)
ValueError: [E966] `nlp.add_pipe` now takes the string name of the registered component factory, not a callable component. Expected string, but got <spacy.pipeline.sentencizer.Sentencizer object at 0x7fddb209eec8> (name: 'None').

- If you created your component with `nlp.create_pipe('name')`: remove nlp.create_pipe and call `nlp.add_pipe('name')` instead.

- If you passed in a component like `TextCategorizer()`: call `nlp.add_pipe` with the string name instead, e.g. `nlp.add_pipe('textcat')`.

- If you're using a custom component: Add the decorator `@Language.component` (for function components) or `@Language.factory` (for class components / factories) to your custom component and assign it a name, e.g. `@Language.component('your_name')`. You can then run `nlp.add_pipe('your_name')` to add it to the pipeline.

Unfortunately, I can't move past this last error to continue debugging. Let me know how I can help, or if you have any ideas I can try. Thanks!

[lintool]

hey @joelrorseth thanks for noting these issues. I just pinged you on Slack as follow-up.

I don't think anyone has tried to replicate this for a while, so it's great that you're working on it... let me loop in @ronakice here.

But tl;dr - we'll get someone from our end to help you store through everything!