[Maintenance] error when running test_end2end

[xaclincoln]

Describe the feature
pip install -e .
python tests/test_end2end.py

Traceback (most recent call last):
File "tests/test_end2end.py", line 3, in
from convlab2.nlu.jointBERT.multiwoz import BERTNLU
File "/home/lincoln/deep-learning/ConvLab-2/convlab2/nlu/jointBERT/multiwoz/init.py", line 1, in
from convlab2.nlu.jointBERT.multiwoz.nlu import BERTNLU
File "/home/lincoln/deep-learning/ConvLab-2/convlab2/nlu/jointBERT/multiwoz/nlu.py", line 10, in
from convlab2.nlu.jointBERT.dataloader import Dataloader
File "/home/lincoln/deep-learning/ConvLab-2/convlab2/nlu/jointBERT/dataloader.py", line 4, in
from transformers import BertTokenizer
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/transformers/init.py", line 25, in
from .data import (is_sklearn_available,
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/transformers/data/init.py", line 6, in
from .metrics import is_sklearn_available
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/transformers/data/metrics/init.py", line 25, in
from sklearn.metrics import matthews_corrcoef, f1_score
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/sklearn/init.py", line 64, in
from .base import clone
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/sklearn/base.py", line 14, in
from .utils.fixes import signature
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/sklearn/utils/init.py", line 14, in
from . import _joblib
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/sklearn/utils/_joblib.py", line 22, in
from ..externals import joblib
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/sklearn/externals/joblib/init.py", line 119, in
from .parallel import Parallel
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/sklearn/externals/joblib/parallel.py", line 28, in
from ._parallel_backends import (FallbackToBackend, MultiprocessingBackend,
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 22, in
from .executor import get_memmapping_executor
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/sklearn/externals/joblib/executor.py", line 14, in
from .externals.loky.reusable_executor import get_reusable_executor
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/sklearn/externals/joblib/externals/loky/init.py", line 12, in
from .backend.reduction import set_loky_pickler
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/sklearn/externals/joblib/externals/loky/backend/reduction.py", line 125, in
from sklearn.externals.joblib.externals import cloudpickle # noqa: F401
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/sklearn/externals/joblib/externals/cloudpickle/init.py", line 3, in
from .cloudpickle import *
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py", line 167, in
_cell_set_template_code = _make_cell_set_template_code()
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py", line 148, in _make_cell_set_template_code
return types.CodeType(
TypeError: an integer is required (got type bytes)

Expected behavior

Additional context
python: 3.8
OS: ubuntu 18.04
Tried with other NLU modules, same errors occured.

[zqwerty]

it seems like there is inconsistency in the python environment, specifically, the Transformers package.

File "/home/lincoln/deep-learning/ConvLab-2/convlab2/nlu/jointBERT/dataloader.py", line 4, in
from transformers import BertTokenizer
File "/home/lincoln/anaconda3/envs/convlab/lib/python3.8/site-packages/transformers/init.py", line 25, in
from .data import (is_sklearn_available,

We've tested on python 3.6&3.7. You could try to use this version.

[wufangjie]

I meet this error too, and I fixed this by uncommenting the scikit-learn's version in setup.py

see: joblib/joblib#917

[zqwerty]

Thanks for the information!