Make use of anaconda.org/huggingface

[jayfurmanek]

Transformers now has an official conda channel at: http://anaconda.org/huggingface

Some notes:

• transformers itself is noarch so we could maybe stomp building it
• tokenizers is not noarch - so we'll need to continue building that for ppc64le at least
• there is also huggingface-datasets and sacremoses in the channel (both noarch)

the channel includes python_abi package, which is something that forge uses, so I'm curious if they include that here to be compatible with both defaults and forge - in which case maybe we steal that idea..

So let's make use of this. We could do something like the following:

• Remove the packages that already exist in the HuggingFace channel from transformers-env.yaml.
• Add the HuggingFace channel to the top of transformers-env.yaml.
• List transformers, and whichever other packages we care about as external dependencies in transformers-env.yaml.
• Investigate and Adjust any dependencies on our side to be compatible with the packages in this channel.

transformers 4.3.3 py_0
-----------------------
file name   : transformers-4.3.3-py_0.tar.bz2
name        : transformers
version     : 4.3.3
build       : py_0
build number: 0
size        : 1.7 MB
license     : Apache License 2.0
subdir      : noarch
url         : https://conda.anaconda.org/huggingface/noarch/transformers-4.3.3-py_0.tar.bz2
md5         : c765cb79a4283ff25d3ea9463b6a2e78
timestamp   : 2021-02-27 01:17:51 UTC
dependencies: 
  - dataclasses
  - filelock
  - numpy >=1.17
  - packaging
  - protobuf
  - python
  - regex !=2019.12.17
  - requests
  - sacremoses
  - tokenizers >=0.10.1,<0.11.0
  - tqdm >=4.27

It looks like there will be some missing dependencies (multiprocess, python-xxhash) for datasets. Forge likely has those and we can add them as remote recipes in the env file.

datasets 1.2.1 py_0
-------------------
file name   : datasets-1.2.1-py_0.tar.bz2
name        : datasets
version     : 1.2.1
build       : py_0
build number: 0
size        : 158 KB
license     : Apache License 2.0
subdir      : noarch
url         : https://conda.anaconda.org/huggingface/noarch/datasets-1.2.1-py_0.tar.bz2
md5         : a220db03876e20a8a96413d1bc07343b
timestamp   : 2021-01-14 15:02:04 UTC
dependencies: 
  - dataclasses
  - dill
  - multiprocess
  - numpy >=1.17
  - pandas
  - pip
  - pyarrow >=0.17.1
  - python
  - python-xxhash
  - requests >=2.19.0
  - tqdm >=4.27,<4.50.0

[jayfurmanek]

Official conda recipes are at:
https://github.com/huggingface/transformers/tree/master/.github/conda
https://github.com/huggingface/tokenizers/tree/master/.github/conda

[jayfurmanek]

also:

2. SentencePiece is removed from the required dependencies
The requirement on the SentencePiece dependency has been lifted from the setup.py. This is done so that we may have a 
channel on anaconda cloud without relying on conda-forge. This means that the tokenizers that depend on the SentencePiece 
library will not be available with a standard transformers installation.

[jayfurmanek]

This env file seems to work with a couple caveats:

channels:
  - defaults
  - HuggingFace

packages:
  - feedstock : rust-nightly      #[ppc64le]
  - feedstock : semantic         #[ppc64le]
  - feedstock : tokenizers       #[ppc64le]
  - feedstock : protobuf          #[ppc64le and python == '3.8']
  - feedstock : sentencepiece

external_dependencies:
  - sacremoses
#  - transformers

• tokenizers needs to be updated to version 0.9.4 or later since the versions of transformers in the huggingface channel
• currently, having transformers in external_dependencies will fail the conda env validation check on ppc64le since it depends on tokenizers and we have to build that on ppc64le
• adding in huggingface datasets is further complicated as it has a few dependencies that are resident in conda-forge. The huggingface channel has its own python_abi package in order to prevent pulling in a string of unintended packages from forge. We could do the same, but if conda-build finds python_abi in an enabled channel ("conda-forge"), it will skip it, so that one would have to be built separate.
• python-xxhash xxhash and multiprocess are in conda-forge for both x86-64 and ppc64le.

[cdeepali]

We decided to build transformers and tokenizers because import is not working with tokenizers from HuggingFace channel and transfomers-cli command also throws error.

ImportError: /lib64/libm.so.6: version GLIBC_2.29' not found `