Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python3Packages.transformers: 2.2.1 -> 3.0.1 #91915

Merged
merged 2 commits into from Jul 3, 2020

Conversation

@danieldk
Copy link
Member

danieldk commented Jul 1, 2020

Motivation for this change

Changelog:
https://github.com/huggingface/transformers/releases/tag/v3.0.0

Also add myself as a maintainer.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

cc: @pashashocky

@danieldk
Copy link
Member Author

danieldk commented Jul 1, 2020

@ofborg build python37Packages.transformers python38Packages.transformers

Copy link
Contributor

drewrisinger left a comment

Suggest using pytestCheckHook for disabling these tests.

@danieldk danieldk force-pushed the danieldk:transformers-3.0.0 branch from dfb585b to 346f75e Jul 2, 2020
@danieldk danieldk requested a review from drewrisinger Jul 2, 2020
Copy link
Contributor

drewrisinger left a comment

  • Diff LGTM
  • Commits: Squash add maintainer. Remove tokenizers when #91342 is approved
  • Builds cleanly
  • DOESN'T run. See log below:

python38.transformers: Willing to accept, maybe include pytorch/tensorflow in inputs?? No strong opinion. (also, the spacing in this error message makes me want to file a PR... grr :D )

>>> import transformers
Neither PyTorch nor TensorFlow >= 2.0 have been found.Models won't be available and only tokenizers, configurationand file/data utilities can be used.

python37Packages.transformers: getting the same issue when launched from command-line or via transformers-cli. Not sure what's causing this, seems like it's trying to use py38.numpy for some reason??

[nix-shell:~/.cache/nixpkgs-review/pr-91915/results/python37Packages.transformers]$ ./bin/transformers-cli
Traceback (most recent call last):
  File "/nix/store/0ryfy7j7azg5sgnh75sxlxbkbnd9vvbj-python3.8-numpy-1.18.5/lib/python3.8/site-packages/numpy/core/__init__.py", line 24, in <module>
    from . import multiarray
  File "/nix/store/0ryfy7j7azg5sgnh75sxlxbkbnd9vvbj-python3.8-numpy-1.18.5/lib/python3.8/site-packages/numpy/core/multiarray.py", line 14, in <module>
    from . import overrides
  File "/nix/store/0ryfy7j7azg5sgnh75sxlxbkbnd9vvbj-python3.8-numpy-1.18.5/lib/python3.8/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/rsxk262fy8xv1dsmpqb6klbw4sf1q4hf-python3.7-transformers-3.0.0/bin/.transformers-cli-wrapped", line 6, in <module>
    from transformers.commands.transformers_cli import main
  File "/nix/store/8ndbww1p3x936vjcxrjawa112wkvgc15-python3.8-transformers-3.0.0/lib/python3.8/site-packages/transformers/__init__.py", line 48, in <module>
    from .data import (
  File "/nix/store/8ndbww1p3x936vjcxrjawa112wkvgc15-python3.8-transformers-3.0.0/lib/python3.8/site-packages/transformers/data/__init__.py", line 6, in <module>
    from .processors import (
  File "/nix/store/8ndbww1p3x936vjcxrjawa112wkvgc15-python3.8-transformers-3.0.0/lib/python3.8/site-packages/transformers/data/processors/__init__.py", line 5, in <module>
    from .glue import glue_convert_examples_to_features, glue_output_modes, glue_processors, glue_tasks_num_labels
  File "/nix/store/8ndbww1p3x936vjcxrjawa112wkvgc15-python3.8-transformers-3.0.0/lib/python3.8/site-packages/transformers/data/processors/glue.py", line 24, in <module>
    from ...tokenization_utils import PreTrainedTokenizer
  File "/nix/store/8ndbww1p3x936vjcxrjawa112wkvgc15-python3.8-transformers-3.0.0/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 26, in <module>
    from .tokenization_utils_base import (
  File "/nix/store/8ndbww1p3x936vjcxrjawa112wkvgc15-python3.8-transformers-3.0.0/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 30, in <module>
    import numpy as np
  File "/nix/store/0ryfy7j7azg5sgnh75sxlxbkbnd9vvbj-python3.8-numpy-1.18.5/lib/python3.8/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/nix/store/0ryfy7j7azg5sgnh75sxlxbkbnd9vvbj-python3.8-numpy-1.18.5/lib/python3.8/site-packages/numpy/core/__init__.py", line 50, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.7 from "/nix/store/73iskwap4wblkwbq2jy9cndmrn49n6ss-python3-3.7.7/bin/python3.7"
  * The NumPy version is: "1.18.5"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'
@danieldk
Copy link
Member Author

danieldk commented Jul 2, 2020

* [ ]  Commits: Squash add maintainer. Remove tokenizers when #91342 is approved

I think it's nice to have that in a commit separate from the version bump.

python38.transformers: Willing to accept, maybe include pytorch/tensorflow in inputs?? No strong opinion. (also, the

@pashashocky probably excluded these dependencies intentionally. transformers works with either, so I think it's up to the user to pick one. We use transformers with PyTorch and it would be nasty if we had to pull in Tensorflow as an additional dependency every time. Also Tensorflow has had quite a history of being broken frequently in nixpkgs.

spacing in this error message makes me want to file a PR... grr :D )

😆

python37Packages.transformers: getting the same issue when launched from command-line or via transformers-cli. Not sure what's causing this, seems like it's trying to use py38.numpy for some reason??

Let me check.

@danieldk
Copy link
Member Author

danieldk commented Jul 2, 2020

python37Packages.transformers: getting the same issue when launched from command-line or via transformers-cli. Not sure what's causing this, seems like it's trying to use py38.numpy for some reason??

Let me check.

The command seems to work fine?

> $(nix-build --no-out-link -A python3Packages.transformers)/bin/transformers-cli
Neither PyTorch nor TensorFlow >= 2.0 have been found.Models won't be available and only tokenizers, configurationand file/data utilities can be used.
usage: transformers-cli <command> [<args>]

positional arguments:
  {convert,download,env,run,serve,login,whoami,logout,s3,upload}
[...]
Copy link
Contributor

drewrisinger left a comment

Confirm that everything looks fine & builds. Approved pending removal of python3Packages.tokenizers commit from this PR.

@danieldk danieldk force-pushed the danieldk:transformers-3.0.0 branch from 346f75e to e516bdc Jul 2, 2020
@danieldk
Copy link
Member Author

danieldk commented Jul 2, 2020

Rebased now that python3Packages.tokenizers is merged.

Copy link
Contributor

drewrisinger left a comment

  • Diff LGTM
  • Commits LGTM, minus subjective style of adding maintainer in separate commit
  • Builds clean via nix-review
  • tested via pytest & trivially w/ transformer-cli (same issue with python37Packages.transformers-cli as before, maybe re-test with that command & python37Packges.transformers in your test, vs python3Packages.transformers).
@danieldk
Copy link
Member Author

danieldk commented Jul 3, 2020

  • tested via pytest & trivially w/ transformer-cli (same issue with python37Packages.transformers-cli as before, maybe re-test with that command & python37Packges.transformers in your test, vs python3Packages.transformers).

Ah, I see the problem you are running into. If you run nix-review, the Python 3.8 modules end up in PYTHONPATH. Try to run nix-review as follows: nix-review pr 91915 -p python37Packages.transformers

Then transformers-cli should run as expected.

@danieldk danieldk force-pushed the danieldk:transformers-3.0.0 branch from e516bdc to 7a983c6 Jul 3, 2020
@danieldk danieldk changed the title python3Packages.transformers: 2.2.1 -> 3.0.0 python3Packages.transformers: 2.2.1 -> 3.0.1 Jul 3, 2020
@danieldk
Copy link
Member Author

danieldk commented Jul 3, 2020

Updated to transformers 3.0.1.

@FRidh FRidh merged commit 4855aa6 into NixOS:master Jul 3, 2020
14 checks passed
14 checks passed
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="7a983c6"; rev="7a983c6c208bff07f3cc0974c2fb70d9f2346d67"; } ./pkgs/t
Details
grahamcofborg-eval-lib-tests nix-build --arg pkgs import ./. {} ./lib/tests/release.nix
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="7a983c6"; rev="7a983c6c208bff07f3cc0974c2fb70d9f2346d67"; } ./nixos/
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="7a983c6"; rev="7a983c6c208bff07f3cc0974c2fb70d9f2346d67"; } ./nixos/
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="7a983c6"; rev="7a983c6c208bff07f3cc0974c2fb70d9f2346d67"; } ./nixos/
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="7a983c6"; rev="7a983c6c208bff07f3cc0974c2fb70d9f2346d67"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="7a983c6"; rev="7a983c6c208bff07f3cc0974c2fb70d9f2346d67"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="7a983c6"; rev="7a983c6c208bff07f3cc0974c2fb70d9f2346d67"; } ./pkgs/t
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
@danieldk danieldk deleted the danieldk:transformers-3.0.0 branch Jul 3, 2020
@drewrisinger
Copy link
Contributor

drewrisinger commented Jul 4, 2020

@danieldk yeah, I figured something like that was happening, but I didn't have time to debug it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.