Skip to content
This repository has been archived by the owner on Nov 22, 2022. It is now read-only.

Specify CFLAGS to install fairseq in MacOS #1175

Closed
wants to merge 2 commits into from

Conversation

hudeven
Copy link
Contributor

@hudeven hudeven commented Nov 30, 2019

Summary:
Currently, we installs fairseq by requirements.txt which is equivalent to pip install fairseq

According to https://github.com/pytorch/fairseq, to install fairseq in MacOS, we need to run
CFLAGS="-stdlib=libc++" pip install fairseq

Before this diff:
Can't install PyText in MacOS by
$ ./install_deps

Installing collected packages: click, pycparser, cffi, fastBPE, numpy, regex, portalocker, typing, sacrebleu, torch, tqdm, fairseq, attrs, hypothesis, joblib, typing-extensions, six, setuptools, protobuf, onnx, python-dateutil, certifi, urllib3, idna, chardet, requests, jmespath, docutils, botocore, s3transfer, boto3, pytorch-pretrained-bert, pytz, pandas, scipy, torchtext, wheel, grpcio, werkzeug, markdown, absl-py, tensorboard, sentencepiece, pytext-nlp
Running setup.py install for pycparser ... done
Running setup.py install for fastBPE ... error
ERROR: Command errored out with exit status 1:
command: /Users/stevenliu/workspace/sphinx/pytext/pytext_venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sg/jcdjt2vn6pndx_d0vmf0l2n15y5jp7/T/pip-install-x0x9inoi/fastBPE/setup.py'"'"'; file='"'"'/private/var/folders/ sg/jcdjt2vn6pndx_d0vmf0l2n15y5jp7/T/pip-install-x0x9inoi/fastBPE/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/sg/jcdjt2vn6pndx_d0vmf0l2n15y5jp7/T/pip-record-1kds02_a/install-record.txt --single-version-externally-managed --compile --install-headers /Users/stevenliu/workspace/sphinx/pytext/pytext_venv/include/site/python3.7/fastBPE
cwd: /private/var/folders/sg/jcdjt2vn6pndx_d0vmf0l2n15y5jp7/T/pip-install-x0x9inoi/fastBPE/
Complete output (16 lines):
running install
running build
running build_py
package init file 'fastBPE/init.py' not found (or not a regular file)
running build_ext
building 'fastBPE' extension
creating build
creating build/temp.macosx-10.7-x86_64-3.7
creating build/temp.macosx-10.7-x86_64-3.7/fastBPE
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/stevenliu/ anaconda3/include -arch x86_64 -I/Users/stevenliu/anaconda3/include -arch x86_64 -I/Users/stevenliu/workspace/sphinx/pytext/pytext_venv/ include -I/Users/stevenliu/anaconda3/include/python3.7m -c fastBPE/fastBPE.cpp -o build/temp.macosx-10.7-x86_64-3.7/fastBPE/fastBPE.o -std=c++11 -Ofast -pthread
warning: include path for stdlibc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
fastBPE/fastBPE.cpp:603:10: fatal error: 'ios' file not found
#include "ios"
^~~~~
1 warning and 1 error generated.
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /Users/stevenliu/workspace/sphinx/pytext/pytext_venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sg/jcdjt2vn6pndx_d0vmf0l2n15y5jp7/T/pip-install-x0x9inoi/fastBPE/setup.py'"'"'; file='"'"'/private/var/folders/sg/jcdjt2vn6pndx_d0vmf0l2n15y5jp7/T/pip-install-x0x9inoi/fastBPE/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/sg/jcdjt2vn6pndx_d0vmf0l2n15y5jp7/T/pip-record-1kds02_a/install-record.txt --single-version-externally-managed --compile --install-headers /Users/stevenliu/workspace/sphinx/pytext/pytext_venv/include/site/python3.7/fastBPE Check the logs for full command output.

BTW, made activation_venv executable as install_deps

Differential Revision: D18753339

@facebook-github-bot facebook-github-bot added CLA Signed Do not delete this pull request or issue due to inactivity. fb-exported labels Nov 30, 2019
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D18753339

…ed-bert

Differential Revision: D18753252

fbshipit-source-id: 18bfe1d38d1afca4eb48a2e2e33ddc1b137aa047
Summary:
Pull Request resolved: facebookresearch#1175

Currently, we installs fairseq by requirements.txt which is equivalent to  ```pip install fairseq```

According to https://github.com/pytorch/fairseq, to install fairseq in MacOS, we need to run
```CFLAGS="-stdlib=libc++" pip install fairseq```

Before this diff:
Can't install PyText in MacOS by
$ ./install_deps

  Installing collected packages: click, pycparser, cffi, fastBPE, numpy, regex, portalocker, typing, sacrebleu, torch, tqdm, fairseq, attrs,   hypothesis, joblib, typing-extensions, six, setuptools, protobuf, onnx, python-dateutil, certifi, urllib3, idna, chardet, requests, jmespath,   docutils, botocore, s3transfer, boto3, pytorch-pretrained-bert, pytz, pandas, scipy, torchtext, wheel, grpcio, werkzeug, markdown, absl-py,   tensorboard, sentencepiece, pytext-nlp
      Running setup.py install for pycparser ... done
      Running setup.py install for fastBPE ... error
      ERROR: Command errored out with exit status 1:
       command: /Users/stevenliu/workspace/sphinx/pytext/pytext_venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] =   '"'"'/private/var/folders/sg/jcdjt2vn6pndx_d0vmf0l2n15y5jp7/T/pip-install-x0x9inoi/fastBPE/setup.py'"'"'; __file__='"'"'/private/var/folders/  sg/jcdjt2vn6pndx_d0vmf0l2n15y5jp7/T/pip-install-x0x9inoi/fastBPE/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"',   open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install   --record /private/var/folders/sg/jcdjt2vn6pndx_d0vmf0l2n15y5jp7/T/pip-record-1kds02_a/install-record.txt --single-version-externally-managed   --compile --install-headers /Users/stevenliu/workspace/sphinx/pytext/pytext_venv/include/site/python3.7/fastBPE
           cwd: /private/var/folders/sg/jcdjt2vn6pndx_d0vmf0l2n15y5jp7/T/pip-install-x0x9inoi/fastBPE/
      Complete output (16 lines):
      running install
      running build
      running build_py
      package init file 'fastBPE/__init__.py' not found (or not a regular file)
      running build_ext
      building 'fastBPE' extension
      creating build
      creating build/temp.macosx-10.7-x86_64-3.7
      creating build/temp.macosx-10.7-x86_64-3.7/fastBPE
      gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/stevenliu/  anaconda3/include -arch x86_64 -I/Users/stevenliu/anaconda3/include -arch x86_64 -I/Users/stevenliu/workspace/sphinx/pytext/pytext_venv/  include -I/Users/stevenliu/anaconda3/include/python3.7m -c fastBPE/fastBPE.cpp -o build/temp.macosx-10.7-x86_64-3.7/fastBPE/fastBPE.o   -std=c++11 -Ofast -pthread
      warning: include path for stdlibc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead   [-Wstdlibcxx-not-found]
      fastBPE/fastBPE.cpp:603:10: fatal error: 'ios' file not found
      #include "ios"
               ^~~~~
      1 warning and 1 error generated.
      error: command 'gcc' failed with exit status 1
      ----------------------------------------
  ERROR: Command errored out with exit status 1: /Users/stevenliu/workspace/sphinx/pytext/pytext_venv/bin/python3 -u -c 'import sys, setuptools,   tokenize; sys.argv[0] = '"'"'/private/var/folders/sg/jcdjt2vn6pndx_d0vmf0l2n15y5jp7/T/pip-install-x0x9inoi/fastBPE/setup.py'"'"';   __file__='"'"'/private/var/folders/sg/jcdjt2vn6pndx_d0vmf0l2n15y5jp7/T/pip-install-x0x9inoi/fastBPE/setup.py'"'"';f=getattr(tokenize,   '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/sg/jcdjt2vn6pndx_d0vmf0l2n15y5jp7/T/pip-record-1kds02_a/install-record.txt --single-version-externally-managed --compile --install-headers /Users/stevenliu/workspace/sphinx/pytext/pytext_venv/include/site/python3.7/fastBPE Check the logs for full command output.

BTW, made activation_venv executable as install_deps

Differential Revision: D18753339

fbshipit-source-id: 9507101610de0f01fef5c766a315cf3269068565
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D18753339

@facebook-github-bot
Copy link
Contributor

This pull request has been merged in 5cb79b7.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
CLA Signed Do not delete this pull request or issue due to inactivity. fb-exported Merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants