Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support anaconda python distribution(s) #7413

Open
ben-albrecht opened this issue Sep 20, 2017 · 6 comments
Open

Support anaconda python distribution(s) #7413

ben-albrecht opened this issue Sep 20, 2017 · 6 comments
Labels
area: BTR area: Third-Party type: Feature Request

Comments

@ben-albrecht
Copy link
Member

Add support and testing for anaconda python distribution(s), such as miniconda.

miniconda has been shown to not work in a couple of cases, due to virtualenv issues:

Once supported, we should include an anaconda distribution in our python testing (#7206)
@ben-albrecht ben-albrecht mentioned this issue Sep 20, 2017
Closed
@ben-albrecht
Copy link
Member Author

Note that anaconda is a popular python distribution in HPC (especially on Crays), so I'd strongly advocate for supporting it.

@ronawho
Copy link
Contributor

ronawho commented Nov 29, 2018

#9102 may have helped with this.

With #11728, I'm able to build/run with miniconda-latest (based on python 2.7.15) and miniconda3-latest (based on python 3.7.1). I was able to build test-venv and chpldoc and run release/examples

That said, older versions of miniconda2 still fail for me while trying to build the test-venv. The last 2 or 3 versions of miniconda3 work for me, but older versions also run into test-venv build issues.

@ben-albrecht
Copy link
Member Author

I just tested make docs and make test-venv with a few cases that caused failures in the past. Here are my findings:

✅ miniconda (latest: Python 3.8.5, conda 4.8.5)

✅ miniconda inside a conda environment (latest: Python 3.8.5, conda 4.8.5)

❌ Anaconda3-2020.11 installed via pyenv

Build output:

> make chpldoc
...
cd chpl-venv && /Library/Developer/CommandLineTools/usr/bin/make chpldoc-venv
python3 -m venv $CHPL_HOME/projects/chapel/master/third-party/chpl-venv/build/build-venv
export PATH="$CHPL_HOME/projects/chapel/master/third-party/chpl-venv/build/build-venv/bin:$PATH" && \
        export VIRTUAL_ENV=$CHPL_HOME/projects/chapel/master/third-party/chpl-venv/build/build-venv && \
        python3 -m pip install \
        --upgrade   wheel && \
        touch $CHPL_HOME/projects/chapel/master/third-party/chpl-venv/build/build-venv/ok
Collecting wheel
  Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel
Successfully installed wheel-0.36.2
WARNING: You are using pip version 20.1.1; however, version 21.0.1 is available.
You should consider upgrading via the '$CHPL_HOME/projects/chapel/master/third-party/chpl-venv/build/build-venv/bin/python3 -m pip install --upgrade pip' command.
export PATH="$CHPL_HOME/projects/chapel/master/third-party/chpl-venv/build/build-venv/bin:$PATH" && \
        export VIRTUAL_ENV=$CHPL_HOME/projects/chapel/master/third-party/chpl-venv/build/build-venv && \
        python3 -m pip install --upgrade   \
          --target $CHPL_HOME/projects/chapel/master/third-party/chpl-venv/install/chpldeps \
          -r $CHPL_HOME/projects/chapel/master/third-party/chpl-venv/test-requirements.txt \
          -r $CHPL_HOME/projects/chapel/master/third-party/chpl-venv/chpldoc-requirements.txt \
          -r $CHPL_HOME/projects/chapel/master/third-party/chpl-venv/c2chapel-requirements.txt && \
        cp chpldeps-main.py $CHPL_HOME/projects/chapel/master/third-party/chpl-venv/install/chpldeps/__main__.py
Processing $CHPL_HOME/Library/Caches/pip/wheels/a7/c1/ea/cf5bd31012e735dc1dfea3131a2d5eae7978b251083d6247bd/PyYAML-5.3.1-cp38-cp38-macosx_10_14_x86_64.whl
Collecting filelock==3.0.12
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting argcomplete==1.12.1
  Using cached argcomplete-1.12.1-py2.py3-none-any.whl (38 kB)
Collecting Jinja2==2.11.2
  Using cached Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting MarkupSafe==1.1.1
  Using cached MarkupSafe-1.1.1-cp38-cp38-macosx_10_9_x86_64.whl (16 kB)
Collecting Pygments==2.7.1
  Using cached Pygments-2.7.1-py3-none-any.whl (944 kB)
Collecting Sphinx==3.2.1
  Using cached Sphinx-3.2.1-py3-none-any.whl (2.9 MB)
Collecting docutils==0.16
  Using cached docutils-0.16-py2.py3-none-any.whl (548 kB)
Collecting sphinx-rtd-theme==0.5.0
  Using cached sphinx_rtd_theme-0.5.0-py2.py3-none-any.whl (10.8 MB)
Collecting sphinxcontrib-chapeldomain==0.0.19
  Using cached sphinxcontrib_chapeldomain-0.0.19-py3-none-any.whl (19 kB)
Collecting babel==2.8.0
  Using cached Babel-2.8.0-py2.py3-none-any.whl (8.6 MB)
Processing $CHPL_HOME/Library/Caches/pip/wheels/3f/a9/a7/44a27bb67d962845df8df4ac988e1f369184e307392ce37c5b/pycparser-2.17-py2.py3-none-any.whl
Collecting sphinxcontrib-devhelp
  Using cached sphinxcontrib_devhelp-1.0.2-py2.py3-none-any.whl (84 kB)
Collecting sphinxcontrib-applehelp
  Using cached sphinxcontrib_applehelp-1.0.2-py2.py3-none-any.whl (121 kB)
Collecting snowballstemmer>=1.1
  Using cached snowballstemmer-2.1.0-py2.py3-none-any.whl (93 kB)
Collecting sphinxcontrib-serializinghtml
  Using cached sphinxcontrib_serializinghtml-1.1.4-py2.py3-none-any.whl (89 kB)
Collecting sphinxcontrib-jsmath
  Using cached sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl (5.1 kB)
Collecting alabaster<0.8,>=0.7
  Using cached alabaster-0.7.12-py2.py3-none-any.whl (14 kB)
Collecting setuptools
  Using cached setuptools-53.0.0-py3-none-any.whl (784 kB)
Collecting sphinxcontrib-qthelp
  Using cached sphinxcontrib_qthelp-1.0.3-py2.py3-none-any.whl (90 kB)
Collecting imagesize
  Using cached imagesize-1.2.0-py2.py3-none-any.whl (4.8 kB)
Collecting requests>=2.5.0
  Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting packaging
  Using cached packaging-20.9-py2.py3-none-any.whl (40 kB)
Collecting sphinxcontrib-htmlhelp
  Using cached sphinxcontrib_htmlhelp-1.0.3-py2.py3-none-any.whl (96 kB)
Collecting six
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting pytz>=2015.7
  Using cached pytz-2021.1-py2.py3-none-any.whl (510 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<5,>=3.0.2
  Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Installing collected packages: PyYAML, filelock, argcomplete, MarkupSafe, Jinja2, Pygments, sphinxcontrib-devhelp, sphinxcontrib-applehelp, snowballstemmer, sphinxcontrib-serializinghtml, sphinxcontrib-jsmath, alabaster, setuptools, sphinxcontrib-qthelp, imagesize, docutils, certifi, urllib3, idna, chardet, requests, pyparsing, packaging, pytz, babel, sphinxcontrib-htmlhelp, Sphinx, sphinx-rtd-theme, six, sphinxcontrib-chapeldomain, pycparser
Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 PyYAML-5.3.1 Pygments-2.7.1 Sphinx-3.2.1 alabaster-0.7.12 argcomplete-1.12.1 babel-2.8.0 certifi-2020.12.5 chardet-4.0.0 docutils-0.16 filelock-3.0.12 idna-2.10 imagesize-1.2.0 packaging-20.9 pycparser-2.17 pyparsing-2.4.7 pytz-2021.1 requests-2.25.1 setuptools-53.0.0 six-1.15.0 snowballstemmer-2.1.0 sphinx-rtd-theme-0.5.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-chapeldomain-0.0.19 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-1.0.3 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.4 urllib3-1.26.3
WARNING: You are using pip version 20.1.1; however, version 21.0.1 is available.
You should consider upgrading via the '$CHPL_HOME/projects/chapel/master/third-party/chpl-venv/build/build-venv/bin/python3 -m pip install --upgrade pip' command.
cd compiler && /Library/Developer/CommandLineTools/usr/bin/make chpldoc
make[1]: Nothing to be done for `chpldoc'.
cd chpl-venv && /Library/Developer/CommandLineTools/usr/bin/make chpldoc-venv
make[3]: Nothing to be done for `chpldoc-venv'.
cd man && /Library/Developer/CommandLineTools/usr/bin/make chpldoc
make[2]: Nothing to be done for `chpldoc'.
cd man && /Library/Developer/CommandLineTools/usr/bin/make
***** ./ *****
mkdir -p man1
sed "/confchpl.rst/r confchpl.rst" chpl.rst > chpl.rst.tmp
$CHPL_HOME/.pyenv/shims/rst2man.py chpl.rst.tmp > chpl.rst.tmp.out
mv chpl.rst.tmp.out man1/chpl.1
rm -f chpl.rst.tmp
cd man && /Library/Developer/CommandLineTools/usr/bin/make chpldoc
make[1]: Nothing to be done for `chpldoc'.
cd doc && /Library/Developer/CommandLineTools/usr/bin/make docs

Confirming that sphinx-build is available...
OK

Collecting syntax from 'rst/language/spec' into 'rst/language/spec/syntax.rst'

Generating module docs from $CHPL_HOME/modules/ into rst/modules
mkdir -p standard
cd standard && python3 $CHPL_HOME/projects/chapel/master/util/config/make_sys_basic_types.py --doc SysCTypes.chpl
Generating module documentation with chpldoc
Running post-processing scripts
done
Copying generated module documentation to $CHPL_HOME/projects/chapel/master/build/doc
Updating generated module documentation in $CHPL_HOME/projects/chapel/master/doc/rst
Copying $CHPL_HOME/projects/chapel/master/build/doc/modules/packages/AtomicObjects.rst to $CHPL_HOME/projects/chapel/master/doc/rst/modules/packages/AtomicObjects.rst
Copying $CHPL_HOME/projects/chapel/master/build/doc/modules/packages/Sort.rst to $CHPL_HOME/projects/chapel/master/doc/rst/modules/packages/Sort.rst
Copying $CHPL_HOME/projects/chapel/master/build/doc/modules/packages/UnrolledLinkedList.rst to $CHPL_HOME/projects/chapel/master/doc/rst/modules/packages/UnrolledLinkedList.rst
Copying $CHPL_HOME/projects/chapel/master/build/doc/modules/standard/Heap.rst to $CHPL_HOME/projects/chapel/master/doc/rst/modules/standard/Heap.rst
Copying $CHPL_HOME/projects/chapel/master/build/doc/modules/standard/Path.rst to $CHPL_HOME/projects/chapel/master/doc/rst/modules/standard/Path.rst
Copying $CHPL_HOME/projects/chapel/master/build/doc/modules/standard/Sys.rst to $CHPL_HOME/projects/chapel/master/doc/rst/modules/standard/Sys.rst
Copying $CHPL_HOME/projects/chapel/master/build/doc/builtins/ChapelArray.rst to $CHPL_HOME/projects/chapel/master/doc/rst/builtins/ChapelArray.rst
Copying $CHPL_HOME/projects/chapel/master/build/doc/builtins/OwnedObject.rst to $CHPL_HOME/projects/chapel/master/doc/rst/builtins/OwnedObject.rst
Removing generated files and directories in modules/

Generating primers from $CHPL_HOME/test/release/examples to rst/primers
Copying ../build/doc/primers/parIters.rst to rst/primers/parIters.rst

Generating hellos from $CHPL_HOME/test/release/examples to rst/examples
Creating directory:  ../build/doc/examples/hellos

Creating symlinks

Building HTML documentation
python3 $CHPL_HOME/projects/chapel/master/third-party/chpl-venv/install/chpldeps sphinx-build -b html -d ../build/doc/doctrees  -W -n rst ../build/doc/html
Running Sphinx v3.2.1

Extension error:
Could not import extension sphinxcontrib.chapeldomain (exception: No module named 'sphinxcontrib.chapeldomain')
make[2]: *** [html-release] Error 2
make[2]: *** [error_docs] Error 1
make[1]: *** [docs] Error 2
make: *** [docs] Error 2

@bradcray
Copy link
Member

bradcray commented Feb 2, 2021

Thanks for taking a look after the discussion today, @ben-albrecht!

Cutting to the chase, the problematic error is:

Extension error:
Could not import extension sphinxcontrib.chapeldomain (exception: No module named 'sphinxcontrib.chapeldomain')

The mention of sphinxcontrib.domain makes me want to tag @e-kayrakli and @lydia-duncan who've touched that tech recently (where my main curiosity is whether it's clear whether this is a simple fix or not).

@lydia-duncan
Copy link
Member

That seems like it would be a simple thing that's been overlooked when making the Sphinx domain but without digging into how python packages I don't know what the fix is off the top of my head.

@lydia-duncan
Copy link
Member

It does seem like the situation has improved since last we looked

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area: BTR area: Third-Party type: Feature Request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ronawho @lydia-duncan @ben-albrecht @bradcray