Installing on Princeton Della issues

[iparask]

I tried to install SO tools on Della using soconda.sh. The loaded modules are:

(base) [ip8725@della8 soconda]$ module list
Currently Loaded Modulefiles:
 1) anaconda3/2023.9   2) intel-tbb/2021.7.0   3) intel-rt/2022.2.0   4) intel-mkl/2022.2.0   5) intel/2022.2.0

To install I did ./soconda.sh -e $SCRATCH/conda_envs. The base Anaconda environment was loaded. There are few issues that occurred.

It could not find a set of executables such as conda-index, pip, conda-build etc.

It gave an error for unreachable channel:

UnavailableInvalidChannel: HTTP 404 UNAVAILABLE OR INVALID for channel conda-bld <file:///scratch/gpfs/ip8725/conda_envs/soconda_0.0.5.dev78/conda-bld>

And last it was trying to create a python3.6 under /usr/lib when trying to install quaternionarray:

Collecting quaternionarray
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/27/1b/4f2bcfd1bf3d78173cf2c05d79b3c129ed789e99e072c220bf5041b31076/quaternionarray-0.6.2.tar.gz
Installing collected packages: quaternionarray
  Running setup.py install for quaternionarray: started
    Running setup.py install for quaternionarray: finished with status 'error'
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-e54fzury/quaternionarray/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-59_7rwzr-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/quaternionarray
    copying quaternionarray/__init__.py -> build/lib/quaternionarray
    copying quaternionarray/test_quaternionarray.py -> build/lib/quaternionarray
    running install_lib
    creating /usr/local/lib/python3.6
    error: could not create '/usr/local/lib/python3.6': Permission denied

    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-e54fzury/quaternionarray/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-59_7rwzr-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-e54fzury/quaternionarray/
Checking dependencies for package "pysm3"
./soconda.sh: line 226: pipgrip: command not found
Installing package pysm3
Collecting pysm3
  Downloading https://files.pythonhosted.org/packages/17/12/9f2b40d37df9b420004d994e4973d4661ee9bcf6821051ce97b54e74ee77/pysm3-3.3.0-py3-none-any.whl (45.1MB)
Installing collected packages: pysm3
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3.6/site-packages/pip/commands/install.py", line 365, in run
    strip_file_prefix=options.strip_file_prefix,
  File "/usr/lib/python3.6/site-packages/pip/req/req_set.py", line 789, in install
    **kwargs
  File "/usr/lib/python3.6/site-packages/pip/req/req_install.py", line 854, in install
    strip_file_prefix=strip_file_prefix
  File "/usr/lib/python3.6/site-packages/pip/req/req_install.py", line 1069, in move_wheel_files
    strip_file_prefix=strip_file_prefix,
  File "/usr/lib/python3.6/site-packages/pip/wheel.py", line 345, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/usr/lib/python3.6/site-packages/pip/wheel.py", line 287, in clobber
    ensure_dir(dest)  # common for the 'include' path
  File "/usr/lib/python3.6/site-packages/pip/utils/__init__.py", line 83, in ensure_dir
    os.makedirs(path)
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6'

Should I provide the path to the base conda even if it is loaded or should I create an environment first that the script will use as the base?

[tskisner]

Thanks @iparask , I have mostly been testing with a conda-forge base env. I can test again with an Anaconda base. Also this problem might be fixed by #7 very soon.

[Bai-Chiang]

I'm using following steps to set up a SO environment on my local machine.

Creating an empty conda environment

conda create --no-default-packages -n soconda-test
conda activate soconda-test

If conda command not found, make sure anaconda/conda is loaded.

Then clone and run this script. This script should run with soconda-test environment activated, it does not create new environment.

This script installs all packages (like sotodlib so3g etc), but won't setup modules and jupyter kernel stuff, since I don't need these on my local machine. And whether you are using base channel or conda-forge in base environment doesn't matter, the script will create a config file in new environment to force using conda-forge.

[iparask]

@Bai-Chiang, I will try the script you are proposing and see. Another issue I faced had to do with the MPI version I loaded. But I think that is a question for the machine admins.

I will post the steps somewhere.

@tskisner do you think that a PR with instructions on how to create the environments on Della and Tiger at Princeton will be useful?

[iparask]

The next issue I face is this:

 /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev83/conda-bld/pixell_1697734606021/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.10/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /scratch/gpfs/ip8725/conda_envs/rp_env_0.0.5.dev83/conda-bld/pixell_1697734606021/work/cython/srcsim.pyx
    tree = Parsing.p_module(s, pxd, full_module_name)

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
        cdef int nx = map.shape[2]
        # I can't figure out how to make a memoryview of an array of pointers, so let's just do
        # things brute force
        cdef float ** map_ = <float**>calloc(ncomp, sizeof(float*));
        for i in range(ncomp):
                map_[i] = &map[i,0,0]
                  ^
  ------------------------------------------------------------

  srcsim.pyx:42:16: Compiler crash in AnalyseExpressionsTransform

  ModuleNode.body = StatListNode(srcsim.pyx:1:0)

Do you think it has to do with pixels or Della specifically?

[tskisner]

Hi @iparask , sorry for slow response today, am slightly sick. I have not yet tested with Anaconda base, will try to do that soon.

[Bai-Chiang]

Related cython/cython#5771 and simonsobs/pixell#235 (comment)

cython=3.0.4 is released to conda-forge yesterday. Upstream now limit cython to version 3.0.3

Could you try setting cython=3.0.3 in packages_conda.txt and pkgs/pixell/meta.yml?

[iparask]

@Bai-Chiang, that did the trick for Della. I'll close this issue. Thank you.