Error specifying commission_alpha - TypeError: 'NoneType' object is not iterable

[valpasq]

Tried to run YATSM with commission_alpha=0.5 using config file /projectnb/landsat/projects/Massachusetts/p013r030/p013r030_config.yaml

YATSM:
    algorithm: "CCDCesque"
    prediction: "GLMNET_Lasso20"
    design_matrix: "1 + x + harm(x, 1) + harm(x, 2)"
    reverse: False
    commission_alpha: 0.05

But got the following error:

  File "/usr3/graduate/valpasq/venv/bin/yatsm", line 8, in <module>
    load_entry_point('yatsm==0.5.6-beta', 'console_scripts', 'yatsm')()
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 700, in __call__
    return self.main(*args, **kwargs)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 680, in main
    rv = self.invoke(ctx)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 1027, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 873, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/core.py", line 508, in invoke
    return callback(*args, **kwargs)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/click/decorators.py", line 16, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/cli/line.py", line 178, in line
    yatsm, cfg['YATSM']['commission_alpha'])
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/algorithms/postprocess.py", line 127, in commission_test
    for i_m, _m in enumerate(_models):
TypeError: 'NoneType' object is not iterable

[ceholden]

Fixed in development for v0.6.0. I'm going to yank all of the v0.5.0 releases when v0.6.0 comes out because there were so many bugs. In retrospect, I'm thinking of it as an off cycle "developer only" release that sorts out the bugs and API design for the publicly consumable v0.6.0 release.

You can either stick with your current install or try to upgrade to the development version of v0.6.0. If you do upgrade, there's some stuff that's changed that may make things incompatible with the previous release (mostly the estimator pickles). If you do a git pull, please make sure to run python setup.py clean to remove all built files from the v0.5.x release and then re-install.

See the section in the changelog for this release.

[valpasq]

I just did a git pull then tried to run python setup.py clean, but got the following error (including all of it just in case):

ERROR:root:Could not produce pickles from package "glment". Check if it is installed
Traceback (most recent call last):
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/regression/pickles/serialize.py", line 28, in <module>
    'glmnet_Lasso20.pkl': glmnet.Lasso(lambdas=20),
TypeError: Lasso() got an unexpected keyword argument 'lambdas'

INFO:root:Serializing estimators to pickles...
Traceback (most recent call last):
  File "setup.py", line 54, in <module>
    serialize_pickles.make_pickles()
  File "/usr3/graduate/valpasq/Documents/yatsm/yatsm/regression/pickles/serialize.py", line 63, in make_pickles
    jl.dump(obj, os.path.join(here, fname), compress=5)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 369, in dump
    pickler.dump(value)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 249, in save
    return Pickler.save(self, obj)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 249, in save
    return Pickler.save(self, obj)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 681, in _batch_setitems
    save(v)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 249, in save
    return Pickler.save(self, obj)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 249, in save
    return Pickler.save(self, obj)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 681, in _batch_setitems
    save(v)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/sklearn/externals/joblib/numpy_pickle.py", line 249, in save
    return Pickler.save(self, obj)
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/project/earth/packages/Python-2.7.5/lib/python2.7/pickle.py", line 753, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <function mad at 0x2c65398>: it's not the same object as yatsm.regression.robust_fit.mad

From the first line of the error (ERROR:root), looks like there might be a typo---glment instead of glmnet.

[ceholden]

Looks like there's some recursion going on with the second traceback, but FYI please always include in the future the entire traceback as you've done here. The tracebacks with CLI programs that use click can be quite long but there's some useful information toward the top usually.

Two problems:

1. Lasso() not accepting lambdas as a keyword argument. This is likely due to you not being in the correct Python environment, or not having the correct version of glmnet installed. Please try again in the virtualenv you setup as I can't reproduce it when in conda or in a venv based on python/2.7.5 module

2. As far as the second error related to pickling goes, I can't reproduce it locally or on Travis.CI using conda. My guess is that it's your Python environment on the cluster causing issues again, likely due to a very old version of numba. I was just able to reproduce the same thing with the python/2.7.5. module and I've added a version check to yatsm.accel to compensate.

[valpasq]

Pulled again and then took a closer look at the virtual environment, and even through I was using module load Python/2.7.5_nopath, glmnet was still coming from the site packages copy, not the virtual environment. I added unset PYTHONPATH back into my modules.sh file and it looks like I'm using the right glmnet now.

But I have two new errors.

First, when I try to run pip install -r requirements/all.txt, it looks like my numba is up-to-date now, but I get the following error:

(venv)[valpasq@geo yatsm]$ pip install -r requirements/all.txt 
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.8.0 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from -r requirements/common.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): scipy>=0.14.0 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from -r requirements/common.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): Cython>=0.20.1 in /usr3/graduate/valpasq/venv/lib/python2.7/site-packages (from -r requirements/common.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): statsmodels>=0.5.0 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from -r requirements/common.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): scikit-learn>=0.15.1 in /usr3/graduate/valpasq/venv/lib/python2.7/site-packages (from -r requirements/common.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): pandas>=0.16.0 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from -r requirements/common.txt (line 8))
Requirement already satisfied (use --upgrade to upgrade): patsy>=0.3.0 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from -r requirements/common.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): click>=4.0 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from -r requirements/common.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): click-plugins>=1.0 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from -r requirements/common.txt (line 12))
Requirement already satisfied (use --upgrade to upgrade): matplotlib>=1.4.2 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from -r requirements/common.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): palettable>=2.0.0 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from -r requirements/common.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): pyyaml>=3.11 in /usr3/graduate/valpasq/venv/lib/python2.7/site-packages (from -r requirements/common.txt (line 17))
Requirement already satisfied (use --upgrade to upgrade): rpy2>=2.5.6 in /usr3/graduate/valpasq/venv/lib/python2.7/site-packages (from -r requirements/pheno.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): numba>=0.22 in /usr3/graduate/valpasq/venv/lib/python2.7/site-packages (from -r requirements/accel.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): glmnet from git+git://github.com/ceholden/glmnet-python.git in /usr3/graduate/valpasq/venv/lib/python2.7/site-packages (from -r requirements/accel.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): scandir>=1.1 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from -r requirements/accel.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from pandas>=0.16.0->-r requirements/common.txt (line 8))
Requirement already satisfied (use --upgrade to upgrade): pytz>=2011k in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from pandas>=0.16.0->-r requirements/common.txt (line 8))
Requirement already satisfied (use --upgrade to upgrade): six>=1.4 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from matplotlib>=1.4.2->-r requirements/common.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): pyparsing>=1.5.6 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from matplotlib>=1.4.2->-r requirements/common.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): nose>=0.11.1 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from matplotlib>=1.4.2->-r requirements/common.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): mock in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from matplotlib>=1.4.2->-r requirements/common.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): singledispatch in /usr3/graduate/valpasq/venv/lib/python2.7/site-packages (from rpy2>=2.5.6->-r requirements/pheno.txt (line 2))
Downloading/unpacking llvmlite (from numba>=0.22->-r requirements/accel.txt (line 2))
  Running setup.py egg_info for package llvmlite

Requirement already satisfied (use --upgrade to upgrade): enum34 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from numba>=0.22->-r requirements/accel.txt (line 2))
Downloading/unpacking funcsigs (from numba>=0.22->-r requirements/accel.txt (line 2))
  Running setup.py egg_info for package funcsigs

Installing collected packages: llvmlite, funcsigs
  Running setup.py install for llvmlite
    got version from file /usr3/graduate/valpasq/venv/build/llvmlite/llvmlite/_version.py {'version': '0.8.0', 'full': 'cb57c3dc90efd291822e07839caee2cfebfff5d4'}
    /usr3/graduate/valpasq/venv/bin/python /usr3/graduate/valpasq/venv/build/llvmlite/ffi/build.py
    LLVM version... Traceback (most recent call last):
      File "/usr3/graduate/valpasq/venv/build/llvmlite/ffi/build.py", line 114, in <module>
        main()
      File "/usr3/graduate/valpasq/venv/build/llvmlite/ffi/build.py", line 104, in main
        main_posix('linux', '.so')
      File "/usr3/graduate/valpasq/venv/build/llvmlite/ffi/build.py", line 93, in main_posix
        "to the path for llvm-config" % (llvm_config,))
    RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config
    error: command '/usr3/graduate/valpasq/venv/bin/python' failed with exit status 1
    Complete output from command /usr3/graduate/valpasq/venv/bin/python -c "import setuptools;__file__='/usr3/graduate/valpasq/venv/build/llvmlite/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /scratch/pip-6DQaU7-record/install-record.txt --single-version-externally-managed --install-headers /usr3/graduate/valpasq/venv/include/site/python2.7:
    running install

running build

got version from file /usr3/graduate/valpasq/venv/build/llvmlite/llvmlite/_version.py {'version': '0.8.0', 'full': 'cb57c3dc90efd291822e07839caee2cfebfff5d4'}

running build_ext

/usr3/graduate/valpasq/venv/bin/python /usr3/graduate/valpasq/venv/build/llvmlite/ffi/build.py

LLVM version... Traceback (most recent call last):

  File "/usr3/graduate/valpasq/venv/build/llvmlite/ffi/build.py", line 114, in <module>

    main()

  File "/usr3/graduate/valpasq/venv/build/llvmlite/ffi/build.py", line 104, in main

    main_posix('linux', '.so')

  File "/usr3/graduate/valpasq/venv/build/llvmlite/ffi/build.py", line 93, in main_posix

    "to the path for llvm-config" % (llvm_config,))

RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config

error: command '/usr3/graduate/valpasq/venv/bin/python' failed with exit status 1

----------------------------------------
Command /usr3/graduate/valpasq/venv/bin/python -c "import setuptools;__file__='/usr3/graduate/valpasq/venv/build/llvmlite/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /scratch/pip-6DQaU7-record/install-record.txt --single-version-externally-managed --install-headers /usr3/graduate/valpasq/venv/include/site/python2.7 failed with error code 1 in /usr3/graduate/valpasq/venv/build/llvmlite
Storing complete log in /usr3/graduate/valpasq/.pip/pip.log

Despite this error, I tried cleaning and re-installing yatsm using `pip install -e .``, and get the following error.

(venv)[valpasq@geo yatsm]$ pip install -e .
Obtaining file:///usr3/graduate/valpasq/Documents/yatsm
  Running setup.py egg_info for package from file:///usr3/graduate/valpasq/Documents/yatsm

Requirement already satisfied (use --upgrade to upgrade): numpy in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): scipy in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): Cython in /usr3/graduate/valpasq/venv/lib/python2.7/site-packages (from yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): statsmodels in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): scikit-learn in /usr3/graduate/valpasq/venv/lib/python2.7/site-packages (from yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): matplotlib in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): click in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): click-plugins in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): palettable in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): patsy in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): six>=1.4 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from matplotlib->yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from matplotlib->yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): pytz in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from matplotlib->yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): pyparsing>=1.5.6 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from matplotlib->yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): nose>=0.11.1 in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from matplotlib->yatsm==0.6.0dev)
Requirement already satisfied (use --upgrade to upgrade): mock in /project/earth/packages/Python-2.7.5/lib/python2.7/site-packages (from matplotlib->yatsm==0.6.0dev)
Installing collected packages: yatsm
  Running setup.py develop for yatsm
    Building estimator pickles
    INFO:root:Serializing estimators to pickles...
    INFO:root:Wrote pickles.json to yatsm/regression/pickles/pickles.json

    building 'yatsm._cyprep' extension
    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/numpy/core/include -I/project/earth/packages/Python-2.7.5/include/python2.7 -c yatsm/_cyprep.c -o build/temp.linux-x86_64-2.7/yatsm/_cyprep.o -O3
    gcc: yatsm/_cyprep.c: No such file or directory
    gcc: no input files
    error: command 'gcc' failed with exit status 1
    Complete output from command /usr3/graduate/valpasq/venv/bin/python -c "import setuptools; __file__='/usr3/graduate/valpasq/Documents/yatsm/setup.py'; exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop --no-deps:
    running develop

Building estimator pickles

INFO:root:Serializing estimators to pickles...

INFO:root:Wrote pickles.json to yatsm/regression/pickles/pickles.json

running egg_info

writing requirements to yatsm.egg-info/requires.txt

writing yatsm.egg-info/PKG-INFO

writing top-level names to yatsm.egg-info/top_level.txt

writing dependency_links to yatsm.egg-info/dependency_links.txt

writing entry points to yatsm.egg-info/entry_points.txt

warning: manifest_maker: standard file '-c' not found



reading manifest file 'yatsm.egg-info/SOURCES.txt'

writing manifest file 'yatsm.egg-info/SOURCES.txt'

running build_ext

building 'yatsm._cyprep' extension

creating build

creating build/temp.linux-x86_64-2.7

creating build/temp.linux-x86_64-2.7/yatsm

gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/project/earth/packages/Python-2.7.5/lib/python2.7/site-packages/numpy/core/include -I/project/earth/packages/Python-2.7.5/include/python2.7 -c yatsm/_cyprep.c -o build/temp.linux-x86_64-2.7/yatsm/_cyprep.o -O3

gcc: yatsm/_cyprep.c: No such file or directory

gcc: no input files

error: command 'gcc' failed with exit status 1

----------------------------------------
Command /usr3/graduate/valpasq/venv/bin/python -c "import setuptools; __file__='/usr3/graduate/valpasq/Documents/yatsm/setup.py'; exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop --no-deps failed with error code 1 in /usr3/graduate/valpasq/Documents/yatsm
Storing complete log in /usr3/graduate/valpasq/.pip/pip.log

I noticed I got this same error before the issue with the requirements. I looked in your /yatsm/yatsm/ directory on GitHub and I see /yatsm/_cyprep.c, but I don't see this file in my own copy. I tried pulling again, but everything is up to date.

Sorry for the ongoing issues. If it would be easier to troubleshoot in person, I'll be in the office all day tomorrow.

[ceholden]

Yeah good idea -- I'm not going to try to debug installation issues on Github again when I bet I could get it working in 5 minutes in person. The original issue is fixed so I'm closing the ticket.

FYI you can't upgrade Numba on the cluster without recompiling quite a lot of stuff because our gcc is so old.