Developer instructions are no longer valid

[rhyolight]

See original issue, which was filed against nupic.core: numenta/nupic.core-legacy#1069

Please continue discussion of this issue here.

[vitaly-krugl]

There are at least two developer workflows that come to mind (not to mention the anticipated Bambo-CI release/deployment workflow)

1. The nupic-only developer workflow
   • Clones nupic, installs it via python setup.py develop or similar, makes changes only in nupic
   • Relies on automatically getting the version of nupic.bindings from PyPi that's compatible with the cloned version of nupic.
2. The nupic.core/nupic.bindings developer workflow
   • Wants to make changes in nupic.core and possibly also in nupic; likely also wants the benefit of nupic's tests.
   • Needs to work with nupic.bindings version that's independent of the version that's pinned down in nupic

The ">=" solution proposed in PR #3325 would introduce subtle incompatibilities for the nupic-only developer workflow by pulling in newer builds of nupic.bindings from PyPi that are incompatible with nupic master that they cloned.

The premise of the new bamboo-ci-based workflow is that we would make more frequent nupic.bindings deployments to PyPi, including pre-releases, and update nupic's nupic.bindings dependency as needed.

My initial idea (TLDR that didn't pan out) for a solution was based on the wheel maintainers' recommendations in https://bitbucket.org/pypa/wheel/issues/172/wheel-doesnt-support-environment-markers as well as discussions in https://packaging.python.org/requirements/ and https://caremad.io/2013/07/setup-vs-requirement/, involving the separation of requirements into abstract/loosely-versioned dependencies in install_requires (arg to setup() in setup.py) and concrete requirements with pinned-down versions in requirements.txt. After a little experimentation, I couldn't find a way to build in the pinned-down requirements into the nupuic wheel without having those pinned-down requirements in install_requires. The pip wheel ... options that I was counting on did not have the desired effect. It looks like the construct proposed by those references only works only if you have access to the relevant requirements.txt, which implies that you're installing from the source tree.

Considering the two workflows described at the top of this comment, I am going to propose that we go with a simple workflow that doesn't involve any code changes, just documentation of the workflow:

1. nupic master will have a pinned-down exact (==) dependency on a specific version of nupic.bindings that's in PyPi (as it does at the moment).
   • Thus, installing nupic from source should always pull in a compatible version of nupic.bindings, addressing the nupic-only developer workflow, among others.
   • This avoids the incompatibility issue described in the nupic-only developer workflow.
2. The nupic.core/nupic.bindings developer who also needs to install nupic for modification and/or take advantage of nupic's various tests, would need to
   2.1 Install nupic (need to do this anyway)
   2.2 Uninstall nupic.bindings
   2.3 Build/install the desired version of nupic.bindings (need to do this anyway)

[vitaly-krugl]

Here is a more complete list of workflows that were discussed offline:

1. It should always be clear in nupic master with which release of nupic.bindings it is compatible.
2. Incremental nupic builds in the build/ci system (e.g., nupic PR builds)
   • These need to be validated against a specific version of nupic.bindings (not loose>=).
   • Testing against a non-specific or unintended version of nupic.bindings will produce inconsistent and/or failing builds
3. nupic builds intended for release/deployment
   • These builds also need to be based on specific, locked-down versions of nupic.bindings, and the resulting artifacts (e.g., wheels) must convey the dependency on the exact locked-down version of nupic.bindings, such that installation of the artifact from PyPi will bring in the specific version of nupic.bindings.
4. The nupic-only developer workflow
   • Clones nupic, installs it via python setup.py develop or similar, makes changes only in nupic
   • Relies on automatically getting the version of nupic.bindings from PyPi that's compatible with the cloned version of nupic.
5. The nupic.core/nupic.bindings developer workflow
   • Wants to make changes in nupic.core and possibly also in nupic; likely also wants the benefit of nupic's tests.
   • Needs to work with nupic.bindings version that's independent of the version that's pinned down in nupic

[vitaly-krugl]

Considering the updated workflows described in my preceding comment, and in the interest of time, we decided to forego PR #3325 for the time being, and solve the problem by updating the developer workflow documentation. As we flesh out our formal bamboo-based build and release process, we may still revisit the concepts in PR #3325.

The developer worflow documentation changes would consist of:

The nupic.core/nupic.bindings developer who also needs to install nupic for modification and/or take advantage of nupic's various tests, would need to

1. Install nupic (need to do this anyway)
2. Uninstall nupic.bindings
3. Build/install the desired version of nupic.bindings (need to do this anyway)

[vitaly-krugl]

I also posted a question about this on the wheel-builders list here: https://mail.python.org/pipermail/wheel-builders/2016-September/000220.html.

And my original question in the python wheel bitbucket issue is here: https://bitbucket.org/pypa/wheel/issues/172/wheel-doesnt-support-environment-markers#comment-30414396

[vitaly-krugl]

The original issue that was filed in nupic.core numenta/nupic.core-legacy#1069 asks to update the developer-installation documentation in nupic.core. Upon re-read of that documentation, I find no fault with it, since it focuses primarily on the standalone builds of nupic.core/nupic.bindings and doesn't discuss installing nupic

This option is for developers that would like the ability to do incremental builds of the C++ or for those that are using the C++ libraries directly.

I am updating the nupic developer installation documentation, instead, since that is where the conflict comes up.

[rhyolight]

@vitaly-krugl Sounds good Vitaly, thanks.

[vitaly-krugl]

I updated the https://github.com/numenta/nupic/wiki/Installing-and-Building-NuPIC#compiling section under "Installation from Source Code".

Please review.

[rhyolight]

Here is the diff. Looks okay to me.

However, does it address this concern that @oxtopus brought up in the original issue he filed?

---

Yet, when I pip uninstall nupic.bindings, it removes the link to the newer dev version which is unexpected, and even makes it so I can't remove the older 0.4.6 version, requiring manual intervention (i.e. rming the .egg file from site-packages):

$ pip uninstall nupic.bindings
Uninstalling nupic.bindings-0.4.7.dev0:
  /Users/amarshall/Library/Python/2.7/lib/python/site-packages/nupic.bindings.egg-link
Proceed (y/n)? y
  Successfully uninstalled nupic.bindings-0.4.7.dev0
$ pip uninstall nupic.bindings
Can't uninstall 'nupic.bindings'. No files were found to uninstall.
$ python -c 'import nupic.bindings; print nupic.bindings'
<module 'nupic.bindings' from '/Users/amarshall/Library/Python/2.7/lib/python/site-packages/nupic.bindings-0.4.6-py2.7-macosx-10.9-intel.egg/nupic/bindings/__init__.pyc'>

[vitaly-krugl]

@rhyolight, the new instructions in the nupic wiki should address @oxtopus's concerns by reversing the installation order of nupic and nupic.bindings and adding the pip uninstall nupic.bindings step in-between.

The various python mechanisms for package/installation management unfortunately are not fully compatible with each other, leaving room for getting an installation into an undesired state.

[oxtopus]

No. It does not. As I documented in the original issue, the instruction to pip uninstall nupic.bindings removes the .egg-link file (the one we actually want) and not the .egg (the one we actually want to remove).

Only way I know, as I documented, is to manually remove it e.g. rm

[vitaly-krugl]

@oxtopus, if you start with a clean system, then install nupic from source, then uninstall nupic.bindings, and finally build/install nupic.bindings from source, how can you possibly end up with a messed up system?

[vitaly-krugl]

@rhyolight and @oxtopus: I updated https://github.com/numenta/nupic/wiki/Installing-and-Building-NuPIC#nupic: Added instructions for uninstalling nupic and nupic.bindings before beginning to install nupic from source in order to get a clean install.

This would have been a good idea even before the latest dependency changes, since multiple installations from wheel (pip) and via python setup.py ... (setuptools) don't necessarily play nice together.

[vitaly-krugl]

@rhyolight, it sounds like we're okay with closing this issue?

[vitaly-krugl]

@rhyolight, the various build videos are a different story, though. We can't update those as easily as we can update the text in the wiki. What do you propose for them?

[vitaly-krugl]

@rhyolight and @oxtopus: I validated the updated build instructions from the nupic wiki on my dev laptop and ended up with a clean installation of both nupic and nupic.bindings, and the tests are doing well.

I think that if someone has a wedged installation of something in site-packages, then our instructions won't help them, and they will need to rely on their python skills to unwedge things. That said, it might be worthwhile when we hove more time to understand how our nupic.bindings installations manage get wedged, such that we have to rely on rm to clean up. @rhyolight: perhaps we can get some help from an outside nupic developer who is eager to land a helping hand?

[oxtopus]

Re:

how can you possibly end up with a messed up system?

Very easily. For example, by running python setup.py develop at any point in the future, which is a near certainty for someone actively working in both projects.

Expecting the user to always start from a clean system is at odds with an incremental build system that is supposed to support in-place editing. We might as well not allow it at all and instead remove support for the python setup.py develop workflow.

[vitaly-krugl]

#3327 (comment) describes the reasons behind pursuing the current approach versus the alternative. We will almost certainly revisit this in the very near future as part of our bamboo-ci build/release/deploy workflows. At that point, I hope that we will have a solution that we can all be happy with and proud of.

[rhyolight]

I want to do a final pass on this today. Thanks @vitaly-krugl I will take responsibility for this from here.

[rhyolight]

After uninstalling both nupic and nupic.bindings, here is how I followed the current instructions...

# Nothing up my sleeve
> pip freeze | grep nupic

# virtualenv for nupic
> virtualenv venv
New python executable in venv/bin/python
Installing setuptools, pip...done.
> . venv/bin/activate

# First, install nupic
cd $NUPIC
python setup.py install

# The explicit version of nupic.bindings is installed now
> nupic-version
nupic 0.5.6.dev0
nupic.bindings 0.4.6

# So uninstall the version of nupic.bindings it installed
> pip uninstall nupic.bindings
Uninstalling nupic.bindings-0.4.6:
  /Users/mtaylor/nta/nupic/venv/bin/nupic-bindings-check
  /Users/mtaylor/nta/nupic/venv/lib/python2.7/site-packages/nupic.bindings-0.4.6-py2.7-macosx-10.9-intel.egg
Proceed (y/n)? y
  Successfully uninstalled nupic.bindings-0.4.6

# Only nupic is installed now
> pip freeze | grep nupic
-e git+git@github.com:rhyolight/nupic.git@d2aad354226bba6b554c706e74e18b3ed4415a0f#egg=nupic

# So I'll go install nupic.bindings from source, deferring to the ncore readme as directed...

# First deactivate virtualenv to do the C++ build (I don't think it matters whether I do this
# within virtualenv or not, but jumping out anyway just to prove we don't need to be in one.)
> deactivate
> mkdir -p $NUPIC_CORE/build/scripts
> cd $NUPIC_CORE/build/scripts
> cmake $NUPIC_CORE -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../release -DPY_EXTENSIONS_DIR=$NUPIC_CORE/bindings/py/nupic/bindings
> make -j6
> make install
# All output looks good

# Going back to $NUPIC to re-enter virtualenv before installing 
> cd $NUPIC
> . venv/bin/activate

# Now heading to nupic.core source to install nupic.bindings at tip of master
> cd $NUPIC_CORE
> python setup.py install

> nupic-version
nupic 0.5.6.dev0
nupic.bindings 0.4.7.dev0

# Go into nupic dev mode
> cd $NUPIC
> python setup.py develop
running develop
running egg_info
writing requirements to src/nupic.egg-info/requires.txt
writing src/nupic.egg-info/PKG-INFO
writing namespace_packages to src/nupic.egg-info/namespace_packages.txt
writing top-level names to src/nupic.egg-info/top_level.txt
writing dependency_links to src/nupic.egg-info/dependency_links.txt
reading manifest file 'src/nupic.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'src/nupic.egg-info/SOURCES.txt'
running build_ext
Creating /Users/mtaylor/nta/nupic/venv/lib/python2.7/site-packages/nupic.egg-link (link to src)
nupic 0.5.6.dev0 is already the active version in easy-install.pth

Installed /Users/mtaylor/nta/nupic/src
Processing dependencies for nupic==0.5.6.dev0
Searching for nupic.bindings==0.4.6
Reading https://pypi.python.org/simple/nupic.bindings/
Downloading https://pypi.python.org/packages/a4/5c/2b97fe9235c044f2fa5a61885ca18c1757c516f9173a625727a0635945dc/nupic.bindings-0.4.6-py2.7-macosx-10.9-intel.egg#md5=295b607d4a598c115b174652ca988be6
nupicBest match: nupic.bindings 0.4.6
Processing nupic.bindings-0.4.6-py2.7-macosx-10.9-intel.egg

# Uh oh, we just went back to nupic.bindings==0.4.6
> nupic-version
nupic 0.5.6.dev0
nupic.bindings 0.4.6

# I can fix this by re-installing from nupic.core
> cd $NUPIC_CORE
> python setup.py develop

> nupic-version
nupic 0.5.6.dev0
nupic.bindings 0.4.7.dev0

[rhyolight]

So, given the above, there is still a problem with running python setup.py develop in nupic, because it will reinstall the explicit version of nupic.bindings. Maybe there is a way to specific dependencies differently between python setup.py install and python setup.py develop, but the work of setuptools/distutils is so confusing I'm afraid to even look.

I think I can update docs to indicate that when switching to "develop" mode in nupic, that nupic.bindings must be manually reinstalled at a previous version, if desired.

[rhyolight]

I also want to show that we can move around to different version of nupic.bindings in this flow if desired. So, starting where I left off in the previous command chain...

# Still in venv because why not?
> cd $NUPIC_CORE
# Switch to older version of nupic.core
> git checkout 0.4.1
HEAD is now at 3c64788... Release 0.4.1.
> mkdir -p $NUPIC_CORE/build/scripts
> cd $NUPIC_CORE/build/scripts
> cmake $NUPIC_CORE -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../release -DPY_EXTENSIONS_DIR=$NUPIC_CORE/bindings/py/nupic/bindings
> make -j6
> make install
# All output looks good

# Install nupic.bindings at 0.4.1
> cd $NUPIC_CORE
> python setup.py install
# version check
> nupic-version
nupic 0.5.6.dev0
nupic.bindings 0.4.1

When I run a hotgym example in this state, there is an error because of an API change in RecordSensor. But this proves that we have mobility to install different versions of nupic.core (in dev mode) and run against nupic (in dev mode).

But every time python setup.py develop is run, the developer must manually revert to the desired previous version of nupic.core if necessary.

With this knowledge, I'm going to run through the installation instructions in the NuPIC README and our wiki and do some cleanup before closing.

[rhyolight]

After reviewing our compilation instructions for nupic, I added this paragraph immediately after the instruction to run python setup.py develop:

Calling python setup.py develop will cause the installation of the latest stable release of nupic.bindings. If you are a developer trying to run nupic code against an older (or newer) version of nupic.bindings that what is declared in the requirements.txt file, you must re-install nupic.bindings from source manually to override the explicit version. You can do this easily by moving into the nupic.core checkout directory and re-running python setup.py develop|install there, which will install the version associated with the build nupic.core package there. View the nupic.core README for compilation instructions.

[rhyolight]

Another issue is that the https://github.com/numenta/nupic/wiki/Numenta-Developer-Setup is broken, but I'd rather defer that document update for now. I'm going to mark it as deprecated temporarily.