Upgrade to support IPython/Jupyter 4

[lbustelo]

Currently the project is only installable on a IPython 3.2.x. This move need to be done in conjunction with dashboards #49.

[wellecks]

Looking into this, will look at the PR above as a starting point

[lbustelo]

Before getting to far into this... would be nice to take a look at #90

[wellecks]

Current TODO:

• 1. Change Dockerfile to pull from jupyter/all-spark-notebook or create separate make build in Makefile
  • push cloudet/all-spark-notebook-bower image
• 2. Change hardcoded filepaths in Makefile, setup.py, .gitignore, and other affected files
  • ensure make sdist, make server, make install work
• 3. Change import statements in example notebooks and backend Widget code
• 4. Fix declarative widget initialization
  • load_ipython_extension now necessary for extension initialization
  • handle request_state message in python
  • handle request_state message in scala
• 5. Find broken declarative widget functionality
  • add argument to _handle_custom_event_msg
  • Error: Couldn't create model on page refresh with developer tools window open
• 6. Fix broken declarative widget functionality
  • function
    • Error while invoking function: Can't clean for JSON: 0 in python dataframe function
      • numpy.int64 in the index and column list was not understood
  • dataframe
    • Invalid DataFrame variable name on all python DataFrame examples
      • sync message was arriving before sync_data. Now sends an update on sync_data
  • channels
    • no issues when running example notebook
  • import
    • no issues when running example notebook
  • chart
    • no issues when running example notebook
  • explorer
    • no issues when running example notebook
  • table
    • no issues when running example notebook
• 7. dependency modules, e.g. ipykernel and ipywidgets not visible in make test-py
  • the make target was using the old docker image

[wellecks]

regarding (1) above, our all-spark-notebook-bower now only installs bower. We can do away with the entire image by introducing a make build step, similar to the dashboards Makefile

[wellecks]

regarding (4),

"Error: Could not set widget manager state.↵ at …in.js?v=40e10638fcf65fc1c057bff31d165e9d:12785:33", message: "Could not set widget manager state."

now occurs when the notebook loads

[wellecks]

(5) and (6) are intentionally vague for now

[parente]

/cc @dalogsdon @jhpedemonte since findings here might inform widget related work in jupyter/dashboards#105 (and vice versa)

[wellecks]

Just FYI - I'm updating the checkbox TODO above as I discover / fix issues

[wellecks]

Opened an issue on the spark kernel that I discovered when working on handling the request_state in Scala

Specific problem arises here due to the missing parent_header msg_id

As a result, the model for the Scala widgets does not initialize

[wellecks]

I fixed the issue on the kernel PR here. Progress on this task is blocked until it's merged and the jupyter/all-spark-notebook image is updated with the new kernel

[parente]

@wellecks Could you please open a PR against docker-stacks asking for a rebuild to account for the spark kernel fix, and refer to that kernel PR so we know when the rebuild can happen? Thanks.

[wellecks]

Rebasing with the current changes; it looks like the python 2/3 support will make the rebase non-trivial