Javascript error when updating table display (PR #87 export-data)

[GenevieveBuckley]

Re: PR #87

I'm getting a javascript error related to updating the datatable with selected data from the scatterplot. It doesn't seem like a fatal error, but should still be fixed:

error handling message Message 'PATCH-DOC' (revision 1): DeserializationError("Seq(Any) expected a list or None, got {'0': -0.43731416321298383, '1': -0.4933948783766794}",)

Plan of attack - set the table.source to a CDSView of the scatterplot source selected indices.

Some other possibly related info:

• DeserializationError when editing Data Table bokeh/bokeh#7417 (comment)
• DataTable data model fixes bokeh/bokeh#8310

[bryevdv]

Those are definitely the relevant issue/PR. The work has been merged and those error messages should go away with the next release of Bokeh. If you'd like to try things out early, dev builds are installable now: https://bokeh.pydata.org/en/latest/docs/installation.html#developer-builds

[jni]

@bryevdv thanks so much for chiming in! I'll give the dev builds a go.

[GenevieveBuckley]

So I just tried adding https://github.com/bokeh/bokeh.git to the pip install section of ourenvironment.yml but conda had trouble unpacking bokeh. Not sure if I've done something silly because I'm tired, but I thought that should have worked.

environment.yml file:

name: mic
dependencies:
- python=3.6*
- click
- cytoolz
- h5py
- matplotlib
- networkx
- numpy
- pandas
- pip
- pymongo=3*
- scikit-image
- scikit-learn
- scipy
- setuptools
- six
- toolz
- imageio
- pip:
  - coverage>=4.0
  - pytest>=2.8
  - pytest-cov>=2.2
  - sh>=1.11
  - https://github.com/bokeh/bokeh.git

Traceback:

$ conda env create -f environment.yml 
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.10
  latest version: 4.5.11

Please update conda by running

    $ conda update -n base -c defaults conda



Downloading and Extracting Packages
wheel-0.32.1         | 35 KB     | ##################################### | 100% 
scikit-learn-0.20.0  | 5.4 MB    | ##################################### | 100% 
numpy-base-1.15.2    | 4.1 MB    | ##################################### | 100% 
cloudpickle-0.5.6    | 26 KB     | ##################################### | 100% 
matplotlib-3.0.0     | 6.8 MB    | ##################################### | 100% 
tornado-5.1.1        | 662 KB    | ##################################### | 100% 
networkx-2.2         | 2.0 MB    | ##################################### | 100% 
pyparsing-2.2.2      | 97 KB     | ##################################### | 100% 
cytoolz-0.9.0.1      | 350 KB    | ##################################### | 100% 
pymongo-3.7.1        | 990 KB    | ##################################### | 100% 
dask-core-0.19.3     | 1.2 MB    | ##################################### | 100% 
pillow-5.3.0         | 550 KB    | ##################################### | 100% 
pywavelets-1.0.1     | 4.3 MB    | ##################################### | 100% 
numpy-1.15.2         | 48 KB     | ##################################### | 100% 
mkl_fft-1.0.6        | 139 KB    | ##################################### | 100% 
six-1.11.0           | 21 KB     | ##################################### | 100% 
click-7.0            | 118 KB    | ##################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Collecting https://github.com/bokeh/bokeh.git (from -r /Users/genevieb/Documents/GitHub/Code_repositories/microscopium/condaenv.9_lsiz22.requirements.txt (line 5))
  Downloading https://github.com/bokeh/bokeh.git
     / 245kB 93kB/s
  Cannot unpack file /private/var/folders/gz/6f0v_zhs09x38mb4td_h6py80000gn/T/pip-unpack-jave95nr/bokeh.git (downloaded from /private/var/folders/gz/6f0v_zhs09x38mb4td_h6py80000gn/T/pip-req-build-olw3aj30, content-type: text/html; charset=utf-8); cannot detect archive format
Cannot determine archive format of /private/var/folders/gz/6f0v_zhs09x38mb4td_h6py80000gn/T/pip-req-build-olw3aj30

CondaValueError: pip returned an error

[jni]

Should be git+https

[GenevieveBuckley]

Thanks! I knew it'd be something silly.

Unfortunately there is a user input prompt that seems to be broken, I can't pip install:

1. adding git+https://github.com/bokeh/bokeh.git to the environment.yml file
2. from the terminal into an existing conda environment with pip install git+https://github.com/bokeh/bokeh.git
3. locally from my cloned bokeh repository, with pip install .

Same traceback and error message each time:

$ conda env create -f environment.yml 
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.10
  latest version: 4.5.11

Please update conda by running

    $ conda update -n base -c defaults conda


Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Collecting git+https://github.com/bokeh/bokeh.git (from -r /Users/genevieb/Documents/GitHub/Code_repositories/microscopium/condaenv.8ikywm76.requirements.txt (line 5))
  Cloning https://github.com/bokeh/bokeh.git to /private/var/folders/gz/6f0v_zhs09x38mb4td_h6py80000gn/T/pip-req-build-l3mtj88r
    Complete output from command python setup.py egg_info:
    
    Bokeh includes a JavaScript library (BokehJS) that has its own
    build process. How would you like to handle BokehJS:
    
    1) build and install fresh BokehJS
    2) install last built BokehJS from bokeh/bokehjs/build
    
    Choice? Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/gz/6f0v_zhs09x38mb4td_h6py80000gn/T/pip-req-build-l3mtj88r/setup.py", line 109, in <module>
        bokehjs_action = build_or_install_bokehjs()
      File "/private/var/folders/gz/6f0v_zhs09x38mb4td_h6py80000gn/T/pip-req-build-l3mtj88r/_setup_support.py", line 128, in build_or_install_bokehjs
        jsbuild = jsbuild_prompt()
      File "/private/var/folders/gz/6f0v_zhs09x38mb4td_h6py80000gn/T/pip-req-build-l3mtj88r/_setup_support.py", line 253, in jsbuild_prompt
        value = input("Choice? ")
    EOFError: EOF when reading a line
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/gz/6f0v_zhs09x38mb4td_h6py80000gn/T/pip-req-build-l3mtj88r/

CondaValueError: pip returned an error

[jni]

@GenevieveBuckley check out the link shared by @bryevdv. Looks like building Bokeh is quite tricky so they provide prerelease conda and pip packages. The installation from source instructions are quite unconventional!

https://bokeh.pydata.org/en/latest/docs/dev_guide/setup.html#devguide-setup

[GenevieveBuckley]

Good point, reading the documentation would be a good start.

Ok, so I can get a bokeh development install working well using the steps from your link here: https://bokeh.pydata.org/en/latest/docs/dev_guide/setup.html#devguide-setup
I've tested it like they suggest you do at the end, and both examples with the sample data work well.

I've also been able to create a new microscopium conda env with a bokeh development install. The bokeh sample data tests work.

In the old microscopium conda env (using bokeh 0.13) I can run python microscopium/serve.py tests/testdata/images/data.csv and everything works.

But... putting the last two bits together is a problem. Once I switch to the conda env with the development version of bokeh, our web app no longer updates the figure with the selected images. To make things trickier, there are no python or javascript errors being reported.

$ python microscopium/serve.py tests/testdata/images/data.csv
ready!
WARNING:bokeh.server.util:Host wildcard '*' will allow connections originating from multiple (or possibly all) hostnames or IPs. Use non-wildcard values to restrict access explicitly

Javascript logs:

properties.js:14 [bokeh] setting log level to: 'info'
connection.js:183 [bokeh] Websocket connection 0 is now open
data_table.js:180 [bokeh] jquery-ui is required to enable DataTable.reorderable
document.js:162 [bokeh] document idle at 358 ms
notebook.js:60 Bokeh items were rendered successfully

[jni]

@GenevieveBuckley Yikes! That's scary, and suggests that whatever we're doing in 0.13 might go away soon. =(

[bryevdv]

@jni @GenevieveBuckley The issue is almost certainly that a very bad bug in 0.13 related to selections has been fixed, but that may also require code changes for some users. The good news is the new (proper) way to handle selections is now much more rigorously maintained under new integration tests.

TLDR, BokehJS used to replace entire Selection objects (so: source.on_change('selected', ...) is what people used) but this should have never been allowed to happen. We can't make that workable, so things have changed so that BokehJS only ever updates the existing selection (so: source.selected.on_change('indices', ...) is what will always be correct going forward)

I will try to produce a patch for your code by tomorrow morning.

[bryevdv]

@jni @GenevieveBuckley This patch appears to be what is needed to get things working with 1.0dev builds:

diff --git a/microscopium/serve.py b/microscopium/serve.py
index b13fbdc..309cbb6 100644
--- a/microscopium/serve.py
+++ b/microscopium/serve.py
@@ -300,17 +300,17 @@ def make_makedoc(filename, color_column=None):

         def load_selected(attr, old, new):
             """Update images and table to display selected data."""
-            print('new index: ', new.indices)
+            print('new index: ', new)
             # Update images & table
-            if len(new.indices) == 1:  # could be empty selection
-                update_image_canvas_single(new.indices[0], data=dataframe,
+            if len(new) == 1:  # could be empty selection
+                update_image_canvas_single(new[0], data=dataframe,
                                            source=image_holder)
-            elif len(new.indices) > 1:
-                update_image_canvas_multi(new.indices, data=dataframe,
+            elif len(new) > 1:
+                update_image_canvas_multi(new, data=dataframe,
                                           source=image_holder)
-            update_table(new.indices, dataframe, table)
+            update_table(new, dataframe, table)

-        source.on_change('selected', load_selected)
+        source.selected.on_change('indices', load_selected)
         page_content = layout([
             [embed, image_plot],
             controls,

apologies for the churn, we all make mistakes unfortunately, and this one came in because our integration testing infrastructure fell over for quite a long time. But there are tests in place now to maintain that this kind of pattern always works as expected, and this is the correct approach from 1.0 onward.

[jni]

@bryevdv Great! Thank you so much for looking at this! We will pin to 0.13 for now and keep this patch and issue in mind for when it's time to upgrade.

Incidentally, we ourselves are not testing any Bokeh functionality as we have no experience with writing UI tests. So if you can point us to anything you're doing that we could maybe implement for our own tests, that would be incredibly useful!

[GenevieveBuckley]

Wow, thanks so much @bryevdv! That is extremely useful to know. We'll upgrade when there's a stable release of bokeh 1.0.0

I also second Juan's comments - we're not sure how to get started writing UI tests, so if you have any pointers for newbies here that would be highly appreciated.