rows x columns is not JSON serializable?

[chinobing]

I am trying to share the data between callbacks with "Storing Data in Hidden Div" method.

1. First of all, I get multiple stock prices and put it into a pd.DataFrame, then I save it as a json file.
2. I follow the tutorials to pd.read_json file first, the save it to the immediate Div. here is the code:

#json data processing
def data_table():
    df = pd.read_json('matrix_table.json')
    return df
    
@app.callback(dash.dependencies.Output('intermediate-value', 'children'), 
              [dash.dependencies.Input('stock-ticker-input', 'value')])
def clean_data(value):
     # if "Dropdown" has value input
     if value:
         cleaned_df = data_table()
    
         return cleaned_df  

@app.callback(dash.dependencies.Output('table', 'children'), 
              [dash.dependencies.Input('intermediate-value', 'children')])
def update_table(jsonified_cleaned_data):

    return jsonified_cleaned_data

it turns out to be an error message.

[chinobing]

I have tried another method "Caching and Signaling" without using to_json. But it turned out to be the same issue.

127.0.0.1 - - [11/Nov/2017 11:07:05] "POST /_dash-update-component HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1994, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1985, in wsgi_app
    response = self.handle_exception(e)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/dash/dash.py", line 541, in dispatch
    return self.callback_map[target_id]['callback'](*args)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/dash/dash.py", line 509, in add_context
    cls=plotly.utils.PlotlyJSONEncoder),
  File "/Users/BingWong/anaconda/lib/python3.4/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/plotly/utils.py", line 136, in encode
    encoded_o = super(PlotlyJSONEncoder, self).encode(o)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/simplejson/encoder.py", line 291, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/simplejson/encoder.py", line 373, in iterencode
    return _iterencode(o, 0)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/plotly/utils.py", line 204, in default
    return _json.JSONEncoder.default(self, obj)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/simplejson/encoder.py", line 268, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError:             002190  300169
date                      
2017-10-12   28.42    7.88
2017-10-11   28.41    7.94
2017-10-10   28.72    8.00
2017-10-09   28.50    7.91
2017-09-29   28.00    7.74
2017-09-28   27.70    7.48
2017-09-27   28.09    7.55
2017-09-26   28.20    7.40
2017-09-25   27.65    7.51
2017-09-22   28.32    7.64
2017-09-21   28.92    7.62
2017-09-20   29.52    7.74
2017-09-19   28.79    7.67
2017-09-18   29.51    7.73
2017-09-15   29.04    7.67
2017-09-14   28.50    7.73
2017-09-13   28.91    7.68 is not JSON serializable

[chinobing]

here is code I used:

import copy
import dash
from dash.dependencies import Input, Output
import dash_html_components as html
import dash_core_components as dcc
import datetime
from flask_caching import Cache
import numpy as np
import os
import pandas as pd
import time


app = dash.Dash(__name__)
cache = Cache(app.server, config={
    'CACHE_TYPE': 'filesystem',
    'CACHE_DIR': 'cache'
})

stock_ticker=['300169','002190','002405','600069','000039','600668']


app.layout = html.Div([
    dcc.Dropdown(
        id='dropdown',
        options=[{'label': i, 'value': i} for i in stock_ticker],
        value=['300169','002190'],
        multi= True
    ),
    html.Div([
        html.Div(dcc.Graph(id='graph-1'), className="six columns"),
    ], className="row"),

    # hidden signal value
    html.Div(id='signal', style={'display': 'none'})
])


# perform expensive computations in this "global store"
# these computations are cached in a globally available
# redis memory store which is available across processes
# and for all time.
@cache.memoize()
def global_store(value):
    # simulate expensive query
    print('Computing value with {}'.format(value))
    
    # it is supposed to be the real-time data downloaded online,
    # But I saved it as csv file for testing purpose.
    price = pd.read_csv('matrix_table.csv')
    return price[value]



@app.callback(Output('signal', 'children'), [Input('dropdown', 'value')])
def compute_value(value):
    # compute value and send a signal when done
    # calculate correlation
    close = global_store(value)
    rets=close.pct_change()
    df=rets.corr()
    return df


@app.callback(Output('graph-1', 'figure'), [Input('signal', 'children')])
def update_graph_1(value):
    # generate_figure gets data from `global_store`.
    # the data in `global_store` has already been computed
    # by the `compute_value` callback and the result is stored
    # in the global redis cached
    
    #just simply display data as H3 format.
    print(value)
    return html.H3(value)





# Dash CSS
app.css.append_css({
    "external_url": "https://codepen.io/chriddyp/pen/bWLwgP.css"})
# Loading screen CSS
app.css.append_css({
    "external_url": "https://codepen.io/chriddyp/pen/brPBPO.css"})

if __name__ == '__main__':
    app.run_server(debug=True, processes=6)

here is the error message:

Traceback (most recent call last):
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1994, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1985, in wsgi_app
    response = self.handle_exception(e)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/dash/dash.py", line 541, in dispatch
    return self.callback_map[target_id]['callback'](*args)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/dash/dash.py", line 509, in add_context
    cls=plotly.utils.PlotlyJSONEncoder),
  File "/Users/BingWong/anaconda/lib/python3.4/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/plotly/utils.py", line 136, in encode
    encoded_o = super(PlotlyJSONEncoder, self).encode(o)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/simplejson/encoder.py", line 291, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/simplejson/encoder.py", line 373, in iterencode
    return _iterencode(o, 0)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/plotly/utils.py", line 204, in default
    return _json.JSONEncoder.default(self, obj)
  File "/Users/BingWong/anaconda/lib/python3.4/site-packages/simplejson/encoder.py", line 268, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError:           300169    002190    000039    600668    002405    600069
300169  1.000000  0.426461  0.307540  0.498250  0.441630  0.465351
002190  0.426461  1.000000  0.745591  0.216099  0.051173  0.219213
000039  0.307540  0.745591  1.000000  0.356857 -0.000641  0.094726
600668  0.498250  0.216099  0.356857  1.000000  0.612720 -0.045782
002405  0.441630  0.051173 -0.000641  0.612720  1.000000  0.421885
600069  0.465351  0.219213  0.094726 -0.045782  0.421885  1.000000 is not JSON serializable

csv file:

matrix_table.zip

[ned2]

It looks like you're returning a Pandas DataFrame and Series from your callback. Dash callback return values need to be serialised into JSON to be sent to the client. The JSON encoder that's being used (looks like it could be either simplejson or the standard-lib json depending on which is available) can convert a limited range of objects to JSON, but if you want automatic conversion of other things such as DataFrame you'd need customize the JSON encoder.

What you probably want to do is just convert the results of your callbacks into data structures that can be encoded automatically, such as lists and dictionaries. For example, with the callback that returns a column, I think you could return pd.read_csv('matrix_table.csv')[value].tolist()

[chinobing]

@ned2 Thanks for your suggestion. I have tried it out but it turned out to be the same error. I think I may need to slice the dataframe into lists.

[chriddyp]

For future googlers, see https://community.plot.ly/t/rows-x-columns-is-not-json-serializable/6747/3 for a continuation of this topic.