Bokeh ceases to stream data using ColumnDataSource after intermittent time has elasped in v0.12.15

[jzybert]

Software/Hardware Information:
Python Version: 3.6.4
Bokeh Version: 0.12.15
Flask Version: 0.12.2
OS: Windows 10 Pro
Hardware: Intel Core i5, 64-bit
Browser: Chrome v66.0.3359.139 (Official Build) (64-bit)

Use Case:
Before explaining the issue, let me explain the software I'm developing so as to better understand the use case. I'm creating a GUI which communicates over UART to a microcontroller. The microcontroller send the GUI telemetry data about the system's current state and the GUI's job is to graph it. The GUI also has other button functionality, but the main interaction with Bokeh is via graphing telemetry.
The telemetry packets contain six different integer values for which the Bokeh graph has six independent lines. When a packet is received, it is timestamped and, using a ColumnDataSource, the six different integer values are graphed on two y-axes with a shared x-axis for time using a DatetimeTickFormatter.

Expected Result:
Using the ColumnDataSource to stream in the aforementioned values, I'm able to get live-updating graphing of the telemetry data. New data comes in and it's streamed at around 20 Hz to the Bokeh plot. (the plot keeps 3600 data points, ~2mins). On Bokeh version 0.12.14, everything works smoothly. The telemetry will continue to stream without issue tested up to days worth of this program running.

Issue:
The issue results only in Bokeh version 0.12.15. The streaming will be going along fine and then at a random time period, the graph stops displaying new telemetry data. (The random time period is anywhere from a minute to a few minutes). I lose all connection to the graph it seems. No more live-updating, no ability to clear the data source, even if I pause and then play again no more data will be streamed. I'm forced to re-call bokeh serve --allow-websocket-origin=localhost:5000 in order to see data again.

Code: (I'm not able to post my entire code because this is software for the company I work for, but here's the code for creating the graph and streaming to it)

from __future__ import with_statement

import atexit
import configparser
from ctypes import c_short
from copy import copy
from datetime import datetime, timedelta
import logging
import math
import os
from shutil import copy2
import smtplib
import struct
import subprocess
import sys
from threading import Event, Thread, Lock
from time import sleep
import time
from queue import Queue, Empty
import webbrowser

from bokeh.client import push_session
from bokeh.embed import server_session
from bokeh.layouts import column, row, widgetbox
from bokeh.models import ColumnDataSource, CustomJS, HoverTool, LinearAxis,\
    NumeralTickFormatter, Range1d, DataRange1d, DatetimeTickFormatter
from bokeh.models.tools import WheelZoomTool, ResetTool
from bokeh.models.widgets import CheckboxGroup, Panel, Tabs
from bokeh.plotting import figure, curdoc, Figure
from flask import Flask, jsonify, render_template, request
from flask_uploads import configure_uploads, TEXT, UploadSet, UploadNotAllowed
from flask_socketio import SocketIO

app = Flask(__name__)
socketio = SocketIO(app, async_mode="threading")

stop_threads_event = Event()

def signal_threads_to_exit():
    """
    Signals all of the threads to exit and waits for it to happen.
    Closes the current port.

    :return: 0 on success
    """
    stop_threads_event.set()
    ccc.empty_short_graph()
    ccc.empty_long_graph()
    if thread_send_and_receive.isAlive():
        thread_send_and_receive.join()
    if thread_update_graph.isAlive():
        thread_update_graph.join()
    if thread_bokeh_server.isAlive():
        thread_bokeh_server.join()
    stop_threads_event.clear()
    return 0


def exit_handler():
    """
    Called when the script exits, Ctrl+C is clicked, or the terminal
    window is exited. Properly kills the bokeh subprocess and signals
    all threads to exit (if they haven't already). Exits the script
    with return code 0.
    """
    if bokeh_subprocess != 0:
        bokeh_subprocess.terminate()
    if "signal_threads_to_exit" in dir(os):
        signal_threads_to_exit()
    sys.exit(0)

atexit.register(exit_handler)
bokeh_subprocess = 0


class CommandControlCenter:
    """
    A class that contains the information about the Command Control 
    Center.
    """
    def __init__(self, hide_short_graph=False, hide_long_graph=False):
        self.is_in_playback_mode = False
        self.test_name = ""
        self.doc = curdoc()
        self.doc.title = "Command Control Center"
        self.graph_time = datetime.now().strftime("%Y:%m:%d:%H:%M:%S")
        self.chosen_port = None
        self.port_open = False
        self.board = ""
        self.current_mode = ""
        self.current_user = (None, None, None)
        self.current_direction = ""
        self.file_path = ""
        self.password = ""
        self.abs_values={
            "direction": False,
            "volt_A": False, "volt_B": False, "emit_A": False,
            "emit_B": False, "extr_A": False, "extr_B": False
        }
        # Initialize graphs
        self.hide_short_graph = hide_short_graph
        self.hide_long_graph = hide_long_graph
        if hide_long_graph and hide_short_graph:
            self.hide_short_graph = False
        if not hide_short_graph and not hide_long_graph:
            self.short_graph = GraphTelemetry(400, 1200, 3600)
            self.long_graph = GraphTelemetry(400, 1200, 4320)
        elif hide_short_graph and not hide_long_graph:
            self.long_graph = GraphTelemetry(800, 1200, 4320)
        elif hide_long_graph:
            self.short_graph = GraphTelemetry(800, 1200, 3600)
        self.playback_index_to_graph = 0
        # Add graphs to document
        checkbox = CheckboxGroup(labels=["", "", "", "", "", ""],\
            active=[0, 1, 2, 3, 4, 5], width=40)
        plot_lines = dict(checkbox=checkbox)
        if ((hide_short_graph and not hide_long_graph) 
                or (not hide_short_graph and not hide_long_graph)):
            long_lines = dict(
                lg_c1_l0=self.long_graph.channel1.plot_lines[0],
                lg_c1_l1=self.long_graph.channel1.plot_lines[1],
                lg_c1_l2=self.long_graph.channel1.plot_lines[2],
                lg_c1_l3=self.long_graph.channel1.plot_lines[3],
                lg_c1_l4=self.long_graph.channel1.plot_lines[4],
                lg_c1_l5=self.long_graph.channel1.plot_lines[5])
            plot_lines.update(long_lines)
        if (hide_long_graph) or (not hide_short_graph and not hide_long_graph):
            short_lines = dict(
                sg_c1_l0=self.short_graph.channel1.plot_lines[0],
                sg_c1_l1=self.short_graph.channel1.plot_lines[1],
                sg_c1_l2=self.short_graph.channel1.plot_lines[2],
                sg_c1_l3=self.short_graph.channel1.plot_lines[3],
                sg_c1_l4=self.short_graph.channel1.plot_lines[4],
                sg_c1_l5=self.short_graph.channel1.plot_lines[5])
            plot_lines.update(short_lines)
        checkbox.callback = CustomJS(args=plot_lines, code="""
        if (sg_c1_l0) {
            sg_c1_l0.visible = checkbox.active.includes(0);
            sg_c1_l1.visible = checkbox.active.includes(1);
            sg_c1_l2.visible = checkbox.active.includes(2);
            sg_c1_l3.visible = checkbox.active.includes(3);
            sg_c1_l4.visible = checkbox.active.includes(4);
            sg_c1_l5.visible = checkbox.active.includes(5);
        }
        if (lg_c1_l0) {
            lg_c1_l0.visible = checkbox.active.includes(0);
            lg_c1_l1.visible = checkbox.active.includes(1);
            lg_c1_l2.visible = checkbox.active.includes(2);
            lg_c1_l3.visible = checkbox.active.includes(3);
            lg_c1_l4.visible = checkbox.active.includes(4);
            lg_c1_l5.visible = checkbox.active.includes(5);
        }""")
        if hide_long_graph:
            plots = column(self.short_graph.channel1.plot)
        elif hide_short_graph and not hide_long_graph:
            plots = column(self.long_graph.channel1.plot)
        else:
            plots = column(
                self.short_graph.channel1.plot,
                self.long_graph.channel1.plot
            )
        layout = row(plots, widgetbox(checkbox, width=40))
        self.doc.add_root(layout)

    def check_password(self):
        """
        Verifies if the CCC password is correct.

        :return: True if the password is correct
        """
        from constants import PASSWORD
        return self.password == PASSWORD
    
    def reset_graph(self):
        """
        Resets graph time to current system time and resets the data
        sources of each graph.
        """
        self.graph_time = datetime.now().strftime("%Y:%m:%d:%H:%M:%S")
        if not self.hide_short_graph:
            self.short_graph.reset_graph()
        if not self.hide_long_graph:
            self.long_graph.reset_graph()

    def append_to_short_graph(self, data):
        """
        Appends data the short graph.

        :param data: the data to add
        """
        if not self.hide_short_graph:
            self.short_graph.append_to_buffer(data)

    def append_to_long_graph(self, data):
        """
        Appends data the long graph.

        :param data: the data to add
        """
        if not self.hide_long_graph:
            self.long_graph.append_to_buffer(data)

    def empty_short_graph(self):
        """
        Empties the telemetry buffer for the short graph.
        """
        if not self.hide_short_graph:
            self.short_graph.empty_buffer()

    def empty_long_graph(self):
        """
        Empties the telemetry buffer for the long graph.
        """
        if not self.hide_long_graph:
            self.long_graph.empty_buffer()

    def pop_and_empty_short_graph(self, index=None):
        """
        Gets the data the index from the short graph. Clears the short
        graph buffer.
        """
        if not self.hide_short_graph:
            return self.short_graph.pop_and_empty_buffer(
                self.is_in_playback_mode, index
            )

    def pop_and_empty_long_graph(self, index=None):
        """
        Gets the data the index from the long graph. Clears the long
        graph buffer.
        """
        if not self.hide_long_graph:
            return self.long_graph.pop_and_empty_buffer(
                self.is_in_playback_mode, index
            )


class ChannelPlot():
    """
    Class representing a telemetry plot for a single channel.
    """
    def __init__(self, channel_number, plot_height, plot_width):
        self.channel_number = channel_number
        self.data_source_lock = Lock()
        self.data_source = ColumnDataSource(dict(
            formatted_date=[], time=[], 
            volt_A=[], volt_B=[], 
            emit_A=[], emit_B=[], 
            extr_A=[], extr_B=[]
        ))
        self.plot = figure(
            plot_height=plot_height,
            plot_width=plot_width,
            tools=[HoverTool(
                tooltips=[
                    ("Time", "@formatted_date"),
                    ("Voltage A", "@volt_A"),
                    ("Voltage B", "@volt_B"),
                    ("Emitter Current A", "@emit_A"),
                    ("Emitter Current B", "@emit_B"),
                    ("Extractor Current A", "@extr_A"),
                    ("Extractor Current B", "@extr_B")
                ]
            )],
            toolbar_location=None
        )
        self.plot_lines = []
        self.plot.yaxis.visible = False
        self.plot.xaxis.axis_label = "Time (hh:mm:ss)"
        self.plot.xaxis.formatter = DatetimeTickFormatter(
            milliseconds=["%H:%M:%S"], seconds=["%H:%M:%S"],
            minsec=["%H:%M:%S"], minutes=["%H:%M:%S"],
            hourmin=["%H:%M:%S"], hours=["%H:%M:%S"],
            days=["%H:%M:%S"], months=["%H:%M:%S"], years=["%H:%M:%S"])
        self.plot.extra_y_ranges = {
            "voltage": DataRange1d(range_padding=0.5),
            "current": DataRange1d(range_padding=0.25)
        }
        self.plot.add_layout(LinearAxis(y_range_name="voltage",
                                         axis_label="Voltage (V)"), 'right')
        self.plot.add_layout(LinearAxis(y_range_name="current",
                                         axis_label="Current (µA)"), 'left')
        self.init_lines()
        self.plot.legend.padding = -67
        self.plot.legend.visible = False

    def init_lines(self):
        """
        Sets up the lines of the graph in accordance to the source data.
        """
        vA = self.plot.line(x="time", y="volt_A", color="red", line_width=1,
                        y_range_name="voltage", legend="Voltage A", 
                        source=self.data_source)
        vB = self.plot.line(x="time", y="volt_B", color="green", line_width=1,
                        y_range_name="voltage", legend="Voltage B", 
                        source=self.data_source)
        emA = self.plot.line(x="time", y="emit_A", color="gold", line_width=1, 
                        y_range_name="current", legend="Emitter A", 
                        source=self.data_source)
        emB = self.plot.line(x="time", y="emit_B", color="blue", line_width=1,
                        y_range_name="current", legend="Emitter B", 
                        source=self.data_source)
        exA = self.plot.line(x="time", y="extr_A", color="orangered", line_width=1,
                        y_range_name="current", legend="Extractor A", 
                        source=self.data_source)
        exB = self.plot.line(x="time", y="extr_B", color="purple", line_width=1,
                        y_range_name="current", legend="Extractor B", 
                        source=self.data_source)
        self.plot_lines.extend([vA, vB, emA, emB, exA, exB])
        self.plot.extra_y_ranges["voltage"].renderers = [
            self.plot_lines[0], self.plot_lines[1]
        ]
        self.plot.extra_y_ranges["current"].renderers = [
            self.plot_lines[2], self.plot_lines[3], 
            self.plot_lines[4], self.plot_lines[5]
        ]

class GraphTelemetry:
    """
    A class that contains the plotting information for 
    a graphing telemetry data.
    """
    def __init__(self, height, width, max_data_points):
        self.max_data_points = max_data_points
        self.telemetry_buffer_lock = Lock()
        self.telemetry_buffer = []
        # Channels
        self.channel1 = ChannelPlot(1, height, width)
        # Tabs
        self.tab1 = Panel(child=self.channel1.plot, title="Channel 1")
        self.tabs = Tabs(
            tabs=[self.tab1], 
            width=1010,
        )
        
    def stream_data(self, data_points, channel_number):
        """
        Streams the data_points dictionary to the data source.

        :param data_points: a dictionary of column names mapped to
                            an array of values to stream
        :param channel_number: the channel to stream to
        """
        with self.channel1.data_source_lock:
            self.channel1.data_source.stream(data_points, self.max_data_points)

    def reset_graph(self):
        """
        Resets the data property of ColumnDataSource to remove any
        stored data.
        """
        with self.channel1.data_source_lock:
            self.channel1.data_source.data = dict(
                formatted_date=[], time=[], 
                volt_A=[], volt_B=[], 
                emit_A=[], emit_B=[], 
                extr_A=[], extr_B=[]
            )

    def append_to_buffer(self, data):
        """
        Append data to the telemetry buffer.
        This is a thread-safe function.

        :param data: the data to append to the buffer
        """
        self.telemetry_buffer_lock.acquire()
        self.telemetry_buffer.append(data)
        self.telemetry_buffer_lock.release()

    def pop_from_buffer(self, index=0):
        """
        Pop from the the telemetry buffer at index.
        This is a thread-safe function.

        :param index: the index to pop from, default 0\n
        :return: the data at the index
                 None if the buffer is empty
                 EINVAL if the index is out of bounds
        """
        self.telemetry_buffer_lock.acquire()
        if len(self.telemetry_buffer) == 0:
            self.telemetry_buffer_lock.release()
            return None
        if len(self.telemetry_buffer) > index:
            self.telemetry_buffer_lock.release()
            return 7
        data = self.telemetry_buffer.pop(index)
        self.telemetry_buffer_lock.release()
        return data

    def empty_buffer(self):
        """
        Empty the telemetry buffer.
        This is a thread-safe function.
        """
        self.telemetry_buffer_lock.acquire()
        self.telemetry_buffer.clear()
        self.telemetry_buffer_lock.release()

    def pop_and_empty_buffer(self, is_in_playback_mode, index=None):
        """
        Pop from the the telemetry buffer at index and
        clear the buffer. This is a thread-safe function.
        Useful when you want to make sure nothing gets
        added to the buffer after you pop but before you
        clear.

        Will return the last element if the index is
        greater than any index available.

        :param index: the index to pop from, default 0\n
        :return: the data at the index
        """
        self.telemetry_buffer_lock.acquire()
        # List is empty
        if len(self.telemetry_buffer) == 0:
            self.telemetry_buffer_lock.release()
            return []
        # Index is greater than length of buffer
        if not index or len(self.telemetry_buffer) <= index:
            data = copy(self.telemetry_buffer)
            self.telemetry_buffer.clear()
            self.telemetry_buffer_lock.release()
            return data
        # Index within buffer bounds
        data = self.telemetry_buffer[:index]
        del self.telemetry_buffer[:index]
        self.telemetry_buffer_lock.release()
        return data


ccc = CommandControlCenter(hide_short_graph=False, hide_long_graph=False)


class FakeTelemetry:
    def __init__(self):
        self.relay_dir = 0
        self.voltage_A = 10
        self.voltage_B = 20
        self.emitter_A = 30
        self.emitter_B = 40
        self.extractor_A = 50
        self.extractor_B = 60


def bokeh_thread():
    """
    Thread that kicks off the bokeh server in a subprocess.
    The subprocess is properly terminated upon exit of script.
    """
    global bokeh_subprocess
    bokeh_args = ["bokeh", 
                  "serve", 
                  "--allow-websocket-origin=localhost:{}".format(5000)]
    try:
        bokeh_subprocess = subprocess.Popen(bokeh_args)
    except FileNotFoundError:
        print("Could not find path to bokeh module. `bokeh serve` will fail " +
              "and throw an error. Check your PATH variable.")


def command_thread():
    """
    Thread that sends and receives commands to and from the
    microcontroller via the serial_traffic functions.

    Will send the GET_ALL_TELEMETRY command if no other commands are
    currently queued up in Tx.
    """
    while True:
        if stop_threads_event.isSet():
            break
        timestamp = datetime.now().strftime("%Y:%m:%d:%H:%M:%S.%f")[:-3]
        data = [FakeTelemetry()]
        if not ccc.hide_short_graph:
            ccc.append_to_short_graph((timestamp, data))
        else:
            ccc.append_to_long_graph((timestamp, data))
        sleep(0.2)


def set_telemetry_values(timestamp, channel_stream, data):
    """
    Helper function to set the channels to the telemetry values.

    :param timestamp: the time for this data
    :param channel_stream: the array with all the channel data
    :param data: the telemetry data
    """
    global ccc
    abs_vals = ccc.abs_values
    direction = data.relay_dir
    # Time
    formatted_date = timestamp.strftime("%H:%M:%S.%f")[:-3]
    channel_stream["formatted_date"].append(formatted_date)
    channel_stream["time"].append(timestamp)
    # High Voltage A
    vA_pos = direction == 2 or direction == 0 or abs_vals["volt_A"]
    vA = data.voltage_A if vA_pos else 0 - data.voltage_A
    channel_stream["volt_A"].append(vA)
    # High Voltage B
    vB_pos = direction == 1 or direction == 0 or abs_vals["volt_B"]
    vB = data.voltage_B if vB_pos else 0 - data.voltage_B
    channel_stream["volt_B"].append(vB)
    # Emitter Current A
    emA = abs(data.emitter_A) if abs_vals["emit_A"]\
        else 0 - data.emitter_A if (direction == 1) else data.emitter_A
    channel_stream["emit_A"].append(emA)
    # Emitter Current B
    emB = abs(data.emitter_B) if abs_vals["emit_B"]\
        else 0 - data.emitter_B if (direction == 1) else data.emitter_B
    channel_stream["emit_B"].append(emB)
    # Extractor Current A
    exA = abs(data.extractor_A) if abs_vals["extr_A"]\
        else 0 - data.extractor_A if (direction == 1) else data.extractor_A
    channel_stream["extr_A"].append(exA)
    # Extractor Current B
    exB = abs(data.extractor_B) if abs_vals["extr_B"]\
        else 0 - data.extractor_B if (direction == 1) else data.extractor_B
    channel_stream["extr_B"].append(exB)


def update_graph_thread():
    """
    Updates the Bokeh plots at 20 Hz with new telemetry data pulled from
    the telemetry buffer in CommandControlCenter.
    """
    INDEX_TO_PULL = None
    TIME_TO_SLEEP = 0.2
    while True:
        if stop_threads_event.isSet():
            break
        # Safely pulls recent telemetry data from the buffer
        if not ccc.hide_short_graph:
            all_telemetry = ccc.pop_and_empty_short_graph(index=INDEX_TO_PULL)
        else:
            all_telemetry = ccc.pop_and_empty_long_graph(index=INDEX_TO_PULL)
        if all_telemetry:
            channel_1_stream = dict(formatted_date=[], time=[], volt_A=[], volt_B=[], emit_A=[], emit_B=[], extr_A=[], extr_B=[])
            # For each (timestamp, telemetry_array)
            for timestamp, telemetry_channels_array in all_telemetry:
                # Convert datetime string into datetime object
                timestamp_format = "{}-{}-{} {}".format(timestamp[:4], timestamp[5:7], timestamp[8:10], timestamp[11:])
                timestamp = datetime.strptime(timestamp_format, "%Y-%m-%d %H:%M:%S.%f")
                # Update the ccc graph time
                ccc.graph_time = timestamp.strftime("%Y:%m:%d:%H:%M:%S")
                # For each channel in the telemetry data
                channel_telemetry = telemetry_channels_array[0]
                channel_stream = channel_1_stream
                # If data exists
                if channel_telemetry:
                    # Parse out telemetry packet
                    set_telemetry_values(timestamp, channel_stream, channel_telemetry)
            # Graph short graph
            if not ccc.hide_short_graph:
                short_graph = ccc.short_graph
                channel_data_short = channel_stream
                short_graph.stream_data(channel_data_short, 1)
            # Sleep for a little bit of time to reduce the sluggishness
            sleep(TIME_TO_SLEEP)

@app.route("/", methods=["GET"])
def index():
    """
    The initial route that creates the server and embeds the Bokeh
    plot into index.html.

    :return: the template
    """
    session = push_session(ccc.doc)
    script = server_session(None, session_id=session.id)
    return render_template("index.html", script=script)

thread_update_graph = Thread(target=update_graph_thread)
thread_bokeh_server = Thread(target=bokeh_thread)
thread_send_and_receive = Thread(target=command_thread)

# Start Threads
thread_update_graph.start()
thread_send_and_receive.start()
thread_bokeh_server.start()
# Open the CCC webpage
webbrowser.open("http://localhost:{}".format(5000))
# Start the Flask Application
socketio.run(app, port=5000, debug=False)

In the Python Terminal:
2018-05-08 13:14:35,003 Starting Command Control Center version 1.8.8
2018-05-08 13:14:36,867 Starting Bokeh server version 0.12.14 (running on Tornado 4.5.3)
2018-05-08 13:14:36,876 Bokeh app running at: http://localhost:5006/
2018-05-08 13:14:36,877 Starting Bokeh server with process id: 1508
2018-05-08 13:14:36,964 101 GET /ws?bokeh-protocol-version=1.0&bokeh-session-id=3g8eVlcWvxTrP27DDxnwzDbOe83i2PSnckGWgEJSrKGp (::1) 1.00ms
2018-05-08 13:14:36,964 WebSocket connection opened
2018-05-08 13:14:36,966 ServerConnection created
2018-05-08 13:14:37,173 200 GET /autoload.js?bokeh-autoload-element=2e38afe5-1be5-41c2-ab4f-4612d0784631&bokeh-absolute-url=http://localhost:5006&bokeh-session-id=3g8eVlcWvxTrP27DDxnwzDbOe83i2PSnckGWgEJSrKGp (::1) 9.00ms
2018-05-08 13:14:38,146 101 GET /ws?bokeh-protocol-version=1.0&bokeh-session-id=3g8eVlcWvxTrP27DDxnwzDbOe83i2PSnckGWgEJSrKGp (::1) 0.00ms
2018-05-08 13:14:38,147 WebSocket connection opened
2018-05-08 13:14:38,149 ServerConnection created

Image:

It's hard to tell since this is a static image, but the graph has stopped displaying date but I know for a fact data is still coming in and is being logged.

I will do my best to provide as much information as possible as I'm able to. Unfortunately, I'm still not clear what caused this to happen in 0.12.15 and I'm limited in the code I'll be able to show.

[bryevdv]

I have to be completely honest and say that without some kind of complete minimal reproducer it's unlikely in the extreme that we will be able to determine any corrective course. I understand that code can be proprietary, but in those cases, more of the burden will have to shift on to the issue reporter out of sheer necessity. At this point, I only have two suggestions:

• check the browser JS console log for any errors or messages
• do a bisect to narrow down where the problem started

Doing a real bisection means getting a dev env set up and will be a tedious slog (I've had to do it many times) but may ultimately be the only option if there is no way to provide an MRE. However, you can first try to narrow things down by "bisecting" with dev builds:

https://bokeh.pydata.org/en/latest/docs/installation.html#developer-builds

These can be conda/pip installed so it should be much easier for a start. We are looking for the first 0.12.15dev build version that fails.

[bryevdv]

@jzybert will you be able to check the JS console, and try to test out using various dev builds?

[bryevdv]

closing for lack of response