bug: OT-2 Robot Server Won't Start - Smoothieboard Com Issue?

[dabdoue]

Overview

Hello,
I am having an issue with my OT-2 in that I am unable to connect to it over USB. When SSHing into the pi, I found that the robot server service is not able to complete. The issue appears to be that when running the robot server service, it tries to update the firmware of my smoothieboard, but there is some kind of communication error between the pi and the smoothieboard when this happens. Because of the communication error, the robot server continually restarts, trying to communicate with the smoothieboard, but always failing to do so. I recently replaced the smoothieboard hoping the issue would be fixed, but unfortunately, it was not fixed. When I went to replace the board, I found that everything in that pipette assembly: motors, metal casing, the board itself, were all hot to the touch. Now after putting in the new board this is no longer the case, however I'm still unable to get my OT-2 to work properly. Perhaps another piece of hardware is not functioning correctly? Below is a full loop of the log of the robot server service trying to start.

Thank you

Steps to reproduce

Happens when turning on the robot. No amount of cycling nor time spent without power has been able to fix the issue.

Current behavior

Feb 27 20:18:35 0402e3 systemd[1]: Starting Opentrons Robot HTTP Server…
Feb 27 20:19:29 0402e3 uvicorn[1161]: INFO: Started server process [1161]
Feb 27 20:19:29 0402e3 uvicorn[1161]: INFO: Waiting for application startup.
Feb 27 20:19:29 0402e3 uvicorn[1161]: /data/robot_settings.json not found. Loading defaults
Feb 27 20:19:29 0402e3 uvicorn[1161]: Application startup complete.
Feb 27 20:19:29 0402e3 uvicorn[1161]: Uvicorn running on unix socket /run/aiohttp.sock (Press CTRL+C to quit)
Feb 27 20:19:29 0402e3 uvicorn[1161]: Stopped opentrons-gpio-setup.
Feb 27 20:19:29 0402e3 uvicorn[1161]: /data/robot_settings.json not found. Loading defaults
Feb 27 20:19:29 0402e3 opentrons-api[1161]: API server version: 7.0.1
Feb 27 20:19:29 0402e3 opentrons-api[1161]: Robot Name: opentrons
Feb 27 20:19:30 0402e3 opentrons-api[1161]: Connecting to motor controller at port /dev/ttyAMA0
Feb 27 20:19:30 0402e3 uvicorn[1161]: Using directory /data/opentrons_robot_server for persistence.
Feb 27 20:19:30 0402e3 opentrons-api[1161]: /data/robot_settings.json not found. Loading defaults
Feb 27 20:19:30 0402e3 opentrons-api[1161]: Registering Central Routing Board Revision GPIOs
Feb 27 20:19:30 0402e3 opentrons-api[1161]: Configuring GPIOs by central routing roard revision
Feb 27 20:19:30 0402e3 opentrons-api[1161]: Setting up GPIOs
Feb 27 20:19:30 0402e3 uvicorn[1161]: BEGIN (implicit)
Feb 27 20:19:30 0402e3 uvicorn[1161]: BEGIN
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: PRAGMA main.table_info(“migration”)
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: PRAGMA main.table_info(“protocol”)
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: PRAGMA main.table_info(“analysis”)
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: PRAGMA main.table_info(“run”)
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: PRAGMA main.table_info(“action”)
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: COMMIT
Feb 27 20:19:30 0402e3 uvicorn[1161]: BEGIN (implicit)
Feb 27 20:19:30 0402e3 uvicorn[1161]: BEGIN
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: PRAGMA main.table_info(“run”)
Feb 27 20:19:30 0402e3 uvicorn[1161]: [raw sql] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: SELECT migration.id, migration.created_at, migration.version
FROM migration ORDER BY migration.version DESC
Feb 27 20:19:30 0402e3 uvicorn[1161]: [generated in 0.00212s] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: Database has schema version 2. no migrations needed.
Feb 27 20:19:30 0402e3 uvicorn[1161]: SELECT analysis.id, analysis.completed_analysis
FROM analysis
WHERE analysis.completed_analysis_as_document IS NULL
Feb 27 20:19:30 0402e3 uvicorn[1161]: [generated in 0.00205s] ()
Feb 27 20:19:30 0402e3 uvicorn[1161]: COMMIT
Feb 27 20:19:30 0402e3 opentrons-api[1161]: Connecting to smoothie at port /dev/ttyAMA0
Feb 27 20:19:30 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:19:31 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:19:35 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:19:35 0402e3 opentrons-api-serial[1161]: smoothie: retry number 0/3
Feb 27 20:19:35 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:19:36 0402e3 uvicorn[1161]: - “GET /wifi/list HTTP/1.1” 200
Feb 27 20:19:36 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:19:40 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:19:40 0402e3 opentrons-api-serial[1161]: smoothie: retry number 1/3
Feb 27 20:19:40 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:19:41 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:19:44 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:19:44 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:19:46 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:19:46 0402e3 opentrons-api-serial[1161]: smoothie: retry number 2/3
Feb 27 20:19:46 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:19:46 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:19:51 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:19:51 0402e3 opentrons-api-serial[1161]: smoothie: retry number 3/3
Feb 27 20:19:51 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:19:51 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:19:56 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:19:56 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:19:56 0402e3 opentrons-api-serial[1161]: smoothie: retry number 0/3
Feb 27 20:19:56 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:19:59 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:19:59 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:20:01 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:01 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:20:01 0402e3 opentrons-api-serial[1161]: smoothie: retry number 1/3
Feb 27 20:20:02 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:20:06 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:07 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:20:07 0402e3 opentrons-api-serial[1161]: smoothie: retry number 2/3
Feb 27 20:20:07 0402e3 opentrons-api-serial[1161]: smoothie: Write → b’\r\n\r\n’
Feb 27 20:20:11 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:12 0402e3 opentrons-api-serial[1161]: smoothie: Read ← b’\r\n\r\n’
Feb 27 20:20:12 0402e3 opentrons-api-serial[1161]: smoothie: retry number 3/3
Feb 27 20:20:12 0402e3 opentrons-api[1161]: [2.5K blob data]
Feb 27 20:20:12 0402e3 opentrons-api[1161]: Setting programming mode
Feb 27 20:20:13 0402e3 opentrons-api[1161]: lpc21isp -wipe -donotstart /usr/lib/firmware/smoothie-v1.1.0-25e5cea.hex /dev/ttyAMA0 115200 12000
Feb 27 20:20:13 0402e3 opentrons-api[1161]: created lpc21isp subproc in 0.02816152572631836
Feb 27 20:20:14 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:20:14 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:20:16 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:21 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:26 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:30 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:20:30 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:20:31 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:36 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:41 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:45 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:20:45 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:20:46 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:51 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:20:56 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:00 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:21:00 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:21:01 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:03 0402e3 opentrons-api[1161]: ran lpc21isp subproc in 50.2349579334259
Feb 27 20:21:03 0402e3 opentrons-api[1161]: Smoothie update failed: 1 b"lpc21isp version 1.97\nFile /usr/lib/firmware/smoothie-v1.1.0-25e5cea.hex:\n\tloaded…\nStart Address = 0x0003C379\n\tconverted to binary format…\n\timage size : 368852\nImage size : 368852\nSynchronizing (ESC to abort)… no answer on ‘?’\n" b’’
Feb 27 20:21:03 0402e3 opentrons-api[1161]: Exception in Thread Manager build
Traceback (most recent call last):
File “usr/lib/python3.7/site-packages/opentrons/hardware_control/thread_manager.py”, line 276, in _build_and_start_loop
File “usr/lib/python3.7/asyncio/base_events.py”, line 573, in run_until_complete
File “usr/lib/python3.7/site-packages/opentrons/hardware_control/api.py”, line 214, in build_hardware_controller
File “usr/lib/python3.7/site-packages/opentrons/hardware_control/backends/controller.py”, line 315, in update_firmware
File “usr/lib/python3.7/site-packages/opentrons/drivers/smoothie_drivers/driver_3_0.py”, line 1914, in update_firmware
RuntimeError: Failed to program smoothie: 1: b’’
Feb 27 20:21:03 0402e3 uvicorn[1161]: Exception during hardware background initialization.
Traceback (most recent call last):
File “usr/lib/python3.7/site-packages/robot_server/hardware.py”, line 505, in _initialize_hardware_api
File “usr/lib/python3.7/site-packages/robot_server/hardware.py”, line 430, in _initialize_ot2_robot
File “usr/lib/python3.7/site-packages/opentrons/init.py”, line 149, in initialize
File “usr/lib/python3.7/site-packages/opentrons/init.py”, line 131, in _create_thread_manager
File “usr/lib/python3.7/site-packages/opentrons/hardware_control/thread_manager.py”, line 264, in managed_thread_ready_async
opentrons.hardware_control.thread_manager.ThreadManagerException: Failed to create Managed Object
Feb 27 20:21:06 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:11 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:15 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:21:15 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:21:16 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:21 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:26 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:30 0402e3 uvicorn[1161]: Error response: 503 - LegacyError -
Feb 27 20:21:30 0402e3 uvicorn[1161]: - “GET /health HTTP/1.1” 503
Feb 27 20:21:31 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:35 0402e3 systemd[1]: opentrons-robot-server.service: Start operation timed out. Terminating.
Feb 27 20:21:35 0402e3 uvicorn[1161]: Shutting down
Feb 27 20:21:36 0402e3 uvicorn[1161]: Waiting for connections to close. (CTRL+C to force quit)
Feb 27 20:21:36 0402e3 uvicorn[1161]: - “GET /networking/status HTTP/1.1” 200
Feb 27 20:21:36 0402e3 uvicorn[1161]: Waiting for application shutdown.
Feb 27 20:21:36 0402e3 uvicorn[1161]: Pool recreating
Feb 27 20:21:36 0402e3 uvicorn[1161]: Application shutdown complete.
Feb 27 20:21:36 0402e3 uvicorn[1161]: Finished server process [1161]
Feb 27 20:21:38 0402e3 systemd[1]: opentrons-robot-server.service: Failed with result ‘timeout’.
Feb 27 20:21:38 0402e3 systemd[1]: Failed to start Opentrons Robot HTTP Server.
Feb 27 20:21:38 0402e3 systemd[1]: opentrons-robot-server.service: Service RestartSec=100ms expired, scheduling restart.
Feb 27 20:21:38 0402e3 systemd[1]: opentrons-robot-server.service: Scheduled restart job, restart counter is at 3.
Feb 27 20:21:38 0402e3 systemd[1]: Stopped Opentrons Robot HTTP Server.
Feb 27 20:21:38 0402e3 systemd[1]: Starting Opentrons Robot HTTP Server…

Expected behavior

Robot server should complete so I can communicate over USB

Operating system

Windows

System and robot setup or anything else?

No response