create3 not scanning for network

[sdobbelaere]

Please provide the following information:

• OS: Ubuntu 22
• ROS Distro: Humble
• Built from source or installed: deb package
• Package version: Current Firmware Version: H.2.2 (Humble)

Expected behaviour
List of SSIDs to select network on the create3 web interface.

Actual behaviour
Dropdown remains empty. Digging in logs, the message "network is down" keeps coming back.

What I've done so far:

• restarted the application
• restarted the robot
• cold restarted the robot
• restored to factory settings
• re-flashed the turtlebot4 sd-card to the humble image
• Upgraded through apt upgrade
• re-uploaded the humble 2.2 firmware for icreate

every time followed by looking in the dropdown and re-scanning for networks.

In the logs, I keep seeing a message Interface doesn't support scanning : Network is down for wlan0.
Similarly, manually setting the SSID and password don't work.
Logs

Aside from the issue at hand, It would be nice to better understand the relation between the create3 and the rest of the bot. They seem to be 2 independent systems, yet not entirely. I'm having a hard time identifying the relation.

[sdobbelaere]

Looking through the logs I do see a lot of "missing file" errors and network errors overall. Not sure if that should be the case?

Sep  3 12:21:42 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/aos_kvs_helper.c:51 errno:No such file or directory) Failed to stat file product.regdom
Sep  3 12:21:44 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/main.c:347 errno:No such file or directory) failed to set regdomain
Sep  3 12:21:44 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/main.c:353 errno:No such file or directory) Could not find product.regdom key
Sep  3 12:21:51 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/aos_kvs_helper.c:51 errno:No such file or directory) Failed to stat file network.ssid
Sep  3 12:21:51 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/aos_misc.c:754 errno:No such file or directory) Failed to stat file /opt/irobot/persistent/opt/irobot/data/bbk/user_wpa_supplicant.conf
Sep  3 12:21:51 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/aos_misc.c:801 errno:No such file or directory) Failed to open file: /opt/irobot/persistent/opt/irobot/data/bbk/user_wpa_supplicant.conf
Sep  3 12:21:51 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/aos_kvs_helper.c:51 errno:No such file or directory) Failed to stat file robot.disc_reconnect_sleep_sec
Sep  3 12:21:51 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/kvs_json.c:21 errno:No such file or directory) Failed to retrieve key "robot.disc_reconnect_sleep_sec"
Sep  3 12:21:51 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/aos_kvs_helper.c:51 errno:No such file or directory) Failed to stat file network.watchdog_interval_min
Sep  3 12:21:51 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/kvs_json.c:21 errno:No such file or directory) Failed to retrieve key "network.watchdog_interval_min"
Sep  3 12:21:52 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/aos_misc.c:86 errno:No such file or directory)  MODULE : network_manager CMD (sudo network_dhcp_arp_control.sh disable) failed with status = 1, ret = 256
Sep  3 12:21:57 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/mqtt_client.c:695 errno:Resource temporarily unavailable) firr_transfer failed
Sep  3 12:21:57 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/main.c:184 errno:Resource temporarily unavailable) Error publishing sta:0 FIRR request.
Sep  3 12:21:57 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/aos_kvs_helper.c:51 errno:No such file or directory) Failed to stat file network.addr_type
Sep  3 12:21:57 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/kvs_json.c:21 errno:No such file or directory) Failed to retrieve key "network.addr_type"
Sep  3 12:21:57 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/aos_kvs_helper.c:51 errno:No such file or directory) Failed to stat file network.ssid
Sep  3 12:21:57 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/network_utils.c:262 errno:No such file or directory) cannot load ssid
Sep  3 12:22:21 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/mqtt_client.c:695) firr_transfer failed
Sep  3 12:22:21 iRobot-6F2890FA31C0486E80A20C80E321CE4D daemon.err network[1462]: [ERROR] (src/discovery.c:407) falied to get cap from tsprime

[hilary-luo]

If the Network is down, it can be brought up by doing the following series of steps (Ignore the step numbers, do them in the order that I am showing here):

These are a subset of the create3 setup guide found here: https://edu.irobot.com/create3-setup. You can go through all of the steps listed there but you should have already done the update step and the mutli-robot namespacing should be done through the turtlebot4-setup tool not through the create3 interface.

Additionally, please note that for discovery server, the create3 should not be connected to the internet. It is only for simple discovery where the create3 should be directly connected to the internet.

[sdobbelaere]

Thank you @hilary-luo
The updates have been done already indeed. Without the wifi working I ended up switching to discovery network. But that seems to be bringing down my wifi as soon as I do something in Rviz.
Hoping that with simple discovery this will run as expected.

[hilary-luo]

Hmmm, you can give it a try but simple discovery does result in more network traffic so if you are seeing an issue with the network not being able to handle the traffic then this is unlikely to fix it. Good luck and feel free to let me know if you are looking for more assistance.

[sdobbelaere]

It would seem I'm not out of the woods yet.
So I've moved things over to simple discovery and both the bot and pi are connected to the same wifi.

However, when launching diagnostics no data is coming though. Running Rviz is showing OK for the camera but no image is showing. Yet, using the teleop keyboard on my pc I can make the bot move.

If I compare the topic list from the PI to the pc, there seems to be a significant difference. Does that mean that icreate and the bot aren't really communicating well? What would be the right steps to try and resolve this?

--- /Users/sascha/Downloads/bot/ubuntu_pi.txt	Tue Sep  5 14:17:51 2023
+++ /Users/sascha/Downloads/bot/ubuntu_pc.txt	Tue Sep  5 14:18:32 2023
@@ -1,34 +1,35 @@
 /battery_state
+/clicked_point
+/cliff_intensity
+/cmd_audio
+/cmd_lightring
 /cmd_vel
-/color/preview/image
-/diagnostics
-/diagnostics_agg
-/diagnostics_toplevel_state
 /dock_status
-/function_calls
+/goal_pose
 /hazard_detection
-/hmi/buttons
-/hmi/display
-/hmi/display/message
-/hmi/led
 /imu
+/initialpose
 /interface_buttons
-/ip
-/joint_states
-/joy
-/joy/set_feedback
+/ir_intensity
+/ir_opcode
+/kidnap_status
+/mobility_monitor/transition_event
 /mouse
 /oakd/rgb/preview/camera_info
 /oakd/rgb/preview/image_raw
-/oakd/rgb/preview/image_raw/compressed
-/oakd/rgb/preview/image_raw/compressedDepth
-/oakd/rgb/preview/image_raw/theora
+/odom
 /parameter_events
 /robot_description
+/robot_state/transition_event
 /rosout
 /scan
-/stereo/depth
+/slip_status
+/static_transform/transition_event
+/stop_status
 /tf
 /tf_static
 /wheel_status
+/wheel_ticks
+/wheel_vels
+

It's also worth noting that the display on the bot isn't giving battery info, and there are only 3 of the 5 led lights on. (left, middle and right)

[sdobbelaere]

So for now I've given up on simple discover - instead I'm trying with the discovery server again. However the moment I run ros2 launch turtlebot4_viz view_model.launch.py with lidar and camera the wifi is saturated.

Do you have any thoughts on ways to reduce the network traffic? Or maybe some thoughts on wifi routers that have more bandwidth?

[hilary-luo]

I agree that it seems as though the devices were not all set up properly in simple discovery. But as you say, we can continue with discovery server.

In regards to the network, have you connected the raspberry pi to a 5GHz network? That would be the first place to start, in order for your system to be able to support full functionality. What are the symptoms that make you say that the wifi is saturated? I am running my turtlebots off of a dlink router I got from a thrift shop, I just made sure that it supported 5 GHz.

In the meantime, there are some things you can do to limit the network traffic but it involves reducing which features that you use. When you say that when you launch rviz with lidar and camera, the wifi is saturated, are you trying to actually display the camera feed in rviz or just launching rviz to begin with? You should be able to launch rviz without subscribing to / displaying the camera feed. That should really cut down on the network traffic. Also, if you have diagnostics on, try turning it off (can be turned off through the turtlebot-setup tool). That should also reduce the network traffic.

[sdobbelaere]

I agree that it seems as though the devices were not all set up properly in simple discovery. But as you say, we can continue with discovery server.

Fair enough. I'll probably like to circle back to this once the networking issue is resolved. As you say simple discovery will increase the network traffic even more. So that sounds like #1 to deal with.

In regards to the network, have you connected the raspberry pi to a 5GHz network? That would be the first place to start, in order for your system to be able to support full functionality. What are the symptoms that make you say that the wifi is saturated? I am running my turtlebots off of a dlink router I got from a thrift shop, I just made sure that it supported 5 GHz.

So the wifi is pre-dual band. I read in the docs that dual-band would work better, never realising that this dual band is really a requirement more than an optional "improvement".

The AP I ordered last night is a Zyxell NWA210X. Probably overkill judging from your response - but this current pre-dual band router I'm using isn't cutting it.

In the meantime, there are some things you can do to limit the network traffic but it involves reducing which features that you use. When you say that when you launch rviz with lidar and camera, the wifi is saturated, are you trying to actually display the camera feed in rviz or just launching rviz to begin with? You should be able to launch rviz without subscribing to / displaying the camera feed. That should really cut down on the network traffic. Also, if you have diagnostics on, try turning it off (can be turned off through the turtlebot-setup tool). That should also reduce the network traffic.

Hmm, can't seem to find that option in the turtle-bot setup. I did see some options like this when running ros2 service list but I'm not sure where to find the docs to identify the service-type and request with these calls. Can you point me in the right direction?

[hilary-luo]

Investing in a good quality AP sets you up well for future work.

I assume that you are referring to finding the option to turn off diagnostics. It is actually a setting on the robot that you will change, not accessed through ROS2 services. SSH into the robot and run turtlebot4-setup -> go under ROS Setup -> Bash Setup -> Select TURTLEBOT4_DIAGNOSTICS -> Disabled -> Save -> Back out to the main menu -> Apply Settings -> Yes

[sdobbelaere]

Success! So for sure, buy yourself a modern Wifi Access point. Lol
I made a pull-request on the docs to stipulate that you really want to have 5G wifi. None of this old stuff - it just doesn't have the bandwidth.

I'll try getting the simple discovery running as well.

[ramakrishnasakethi]

If the Network is down, it can be brought up by doing the following series of steps (Ignore the step numbers, do them in the order that I am showing here):

These are a subset of the create3 setup guide found here: https://edu.irobot.com/create3-setup. You can go through all of the steps listed there but you should have already done the update step and the mutli-robot namespacing should be done through the turtlebot4-setup tool not through the create3 interface.

Additionally, please note that for discovery server, the create3 should not be connected to the internet. It is only for simple discovery where the create3 should be directly connected to the internet.

I do not understand the step 3. Where is the listing supposed to be exactly? I work with Ubuntu 22.04 in a Virtual Machine.

[ramakrishnasakethi]

If the Network is down, it can be brought up by doing the following series of steps (Ignore the step numbers, do them in the order that I am showing here):

These are a subset of the create3 setup guide found here: https://edu.irobot.com/create3-setup. You can go through all of the steps listed there but you should have already done the update step and the mutli-robot namespacing should be done through the turtlebot4-setup tool not through the create3 interface.

Additionally, please note that for discovery server, the create3 should not be connected to the internet. It is only for simple discovery where the create3 should be directly connected to the internet.

Here's the downloaded log file for your reference:
messages(1).txt

Do you recommend any resource(books, video tutorials, blog articles etc..) to make sense of the messages in the log file? That would be much appreciated. Thank you.

[sdobbelaere]

If the Network is down, it can be brought up by doing the following series of steps (Ignore the step numbers, do them in the order that I am showing here):
These are a subset of the create3 setup guide found here: https://edu.irobot.com/create3-setup. You can go through all of the steps listed there but you should have already done the update step and the mutli-robot namespacing should be done through the turtlebot4-setup tool not through the create3 interface.
Additionally, please note that for discovery server, the create3 should not be connected to the internet. It is only for simple discovery where the create3 should be directly connected to the internet.

I do not understand the step 3. Where is the listing supposed to be exactly? I work with Ubuntu 22.04 in a Virtual Machine.

It's in the create3 screen. The wifi SSID input field will turn into a dropdown once the wifi is live.
I must admit it's had me puzzled as well - when I first got the robot running it was always showing. But now only when you follow the instructions with the two-buttons.

Also it won't show you which SSID you're actually connected to. But if you're connected to a wifi and all went well you'll see an ip-address.

[ramakrishnasakethi]

If the Network is down, it can be brought up by doing the following series of steps (Ignore the step numbers, do them in the order that I am showing here):
These are a subset of the create3 setup guide found here: https://edu.irobot.com/create3-setup. You can go through all of the steps listed there but you should have already done the update step and the mutli-robot namespacing should be done through the turtlebot4-setup tool not through the create3 interface.
Additionally, please note that for discovery server, the create3 should not be connected to the internet. It is only for simple discovery where the create3 should be directly connected to the internet.

I do not understand the step 3. Where is the listing supposed to be exactly? I work with Ubuntu 22.04 in a Virtual Machine.

It's in the create3 screen. The wifi SSID input field will turn into a dropdown once the wifi is live. I must admit it's had me puzzled as well - when I first got the robot running it was always showing. But now only when you follow the instructions with the two-buttons.

Also it won't show you which SSID you're actually connected to. But if you're connected to a wifi and all went well you'll see an ip-address.

I do not get the dropdown activated. There is an Identify Robot button on top as shown in the picture below, which when I click the light ring turns solid cyan,

Assuming that Create-3 is then connected to the Computer's Wi Fi, I tried opening the 192.168.10.1 page mentioned in the Instruction steps and it would not open,

A snapshot of the logs corresponding to the action for your reference,

Also, when I try and download the logs, the logs that are downloaded seem to be random and not corresponding to the most relevant and latest ones. Is this a bug or have I misunderstood something? @hilary-luo

[hilary-luo]

@ramakrishnasakethi Please discuss your issue on your open ticket. If you wish to ask other people questions you can tag them on your ticket but for my own sanity I will be responding to you with troubleshooting steps etc. on your ticket only (or tickets when you have multiple independent issues open).

I have responded there about the fact that you have not successfully enabled the hotspot / wifi on your create3 robot.

@sdobbelaere Would you agree that this issue is now resolved for you and the ticket can be closed? Thank you for entering the PR, I (or a colleague) will review and get back to you.

[sdobbelaere]

Indeed, ticket resolved. Thank you for your help @hilary-luo