No messages on joint_states topic - sometimes

[rbbg]

Hi,

I am trying to communicate with iiwa 7 r800 using the iiwa_stack. The current Sunrise version is 1.9 and I am connecting to the java side to using a USB NIC (as discussed in #7). The configuration seems to work fine (the java application can connect to the master and rostopic list shows me all the topics) but no messages seem to be published. rostopic echo is not able to read any messages on any of the topics. However sometimes, -very sporadically- it does work. Have you ever run into these kind of problems?

[SalvoVirga]

I never used a USB/Ethernet adapter myself, so I cannot help you if the problem is that.
Nevertheless, make sure of the following:

• your Ethernet interface within Windows7 Embedded has the right IP.
  In your case this regards the Ethernet adapter, be sure that it has an IP in the same subnet as your ROS Master;
• that the .bashrc contains the right IPs;
• the config.txt in Sunrise Workbench has the right IP

If you are sure that all these points are correct and your roscore echo still doesn't give you anything, then I would say it is something relate to the USB NIC.
If that is the case, I cannot guide you any better. Personally I would not suggest you to use an adapter, from #7 it doesn't sound any easier or safer than the change I suggest in the wiki.

In particular

the installation of the USB3-Ethernet adapter somehow screwed up the configuration of the “Realtime OS Virtual Network Adapter”

scares be a bit. Once I made my cabinet unable to boot while messing around that “Realtime OS Virtual Network Adapter”, which is not nice (I have my back covered, so I can mess around).

Let me know if checking those three points your find out something!

[rbbg]

It is actually the exact same setup as discussed in #7, So the configuration has been largely working for quite a while. However, it doesn't seem to be very robust and a restart is often required to get it to work.

I have been doing a couple of more tests. First of all, when only running the ROSMonitor and a single roscore, there are no joint_states being published on the /iiwa/joint_states topic, but there are messages being published on the /iiwa/state/... topics so that threw me for a loop a bit.

Also, if you kill the roscore before stopping the application, it complains about active threads and often requires a reboot to work again as well. Did you ever have the situation where there seems to be a connection, but you cannot read any published messages? I have seen this happening at least a couple of times.

[SalvoVirga]

Oh, I see now! Sorry, I thought you meant in general all the 'state' topics had that behaviour.
/iiwa/joint_states is a bit different from the others. It is optional and it is regulated by the ROS param /iiwa/publishJointStates . (a bit like /iiwa/toolName allows to select the tool to use).
When you run either ROSMonitor or ROSSmartServo, that parameters is checked. If it is not set or set to false, the joint_state is not published, else it is.

You can define this parameter in your launch file accordingly to your needs, by default it is not set so if you just run roscore and one of the Sunrise applications joint_states won't be published.
Another case is if you run MoveIt!. MoveIt! uses the /joint_states topic for determining the current state information. So the internal publisher on the robot will still not publish, but MoveIt! itself will. I guess that is what you see sometimes happening.
The parameters /iiwa/publishJointStates is exactly meant to provide the joint_states information without using MoveIt! if that is needed.

On the second issue. I don't recall complains about active threads, but yes. Sometimes when you kill roscore, keep the Sunrise application running and start a new roscore; there seems to be a connection but somehow messages are not published. That is also a mystery to me, but honestly I never tried to dig into the reason behind this. It usually just requires to restart the Sunrise application, which takes few seconds (I guess that is what you meant and not a complete reboot of the robot). I will like to understand what is really happening, but it has a low priority right now.

Let me know if this clarify things a bit, I should probably write this down in the Wiki as well!

[gavanderhoorn]

Just curious: what was the reason for making not publishing JointState msgs the default? It's about the most basic type for a ROS robot driver to publish.

And since everything is namespaced, the risk of collisions with other publishers seems small, if not impossible.

[SalvoVirga]

We basically tried to follow what is done for the UR5 or the LBR 4+, i.e. having an hardware_interface that provides a JointPositionInterface and a JointStateInterface. That is what iiwa_hw does.
So when launching MoveIt! with the iiwa, JointState msgs are published by iiwa_hw.

Now, the iiwa and iiwa_stack are kinda an exception, since we have state publishers directly on board of the cabinet. We added the internal publisher for JointState and the relative parameters to allow that also for users that don't care/want to use MoveIt or iiwa_hw.
The question is, what is the best design to follow? Having the internal publisher running by default and the JointStateInterface in iiwa_hw being optional for MoveIt users? Having the internal publisher always running and no JointStateInterface in our hardware_interface? (that would depart from what is the current standard for hardware interfaces).

Suggestions and opinion are definitely welcome! (Finally someone actually helping improving our work!)

[rbbg]

Ah! That clarifies the bit about the joint_states not being published! However, I still sometimes have the situation where they are not published, even though the /iiwa/publishJointStates parameter is set to true. This happens after there has been an 'unclean' termination of the Sunrise application (happens to both the ROSMonitor and ROSSmartServo)

On the second issue. I don't recall complains about active threads, but yes. Sometimes when you kill roscore, keep the Sunrise application running and start a new roscore; there seems to be a connection but somehow messages are not published. That is also a mystery to me, but honestly I never tried to dig into the reason behind this. It usually just requires to restart the Sunrise application, which takes few seconds (I guess that is what you meant and not a complete reboot of the robot). I will like to understand what is really happening, but it has a low priority right now.

What I was talking about is when you kill the roscore and then kill the Sunrise app (while the roscore is still dead). The application doesn't seem to be terminated correctly and the error The task didnt terminate with 15.0s: Task thread group still has active threads. shows up on the HMI. Sometimes, this seems to interfere with the normal running of the application and I don't see any messages on the iiwa/joint_states topic (with the parameter set to true). The only thing that always fixes it is a controller reboot.

Anyhow, the other problem we were having was a seemingly correct connection to the roscore, but none of the topics were publishing messages. This turned out to be connected to a ntp_with_host setting to true while there was no ntp server running on the host. This caused the publish() not to go through. I will test this some more tomorrow but I'll leave this here for now in case anyone else runs into similar issues.

[SalvoVirga]

Anyhow, the other problem we were having was a seemingly correct connection to the roscore, but none of the topics were publishing messages. This turned out to be connected to a ntp_with_host setting to true while there was no ntp server running on the host. This caused the publish() not to go through. I will test this some more tomorrow but I'll leave this here for now in case anyone else runs into similar issues.

Very true, that is why I set the default value of ntp_with_host to false since #17 , someone else had the same problem. I told myself many times to write down something about the NTP in the Wiki, this is another one.

As for the rest. Yes, ROSSmartServo doesn't quit cleanly. Both me and @marcoesposito1988 tried all we could think of to be sure everything is disposed at the right time. I remember we achieved that for ROSMonitor, but in ROSSmartServo something hangs when it shouldn't.
But I don't recall having issues with iiwa/joint_states. I will try to replicate the issue in the next days, finding a free slot in the busy schedule of our beloved robot.

Give us more comments and updates if you have, and feel free to open a MR if you have a fix.

[gavanderhoorn]

@SalvoVirga wrote:

Now, the iiwa and iiwa_stack are kinda an exception, since we have state publishers directly on board of the cabinet. We added the internal publisher for JointState and the relative parameters to allow that also for users that don't care/want to use MoveIt or iiwa_hw.

I'm not sure, but you seem to be implying here that the ros_control compatible bits are "for MoveIt" only. That is definitely not the case. The joint_state_controller and joint_trajectory_controller expose the ROS standard JointMessage and JointTrajectoryAction interfaces, which can be very well used without any MoveIt in the loop (in fact, we do it often in ROS-Industrial, with Descartes fi).

The question is, what is the best design to follow? Having the internal publisher running by default and the JointStateInterface in iiwa_hw being optional for MoveIt users? Having the internal publisher always running and no JointStateInterface in our hardware_interface? (that would depart from what is the current standard for hardware interfaces).

Well. What sort of use of the bare driver (ie: without the ros_control parts layered on top) do you see right now / expect? If not much, then I'd remove the JointState publisher: users not wanting to use ros_control can use the various msgs in iiwa_msgs directly. All other users would use the ros_control managed JointState and JointTrajectoryAction interfaces.

---

I didn't include it above, but another option would be to migrate the Action server to the rosjava side as well. That would conceptually make sense given the fact that there is already a JointState publisher. There is an implementation of actionlib for rosjava (here), so the bare essentials are available, but a Java JointTrajectoryAction server would need quite some development to get it up to the level of the joint_trajectory_controller of ros_control (trajectory replacement, splining, etc).

[marcoesposito1988]

Well. What sort of use of the bare driver (ie: without the ros_control parts layered on top) do you see right now / expect?

I was using it for visual servoing: I only had a rosmaster, the robot_state_publisher to have the joints in TF, and a node running. Taking the joint_state_publisher out helped reduce the latency.

I know that this is quite an edge case, but if it does not disturb too much I would leave the joint_state publishing in.

[gavanderhoorn]

@marcoesposito1988 wrote:

I was using it for visual servoing: I only had a rosmaster, the robot_state_publisher to have the joints in TF, and a node running. Taking the joint_state_publisher out helped reduce the latency.

Are you talking about a separate joint_state_publisher?

[marcoesposito1988]

Yes: the standard MoveIt - ros_control configuration includes (in my experience) an hw interface (in our case iiwa_ros), then a joint_state_publisher and then robot_state_publisher on top. I had neither joint_state_publisher nor iiwa_ros running.

I was publishing cartesian commands directly to /iiwa/command...

[gavanderhoorn]

Without seeing the launch files this is a bit of a guess, but MoveIt by itself does not publish any joint states unless it is configured to do so.

A standard demo.launch will make it publish joint states, but that is actually the fake_controller that publishes those. The joint_state_publisher instance is then used to assemble a 'complete' state of the entire scene (could be multiple robots / controllers) by use of the source_list parameter.

In a setup with a real robot (ie: a ros_control hardware_interface publishing JointState msgs), there is no need for that joint_state_publisher instance (unless you have other sources of JointState msgs and you'd like to incorporate those in a 'unified JointState').

That's why I asked whether you were referring to a separate joint_state_publisher: it's not a standard component in a 'normal' (as far as those exist) moveit setup for a real-robot.

[rbbg]

As a closing remark, the issues we were experiencing were related to the NTP server and sometimes to ROSSmartServo not exiting cleanly.