significant delay

[QuantuMope]

Hello Ravi,
I am using your ros package in an attempt to use openpose for real time skeleton estimation.

The problem is that I am experiencing significant delay between the realtime video feed and the outputted skeleton.
I see from your example gif on the front page README that you were more or less able to get realtime output.

Do you have any idea what could be causing this delay? I am using quite a powerful machine with Openpose 1.6 and a realsense camera. Aside from the delay, the framerate is stable. This leads me to think that this is not openpose related.

Any help would be greatly appreciated.
Thanks in advance.

[chrisywong]

I was just about to post about this problem too. I'm thinking the delay is from openpose itself (since I notice the lag there). Maybe the real time output from the example gif is using saved data rather than an actual real time feed?

This delay also produces another issue particular to the 3D projection and fast motions. If the motion is fast enough (e.g. swinging your arms quickly) that the delayed skeleton no longer overlaps with the person, the skeleton actually gets projected onto the background. What this tells me is that perhaps the data used for 3D projection in ros_openpose is using unsynced (or improperly synced) keypoints from openpose and the real time point cloud data.

E.g. in the screenshot below where I'm waving my arm, OpenPose is displaying an old image, which corresponds to the position of the skeleton, but the point cloud image is more updated and showing my arm already extended. You can't see as well but the skeleton keypoint is projected onto the background.

I'm trying to see if I can do anything with timestamps to try and make it work.

[QuantuMope]

Yes, this is pretty much exactly the problem I am having as well. If arm movements are too fast, the skeleton lags behind considerably as you've shown.

I was unaware that the 3D projection actually uses the current frame. That would indeed result in quite inaccurate results.
I can try to see if this is also the case on my end and report back.

If the delay is indeed solely due to openpose, it seems like there's no clear solution other than to find other more efficient methods for skeleton recognition. This is my first time using openpose but I was led to believe that it was more-or-less suitable for realtime operation.

[chrisywong]

From what I see, the delay isn't TOO bad (<1 sec). If we can somehow figure out the exact delay from OpenPose, we could feed the old colour/depth images and at least have them sync'd up.

Also I'm not 100% sure what I said is the case. This is just my deduction so far from digging into the code, but I'm not deeply familiar with it just yet.

[QuantuMope]

From openpose forum posts I've read so far, it seems like we should expect a 250 ms or less delay with a good enough GPU. I think the delay exceeds this though.

I did notice that in the code the ros header time is assigned by the output worker after the openpose computation. This would result in skeleton frames being published with a time stamp that is later than the actual input time which results in the delay/mismatch.

[ravijo]

Hello @QuantuMope and @chrisywong

Thank you very much for bringing up this issue.

I was using GTX 1080 with Intel i7 having a 16GB memory machine. My environment details can be found here

I don't remember but back then I compiled many packages (even librealsense too, maybe OpenCV with Cuda support too (unsure about OpenCV, I forgot)) from the source. In one of my projects, I have noticed the compiling packages from source gives better performance. So, please give it a try in this way if you get time.

Next, the 3D projection code is not optimized right now. You can see that for each detection keypoint in 2D space, the 3D point is calculated in a sequence. I did use OpenMP but I think that the overhead of creating threads is too much in our case. However, in principle, all 2D points can be converted to 3D using parallel computation to speed up the delay.

Another thing you assumed correctly is about the data used for 3D projection in ros_openpose. Right now, the latest available depth image is used for 3D projection. Please see here. In the header file, this is what happening. While developing this package, as two independent workers were created (op::WorkerProducer and op::WorkerConsumer), I assumed that it is more or less will work properly. I remember that in order to use the depth image corresponding to the color image, once I locked the depth image while acquiring the color image and later on used that depth image after receiving key points from OpenPose, it worked. I wasn't satisfied with the overall performance.

In the past, you may notice in older commits, that I used message_filters::TimeSynchronizer from ROS but it actually added delay because it needs to wait until it receives frames with the same timestamp. Please see here. In this implementation, we need to provide queue size to the TimeSynchronizer. At that time, I wanted to make the queue size as small as possible but it turned out that this number somehow depends on the processing capabilities of the machine. I do remember that I tried message_filters::sync_policies::ApproximateTime given by ROS too but I wasn't satisfied by the overall performance.

Other than this, once I remember one person said that opWrapper.waitAndEmplace() style is giving better performance. You can see an example code here. I have not checked it but you can easily code it in ros_openpose with minimal effort. Please give it a try if necessary.

Thank you so much. Please let me know the outcome.

[QuantuMope]

Thanks for the reply @ravijo

I'm surprised message_filters::sync_policies::ApproximateTime didn't result in good performance. In my experience, messages are received at a stable rate so long as the image and depth stream have a similar fps.

I will try to use opWrapper.waitAndEmplace() and see if there is a noticeable boost in performance.

[chrisywong]

Have either of you noticed that the skeleton and point cloud are not aligned in RViz? All points seem to be shifted to the left by about 0.02m. I just want to see if it's a general thing or if it's something related to my setup and/or parameters. I'm using /camera/aligned_depth_to_color/image_raw for my Intel RealSense D435.

[ravijo]

@chrisywong

Hi, frankly speaking, while writing this package, my application was not expecting large movements and such nice precision. I did not verify this 2cm shift.

However, what I have seen that there were some similar-looking TF frames for the camera. Right now, I am using camera_depth_optical_frame as shown here.

You can echo all TF frames for the camera and there is a possibility that you can find similar TF (no relative rotation and no shift in the z-direction). If so, please change this parameter. I am not so sure though.

The other hack can be publishing a child TF of camera_depth_optical_frame with no rotation but only 2cm shift in x (or y) direction as you noticed.

[QuantuMope]

@chrisywong
Sounds like your images are published in the frame of camera_color_optical_frame but the transforms are being done with respect to camera_depth_optical_frame. The 0.02m shift makes sense as the depth lens is more or less 2cm away from the color lens on a realsense. Since you are already using /camera/aligned_depth_to_color/image_raw I would change the frame to camera_color_optical_frame in the location that Ravi linked above, which should solve the problem.

[ravijo]

@QuantuMope Perfect 👍

This is what I was trying to say. It should work. Thanks a lot.

[QuantuMope]

Hello @chrisywong and @ravijo.

I apologize for the late update.
Just wanted to update that I created a new ros wrapper using message_filters::sync_policies::ApproximateTime and opWrapper.waitAndEmplace() and was able to reduce latency considerably. On my system, I am experiencing an average of 100 ms per frame. I think if real-time performance is important, it is worth going with a synchronous approach. I will close this issue since the problem has at least been solved for me.

[ravijo]

Hi @QuantuMope

Thank you so much for the update. I am glad that the approach worked well on your side.

By the way, would you mind making a pull request to share your code?

I am thinking to add your contribution to the dev branch by acknowledging your kind efforts too.

[QuantuMope]

Sure but I just wanted to clarify two things.

1. The code I wrote does not reuse any of the original ros_openpose code. Since your code is asynchronous, I thought it would be faster to just rewrite the code from scratch. Still it could prove useful to have two separate modules. One that prioritizes latency (synchronous) and one that prioritizes fps (asynchronous).
2. I wrote the ros wrapper in Python for speedy prototyping.

Given this, let me know if you'd still like me to submit a pull request. If you'd like the code to stay C++ (since using openpose through Python requires additional compilation steps), if I have some time, I can even convert the code to C++ and submit a pull request then. Just let me know.

[ravijo]

I understand.

I think it is all right to submit a pull request for the Python version only. I will create a new branch, say python, and will be adding to it.

[chrisywong]

Hi @QuantuMope, I'd definitely be interested in trying out your synchronized version! I'm not 100% sure which one I'll end up using for my application, but it'd be good to compare to see which one is more suitable for me.

[ravijo]

@chrisywong

If you are in hurry, implementing opWrapper.waitAndEmplace() and message_filters::sync_policies::ApproximateTime is rather easy once you already have ros_openpose. I did that in the past but did not push it to GitHub.

Meanwhile, we can wait for @QuantuMope for a pull request.

[QuantuMope]

I apologize for the delay. I am away from my lab for a few days and I want to fully bug-proof the code on a realsense camera before submitting a pull request. I should be back in my lab and be able to get my hands on the camera come this Thursday and will submit it then.

[ravijo]

Have a nice relaxing break @QuantuMope

[QuantuMope]

@ravijo @chrisywong
Just an FYI that the pull request is submitted. Sorry for the delay, got access to a Realsense just today.

From tests, latency is greatly reduced on synchronous version but has worse frame rate than the original wrapper so the best version to use will depend on the usage case.

[ravijo]

Thank you so much @QuantuMope

I sincerely acknowledge your contribution.

The latest commit 0ed2a72 now includes support for the synchronous version provided by you.

In the documentation, I quickly added the relevant information under Implementation Versions Info..

Thank you again.

@chrisywong

You may like to pull the latest changes from the repository. Please let us know if you face any issues.

[QuantuMope]

Hello @ravijo,

Just wanted to notify that I plan on implementing the synchronous node in C++ when I have some free time in the coming weeks. Will submit a pull request when I have it complete and tested. It may be worth keeping both the C++ and Python versions though in case users want to modify the nodes in their language of choice.

[ravijo]

Hello @QuantuMope

Thank you so much.

Yes, I agree with your suggestion.

Thanks again.

[chrisywong]

Hi all,

I'm trying to use the synchronous script but am running into issues. When I try to run the launch file, I get this error:

[ERROR] [1617024825.486954]: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder?
ImportError: cannot import name pyopenpose

I did enable BUILD_PYTHON when building OpenPose (both v1.7.0 and v1.6.0), as:

$ find / -iname "pyopenpose*"
/usr/local/python/pyopenpose.cpython-36m-x86_64-linux-gnu.so
/usr/local/python/pyopenpose.cpython-37m-x86_64-linux-gnu.so
/usr/local/python/openpose/pyopenpose.cpython-36m-x86_64-linux-gnu.so
/usr/local/python/openpose/pyopenpose.cpython-37m-x86_64-linux-gnu.so
/usr/local/python/openpose/CMakeFiles/pyopenpose.dir

This occurs on the line from openpose import pyopenpose as op in ros_openpose_synchronous.py with the default path <arg name="py_openpose_path" default="/usr/local/python/" />

Any ideas?

Would this somehow be a conflict between having both Python 2.7 and 3.6 on my system?

[luorui93]

Hi all,

I'm trying to use the synchronous script but am running into issues. When I try to run the launch file, I get this error:

[ERROR] [1617024825.486954]: OpenPose library could not be found. Did you enable `BUILD_PYTHON` in CMake and have this Python script in the right folder?
ImportError: cannot import name pyopenpose

I did enable BUILD_PYTHON when building OpenPose (both v1.7.0 and v1.6.0), as:

$ find / -iname "pyopenpose*"
/usr/local/python/pyopenpose.cpython-36m-x86_64-linux-gnu.so
/usr/local/python/pyopenpose.cpython-37m-x86_64-linux-gnu.so
/usr/local/python/openpose/pyopenpose.cpython-36m-x86_64-linux-gnu.so
/usr/local/python/openpose/pyopenpose.cpython-37m-x86_64-linux-gnu.so
/usr/local/python/openpose/CMakeFiles/pyopenpose.dir

This occurs on the line from openpose import pyopenpose as op in ros_openpose_synchronous.py with the default path <arg name="py_openpose_path" default="/usr/local/python/" />

Any ideas?

Would this somehow be a conflict between having both Python 2.7 and 3.6 on my system?

First make sure you've successfully built the python openpose. Check the if there is a python folder in your openpose build folder.
If you've already built it without problems, to use the default "/usr/local/python" as the openpose module path you have to do

sudo make install

in the build folder to install the modules.
Or you can just change the default path in launch file to something like "path_to_your_openpose/build/python"

And if your default python for ROS is python2, you have to make sure you built openpose with python2 too. The default one is python3, you have to change the cmake flag in configuration.
Refer to this link:
https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/03_python_api.md#compatibility

[chrisywong]

I did do sudo make install (that's why I have those files located in /usr/local/python/). Pointing directly to ~/catkin_ws/src/openpose/build/python/openpose/ did not work either.

[luorui93]

I did do sudo make install (that's why I have those files located in /usr/local/python/). Pointing directly to ~/catkin_ws/src/openpose/build/python/openpose/ did not work either.

Then it's probably you built a python 3 module but your ROS is looking for a python 2 version. Check the link I pasted before and make sure you build with Python2 executable and libraries.

[chrisywong]

Tried building Python 2 vs 3 and still nope. Even pointing to build folder specifically for python 2.7 (and verified version in CMakeCache.txt)

[luorui93]

I did do sudo make install (that's why I have those files located in /usr/local/python/). Pointing directly to ~/catkin_ws/src/openpose/build/python/openpose/ did not work either.

I think you should point to absolute_path/catkin_ws/src/openpose/build/python . Also don't use "~", type the absolute path like /home/user_name/catkin_ws/...... Python won't interpret "~"

And maybe you can double check if you have a similar layout like this in your build folder if you run $ ls python/openpose

$ ls openpose/
CMakeFiles           __init__.py   Makefile
cmake_install.cmake  __init__.pyc  pyopenpose.so

[chrisywong]

Yes, I am using the absolute path. I just wrote ~ here for convenience.

Neither py2.7 nor py3.6 build/python/openpose/ folders has __init__.pyc, while py2.7 has pyopenpose.so but py3.6 has pyopenpose.cpython-36m-x86_64-linux-gnu.so (which, if I rename/copy/symlink to pyopenpose.so still doesn't help)

[luorui93]

pyc file is not important, as long as you have __init__.py.
Looks like your module should be built correctly.
By the way, my last comment not only points out the absolute path, you should also only set the path to .../build/python instead of .../build/python/openpose.
If that's still correct, I'm out of ideas...

[ravijo]

@chrisywong

Sorry to hear about the issue.

Thank you @luorui93 for providing detailed explanations to fix the problem. You have clearly stated all possible ideas! Thanks again.

In the case of the ImportError exception, I always try to import the module manually in python interpreter first to make debugging a lot easier.

First, please switch to tutorial_api_python folder provided by OpenPose team and then import pyopenpose as shown below:

ravi@dell:~$ cd /home/ravi/openpose/examples/tutorial_api_python
ravi@dell:~$ python
Python 2.7.17 (default, Feb 27 2021, 15:10:58) 
[GCC 7.5.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('../../python')
>>> from openpose import pyopenpose as op
>>> exit()

Above, I assumed that you are going to use pyopenpose module with standard ROS. By saying standard ROS, I just mean that ROS which by default works with Python 2 version. However, you are in Ubuntu 20, you must be using ROS with Python3. So in such case, please use python3 to import pyopenpose as shown below:

ravi@dell:~/openpose/examples/tutorial_api_python$ python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('../../python')
>>> from openpose import pyopenpose as op
>>> exit()

To see all python versions available, please type python and then press the tab key twice. You may see similar results:

ravi@dell:~$ python
python              python2-pasteurize  python3.6m-config   python3m-config
python2             python2-qr          python3.7           python-config
python2.7           python3             python3.7m          pythontex
python2.7-config    python3.6           python3.8           pythontex3
python2-config      python3.6-config    python3-config      
python2-futurize    python3.6m          python3m            

If you can not import pyopenpose as shown above, I highly doubt that the issue is related to OpenPose installation/building. Please follow the installation documentation properly.

Next, please read the following:

• Please identify the python version of your ROS. You must use the same python for building OpenPose (as said by @luorui93 too). To find out your ROS python version, PYTHON_EXECUTABLE, PYTHON_INCLUDE_DIR, and PYTHON_LIBRARY variables, you can write the following code in any of your working ROS packages:

  import rospy
  import sys
  from distutils.sysconfig import get_python_inc
  import distutils.sysconfig as sysconfig
  
  if __name__ == '__main__':
      rospy.init_node('example_node', anonymous=True)
      print(sys.version) # may print as '2.7.17 (default, Feb 27 2021, 15:10:58) \n[GCC 7.5.0]'
      print(sys.executable)
      print(get_python_inc())
      print(sysconfig.get_config_var('LIBDIR'))
  

  Then please execute the script as rosrun my_package my_script.py
• Last but not least, I also noticed that you have kept OpenPose inside the ROS workspace. Inside a ROS workspace, we are required to execute source devel/setup.bash. This changes many environment variables. I suggest you keep OpenPose in $HOME or somewhere else so that OpenPose installation doesn't get affected by ROS workspace.

PS: I have reopened this issue. However, I kindly request you to open a new issue with these details. Because this issue, i.e., #31 was originally created for discussion on processing delay.

[chrisywong]

Solved this issue, see #41.