Compress data before sending

[matteo-ronchetti]

Data that is sent to Python API is uncompressed, therefore connecting to a remote server running Carla requires a lot of bandwidth. In particular uncompressed camera data are very large, for example a 1920x1080 image occupies 8.3Mb.
It would be useful to have some compression before sending the data. A few ideas:

• Don't send unrequired channels, for example the A channel when it is not used
• Use lossless image compression on the single frames (for example PNG)
• Use lossless video compression

[soulslicer]

currently my solution is to run the carla ros-bridge on my server. and then produce a compressed stream to subscribe to.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[OlivierDelbeke]

I spent some time on this one, and implemented a JPEG compression/decompression serializer. For a 1280x1024 frame size, the required bandwidth drops from 91 Mb/s to 13 Mb/s, i.e. from an unreasonable value to a reasonable one.When the server and the client are on different machines of the same ethernet network (or probably also on machines with a good internet connection), Carla works as well as when everything runs on the same machine. The drawback is an additional delay of ~13 ms (10ms for compression and 3ms for decompression). If you're interested by this I would need some help to clean the code (I'm not a C++ expert). I based the change on 0.9.9.

[jimmyw404]

@OlivierDelbeke I'd be interested in the jpeg compression/decompression feature if you want to share it (maybe put it in a branch?)

[OlivierDelbeke]

@jimmyw404 Thanks for your interest. I tried to push it, but got a 403 error.

[OlivierDelbeke]

@jimmyw404 and everyone else interested, I forked the project to my account and pushed my changes to a branch called "experimental/image_compression". I'm not sure that this is the right way to go, but at at least, the code is available. Only Linux is supported, and you have to put the jpeg libraries somewhere in the UE4 plugin directory ("Dependencies")) to get it compiled, but that should be all.

[jimmyw404]

@OlivierDelbeke Looks good to me! I think this is something that should be integrated into CARLA as an option to the RGB cameras, but until then this will save me some time. I see that you persisted the alpha channel in the RGB images, which is probably a good idea to avoid changing the nature of the data, but is that used for anything? I was surprised to see it in the RGB camera.

[OlivierDelbeke]

The alpha channel is probably not necessary ; not sending it at all sounds like a good idea of improvement. The first thing the pygame-based carla clients do, is to strip that channel off (and it costs 3 ms per frame).

[TomCC7]

I wonder is there any update on this issue? Has similar functions been added to the new releases?

[OlivierDelbeke]

I don't think so. At least not in 0.9.13.

[PatrickFSLin]

@OlivierDelbeke
I am trying your your branch jpeg com/decom to Carla 0.9.13 and want to know the details about "you have to put the jpeg libraries somewhere in the UE4 plugin directory ("Dependencies")) to get it compiled"
Could you explain more about this step? Thanks

[kevin1zc]

@OlivierDelbeke I am trying your your branch jpeg com/decom to Carla 0.9.13 and want to know the details about "you have to put the jpeg libraries somewhere in the UE4 plugin directory ("Dependencies")) to get it compiled" Could you explain more about this step? Thanks

# ==============================================================================
# -- Get and compile libjpeg 8 ------------------------------
# ==============================================================================

LIBJPEG_VERSION=8
LIBJPEG_REPO=http://ijg.org/files/jpegsrc.v${LIBJPEG_VERSION}.tar.gz
LIBJPEG_BASENAME=jpeg-${LIBJPEG_VERSION}
LIBJPEG_INSTALL=${LIBJPEG_BASENAME}-install

LIBJPEG_INCLUDE=${PWD}/${LIBJPEG_BASENAME}-install/include/
LIBJPEG_LIB=${PWD}/${LIBJPEG_BASENAME}-install/lib/

if [[ -d ${LIBJPEG_INSTALL} ]] ; then
  log "Libjpeg already installed."
else
  log "Retrieving libjpeg."
  wget ${LIBJPEG_REPO}

  log "Extracting libjpeg."
  tar -xf jpegsrc.v${LIBJPEG_VERSION}.tar.gz
  mv ${LIBJPEG_BASENAME} ${LIBJPEG_BASENAME}-source

  pushd ${LIBJPEG_BASENAME}-source >/dev/null

  export CFLAGS="-fPIC"
  ./configure --prefix=${CARLA_BUILD_FOLDER}/${LIBJPEG_INSTALL} --enable-shared
  make -j$(nproc)
  make install

  popd >/dev/null

  rm -Rf jpegsrc.v${LIBJPEG_VERSION}.tar.gz
  rm -Rf ${LIBJPEG_BASENAME}-source
fi

mkdir -p ${LIBCARLA_INSTALL_SERVER_FOLDER}/lib/
mkdir -p ${LIBCARLA_INSTALL_SERVER_FOLDER}/include/
cp -p -r ${LIBJPEG_INCLUDE}/* ${LIBCARLA_INSTALL_SERVER_FOLDER}/include/
cp -p ${LIBJPEG_LIB}/*.so ${LIBCARLA_INSTALL_SERVER_FOLDER}/lib/

[PatrickFSLin]

Thanks, @kevin1zc

[PatrickFSLin]

@kevin1zc
I have tested deeply and got the question:
Should I use the specific version libjpeg v9e?
Because it always appears the message "Wrong JPEG library version library is 90 caller expects 80"
However, when I remove all libjpeg8 related .so files in /usr/lib/x86_64-linux-gnu
The message turns into " libjpeg.so.8: cannot open shared object file: No such file or directory"

Thanks

[kevin1zc]

@PatrickFSLin Looks like we should go for v8. I have updated my script above. Also, you need to add the dependency in <project>/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Carla.Build.cs: add this line

AddDynamicLibrary(Path.Combine(LibCarlaInstallPath, "lib", "libjpeg.so"));

inside the private void AddCarlaServerDependency(ReadOnlyTargetRules Target) function.

[PatrickFSLin]

@kevin1zc
Thanks, I have tested with your updates, and it seems to be one more step close.
However, now it appears a message "Bogus input colorspace"
very similar to https://www.cnblogs.com/oloroso/p/13323527.html
Not very sure that the message is casued by libjpeg-turbo8 as cannot be uninstalled in ubuntu
Any suggestions?

[kevin1zc]

@PatrickFSLin I don't have this error on my side. FYI, I'm using Ubuntu 20.04 and carla master branch.

[PatrickFSLin]

@kevin1zc
Thanks for your prompt response.
I will try it again on a refreshed [ubuntu + carla master branch]

[PatrickFSLin]

I have tried this method of image compression several times. However, only once I can successfully build up and run, but with lack of libjpeg.so.8 in /usr/lib/x86_64-linux-gnu and the side effect is that I cannot open "Ubuntu Settings" with UI.
The method requires JCS_EXT_BGRA as color input space quite fitting carla RGBA camera, and only libjpeg-turbo supports the extension of color space.
So I manually add libjpeg-turbo's .a and .so file into CarlaDependencies, and then I encountered another issue is that "Wrong JPEG library version: library is 62, caller expects 80"

Is there anyone tried image compression method successfully on Ubuntu 20.04 and carla master branch without any side effects?

[kevin1zc]

@PatrickFSLin That's weird since the code should not look for libjpeg under /usr/lib/x86_64-linux-gnu at all. Did you add this line AddDynamicLibrary(Path.Combine(LibCarlaInstallPath, "lib", "libjpeg.so")); at the correct position? (i.e. outside of IsWindows block)

[PatrickFSLin]

@kevin1zc Yes, I am quite sure that I add AddDynamicLibrary(Path.Combine(LibCarlaInstallPath, "lib", "libjpeg.so")); at else block of if (IsWindows(Target)). The only different thing is libjpeg.a required while building carla, and I manually add it into CarlaDependencies

I REALLY need this feature both for PC-PC and PC-(embedded platform) to test ADAS software
I've tried everything I knew and keep going to make it work

@kevin1zc Thanks for your guide for resolving several issues I've encountered

[PatrickFSLin]

@kevin1zc
Finally, I solved the issue.
First, I still think if you want to JPEF com/decom with OlivierDelbeke's implementation, you should put libjpeg-turbo (.so) into CarlaDependencies because JCS_EXT_RGBA is needed.
And I make the mistake for assuming libjpeg-turbo only depend on libjpeg62, so I build libjpeg-turbo with default cmake settings and hence get libjpeg.so.62 which cannot be compatible with LibCarla
So, the key step is cmake libjpeg-turbo with the flag -DWITH_JPEG8=1
Else remains the same, and DONE!