rviz + moveit_setup_assistant on osx need macos_min_version=10.9 & MACOSX_DEPLOYMENT_TARGET=10.9

[sonelu]

After the full rebuild #62 rviz is broken.

When starting a launch file or simply rviz, there is a blank window and the process hangs:

The bug was originally discussed in #63.

Thanks to @jc-bao comment by downgrading to rviz 1.14.4 the problem goes away.

[wolfv]

Thanks again for testing and all! I can locally also reproduce this issue. I can get further when I add -s "" to the command line to disable the splash screen. But still, no proper window is created. I am very unsure where the bug is ... in RViz or in the udnerlying Qt. Maybe @rhaschke has ideas?

EDIT Here is my log:

(ros) wolfv@Wolfs-Mac-mini ~ % rviz --verbose -s ""
[ INFO] [1615461680.098935000]: rviz version 1.14.5
[ INFO] [1615461680.098982000]: compiled against Qt version 5.12.9
[ INFO] [1615461680.098991000]: compiled against OGRE version 1.10.12 (Xalafu)
[ INFO] [1615461680.106224000]: Forcing OpenGl version 0.
[ INFO] [1615461681.019575000]: Stereo is NOT SUPPORTED
[ INFO] [1615461681.019629000]: OpenGL device: Intel(R) UHD Graphics 630
[ INFO] [1615461681.019892000]: OpenGl version: 2.1 (GLSL 1.2).
qt.qpa.window: <QNSWindow: 0x7fea6592d6f0; contentView=<QNSView: 0x7fea6592d430; QCocoaWindow(0x7fea6592a810, window=QWidgetWindow(0x7fea65929c80, name="QWidgetClassWindow"))>> has active key-value observers (KVO)! These will stop working now that the window is recreated, and will result in exceptions when the observers are removed. Break in QCocoaWindow::recreateWindowIfNeeded to debug.
qt.qpa.window: <QNSWindow: 0x7fea6455b890; contentView=<QNSView: 0x7fea6455b4c0; QCocoaWindow(0x7fea6455b3a0, window=QWidgetWindow(0x7fea64556b80, name="QWidgetClassWindow"))>> has active key-value observers (KVO)! These will stop working now that the window is recreated, and will result in exceptions when the observers are removed. Break in QCocoaWindow::recreateWindowIfNeeded to debug.

[rhaschke]

The only thing that changed w.r.t. rendering since 1.14.4 is ros-visualization/rviz#1588.
Could you check whether glGetString() doesn't return? Or, maybe, the char* cast doesn't work on MacOS?

[rhaschke]

Given your log, @wolfv, this looks like a Qt issue. Did you change the Qt version as well? Could you try building rviz 1.14.4 in the current environment to exclude that rviz is the culprit?

[wolfv]

Thanks @rhaschke. Indeed I need to try rebuilding the older RViz. So far I am running into another issue with some resource (the Liberation font) being already added...

[rhaschke]

Do you have an error message? There are two fontdef files in rviz/ogre_media/fonts, one was intended for Ogre 1.9 and the other one for Ogre >= 1.10, which uses another syntax. Maybe, in your config, both are considered?

[Tobias-Fischer]

I think this issues is fixed by setting mesa_workaround to false for OSX & Windows. See #70 but awaiting confirmation from @wolfv

[wolfv]

I tried this in rviz that I built in a catkin workspace and it didn't seem to fix it yet. Did you also try locally?

[Tobias-Fischer]

I did. It works fine for me:

[ INFO] [1615499689.750586000]: rviz version 1.14.5
[ INFO] [1615499689.750767000]: compiled against Qt version 5.12.9
[ INFO] [1615499689.750796000]: compiled against OGRE version 1.10.12 (Xalafu)
[ INFO] [1615499689.759184000]: Forcing OpenGl version 0.
[ INFO] [1615499690.885723000]: Stereo is NOT SUPPORTED
[ INFO] [1615499690.885790000]: OpenGL device: Intel(R) Iris(TM) Graphics 6100
[ INFO] [1615499690.885822000]: OpenGl version: 2.1 (GLSL 1.2).
Debug:   class_loader.ClassLoader: Destroying class loader, unloading associated library...

[Tobias-Fischer]

(that's build using boa)

[Tobias-Fischer]

I uploaded the package here if you want to try: https://anaconda.org/TobiasRobotics/ros-noetic-rviz/1.14.5/download/osx-64/ros-noetic-rviz-1.14.5-py38h5347e94_5.tar.bz2

[rhaschke]

Given the fact that your OpenGL version is 2.10 only, mesa_workaround should be false anyway (because 210 < 320), I cannot believe that #70 will fix the issue. Also, I'm confused that you deleted ogre_media/fonts/ogre1.10.fontdef instead of ogre_media/fonts/ogre1.9.fontdef, because you are building against Ogre 1.10.

[wolfv]

Indeed, your package works for me locally, too @Tobias-Fischer! Awesome :) Maybe my local catkin build was somehow different from the proper build.

@rhaschke I think talking to opengl directly as is done here is not really cross-platform anyways (I thought this is usually abstracted away through OGRE, right? We had compile issues with the glGetVersion and GL_VERSION symbols on Windows, before).

[wolfv]

@rhaschke, I have prototyped a Github Actions pipeline to continously test RViz builds on top of these conda packages: https://github.com/RoboStack/rviz/runs/2076163521?check_suite_focus=true

You can see the code here: https://github.com/RoboStack/rviz/blob/noetic-add-test/.github/workflows/ci.yml and the environment setup file here: https://github.com/RoboStack/rviz/blob/noetic-add-test/.github/workflows/ci_env.yml

As you can see, these conda packages could make cross-platform testing much simpler. Would you be interested in such an action upstream?

[rhaschke]

Sure, having github actions available for cross-platform testing would be great. However, for the rviz repo, these actions should be independent of conda, shouldn't they? By the way, I will be happy to apply your patches upstream as well. Please file PRs.

[wolfv]

But we need to get all dependencies of RViz from somewhere. Using conda packages for the dependencies (Qt, OGRE, ros-noetic-std-msgs ...) makes this fast and simple (since they are binary packages, it takes no time at all to install them, vs building all deps from source).

[Tobias-Fischer]

@rhaschke - my best guess is that one of the functions never returns and therefore rviz gets stuck at the splash screen. But really just a guess.

Re the fontdef - I was meant to open a bug report upstream about this. The 1.10 fontdef doesn't work, I've tried it and it results in some error (can't remember which one). My suspicion is that the new fontdef is only compatible with OGRE >1.10 but not =1.10. I tried to dig to see when they did the relevant change but couldn't trace it down exactly. Long story short, we need the old fontdef and that one is working nicely ..

[wolfv]

@rhaschke any ideas on how to fix these tests? https://github.com/RoboStack/rviz/runs/2095215307?check_suite_focus=true they all seem to fail... do I need to set some other variable?

[rhaschke]

The test target isn't called test for catkin, but run_tests. So change to ninja run_tests

[wolfv]

wonderful, that worked great for OS X and Linux: https://github.com/RoboStack/rviz/runs/2096713763?check_suite_focus=true

Now I just need to fix Windows!

[Tobias-Fischer]

So oddly enough rviz still cannot be started with the latest build from robostack, but it works fine with my build from https://anaconda.org/TobiasRobotics/ros-noetic-rviz/1.14.5/download/osx-64/ros-noetic-rviz-1.14.5-py38h5347e94_5.tar.bz2

Any idea how to debug this further?

[Tobias-Fischer]

@sonelu - could you try https://anaconda.org/TobiasRobotics/ros-noetic-rviz/1.14.5/download/osx-64/ros-noetic-rviz-1.14.5-py38h5347e94_5.tar.bz2 and see whether it works for you?

[sonelu]

Yes, that one works for me too. Any idea what's different from the one on robostack channel?

[sonelu]

Actually there are 3 of them in the channel and two have the same build number 6:

(ros1) Alex3-2:~ Alex$ conda search ros-noetic-rviz=1.14.5
Loading channels: done
# Name                       Version           Build  Channel             
ros-noetic-rviz               1.14.5  py38hb993ecd_6  robostack           
ros-noetic-rviz               1.14.5  py38he3dbe5f_5  robostack           
ros-noetic-rviz               1.14.5  py38he3dbe5f_6  robostack  

I'm not sure how conda picks them, probably by date?

(ros1) Alex3-2:~ Alex$ conda search ros-noetic-rviz=1.14.5 --info
Loading channels: done
ros-noetic-rviz 1.14.5 py38hb993ecd_6
-------------------------------------
file name   : ros-noetic-rviz-1.14.5-py38hb993ecd_6.tar.bz2
name        : ros-noetic-rviz
version     : 1.14.5
build       : py38hb993ecd_6
build number: 6
size        : 2.6 MB
subdir      : osx-64
url         : https://conda.anaconda.org/robostack/osx-64/ros-noetic-rviz-1.14.5-py38hb993ecd_6.tar.bz2
md5         : f967bcebce96342ceb14da255ce0c0dc
timestamp   : 2021-03-15 01:16:05 UTC
dependencies: 
  - __osx >=10.14
  - ...

ros-noetic-rviz 1.14.5 py38he3dbe5f_5
-------------------------------------
file name   : ros-noetic-rviz-1.14.5-py38he3dbe5f_5.tar.bz2
name        : ros-noetic-rviz
version     : 1.14.5
build       : py38he3dbe5f_5
build number: 5
size        : 2.6 MB
subdir      : osx-64
url         : https://conda.anaconda.org/robostack/osx-64/ros-noetic-rviz-1.14.5-py38he3dbe5f_5.tar.bz2
md5         : b4c9c138584a3202998e6a8b5cf8cdc4
timestamp   : 2021-03-08 01:56:04 UTC
dependencies: 
  - __osx >=10.14
  - ...

ros-noetic-rviz 1.14.5 py38he3dbe5f_6
-------------------------------------
file name   : ros-noetic-rviz-1.14.5-py38he3dbe5f_6.tar.bz2
name        : ros-noetic-rviz
version     : 1.14.5
build       : py38he3dbe5f_6
build number: 6
size        : 2.6 MB
subdir      : osx-64
url         : https://conda.anaconda.org/robostack/osx-64/ros-noetic-rviz-1.14.5-py38he3dbe5f_6.tar.bz2
md5         : e81fe44033de5b1b792b77178e9e112d
timestamp   : 2021-03-12 11:19:31 UTC
dependencies: 
  - __osx >=10.14
  - ...

[Tobias-Fischer]

Hi @wolfv - bad news for you ;). I had an intuition that I hadn't upgraded boa on my Mac for a while. So I built rviz 1.14.5 with this commit from Dec 7 2020 and rviz starts fine. After upgrading the latest commit on master, rviz doesn't start anymore. How weird! Any clue what could have changed?

[wolfv]

@Tobias-Fischer can you let me know the conda-build version of the environment that was working? For the boa commit you indicated I had to downgrade conda-build to 3.20.5 (from 3.21.4)

[wolfv]

I could reproduce the (working) build with the older boa. That's a really lucky coincidence!

ok, so I think one change is also that we're loading virtual package in boa with the later release.

The earlier boa picks up build number 3 from robostack, while the later release picks up build number 5. This is related to the __osx package that is not available in the older boa build.

Now the question is:

• is the bug related to the other rebuilt packages (besides some message packages, I don't see many differences) -- we coudl remove the virtual packages from boa - master and see if that works then
• the bug could be related to some CONDA_BUILD_SYSROOT / MACOSX_DEPLOYMENT_TARGET stuff that is more properly handled with the later boa master... but maybe the behavior from before was somehow better
• something else ...

[wolfv]

I back ported virtual packages to the old boa, and teh resulting rviz still works (compiled with otherwise rebuilt packages).

[wolfv]

Checking the two bin/rviz binaries shows the following differences with otool -l bin/rviz

Not working ... 

Load command 7
          cmd LC_LOAD_DYLINKER
      cmdsize 32
         name /usr/lib/dyld (offset 12)
Load command 8
     cmd LC_UUID
 cmdsize 24
    uuid FF58DBA6-0CD2-39CF-AED9-49180FE51FE6
Load command 9
      cmd LC_BUILD_VERSION
  cmdsize 32
 platform 1
    minos 10.14
      sdk 11.1
   ntools 1
     tool 3
  version 530.0
Load command 10
      cmd LC_SOURCE_VERSION
  cmdsize 16
  version 0.0
Load command 11
       cmd LC_MAIN
   cmdsize 24
  entryoff 7760
 stacksize 0
Load command 12


Working ... 

Load command 7
          cmd LC_LOAD_DYLINKER
      cmdsize 32
         name /usr/lib/dyld (offset 12)
Load command 8
     cmd LC_UUID
 cmdsize 24
    uuid 69D7580D-1A9B-31AE-9C27-9E72A0A99E65
Load command 9
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.9
      sdk 11.1
Load command 10
      cmd LC_SOURCE_VERSION
  cmdsize 16
  version 0.0
Load command 11
       cmd LC_MAIN
   cmdsize 24
  entryoff 7760
 stacksize 0

We could try to compile with new boa and MACOSX_DEPLOYMENT_TARGET set to 10.9... I'll have to stop for tonight but will take this up tomorrow.

[wolfv]

I tried setting this in conda_build_cofnig.yaml:

# Project overrides
macos_min_version:             # [osx]
  - 10.9                      # [osx]
MACOSX_DEPLOYMENT_TARGET:      # [osx]
  - 10.9                      # [osx]

and built again with latest boa and it seems to work. Can you try that, too @Tobias-Fischer ?

[wolfv]

in that case, the otool -l result is again

     cmd LC_UUID
 cmdsize 24
    uuid 1C015443-5AE9-31A5-9447-FEFF9AC0A396
Load command 9
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.9
      sdk 11.1

[Tobias-Fischer]

Yes, can confirm that setting the version to 10.9 makes it work (can't make sense of it though - can you?)

[wolfv]

@Tobias-Fischer thanks a lot for checking! I don't know either what's' going on exactly.

We can do two more things:

• check with conda-build if boa does something differently / wrong wrt to these flags (although I somewhat doubt it, but maybe)
• try a couple of different deployment target values

Maybe there is something in the ${CMAKE_ARGS} that we added that also influences some other stuff down the line in the cmake files of either ROS or Qt, or OGRE?

[Tobias-Fischer]

Quick test - I removed passing the ${CMAKE_ARGS} and setting the osx version back to 10.14, and it still does not work.

[wolfv]

thanks. yeah, setting the deployment target will add some environment variables that should change some compiler flags etc.
I'll ask on the conda-forge channel if someone has seen something like this before.

We can have the deployment target more specific (either only for the packages that need it, or we can add a conda_build_config specific to rviz to set it to the lower osx version).

What do you think?

[Tobias-Fischer]

Hmm .. ideally we wanna find out what's going on (as we'll likely run into a similar problem in the future), but for now yeah we can add a conda_build_config specific to rviz to have a working version in our channel.

[sonelu]

@sonelu - could you try https://anaconda.org/TobiasRobotics/ros-noetic-rviz/1.14.5/download/osx-64/ros-noetic-rviz-1.14.5-py38h5347e94_5.tar.bz2 and see whether it works for you?

@Tobias-Fischer btw: after installing the 1.14.5 from the tar above, although rviz works, I'm getting inconsistencies in the conda environment and I cannot do much there (ex. add other packages). It takes a very long time to check the consistency when installing new packages and then comes with a huge list of incompatibilities (I mean almost everything seems to be incompatible with everything...). I needed to downgrade to the 1.14.4 from robostack to be able to keep working.

[Tobias-Fischer]

@sonelu - I uploaded https://anaconda.org/TobiasRobotics/ros-noetic-rviz/1.14.6/download/osx-64/ros-noetic-rviz-1.14.6-py38h7870788_8.tar.bz2 which shouldn't have these issues. Note that this is still a workaround, we'll need to think about a proper fix.

[sonelu]

@Tobias-Fischer I tried that version and is not working on my machine. if I start rviz from a launch file I don't get any errors, but the rivz window does not show up. And killing the launch with Ctrl-C escalates for the rviz process with SIGTERM to close down, which is a bad sign.

If I run rviz on its own, there is the same empty window as at the top of this thread.

I've seen there is a 1.14.6 on the RoboStack too. I have not checked that one yet. Do you want me to check it?

For the time being the only one that works on my machine is the 1.14.4 (build py38h38af97b_3).

[Tobias-Fischer]

Maybe I uploaded the wrong version, sorry. I'll double check Monday.

[Tobias-Fischer]

Sorry - indeed it was the wrong version. This one should work: https://anaconda.org/TobiasRobotics/ros-noetic-rviz/1.14.6/download/osx-64/ros-noetic-rviz-1.14.6-py38ha61b876_8.tar.bz2

[sonelu]

@Tobias-Fischer that one works on my machine.

[Tobias-Fischer]

Great, thanks for testing @sonelu!

[Tobias-Fischer]

I uploaded this package to the robostack channel while we don't have a proper solution @wolfv. Have you had any further thoughts what the best way forward is? Modify vinca to create a conda_build_config just for rviz with 10.9 as macos version?

[Tobias-Fischer]

The same issue occurs with moveit_setup_assistant - see #132. Any more thoughts @wolfv?

[traversaro]

fyi @nkalupahana i guess the problems you saw could be related to this.

[Tobias-Fischer]

This should not happen in the future anymore due to c2e2640

However, there might be other GUI applications affected that we don't yet know about which might need to be included in that hack

[Tobias-Fischer]

Let's keep track of the new issues in #187