New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catastrophic 2.8 and 2.10 x.plot failure on our new CentOS 7 server #230

Closed
jypeter opened this Issue Aug 3, 2017 · 37 comments

Comments

Projects
None yet
6 participants
@jypeter

jypeter commented Aug 3, 2017

@doutriaux1 and @aashish24 the following very simple plot works fine on our old servers, but crashes python in a most violent way (with a VERY verbose output, not the usual simple core dumped) on our new CentOS 7 servers, with both 2.8 and 2.10

No problem with cdms2 in other non graphical scripts. I see lots of references to vtk in the long crash output I get. Note that, as usual, I'm connected to the servers with a VM. But with the same VM the plot works fine on the old servers, and fails on the new ones

(cdatm15) bash-4.1$ python
Python 2.7.13 | packaged by conda-forge | (default, May  2 2017, 12:48:11) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np, vcs
>>> i10 = np.identity(10)
>>> x = vcs.init()
>>> x.plot(i10)
<vcs.displayplot.Dp object at 0x2aded0d1fd70>
>>> 

Old servers

(cdatm15) bash-4.1$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.9 (Santiago)

New servers

jypeter@obelix5 - ...jypeter - 43 >cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core) 

when I type x.plot(i10), a graphical window opens, and then quickly closes, and my terminal is swamped by the following output

>>> x.plot(i10)
*** Error in `python': free(): invalid pointer: 0x00002ab70e702f60 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c503)[0x2ab6c16e2503]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/./libstdc++.so.6(_ZNSs9_M_mutateEmmm+0x189)[0x2ab6dd6f82b9]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/./libstdc++.so.6(_ZNSs15_M_replace_safeEmmPKcm+0x1e)[0x2ab6dd6f872e]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/./libstdc++.so.6(_ZNSs7replaceEmmPKcm+0x5f)[0x2ab6dd6f984f]
/usr/lib64/dri/swrast_dri.so(+0x3e7642)[0x2ab70b972642]
/usr/lib64/dri/swrast_dri.so(+0x3e55a4)[0x2ab70b9705a4]
/usr/lib64/dri/swrast_dri.so(+0x67cd0e)[0x2ab70bc07d0e]
/usr/lib64/dri/swrast_dri.so(+0x675240)[0x2ab70bc00240]
/usr/lib64/dri/swrast_dri.so(+0x6608d6)[0x2ab70bbeb8d6]
/usr/lib64/dri/swrast_dri.so(+0x323602)[0x2ab70b8ae602]
/usr/lib64/dri/swrast_dri.so(+0x20eb41)[0x2ab70b799b41]
/usr/lib64/dri/swrast_dri.so(+0x1f8219)[0x2ab70b783219]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/python2.7/site-packages/vtk/../../../libvtkRenderingOpenGL-7.1.so.1(_ZN17vtkOpenGLRenderer5ClearEv+0x216)[0x2ab6fa500d14]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/python2.7/site-packages/vtk/../../../libvtkRenderingOpenGL-7.1.so.1(_ZN15vtkOpenGLCamera6RenderEP11vtkRenderer+0x6a4)[0x2ab6fa4a2cd6]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/python2.7/site-packages/vtk/../../../libvtkRenderingCore-7.1.so.1(_ZN11vtkRenderer12UpdateCameraEv+0x16d)[0x2ab6ea5c47bd]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/python2.7/site-packages/vtk/../../../libvtkRenderingOpenGL-7.1.so.1(_ZN17vtkOpenGLRenderer12DeviceRenderEv+0x152)[0x2ab6fa4fb40e]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/python2.7/site-packages/vtk/../../../libvtkRenderingCore-7.1.so.1(_ZN11vtkRenderer6RenderEv+0x975)[0x2ab6ea5c424b]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/python2.7/site-packages/vtk/../../../libvtkRenderingCore-7.1.so.1(_ZN21vtkRendererCollection6RenderEv+0xc9)[0x2ab6ea5c2007]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/python2.7/site-packages/vtk/../../../libvtkRenderingCore-7.1.so.1(_ZN15vtkRenderWindow14DoStereoRenderEv+0xfb)[0x2ab6ea5db8e5]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/python2.7/site-packages/vtk/../../../libvtkRenderingCore-7.1.so.1(_ZN15vtkRenderWindow10DoFDRenderEv+0x61f)[0x2ab6ea5db7e7]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/python2.7/site-packages/vtk/../../../libvtkRenderingCore-7.1.so.1(_ZN15vtkRenderWindow10DoAARenderEv+0x7b0)[0x2ab6ea5db1c6]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/python2.7/site-packages/vtk/../../../libvtkRenderingCore-7.1.so.1(_ZN15vtkRenderWindow6RenderEv+0x83f)[0x2ab6ea5da7ad]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/python2.7/site-packages/vtk/../../../libvtkRenderingOpenGL-7.1.so.1(_ZN22vtkXOpenGLRenderWindow6RenderEv+0x8b)[0x2ab6fa5c4b59]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/lib/python2.7/site-packages/vtk/../../../libvtkRenderingOpenGLPython27D-7.1.so.1(+0xb3cc6)[0x2ab6fa0ffcc6]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6f89)[0x2ab6c0a2daa9]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d)[0x2ab6c0a301dd]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(+0x86085)[0x2ab6c09a7085]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(PyObject_Call+0x43)[0x2ab6c0974ef3]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3597)[0x2ab6c0a2a0b7]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d)[0x2ab6c0a301dd]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6945)[0x2ab6c0a2d465]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d)[0x2ab6c0a301dd]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6945)[0x2ab6c0a2d465]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d)[0x2ab6c0a301dd]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x2ab6c0a30312]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(PyRun_InteractiveOneFlags+0x157)[0x2ab6c0a5a077]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(PyRun_InteractiveLoopFlags+0x4e)[0x2ab6c0a5a27e]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(PyRun_AnyFileExFlags+0x3e)[0x2ab6c0a5aa6e]
/home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/../lib/libpython2.7.so.1.0(Py_Main+0xc4d)[0x2ab6c0a7012d]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x2ab6c1687b35]
python[0x4007b1]
======= Memory map: ========
00400000-00401000 r-xp 00000000 00:2b 49808663                           /home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/python2.7
00600000-00601000 rw-p 00000000 00:2b 49808663                           /home/share/unix_files/cdat/miniconda2b/envs/cdatm15/bin/python2.7
00924000-026c4000 rw-p 00000000 00:00 0                                  [heap]
2ab6c06fe000-2ab6c071e000 r-xp 00000000 08:02 790021                     /usr/lib64/ld-2.17.so
2ab6c071e000-2ab6c0720000 rw-p 00000000 00:00 0 

[... pages after pages of references to the libraries in my conda environment]

2ab70fd0e000-2ab70fd0f000 ---p 00000000 00:00 0 
2ab70fd0f000-2ab710d27000 rw-p 00000000 00:00 0                          [stack:16902]
2ab714000000-2ab714021000 rw-p 00000000 00:00 0 
2ab714021000-2ab718000000 ---p 00000000 00:00 0 
7ffe5ed01000-7ffe5ed24000 rw-p 00000000 00:00 0                          [stack]
7ffe5ed55000-7ffe5ed57000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted (core dumped)
(cdatm15) bash-4.2$ 

You can access the full error dump I got for 2.8 and 2.10 in this folder

Let me know if you need more information on the server hardware/software, etc... All our disks are now on these servers, so I'd rather like to be able to run vcs there... Must be something wrong with opengl or something related

@danlipsa

This comment has been minimized.

Contributor

danlipsa commented Aug 3, 2017

@jypeter Indeed, the problem is OpenGL through vnc. Note it is the same vnc client but not the same vnc server (as the vnc server runs on the newer machine). Here we are running vcs remotely through Virtual GL + TurboVNC. See the doc that @sankhesh put together.
https://github.com/UV-CDAT/uvcdat/wiki/Remote-server-setup-for-VNC
It would be nice if @doutriaux1 would try this setup on the new machine. He tried it on the old redhat 6 and did not work.

@durack1

This comment has been minimized.

Member

durack1 commented Aug 3, 2017

@danlipsa @doutriaux1 @aashish24 it would be great to augment the testing suite to include redhat7/CENTOS7 in addition to the redhat6 machines that we're currently using

@aashish24

This comment has been minimized.

Contributor

aashish24 commented Aug 3, 2017

@danlipsa @doutriaux1 @aashish24 it would be great to augment the testing suite to include redhat7/CENTOS7 in addition to the redhat6 machines that we're currently using

@durack1 I think it will be a a hard thing since for testing we switched to circleci and gave up on buildbot which would have allowed us to setup our own testing infrastructure. I like the idea/suggestions though so if there are any other easy way, I am all for it.

@danlipsa

This comment has been minimized.

Contributor

danlipsa commented Aug 4, 2017

@aashish24 @doutriaux1 @durack1 @williams13 We'll have to do something (bring back buildbot?) to be able to test on-screen vcs on both linux and mac. Right now we have regressions for window close and window resize that we fixed a year ago. We have a test that passes for mesa but fails for on-screen driver.

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 7, 2017

@jypeter

This comment has been minimized.

jypeter commented Aug 7, 2017

@danlipsa I have started reading your VNC wiki page. I can check with our sysadmins if they have a test server and time to do all the things you mention.

The trouble is that, as far as I know, I'm not using VNC!. I'm running Fedora Core 20 (Fedora release 20 (Heisenbug)) in my VirtualBox VM, on a Windows 7 machine with a basic NVIDIA NVS 310 card.

I think I understand that maybe using TurboVNC would allow me to squeeze some extra performance when running vcs on a remote server (which would be nice, because it is often painfully slow). But would not using this be a duplicate of using my VM+ssh, or an X server (Cygwin or VcXsrv) + ssh?

I would like first to be able to run vcs run out-of-the-box on the new server, or understand why it is not running. glxgears is running fine when started locally on my VM, or when started on the old and the new server

I have added below the output of glxinfo in my VM, and on the old and the new servers (when connected from my VM). I have also added what GL stuff we have on the remote servers

glxinfo in the VM

jypeter@lsce3017.extra.cea.fr - ...jypeter - 44 >glxinfo
name of display: :0.0
display: :0  screen: 0
direct rendering: Yes
server glx vendor string: Chromium
server glx version string: 1.3 Chromium
server glx extensions:
    GLX_ARB_get_proc_address, GLX_ARB_multisample, 
    GLX_EXT_texture_from_pixmap, GLX_SGIX_fbconfig
client glx vendor string: Chromium
client glx version string: 1.3 Chromium
client glx extensions:
    GLX_ARB_get_proc_address, GLX_ARB_multisample, 
    GLX_EXT_texture_from_pixmap, GLX_SGIX_fbconfig
GLX version: 1.3
GLX extensions:
    GLX_ARB_get_proc_address, GLX_ARB_multisample, 
    GLX_EXT_texture_from_pixmap, GLX_SGIX_fbconfig
OpenGL vendor string: Humper
OpenGL renderer string: Chromium
OpenGL version string: 2.1 Chromium 1.9
OpenGL shading language version string: 4.50 NVIDIA
OpenGL extensions:
    GL_ARB_depth_texture, GL_ARB_draw_buffers, GL_ARB_fragment_program, 
    GL_ARB_fragment_shader, GL_ARB_multisample, GL_ARB_multitexture, 
    GL_ARB_occlusion_query, GL_ARB_pixel_buffer_object, 
    GL_ARB_point_parameters, GL_ARB_point_sprite, GL_ARB_shader_objects, 
    GL_ARB_shader_texture_lod, GL_ARB_shading_language_100, GL_ARB_shadow, 
    GL_ARB_texture_border_clamp, GL_ARB_texture_compression, 
    GL_ARB_texture_cube_map, GL_ARB_texture_env_add, 
    GL_ARB_texture_env_combine, GL_ARB_texture_env_crossbar, 
    GL_ARB_texture_env_dot3, GL_ARB_texture_float, 
    GL_ARB_texture_mirrored_repeat, GL_ARB_texture_non_power_of_two, 
    GL_ARB_texture_rectangle, GL_ARB_transpose_matrix, 
    GL_ARB_vertex_buffer_object, GL_ARB_vertex_program, GL_ARB_vertex_shader, 
    GL_ARB_window_pos, GL_ATI_texture_mirror_once, GL_CR_bounding_box, 
    GL_CR_cursor_position, GL_CR_head_spu_name, GL_CR_performance_info, 
    GL_CR_print_string, GL_CR_readback_barrier_size, GL_CR_saveframe, 
    GL_CR_server_id_sharing, GL_CR_server_matrix, GL_CR_state_parameter, 
    GL_CR_synchronization, GL_CR_tile_info, GL_CR_tilesort_info, 
    GL_CR_window_size, GL_EXT_blend_color, GL_EXT_blend_equation_separate, 
    GL_EXT_blend_func_separate, GL_EXT_blend_minmax, GL_EXT_blend_subtract, 
    GL_EXT_compiled_vertex_array, GL_EXT_draw_range_elements, 
    GL_EXT_fog_coord, GL_EXT_framebuffer_blit, GL_EXT_framebuffer_object, 
    GL_EXT_multi_draw_arrays, GL_EXT_secondary_color, GL_EXT_shadow_funcs, 
    GL_EXT_stencil_two_side, GL_EXT_stencil_two_side, GL_EXT_stencil_wrap, 
    GL_EXT_texture3D, GL_EXT_texture_compression_s3tc, 
    GL_EXT_texture_cube_map, GL_EXT_texture_edge_clamp, 
    GL_EXT_texture_env_add, GL_EXT_texture_env_combine, 
    GL_EXT_texture_env_dot3, GL_EXT_texture_filter_anisotropic, 
    GL_EXT_texture_lod_bias, GL_EXT_texture_object, GL_EXT_texture_sRGB, 
    GL_IBM_rasterpos_clip, GL_IBM_texture_mirrored_repeat, GL_NV_fog_distance, 
    GL_NV_fragment_program, GL_NV_fragment_program2, 
    GL_NV_fragment_program_option, GL_NV_register_combiners, 
    GL_NV_register_combiners2, GL_NV_texgen_reflection, 
    GL_NV_texture_rectangle, GL_NV_vertex_program, GL_NV_vertex_program1_1, 
    GL_NV_vertex_program2, GL_NV_vertex_program2_option, 
    GL_NV_vertex_program3, GL_SGIS_generate_mipmap

120 GLX Visuals
    visual  x   bf lv rg d st  colorbuffer  sr ax dp st accumbuffer  ms  cav
  id dep cl sp  sz l  ci b ro  r  g  b  a F gb bf th cl  r  g  b  a ns b eat
----------------------------------------------------------------------------
0x021 24 tc  0  32  0 r  y y   8  8  8  8 .  .  0 24  8 16 16 16 16  0 0 None
0x104 24 tc  0  32  0 r  y y   8  8  8  8 .  .  0 24  8 16 16 16 16  0 0 None

glxinfo on the old servers (RH 6)

jypeter@lsce3017.extra.cea.fr - ...jypeter - 45 >ssh asterix
Last login: Mon Aug  7 15:14:26 2017 from lsce3046.extra.cea.fr
jypeter@asterix2 - ...jypeter - 41 >glxinfo
name of display: localhost:11.0
display: localhost:11  screen: 0
direct rendering: Yes
server glx vendor string: SGI
server glx version string: 1.4
server glx extensions:
    GLX_ARB_multisample, GLX_EXT_visual_info, GLX_EXT_visual_rating, 
    GLX_EXT_import_context, GLX_EXT_texture_from_pixmap, GLX_OML_swap_method, 
    GLX_SGI_make_current_read, GLX_SGIS_multisample, GLX_SGIX_fbconfig, 
    GLX_SGIX_pbuffer, GLX_MESA_copy_sub_buffer
client glx vendor string: Mesa Project and SGI
client glx version string: 1.4
client glx extensions:
    GLX_ARB_create_context, GLX_ARB_create_context_profile, 
    GLX_ARB_create_context_robustness, GLX_ARB_fbconfig_float, 
    GLX_ARB_framebuffer_sRGB, GLX_ARB_get_proc_address, GLX_ARB_multisample, 
    GLX_EXT_import_context, GLX_EXT_visual_info, GLX_EXT_visual_rating, 
    GLX_EXT_fbconfig_packed_float, GLX_EXT_framebuffer_sRGB, 
    GLX_EXT_create_context_es2_profile, GLX_MESA_copy_sub_buffer, 
    GLX_MESA_multithread_makecurrent, GLX_MESA_query_renderer, 
    GLX_MESA_swap_control, GLX_OML_swap_method, GLX_OML_sync_control, 
    GLX_SGI_make_current_read, GLX_SGI_swap_control, GLX_SGI_video_sync, 
    GLX_SGIS_multisample, GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, 
    GLX_SGIX_visual_select_group, GLX_EXT_texture_from_pixmap, 
    GLX_INTEL_swap_event, GLX_EXT_buffer_age
GLX version: 1.4
GLX extensions:
    GLX_ARB_get_proc_address, GLX_ARB_multisample, GLX_EXT_import_context, 
    GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_MESA_copy_sub_buffer, 
    GLX_MESA_multithread_makecurrent, GLX_MESA_query_renderer, 
    GLX_OML_swap_method, GLX_SGI_make_current_read, GLX_SGIS_multisample, 
    GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, GLX_EXT_texture_from_pixmap
OpenGL vendor string: VMware, Inc.
OpenGL renderer string: Gallium 0.4 on llvmpipe (LLVM 3.6, 128 bits)
OpenGL version string: 2.1 Mesa 11.0.7
OpenGL shading language version string: 1.30
OpenGL extensions:
    GL_ARB_multisample, GL_EXT_abgr, GL_EXT_bgra, GL_EXT_blend_color, 
    GL_EXT_blend_minmax, GL_EXT_blend_subtract, GL_EXT_copy_texture, 
    GL_EXT_polygon_offset, GL_EXT_subtexture, GL_EXT_texture_object, 
    GL_EXT_vertex_array, GL_EXT_compiled_vertex_array, GL_EXT_texture, 
    GL_EXT_texture3D, GL_IBM_rasterpos_clip, GL_ARB_point_parameters, 
    GL_EXT_draw_range_elements, GL_EXT_packed_pixels, GL_EXT_point_parameters, 
    GL_EXT_rescale_normal, GL_EXT_separate_specular_color, 
    GL_EXT_texture_edge_clamp, GL_SGIS_generate_mipmap, 
    GL_SGIS_texture_border_clamp, GL_SGIS_texture_edge_clamp, 
    GL_SGIS_texture_lod, GL_ARB_framebuffer_sRGB, GL_ARB_multitexture, 
    GL_EXT_framebuffer_sRGB, GL_IBM_multimode_draw_arrays, 
    GL_IBM_texture_mirrored_repeat, GL_ARB_texture_cube_map, 
    GL_ARB_texture_env_add, GL_ARB_transpose_matrix, 
    GL_EXT_blend_func_separate, GL_EXT_fog_coord, GL_EXT_multi_draw_arrays, 
    GL_EXT_secondary_color, GL_EXT_texture_env_add, GL_EXT_texture_lod_bias, 
    GL_INGR_blend_func_separate, GL_NV_blend_square, GL_NV_light_max_exponent, 
    GL_NV_texgen_reflection, GL_NV_texture_env_combine4, 
    GL_SUN_multi_draw_arrays, GL_ARB_texture_border_clamp, 
    GL_ARB_texture_compression, GL_EXT_framebuffer_object, 
    GL_EXT_texture_env_combine, GL_EXT_texture_env_dot3, GL_MESA_window_pos, 
    GL_NV_packed_depth_stencil, GL_NV_texture_rectangle, GL_ARB_depth_texture, 
    GL_ARB_occlusion_query, GL_ARB_shadow, GL_ARB_texture_env_combine, 
    GL_ARB_texture_env_crossbar, GL_ARB_texture_env_dot3, 
    GL_ARB_texture_mirrored_repeat, GL_ARB_window_pos, 
    GL_EXT_stencil_two_side, GL_EXT_texture_cube_map, GL_NV_depth_clamp, 
    GL_NV_fog_distance, GL_APPLE_packed_pixels, GL_APPLE_vertex_array_object, 
    GL_ARB_draw_buffers, GL_ARB_fragment_program, GL_ARB_fragment_shader, 
    GL_ARB_shader_objects, GL_ARB_vertex_program, GL_ARB_vertex_shader, 
    GL_ATI_draw_buffers, GL_ATI_texture_env_combine3, GL_EXT_shadow_funcs, 
    GL_EXT_stencil_wrap, GL_MESA_pack_invert, GL_MESA_ycbcr_texture, 
    GL_NV_primitive_restart, GL_ARB_depth_clamp, 
    GL_ARB_fragment_program_shadow, GL_ARB_half_float_pixel, 
    GL_ARB_occlusion_query2, GL_ARB_point_sprite, GL_ARB_shading_language_100, 
    GL_ARB_sync, GL_ARB_texture_non_power_of_two, GL_ARB_vertex_buffer_object, 
    GL_ATI_blend_equation_separate, GL_EXT_blend_equation_separate, 
    GL_OES_read_format, GL_ARB_color_buffer_float, GL_ARB_pixel_buffer_object, 
    GL_ARB_texture_compression_rgtc, GL_ARB_texture_rectangle, 
    GL_ATI_texture_compression_3dc, GL_EXT_packed_float, 
    GL_EXT_pixel_buffer_object, GL_EXT_texture_compression_rgtc, 
    GL_EXT_texture_mirror_clamp, GL_EXT_texture_rectangle, 
    GL_EXT_texture_sRGB, GL_EXT_texture_shared_exponent, 
    GL_ARB_framebuffer_object, GL_EXT_framebuffer_blit, 
    GL_EXT_framebuffer_multisample, GL_EXT_packed_depth_stencil, 
    GL_ARB_vertex_array_object, GL_ATI_separate_stencil, 
    GL_ATI_texture_mirror_once, GL_EXT_draw_buffers2, GL_EXT_draw_instanced, 
    GL_EXT_gpu_program_parameters, GL_EXT_texture_array, 
    GL_EXT_texture_compression_latc, GL_EXT_texture_integer, 
    GL_EXT_texture_sRGB_decode, GL_EXT_timer_query, GL_OES_EGL_image, 
    GL_ARB_copy_buffer, GL_ARB_depth_buffer_float, GL_ARB_draw_instanced, 
    GL_ARB_half_float_vertex, GL_ARB_instanced_arrays, 
    GL_ARB_map_buffer_range, GL_ARB_texture_rg, GL_ARB_texture_swizzle, 
    GL_ARB_vertex_array_bgra, GL_EXT_texture_swizzle, 
    GL_EXT_vertex_array_bgra, GL_NV_conditional_render, 
    GL_AMD_conservative_depth, GL_AMD_draw_buffers_blend, 
    GL_AMD_seamless_cubemap_per_texture, GL_AMD_shader_stencil_export, 
    GL_ARB_ES2_compatibility, GL_ARB_blend_func_extended, GL_ARB_debug_output, 
    GL_ARB_draw_buffers_blend, GL_ARB_draw_elements_base_vertex, 
    GL_ARB_explicit_attrib_location, GL_ARB_fragment_coord_conventions, 
    GL_ARB_provoking_vertex, GL_ARB_sampler_objects, GL_ARB_seamless_cube_map, 
    GL_ARB_shader_stencil_export, GL_ARB_shader_texture_lod, 
    GL_ARB_texture_cube_map_array, GL_ARB_texture_gather, 
    GL_ARB_texture_multisample, GL_ARB_texture_rgb10_a2ui, 
    GL_ARB_uniform_buffer_object, GL_ARB_vertex_type_2_10_10_10_rev, 
    GL_EXT_provoking_vertex, GL_EXT_texture_snorm, 
    GL_MESA_texture_signed_rgba, GL_ARB_get_program_binary, GL_ARB_robustness, 
    GL_ARB_separate_shader_objects, GL_ARB_shader_bit_encoding, 
    GL_ARB_timer_query, GL_ARB_transform_feedback2, 
    GL_ARB_transform_feedback3, GL_ARB_base_instance, 
    GL_ARB_compressed_texture_pixel_storage, GL_ARB_conservative_depth, 
    GL_ARB_internalformat_query, GL_ARB_map_buffer_alignment, 
    GL_ARB_shading_language_420pack, GL_ARB_shading_language_packing, 
    GL_ARB_texture_storage, GL_ARB_transform_feedback_instanced, 
    GL_EXT_framebuffer_multisample_blit_scaled, GL_EXT_transform_feedback, 
    GL_AMD_shader_trinary_minmax, GL_ARB_ES3_compatibility, 
    GL_ARB_clear_buffer_object, GL_ARB_explicit_uniform_location, 
    GL_ARB_invalidate_subdata, GL_ARB_program_interface_query, 
    GL_ARB_stencil_texturing, GL_ARB_texture_query_levels, 
    GL_ARB_texture_storage_multisample, GL_ARB_texture_view, 
    GL_ARB_vertex_attrib_binding, GL_KHR_debug, GL_ARB_buffer_storage, 
    GL_ARB_multi_bind, GL_ARB_seamless_cubemap_per_texture, 
    GL_ARB_texture_mirror_clamp_to_edge, GL_ARB_texture_stencil8, 
    GL_ARB_vertex_type_10f_11f_11f_rev, GL_EXT_shader_integer_mix, 
    GL_ARB_clip_control, GL_ARB_conditional_render_inverted, 
    GL_ARB_get_texture_sub_image, GL_EXT_polygon_offset_clamp, 
    GL_KHR_context_flush_control

120 GLX Visuals
    visual  x   bf lv rg d st  colorbuffer  sr ax dp st accumbuffer  ms  cav
  id dep cl sp  sz l  ci b ro  r  g  b  a F gb bf th cl  r  g  b  a ns b eat
----------------------------------------------------------------------------
0x021 24 tc  0  32  0 r  y .   8  8  8  8 .  .  0 24  8  0  0  0  0  0 0 None
0x104 24 tc  0  32  0 r  . .   8  8  8  8 .  .  0  0  0  0  0  0  0  0 0 None
jypeter@asterix2 - ...jypeter - 44 >rpm -qa | \grep -i gl | \grep -v glib
glx-utils-11.0.7-4.el6.x86_64
libglade2-2.6.4-3.1.el6.x86_64
libglade2-devel-2.6.4-3.1.el6.x86_64
glpk-4.40-1.1.el6.x86_64
mesa-libEGL-11.0.7-4.el6.x86_64
mesa-libGLU-11.0.7-4.el6.x86_64
libgssglue-0.1-11.el6.x86_64
freeglut-devel-2.6.0-1.el6.x86_64
ftgl-2.1.3-0.3.rc5.el6.x86_64
mesa-libGLU-devel-11.0.7-4.el6.x86_64
gegl-0.1.2-6.el6.x86_64
mesa-libGLU-11.0.7-4.el6.i686
mesa-libGL-11.0.7-4.el6.x86_64
wxGTK-gl-2.8.12-1.el6.x86_64
mesa-libGL-11.0.7-4.el6.i686
mesa-libGL-devel-11.0.7-4.el6.x86_64
xorg-x11-drv-glint-1.2.8-11.el6.x86_64
pygtk2-libglade-2.16.0-3.el6.x86_64
freeglut-2.6.0-1.el6.x86_64

glxinfo on the new servers (CentOS 7)

jypeter@lsce3017.extra.cea.fr - ...jypeter - 46 >ssh obelix5.lsce.ipsl.fr
Last login: Mon Aug  7 16:17:24 2017 from lsce3046.extra.cea.fr 
jypeter@obelix5 - ...jypeter - 41 >glxinfo
name of display: localhost:36.0
display: localhost:36  screen: 0
direct rendering: Yes
server glx vendor string: SGI
server glx version string: 1.4
server glx extensions:
    GLX_ARB_multisample, GLX_EXT_import_context, GLX_EXT_texture_from_pixmap, 
    GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_MESA_copy_sub_buffer, 
    GLX_OML_swap_method, GLX_SGIS_multisample, GLX_SGIX_fbconfig, 
    GLX_SGIX_pbuffer, GLX_SGI_make_current_read
client glx vendor string: Mesa Project and SGI
client glx version string: 1.4
client glx extensions:
    GLX_ARB_create_context, GLX_ARB_create_context_profile, 
    GLX_ARB_create_context_robustness, GLX_ARB_fbconfig_float, 
    GLX_ARB_framebuffer_sRGB, GLX_ARB_get_proc_address, GLX_ARB_multisample, 
    GLX_EXT_buffer_age, GLX_EXT_create_context_es2_profile, 
    GLX_EXT_create_context_es_profile, GLX_EXT_fbconfig_packed_float, 
    GLX_EXT_framebuffer_sRGB, GLX_EXT_import_context, 
    GLX_EXT_texture_from_pixmap, GLX_EXT_visual_info, GLX_EXT_visual_rating, 
    GLX_INTEL_swap_event, GLX_MESA_copy_sub_buffer, 
    GLX_MESA_multithread_makecurrent, GLX_MESA_query_renderer, 
    GLX_MESA_swap_control, GLX_OML_swap_method, GLX_OML_sync_control, 
    GLX_SGIS_multisample, GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, 
    GLX_SGIX_visual_select_group, GLX_SGI_make_current_read, 
    GLX_SGI_swap_control, GLX_SGI_video_sync
GLX version: 1.4
GLX extensions:
    GLX_ARB_get_proc_address, GLX_ARB_multisample, GLX_EXT_import_context, 
    GLX_EXT_texture_from_pixmap, GLX_EXT_visual_info, GLX_EXT_visual_rating, 
    GLX_MESA_copy_sub_buffer, GLX_MESA_multithread_makecurrent, 
    GLX_MESA_query_renderer, GLX_OML_swap_method, GLX_SGIS_multisample, 
    GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, GLX_SGI_make_current_read
OpenGL vendor string: VMware, Inc.
OpenGL renderer string: Gallium 0.4 on llvmpipe (LLVM 3.8, 128 bits)
OpenGL version string: 2.1 Mesa 11.2.2
OpenGL shading language version string: 1.30
OpenGL extensions:
    GL_AMD_conservative_depth, GL_AMD_draw_buffers_blend, 
    GL_AMD_seamless_cubemap_per_texture, GL_AMD_shader_stencil_export, 
    GL_AMD_shader_trinary_minmax, GL_APPLE_packed_pixels, 
    GL_APPLE_vertex_array_object, GL_ARB_ES2_compatibility, 
    GL_ARB_ES3_compatibility, GL_ARB_arrays_of_arrays, GL_ARB_base_instance, 
    GL_ARB_blend_func_extended, GL_ARB_buffer_storage, 
    GL_ARB_clear_buffer_object, GL_ARB_clip_control, 
    GL_ARB_color_buffer_float, GL_ARB_compressed_texture_pixel_storage, 
    GL_ARB_conditional_render_inverted, GL_ARB_conservative_depth, 
    GL_ARB_copy_buffer, GL_ARB_debug_output, GL_ARB_depth_buffer_float, 
    GL_ARB_depth_clamp, GL_ARB_depth_texture, GL_ARB_draw_buffers, 
    GL_ARB_draw_buffers_blend, GL_ARB_draw_elements_base_vertex, 
    GL_ARB_draw_instanced, GL_ARB_explicit_attrib_location, 
    GL_ARB_explicit_uniform_location, GL_ARB_fragment_coord_conventions, 
    GL_ARB_fragment_program, GL_ARB_fragment_program_shadow, 
    GL_ARB_fragment_shader, GL_ARB_framebuffer_object, 
    GL_ARB_framebuffer_sRGB, GL_ARB_get_program_binary, 
    GL_ARB_get_texture_sub_image, GL_ARB_half_float_pixel, 
    GL_ARB_half_float_vertex, GL_ARB_instanced_arrays, 
    GL_ARB_internalformat_query, GL_ARB_invalidate_subdata, 
    GL_ARB_map_buffer_alignment, GL_ARB_map_buffer_range, GL_ARB_multi_bind, 
    GL_ARB_multisample, GL_ARB_multitexture, GL_ARB_occlusion_query, 
    GL_ARB_occlusion_query2, GL_ARB_pixel_buffer_object, 
    GL_ARB_point_parameters, GL_ARB_point_sprite, 
    GL_ARB_program_interface_query, GL_ARB_provoking_vertex, 
    GL_ARB_robustness, GL_ARB_sampler_objects, GL_ARB_seamless_cube_map, 
    GL_ARB_seamless_cubemap_per_texture, GL_ARB_separate_shader_objects, 
    GL_ARB_shader_bit_encoding, GL_ARB_shader_objects, 
    GL_ARB_shader_stencil_export, GL_ARB_shader_texture_lod, 
    GL_ARB_shading_language_100, GL_ARB_shading_language_420pack, 
    GL_ARB_shading_language_packing, GL_ARB_shadow, GL_ARB_stencil_texturing, 
    GL_ARB_sync, GL_ARB_texture_border_clamp, GL_ARB_texture_compression, 
    GL_ARB_texture_compression_rgtc, GL_ARB_texture_cube_map, 
    GL_ARB_texture_cube_map_array, GL_ARB_texture_env_add, 
    GL_ARB_texture_env_combine, GL_ARB_texture_env_crossbar, 
    GL_ARB_texture_env_dot3, GL_ARB_texture_gather, 
    GL_ARB_texture_mirror_clamp_to_edge, GL_ARB_texture_mirrored_repeat, 
    GL_ARB_texture_multisample, GL_ARB_texture_non_power_of_two, 
    GL_ARB_texture_query_levels, GL_ARB_texture_rectangle, GL_ARB_texture_rg, 
    GL_ARB_texture_rgb10_a2ui, GL_ARB_texture_stencil8, 
    GL_ARB_texture_storage, GL_ARB_texture_storage_multisample, 
    GL_ARB_texture_swizzle, GL_ARB_texture_view, GL_ARB_timer_query, 
    GL_ARB_transform_feedback2, GL_ARB_transform_feedback3, 
    GL_ARB_transform_feedback_instanced, GL_ARB_transpose_matrix, 
    GL_ARB_uniform_buffer_object, GL_ARB_vertex_array_bgra, 
    GL_ARB_vertex_array_object, GL_ARB_vertex_attrib_binding, 
    GL_ARB_vertex_buffer_object, GL_ARB_vertex_program, GL_ARB_vertex_shader, 
    GL_ARB_vertex_type_10f_11f_11f_rev, GL_ARB_vertex_type_2_10_10_10_rev, 
    GL_ARB_window_pos, GL_ATI_blend_equation_separate, GL_ATI_draw_buffers, 
    GL_ATI_separate_stencil, GL_ATI_texture_compression_3dc, 
    GL_ATI_texture_env_combine3, GL_ATI_texture_mirror_once, GL_EXT_abgr, 
    GL_EXT_bgra, GL_EXT_blend_color, GL_EXT_blend_equation_separate, 
    GL_EXT_blend_func_separate, GL_EXT_blend_minmax, GL_EXT_blend_subtract, 
    GL_EXT_compiled_vertex_array, GL_EXT_copy_texture, GL_EXT_draw_buffers2, 
    GL_EXT_draw_instanced, GL_EXT_draw_range_elements, GL_EXT_fog_coord, 
    GL_EXT_framebuffer_blit, GL_EXT_framebuffer_multisample, 
    GL_EXT_framebuffer_multisample_blit_scaled, GL_EXT_framebuffer_object, 
    GL_EXT_framebuffer_sRGB, GL_EXT_gpu_program_parameters, 
    GL_EXT_multi_draw_arrays, GL_EXT_packed_depth_stencil, 
    GL_EXT_packed_float, GL_EXT_packed_pixels, GL_EXT_pixel_buffer_object, 
    GL_EXT_point_parameters, GL_EXT_polygon_offset, 
    GL_EXT_polygon_offset_clamp, GL_EXT_provoking_vertex, 
    GL_EXT_rescale_normal, GL_EXT_secondary_color, 
    GL_EXT_separate_specular_color, GL_EXT_shader_integer_mix, 
    GL_EXT_shadow_funcs, GL_EXT_stencil_two_side, GL_EXT_stencil_wrap, 
    GL_EXT_subtexture, GL_EXT_texture, GL_EXT_texture3D, 
    GL_EXT_texture_array, GL_EXT_texture_compression_latc, 
    GL_EXT_texture_compression_rgtc, GL_EXT_texture_cube_map, 
    GL_EXT_texture_edge_clamp, GL_EXT_texture_env_add, 
    GL_EXT_texture_env_combine, GL_EXT_texture_env_dot3, 
    GL_EXT_texture_integer, GL_EXT_texture_lod_bias, 
    GL_EXT_texture_mirror_clamp, GL_EXT_texture_object, 
    GL_EXT_texture_rectangle, GL_EXT_texture_sRGB, GL_EXT_texture_sRGB_decode, 
    GL_EXT_texture_shared_exponent, GL_EXT_texture_snorm, 
    GL_EXT_texture_swizzle, GL_EXT_timer_query, GL_EXT_transform_feedback, 
    GL_EXT_vertex_array, GL_EXT_vertex_array_bgra, 
    GL_IBM_multimode_draw_arrays, GL_IBM_rasterpos_clip, 
    GL_IBM_texture_mirrored_repeat, GL_INGR_blend_func_separate, 
    GL_KHR_context_flush_control, GL_KHR_debug, GL_MESA_pack_invert, 
    GL_MESA_texture_signed_rgba, GL_MESA_window_pos, GL_MESA_ycbcr_texture, 
    GL_NV_blend_square, GL_NV_conditional_render, GL_NV_depth_clamp, 
    GL_NV_fog_distance, GL_NV_light_max_exponent, GL_NV_packed_depth_stencil, 
    GL_NV_primitive_restart, GL_NV_texgen_reflection, 
    GL_NV_texture_env_combine4, GL_NV_texture_rectangle, GL_OES_EGL_image, 
    GL_OES_read_format, GL_SGIS_generate_mipmap, GL_SGIS_texture_border_clamp, 
    GL_SGIS_texture_edge_clamp, GL_SGIS_texture_lod, GL_SUN_multi_draw_arrays

120 GLX Visuals
    visual  x   bf lv rg d st  colorbuffer  sr ax dp st accumbuffer  ms  cav
  id dep cl sp  sz l  ci b ro  r  g  b  a F gb bf th cl  r  g  b  a ns b eat
----------------------------------------------------------------------------
0x021 24 tc  0  32  0 r  y .   8  8  8  8 .  .  0 24  8  0  0  0  0  0 0 None
0x104 24 tc  0  32  0 r  . .   8  8  8  8 .  .  0  0  0  0  0  0  0  0 0 None
jypeter@obelix5 - ...jypeter - 49 >rpm -qa | \grep -i gl | \grep -v glib
mesa-libGLw-8.0.0-4.el7.x86_64
glusterfs-3.7.9-12.el7.centos.x86_64
freeglut-2.8.1-3.el7.x86_64
mesa-libglapi-11.2.2-2.20160614.el7.x86_64
mesa-libGLU-devel-9.0.0-4.el7.x86_64
gl-manpages-1.1-7.20130122.el7.noarch
mesa-libEGL-11.2.2-2.20160614.el7.x86_64
mesa-libGLU-9.0.0-4.el7.x86_64
glusterfs-client-xlators-3.7.9-12.el7.centos.x86_64
gl2ps-1.3.8-2.el7.x86_64
glusterfs-fuse-3.7.9-12.el7.centos.x86_64
mesa-libGL-devel-11.2.2-2.20160614.el7.x86_64
mesa-libGL-11.2.2-2.20160614.el7.x86_64
glx-utils-8.2.0-3.el7.x86_64
mesa-libGLw-devel-8.0.0-4.el7.x86_64
mesa-libEGL-devel-11.2.2-2.20160614.el7.x86_64
fftw-libs-single-3.3.3-8.el7.x86_64
mesa-libGLES-11.2.2-2.20160614.el7.x86_64
kwin-gles-libs-4.11.19-8.el7.x86_64
glpk-4.52.1-2.el7.x86_64
glusterfs-libs-3.7.9-12.el7.centos.x86_64
freeglut-devel-2.8.1-3.el7.x86_64
@jypeter

This comment has been minimized.

jypeter commented Aug 8, 2017

@doutriaux1 and @danlipsa Has anybody tried vcs (either 2.8 or 2.10) on a Centos 7 or RH7 machine? I have just run some additional tests, and got the usual catastrophic final crash. Note that glxgears worked in all cases

  • does not work on a Linux desktop connected to the remote Centos 7 server. So it seems the problem is not due to my using a Linux running in a VM to connect to the server
  • does not work when connected to the remote server with XDMCP with a basic xfce display manager
  • does not work when logged directly to the server! The error dump I got, typescript-LOCAL-2.8.0.txt, is located in the same folder as the other dumps. The only difference is that the dump is only 528 lines long

@doutriaux1 doutriaux1 added the bug label Aug 8, 2017

@doutriaux1 doutriaux1 added this to the 2.12 milestone Aug 8, 2017

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 8, 2017

@aashish24 we really need to address this in a straight forward way. It is making us look really bad.

@danlipsa

This comment has been minimized.

Contributor

danlipsa commented Aug 10, 2017

@jypeter Do you always run inside a VM? If that is true it means that you depend on the OpenGL implementation in the VM which might not work correctly. In this case I would recomment using OSMesa.
conda install mesalib -c uvcdat/label/nightly -c conda-forge -c uvcdat

@jypeter

This comment has been minimized.

jypeter commented Aug 11, 2017

@danlipsa If you look at my previous comment, you'll see in the additional tests that I've made that 2.8.0 did not work either on a) a linux desktop connecting to a remote server, b) directly on the server. I got similar results with 2.10

Note also that 2.8 and 2.10 are not installed in the VM. I only use my VM to connect to the remote servers (basically instead of running a local X server)

Just to be sure, I have run my test again from a ubuntu desktop. You will find the results in the test_170811 folder of the folder where I have already put log files. Note that, again, glxgears ran fine on the local desktop, and on the remote server where I connected from the local desktop

  • glxinfo_on_ubuntu_desktop.txt: output of glxinfo on the local desktop
  • xdpyinfo_on_ubuntu_desktop.txt: output of xdpyinfo on the local desktop
  • glxinfo_on_CentOS7_server_from_ubuntu_desktop.txt: output of glxinfo on the CentOS7 server, after connecting from the local desktop
  • typescript_2.10_on_CentOS7_server_from_ubuntu_desktop.txt: what I got when initializing and running 2.10
@jypeter

This comment has been minimized.

jypeter commented Aug 11, 2017

@danlipsa I have also just installed mesalib the following way

(cdatm15) bash-4.2$ conda install -n cdatm15  -c uvcdat/label/nightly -c conda-forge -c uvcdat mesalib
Fetching package metadata ...............
Solving package specifications: .

Package plan for installation in environment /home/share/unix_files/cdat/miniconda2/envs/cdatm15:

The following NEW packages will be INSTALLED:

    llvmdev:       4.0.0-default_0   conda-forge
    mesalib:       17.1.4-3          conda-forge
    pthread-stubs: 0.3-1             conda-forge
    system:        5.8-2                        

The following packages will be DOWNGRADED due to dependency conflicts:

    vtk-cdat:      7.1.0.2.10-py27_1 uvcdat      --> 7.1.0.2.10-py27_mesalib_0 uvcdat [mesalib]

Proceed ([y]/n)? y

pthread-stubs- 100% |######################################################################| Time: 0:00:00   1.34 MB/s
system-5.8-2.t 100% |######################################################################| Time: 0:00:00  10.99 MB/s
llvmdev-4.0.0- 100% |######################################################################| Time: 0:00:57   1.96 MB/s
mesalib-17.1.4 100% |######################################################################| Time: 0:00:03   3.02 MB/s
vtk-cdat-7.1.0 100% |######################################################################| Time: 0:00:16   6.25 MB/s
(cdatm15) bash-4.2$ python
Python 2.7.13 | packaged by conda-forge | (default, May  2 2017, 12:48:11) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import numpy as np, vcs
>>> i10 = np.identity(10)
>>> x = vcs.init()
>>> x.plot(i10)
<vcs.displayplot.Dp object at 0x2b3f773b4b40>
>>> x.pdf('test_mesalib.pdf')
>>> x.png('test_mesalib.png')
>>> 

So, the result is... No crash, I get the simple plot I was expecting in the pdf and png files BUT I don't get interactive plots. Is this what I should be expecting? Is this the solution for generating plots on a headless display, and you don't even need to use bg=1? That's useful to know, but I'm looking for interactive graphics. At least a graphical window I can look at, even if I don't do and x.interact() (that can sometimes be painfully slow)

An extra question: once I have installed mesalib, is there a way I can go back to interactive graphics, or do I have to uninstall mesalib (and other packages?)?

@danlipsa

This comment has been minimized.

Contributor

danlipsa commented Aug 11, 2017

So, the result is... No crash, I get the simple plot I was expecting in the pdf and png files BUT I don't get interactive plots. Is this what I should be expecting? Is this the solution for generating plots on a headless display, and you don't even need to use bg=1?

@jypeter Indeed, this is it.

An extra question: once I have installed mesalib, is there a way I can go back to interactive graphics, or do I have to uninstall mesalib (and other packages?)?

I usually just install another conda env that has interactive graphics (direct hardware access). Probably you could replace the vtk pakage and remove mesa, but I have never done it.

@danlipsa

This comment has been minimized.

Contributor

danlipsa commented Aug 11, 2017

@aashish24 @doutriaux1 @jypeter This does seem to be a OpenGL installation problem. A solution to this kind of problem is to ship mesa (not osmesa) with our packages and switch at runtime which opengl implementation we use. Maybe by default we use the driver install on your system. If that does not work, the user can switch to the software renderer (mesa). ParaView does that

https://blog.kitware.com/messing-with-mesa-for-paraview-5-0vtk-7-0/

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 14, 2017

@danlipsa you mean like using a conda-based mesa?

@jypeter

This comment has been minimized.

jypeter commented Aug 14, 2017

I'm on holidays this week, so I won't be able to test things.

I will try to remember the mesalib trick in a cloned environment for offline graphics (this should probably go to a uvcdat cookbook). But in order to have offline graphics, you need to be able to test things interactively

@danlipsa

This comment has been minimized.

Contributor

danlipsa commented Aug 14, 2017

@jypeter The idea is to be able to switch at runtime between the system installed OpenGL driver (which is usually hardware accelerated and might not be installed correctly - especially on Linux) and a software OpenGL driver that will be installed by us in coda and will always work (but might be slower than the hardware driver).

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 15, 2017

@danlipsa do you know what the magic is for the paraview --mesa-llvm listed at the end? Maybe we could do the same for uvcdat?

@danlipsa

This comment has been minimized.

Contributor

danlipsa commented Aug 15, 2017

@doutriaux1 I think we only need to pass LD_LIBRARY_PATH pointing to the libGL.* from mesa (installed by conda). I wanted to try this but you are welcome to try it.
Just install the regular VTK (the one that uses the system OpenGL (hardware)) and mesa without dependencies so that it does not replaces VTK. Than launch python with the LD_LIBRARY_PATH pointing to the mesa libGL.*. If we want a command line we need a small executable to set this env variable.

@danlipsa

This comment has been minimized.

Contributor

danlipsa commented Aug 15, 2017

@doutriaux1 Or python script.

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 15, 2017

@danlipsa but when we use the offscreen mesa we send a special build keyword to the VTK build. Are you saying I could use the regular VTK build and simply points my LD_LIBRARY_PATH to the mesalib and it would get me the offscreen as well?

@danlipsa

This comment has been minimized.

Contributor

danlipsa commented Aug 15, 2017

@doutriaux1 Our special VTK build is 'headless' which means it does not use X. If we point LD_LIBRARY_PATH to mesa's libGL.* we'll still get onscreen (but software) rendering. I think the headless VTK uses different libraries than libGL.*.

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 15, 2017

@danlipsa ok thanks for clarification I rwad the articlle too fast it does say:

ParaView 5.0 Linux binaries distributed from paraview.org (including the release candidates) already package binaries for the  3 variants of Onscreen Mesa: Mesa+llvmpipe, Mesa+openswr+avx, Mesa+openswr+avx2

I wrongfully assumed at first that Mesa+llvmpipe was the offscreen one.

@danlipsa

This comment has been minimized.

Contributor

danlipsa commented Aug 17, 2017

@jypeter I just learn a trick that you can try in your virtual box.
Shut down you VM
Set Settings > Display > Acceleration > Enable 3D Acceleration
Then start you VM.
Without this, OpenGL does not work in Virtual Box.

@jypeter

This comment has been minimized.

jypeter commented Aug 22, 2017

@danlipsa 3D acceleration has been enabled in my VM for a long time, and I get the following if I run glxgears in local terminal in the Fedora 20 running in the VM. I see the cogs turning the way they should

jypeter@lsce3017.extra.cea.fr - ...jypeter - 45 >glxgears -info
GL_RENDERER   = Chromium
GL_VERSION    = 2.1 Chromium 1.9
GL_VENDOR     = Humper

glxgears also runs fine when I connect to our CentOS server, apparently in software rendering

jypeter@obelix5 - ...jypeter - 12 >glxgears -info
GL_RENDERER   = Gallium 0.4 on llvmpipe (LLVM 3.8, 128 bits)
GL_VERSION    = 2.1 Mesa 11.2.2
GL_VENDOR     = VMware, Inc.

I have even checked that glxgears keeps on working after I activate cdat

bash-4.2$ export PATH="/home/share/unix_files/cdat/miniconda2b/bin:$PATH"
bash-4.2$ source activate uvcdat-2.10
(uvcdat-2.10) bash-4.2$ glxgears -info
GL_RENDERER   = Gallium 0.4 on llvmpipe (LLVM 3.8, 128 bits)
GL_VERSION    = 2.1 Mesa 11.2.2
GL_VENDOR     = VMware, Inc.

And then I get the usual very verbose crash, that mentions lots of libraries, both in /usr/lib64 and minconda2/envs. The giant backtrace always consistently starts with a memory related error

>>> x.plot(i10)
*** Error in `python': free(): invalid pointer: 0x00002b01f1c51f60 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c503)[0x2b01a39ad503]

[LOTS and LOTS more]

7ffe5ed01000-7ffe5ed24000 rw-p 00000000 00:00 0                          [stack]
7ffe5ed55000-7ffe5ed57000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted (core dumped)

By the way, why am I getting such a looong error output, instead of just the usual Aborted (core dumped). Is this extra verbose output coming from the python interpreter or the system?

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 22, 2017

@danlipsa that sounds awfully similar to what I experience on my ubuntu since 2.10.

@jypeter

This comment has been minimized.

jypeter commented Aug 22, 2017

Note that on the CentOS 7 server, I get the same failure with both 2.8 and 2.10

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 22, 2017

@jypeter @danlipsa @durack1 @dnadeau4 I just randomly found a solution for my ubunut. It appears to be a mismatch between ANACONDA's stdc++ and the system

see: https://stackoverflow.com/questions/35911302/cannot-launch-emulator-on-linux-ubuntu-15-10

in my case I ran:

LD_PRELOAD='/usr/lib/x86_64-linux-gnu/libstdc++.so.6' python

And I am up and running for my ubuntu

the locate shows on my system:

...
/home/doutriaux1/anaconda2/envs/test/lib/libstdc++.so.6
/home/doutriaux1/anaconda2/envs/test/lib/libstdc++.so.6.0.19
/home/doutriaux1/anaconda2/envs/test/lib/libstdc++.so.6.0.19-gdb.py
/home/doutriaux1/anaconda2/envs/test/lib/libstdc++.so.6.0.21
...
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22

So the system one is 6.0.22 whereas the anaconda one is 6.0.21 (or 19)

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 22, 2017

Note that :

rm /home/doutriaux1/anaconda2/envs/test/lib/libstdc++*

Also fixes the issue and does not seem to affect anything since it's now picking up the system stc++

@dnadeau4 @danlipsa should we try to remove gcc from our conda builds for linux?

@danlipsa

This comment has been minimized.

Contributor

danlipsa commented Aug 23, 2017

@doutriaux1 I think the reason you added gcc was to be able to build on redhat 6 which has an older gcc that what VTK requires.

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 23, 2017

@danlipsa yes we wanted a consistent gcc to build everything. After some investigation with @dnadeau4 it is definitely graphic card drivers related. logging in from a mac and using X forwarding works fine, but on the machine itself (using the local drivers) it seg faults. Will try to build all of our package w/o runtime gcc or libgcc to see if it helps....

@jypeter

This comment has been minimized.

jypeter commented Aug 23, 2017

I have the following on our CentOS 7 server

bash-4.2$ source activate uvcdat-2.10
(uvcdat-2.10) bash-4.2$ locate libstdc++.so.6
/usr/lib/libstdc++.so.6
/usr/lib/libstdc++.so.6.0.19
/usr/lib64/libstdc++.so.6
/usr/lib64/libstdc++.so.6.0.19
/usr/local/install/gcc-5.2.0/lib64/libstdc++.so.6
/usr/local/install/gcc-5.2.0/lib64/libstdc++.so.6.0.21
/usr/local/install/gcc-5.2.0/lib64/libstdc++.so.6.0.21-gdb.py
/usr/local/install/intel-fcomp-2013/composer_xe_2013.2.146/bin/sourcechecker/lib/ia32/pinruntime/libstdc++.so.6
/usr/local/install/intel-fcomp-2013/composer_xe_2013.2.146/bin/sourcechecker/lib/ia32/pinruntime/libstdc++.so.6.0.10
/usr/local/install/intel-fcomp-2013/composer_xe_2013.2.146/bin/sourcechecker/lib/intel64/pinruntime/libstdc++.so.6
/usr/local/install/intel-fcomp-2013/composer_xe_2013.2.146/bin/sourcechecker/lib/intel64/pinruntime/libstdc++.so.6.0.10
/usr/local/install/python-2.7/envs/4artic/lib/libstdc++.so.6
/usr/local/install/python-2.7/envs/4artic/lib/libstdc++.so.6.0.21
/usr/local/install/python-2.7/lib/libstdc++.so.6
/usr/local/install/python-2.7/lib/libstdc++.so.6.0.21
/usr/local/install/python-2.7/pkgs/libgcc-5.2.0-0/lib/libstdc++.so.6
/usr/local/install/python-2.7/pkgs/libgcc-5.2.0-0/lib/libstdc++.so.6.0.21
/usr/share/gdb/auto-load/usr/lib/libstdc++.so.6.0.19-gdb.py
/usr/share/gdb/auto-load/usr/lib/libstdc++.so.6.0.19-gdb.pyc
/usr/share/gdb/auto-load/usr/lib/libstdc++.so.6.0.19-gdb.pyo
/usr/share/gdb/auto-load/usr/lib64/libstdc++.so.6.0.19-gdb.py
/usr/share/gdb/auto-load/usr/lib64/libstdc++.so.6.0.19-gdb.pyc
/usr/share/gdb/auto-load/usr/lib64/libstdc++.so.6.0.19-gdb.pyo

(uvcdat-2.10) bash-4.2$ which python
/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10/bin/python

(uvcdat-2.10) bash-4.2$ find /home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10 -name 'libstdc++.so.6*'
/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10/lib/libstdc++.so.6.0.21
/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10/lib/libstdc++.so.6.0.19-gdb.py
/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10/lib/libstdc++.so.6.0.19
/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10/lib/libstdc++.so.6
(uvcdat-2.10) bash-4.2$ LD_PRELOAD='/usr/lib64/libstdc++.so.6'

I have tried @doutriaux1 trick, and it does work! Congratulations! The interactive mode in software opengl is still painfully slow (even for such a simple plot), but it's better than nothing

(uvcdat-2.10) bash-4.2$ LD_PRELOAD='/usr/lib64/libstdc++.so.6' python
Python 2.7.13 | packaged by conda-forge | (default, May  2 2017, 12:48:11) 
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np, vcs
>>> i10 = np.identity(10)
>>> x = vcs.init()
>>> x.plot(i10)
<vcs.displayplot.Dp object at 0x2ba0824ea4b0>
>>> x.interact()
/home/share/unix_files/cdat/miniconda2/envs/uvcdat-2.10/lib/python2.7/site-packages/vcs/VTKPlots.py:119: UserWarning: Press 'Q' to exit interactive mode and continue script execution
  "Press 'Q' to exit interactive mode and continue script execution")
>>>

Now, I can't conveniently tell my users to use this trick. Is there a way to make this work out-of-the-box? I could try @doutriaux1 other trick and remove the anaconda2/envs/uvcdat-2.10/lib/libstdc++* files, but do I risk other side effects?

Note that in my case, out system as the same 19 version as uvcdat-2.10. Could the problem come from the libstdc++.so.6.0.21 file?

I have checked where libstdc++ was coming from

(uvcdat-2.10) bash-4.2$ rpm -qf /usr/lib/libstdc++.so.6
libstdc++-4.8.5-11.el7.i686

(uvcdat-2.10) bash-4.2$ rpm -ql libstdc++-4.8.5-11.el7.i686
/usr/lib/libstdc++.so.6
/usr/lib/libstdc++.so.6.0.19
/usr/share/gcc-4.8.2
/usr/share/gcc-4.8.2/python
/usr/share/gcc-4.8.2/python/libstdcxx
/usr/share/gcc-4.8.2/python/libstdcxx/__init__.py
/usr/share/gcc-4.8.2/python/libstdcxx/__init__.pyc
/usr/share/gcc-4.8.2/python/libstdcxx/__init__.pyo
/usr/share/gcc-4.8.2/python/libstdcxx/v6
/usr/share/gcc-4.8.2/python/libstdcxx/v6/__init__.py
/usr/share/gcc-4.8.2/python/libstdcxx/v6/__init__.pyc
/usr/share/gcc-4.8.2/python/libstdcxx/v6/__init__.pyo
/usr/share/gcc-4.8.2/python/libstdcxx/v6/printers.py
/usr/share/gcc-4.8.2/python/libstdcxx/v6/printers.pyc
/usr/share/gcc-4.8.2/python/libstdcxx/v6/printers.pyo
/usr/share/gcc-4.8.5
/usr/share/gdb
/usr/share/gdb/auto-load
/usr/share/gdb/auto-load/usr
/usr/share/gdb/auto-load/usr/lib
/usr/share/gdb/auto-load/usr/lib/libstdc++.so.6.0.19-gdb.py
/usr/share/gdb/auto-load/usr/lib/libstdc++.so.6.0.19-gdb.pyc
/usr/share/gdb/auto-load/usr/lib/libstdc++.so.6.0.19-gdb.pyo

(uvcdat-2.10) bash-4.2$ rpm -qa | grep libstdc++
libstdc++-devel-4.8.5-11.el7.x86_64
libstdc++-4.8.5-11.el7.x86_64
libstdc++-4.8.5-11.el7.i686
@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 23, 2017

@jypeter I will try and see if building all of our package on linux w/o runtime libgcc fixes things.

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 23, 2017

actually esmf needs libgcc at runtime so we won't be able to use the trick... The other possibility is to add to the source activate code. Having some magic bash code that locates the system's stdc++ and add the LD_PRELOAD to the env...

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 23, 2017

@jypeter I'm building a conda version that might work out of the box. When it's built and up can i ask you you to try it as well?

Thanks.

@jypeter

This comment has been minimized.

jypeter commented Aug 23, 2017

Yep, no pb. Just give me the full conda command I should use. I'd rather not lose esmf, now that we have it at last! Though it proves to be harder to use than I expected

This is becoming my longest github issue

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Aug 23, 2017

@jypeter esmf is integrated in cdms though, so it should be transparent to you.

@danlipsa

This comment has been minimized.

Contributor

danlipsa commented Aug 23, 2017

@doutriaux1 Not sure if you found a solution for using a newer gcc on redhat-6. This is the advice that Ben, our build expert give us on Jan 20th.

Ben, our build expert advised using devtoolset-4
https://www.softwarecollections.org/en/scls/rhscl/devtoolset-4/
to compile rather than gcc from conda.
They have a better way to switch the toolset that switches the library paths as well.
They also have a newer compiler.
Not sure if you can install/require this package when installing using conda.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment